Hyperspectral data classification method based on spatial dilation convolution and spectral dilation convolution

By using spatially extended convolution and spectral extension convolution methods in hyperspectral data classification, the spatial and spectral characteristics of hyperspectral images are separated and extracted, and the problem of low classification accuracy of hyperspectral data in the existing technology is solved, and a higher Feature extraction accuracy and classification accuracy.

CN116468906BActive Publication Date: 2024-11-22CHINESE ACAD OF SURVEYING & MAPPING
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202310449506.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-04-24
Publication Date
2024-11-22
Estimated Expiration
2043-04-24

AI Technical Summary

Technical Problem

The existing hyperspectral data classification technology has a small sample data volume, and the two-dimensional convolutional neural network and three-dimensional convolutional neural network have low feature extraction accuracy for hyperspectral features, resulting in low classification accuracy.

Method used

Using the hyperspectral data classification method based on spatially extended convolution and spectrally extended convolution, the first hyperspectral image and the second hyperspectral image are separated, and the spatial feature extraction network and spectral feature extraction are used respectively. The network performs feature extraction, combining feature fusion layer and feature classification network, to improve feature extraction accuracy.

Benefits of technology

It improves the classification accuracy of hyperspectral data, enhances the ability of feature extraction network, and can effectively classify even when the sample data is small, improving the ability to identify hyperspectral data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116468906B_ABST
    Figure CN116468906B_ABST
Patent Text Reader

Abstract

The present application relates to a hyperspectral data classification method, device, computer equipment, storage medium and computer program product based on spatial extended convolution and spectral extended convolution. The method comprises: obtaining an initial hyperspectral image, and determining a first hyperspectral image and a second hyperspectral image according to the initial hyperspectral image; determining a spatial feature map according to the first hyperspectral image and a spatial feature extraction network; the spatial feature extraction network extracts spatial features based on spatial extended convolution; determining a spectral feature map according to the second hyperspectral image and a spectral feature extraction network; the spectral feature extraction network extracts spectral features based on spectral extended convolution; inputting the spatial feature map and the spectral feature map into a feature fusion layer to obtain a fused feature map; inputting the fused feature map into a feature classification network to obtain a classification result corresponding to the initial hyperspectral image. This scheme can improve the feature extraction accuracy, thereby improving the classification accuracy of hyperspectral data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of hyperspectral data technology, and in particular to a hyperspectral data method, device, computer equipment and storage medium. Background Art

[0002] With the development of hyperspectral data technology, hyperspectral data classification technology has emerged, which can classify hyperspectral data so that hyperspectral data has corresponding classification labels. Hyperspectral data classification technology is often used in the field of ground object classification of remote sensing hyperspectral images, that is, to classify the ground objects in remote sensing hyperspectral images so that each ground object contained in the remote sensing hyperspectral images has a corresponding classification label.

[0003] In the traditional hyperspectral data classification technology, a frame of hyperspectral image containing the object to be classified is obtained, and the hyperspectral image is input into the target classification model to obtain the classification map corresponding to the object to be classified.

[0004] The target classification model adopted by the current hyperspectral data classification technology is generally based on a two-dimensional convolutional neural network or a three-dimensional convolutional neural network to extract hyperspectral features. However, when the sample set contains a small amount of sample data, both the two-dimensional convolutional neural network and the three-dimensional convolutional neural network have low feature extraction accuracy for hyperspectral features. Therefore, the classification accuracy of the classification map corresponding to the object to be classified is also low. Summary of the invention

[0005] Based on this, it is necessary to provide a hyperspectral data classification method, device, computer equipment, computer-readable storage medium and computer program product based on spatial extended convolution and spectral extended convolution, which can improve the feature extraction accuracy and thus improve the classification accuracy of hyperspectral data in response to the above technical problems.

[0006] In a first aspect, the present application provides a hyperspectral data classification method based on spatial extension convolution and spectral extension convolution. The method comprises:

[0007] Acquire an initial hyperspectral image, and determine a first hyperspectral image and a second hyperspectral image according to the initial hyperspectral image;

[0008] Determine a spatial feature map according to the first hyperspectral image and a spatial feature extraction network; the spatial feature extraction network extracts spatial features based on spatial dilation convolution;

[0009] Determine a spectral feature graph according to the second hyperspectral image and a spectral feature extraction network; the spectral feature extraction network extracts spectral features based on spectral expansion convolution;

[0010] Inputting the spatial feature map and the spectral feature map into a feature fusion layer to obtain a fused feature map;

[0011] The fused feature map is input into a feature classification network to obtain a classification result corresponding to the initial hyperspectral image.

[0012] In one embodiment, determining the first hyperspectral image and the second hyperspectral image according to the initial hyperspectral image includes:

[0013] Performing principal component analysis on the initial hyperspectral image to obtain a principal component hyperspectral image;

[0014] The initial hyperspectral image is used as the first hyperspectral image, and the main component hyperspectral image is used as the second hyperspectral image; or,

[0015] Using the initial hyperspectral image as the second hyperspectral image, and using the principal component hyperspectral image as the first hyperspectral image; or,

[0016] The principal component hyperspectral image is used as both the first hyperspectral image and the second hyperspectral image.

[0017] In one embodiment, the spatial feature extraction network includes a first input layer and a first residual block, the first residual block includes a spatial convolution network including the spatial dilation convolution and a spatial feature fusion layer; the determining of the spatial feature map according to the first hyperspectral image and the spatial feature extraction network includes:

[0018] Determine a first hyperspectral vector sequence corresponding to the first hyperspectral image according to the first hyperspectral image and the first input layer;

[0019] Inputting the first hyperspectral vector sequence into the spatial convolution network including the spatial dilation convolution to obtain a spatial feature vector sequence;

[0020] The spatial feature vector sequence and the first hyperspectral vector sequence are input into the spatial feature fusion layer to obtain a spatial feature map.

[0021] In one embodiment, the spectral feature extraction network includes a second input layer and a second residual block, the second residual block includes a spectral convolution network including the spectral expansion convolution and a spectral feature fusion layer; the determining of the spectral feature map according to the second hyperspectral image and the spectral feature extraction network includes:

[0022] Determine a second hyperspectral vector sequence corresponding to the second hyperspectral image according to the second hyperspectral image and the second input layer;

[0023] Inputting the second hyperspectral vector sequence into the spectral convolution network including the spectral expansion convolution to obtain a spectral feature vector sequence;

[0024] The spectral feature vector sequence and the second hyperspectral vector sequence are input into the spectral feature fusion layer to obtain a spectral feature map.

[0025] In one embodiment, the feature classification network includes an attention mechanism and a classification network; the step of inputting the fused feature map into the feature classification network to obtain a classification result corresponding to the initial hyperspectral image includes:

[0026] Inputting the fused feature map into the attention mechanism to obtain a fused feature weighted map;

[0027] The fused feature weighted graph is input into the classification network to obtain a classification result corresponding to the initial hyperspectral image.

[0028] In one embodiment, the acquiring an initial hyperspectral image and determining a first hyperspectral image and a second hyperspectral image according to the initial hyperspectral image comprises:

[0029] Acquire multiple frames of initial hyperspectral images containing the target to be classified, and acquire a frame of target hyperspectral image from each of the initial hyperspectral images;

[0030] Determining a first hyperspectral image and a second hyperspectral image according to the target hyperspectral image;

[0031] The step of inputting the fused feature map into a feature classification network to obtain a classification result corresponding to the initial hyperspectral image includes:

[0032] Inputting the fused feature map into a feature classification network to obtain a target classification result corresponding to the target hyperspectral image, wherein the target classification result includes a target classification label corresponding to each pixel in the target hyperspectral image;

[0033] For any pixel point in the target classification result, determine the target feature point of the target to be classified corresponding to the pixel point, and determine the attribute of the target feature point; the attribute of the target feature point includes classified or to be classified;

[0034] In the case where the attribute of the target feature point corresponding to the pixel point is the classified one, the target classification label of the target feature point corresponding to the pixel point is used as the classification label of the pixel point; or

[0035] In the case where the attribute of the target feature point corresponding to the pixel point is the one to be classified, determining whether the target feature point corresponding to the pixel point has appeared in the historical target classification results;

[0036] If the target classification label corresponding to the pixel point has appeared, the target classification label corresponding to the pixel point is used as the classification label corresponding to the pixel point, and the target classification label is used as the classification label of the target feature point, and the attribute of the target feature point corresponding to the pixel point is updated to the classified one; or,

[0037] If the target classification label corresponding to the pixel point does not appear, the step of obtaining a frame of target hyperspectral image from each of the initial hyperspectral images is returned to execution until any pixel point in the new target hyperspectral image has a corresponding classification label.

[0038] In a second aspect, the present application also provides a hyperspectral data classification device based on spatial extension convolution and spectral extension convolution. The device comprises:

[0039] An acquisition module, used for acquiring an initial hyperspectral image, and determining a first hyperspectral image and a second hyperspectral image according to the initial hyperspectral image;

[0040] A spatial feature extraction module, used to determine a spatial feature map according to the first hyperspectral image and a spatial feature extraction network; the spatial feature extraction network performs spatial feature extraction based on spatial dilation convolution;

[0041] A spectral feature extraction module, used to determine a spectral feature map according to the second hyperspectral image and a spectral feature extraction network; the spectral feature extraction network extracts spectral features based on spectral expansion convolution;

[0042] A feature fusion module, used for inputting the spatial feature map and the spectral feature map into a feature fusion layer to obtain a fused feature map;

[0043] The classification module is used to input the fused feature map into a feature classification network to obtain a classification result corresponding to the initial hyperspectral image.

[0044] In one embodiment, the acquisition module is specifically used for:

[0045] Performing principal component analysis on the initial hyperspectral image to obtain a principal component hyperspectral image;

[0046] The initial hyperspectral image is used as the first hyperspectral image, and the main component hyperspectral image is used as the second hyperspectral image; or,

[0047] Using the initial hyperspectral image as the second hyperspectral image, and using the principal component hyperspectral image as the first hyperspectral image; or,

[0048] The principal component hyperspectral image is used as both the first hyperspectral image and the second hyperspectral image.

[0049] In one embodiment, the spatial feature extraction network includes a first input layer and a first residual block, the first residual block includes a spatial convolution network including the spatial dilation convolution and a spatial feature fusion layer; the spatial feature extraction module is specifically used for:

[0050] Determine a first hyperspectral vector sequence corresponding to the first hyperspectral image according to the first hyperspectral image and the first input layer;

[0051] Inputting the first hyperspectral vector sequence into the spatial convolution network including the spatial dilation convolution to obtain a spatial feature vector sequence;

[0052] The spatial feature vector sequence and the first hyperspectral vector sequence are input into the spatial feature fusion layer to obtain a spatial feature map.

[0053] In one embodiment, the spectral feature extraction network includes a second input layer and a second residual block, the second residual block includes a spectral convolution network including the spectral expansion convolution and a spectral feature fusion layer; the spectral feature extraction module is specifically used for:

[0054] Determine a second hyperspectral vector sequence corresponding to the second hyperspectral image according to the second hyperspectral image and the second input layer;

[0055] Inputting the second hyperspectral vector sequence into the spectral convolution network including the spectral expansion convolution to obtain a spectral feature vector sequence;

[0056] The spectral feature vector sequence and the second hyperspectral vector sequence are input into the spectral feature fusion layer to obtain a spectral feature map.

[0057] In one embodiment, the feature classification network includes an attention mechanism and a classification network; the classification module is specifically used for:

[0058] Inputting the fused feature map into the attention mechanism to obtain a fused feature weighted map;

[0059] The fused feature weighted graph is input into the classification network to obtain a classification result corresponding to the initial hyperspectral image.

[0060] In one embodiment, the acquisition module is specifically used for:

[0061] Acquire multiple frames of initial hyperspectral images containing the target to be classified, and acquire a frame of target hyperspectral image from each of the initial hyperspectral images;

[0062] Determining a first hyperspectral image and a second hyperspectral image according to the target hyperspectral image;

[0063] The classification module is specifically used for:

[0064] Inputting the fused feature map into a feature classification network to obtain a target classification result corresponding to the target hyperspectral image, wherein the target classification result includes a target classification label corresponding to each pixel in the target hyperspectral image;

[0065] For any pixel point in the target classification result, determine the target feature point of the target to be classified corresponding to the pixel point, and determine the attribute of the target feature point; the attribute of the target feature point includes classified or to be classified;

[0066] In the case where the attribute of the target feature point corresponding to the pixel point is the classified one, the target classification label of the target feature point corresponding to the pixel point is used as the classification label of the pixel point; or

[0067] In the case where the attribute of the target feature point corresponding to the pixel point is the one to be classified, determining whether the target feature point corresponding to the pixel point has appeared in the historical target classification results;

[0068] If the target classification label corresponding to the pixel point has appeared, the target classification label corresponding to the pixel point is used as the classification label corresponding to the pixel point, and the target classification label is used as the classification label of the target feature point, and the attribute of the target feature point corresponding to the pixel point is updated to the classified one; or,

[0069] If the target classification label corresponding to the pixel point does not appear, the step of obtaining a frame of target hyperspectral image from each of the initial hyperspectral images is returned to execution until any pixel point in the new target hyperspectral image has a corresponding classification label.

[0070] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps described in the first aspect are implemented.

[0071] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps described in the first aspect are implemented.

[0072] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps described in the first aspect are performed.

[0073] The hyperspectral data classification method, device, computer equipment, storage medium and computer program product based on spatial extended convolution and spectral extended convolution obtain an initial hyperspectral image, and determine the first hyperspectral image and the second hyperspectral image according to the initial hyperspectral image; determine the spatial feature map according to the first hyperspectral image and the spatial feature extraction network; the spatial feature extraction network extracts spatial features based on spatial extended convolution; determines the spectral feature map according to the second hyperspectral image and the spectral feature extraction network; the spectral feature extraction network extracts spectral features based on spectral extended convolution; inputs the spatial feature map and the spectral feature map into the feature fusion layer to obtain a fused feature map; inputs the fused feature map into the feature classification network to obtain the classification result corresponding to the initial hyperspectral image. In the above method, the first hyperspectral image and the corresponding second hyperspectral image corresponding to the initial hyperspectral image are determined, and the spatial feature map is determined based on the first hyperspectral image, and the spectral feature map is determined based on the second hyperspectral image. Since the image is separately extracted from each dimension, a feature map that pays more attention to the feature of the dimension can be obtained, that is, the feature extraction accuracy is improved. In addition, compared with the traditional fixed convolution kernel, the extended convolution (spatial extended convolution or spectral extended convolution) has a larger receptive field, which can better focus on global information, thereby improving the feature extraction accuracy. It can be understood that compared with the existing hyperspectral data classification technology, since this scheme improves the feature extraction accuracy of the above feature extraction network (including spatial feature extraction network and spectral feature extraction network), even if the hyperspectral data classification network is trained based on a sample set with a smaller sample data volume (the hyperspectral data classification network of this scheme includes spatial feature extraction network, spectral feature extraction network, feature fusion layer and feature classification network), this scheme can also improve the classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 1 is a flow chart of a hyperspectral data classification method based on spatial extension convolution and spectral extension convolution in one embodiment;

[0075] Figure 2 is a flow chart of a hyperspectral data classification method based on spatial extension convolution and spectral extension convolution in another embodiment;

[0076] Figure 3 A schematic diagram of the structure of a spatial feature extraction network and a spectral feature extraction network in one embodiment;

[0077] Figure 4 A schematic diagram of the structure of a spatial feature extraction network in one embodiment;

[0078] Figure 5 A schematic diagram of the structure of a spectral feature extraction network in one embodiment;

[0079] Figure 6 is a schematic diagram of the structure of an attention mechanism in one embodiment;

[0080] Figure 7 is a schematic flow chart of a method for determining a first hyperspectral image and a second hyperspectral image in one embodiment;

[0081] Figure 8 is a flow chart of a method for determining a classification result corresponding to an initial hyperspectral image in one embodiment;

[0082] Fig. 9 is a structural block diagram of a hyperspectral data classification device based on spatial extension convolution and spectral extension convolution in one embodiment;

[0083] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0084] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0085] In one embodiment, Figure 1 As shown, a hyperspectral data classification method based on spatial expansion convolution and spectral expansion convolution is provided. This embodiment takes the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0086] Step 102: acquiring an initial hyperspectral image, and determining a first hyperspectral image and a second hyperspectral image according to the initial hyperspectral image.

[0087] In an embodiment of the present application, a terminal acquires an initial hyperspectral image containing a target to be classified. Optionally, the number of initial hyperspectral images may be one or more. Specifically, the terminal acquires an initial hyperspectral image containing a target to be classified through a collection device. The collection device is a device that can collect hyperspectral images. In one embodiment, the collection device is a drone equipped with a hyperspectral sensor. The target to be classified is determined according to the actual application, and is not limited in the present application. In one embodiment, the target to be classified may be a ground object. The initial hyperspectral image is an image containing hyperspectral information. In one embodiment, the initial hyperspectral image is a remote sensing hyperspectral image. In one embodiment, the terminal uses the initial hyperspectral image as both the first hyperspectral image and the second hyperspectral image. In another embodiment, the terminal performs dimensionality reduction processing on the initial hyperspectral image to obtain the initial hyperspectral image after dimensionality reduction. The terminal uses the initial hyperspectral image after dimensionality reduction as both the first hyperspectral image and the second hyperspectral image, or uses the initial hyperspectral image after dimensionality reduction as the first hyperspectral image and the initial hyperspectral image as the second hyperspectral image, or uses the initial hyperspectral image after dimensionality reduction as the second hyperspectral image and the initial hyperspectral image as the first hyperspectral image. The dimensionality reduction processing includes, but is not limited to, principal component analysis (PCA), maximum noise fraction (MNF), minimum / maximum autocorrelation factor (MAF), and independent component analysis (ICA).

[0088] Step 104: determining a spatial feature map according to the first hyperspectral image and the spatial feature extraction network.

[0089] Among them, the spatial feature extraction network performs spatial feature extraction based on spatial dilated convolution.

[0090] In the embodiments of the present application, Figure 2 , Figure 3 and Figure 4As shown, the terminal divides the first hyperspectral image into blocks according to a preset first block strategy to obtain a plurality of first hyperspectral sub-images. In one embodiment, the first hyperspectral sub-image is a three-dimensional (3D) image, and the size of the first hyperspectral sub-image is 13×13×band, that is, the first hyperspectral image is divided into blocks in two dimensions of width and height without changing the hyperspectral information corresponding to the plane image formed based on the width and height. The terminal simultaneously inputs all the first hyperspectral sub-images corresponding to the first hyperspectral image into the spatial feature extraction network, and outputs a spatial feature map corresponding to the first hyperspectral image. Among them, the spatial dilated convolution is a three-dimensional dilated convolution, the spatial feature map is composed of multiple spatial feature sub-map sequences, each spatial feature sub-map sequence includes multiple spatial feature sub-maps, the spatial feature sub-map sequence is used to represent the corresponding entire spectral information (corresponding information between wavelength and amplitude), the spatial feature sub-map is used to represent the amplitude information corresponding to a certain image plane information at a certain wavelength, the image plane information includes the image plane size and the position information of the sub-map in the whole image, the sub-map includes the spatial feature sub-map and the spectral feature sub-map, and the whole image includes the first hyperspectral image and the second hyperspectral image. Specifically, in Figure 2 Among them, one of the dimensions is 13×13×B bands (B bands The first hyperspectral sub-image is taken as an example. The processing of multiple first hyperspectral sub-images is similar and will not be described in detail. bands The first hyperspectral sub-image of is input into the spatial feature extraction network, such as Figure 3 As shown, the output is a spatial feature sub-graph sequence of size (7×7, 32b2). Among them, the spatial feature sub-graph sequence includes 32b2 spatial feature sub-graphs, and the image plane size of each spatial feature sub-graph is 7×7. The determination of b2 is related to the size of the spatial expansion convolution, and 32 is related to the size of the convolution in the first input layer of the spatial feature extraction network and the size of the spatial expansion convolution. It can be understood that the example is only for explanation. In actual applications, the specific size of the output results of each network mentioned in this application can be adjusted according to the parameters in each network. Figure 2 , Figure 3 , Figure 4 and Figure 5 The n in (n, m) that appears in the formula represents the size of the output result. Optionally, n can be three-dimensional, such as n1×n2×n3, or two-dimensional, such as n1×n2. The m in (n, m) represents the number of vectors contained in the output result.

[0091] Step 106, determining a spectral feature graph according to the second hyperspectral image and a spectral feature extraction network; the spectral feature extraction network extracts spectral features based on spectral expansion convolution.

[0092] In the embodiments of the present application, Figure 2 , Figure 3 and Figure 5 As shown, the terminal divides the second hyperspectral image into blocks according to a preset second block strategy to obtain a plurality of second hyperspectral sub-image sequences. In one embodiment, the second hyperspectral sub-image is a two-dimensional (2-dimension, 2D) image, and the plane size of the second hyperspectral sub-image in the second hyperspectral sub-image sequence is 13×13. The second hyperspectral sub-image sequence includes B bands The terminal simultaneously inputs all second hyperspectral sub-image sequences corresponding to the second hyperspectral image into the spectral feature extraction network, and outputs a spectral feature map corresponding to the second hyperspectral image. Among them, the spectral expansion convolution is a two-dimensional expansion convolution, and the spectral feature map is composed of multiple spectral feature sub-map sequences, each spectral feature sub-map sequence includes multiple spectral feature sub-maps, and the spectral feature sub-map sequence is used to characterize all spectral information (corresponding information of wavelength and amplitude) corresponding to the same image plane information (including image plane size). The spectral feature sub-map is used to characterize the amplitude information corresponding to a certain image plane information at a certain wavelength. Specifically, in Figure 2 In the example, one of the dimensions is (13×13, B bands ) is used as an example. The processing of multiple second hyperspectral sub-image sequences is similar and will not be described in detail. bands ) The second hyperspectral sub-image sequence is input into the spectral feature extraction network, such as Figure 3 As shown, the output is a spectral feature sub-image sequence of size (7×7, 32). The spectral feature sub-image sequence includes 32 spectral feature sub-images, and the image plane size of each spectral feature sub-image is 7×7. 32 is related to the size of the convolution in the second input layer of the spectral feature extraction network and the size of the spectral expansion convolution.

[0093] Step 108, inputting the spatial feature map and the spectral feature map into the feature fusion layer to obtain a fused feature map.

[0094] In the embodiment of the present application, the terminal inputs the spatial feature map and the spectral feature map into the feature fusion layer, and outputs a fused feature map. Figure 2As shown, the feature fusion layer includes a feature splicing layer, a convolution layer, a rectified linear unit (Linear rectification function, ReLU function) and a batch normalization layer (Batch Normalization, BN layer). In one embodiment, the convolution kernel size of the convolution layer in the feature splicing layer is a 3×3 two-dimensional convolution kernel. The terminal inputs the spatial feature map and the spectral feature map into the feature splicing layer, and implements the alignment splicing in the feature splicing layer, and outputs the spliced ​​feature map. Alignment splicing refers to splicing the spectral information of the spatial feature sub-map sequence and the spectral information of the spectral feature sub-map sequence of the same image plane information. Specifically, taking a spatial feature sub-map sequence 1 with a size of (7×7, cb2) and a spectral feature sub-map sequence 1 with a size of (7×7, c) as an example, the processing process of the spatial feature map containing multiple spatial feature sub-map sequences and the spectral feature map containing multiple spectral feature sub-map sequences is similar, and will not be repeated. The terminal inputs a spatial feature sub-image sequence 1 of size (7×7, cb2) and a spectral feature sub-image sequence 1 of size (7×7, c) into the feature fusion layer, and outputs a spliced ​​feature sub-image sequence 1 of size (7×7, cb2+c). In one embodiment, c is 32. Among them, the spliced ​​feature map includes multiple spliced ​​feature sub-image sequences, and the image plane information corresponding to the spatial feature sub-image sequence 1 and the spectral feature sub-image sequence 1 is the same. The terminal inputs the spliced ​​feature map into the convolution layer in the feature splicing layer to obtain the convolved feature map. The terminal inputs the convolved feature map into the rectified linear unit to obtain the corrected feature map. The terminal inputs the corrected feature map into the batch normalization layer in the feature splicing layer to obtain a fused feature map.

[0095] Step 110, input the fused feature map into a feature classification network to obtain a classification result corresponding to the initial hyperspectral image.

[0096] In an embodiment of the present application, the terminal inputs the fused feature map into a feature classification network, and outputs a classification result corresponding to the initial hyperspectral image. In one embodiment, the classification result is a classification map, wherein any pixel in the classification map corresponds to a classification label, and the types included in the classification labels are pre-set according to actual classification requirements, which are not limited in the present application. The terminal may also pre-store the correspondence between colors and classification labels, the terminal identifies the classification label corresponding to each pixel in the classification map, and determines the target color corresponding to each pixel based on the correspondence between the color and the classification label, and colors the pixels based on the target color to obtain a color classification map corresponding to the classification map. Specifically, Figure 2 shown. Figure 2 1×class means a size of 13×13×B bandsThe first hyperspectral sub-image and a size of (13 × 13, B bands ) The classification result of the second hyperspectral sub-image sequence after being processed by the spatial feature extraction network, the spectral feature extraction network, the feature fusion layer and the feature classification network, where 1×class represents the corresponding 1 classification category (i.e., class).

[0097] In the above-mentioned hyperspectral data classification method based on spatially extended convolution and spectrally extended convolution, the first hyperspectral image and the corresponding second hyperspectral image corresponding to the initial hyperspectral image are determined, and the spatial feature map is determined based on the first hyperspectral image, and the spectral feature map is determined based on the second hyperspectral image. Since the features of the image are extracted separately from each dimension, a feature map that pays more attention to the features of this dimension can be obtained, which improves the feature extraction accuracy. In addition, compared with the traditional fixed convolution kernel, the extended convolution (spatial extended convolution or spectrally extended convolution) has a larger receptive field, so that it can better pay attention to global information, thereby improving the feature extraction accuracy. It can be understood that since this scheme improves the feature extraction accuracy, it also improves the classification accuracy of hyperspectral data.

[0098] In one embodiment, determining a first hyperspectral image and a second hyperspectral image according to an initial hyperspectral image includes:

[0099] The initial hyperspectral image is processed by principal component analysis to obtain a principal component hyperspectral image; the initial hyperspectral image is used as the first hyperspectral image, and the principal component hyperspectral image is used as the second hyperspectral image; or, the initial hyperspectral image is used as the second hyperspectral image, and the principal component hyperspectral image is used as the first hyperspectral image; or, the principal component hyperspectral image is used as both the first hyperspectral image and the second hyperspectral image.

[0100] In an embodiment of the present application, the terminal performs principal component analysis (PCA) on the initial hyperspectral image to obtain a principal component hyperspectral image. Since the principal component analysis can extract the principal components in the initial hyperspectral image, that is, remove the noise and redundant spectral information, the principal component hyperspectral image obtained by the principal component analysis can retain the original spectral information to the maximum extent while achieving dimensionality reduction, thereby improving the data processing efficiency while also ensuring the accuracy of data processing. In one embodiment, the terminal uses the initial hyperspectral image as the first hyperspectral image and the principal component hyperspectral image as the second hyperspectral image. In another embodiment, the terminal uses the initial hyperspectral image as the second hyperspectral image and the principal component hyperspectral image as the first hyperspectral image. In another embodiment, the terminal uses the component hyperspectral image as both the first hyperspectral image and the second hyperspectral image.

[0101] In this embodiment, the initial hyperspectral image is subjected to principal component analysis to obtain a principal component hyperspectral image, and the principal component hyperspectral image is used as the first hyperspectral image and / or the second hyperspectral image. Compared with the initial hyperspectral image, the principal component hyperspectral image achieves dimensionality reduction while retaining the spectral information of the initial hyperspectral image to the maximum extent. Therefore, data processing based on the principal component hyperspectral image can improve processing efficiency while ensuring processing accuracy.

[0102] In one embodiment, the spatial feature extraction network includes a first input layer and a first residual block, the first residual block includes a spatial convolution network including a spatial dilation convolution and a spatial feature fusion layer; according to the first hyperspectral image and the spatial feature extraction network, determining a spatial feature map includes:

[0103] According to the first hyperspectral image and the first input layer, a first hyperspectral vector sequence corresponding to the first hyperspectral image is determined; the first hyperspectral vector sequence is input into a spatial convolution network including a spatial expansion convolution to obtain a spatial feature vector sequence; the spatial feature vector sequence and the first hyperspectral vector sequence are input into a spatial feature fusion layer to obtain a spatial feature map.

[0104] The first input layer is used to vectorize the first hyperspectral image, and the spatial convolutional network is used to extract features of the vectorized first hyperspectral image (ie, the first hyperspectral vector sequence) based on spatial extended convolution.

[0105] In the embodiments of the present application, Figure 3 As shown, the output result of the first input layer is input into the spatial convolution network, and the output result of the first input layer is identically mapped to the spatial feature fusion layer; the output result of the spatial convolution network is input into the spatial feature fusion layer, and is fused with the output result of the first input layer in the spatial feature fusion layer; that is, the spatial convolution network and the spatial feature fusion layer constitute a first residual block. Specifically, the terminal simultaneously inputs all the first hyperspectral sub-images corresponding to the first hyperspectral image into the first input layer, and outputs the first hyperspectral vector sequence corresponding to each first hyperspectral sub-image. The terminal inputs each first hyperspectral vector sequence into the spatial convolution network including the spatial expansion convolution to obtain each spatial feature vector sequence. The terminal inputs each spatial feature vector sequence and each first hyperspectral vector sequence into the spatial feature fusion layer for alignment fusion processing to obtain a spatial feature map including a plurality of spatial feature sub-map sequences. In one embodiment, the spatial convolution network includes a first spatial convolution subnetwork including a first spatial dilation convolution, a second spatial convolution subnetwork including a second spatial dilation convolution, and a third spatial convolution subnetwork including a third spatial dilation convolution, wherein the spatial dilation convolution includes the first spatial dilation convolution, the second spatial dilation convolution, and the third spatial dilation convolution. Specifically, Figure 4As shown. The first spatial convolution subnetwork includes the first spatial convolution layer of the first spatial dilation convolution, the rectified linear unit and the batch normalization layer; the second spatial convolution subnetwork includes the second spatial convolution layer of the second spatial dilation convolution, the rectified linear unit and the batch normalization layer; the third spatial convolution subnetwork includes the third spatial convolution layer of the third spatial dilation convolution, the rectified linear unit and the batch normalization layer. Among them, Figure 4 Here, 32-3×3×3-1 represents the number of output results - the size of the spatial expansion convolution - the expansion rate, that is, 32-3×3×3-1 means that for any first hyperspectral vector sequence, the terminal inputs the first hyperspectral vector sequence into the first spatial convolution layer, and outputs the first hyperspectral vector sequence after convolution containing 32 vectors. Figure 4 The meanings of 32-3×3×3-2 and 32-3×3×3-5 are similar to those of 32-3×3×3-1 above, and are not repeated here. Specifically, the terminal simultaneously inputs all the first hyperspectral sub-images corresponding to the first hyperspectral image into the first input layer, and outputs the first hyperspectral vector sequence corresponding to each first hyperspectral sub-image. The terminal inputs each first hyperspectral vector sequence into the first spatial convolution subnetwork, and outputs each first spatial convolution result (i.e., the first hyperspectral vector sequence after convolution). The terminal inputs each first spatial convolution result into the second spatial convolution subnetwork, and outputs each second spatial convolution result. The terminal inputs each first hyperspectral vector sequence and each second spatial convolution result into the third spatial convolution subnetwork, and outputs each spatial feature vector sequence. The terminal inputs each first hyperspectral vector sequence, each first spatial convolution result, each second spatial convolution result, and each spatial feature vector sequence into the spatial feature fusion layer for alignment fusion processing, and obtains a spatial feature map containing multiple spatial feature submap sequences.

[0106] Specifically, it is shown in the following formula (1).

[0107]

[0108] Where b represents the spatial feature extraction network (or spectral feature extraction network), l represents the lth spatial convolution subnetwork (or lth spectral convolution subnetwork), l is a positive integer, and K b (x) represents the output of the spatial convolutional network (or spectral convolutional network), represents the input result of the (l-1)th spatial convolution subnetwork, F b (x) represents the output of the first input layer (or the second input layer), σ[] represents the activation function linear rectification function (rectified linear unit, ReLU), represents the convolution operation, represents the lth spatial dilation convolution (or spectral dilation convolution), represents the bias in the lth spatial convolution layer (or spectral convolution layer). In one embodiment, l∈{1, 2, 3}, the lth spatial dilation convolution 3×3×3, the lth spectrum expansion convolution is 1×1. Figure 4 In the figure, the expansion rate of the first spatial expansion convolution is 1, the expansion rate of the second spatial expansion convolution is 2, and the expansion rate of the third spatial expansion convolution is 5. These are just examples given in this application. In fact, as long as the expansion rate meets the requirements shown in the following formula (2), it is within the protection scope of this application.

[0109] L i =max[L i+1 -2d i , L i+1 -2(L i+1 -d i ), d i ] Formula (2)

[0110] Among them, L i represents the maximum distance between two non-zero elements in the output feature map of the i-th spatial convolutional subnetwork (or spectral convolutional subnetwork), d i represents the expansion rate of the i-th spatial expansion convolution (or spectral expansion convolution), i is a positive integer, in one embodiment, i∈{1, 2, 3}, where L3=d3. i Satisfying formula (2) can make L2≤K, where K×K×K represents the size of the spatial expansion convolution (or K×K represents the size of the spectral expansion convolution), and the expansion rate d i It meets the requirements.

[0111] In this embodiment, feature extraction of the first hyperspectral vector sequence is performed based on a spatial convolution network including spatial expansion convolution, which can increase the receptive field, so that global information can be better focused, thereby improving the feature extraction accuracy. In addition, since the spatial feature extraction network includes the first residual block, it can be ensured that the original data is not lost during the convolution process, thereby ensuring the feature extraction accuracy. In addition, the spatial feature extraction network focuses on the features in the spatial dimension, while the features in the spectral dimension are extracted by another spectral feature extraction network, thereby avoiding the problem that multi-dimensional feature extraction is placed in the same feature extraction network, resulting in a large number of convolution calculations and reduced operating efficiency.

[0112] In one embodiment, the spectral feature extraction network includes a second input layer and a second residual block, the second residual block includes a spectral convolution network including spectral expansion convolution and a spectral feature fusion layer; according to the second hyperspectral image and the spectral feature extraction network, determining a spectral feature map includes:

[0113] According to the second hyperspectral image and the second input layer, a second hyperspectral vector sequence corresponding to the second hyperspectral image is determined; the second hyperspectral vector sequence is input into a spectral convolution network including spectral expansion convolution to obtain a spectral feature vector sequence; the spectral feature vector sequence and the second hyperspectral vector sequence are input into a spectral feature fusion layer to obtain a spectral feature map.

[0114] The second input layer is used to vectorize the second hyperspectral image, and the spectral convolutional network is used to extract features of the vectorized second hyperspectral image (i.e., the second hyperspectral vector sequence) based on spectral expansion convolution.

[0115] In the embodiments of the present application, Figure 3 As shown, the output result of the second input layer is input into the spectral convolution network, and the output result of the second input layer is identically mapped to the spectral feature fusion layer; the output result of the spectral convolution network is input into the spectral feature fusion layer, and is fused with the output result of the second input layer in the spectral feature fusion layer; that is, the spectral convolution network and the spectral feature fusion layer constitute a second residual block. Specifically, the terminal simultaneously inputs all second hyperspectral sub-images corresponding to the second hyperspectral image into the second input layer, and outputs a second hyperspectral vector sequence corresponding to each second hyperspectral sub-image. The terminal inputs each second hyperspectral vector sequence into a spectral convolution network including spectral expansion convolution to obtain each spectral feature vector sequence. The terminal inputs each spectral feature vector sequence and each second hyperspectral vector sequence into the spectral feature fusion layer for alignment fusion processing to obtain a spectral feature graph containing a plurality of spectral feature sub-graph sequences. In one embodiment, the spectral convolution network includes a first spectral convolution subnetwork including a first spectral extension convolution, a second spectral convolution subnetwork including a second spectral extension convolution, and a third spectral convolution subnetwork including a third spectral extension convolution, wherein the spectral extension convolution includes the first spectral extension convolution, the second spectral extension convolution, and the third spectral extension convolution. Specifically, as Figure 5 As shown. The first spectral convolution subnetwork includes the first spectral convolution layer of the first spectral expansion convolution, the rectified linear unit and the batch normalization layer; the second spectral convolution subnetwork includes the second spectral convolution layer of the second spectral expansion convolution, the rectified linear unit and the batch normalization layer; the third spectral convolution subnetwork includes the third spectral convolution layer of the third spectral expansion convolution, the rectified linear unit and the batch normalization layer. Among them, Figure 5 Here, 32-1×1-1 represents the number of output results - the size of the spectral expansion convolution - the expansion rate, that is, 32-1×1-1 means that for any second hyperspectral vector sequence, the terminal inputs the second hyperspectral vector sequence into the first spectral convolution layer, and outputs a convolved second hyperspectral vector sequence containing 32 vectors. Figure 5The meanings of 32-1×1-2 and 32-1×1-5 are similar to those of 32-1×1-1 above, and are not repeated here. Specifically, the terminal simultaneously inputs all second hyperspectral sub-images corresponding to the second hyperspectral image into the second input layer, and outputs a second hyperspectral vector sequence corresponding to each second hyperspectral sub-image. The terminal inputs each second hyperspectral vector sequence into the first spectral convolution subnetwork, and outputs each first spectral convolution result (i.e., the second hyperspectral vector sequence after convolution). The terminal inputs each first spectral convolution result into the second spectral convolution subnetwork, and outputs each second spectral convolution result. The terminal inputs each first hyperspectral vector sequence and each second spectral convolution result into the third spectral convolution subnetwork, and outputs each spectral feature vector sequence. The terminal inputs each second hyperspectral vector sequence, each first spectral convolution result, each second spectral convolution result, and each spectral feature vector sequence into the spectral feature fusion layer for positional fusion processing, and obtains a spectral feature graph containing multiple spectral feature subgraph sequences. Specifically, as shown in the above formula (1). Figure 5 In the figure, the expansion rate of the first spectral expansion convolution is 1, the expansion rate of the second spectral expansion convolution is 2, and the expansion rate of the third spectral expansion convolution is 5. These are just examples given in the present application. In fact, as long as the expansion rate meets the requirements shown in the above formula (2), it is within the protection scope of the present application.

[0116] In this embodiment, feature extraction of the second high-spectral vector sequence is performed based on a spectral convolution network including spectral expansion convolution, which can increase the receptive field, so that global information can be better focused, thereby improving the feature extraction accuracy. In addition, since the spectral feature extraction network includes a second residual block, it can be ensured that the original data is not lost during the convolution process, thereby ensuring the feature extraction accuracy. In addition, the spectral feature extraction network focuses on the features in the spectral dimension, while the features in the spatial dimension are extracted by another spatial feature extraction network, thereby avoiding the problem that multi-dimensional feature extraction is placed in the same feature extraction network, resulting in a large number of convolution calculations and reducing operating efficiency.

[0117] In one embodiment, the feature classification network includes an attention mechanism and a classification network; the fused feature map is input into the feature classification network to obtain a classification result corresponding to the initial hyperspectral image, including:

[0118] The fused feature map is input into the attention mechanism to obtain the fused feature weighted map; the fused feature weighted map is input into the classification network to obtain the classification result corresponding to the initial hyperspectral image.

[0119] In the embodiments of the present application, Figure 2As shown, the terminal inputs the fused feature map into the attention mechanism, and outputs the fused feature weighted map. The terminal then inputs the fused feature weighted map into the classification network, and outputs the classification result corresponding to the original hyperspectral image. In one embodiment, the attention mechanism is a lightweight attention mechanism. Figure 6 As shown, the attention mechanism includes a global average pooling layer, a concatenation layer including a concat function, a two-dimensional convolution layer, a batch normalization layer, a first hidden layer including an h-swish activation function, a second hidden layer including a split function, a third hidden layer including a sigmoid function, and an output layer including a dot product operation. Specifically, the terminal inputs the fused feature map into the global average pooling layer, and outputs the pooling result in the height dimension and the pooling result in the width dimension. The terminal inputs the pooling result in the height dimension and the pooling result in the width dimension into the concatenation layer including the concat function, and obtains the concatenated pooling result. The terminal inputs the concatenated pooling result into the two-dimensional convolution layer in the attention mechanism, and outputs the first weighted map. The terminal inputs the initial weighted map into the batch normalization layer in the attention mechanism, and outputs the second weighted map. The terminal inputs the second weighted map into the first hidden layer including the h-swish activation function, and outputs the third weighted map. The terminal inputs the third weighted graph into the second hidden layer including the split function, and obtains the third weighted graph in the height dimension and the third weighted graph in the width dimension. The terminal inputs the third weighted graph in the height dimension and the third weighted graph in the width dimension into the output layer including the dot product operation, and outputs the fused feature weighted graph. Figure 2 As shown, the feature classification network includes a batch normalization layer, a rectified linear unit, a (two-dimensional) global average pooling layer, a fully connected layer, and an output layer containing a dropout function.

[0120] In this embodiment, an attention mechanism is added before the classification network, which can pay more attention to more important features and ignore less important features, thereby improving the classification accuracy of subsequent classification processing based on the fused feature weighted graph.

[0121] In one embodiment, Figure 7 As shown, step 102 includes:

[0122] S102-1, obtaining multiple frames of initial hyperspectral images containing targets to be classified, and obtaining a frame of target hyperspectral image from each of the initial hyperspectral images.

[0123] In an embodiment of the present application, the terminal obtains multiple frames of initial hyperspectral images containing the target to be classified, and obtains a frame of target hyperspectral image from each initial hyperspectral image. Optionally, the terminal can randomly obtain a frame of initial hyperspectral image from each initial hyperspectral image as the target hyperspectral image; or, the terminal can also pre-store a screening strategy, and the terminal obtains a frame of target hyperspectral image from each initial hyperspectral image according to the screening strategy. Exemplarily, the screening strategy can be to obtain the initial hyperspectral image with the latest acquisition time among the initial hyperspectral images. It can be understood that the target hyperspectral image obtained each time is different.

[0124] S102-2: Determine a first hyperspectral image and a second hyperspectral image according to the target hyperspectral image.

[0125] In the embodiment of the present application, the terminal determines the first hyperspectral image and the second hyperspectral image according to the target hyperspectral image. The processing of S102-2 is similar to the processing of determining the first hyperspectral image and the second hyperspectral image according to the initial hyperspectral image, and will not be repeated here.

[0126] like Figure 8 As shown, the classification result corresponding to the initial hyperspectral image includes the classification label corresponding to each pixel in the initial hyperspectral image; step 110 includes:

[0127] S110-1, input the fused feature map into the feature classification network to obtain the target classification result corresponding to the target hyperspectral image.

[0128] Among them, the target classification result includes the target classification label corresponding to each pixel point in the target hyperspectral image.

[0129] In the embodiment of the present application, the terminal inputs the fused feature map into the feature classification network, and outputs the target classification result corresponding to the target hyperspectral image. The processing process of the feature classification network on the fused feature map can refer to the above-mentioned specific processing process, which will not be repeated here.

[0130] S110 - 2 For any pixel point in the target classification result, determine the target feature point of the target to be classified corresponding to the pixel point, and determine the attribute of the target feature point.

[0131] The attributes of the target feature points include classified or to be classified.

[0132] In an embodiment of the present application, for any pixel point in the target classification result, the terminal calculates the target feature point of the target to be classified corresponding to the pixel point according to the position information of the pixel point in the rectangular coordinate system of the image, the internal and external orientation elements of the acquisition device, and the position information of the target to be classified in the ground measurement coordinate system. Among them, the acquisition device is a device that collects the initial hyperspectral image corresponding to the target classification result. It can be understood that according to the collinearity of the three points of the photographic center, the pixel point, and the target feature point corresponding to the pixel point corresponding to the acquisition device, the terminal can calculate the feature point on the target to be classified corresponding to the pixel point in the target classification result according to the position information of the photographic center in the ground measurement coordinate system when the acquisition device collects the initial hyperspectral image corresponding to the target classification result, the position information of the pixel point in the target classification result in the ground measurement coordinate system, and the position information of the target to be classified in the ground measurement coordinate system. In this application, how to obtain the position information of the target to be classified in the ground measurement coordinate system is not limited, and it can be obtained by field measurement or by three-dimensional reconstruction based on the initial hyperspectral image containing the target to be classified. The terminal determines the position information of the acquisition device in the ground measurement coordinate system when the acquisition device acquires the initial hyperspectral image corresponding to the target classification result (referred to as the first position information for the convenience of distinction) based on the internal and external orientation elements of the acquisition device and the position information of the pixel points in the rectangular coordinate system of the image, and obtains the position information of the pixel points in the target classification result in the ground measurement coordinate system (referred to as the second position information for the convenience of distinction). The terminal calculates the target feature points of the target to be classified corresponding to the pixel points in the target classification result based on the first position information, the second position information and the position information of the target to be classified in the ground measurement coordinate system. The terminal determines the attributes of the target feature points of the target to be classified corresponding to the pixel points in the target classification result. If the attributes of the corresponding target feature points of the target to be classified are classified, the terminal executes S110-3; if the attributes of the corresponding target feature points of the target to be classified are to be classified, the terminal executes S110-4. It can be understood that if a pixel in the target classification result has no corresponding target feature point on the target to be classified, it means that the pixel is a pixel that does not need to be classified. The terminal can directly set the classification result of the pixel as a label that does not need to be classified, and continue to process the next pixel. Exemplarily, the label that does not need to be classified can be "null". Exemplarily, assuming that the target to be classified is farmland A, if the initial hyperspectral image contains farmland A and part of farmland B, then the pixels of the farmland B part are pixels that do not need to be classified.

[0133] S110 - 3 , when the attribute of the target feature point corresponding to the pixel point is classified, the target classification label of the target feature point corresponding to the pixel point is used as the classification label of the pixel point.

[0134] In the embodiment of the present application, when the attribute of the target feature point corresponding to the pixel point in the target classification result is classified, the terminal uses the target classification label of the target feature point corresponding to the pixel point as the classification label of the pixel point. After executing S110-3, continue to process the next pixel point in the target classification result, that is, jump to S110-2, until all the pixels in the target classification result pass through S110-2, and obtain the classification result corresponding to the current target hyperspectral image.

[0135] S110-4, when the attribute of the target feature point corresponding to the pixel point is to be classified, determine whether the target classification label corresponding to the pixel point has appeared in the historical target classification results of the target feature point corresponding to the pixel point.

[0136] In an embodiment of the present application, when the attribute of the target feature point corresponding to the pixel in the target classification result is to be classified, the terminal determines whether the target classification label corresponding to the pixel has appeared in each historical target classification result of the target feature point corresponding to the pixel. If the target classification label corresponding to the pixel has appeared, S110-5 is executed; if the target classification label corresponding to the pixel does not appear, S110-6 is executed. Exemplarily, it is assumed that the target hyperspectral image at a historical moment includes target hyperspectral image 1 and target hyperspectral image 2, wherein the target classification label corresponding to pixel 1 in target hyperspectral image 1 is target classification label 1, and the target classification label corresponding to pixel 2 in target hyperspectral image 2 is target classification label 2, and pixel 1 and pixel 2 both correspond to target feature point A of the target to be classified, then each historical target classification result of target feature point A includes target classification label 1 corresponding to pixel 1 and target classification label 2 corresponding to pixel 2. The target hyperspectral image at the current moment is target hyperspectral image 3, and pixel 3 in target hyperspectral image 3 corresponds to target feature point A of the target to be classified. If the target classification label corresponding to pixel 3 is target classification label 1, then the target classification label 1 corresponding to pixel 3 has appeared in the historical target classification results {target classification label 1, target classification label 2} of the target feature point A corresponding to pixel 3, and the terminal executes S110-5; if the target classification label corresponding to pixel 3 is target classification label 3, then the target classification label 3 corresponding to pixel 3 has not appeared in the historical target classification results {target classification label 1, target classification label 2} of the target feature point A corresponding to pixel 3, and the terminal executes S110-6.

[0137] S110-5, if the target classification label corresponding to the pixel point has appeared, the target classification label corresponding to the pixel point is used as the classification label corresponding to the pixel point, and the target classification label is used as the classification label of the target feature point, and the attribute of the target feature point corresponding to the pixel point is updated to classified.

[0138] In the embodiment of the present application, if the target classification label corresponding to the pixel point has appeared, the terminal uses the target classification label corresponding to the pixel point as the classification label corresponding to the pixel point, and uses the target classification label of the pixel point as the classification label of the target feature point corresponding to the pixel point, and updates the attribute of the target feature point corresponding to the pixel point to classified. After executing S110-5, continue to process the next pixel point in the target classification label, that is, jump to S110-2, until all the pixels in the target classification label pass through S110-2, and obtain the classification label corresponding to the current target hyperspectral image.

[0139] S110-6, if the target classification label corresponding to the pixel point does not appear, return to the step of acquiring a frame of target hyperspectral image from each initial hyperspectral image until any pixel point in the new target hyperspectral image has a corresponding classification label.

[0140] In the embodiment of the present application, if the target classification label corresponding to the pixel point does not appear, the terminal returns to execute the step of obtaining a frame of target hyperspectral image from each initial hyperspectral image until any pixel point in the new target hyperspectral image has a corresponding classification label, and obtains the classification result corresponding to the current target hyperspectral image.

[0141] In this embodiment, the obtained classification labels are all the same target classification labels, thereby reducing the accuracy error caused by the error of a single classification, that is, improving the classification accuracy.

[0142] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0143] Based on the same inventive concept, the embodiment of the present application also provides a hyperspectral data classification device based on spatially extended convolution and spectrally extended convolution for implementing the hyperspectral data classification method based on spatially extended convolution and spectrally extended convolution mentioned above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of one or more embodiments of the hyperspectral data classification device based on spatially extended convolution and spectrally extended convolution provided below can refer to the limitations of the hyperspectral data classification method based on spatially extended convolution and spectrally extended convolution above, and will not be repeated here.

[0144] In one embodiment, Fig. 9 As shown, a hyperspectral data classification device based on spatial extension convolution and spectral extension convolution is provided, comprising:

[0145] An acquisition module 902 is used to acquire an initial hyperspectral image, and determine a first hyperspectral image and a second hyperspectral image according to the initial hyperspectral image;

[0146] A spatial feature extraction module 904 is used to determine a spatial feature map according to the first hyperspectral image and a spatial feature extraction network; the spatial feature extraction network extracts spatial features based on spatial dilation convolution;

[0147] The spectral feature extraction module 906 is used to determine a spectral feature map according to the second hyperspectral image and the spectral feature extraction network; the spectral feature extraction network extracts spectral features based on spectral expansion convolution;

[0148] A feature fusion module 908 is used to input the spatial feature map and the spectral feature map into the feature fusion layer to obtain a fused feature map;

[0149] The classification module 910 is used to input the fused feature map into the feature classification network to obtain the classification result corresponding to the initial hyperspectral image.

[0150] In one embodiment, the acquisition module 902 is specifically used for:

[0151] Performing principal component analysis on the initial hyperspectral image to obtain a principal component hyperspectral image;

[0152] The initial hyperspectral image is used as the first hyperspectral image, and the principal component hyperspectral image is used as the second hyperspectral image; or,

[0153] The initial hyperspectral image is used as the second hyperspectral image, and the principal component hyperspectral image is used as the first hyperspectral image; or,

[0154] The principal component hyperspectral image is used as both the first hyperspectral image and the second hyperspectral image.

[0155] In one embodiment, the spatial feature extraction network includes a first input layer and a first residual block, the first residual block includes a spatial convolution network including a spatial expansion convolution and a spatial feature fusion layer; the spatial feature extraction module 904 is specifically used for:

[0156] Determine a first hyperspectral vector sequence corresponding to the first hyperspectral image according to the first hyperspectral image and the first input layer;

[0157] Inputting the first hyperspectral vector sequence into a spatial convolutional network including spatial dilation convolution to obtain a spatial feature vector sequence;

[0158] The spatial feature vector sequence and the first hyperspectral vector sequence are input into the spatial feature fusion layer to obtain a spatial feature map.

[0159] In one embodiment, the spectral feature extraction network includes a second input layer and a second residual block, the second residual block includes a spectral convolution network including spectral expansion convolution and a spectral feature fusion layer; the spectral feature extraction module 906 is specifically used for:

[0160] Determine a second hyperspectral vector sequence corresponding to the second hyperspectral image according to the second hyperspectral image and the second input layer;

[0161] Inputting the second hyperspectral vector sequence into a spectral convolution network including spectral expansion convolution to obtain a spectral feature vector sequence;

[0162] The spectral feature vector sequence and the second hyperspectral vector sequence are input into the spectral feature fusion layer to obtain a spectral feature map.

[0163] In one embodiment, the feature classification network includes an attention mechanism and a classification network; the classification module 910 is specifically used to:

[0164] Input the fused feature map into the attention mechanism to obtain the fused feature weighted map;

[0165] The fused feature weighted graph is input into the classification network to obtain the classification result corresponding to the initial hyperspectral image.

[0166] In one embodiment, the acquisition module 902 is specifically used for:

[0167] Acquire multiple frames of initial hyperspectral images containing the target to be classified, and acquire a frame of target hyperspectral image from each of the initial hyperspectral images;

[0168] Determining a first hyperspectral image and a second hyperspectral image according to the target hyperspectral image;

[0169] The classification module 910 is specifically used for:

[0170] The fused feature map is input into the feature classification network to obtain the target classification result corresponding to the target hyperspectral image, and the target classification result includes the target classification label corresponding to each pixel point in the target hyperspectral image;

[0171] For any pixel point in the target classification result, determine the target feature point of the target to be classified corresponding to the pixel point, and determine the attribute of the target feature point; the attribute of the target feature point includes classified or to be classified;

[0172] When the attribute of the target feature point corresponding to the pixel point is classified, the target classification label of the target feature point corresponding to the pixel point is used as the classification label of the pixel point; or

[0173] When the attribute of the target feature point corresponding to the pixel point is to be classified, it is determined whether the target classification label corresponding to the pixel point has appeared in the historical target classification results of the target feature point corresponding to the pixel point;

[0174] If the target classification label corresponding to the pixel point has appeared, the target classification label corresponding to the pixel point is used as the classification label corresponding to the pixel point, and the target classification label is used as the classification label of the target feature point, and the attribute of the target feature point corresponding to the pixel point is updated to be classified; or,

[0175] If the target classification label corresponding to the pixel does not appear, the process returns to the step of obtaining a frame of target hyperspectral image from each initial hyperspectral image until any pixel in the new target hyperspectral image has a corresponding classification label.

[0176] Each module in the hyperspectral data classification device based on spatial expansion convolution and spectral expansion convolution can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0177] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.10As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a hyperspectral data classification method based on spatial expansion convolution and spectral expansion convolution is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0178] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0179] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0180] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0181] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0182] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0183] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0184] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0185] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A hyperspectral data classification method based on spatial extended convolution and spectral extended convolution, characterized in that: The method comprises: Acquire an initial hyperspectral image, and determine a first hyperspectral image and a second hyperspectral image according to the initial hyperspectral image, wherein the first hyperspectral image is the initial hyperspectral image, the second hyperspectral image is obtained by reducing the dimension of the initial hyperspectral image, and the second hyperspectral image is composed of a plurality of second hyperspectral sub-image sequences, each of which is obtained by dividing the second hyperspectral image into blocks, and each second hyperspectral sub-image in the second hyperspectral sub-image sequence is a two-dimensional image; Determine a spatial feature map according to the first hyperspectral image and a spatial feature extraction network; the spatial feature extraction network extracts spatial features based on spatial dilation convolution; Determine a spectral feature graph according to the second hyperspectral image and a spectral feature extraction network; the spectral feature extraction network extracts spectral features based on spectral expansion convolution; Inputting the spatial feature map and the spectral feature map into a feature fusion layer to obtain a fused feature map; Input the fused feature map into the global average pooling layer to obtain a pooling result of the height dimension of the fused feature map and a pooling result of the width dimension of the fused feature map; Input the height dimension pooling result and the width dimension pooling result into a concatenation layer including a concat function to obtain a concatenated pooling result; Input the pooling result into a two-dimensional convolutional layer to obtain a first weighted graph, input the first weighted graph into a batch normalization layer to obtain a second weighted graph, input the second weighted graph into a first hidden layer including an h-swish activation function to obtain a third weighted graph, input the third weighted graph into a second hidden layer including a split function to obtain a third weighted graph in a height dimension and a third weighted graph in a width dimension; Inputting the third weighted graph on the height dimension and the third weighted graph on the width dimension into the output layer to obtain a fused feature weighted graph; The fused feature weighted graph is input into a classification network to obtain a classification result corresponding to the initial hyperspectral image.

2. The method according to claim 1, characterized in that Determining the first hyperspectral image and the second hyperspectral image according to the initial hyperspectral image includes: Performing principal component analysis on the initial hyperspectral image to obtain a principal component hyperspectral image; The initial hyperspectral image is used as a first hyperspectral image, and the principal component hyperspectral image is used as a second hyperspectral image.

3. The method according to claim 1 or 2, characterized in that: The spatial feature extraction network includes a first input layer and a first residual block, wherein the first residual block includes a spatial convolution network including the spatial dilation convolution and a spatial feature fusion layer; and determining a spatial feature map according to the first hyperspectral image and the spatial feature extraction network includes: Determine a first hyperspectral vector sequence corresponding to the first hyperspectral image according to the first hyperspectral image and the first input layer; Inputting the first hyperspectral vector sequence into the spatial convolution network including the spatial dilation convolution to obtain a spatial feature vector sequence; The spatial feature vector sequence and the first hyperspectral vector sequence are input into the spatial feature fusion layer to obtain a spatial feature map.

4. The method according to claim 1 or 2, characterized in that: The spectral feature extraction network includes a second input layer and a second residual block, the second residual block includes a spectral convolution network including the spectral expansion convolution and a spectral feature fusion layer; the determining of the spectral feature map according to the second hyperspectral image and the spectral feature extraction network includes: Determine a second hyperspectral vector sequence corresponding to the second hyperspectral image according to the second hyperspectral image and the second input layer; Inputting the second hyperspectral vector sequence into the spectral convolution network including the spectral expansion convolution to obtain a spectral feature vector sequence; The spectral feature vector sequence and the second hyperspectral vector sequence are input into the spectral feature fusion layer to obtain a spectral feature map.

5. The method according to claim 1, characterized in that The first hyperspectral image is a three-dimensional image.

6. The method according to claim 1, characterized in that The step of acquiring an initial hyperspectral image and determining a first hyperspectral image and a second hyperspectral image according to the initial hyperspectral image includes: Acquire multiple frames of initial hyperspectral images containing the target to be classified, and acquire a frame of target hyperspectral image from each of the initial hyperspectral images; Determining a first hyperspectral image and a second hyperspectral image according to the target hyperspectral image; The step of inputting the fused feature map into a feature classification network to obtain a classification result corresponding to the initial hyperspectral image includes: Inputting the fused feature map into a feature classification network to obtain a target classification result corresponding to the target hyperspectral image, wherein the target classification result includes a target classification label corresponding to each pixel in the target hyperspectral image; For any pixel point in the target classification result, determine the target feature point of the target to be classified corresponding to the pixel point, and determine the attribute of the target feature point; the attribute of the target feature point includes classified or to be classified; In the case where the attribute of the target feature point corresponding to the pixel point is the classified one, the target classification label of the target feature point corresponding to the pixel point is used as the classification label of the pixel point; or In the case where the attribute of the target feature point corresponding to the pixel point is the one to be classified, determining whether the target feature point corresponding to the pixel point has appeared in the historical target classification results; If the target classification label corresponding to the pixel point has appeared, the target classification label corresponding to the pixel point is used as the classification label corresponding to the pixel point, and the target classification label is used as the classification label of the target feature point, and the attribute of the target feature point corresponding to the pixel point is updated to the classified one; or, If the target classification label corresponding to the pixel point does not appear, the step of obtaining a frame of target hyperspectral image from each of the initial hyperspectral images is returned to execution until any pixel point in the new target hyperspectral image has a corresponding classification label.

7. A hyperspectral data classification device based on spatial extension convolution and spectral extension convolution, characterized in that: The device comprises: an acquisition module, configured to acquire an initial hyperspectral image, and determine a first hyperspectral image and a second hyperspectral image according to the initial hyperspectral image, wherein the first hyperspectral image is the initial hyperspectral image, the second hyperspectral image is obtained by reducing the dimension of the initial hyperspectral image, and the second hyperspectral image is composed of a plurality of second hyperspectral sub-image sequences, each of which is obtained by dividing the second hyperspectral image into blocks, and each second hyperspectral sub-image in the second hyperspectral sub-image sequence is a two-dimensional image; A spatial feature extraction module, used to determine a spatial feature map according to the first hyperspectral image and a spatial feature extraction network; the spatial feature extraction network performs spatial feature extraction based on spatial dilation convolution; A spectral feature extraction module, used to determine a spectral feature map according to the second hyperspectral image and a spectral feature extraction network; the spectral feature extraction network extracts spectral features based on spectral expansion convolution; A feature fusion module, used for inputting the spatial feature map and the spectral feature map into a feature fusion layer to obtain a fused feature map; A classification module is used to input the fused feature map into a global average pooling layer to obtain a pooling result of the height dimension of the fused feature map and a pooling result of the width dimension of the fused feature map; input the pooling result of the height dimension and the pooling result of the width dimension into a splicing layer including a concat function to obtain a spliced ​​pooling result; input the pooling result into a two-dimensional convolutional layer to obtain a first weighted map, input the first weighted map into a batch normalization layer to obtain a second weighted map, input the second weighted map into a first hidden layer including an h-swish activation function to obtain a third weighted map, input the third weighted map into a second hidden layer including a split function to obtain a third weighted map in the height dimension and a third weighted map in the width dimension; input the third weighted map in the height dimension and the third weighted map in the width dimension into an output layer to obtain a fused feature weighted map; input the fused feature weighted map into a classification network to obtain a classification result corresponding to the initial hyperspectral image.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • SAR and optical image classification method based on multi-scale attention feature fusion

    CN113420838A

  • Hyperspectral image classification method based on multi-scale spatial-spectral feature joint learning

    CN114842264A

  • Hyperspectral image classification method and device, electronic equipment and storage medium

    CN114926694A

  • Hyperspectral image classification method based on mixed spectral network and multi-head self-attention mechanism

    CN115222994A

  • Hyperspectral remote sensing image classification method based on attention joint network

    CN115564996A