Classification method and apparatus for hyperspectral images

Abstract

The application provides a hyperspectral image classification method and device, and relates to the technical field of image processing. The method comprises the following steps: inputting a hyperspectral image to be classified into a first convolution module in a spectral image classification model, performing dimension reduction processing on the hyperspectral image through the first convolution module, inputting shallow spatial spectral features obtained through dimension reduction into a spatial feature mixing module and a spectral feature mixing module in the spectral image classification model respectively, extracting a multi-scale spatial feature map of the shallow spatial spectral features through the spatial feature mixing module, and extracting a multi-scale spectral feature map of the shallow spatial spectral features through the spectral feature mixing module; inputting a superposition result of the multi-scale spatial feature map and the multi-scale spectral feature map into a second convolution module in the spectral image classification model, performing classification on the superposition result through the second convolution module, and obtaining a classification result of the hyperspectral image. Thus, the accuracy of the classification result can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a hyperspectral image classification method and device. BACKGROUND

[0002] A hyperspectral image is a three-dimensional data composed of hundreds of continuous spectral bands, which contains rich spectral information and can extract the detailed features of different ground objects. The purpose of hyperspectral image classification is to determine a class for each pixel.

[0003] At present, when classifying a hyperspectral image, the traditional method is to first divide the hyperspectral image into multiple image blocks, and then classify the hyperspectral image based on the image block method.

[0004] However, the above image block method limits the understanding of the entire hyperspectral image and cannot model the long-distance dependency in the entire hyperspectral image, resulting in low accuracy of the classification result of the hyperspectral image. SUMMARY

[0005] The present application provides a hyperspectral image classification method and device, which improves the accuracy of the classification result of the hyperspectral image.

[0006] The present application provides a hyperspectral image classification method, which can include:

[0007] Obtaining a hyperspectral image to be classified;

[0008] Inputting the hyperspectral image into a first convolution module in a spectral image classification model, performing dimension reduction processing on the hyperspectral image through the first convolution module, and obtaining shallow spatial spectral features;

[0009] Inputting the shallow spatial spectral features into a spatial feature mixing module and a spectral feature mixing module in the spectral image classification model, respectively, extracting a multi-scale spatial feature map of the shallow spatial spectral features through the spatial feature mixing module, and extracting a multi-scale spectral feature map of the shallow spatial spectral features through the spectral feature mixing module;

[0010] Inputting the superimposition result of the multi-scale spatial feature map and the multi-scale spectral feature map into a second convolution module in the spectral image classification model, classifying the superimposition result through the second convolution module, and obtaining a classification result of the hyperspectral image.

[0011] According to the hyperspectral image classification method provided in the application, the spatial feature mixing module comprises a first normalization layer, a spatial adaptive feature modulation unit, a second normalization layer and a spatial mixer connected in sequence, the multi-scale spatial feature map of the shallow layer spatial spectral feature is extracted through the spatial feature mixing module, and the multi-scale spatial feature map comprises:

[0012] The first normalization layer is used for normalizing the shallow layer spatial spectral feature to obtain a first spatial spectral feature;

[0013] The spatial adaptive feature modulation unit is used for spatial adaptive feature modulation on the first spatial spectral feature to obtain a first spatial feature;

[0014] The second normalization layer is used for normalizing the first product of the shallow layer spatial spectral feature and the first spatial feature to obtain a second spatial feature;

[0015] The spatial mixer is used for spatial mixing processing on the second spatial feature to obtain a third spatial feature, and the superposition result of the third spatial feature and the first product is the multi-scale spatial feature map.

[0016] According to the hyperspectral image classification method provided in the application, the spatial adaptive feature modulation unit comprises a multi-scale spatial feature generation unit, a first convolution layer and a first activation function connected in sequence, the spatial adaptive feature modulation unit is used for spatial adaptive feature modulation on the first spatial spectral feature to obtain a first spatial feature, and the method comprises the following steps:

[0017] The multi-scale spatial feature generation unit is used for extracting a plurality of spatial feature maps corresponding to the first spatial spectral feature;

[0018] The first convolution layer is used for convolution on the splicing result of the plurality of spatial feature maps to obtain a spatial feature map after convolution;

[0019] The first activation function is used for nonlinear transformation on the spatial feature map after convolution to obtain a transformed spatial feature map, and the product of the transformed spatial feature map and the first spatial spectral feature is the first spatial feature.

[0020] According to the hyperspectral image classification method provided in the application, the multi-scale spatial feature generation unit comprises a first deep convolution layer and a plurality of spatial feature generation branches connected in parallel, each spatial feature generation branch comprises an adaptive average pooling layer, a second deep convolution layer and a nearest neighbor up-sampling layer connected in sequence, the adaptive average pooling layers comprised by the spatial feature generation branches are different in pooling resolution, and the multi-scale spatial feature generation unit is used for extracting a plurality of spatial feature maps corresponding to the first spatial spectral feature.

[0021] performing deep convolution on the first spatial spectral feature through the first deep convolution layer, obtaining a first spatial feature map;

[0022] generating a branch for each spatial feature, performing pooling processing on the first spatial spectral feature through the adaptive average pooling layer, obtaining a pooled spatial feature; performing deep convolution on the pooled spatial feature through the second deep convolution layer, obtaining a deep-convoluted spatial feature; performing up-sampling on the deep-convoluted spatial feature through the nearest neighbor up-sampling layer, obtaining a second spatial feature map;

[0023] The plurality of spatial feature maps include the first spatial feature map and the second spatial feature map obtained by the branch for generating each spatial feature.

[0024] According to the hyperspectral image classification method provided in the present application, the spectral feature mixing module includes a third normalization layer, a spectral adaptive feature modulation unit, a fourth normalization layer and a spectral mixer connected in sequence, and the multi-scale spectral feature map of the shallow layer spatial spectral feature is extracted through the spectral feature mixing module, which includes:

[0025] performing normalization processing on the shallow layer spatial spectral feature through the third normalization layer, obtaining a second spatial spectral feature;

[0026] performing spectral adaptive feature modulation on the second spatial spectral feature through the spectral adaptive feature modulation unit, obtaining a first spectral feature;

[0027] performing normalization processing on the second product of the shallow layer spatial spectral feature and the first spectral feature through the fourth normalization layer, obtaining a second spectral feature;

[0028] performing spatial mixing processing on the second spectral feature through the spectral mixer, obtaining a third spectral feature, and the superposition result of the third spectral feature and the second product is the multi-scale spectral feature map.

[0029] According to the hyperspectral image classification method provided in the present application, the spectral adaptive feature modulation unit includes a convolution unit, a second convolution layer and a second activation function connected in sequence, and the spectral adaptive feature modulation on the second spatial spectral feature through the spectral adaptive feature modulation unit, obtaining a first spectral feature, includes:

[0030] performing convolution on the second spatial spectral feature through the convolution unit, obtaining a plurality of spectral feature maps;

[0031] convolve the addition result of the plurality of spectral feature maps through the second convolution layer to obtain a first spectral feature map after convolution;

[0032] perform nonlinear transformation on the first spectral feature map through the second activation function to obtain a transformed spectral feature map, and a product of the transformed spectral feature map and the second spatial spectral feature is the first spectral feature.

[0033] According to the classification method of the hyperspectral image provided in the application, the convolution unit includes a plurality of convolution branches connected in parallel, the convolution kernels of the plurality of convolution branches are different, and the convolution of the second spatial spectral feature through the convolution unit obtains a plurality of spectral feature maps, which includes:

[0034] For each convolution branch, the second spatial spectral feature is convolved through the convolution branch to obtain a second spectral feature map after convolution.

[0035] The plurality of spectral feature maps include the second spectral feature maps obtained by the convolution branches.

[0036] The application further provides a classification device of a hyperspectral image, which can include:

[0037] An acquisition unit is configured to acquire a hyperspectral image to be classified.

[0038] A first processing unit is configured to input the hyperspectral image into a first convolution module in a spectral image classification model, perform dimension reduction processing on the hyperspectral image through the first convolution module, and obtain a shallow layer spatial spectral feature.

[0039] A second processing unit is configured to input the shallow layer spatial spectral feature into a spatial feature mixing module and a spectral feature mixing module in the spectral image classification model respectively, extract a multi-scale spatial feature map of the shallow layer spatial spectral feature through the spatial feature mixing module, and extract a multi-scale spectral feature map of the shallow layer spatial spectral feature through the spectral feature mixing module.

[0040] A third processing unit is configured to input a superimposition result of the multi-scale spatial feature map and the multi-scale spectral feature map into a second convolution module in the spectral image classification model, perform classification on the superimposition result through the second convolution module, and obtain a classification result of the hyperspectral image.

[0041] The application further provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to implement the classification method of the hyperspectral image according to any one of the above.

[0042] The application further provides a non-transitory computer-readable storage medium having stored thereon a computer program, the computer program being executed by a processor to implement the hyperspectral image classification method according to any one of the preceding embodiments.

[0043] The application further provides a computer program product comprising a computer program, the computer program being executed by a processor to implement the hyperspectral image classification method according to any one of the preceding embodiments.

[0044] The hyperspectral image classification method and device provided by the embodiments of the application can input the hyperspectral image to be classified into the first convolution module of the spectral image classification model, perform dimension reduction processing on the hyperspectral image through the first convolution module, input the shallow spatial spectral features obtained through dimension reduction into the spatial feature mixing module and the spectral feature mixing module of the spectral image classification model respectively, extract the multi-scale spatial feature maps of the shallow spatial spectral features through the spatial feature mixing module, and extract the multi-scale spectral feature maps of the shallow spatial spectral features through the spectral feature mixing module; input the superposition result of the multi-scale spatial feature maps and the multi-scale spectral feature maps into the second convolution module of the spectral image classification model, perform classification on the superposition result through the second convolution module, and obtain the classification result of the hyperspectral image. In this way, the whole hyperspectral image is taken as the input of the spectral image classification model, the long-distance dependence in the whole hyperspectral image can be effectively captured, and the accuracy of the classification result can be effectively improved through the fusion of the multi-scale spatial feature maps and the multi-scale spectral feature maps considering the long-distance interaction. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0046] Figure 1 A flowchart of a hyperspectral image classification method provided by the embodiments of the present application is shown in the figure.

[0047] Figure 2 A structural diagram of a spectral image classification model provided by the embodiments of the present application is shown in the figure.

[0048] Figure 3 A structural diagram of a spatial feature mixing module provided by the embodiments of the present application is shown in the figure.

[0049] Figure 4A method flow diagram for extracting a multi-scale spatial feature map of shallow layer spatial spectral features by a spatial feature mixing module is provided for the embodiments of the present application.

[0050] Figure 5 A structural diagram of a spatial adaptive feature modulation unit is provided for the embodiments of the present application.

[0051] Figure 6 A structural diagram of a spatial mixer is provided for the embodiments of the present application.

[0052] Figure 7 A structural diagram of a spectral feature mixing module is provided for the embodiments of the present application.

[0053] Figure 8 A method flow diagram for extracting a multi-scale spectral feature map of shallow layer spatial spectral features by a spectral feature mixing module is provided for the embodiments of the present application.

[0054] Figure 9 A structural diagram of a spectral adaptive feature modulation unit is provided for the embodiments of the present application.

[0055] Figure 10 A structural diagram of a spectral mixer is provided for the embodiments of the present application.

[0056] Figure 11 A structural diagram of a hyperspectral image classification device is provided for the embodiments of the present application.

[0057] Figure 12 An entity structural diagram of an electronic device is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0058] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0059] In the embodiments of the present application, “at least one” means one or more, and “multiple” means two or more. “And / or” describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In the textual description of the present application, the character “ / ” generally represents an “or” relationship between the associated objects before and after it.

[0060] The technical scheme provided in the embodiments of the present application can be applied to a hyperspectral image classification scene. The purpose of hyperspectral image classification is to determine a class for each pixel. Considering that a hyperspectral image carries a large amount of spatial information and spectral information, the classification of the hyperspectral image can be implemented by using the large amount of spatial information and spectral information carried by the hyperspectral image.

[0061] At present, when a hyperspectral image is classified, the traditional method is to first divide the hyperspectral image into a plurality of image blocks, and then the classification of the hyperspectral image is implemented based on the image block method.

[0062] However, the image block method limits the understanding of the entire hyperspectral image, cannot model long-distance dependencies in the entire hyperspectral image, and thus results in low accuracy of the classification result of the hyperspectral image.

[0063] In order to effectively improve the accuracy of the classification result, the embodiments of the present application provide a hyperspectral image classification method. In the following, the hyperspectral image classification method provided by the present application will be described in detail through the following specific embodiments. It can be understood that the following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in some embodiments.

[0064] Figure 1 A flowchart of the hyperspectral image classification method provided in the embodiments of the present application is shown in the figure. The method can be executed by a software and / or a hardware device. For example, as shown in the figure, the hyperspectral image classification method can include the following steps. Figure 1

[0065] S101, acquiring a hyperspectral image to be classified.

[0066] For example, in the embodiments of the present application, when the hyperspectral image to be classified is acquired, a plurality of ways can be included, for example, the hyperspectral image can be directly collected, the hyperspectral image sent by other electronic devices can be received, or the hyperspectral image can be acquired from a third-party database, and the like. The specific setting can be made according to actual needs.

[0067] In the embodiments of the present application, in order to implement the classification of the hyperspectral image, a spectral image classification model can be used. For example, in the embodiments of the present application, the structure of the hyperspectral image classification model can be referred to in the figure. Figure 2 Figure 2 The structure of the spectral image classification model provided in the embodiments of the present application is shown in the figure. The spectral image classification model can include a first convolution module, a spatial feature mixing module, a spectral feature mixing module, and a second convolution module. Therefore, after the hyperspectral image to be classified is acquired, the hyperspectral image can be input into the first convolution module, that is, the following S102 is executed. ​​

[0068] S102, input the hyperspectral image into a first convolution module in the spectral image classification model, perform dimension reduction processing on the hyperspectral image through the first convolution module, and obtain shallow spatial spectral features.

[0069] For example, in the embodiment of the present application, the first convolution module can include a 1x1 convolution layer and a 3x3 convolution layer connected in series, which can be set according to actual needs.

[0070] In the embodiment of the present application, the whole hyperspectral image is taken as the input of the spectral image classification model, so that the long-distance dependence in the whole hyperspectral image can be effectively captured in the process of classifying the hyperspectral image, thereby improving the accuracy of the classification result of the hyperspectral image.

[0071] After the dimension reduction processing of the hyperspectral image through the first convolution module, the shallow spatial spectral features are obtained, and the following S103 can be executed:

[0072] S103, input the shallow spatial spectral features into a spatial feature mixing module and a spectral feature mixing module in the spectral image classification model respectively, extract a multi-scale spatial feature map of the shallow spatial spectral features through the spatial feature mixing module, and extract a multi-scale spectral feature map of the shallow spatial spectral features through the spectral feature mixing module.

[0073] In the embodiment of the present application, the spatial feature mixing module and the spectral feature mixing module are combined to realize the classification of the hyperspectral image together. The spatial feature mixing module is used to extract the multi-scale spatial feature map of the shallow spatial spectral features, and the spectral feature mixing module is used to extract the multi-scale spectral feature map of the shallow spatial spectral features. In this way, the long-distance interaction is considered through the fusion of the multi-scale spatial feature map and the multi-scale spectral feature map, thereby further improving the accuracy of the classification result of the hyperspectral image.

[0074] After the multi-scale spatial feature map and the multi-scale spectral feature map of the shallow spatial spectral features are extracted respectively, the following S104 can be executed:

[0075] S104, input the superimposed result of the multi-scale spatial feature map and the multi-scale spectral feature map into a second convolution module in the spectral image classification model, classify the superimposed result through the second convolution module, and obtain the classification result of the hyperspectral image.

[0076] For example, in the embodiment of the present application, the second convolution module can include a 1x1 convolution layer, which can be set according to actual needs.

[0077] It can be seen that, in the embodiment of the present application, when the hyperspectral image is classified, the hyperspectral image to be classified is input into the first convolution module in the spectral image classification model, the hyperspectral image is processed by the first convolution module to reduce the dimension, the shallow spatial spectral features obtained by the dimension reduction are input into the spatial feature mixing module and the spectral feature mixing module in the spectral image classification model respectively, the multi-scale spatial feature maps of the shallow spatial spectral features are extracted by the spatial feature mixing module, and the multi-scale spectral feature maps of the shallow spatial spectral features are extracted by the spectral feature mixing module; the superimposition result of the multi-scale spatial feature maps and the multi-scale spectral feature maps is input into the second convolution module in the spectral image classification model, the superimposition result is classified by the second convolution module, and the classification result of the hyperspectral image is obtained. In this way, the whole hyperspectral image is taken as the input of the spectral image classification model, the long-distance dependence in the whole hyperspectral image can be effectively captured, and the accuracy of the classification result can be effectively improved by considering the long-distance interaction through the fusion of the multi-scale spatial feature maps and the multi-scale spectral feature maps.

[0078] Based on the above Figure 1 The network structure of the spatial feature mixing module for extracting the multi-scale spatial feature maps in the embodiment of the present application can be seen from Figure 3 , Figure 3 The structure diagram of a spatial feature mixing module provided in the embodiment of the present application, which includes a first normalization layer, a spatial adaptive feature modulation unit, a second normalization layer and a spatial mixer connected in series, the multi-scale spatial feature maps of the shallow spatial spectral features are extracted by the spatial feature mixing module.

[0079] In order to facilitate understanding of how to extract the multi-scale spatial feature maps of the shallow spatial spectral features by the spatial feature mixing module in S103, Figure 3 The following embodiments will be described in detail. Figure 4

[0080] Figure 4 The method flow diagram for extracting the multi-scale spatial feature maps of the shallow spatial spectral features by the spatial feature mixing module provided in the embodiment of the present application can be seen from Figure 4 The method can include:

[0081] S401, normalizing the shallow spatial spectral features by the first normalization layer to obtain first spatial spectral features.

[0082] S402, spatially adaptively modulating the first spatial spectral features by the spatial adaptive feature modulation unit to obtain first spatial features.

[0083] ​For example, in the embodiments of the present application, referring to Figure 5 As shown in the figure, Figure 5 The structure diagram of a spatial adaptive feature modulation unit provided in the embodiments of the present application, which comprises a multi-scale spatial feature generation unit, a first convolution layer and a first activation function connected in series.

[0084] In combination with Figure 5 As shown in the spatial adaptive feature modulation unit, when the spatial adaptive feature modulation unit modulates the first spatial spectral feature, the multi-scale spatial feature generation unit can be used to extract a plurality of spatial feature maps corresponding to the first spatial spectral feature; the first convolution layer can be used to convolve the splicing result of the plurality of spatial feature maps to obtain a convolved spatial feature map; and the first activation function can be used to nonlinearly transform the convolved spatial feature map to obtain a transformed spatial feature map, and the product of the transformed spatial feature map and the first spatial spectral feature is the first spatial feature.

[0085] For example, in the embodiments of the present application, the multi-scale spatial feature generation unit can comprise a first deep convolution layer and a plurality of spatial feature generation branches connected in parallel, wherein each spatial feature generation branch comprises an adaptive average pooling layer, a second deep convolution layer and a nearest neighbor up-sampling layer connected in series, and the adaptive average pooling layers included in each spatial feature generation branch have different pooling resolutions. In this way, through the plurality of spatial feature generation branches with different pooling resolutions, a plurality of spatial feature maps of different scales can be extracted to provide a basis for subsequent acquisition of multi-scale spatial feature maps.

[0086] When the multi-scale spatial feature generation unit extracts a plurality of spatial feature maps corresponding to the first spatial spectral feature, the first deep convolution layer can be used to perform deep convolution on the first spatial spectral feature to obtain a first spatial feature map; for each spatial feature generation branch, the adaptive average pooling layer can be used to perform pooling processing on the first spatial spectral feature to obtain a pooled spatial feature; the second deep convolution layer can be used to perform deep convolution on the pooled spatial feature to obtain a deep-convolved spatial feature; and the nearest neighbor up-sampling layer can be used to up-sample the deep-convolved spatial feature to obtain a second spatial feature map; wherein the plurality of spatial feature maps comprise the first spatial feature map and the second spatial feature maps obtained by each spatial feature generation branch.

[0087] For example, in the embodiments of the present application, the first deep convolution layer can comprise a 3x3 deep convolution.

[0088] Taking three spatial feature generation branches as an example, for example, referring to the above Figure 5As shown, the first spatial feature generation branch can include an adaptive average pooling layer with a pooling resolution of 2, a second depth convolution layer with a 3x3 depth convolution, and a nearest neighbor up-sampling layer; the second spatial feature generation branch can include an adaptive average pooling layer with a pooling resolution of 4, a second depth convolution layer with a 3x3 depth convolution, and a nearest neighbor up-sampling layer; and the third spatial feature generation branch can include an adaptive average pooling layer with a pooling resolution of 8, a second depth convolution layer with a 3x3 depth convolution, and a nearest neighbor up-sampling layer. The actual settings can be made according to actual needs.

[0089] After the plurality of spatial feature maps corresponding to the first spatial spectral feature are extracted by the multi-scale spatial feature generation unit, the first convolution layer can be used to convolve the splicing result of the plurality of spatial feature maps. For example, the first convolution layer can include a 3x3 depth convolution to obtain a convolved spatial feature map. The first activation function can be used to perform nonlinear transformation on the convolved spatial feature map. For example, the first activation function can be a Gaussian Error Linear Unit (GELU) activation function to obtain a transformed spatial feature map. The transformed spatial feature map is multiplied by the first spatial spectral feature to obtain a first spatial feature.

[0090] S403. The first product of the shallow spatial spectral feature and the first spatial feature is normalized by the second normalization layer to obtain a second spatial feature.

[0091] S404. The second spatial feature is spatially mixed by the spatial mixer to obtain a third spatial feature. The superimposition result of the third spatial feature and the first product is a multi-scale spatial feature map.

[0092] For example, in the embodiments of the present application, the structure of the spatial mixer can refer to Figure 6 As shown, Figure 6 A structure diagram of a spatial mixer provided in the embodiments of the present application is shown. The spatial mixer can include a 3x3 depth convolution, a GELU activation function, and another 3x3 depth convolution connected in series. The actual settings can be made according to actual needs. In this embodiment, the spatial mixer is only described by way of example, but it does not mean that the embodiments of the present application are limited to this. Figure 6 As shown,

[0093] In combination with the foregoing description, the spatial adaptive feature modulation unit in the spatial feature mixing module can extract the multi-scale spatial feature map of the hyperspectral image; the spatial mixer in the spatial feature mixing module can capture the local information of the hyperspectral image, thereby effectively improving the accuracy of the obtained multi-scale spatial feature map, so that when the hyperspectral image is classified in combination with the multi-scale spatial feature map subsequently, the accuracy of the classification result can be effectively improved.

[0094] Based on any of the above embodiments, for example, in the embodiments of the present application, the spectral feature mixing module for extracting a multi-scale spectral feature map can refer to Figure 7 , Figure 7 A structural diagram of a spectral feature mixing module provided in the embodiments of the present application is shown in the figure, which includes a third normalization layer, a spectral adaptive feature modulation unit, a fourth normalization layer and a spectral mixer connected in series. The multi-scale spectral feature map of the shallow spatial spectral feature is extracted through the spectral feature mixing module.

[0095] In order to facilitate understanding of how to extract the multi-scale spectral feature map of the shallow spatial spectral feature through the spectral feature mixing module shown in S103 above, the following embodiments will be described in detail. Figure 7 Figure 8

[0096] Figure 8 A method flow diagram for extracting the multi-scale spectral feature map of the shallow spatial spectral feature through the spectral feature mixing module provided in the embodiments of the present application is shown in the figure, for example, which can refer to Figure 8 The method can include:

[0097] S801, normalizing the shallow spatial spectral feature through the third normalization layer to obtain the second spatial spectral feature.

[0098] S802, performing spectral adaptive feature modulation on the second spatial spectral feature through the spectral adaptive feature modulation unit to obtain the first spectral feature.

[0099] For example, in the embodiments of the present application, referring to Figure 9 , Figure 9 A structural diagram of a spectral adaptive feature modulation unit provided in the embodiments of the present application is shown in the figure, which includes a convolution unit, a second convolution layer and a second activation function connected in series.

[0100] In combination with Figure 9 ​​The spectral adaptive feature modulation unit is used for performing spectral adaptive feature modulation on the second spatial spectrum feature, and the spectral adaptive feature modulation includes the following steps: performing convolution on the second spatial spectrum feature by using a convolution unit to obtain a plurality of spectral feature maps; performing convolution on a sum result of the plurality of spectral feature maps by using a second convolution layer to obtain a first spectral feature map after convolution; and performing nonlinear transformation on the first spectral feature map by using a second activation function to obtain a transformed spectral feature map, and a product of the transformed spectral feature map and the second spatial spectrum feature being the first spectral feature.

[0101] For example, in the embodiment of the present application, the convolution unit includes a plurality of convolution branches connected in parallel, the convolution kernels of the plurality of convolution branches are different, and when the convolution unit is used to perform convolution on the second spatial spectrum feature, for each convolution branch, the convolution branch is used to perform convolution on the second spatial spectrum feature to obtain a second spectral feature map after convolution; and the plurality of spectral feature maps include the second spectral feature maps obtained by the convolution branches.

[0102] For example, the convolution unit includes three convolution branches, and for example, refer to the above Figure 9 As shown in the figure, one convolution branch can include a 1x1x3 convolution, one convolution branch can include a 1x1x5 convolution, and one convolution branch can include a 1x1x7 convolution, which can be set according to actual needs.

[0103] After the convolution unit is used to perform convolution on the second spatial spectrum feature to obtain a plurality of spectral feature maps, the second convolution layer is used to perform convolution on a sum result of the plurality of spectral feature maps, for example, the second convolution layer can include a 1x1 depth convolution to obtain a first spectral feature map after convolution; and the second activation function is used to perform nonlinear transformation on the first spectral feature map, for example, the second activation function can be a GELU activation function to obtain a transformed spectral feature map, and the transformed spectral feature map is multiplied by the second spatial spectrum feature to obtain a product, which is the first spectral feature.

[0104] S803, performing normalization processing on the second product of the shallow layer spatial spectrum feature and the first spectral feature by using a fourth normalization layer to obtain a second spectral feature.

[0105] S804, performing spatial mixing processing on the second spectral feature by using a spectral mixer to obtain a third spectral feature, and a superimposition result of the third spectral feature and the second product being a multi-scale spectral feature map.

[0106] For example, in the embodiment of the present application, the structure of the spectral mixer can refer to Figure 10 As shown in the figure, Figure 10This is a schematic diagram of a spectral mixer provided in an embodiment of this application. The spectral mixer may include a 1×1 depthwise convolution, a GELU activation function, and another 1×1 depthwise convolution connected in series. The specific configuration can be adjusted according to actual needs. Here, this embodiment of the application only illustrates the structure. Figure 10 The spectral mixer shown is used as an example for illustration, but it does not mean that the embodiments of this application are limited to this.

[0107] Based on the above description, the spectral adaptive feature modulation unit in the spectral feature mixing module can extract multi-scale spectral feature maps of hyperspectral images; the spectral mixer in the spectral feature mixing module can capture local information of hyperspectral images, thereby effectively improving the accuracy of the obtained multi-scale spectral feature maps. This allows for improved accuracy of classification results when combining multi-scale spectral feature maps to classify hyperspectral images.

[0108] The hyperspectral image classification apparatus provided in this application will be described below. The hyperspectral image classification apparatus described below can be referred to in correspondence with the hyperspectral image classification method described above.

[0109] Figure 11 This application provides a schematic diagram of the structure of a hyperspectral image classification device, as illustrated in the embodiments of this application. For example, please refer to [link to example diagram]. Figure 11 As shown, the hyperspectral image classification device 110 may include:

[0110] Acquisition unit 1101 is used to acquire the hyperspectral image to be classified;

[0111] The first processing unit 1102 is used to input the hyperspectral image into the first convolution module in the spectral image classification model, and perform dimensionality reduction processing on the hyperspectral image through the first convolution module to obtain shallow spatial spectral features.

[0112] The second processing unit 1103 is used to input the shallow spatial spectral features into the spatial feature mixing module and the spectral feature mixing module in the spectral image classification model, respectively, and extract the multi-scale spatial feature map of the shallow spatial spectral features through the spatial feature mixing module and the multi-scale spectral feature map of the shallow spatial spectral features through the spectral feature mixing module.

[0113] The third processing unit 1104 is used to input the superposition result of the multi-scale spatial feature map and the multi-scale spectral feature map into the second convolution module in the spectral image classification model, and to classify the superposition result through the second convolution module to obtain the classification result of the hyperspectral image.

[0114] For example, in the embodiment of the present application, the spatial feature mixing module comprises a first normalization layer, a spatial adaptive feature modulation unit, a second normalization layer and a spatial mixer connected in sequence, the second processing unit 1103 is configured to extract a multi-scale spatial feature map of the shallow spatial spectral feature through the spatial feature mixing module, comprising:

[0115] The first normalization layer is configured to normalize the shallow spatial spectral feature to obtain a first spatial spectral feature;

[0116] The spatial adaptive feature modulation unit is configured to perform spatial adaptive feature modulation on the first spatial spectral feature to obtain a first spatial feature;

[0117] The second normalization layer is configured to normalize a first product of the shallow spatial spectral feature and the first spatial feature to obtain a second spatial feature;

[0118] The spatial mixer is configured to perform spatial mixing processing on the second spatial feature to obtain a third spatial feature, and an overlay result of the third spatial feature and the first product is the multi-scale spatial feature map.

[0119] For example, in the embodiment of the present application, the spatial adaptive feature modulation unit comprises a multi-scale spatial feature generation unit, a first convolution layer and a first activation function connected in sequence, and the second processing unit 1103 is configured to perform spatial adaptive feature modulation on the first spatial spectral feature through the spatial adaptive feature modulation unit to obtain a first spatial feature, comprising:

[0120] The multi-scale spatial feature generation unit is configured to extract a plurality of spatial feature maps corresponding to the first spatial spectral feature;

[0121] The first convolution layer is configured to perform convolution on a splicing result of the plurality of spatial feature maps to obtain a convolution spatial feature map;

[0122] The first activation function is configured to perform nonlinear transformation on the convolution spatial feature map to obtain a transformed spatial feature map, and a product of the transformed spatial feature map and the first spatial spectral feature is the first spatial feature.

[0123] For example, in the embodiment of the present application, the multi-scale spatial feature generation unit includes a first deep convolutional layer and a plurality of spatial feature generation branches connected in parallel, each spatial feature generation branch includes an adaptive average pooling layer, a second deep convolutional layer and a nearest neighbor up-sampling layer connected in series, the adaptive average pooling layers included in each spatial feature generation branch have different pooling resolutions, and the second processing unit 1103 is configured to extract a plurality of spatial feature maps corresponding to the first spatial spectral feature through the multi-scale spatial feature generation unit, including:

[0124] performing deep convolution on the first spatial spectral feature through the first deep convolutional layer to obtain a first spatial feature map;

[0125] for each spatial feature generation branch, performing pooling processing on the first spatial spectral feature through the adaptive average pooling layer to obtain a pooled spatial feature, performing deep convolution on the pooled spatial feature through the second deep convolutional layer to obtain a deep-convoluted spatial feature, and performing up-sampling on the deep-convoluted spatial feature through the nearest neighbor up-sampling layer to obtain a second spatial feature map;

[0126] wherein the plurality of spatial feature maps include the first spatial feature map and the second spatial feature maps obtained by the spatial feature generation branches.

[0127] For example, in the embodiment of the present application, the spectral feature mixing module includes a third normalization layer, a spectral adaptive feature modulation unit, a fourth normalization layer and a spectral mixer connected in series, and the second processing unit 1103 is configured to extract a multi-scale spectral feature map of the shallow spatial spectral feature through the spectral feature mixing module, including:

[0128] performing normalization processing on the shallow spatial spectral feature through the third normalization layer to obtain a second spatial spectral feature;

[0129] performing spectral adaptive feature modulation on the second spatial spectral feature through the spectral adaptive feature modulation unit to obtain a first spectral feature;

[0130] performing normalization processing on the second product of the shallow spatial spectral feature and the first spectral feature through the fourth normalization layer to obtain a second spectral feature;

[0131] performing spatial mixing processing on the second spectral feature through the spectral mixer to obtain a third spectral feature, and the superposition result of the third spectral feature and the second product is the multi-scale spectral feature map.

[0132] For example, in the embodiment of the present application, the spectrum adaptive feature modulation unit includes a convolution unit, a second convolution layer and a second activation function connected in series, the second processing unit 1103 is configured to perform spectrum adaptive feature modulation on the second spatial spectrum feature through the spectrum adaptive feature modulation unit to obtain a first spectrum feature, including:

[0133] The second spatial spectrum feature is convolved through the convolution unit to obtain a plurality of spectrum feature maps;

[0134] The addition result of the plurality of spectrum feature maps is convolved through the second convolution layer to obtain a first spectrum feature map after convolution;

[0135] The first spectrum feature map is nonlinearly transformed through the second activation function to obtain a transformed spectrum feature map, and the product of the transformed spectrum feature map and the second spatial spectrum feature is the first spectrum feature.

[0136] For example, in the embodiment of the present application, the convolution unit includes a plurality of convolution branches connected in parallel, the convolution kernels of the plurality of convolution branches are different, and the second processing unit 1103 is configured to perform convolution on the second spatial spectrum feature through the convolution unit to obtain a plurality of spectrum feature maps, including:

[0137] For each convolution branch, the second spatial spectrum feature is convolved through the convolution branch to obtain a second spectrum feature map after convolution;

[0138] The plurality of spectrum feature maps include the second spectrum feature maps obtained by the convolution branches.

[0139] The hyperspectral image classification device 110 provided in the embodiment of the present application can execute the technical solutions of the hyperspectral image classification method in any of the above embodiments, and has similar implementation principles and beneficial effects to the hyperspectral image classification method. For details, refer to the implementation principles and beneficial effects of the hyperspectral image classification method, which will not be described here.

[0140] Figure 12 An entity structure schematic diagram of an electronic device provided in the embodiment of the present application is as follows: Figure 12As shown, the electronic device can include a processor 1210, a communications interface 1220, a memory 1230, and a communications bus 1240, wherein the processor 1210, the communications interface 1220, and the memory 1230 complete mutual communication through the communications bus 1240. The processor 1210 can invoke a logical instruction in the memory 1230 to execute the hyperspectral image classification method described above, which can include: acquiring a hyperspectral image to be classified; inputting the hyperspectral image into a first convolution module in a spectral image classification model, performing dimension reduction processing on the hyperspectral image through the first convolution module to obtain shallow spatial spectral features; inputting the shallow spatial spectral features into a spatial feature mixing module and a spectral feature mixing module in the spectral image classification model, respectively, extracting a multi-scale spatial feature map of the shallow spatial spectral features through the spatial feature mixing module, and extracting a multi-scale spectral feature map of the shallow spatial spectral features through the spectral feature mixing module; inputting a superposition result of the multi-scale spatial feature map and the multi-scale spectral feature map into a second convolution module in the spectral image classification model, classifying the superposition result through the second convolution module, and obtaining a classification result of the hyperspectral image.

[0141] In addition, the logical instructions in the memory 1230 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0142] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the hyperspectral image classification method described above, which can comprise: acquiring a hyperspectral image to be classified; inputting the hyperspectral image into a first convolution module in a spectral image classification model, performing dimension reduction processing on the hyperspectral image through the first convolution module to obtain shallow spatial spectral features; inputting the shallow spatial spectral features into a spatial feature mixing module and a spectral feature mixing module in the spectral image classification model respectively, extracting multi-scale spatial feature maps of the shallow spatial spectral features through the spatial feature mixing module, and extracting multi-scale spectral feature maps of the shallow spatial spectral features through the spectral feature mixing module; inputting a superimposed result of the multi-scale spatial feature maps and the multi-scale spectral feature maps into a second convolution module in the spectral image classification model, performing classification on the superimposed result through the second convolution module to obtain a classification result of the hyperspectral image.

[0143] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the hyperspectral image classification method described above, which can comprise: acquiring a hyperspectral image to be classified; inputting the hyperspectral image into a first convolution module in a spectral image classification model, performing dimension reduction processing on the hyperspectral image through the first convolution module to obtain shallow spatial spectral features; inputting the shallow spatial spectral features into a spatial feature mixing module and a spectral feature mixing module in the spectral image classification model respectively, extracting multi-scale spatial feature maps of the shallow spatial spectral features through the spatial feature mixing module, and extracting multi-scale spectral feature maps of the shallow spatial spectral features through the spectral feature mixing module; inputting a superimposed result of the multi-scale spatial feature maps and the multi-scale spectral feature maps into a second convolution module in the spectral image classification model, performing classification on the superimposed result through the second convolution module to obtain a classification result of the hyperspectral image.

[0144] The apparatus embodiments described above are only schematic, wherein the units illustrated as separate components can or can not be physically separate, and the components illustrated as units can or can not be physical units, i.e., can be located in one place or distributed on a plurality of network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0145] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and the necessary general hardware platform from the above description of the embodiments, and of course, the embodiments can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that contributes to the technical solutions can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0146] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for classifying hyperspectral images, characterized in that, include: Acquire the hyperspectral image to be classified; The hyperspectral image is input into the first convolution module of the spectral image classification model. The first convolution module performs dimensionality reduction on the hyperspectral image to obtain shallow spatial spectral features. The shallow spatial spectral features are respectively input into the spatial feature mixing module and the spectral feature mixing module in the spectral image classification model. The spatial feature mixing module extracts the multi-scale spatial feature map of the shallow spatial spectral features, and the spectral feature mixing module extracts the multi-scale spectral feature map of the shallow spatial spectral features. The superposition result of the multi-scale spatial feature map and the multi-scale spectral feature map is input into the second convolution module in the spectral image classification model. The superposition result is then classified by the second convolution module to obtain the classification result of the hyperspectral image. The spatial feature mixing module comprises a first normalization layer, a spatial adaptive feature modulation unit, a second normalization layer, and a spatial mixer connected in series. The extraction of the multi-scale spatial feature map of the shallow spatial spectral features through the spatial feature mixing module includes: normalizing the shallow spatial spectral features using the first normalization layer to obtain a first spatial spectral feature; performing spatial adaptive feature modulation on the first spatial spectral feature using the spatial adaptive feature modulation unit to obtain a first spatial feature; normalizing the first product of the shallow spatial spectral feature and the first spatial feature using the second normalization layer to obtain a second spatial feature; and performing spatial mixing on the second spatial feature using the spatial mixer to obtain a third spatial feature. The superposition of the third spatial feature and the first product results in the multi-scale spatial feature map.

2. The method according to claim 1, characterized in that, The spatial adaptive feature modulation unit includes a multi-scale spatial feature generation unit, a first convolutional layer, and a first activation function connected in series. The step of performing spatial adaptive feature modulation on the first spatial spectral features through the spatial adaptive feature modulation unit to obtain the first spatial features includes: The multi-scale spatial feature generation unit extracts multiple spatial feature maps corresponding to the first spatial spectral features. The first convolutional layer convolves the concatenated spatial feature maps to obtain the convolved spatial feature map. The convolutional spatial feature map is nonlinearly transformed by the first activation function to obtain the transformed spatial feature map. The product of the transformed spatial feature map and the first spatial spectral feature is the first spatial feature.

3. The method according to claim 2, characterized in that, The multi-scale spatial feature generation unit includes a first deep convolutional layer connected in parallel and multiple spatial feature generation branches. Each spatial feature generation branch includes an adaptive average pooling layer, a second deep convolutional layer, and a nearest neighbor upsampling layer connected in series. The pooling resolution of the adaptive average pooling layer in each spatial feature generation branch is different. The step of extracting multiple spatial feature maps corresponding to the first spatial spectral features through the multi-scale spatial feature generation unit includes: The first spatial spectral features are subjected to deep convolution through the first deep convolutional layer to form a first spatial feature map. For each spatial feature generation branch, the first spatial spectral feature is pooled through the adaptive average pooling layer to obtain the pooled spatial feature; the pooled spatial feature is then depthwise convolved through the second depthwise convolution layer to obtain the depthwise convolutional spatial feature; and the depthwise convolutional spatial feature is then upsampled through the nearest neighbor upsampling layer to obtain the second spatial feature map. The plurality of spatial feature maps include the first spatial feature map and the second spatial feature map obtained from each spatial feature generation branch.

4. The method according to any one of claims 1-3, characterized in that, The spectral feature mixing module comprises a third normalization layer, a spectral adaptive feature modulation unit, a fourth normalization layer, and a spectral mixer, connected in series. The extraction of the multi-scale spectral feature map of the shallow spatial spectral features through the spectral feature mixing module includes: The shallow spatial spectral features are normalized by the third normalization layer to obtain the second spatial spectral features; The second spatial spectral feature is subjected to spectral adaptive feature modulation by the spectral adaptive feature modulation unit to obtain the first spectral feature; The second spectral feature is obtained by normalizing the second product of the shallow spatial spectral feature and the first spectral feature through the fourth normalization layer. The second spectral feature is spatially mixed using the spectral mixer to obtain a third spectral feature. The superposition of the third spectral feature and the product of the second spectral feature is the multi-scale spectral feature map.

5. The method according to claim 4, characterized in that, The spectral adaptive feature modulation unit includes a convolutional unit, a second convolutional layer, and a second activation function connected in series. The step of performing spectral adaptive feature modulation on the second spatial spectral feature through the spectral adaptive feature modulation unit to obtain the first spectral feature includes: The second spatial spectral features are convolved by the convolution unit to obtain multiple spectral feature maps; The sum of the multiple spectral feature maps is convolved by the second convolutional layer to obtain the first convolutional spectral feature map; The first spectral feature map is nonlinearly transformed by the second activation function to obtain the transformed spectral feature map. The product of the transformed spectral feature map and the second spatial spectral feature is the first spectral feature.

6. The method according to claim 5, characterized in that, The convolutional unit includes multiple convolutional branches connected in parallel, each branch having a different convolutional kernel. The second spatial spectral feature is convolved by the convolutional unit to obtain multiple spectral feature maps, including: For each convolution branch, the second spatial spectral features are convolved through the convolution branch to obtain the convolved second spectral feature map; The plurality of spectral feature maps include the second spectral feature map obtained from each of the convolutional branches.

7. A hyperspectral image classification device, characterized in that, include: The acquisition unit is used to acquire the hyperspectral image to be classified. The first processing unit is used to input the hyperspectral image into the first convolution module in the spectral image classification model, and to perform dimensionality reduction processing on the hyperspectral image through the first convolution module to obtain shallow spatial spectral features; The second processing unit is used to input the shallow spatial spectral features into the spatial feature mixing module and the spectral feature mixing module in the spectral image classification model, respectively, and extract the multi-scale spatial feature map of the shallow spatial spectral features through the spatial feature mixing module and the multi-scale spectral feature map of the shallow spatial spectral features through the spectral feature mixing module. The third processing unit is used to input the superposition result of the multi-scale spatial feature map and the multi-scale spectral feature map into the second convolution module in the spectral image classification model, and to classify the superposition result through the second convolution module to obtain the classification result of the hyperspectral image; The spatial feature mixing module comprises a first normalization layer, a spatial adaptive feature modulation unit, a second normalization layer, and a spatial mixer connected in series. Regarding the extraction of the multi-scale spatial feature map of the shallow spatial spectral features through the spatial feature mixing module, the second processing unit is specifically used to: normalize the shallow spatial spectral features through the first normalization layer to obtain a first spatial spectral feature; perform spatial adaptive feature modulation on the first spatial spectral feature through the spatial adaptive feature modulation unit to obtain a first spatial feature; and normalize the first product of the shallow spatial spectral feature and the first spatial feature through the second normalization layer to obtain a second spatial feature. The second spatial feature is spatially mixed by the spatial mixer to obtain a third spatial feature. The superposition result of the third spatial feature and the first product is the multi-scale spatial feature map.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the hyperspectral image classification method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the hyperspectral image classification method as described in any one of claims 1 to 6.

Images