Image classification method and device and readable storage medium

Abstract

The invention relates to an image classification method and device and a readable storage medium. The method comprises the steps of obtaining a to-be-processed hyperspectral image; performing convolution processing on the hyperspectral image by using a residual convolution module of a pre-trained image classification network model to obtain a first image feature; using a channel attention module of the image classification network model to perform channel attention weighting processing on the first image feature to obtain a second image feature; and based on the second image feature, predicting a classification result of a candidate object in the hyperspectral image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image classification method, device and readable storage medium. Background Art

[0002] In recent years, with the development of big data technology and high-performance computing equipment, convolutional neural networks (CNN) have become an important image processing method.

[0003] In related technologies, due to the characteristics of high dimensionality, high correlation, and high noise of hyperspectral images, the computational complexity of image feature extraction of hyperspectral images using convolutional neural networks is high, and the classification effect of hyperspectral images is not ideal. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides an image classification method, device and readable storage medium.

[0005] In a first aspect of the present disclosure, there is provided an image classification method, comprising:

[0006] Acquire a hyperspectral image to be processed;

[0007] For the hyperspectral image, a residual convolution module of a pre-trained image classification network model is used to perform convolution processing to obtain a first image feature;

[0008] Using the channel attention module of the image classification network model, performing channel attention weighted processing on the first image feature to obtain a second image feature;

[0009] Based on the second image feature, a classification result of the candidate object in the hyperspectral image is predicted.

[0010] Optionally, the residual convolution module includes at least two convolution layers, and the at least two convolution layers are cascaded; wherein the output of any one of the at least two convolution layers is transmitted to the output of another convolution layer through a residual unit;

[0011] The residual convolution module is used to fuse image features at different levels in the hyperspectral image to obtain the first image feature.

[0012] Optionally, the at least two convolutional layers include:

[0013] A first depth-separable convolution layer, a second depth-separable convolution layer and a third depth-separable convolution layer; wherein the output of the first depth-separable convolution layer is transmitted to the output of the third depth-separable convolution layer through a residual unit.

[0014] Optionally, the image classification network model includes:

[0015] At least one of the channel attention modules comprises:

[0016] The spatial compression submodule is used to compress the spatial dimension of the input image features;

[0017] The channel feature extraction submodule is used to extract the channel attention features of each channel from the compressed image features;

[0018] The fusion submodule is used to perform weighted processing on the input image features based on the channel attention features of each channel to obtain the second image features.

[0019] Optionally, after acquiring the hyperspectral image to be processed, the method further includes:

[0020] Preprocessing the hyperspectral image;

[0021] Based on the preprocessed hyperspectral image, a plurality of three-dimensional neighborhood block data corresponding to the hyperspectral image is obtained; wherein the central pixel points corresponding to the plurality of three-dimensional neighborhood block data are different;

[0022] The method of performing convolution processing on the hyperspectral image using a residual convolution module of a pre-trained image classification network model to obtain a first image feature includes:

[0023] The plurality of three-dimensional neighborhood block data corresponding to the hyperspectral image are input into a residual convolution module in the image classification network model to obtain the first image feature.

[0024] Optionally, the preprocessing of the hyperspectral image includes:

[0025] Acquiring band data of the hyperspectral image in multiple bands;

[0026] Normalizing the band data of multiple bands of the hyperspectral image;

[0027] Re-determine the hyperspectral image based on the band data of the multiple bands after normalization;

[0028] The redundant band information in the hyperspectral image is removed.

[0029] Optionally, before acquiring the hyperspectral image to be processed, the method further includes:

[0030] Obtaining a training sample set and a classification label corresponding to the training sample set;

[0031] Inputting a plurality of training images in the training sample set into an initial network model to be trained, and obtaining a prediction classification result output by the initial network model;

[0032] Determining a loss function value of the initial network model according to the predicted classification result and the classification label;

[0033] Based on the loss function value, the parameters to be trained of the initial network model are optimized to obtain the image classification network model.

[0034] Optionally, the optimizing the parameters to be trained of the initial network model based on the loss function value to obtain the image classification network model includes:

[0035] Based on the loss function value, an adaptive moment estimation optimization algorithm is used to optimize the parameters to be trained of the initial network model to obtain the image classification network model.

[0036] Optionally, the method further includes:

[0037] Obtaining a test sample set and a classification label corresponding to the test sample set;

[0038] Inputting a plurality of test images in the test sample set into the trained image classification network model to obtain a predicted classification result output by the image classification network model;

[0039] Based on the predicted classification result and the classification label, a classification accuracy test is performed on the trained image classification network model to obtain the classification accuracy of the image classification network model.

[0040] Optionally, the classification accuracy includes at least one of the following:

[0041] Overall classification accuracy, average classification accuracy and Kappa coefficient.

[0042] A second aspect of the present disclosure provides an image classification device, comprising:

[0043] An acquisition module, used for acquiring a hyperspectral image to be processed;

[0044] A first processing module is used to perform convolution processing on the hyperspectral image using a residual convolution module of a pre-trained image classification network model to obtain a first image feature;

[0045] A second processing module is used to use the channel attention module of the image classification network model to perform channel attention weighted processing on the first image feature to obtain a second image feature;

[0046] A prediction module is used to predict the classification result of the candidate object in the hyperspectral image based on the second image feature.

[0047] Optionally, the residual convolution module includes at least two convolution layers, and the at least two convolution layers are cascaded; wherein the output of any one of the at least two convolution layers is transmitted to the output of another convolution layer through a residual unit;

[0048] The residual convolution module is used to fuse image features at different levels in the hyperspectral image to obtain the first image feature.

[0049] Optionally, the at least two convolutional layers include:

[0050] A first depth-separable convolution layer, a second depth-separable convolution layer and a third depth-separable convolution layer; wherein the output of the first depth-separable convolution layer is transmitted to the output of the third depth-separable convolution layer through a residual unit.

[0051] Optionally, the image classification network model includes:

[0052] At least one of the channel attention modules comprises:

[0053] The spatial compression submodule is used to compress the spatial dimension of the input image features;

[0054] The channel feature extraction submodule is used to extract the channel attention features of each channel from the compressed image features;

[0055] The fusion submodule is used to perform weighted processing on the input image features based on the channel attention features of each channel to obtain the second image features.

[0056] Optionally, the image classification device further includes:

[0057] A preprocessing module, used for preprocessing the hyperspectral image;

[0058] A third processing module is used to obtain a plurality of three-dimensional neighborhood block data corresponding to the hyperspectral image based on the preprocessed hyperspectral image; wherein the central pixel points corresponding to the plurality of three-dimensional neighborhood block data are different;

[0059] The first processing module is further used to input the plurality of three-dimensional neighborhood block data corresponding to the hyperspectral image into the residual convolution module in the image classification network model to obtain the first image feature.

[0060] Optionally, the preprocessing module is further used to obtain band data of the hyperspectral image in multiple bands;

[0061] Normalizing the band data of multiple bands of the hyperspectral image;

[0062] Re-determine the hyperspectral image based on the band data of the multiple bands after normalization;

[0063] The redundant band information in the hyperspectral image is removed.

[0064] Optionally, the image classification device further includes:

[0065] A fourth processing module, used to obtain a training sample set and a classification label corresponding to the training sample set;

[0066] Inputting a plurality of training images in the training sample set into an initial network model to be trained, and obtaining a prediction classification result output by the initial network model;

[0067] Determining a loss function value of the initial network model according to the predicted classification result and the classification label;

[0068] Based on the loss function value, the parameters to be trained of the initial network model are optimized to obtain the image classification network model.

[0069] Optionally, the fourth processing module is further used to:

[0070] Based on the loss function value, an adaptive moment estimation optimization algorithm is used to optimize the parameters to be trained of the initial network model to obtain the image classification network model.

[0071] Optionally, the fourth processing module is further used to:

[0072] Obtaining a test sample set and a classification label corresponding to the test sample set;

[0073] Inputting a plurality of test images in the test sample set into the trained image classification network model to obtain a predicted classification result output by the image classification network model;

[0074] Based on the predicted classification result and the classification label, a classification accuracy test is performed on the trained image classification network model to obtain the classification accuracy of the image classification network model.

[0075] Optionally, the classification accuracy includes at least one of the following:

[0076] Overall classification accuracy, average classification accuracy and Kappa coefficient.

[0077] According to a third aspect of the present disclosure, there is provided an image classification device, the device comprising:

[0078] a memory for storing processor-executable instructions;

[0079] A processor connected to the memory;

[0080] Wherein, the processor is configured to execute the image classification method as described in any embodiment of the first aspect of the present disclosure.

[0081] According to a fourth aspect of the present disclosure, a non-temporary computer-readable storage medium is provided. When the instructions in the storage medium are executed by a processor of an image classification device, the image classification device can perform the image classification method as described in any one of the embodiments of the first aspect of the present disclosure.

[0082] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0083] In the disclosed embodiment, by acquiring a hyperspectral image to be processed, and performing convolution processing on the hyperspectral image using a residual convolution module of a pre-trained image classification network model, and performing channel attention weighted processing using a channel attention module of the image classification network model, deep-level image features associated with the channel attention weights can be obtained, and feature information corresponding to multiple different bands of the hyperspectral image can be more effectively extracted, thereby reducing the computational complexity of the hyperspectral image and improving the classification accuracy of the hyperspectral image.

[0084] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0086] Figure 1 The following is a flow chart of an image classification method according to an exemplary embodiment of the present disclosure. Figure 1 .

[0087] Figure 2 The figure is a schematic diagram of an application scenario of an image classification method according to an exemplary embodiment of the present disclosure.

[0088] Figure 3 It is a structural diagram of a residual convolution module according to an exemplary embodiment of the present disclosure.

[0089] Figure 4 The following is a flow chart of an image classification method according to an exemplary embodiment of the present disclosure. Figure 2 .

[0090] Figure 5The following is a flow chart of an image classification method according to an exemplary embodiment of the present disclosure. Figure 3 .

[0091] Figure 6 It is a structural diagram of an image classification network model according to an exemplary embodiment of the present disclosure.

[0092] Figure 7 The following is a flow chart of an image classification method according to an exemplary embodiment of the present disclosure. Figure 4 .

[0093] Figure 8 It is a structural schematic diagram of an image classification device according to an exemplary embodiment of the present disclosure.

[0094] Fig. 9 It is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0095] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices consistent with some aspects of the present disclosure as detailed in the appended claims.

[0096] In the research on hyperspectral image classification based on convolutional neural networks, the classification effect of hyperspectral images is not ideal. The mainstream deep learning model is still driven by a large amount of supervised data, which makes the model classification ability dependent on the quantity and quality of supervised data. It is not only computationally complex, but also prone to small sample problems. It can be seen that better classification calculation methods are needed to improve the current problems in hyperspectral image classification tasks.

[0097] The present invention provides an image classification method. Figure 1 The following is a flow chart of an image classification method according to an exemplary embodiment of the present disclosure. Figure 1 ,like Figure 1 As shown, the image classification method comprises:

[0098] Step S101, obtaining a hyperspectral image to be processed;

[0099] Step S102, for the hyperspectral image, performing convolution processing using a residual convolution module of a pre-trained image classification network model to obtain a first image feature;

[0100] Step S103, using the channel attention module of the image classification network model, performing channel attention weighted processing on the first image feature to obtain a second image feature;

[0101] Step S104: predicting classification results of candidate objects in the hyperspectral image based on the second image features.

[0102] In the embodiment of the present disclosure, a hyperspectral image refers to a spectral resolution of 10 -2 Spectral image (image) within the range of λ. Compared with traditional panchromatic images or red, green and blue (RGB) images, hyperspectral images have more spectral bands, such as visible light bands and short-wave infrared bands.

[0103] In other words, hyperspectral images are equivalent to three-dimensional stereo images. Unlike the single channel of black and white images and the red, green and blue (RGB) three channels of color images, the number of channels of hyperspectral images can be determined by the number of bands. The length and width of hyperspectral images are the same as normal images, but they contain band information corresponding to dozens or even hundreds of bands.

[0104] For example, assuming that a hyperspectral imaging is performed on an object, 200*200 pixels are used and 100 bands are selected, then the hyperspectral image finally formed by imaging is a three-dimensional stereo image of 200*200*100.

[0105] In step S102, for a hyperspectral image containing information of multiple bands, a residual convolution module in a pre-trained image classification network model may be used to perform convolution processing on the hyperspectral image to obtain a first image feature.

[0106] The image classification network model may be a convolutional neural network for classifying candidate objects in a hyperspectral image, and the residual convolution module may be a processing module in the convolutional neural network that can implement residual connections.

[0107] It is understandable that the basic idea of ​​residual connection is to directly connect the input signal to the output in certain layers of the convolutional neural network, thereby introducing "cross-layer connection" in the convolutional neural network. Based on this "cross-layer connection", the convolutional neural network can moderately increase the depth of the model, tap deeper features, and alleviate the overfitting of deep convolutional neural networks, thereby improving the expressiveness and performance of the model.

[0108] The first image feature at least includes: a brightness feature and a classification feature, wherein the brightness feature is used to indicate the grayscale of the hyperspectral image; and the classification feature is used to indicate the granularity of the feature of the hyperspectral image.

[0109] It should be noted that the above is only an exemplary listing of the image features of the embodiment of the present disclosure and does not constitute a limitation on the image features of the embodiment of the present disclosure.

[0110] In some embodiments, the image classification network model may include multiple residual convolution modules, wherein each residual convolution module is used to perform residual processing on the data corresponding to each pixel in the hyperspectral image. Here, the data corresponding to each pixel includes data corresponding to the same pixel in different bands. The residual convolution module can obtain the image features of each pixel of the hyperspectral image to form a first image feature.

[0111] It should be noted that a residual convolution module may also be used to perform residual processing on the data corresponding to all pixels, and there is no specific limitation on this.

[0112] In some embodiments, each of the multiple residual convolution modules in the image classification network model is used to perform residual processing on the band information of a band to obtain image features of the hyperspectral image in multiple different bands, and the image features of the multiple different bands constitute the first image feature.

[0113] In step S103, multiple channel attention modules in the image classification network model can be used to perform channel attention weighted processing on the first image feature obtained in step S102 to obtain a second image feature.

[0114] It can be understood that the image classification network model contains multiple convolution kernels, each of which corresponds to a feature channel. Based on the channel attention mechanism, the weight parameters of different feature channels can be obtained. After obtaining the weight parameters of different feature channels, the feature information of different channels in the first image feature can be weighted, so as to achieve different degrees of strengthening or suppression of the feature information of different channels in the first image feature according to the importance of different feature channels.

[0115] That is to say, based on the channel attention mechanism, the second image features after channel attention weighted processing can be obtained, so as to further extract the image features of the first feature image, so that the image classification network model has better performance in extracting features and stronger generalization ability.

[0116] In some embodiments, each band in a hyperspectral image having multiple bands may correspond to a feature channel. Different feature channels corresponding to different bands have different weight parameters, so that the contribution of different bands to classification accuracy can be adjusted.

[0117] In step S104, based on the second image feature acquired in step S103, a corresponding feature vector can be obtained; further based on the feature vector, a classification result of the candidate object in the hyperspectral image can be predicted.

[0118] The candidate object may be an image region corresponding to one or more pixels in the hyperspectral image.

[0119] It can be understood that by extracting the second image feature, a feature vector corresponding to the second image feature can be constructed, and then the feature vector can be convolved to obtain a classification result corresponding to the feature vector. The classification result corresponding to the feature vector is the classification result of the corresponding pixel point data, or it can be the classification result of the candidate object corresponding to the pixel point data.

[0120] It can be understood that the input to the image classification network model is a hyperspectral image with multiple bands. Then, the second image features obtained by the image classification network model also include feature information corresponding to the multiple bands. Therefore, when predicting the classification results of candidate objects in the hyperspectral image based on the second image features, the band information of multiple bands can be fully considered, thereby improving the accuracy of classifying candidate objects in the hyperspectral image.

[0121] Next, an applicable scenario of the image classification method is introduced in combination with steps S101 to S104.

[0122] Figure 2 is a schematic diagram of an application scenario of an image classification method according to an exemplary embodiment of the present disclosure. Figure 2 As shown, the hyperspectral image can be a hyperspectral remote sensing image of farmland collected by satellites (such as Gaofen-5 satellite, Ziyuan-1 satellite, etc.). Among them, the reflectivity of different crops in different bands shows unique characteristics, for example, different crops show different colors, grayscales, shapes and other characteristics.

[0123] For hyperspectral images collected from farmland, image classification network models can be applied for image classification. Specifically, the image classification network model can be used to extract the spectral features of different crops in different bands, and the spectral features extracted by the model can be associated with the known crop categories based on the association between the spectral features and categories of known crops, thereby achieving classification and identification of crop types.

[0124] After the processed spectral features are associated with known crop categories, it can be learned that the candidate objects in the hyperspectral image include multiple crop planting types such as alfalfa, corn-non-tillage, cornfield, corn, grassland-pasture, grassland-trees, etc., as well as multiple land use types such as buildings-grass-roads, stones-towers, etc.

[0125] Based on the categories of the candidate objects, more detailed crop planting distribution information can be provided, which is helpful for agricultural management and decision-making. At the same time, long-term acquisition of hyperspectral images for the same plot of land can achieve long-term monitoring of the plot of land, provide more data support and decision-making basis, thereby helping to improve crop yield and quality and promote sustainable agricultural development.

[0126] In the disclosed embodiment, by acquiring a hyperspectral image to be processed, and performing convolution processing on the hyperspectral image using a residual convolution module of a pre-trained image classification network model, and performing channel attention weighted processing using a channel attention module of the image classification network model, deep-level image features associated with the channel attention weights can be obtained, and feature information corresponding to multiple different bands of the hyperspectral image can be more effectively extracted, thereby reducing the computational complexity of the hyperspectral image and improving the classification accuracy of the hyperspectral image.

[0127] Optionally, the residual convolution module includes at least two convolution layers, and the at least two convolution layers are cascaded; wherein the output of any one of the at least two convolution layers is transmitted to the output of another convolution layer through a residual unit;

[0128] The residual convolution module is used to fuse image features at different levels in the hyperspectral image to obtain the first image feature.

[0129] In the disclosed embodiment, the residual convolution module includes at least two cascaded convolution layers, and the two cascaded convolution layers can constitute a straight branch in the residual convolution module. At the same time, a direct branch is also provided in the residual convolution module, and the output of any one of the at least two convolution layers can enter the residual unit through the direct branch and be added to the output of the other convolution layer, thereby realizing cross-layer connection in the convolutional neural network.

[0130] It can be understood that after the hyperspectral image is input into the residual convolution module, the feature matrix of the hyperspectral image is convolved in the residual convolution module with two branches. Among them, the feature matrix of the hyperspectral image is convolved in sequence through at least two convolution layers on the straight branch to generate a first output feature matrix. The feature matrix of the hyperspectral image is convolved through a convolution layer no greater than the existing number of layers on the direct branch, or is convolved without passing through any convolution layer to generate a second output feature matrix. The first output feature matrix and the second output feature matrix will be input into the residual unit, and then activated after the feature matrices are added. In this process, the output of any convolution layer is transferred to the output of another convolution layer through the residual unit.

[0131] Among them, the convolution of the feature matrix in the straight branch must go through multiple convolution layers, and the convolution level is deeper; the convolution of the feature matrix in the direct branch only needs to go through some or no convolution layers, and the convolution level is shallower. The feature matrices of the two different branches are processed by the residual unit to achieve the fusion of image features of different convolution levels in the hyperspectral image and obtain the first image feature.

[0132] In one embodiment, the directly connected branch of the residual convolution module can directly skip the convolution of a part of the convolution layers, so that by setting the structure of the convolution layer in the residual convolution module (or setting the weight parameters of the convolution layer), it is possible to retain the convolution layers with good convolution effects and skip the convolution layers with poor convolution effects, thereby effectively performing gradient updates.

[0133] In the disclosed embodiment, by setting a residual convolution module to process the input hyperspectral image, the overfitting of the deep network can be effectively alleviated and the generalization ability of the model can be improved.

[0134] Optional, Figure 3 is a schematic diagram of a residual convolution module according to an exemplary embodiment of the present disclosure. Figure 3 As shown, at least two convolutional layers in the residual convolution module 30 include:

[0135] A first depth-wise separable convolution layer CL1, a second depth-wise separable convolution layer CL2 and a third depth-wise separable convolution layer CL3; wherein the output of the first depth-wise separable convolution layer CL1 is passed to the output of the third depth-wise separable convolution layer CL3 through a residual unit 31.

[0136] In the disclosed embodiment, the convolution layer in the residual convolution module may be a depthwise separable convolution layer. The depthwise separable convolution layer may be composed of two parts: a depthwise convolution layer (DW) and a pointwise convolution layer (PW), which are used to extract feature maps. Compared with conventional convolution operations, the number of parameters and computational cost of the depthwise separable convolution layer are relatively low, and it can effectively process complex data such as hyperspectral images with multi-band data.

[0137] In one embodiment, after the hyperspectral image is input into the deep separable convolution layer, a large number of deep features are generated, and the output of the convolution layer needs to be processed by a batch normalization layer (BN) and a rectified linear unit layer (ReLU) to improve the reliability of the data.

[0138] like Figure 3As shown, the at least two convolutional layers may include three convolutional layers, namely a first depth-separable convolutional layer, a second depth-separable convolutional layer and a third depth-separable convolutional layer. Among them, the first depth-separable convolutional layer, the second depth-separable convolutional layer and the third depth-separable convolutional layer constitute a straight line branch, and the output of the first depth-separable convolutional layer can be connected to the output of the third depth-separable convolutional layer to form a direct branch. After the outputs of the two branches are input into the residual unit, the residual unit will perform matrix addition on the outputs of the two branches. In other words, the output of the first depth-separable convolutional layer is passed to the output of the third depth-separable convolutional layer to achieve cross-layer connection, fully retain the feature information of the first depth-separable layer, and overcome the defect of gradient disappearance.

[0139] In the disclosed embodiment, the output of the first depth-wise separable convolutional layer can be matrix-added with the output of the third depth-wise separable convolutional layer, thereby effectively alleviating the overfitting of the deep network and improving the generalization ability of the model.

[0140] Optionally, the image classification network model includes:

[0141] At least one of the channel attention modules comprises:

[0142] The spatial compression submodule is used to compress the spatial dimension of the input image features;

[0143] The channel feature extraction submodule is used to extract the channel attention features of each channel from the compressed image features;

[0144] The fusion submodule is used to perform weighted processing on the input image features based on the channel attention features of each channel to obtain the second image features.

[0145] In the embodiment of the present disclosure, the image classification network model may include: at least one channel attention module. Each channel attention module includes: a spatial compression submodule, a channel feature extraction submodule and a fusion submodule.

[0146] Among them, the spatial compression submodule is used to perform feature compression from the spatial dimension. It can convert the input three-dimensional feature (hwc dimension, for example (h, w, c)) into a feature whose first two dimensions are both 1 (for example (1, 1, c)), thereby realizing spatial dimension compression of the input first image feature.

[0147] The vector obtained after processing by the spatial compression submodule has a global receptive field to some extent, and the number of output feature channels matches the number of input feature channels, thereby ensuring that the first image feature is globally distributed on the feature channels.

[0148] In one embodiment, the spatial compression submodule may be composed of a fully connected layer that can implement a compression operation, such as a Squeeze operation.

[0149] The channel feature extraction submodule is used to generate weights for each feature channel by introducing the image width (Width, w) parameter, that is, adding parameters to the compressed image features to extract the channel attention features of each channel.

[0150] Among them, after the channel feature extraction submodule introduces the width parameter w of the image, it will also undergo a dimensionality reduction after perceiving the multi-layer data, reducing the number of parameters, and finally obtain the normalized weight between 0 and 1 through an S-type function (such as Sigmoid), that is, the channel attention feature of each feature channel, to explicitly model the correlation between feature channels.

[0151] In one embodiment, the channel feature extraction submodule may be composed of a fully connected layer capable of implementing an excitation operation, such as an Excitation operation.

[0152] The fusion submodule is used to perform matrix multiplication of the channel attention feature output by the channel feature extraction submodule with the first image feature to achieve weighted processing of the first image feature, thereby completing the recalibration of the first image feature in the channel dimension to obtain the second image feature.

[0153] It can be understood that compared with the first image feature, the acquisition of the second image feature adds a channel attention mechanism, so that a more accurate feature vector can be extracted.

[0154] In one embodiment, the fusion submodule may be composed of a fusion unit. The fusion unit may perform a fusion (Scale) operation.

[0155] It should be noted that each of the at least one channel attention module can implement the above-mentioned weighted processing of the channel attention features and the image features to enhance or suppress the features. Therefore, the number of channel attention modules in the image classification network model can be set according to the degree of feature enhancement or suppression required by the model.

[0156] In the disclosed embodiment, by extracting the channel attention features of each channel from the compressed image features, it is possible to perform weighted processing on the input image features to obtain the second image features. Compared with the first image features, the second image features perform different degrees of feature enhancement or feature suppression on different channels, thereby improving the accuracy of the model.

[0157] Optional, Figure 4The following is a flow chart of an image classification method according to an exemplary embodiment of the present disclosure. Figure 2 ,like Figure 4 As shown, after obtaining the hyperspectral image to be processed in step S101, the method further includes:

[0158] Step S401, preprocessing the hyperspectral image;

[0159] Step S402, based on the preprocessed hyperspectral image, obtaining a plurality of three-dimensional neighborhood block data corresponding to the hyperspectral image; wherein the central pixel points corresponding to the plurality of three-dimensional neighborhood block data are different;

[0160] The step S102 of performing convolution processing on the hyperspectral image using a residual convolution module of a pre-trained image classification network model to obtain a first image feature includes:

[0161] Step S403: input the plurality of three-dimensional neighborhood block data corresponding to the hyperspectral image into a residual convolution module in the image classification network model to obtain the first image feature.

[0162] In the embodiment of the present disclosure, after the hyperspectral image to be processed is acquired, the hyperspectral image needs to be preprocessed.

[0163] It is understandable that when a hyperspectral sensor (i.e., spectral imager) is used to simultaneously image multiple bands of an object to be detected, the obtained hyperspectral image contains a large amount of band information, but there is redundant band information, so the hyperspectral image needs to be preprocessed.

[0164] Here, the method of preprocessing the hyperspectral image can be set according to actual needs, and is not specifically limited. Exemplarily, the preprocessing method may include smoothing, normalization, removal of redundant data, multivariate scatter correction, Fourier transform, etc.

[0165] In step S402, through the processing of step S401, the preliminary screening of the data corresponding to the hyperspectral image can be achieved, and then step S402 is executed. Based on the feature that the hyperspectral image has multiple band information, that is, the hyperspectral image has multiple channel data, multiple three-dimensional neighborhood block data corresponding to the hyperspectral data can be obtained.

[0166] The three dimensions in the three-dimensional neighborhood block data refer to the height of the image, the width of the image and the band of the image.

[0167] It is understandable that, when acquiring multiple three-dimensional neighborhood block data corresponding to the hyperspectral data, multiple three-dimensional neighborhood block data can be obtained based on the neighborhood data corresponding to each pixel point in the hyperspectral image. Optionally, when acquiring the neighborhood data corresponding to each pixel point, a three-dimensional neighborhood block of a preset size can be acquired with each pixel point as the center, so that the central pixel points corresponding to the multiple three-dimensional neighborhood block data are different.

[0168] Here, the preset size can be determined according to actual needs and is not specifically limited. For example, each pixel point can be taken as the center to obtain three-dimensional neighborhood block data of 25*25*25 pixels.

[0169] It should be noted that the three-dimensional neighborhood block data constitutes a 3D image. When the 3D image corresponds to the input image classification network model, its input data and output data are both four-dimensional.

[0170] Exemplarily, a three-dimensional neighborhood block data (128, 128, 128, 3) may have four dimensions, wherein the fourth dimension represents the number of color channels.

[0171] In step S403, after obtaining multiple three-dimensional neighborhood block data corresponding to the hyperspectral image, the multiple three-dimensional neighborhood block data can be input into the residual convolution module in the image classification network model, so that the residual convolution module performs convolution pixel by pixel to obtain the first image feature.

[0172] Here, the structure and convolution process of the residual convolution module can refer to the description in one or more of the above embodiments, and for the sake of brevity of the specification, they will not be repeated here.

[0173] Exemplarily, the three-dimensional neighborhood block data (128, 128, 128, 3) can be input into the first depth-separable convolution layer with a convolution kernel size of 3×3×7 in the residual convolution module to generate a large number of first depth features. The first depth feature can be input into the third depth-separable convolution layer with a convolution kernel size of 3×3×5 in the residual convolution module to generate a large number of second depth features. The residual unit can add the first depth feature and the second depth feature to generate a first image feature.

[0174] It should be noted that after executing step S403, the first image feature can be obtained, and then steps S103 to S104 can be executed to complete the classification of the hyperspectral image.

[0175] Optionally, the method further includes: before inputting the image features into a channel attention module as described in step S103, dimensionality reduction processing may be performed on the image features.

[0176] It can be understood that the first image feature is four-dimensional data. Before the four-dimensional data is input into the channel attention module, the four-dimensional data needs to be reduced in dimension to become three-dimensional data so that the first image feature can be convolved through the channel attention module.

[0177] Exemplarily, a reshape operation may be performed on the first image feature to convert a feature map of (x.shape[0], x.shape[1], x.shape[2], x.shape[3]) into a feature map of (x.shape[0], x.shape[1], x.shape[2]×x.shape[3]) format.

[0178] Among them, x.shape[0], x.shape[1], x.shape[2] and x.shape[3] represent the height, width, depth and color channels of the image respectively.

[0179] In the embodiment of the present disclosure, after acquiring the hyperspectral image, the hyperspectral image can also be preprocessed; based on the preprocessed hyperspectral image, a plurality of three-dimensional neighborhood block data can be obtained; thus, the three-dimensional neighborhood block data can be input into the image classification network model to extract the correlation relationship between candidate objects in the hyperspectral image in different bands.

[0180] Optionally, the preprocessing of the hyperspectral image in step S401 includes:

[0181] Acquiring band data of the hyperspectral image in multiple bands;

[0182] Normalizing the band data of multiple bands of the hyperspectral image;

[0183] Re-determine the hyperspectral image based on the band data of the multiple bands after normalization;

[0184] The redundant band information in the hyperspectral image is removed.

[0185] In the embodiment of the present disclosure, when preprocessing the hyperspectral image, the original data collected by the hyperspectral sensor may be used, that is, the band data of the hyperspectral image in multiple bands may be obtained.

[0186] Here, the hyperspectral sensor may be arranged in a satellite, and obtaining the raw data is to obtain a hyperspectral remote sensing image.

[0187] After obtaining the band data of the hyperspectral image in multiple bands, the band data of the multiple bands can be distinguished and normalized by band. Based on the band data of the multiple bands after the normalization process, the bands can be spliced ​​to re-determine the hyperspectral image.

[0188] Here, the normalization process is to calculate the normalized value of each band data.

[0189] For example, the band data Bi of the hyperspectral image may be obtained, the average value Avei and the standard deviation Si of the band data Bi may be obtained, and then the normalized value Ni of the band data may be obtained according to the following formula (1).

[0190] N i =(B i -Ave i ) / S i (1)

[0191] After band stitching of the data after normalization of multiple bands, the hyperspectral image H can be re-determined. Subsequently, the hyperspectral image H can be subjected to redundant band removal processing to remove redundant band information in the hyperspectral image and retain important band information in the hyperspectral image.

[0192] Exemplarily, the redundant band removal process may be a process of removing redundant bands by using a principal component analysis (PCA) technique.

[0193] In the embodiments of the present disclosure, by preprocessing the hyperspectral image, not only can the smoothness and signal-to-noise ratio of the image be improved, thereby enhancing the accuracy of subsequent classification and modeling, but the hyperspectral image can also be reduced in dimensionality, redundant data can be removed, and computational complexity can be reduced.

[0194] Optional, Figure 5 The following is a flow chart of an image classification method according to an exemplary embodiment of the present disclosure. Figure 3 ,like Figure 5 As shown, before obtaining the hyperspectral image to be processed in step S101, the method further includes:

[0195] Step S501, obtaining a training sample set and a classification label corresponding to the training sample set;

[0196] Step S502, inputting a plurality of training images in the training sample set into an initial network model to be trained, and obtaining a prediction classification result output by the initial network model;

[0197] Step S503, determining the loss function value of the initial network model according to the predicted classification result and the classification label;

[0198] Step S504: Based on the loss function value, the parameters to be trained of the initial network model are optimized to obtain the image classification network model.

[0199] In the embodiment of the present disclosure, before obtaining the hyperspectral image to be processed, it is necessary to obtain a trained image classification network model, that is, it is necessary to train the initial network model at least based on the training sample set to obtain a trained image classification network model.

[0200] In step S501, the training sample set may include a plurality of training images, and the plurality of training images may be images of the same category as the hyperspectral image to be processed, for example, they may all be hyperspectral remote sensing images acquired by agricultural satellites.

[0201] The classification labels corresponding to the training sample set may also be the category labels of the candidate objects in the training images, such as plant species labels such as wheat and soybean. The classification labels corresponding to the training sample set can be manually identified with high accuracy and can be used to train the model.

[0202] In some embodiments, based on all classified hyperspectral images, multiple three-dimensional neighborhood block data corresponding to the classified hyperspectral images can be obtained; and some three-dimensional neighborhood data blocks are obtained from the multiple three-dimensional neighborhood block data to form a training sample set and a classification label corresponding to the training sample set.

[0203] In step S502, a plurality of training images in the training sample set may be input into an initial network model to be trained, and based on the initial network model, a prediction classification result of the initial network model for the training sample set may be obtained.

[0204] It can be understood that the initial network model includes at least a residual convolution module and a channel attention module, and the residual convolution module and the channel attention module can perform convolution processing on the training images of the training samples to obtain the predicted classification results output by the initial network model.

[0205] Here, the residual convolution module and the channel attention module can be described in one or more of the above embodiments, and for the sake of brevity of the specification, they will not be repeated here.

[0206] In step S503, the loss function value of the initial network model may be determined according to the predicted classification result of the training sample set obtained in step S502 and the classification label corresponding to the training sample set obtained in step S401.

[0207] It can be understood that the loss function value of the initial network model is determined based on the difference between the category information indicated by the predicted classification result of the training sample set and the category information indicated by the classification label corresponding to the training sample set; the loss function value can reflect the classification accuracy of the initial network model.

[0208] In step S504, based on the loss function value obtained in step S503, the parameters to be trained in the initial network model are correspondingly optimized and adjusted to obtain an optimized model, that is, an image classification network model.

[0209] It can be understood that after obtaining the loss function value of the initial network model, it is possible to determine whether the loss function value meets the conditions for stopping training. If the loss function value does not meet the conditions for stopping training, the parameters to be trained of the initial network model are optimized, and the above training steps are repeated until the loss function value meets the conditions for stopping training, and the image classification network model can be obtained.

[0210] Among them, the parameters to be trained include learning rate, number of iterations, batch size, regularization parameters, etc., and there is no specific limitation on this.

[0211] Optionally, based on the loss function value, the parameters to be trained of the initial network model are optimized to obtain the image classification network model, including: based on the loss function value, using an adaptive moment estimation optimization algorithm to optimize the parameters to be trained of the initial network model to obtain the image classification network model.

[0212] It is understandable that the parameters to be trained of the initial network model can be optimized based on the loss function value by using an adaptive moment estimation optimization algorithm.

[0213] Among them, the adaptive moment estimation optimization algorithm can design independent adaptive learning rates for different parameters by calculating the first-order moment estimation and the second-order moment estimation of the gradient.

[0214] In one embodiment, the adaptive moment estimation optimization algorithm may be implemented by using an adaptive moment estimation (AdaptiveMoment Estimation optimizer, Adam) optimizer.

[0215] Exemplarily, when the initial network model is trained using the training sample set, a learning rate of 0.01 can be used, an adaptive moment estimation optimizer can be used, and a total of 200 rounds of training can be performed at a rate of 10% / 20 rounds to obtain a trained model.

[0216] In the disclosed embodiment, the initial model may be trained based on a training sample set to reduce the gradient vanishing or gradient exploding problems caused by lack of initialization or improper initialization.

[0217] Optionally, before obtaining the hyperspectral image to be processed in step S101, the method further includes:

[0218] Obtaining a test sample set and a classification label corresponding to the test sample set;

[0219] Inputting a plurality of test images in the test sample set into the trained image classification network model to obtain a predicted classification result output by the image classification network model;

[0220] Based on the predicted classification result and the classification label, a classification accuracy test is performed on the trained image classification network model to obtain the classification accuracy of the image classification network model.

[0221] In the disclosed embodiment, the test sample set may include multiple test images, which may be images of the same category as the hyperspectral image to be processed, or may be other images among all labeled images except the images included in the training sample set. The test sample set also has corresponding classification labels.

[0222] In one embodiment, based on all classified hyperspectral images, multiple three-dimensional neighborhood block data corresponding to the classified hyperspectral images can be obtained; then, other three-dimensional neighborhood data blocks except the three-dimensional neighborhood data blocks corresponding to the training sample set can be obtained from the multiple three-dimensional neighborhood block data, and the other three-dimensional neighborhood data blocks can be determined as the three-dimensional neighborhood data blocks corresponding to the test sample set, thereby forming a test sample set and a classification label corresponding to the test sample set.

[0223] Here, the training sample set and the test sample set can be derived from the same image, so that the classification accuracy of the calculated image classification network model is more accurate.

[0224] After obtaining the test sample set, multiple test images in the test sample set can be input into the trained image classification network model to obtain the predicted classification results output by the image classification network model. Based on the category information indicated by the predicted classification results of the test sample set and the category information indicated by the classification labels corresponding to the test sample set, the classification accuracy of the image classification network model can be obtained.

[0225] Optionally, the classification accuracy includes at least one of the following: overall classification accuracy (OA), average classification accuracy (AA) and Kappa coefficient (K).

[0226] It can be understood that if the classification accuracy of the image classification network model obtained based on the test sample set is greater than or equal to a preset threshold, the image classification network model can be used to classify candidate objects in the unprocessed hyperspectral image. If the classification accuracy of the image classification network model obtained based on the test sample set is less than a preset threshold, the image classification network model can be used to classify candidate objects in the unprocessed hyperspectral image.

[0227] In the embodiments of the present disclosure, the image classification network model can be trained based on the test sample set to obtain the model accuracy, thereby further training the image classification network model and improving the accuracy of the model.

[0228] The image classification method in the embodiment of the present disclosure is described below with a specific example.

[0229] Figure 6 FIG. 1 is a schematic diagram of a structure of an image classification network model according to an exemplary embodiment of the present disclosure. Figure 6 As shown, the image classification network model 600 includes a residual convolution module and two channel attention modules. Among them, the residual convolution module can be composed of a depth-separable convolution layer CL1, a depth-separable convolution layer CL2, and a depth-separable convolution layer CL3. The first channel attention module of the two channel attention modules can be composed of a fully connected layer Sq1, a fully connected layer Ex1, and a fused convolution layer SE1; the second channel attention module can be composed of a fully connected layer Sq2, a fully connected layer Ex2, and a fused convolution layer SE1.

[0230] Next, a possible structure of the image classification network model is introduced in combination with the process of classifying hyperspectral images using the image classification network model.

[0231] Step 1: Obtain a hyperspectral image 601, and input the three-dimensional neighborhood block data corresponding to the hyperspectral image 601 into 8 depth-separable convolution layers CL1 with a convolution kernel size of 3×3×7 to generate a large number of first feature maps. At the same time, the output of the depth-separable convolution layer CL1 is processed by a batch normalization BN layer and a linear correction unit ReLU layer.

[0232] Step 2: Input the first feature map outputted in step 1 into the depthwise separable convolutional layer CL2 with 16 convolution kernels of size 3×3×5 to generate a large number of second feature maps. At the same time, the output of the depthwise separable convolutional layer CL2 is processed by the batch normalization BN layer and the linear rectification unit ReLU layer.

[0233] Step 3: Input the second feature map outputted in step 2 into the depthwise separable convolutional layer CL3 with 32 convolution kernels of size 3×3×3 to generate a large number of third feature maps. At the same time, the output of the depthwise separable convolutional layer CL3 is processed by the batch normalization BN layer and the linear rectification unit ReLU layer.

[0234] Step 4: Perform a cross-layer addition operation on the first feature map output by step 1 and the third feature map output by step 3 to obtain a fourth feature map as the input of the subsequent network.

[0235] Step 5: Use the reshape network layer Re to reshape the fourth feature map output in step 4 to achieve dimensionality reduction, thereby converting the feature map of (x.shape[0], x.shape[1], x.shape[2], x.shape[3]) into a feature map of (x.shape[0], x.shape[1], x.shape[2]×x.shape[3]) format.

[0236] Step 6: Pass the output feature map of step 5 through two series-connected fully connected layers: the first fully connected layer Sq1 has 16 neurons to implement the Squeeze operation; the second fully connected layer Ex1 has 256 neurons to implement the Excitation operation.

[0237] Step 7: Use the fused convolutional layer SE1 to perform matrix multiplication on the output feature map of step 5 and the output of step 6 to achieve the fusion (Scale) operation.

[0238] In step eight, the output feature map of step seven is input into a two-dimensional convolutional layer RL with 64 convolution kernels of size 3×3. The activation function of the two-dimensional convolutional layer RL can be a linear rectification function (ReLU).

[0239] Step nine: Input the output feature map of step eight into the global average pooling layer.

[0240] In step 10, the output feature map of step 9 is passed through two series-connected fully connected layers: the first fully connected layer Sq2 has 16 neurons to implement the Squeeze operation; the second fully connected layer Ex2 has 256 neurons to implement the Excitation operation.

[0241] In step 11, the output feature map of step 8 is matrix multiplied with the output of step 9 using the fused convolution layer SE2 to achieve the fusion (Scale) operation.

[0242] In step 12, the output feature map of step 11 is input into a two-dimensional convolution layer with 64 convolution kernels of size 3×3. The activation function of the two-dimensional convolution layer can be a linear rectification function (ReLU).

[0243] In step 13, the output feature map of step 12 is input into the compressed convolution layer FL to convert the multi-dimensional feature map into a one-dimensional feature vector.

[0244] Step 14: Input the output feature vector of step 13 into the fully connected layer FC1 containing 256 neurons, and input the output feature vector into the anti-overfitting (Dropout) layer with a disconnection probability of 0.5.

[0245] Step 15: Input the output feature vector of step 14 into the fully connected layer FC2 containing 128 neurons, and input the output feature vector into the anti-overfitting layer with a disconnection probability of 0.5.

[0246] In step 16, the output of step 15 is input to the fully connected layer FC3 containing 16 neurons, and the output feature vector is input to the activation layer SO, and the activation function (softmax) is used to realize the transformation from the output to the probability function, so as to obtain the hyperspectral image 602 containing the classification result of the candidate object.

[0247] In the disclosed embodiment, based on the image classification network model corresponding to the above structure, the residual convolution module of the pre-trained image classification network model can be used for convolution processing, and the channel attention module of the image classification network model can be used for channel attention weighted processing, so as to obtain deep-level image features associated with the channel attention weights, and more effectively extract feature information corresponding to multiple different bands of the hyperspectral image, thereby reducing the computational complexity of the hyperspectral image and improving the classification accuracy of the hyperspectral image.

[0248] The present disclosure provides an image classification method. Figure 7 The following is a flow chart of an image classification method according to an exemplary embodiment of the present disclosure. Figure 4 ,like Figure 7 As shown, the image classification method may include:

[0249] Step S701, start.

[0250] Step S702: collecting unprocessed hyperspectral images by using a drone or satellite.

[0251] Step S703, performing data preprocessing on the unprocessed hyperspectral image, including: denoising, removing redundant bands, etc.

[0252] Step S704: obtain a data set with classification labels, and divide the data set into a training sample set and a test sample set.

[0253] It is understandable that the training sample set is used to train the initial network model, and the test set is used to verify the classification performance of the image classification network model obtained after training. The method of obtaining the training sample set and the test sample set can refer to the description in one or more of the above embodiments, which will not be repeated here.

[0254] Step S705, building an initial network model corresponding to the image classification network model.

[0255] Understandably, it is possible to build Figure 6 The image classification network model shown, or the image classification network model in one or more of the above embodiments.

[0256] Step S706, using the training sample set in step S704 to train the initial network model in step S605 to obtain an image classification network model.

[0257] Step S707, use the test sample set in step S704 to determine whether the accuracy of the image classification network model is greater than the threshold, if so, execute step S708; if not, execute step S706.

[0258] Step S708, using the image classification network model with verified accuracy to classify the hyperspectral image that has undergone data preprocessing in step S703, and predict the classification results of the candidate objects in the hyperspectral image.

[0259] Step S709, end.

[0260] In the embodiment of the present disclosure, by constructing a model and preprocessing the model, an image classification network model that meets the accuracy requirements can be obtained; by using the image classification model, the preprocessed hyperspectral image can be classified.

[0261] The present disclosure provides an image classification device. Figure 8 is a structural diagram of an image classification device according to an exemplary embodiment of the present disclosure. Figure 8 As shown, the image classification device 800 includes:

[0262] An acquisition module 801 is used to acquire a hyperspectral image to be processed;

[0263] A first processing module 802 is used to perform convolution processing on the hyperspectral image using a residual convolution module of a pre-trained image classification network model to obtain a first image feature;

[0264] A second processing module 803 is used to perform channel attention weighted processing on the first image feature by using the channel attention module of the image classification network model to obtain a second image feature;

[0265] The prediction module 804 is configured to predict the classification result of the candidate object in the hyperspectral image based on the second image feature.

[0266] Optionally, the residual convolution module includes at least two convolution layers, and the at least two convolution layers are cascaded; wherein the output of any one of the at least two convolution layers is transmitted to the output of another convolution layer through a residual unit;

[0267] The residual convolution module is used to fuse image features at different levels in the hyperspectral image to obtain the first image feature.

[0268] Optionally, the at least two convolutional layers include:

[0269] A first depth-separable convolution layer, a second depth-separable convolution layer, and a third depth-separable convolution layer; wherein the output of the first depth-separable convolution layer is transmitted to the output of the third depth-separable convolution layer through a residual unit.

[0270] Optionally, the image classification network model includes:

[0271] At least one of the channel attention modules comprises:

[0272] The spatial compression submodule is used to compress the spatial dimension of the input image features;

[0273] The channel feature extraction submodule is used to extract the channel attention features of each channel from the compressed image features;

[0274] The fusion submodule is used to perform weighted processing on the input image features based on the channel attention features of each channel to obtain the second image features.

[0275] Optionally, the image classification device 800 further includes:

[0276] A preprocessing module, used for preprocessing the hyperspectral image;

[0277] A third processing module is used to obtain a plurality of three-dimensional neighborhood block data corresponding to the hyperspectral image based on the preprocessed hyperspectral image; wherein the central pixel points corresponding to the plurality of three-dimensional neighborhood block data are different;

[0278] The first processing module 802 is further configured to input the plurality of three-dimensional neighborhood block data corresponding to the hyperspectral image into a residual convolution module in the image classification network model to obtain the first image feature.

[0279] Optionally, the preprocessing module is further used to obtain band data of the hyperspectral image in multiple bands;

[0280] Normalizing the band data of multiple bands of the hyperspectral image;

[0281] Re-determine the hyperspectral image based on the band data of the multiple bands after normalization;

[0282] The redundant band information in the hyperspectral image is removed.

[0283] Optionally, the image classification device 800 further includes:

[0284] A fourth processing module, used to obtain a training sample set and a classification label corresponding to the training sample set;

[0285] Inputting a plurality of training images in the training sample set into an initial network model to be trained, and obtaining a prediction classification result output by the initial network model;

[0286] Determining a loss function value of the initial network model according to the predicted classification result and the classification label;

[0287] Based on the loss function value, the parameters to be trained of the initial network model are optimized to obtain the image classification network model.

[0288] Optionally, the fourth processing module is further used to:

[0289] Based on the loss function value, an adaptive moment estimation optimization algorithm is used to optimize the parameters to be trained of the initial network model to obtain the image classification network model.

[0290] Optionally, the fourth processing module is further used to:

[0291] Obtaining a test sample set and a classification label corresponding to the test sample set;

[0292] Inputting a plurality of test images in the test sample set into the trained image classification network model to obtain a predicted classification result output by the image classification network model;

[0293] Based on the predicted classification result and the classification label, a classification accuracy test is performed on the trained image classification network model to obtain the classification accuracy of the image classification network model.

[0294] Optionally, the classification accuracy includes at least one of the following:

[0295] Overall classification accuracy, average classification accuracy and Kappa coefficient.

[0296] Fig. 9 1 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. For example, the electronic device may be an image classification device in one or more of the above embodiments.

[0297] Reference Fig. 9 The electronic device 80 may include one or more of the following components: a processing component 83, a memory 84, a power component 85, a multimedia component 86, an audio component 87, an input / output (I / O) interface 88, a sensor component 89, and a communication component 810.

[0298] The processing component 83 generally controls the overall operation of the electronic device 80, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 83 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 83 may include one or more modules to facilitate the interaction between the processing component 83 and other components. For example, the processing component 83 may include a multimedia module to facilitate the interaction between the multimedia component 86 and the processing component 83.

[0299] The memory 84 is configured to store various types of data to support operations on the electronic device 80. Examples of such data include instructions for any application or method operating on the electronic device 80, contact data, phone book data, messages, pictures, videos, etc. The memory 84 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0300] The power supply assembly 85 provides power to the various components of the electronic device 80. The power supply assembly 85 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 80.

[0301] The multimedia component 86 includes a screen that provides an output interface between the electronic device 80 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 86 includes a front camera and / or a rear camera. When the electronic device 80 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0302] The audio component 87 is configured to output and / or input audio signals. For example, the audio component 87 includes a microphone (MIC), and when the electronic device 80 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 84 or sent via the communication component 810. In some embodiments, the audio component 87 also includes a speaker for outputting audio signals.

[0303] I / O interface 88 provides an interface between processing component 83 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button and lock button.

[0304] The sensor assembly 89 includes one or more sensors for providing various aspects of status assessment for the electronic device 80. For example, the sensor assembly 89 can detect the open / closed state of the electronic device 80, the relative positioning of components, such as the display and keypad of the electronic device 80, and the sensor assembly 89 can also detect the position change of the electronic device 80 or a component of the electronic device 80, the presence or absence of user contact with the electronic device 80, the orientation or acceleration / deceleration of the electronic device 80, and the temperature change of the electronic device 80. The sensor assembly 89 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 89 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 89 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0305] The communication component 810 is configured to facilitate wired or wireless communication between the electronic device 80 and other devices. The electronic device 80 can access a wireless network based on a communication standard, such as Wi-Fi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 810 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 810 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0306] In an exemplary embodiment, the electronic device 80 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above methods.

[0307] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 84 including instructions, and the instructions can be executed by the processor 820 of the electronic device 80 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0308] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0309] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image classification method, characterized in that: The method comprises: Acquire a hyperspectral image to be processed; For the hyperspectral image, a residual convolution module of a pre-trained image classification network model is used to perform convolution processing to obtain a first image feature; Using the channel attention module of the image classification network model, performing channel attention weighted processing on the first image feature to obtain a second image feature; Based on the second image feature, a classification result of the candidate object in the hyperspectral image is predicted.

2. The method according to claim 1, characterized in that The residual convolution module includes at least two convolution layers, and the at least two convolution layers are cascaded; wherein the output of any one of the at least two convolution layers is transmitted to the output of another convolution layer through a residual unit; The residual convolution module is used to fuse image features at different levels in the hyperspectral image to obtain the first image feature.

3. The method according to claim 2, characterized in that The at least two convolutional layers include: A first depth-separable convolution layer, a second depth-separable convolution layer, and a third depth-separable convolution layer; wherein the output of the first depth-separable convolution layer is transmitted to the output of the third depth-separable convolution layer through a residual unit.

4. The method according to claim 1, characterized in that: The image classification network model includes: At least one of the channel attention modules comprises: The spatial compression submodule is used to compress the spatial dimension of the input image features; The channel feature extraction submodule is used to extract the channel attention features of each channel from the compressed image features; The fusion submodule is used to perform weighted processing on the input image features based on the channel attention features of each channel to obtain the second image features.

5. The method according to claim 1, characterized in that After acquiring the hyperspectral image to be processed, the method further includes: Preprocessing the hyperspectral image; Based on the preprocessed hyperspectral image, a plurality of three-dimensional neighborhood block data corresponding to the hyperspectral image is obtained; wherein the central pixel points corresponding to the plurality of three-dimensional neighborhood block data are different; The method of performing convolution processing on the hyperspectral image using a residual convolution module of a pre-trained image classification network model to obtain a first image feature includes: The plurality of three-dimensional neighborhood block data corresponding to the hyperspectral image are input into a residual convolution module in the image classification network model to obtain the first image feature.

6. The method according to claim 5, characterized in that The preprocessing of the hyperspectral image comprises: Acquiring band data of the hyperspectral image in multiple bands; Normalizing the band data of multiple bands of the hyperspectral image; Re-determine the hyperspectral image based on the band data of the multiple bands after normalization; The redundant band information in the hyperspectral image is removed.

7. The method according to claim 1, characterized in that Before acquiring the hyperspectral image to be processed, the method further includes: Obtaining a training sample set and a classification label corresponding to the training sample set; Inputting a plurality of training images in the training sample set into an initial network model to be trained, and obtaining a prediction classification result output by the initial network model; Determining a loss function value of the initial network model according to the predicted classification result and the classification label; Based on the loss function value, the parameters to be trained of the initial network model are optimized to obtain the image classification network model.

8. The method according to claim 7, characterized in that The step of optimizing the parameters to be trained of the initial network model based on the loss function value to obtain the image classification network model comprises: Based on the loss function value, an adaptive moment estimation optimization algorithm is used to optimize the parameters to be trained of the initial network model to obtain the image classification network model.

9. The method according to claim 7, characterized in that: The method further comprises: Obtaining a test sample set and a classification label corresponding to the test sample set; Inputting a plurality of test images in the test sample set into the trained image classification network model to obtain a predicted classification result output by the image classification network model; Based on the predicted classification result and the classification label, a classification accuracy test is performed on the trained image classification network model to obtain the classification accuracy of the image classification network model.

10. The method according to claim 9, characterized in that The classification accuracy includes at least one of the following: Overall classification accuracy, average classification accuracy and Kappa coefficient.

11. An image classification device, characterized in that: The device comprises: An acquisition module, used for acquiring a hyperspectral image to be processed; A first processing module is used to perform convolution processing on the hyperspectral image using a residual convolution module of a pre-trained image classification network model to obtain a first image feature; A second processing module is used to use the channel attention module of the image classification network model to perform channel attention weighted processing on the first image feature to obtain a second image feature; A prediction module is used to predict the classification result of the candidate object in the hyperspectral image based on the second image feature.

12. An image classification device, characterized in that: The device comprises: a memory for storing processor-executable instructions; A processor connected to the memory; The processor is configured to execute the image classification method as described in any one of claims 1 to 11.

13. A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an image classification device, the image classification device is enabled to perform the image classification method according to any one of claims 1 to 11.

Images