Object classification method and device in hyperspectral image, equipment and storage medium

Abstract

The invention provides an object classification method and device for a hyperspectral image, electronic equipment and a readable storage medium, and the method comprises the steps: obtaining an original hyperspectral image, converting the original hyperspectral image into a color space image, calculating a shadow feature map according to the hue component and the brightness component of the color space image, and carrying out the calculation of the shadow feature map; the method comprises the steps of obtaining a shadow feature map of an original hyperspectral image, obtaining a shadow mask according to the shadow feature map, calculating the exposure intensity of the original hyperspectral image according to the shadow mask and the original hyperspectral image, obtaining an exposure image according to the exposure intensity and the original hyperspectral image, and carrying out multi-scale fusion processing on the original hyperspectral image and the exposure image to obtain a processed image. According to the method, the processed image and the original hyperspectral image are processed, the processed image and the original hyperspectral image are fused to obtain the shadow-free hyperspectral image, the dimension reduction processing is performed on the shadow-free hyperspectral image, and the dimension-reduced shadow-free hyperspectral image is input into the classification model for classification, so that the classification accuracy of the image can be improved.

Description

Technical Field

[0001] The present application belongs to the technical field of image classification, and specifically relates to a method, device, electronic device and readable storage medium for classifying objects in a hyperspectral image. Background Art

[0002] Hyperspectral remote sensing technology is one of the core technologies for earth observation. It can not only obtain spatial geometric information of the earth's surface, but also collect rich spectral information of ground objects. It has the characteristics of "spatial and spectral integration", which provides favorable support for the research of remote sensing earth observation.

[0003] The related technology extracts the feature vectors of the hyperspectral image, and then uses the support vector machine to obtain the classification probability matrix based on the similarity of the feature vectors. The classification probability sequence and estimated confidence are obtained by sorting the classification probability matrix. Finally, the α-expansion algorithm of the graph cut theory is used to classify the objects in the hyperspectral image.

[0004] However, for hyperspectral images in complex environments, the object classification methods proposed by related technologies have poor classification accuracy. Summary of the invention

[0005] The present application aims to provide a method, device, electronic device and readable storage medium for classifying objects in hyperspectral images, which at least solves the problem that the object classification methods proposed in the related art have poor classification accuracy for hyperspectral images in complex environments in the prior art.

[0006] In a first aspect, an embodiment of the present application discloses a method for classifying objects in a hyperspectral image, the method comprising:

[0007] Acquire an original hyperspectral image, and convert the original hyperspectral image into a color space image; the original hyperspectral image includes a shadow of an object;

[0008] Calculating a shadow feature map according to the hue component and the brightness component of the color space image, and obtaining a shadow mask according to the shadow feature map; the shadow feature map is used to reflect the hue and brightness of the shadow area in the original hyperspectral image;

[0009] Calculating the exposure intensity of the original hyperspectral image according to the shadow mask and the original hyperspectral image, and obtaining an exposure image according to the exposure intensity and the original hyperspectral image;

[0010] Performing multi-scale fusion processing on the original hyperspectral image and the exposure image to obtain a processed image, and fusing the processed image with the original hyperspectral image to obtain a shadow-free hyperspectral image;

[0011] The shadow-free hyperspectral image is subjected to dimensionality reduction processing to obtain a shadow-free hyperspectral image after dimensionality reduction processing, and the shadow-free hyperspectral image after dimensionality reduction processing is input into a classification model to perform object classification to obtain a classification result.

[0012] In a second aspect, an embodiment of the present application discloses a device for classifying objects in a hyperspectral image, the device comprising:

[0013] An image conversion module, used for acquiring an original hyperspectral image and converting the original hyperspectral image into a color space image; the original hyperspectral image includes a shadow of an object;

[0014] A first calculation module is used to calculate a shadow feature map according to the hue component and the brightness component of the color space image, and obtain a shadow mask according to the shadow feature map; the shadow feature map is used to reflect the hue and brightness of the shadow area in the original hyperspectral image;

[0015] a second calculation module, configured to calculate the exposure intensity of the original hyperspectral image according to the shadow mask and the original hyperspectral image, and obtain an exposure image according to the exposure intensity and the original hyperspectral image;

[0016] A multi-scale fusion module is used to perform multi-scale fusion processing on the original hyperspectral image and the exposure image to obtain a processed image, and fuse the processed image with the original hyperspectral image to obtain a shadow-free hyperspectral image;

[0017] The classification module is used to perform dimensionality reduction processing on the shadow-free hyperspectral image to obtain the shadow-free hyperspectral image after dimensionality reduction processing, and input the shadow-free hyperspectral image after dimensionality reduction processing into the classification model to perform object classification to obtain a classification result.

[0018] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0019] In a fourth aspect, an embodiment of the present application further discloses a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0020] In summary, in the embodiment of the present application, by acquiring the original hyperspectral image and converting the original hyperspectral image into a color space image, a shadow feature map is calculated according to the hue component and brightness component of the color space image, and a shadow mask is obtained according to the shadow feature map, and the exposure intensity of the original hyperspectral image is calculated according to the shadow mask and the original hyperspectral image, and an exposure image is obtained according to the exposure intensity and the original hyperspectral image, so as to amplify the information of the shadow area in the hyperspectral image to different degrees. The original hyperspectral image and the exposure image are subjected to multi-scale fusion processing to obtain a processed image, and the processed image is fused with the original hyperspectral image to obtain a shadow-free hyperspectral image to reconstruct the spectral characteristics of the object in the shadow area, and the hyperspectral image is de-shadowed. The shadow-free hyperspectral image is subjected to dimensionality reduction processing to obtain a shadow-free hyperspectral image after dimensionality reduction processing, and the shadow-free hyperspectral image after dimensionality reduction processing is input into the classification model for object classification to obtain a classification result. Since the present application performs shadow removal on the hyperspectral image and fuses the processed image obtained by multi-scale fusion with the original hyperspectral image, overexposure of the illuminated area is avoided and the spectral characteristics of the original hyperspectral image are effectively reconstructed, thereby improving the classification accuracy when classifying objects in the hyperspectral image. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In the attached picture:

[0022] Figure 1 is a flowchart of the steps of a method for classifying objects in a hyperspectral image provided by an embodiment of the present application;

[0023] Figure 2 is a flowchart of another method for classifying objects in a hyperspectral image provided by an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of a hyperspectral image provided in an embodiment of the present application;

[0025] Figure 4 is a schematic diagram of a shadow-free hyperspectral image provided in an embodiment of the present application;

[0026] Figure 5 This is a schematic diagram of the classification results of a SVM algorithm provided in an embodiment of the present application for a hyperspectral image;

[0027] Figure 6 It is a schematic diagram of the classification results of a PCA algorithm and a SVM algorithm provided in an embodiment of the present application for a hyperspectral image;

[0028] Figure 7 It is a schematic diagram of the classification results of a shadow removal algorithm, a PCA algorithm and a SVM algorithm provided in an embodiment of the present application for a hyperspectral image;

[0029] Figure 8 is a flowchart of the steps of another method for classifying objects in a hyperspectral image provided by an embodiment of the present application;

[0030] Fig. 9 is a block diagram of an object classification device in a hyperspectral image provided by an embodiment of the present application;

[0031] Fig.10 is a block diagram of an electronic device according to an embodiment of the present application;

[0032] Fig.11 It is a block diagram of an electronic device of another embodiment provided by the embodiments of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0035] Hyperspectral remote sensing technology is one of the core technologies for earth observation. It can not only obtain spatial geometric information of the earth's surface, but also collect rich spectral information of ground objects. It has the characteristics of "space-spectrum integration", which provides favorable support for the research of remote sensing earth observation. In recent years, hyperspectral remote sensing technology has been widely used and has played an important role in environmental monitoring, precision agriculture, resource exploration and other fields. However, when hyperspectral remote sensing imaging equipment collects data, due to the influence of objects such as clouds, forests and buildings, sunlight in some areas is blocked, resulting in shadows in the acquired hyperspectral remote sensing images. In the shadow area of ​​the hyperspectral remote sensing image, the brightness and contrast of the image decrease, the spatial information such as the outline, shape and texture of the ground object is weakened, and the spectral information of the ground object also changes, which seriously affects the subsequent interpretation of the image.

[0036] Since the shooting cycle of hyperspectral remote sensing images is long and the shooting cost is high, if the influence of shadows in the images cannot be resolved, the resources of hyperspectral remote sensing technology will be wasted. Therefore, in order to improve the utilization value of hyperspectral remote sensing images, save the cost of data collection, and increase the utilization rate of hyperspectral remote sensing technology, the interpretation of hyperspectral remote sensing images under shadow conditions is very important.

[0037] The classification of objects in hyperspectral remote sensing images under shadow conditions can be divided into two steps: shadow removal and image classification. Shadow detection is a preprocessing step before shadow removal, and its accuracy plays a vital role in the effect of the entire shadow removal. Shadow detection methods include threshold segmentation method, color space change method, physical information method, etc. Different shadow detection methods are suitable for different application scenarios. The methods for removing shadows are relatively mature. Currently, there are the following: shadow removal methods based on gradient elimination, shadow removal methods based on correction models, and shadow removal methods based on deep learning. Compared with color images, hyperspectral remote sensing images contain rich spectral information. According to the similarity of spectra, simple methods such as convolutional neural networks (CNN), K-nearest neighbor (KNN) and support vector machines (SVM) can usually be used to achieve hyperspectral image classification.

[0038] Related technologies remove shadows from hyperspectral images based on the intrinsic representation model. This method first uses the KNN classifier to classify pixels and obtain shadow masks, then decomposes the hyperspectral image into spectral reflectance components and brightness components, and enhances the brightness of shadow pixels in the brightness component to achieve the effect of shadow removal in the reconstructed hyperspectral image. However, when this method enhances the brightness of the shadow area in the brightness component, the shadow boundary area is overexposed, and the shadow edge restoration effect in the reconstructed hyperspectral image is not ideal.

[0039] The hyperspectral image classification method of related technology first extracts the feature vector of the hyperspectral image, then uses the support vector machine to obtain the classification probability matrix based on the similarity of the feature vector, obtains the classification probability sequence and estimated confidence based on the sorting of the classification probability matrix, and finally uses the α-expansion algorithm of the graph cut theory to obtain the image classification result. However, this method has complicated steps and does not consider the differences in spectral features of hyperspectral images under different conditions. It is not effective for hyperspectral image classification under complex environments.

[0040] In an embodiment of the present application, an original hyperspectral image is obtained and converted into a color space image, a shadow feature map is calculated according to the hue component and brightness component of the color space image, and a shadow mask is obtained according to the shadow feature map. The exposure intensity of the original hyperspectral image is calculated according to the shadow mask and the original hyperspectral image, and an exposure image is obtained according to the exposure intensity and the original hyperspectral image, so as to amplify the information of the shadow area in the hyperspectral image to different degrees. The original hyperspectral image and the exposure image are subjected to multi-scale fusion processing to obtain a processed image, and the processed image is fused with the original hyperspectral image to obtain a shadow-free hyperspectral image to reconstruct the spectral characteristics of the object in the shadow area, and the hyperspectral image is de-shadowed. The shadow-free hyperspectral image is subjected to dimensionality reduction processing to obtain a shadow-free hyperspectral image after dimensionality reduction processing, and the shadow-free hyperspectral image after dimensionality reduction processing is input into a classification model for object classification to obtain a classification result. Since the present application performs shadow removal on the hyperspectral image and fuses the processed image obtained by multi-scale fusion with the original hyperspectral image, overexposure of the illuminated area is avoided and the spectral characteristics of the original hyperspectral image are effectively reconstructed, thereby improving the classification accuracy when classifying objects in the hyperspectral image.

[0041] Figure 1 is a flowchart of a method for classifying objects in a hyperspectral image provided by an embodiment of the present application, see Figure 1 , the method may include the following steps:

[0042] Step 101: Acquire an original hyperspectral image, and convert the original hyperspectral image into a color space image; the original hyperspectral image includes the shadow of the object.

[0043] For example, the original hyperspectral image is the hyperspectral data released by the Earth Science and Remote Sensing Society in the 2013 Data Fusion Competition. The size of the hyperspectral data is 349×1905×144.

[0044] For example, the acquired original hyperspectral image needs to be normalized first, the maximum pixel value and the minimum pixel value of each band of the original hyperspectral image are calculated, and the pixel value of the original hyperspectral image is normalized from 0 to 255 to 0 to 1. This is to facilitate subsequent processing.

[0045] For example, the original hyperspectral image is a three-dimensional data, which is width, height and number of bands. The RGB channels of the original hyperspectral image are selected as 59, 40, and 23, that is, the images of the 59th, 40th, and 23th bands of the original hyperspectral image are selected. The images of these three bands can be combined into a three-band RGB image. Convert the RGB image of the original hyperspectral image into a hue-saturation-intensity (HSI) color space image.

[0046] Step 102: Calculate a shadow feature map according to the hue component and the brightness component of the color space image, and obtain a shadow mask according to the shadow feature map; the shadow feature map is used to reflect the hue and brightness of the shadow area in the original hyperspectral image.

[0047] For example, according to the hue component and brightness component of the HSI color space image, the calculation formula for calculating the shadow feature map is as follows:

[0048] F=(H+1) / (I+1)

[0049] Among them, I represents the brightness component of the original hyperspectral image, H represents the hue component of the original hyperspectral image, and F represents the shadow feature map.

[0050] For example, in the shadow feature map, the pixel value of the shadow area is much higher than the pixel value of the illuminated area. Based on this feature, the initial shadow detection map can be obtained by the adaptive threshold segmentation method. The adaptive meaning is that the corresponding threshold can be obtained for different shadow detection maps, and each shadow detection map will get a fixed threshold.

[0051] For example, the noise in the shadow detection image is caused by the small black objects in the original hyperspectral image being mistakenly detected as shadows. The area of ​​such areas mistakenly detected as shadows is small, and the shadows caused by clouds and fog are generally large shadows. Therefore, a minimum shadow area value can be set, and shadows smaller than the set area value can be removed in the initial shadow detection image, and shadows larger than the set area value are retained to obtain a shadow mask.

[0052] Step 103: Calculate the exposure intensity of the original hyperspectral image according to the shadow mask and the original hyperspectral image, and obtain an exposure image according to the exposure intensity and the original hyperspectral image.

[0053] For example, calculating the exposure intensity of the original hyperspectral image according to the shadow mask and the original hyperspectral image may be to determine the shadow area of ​​the original hyperspectral image by performing a dot product of the shadow mask and the original hyperspectral image.

[0054] For example, a shadow mask is used to obtain the average spectral reflectance of the shadow area and the illuminated area in the hyperspectral image. By scaling the proportional relationship between the average spectral reflectance between the illuminated area and the shadow area, the exposure intensity is obtained to obtain a set of exposure intensities, and the hyperspectral image is exposed to different degrees to obtain an exposed image.

[0055] Step 104: perform multi-scale fusion processing on the original hyperspectral image and the exposure image to obtain a processed image, and fuse the processed image with the original hyperspectral image to obtain a shadow-free hyperspectral image.

[0056] For example, in order to avoid obvious seams, a multi-scale fusion processing method is used to perform image fusion at multiple resolutions. First, the image is decomposed into an image pyramid using bandpass filters of different scales, and then the images of each layer are fused separately.

[0057] For example, the original hyperspectral image and the exposure image are subjected to multi-scale fusion processing to obtain a processed image, that is, the pixel value of each pixel in the original hyperspectral image and the exposure image is multiplied by the corresponding weight value, and then all are accumulated. The obtained exposure image sequence and the original hyperspectral image are subjected to multi-scale fusion processing, which can effectively restore the spectral information of the objects in the shadow area.

[0058] For example, the brightness component of the shadow area in the processed image is restored, achieving the effect under normal lighting conditions. However, the illuminated area is overexposed because the illuminated areas of most hyperspectral images in the fused image sequence are overexposed. Therefore, in order to keep the brightness component of the illuminated area in the reconstructed hyperspectral remote sensing image unchanged, it is necessary to fuse the processed image with the original hyperspectral image for a second time.

[0059] For example, the second stage of fusion is to fuse the processed image with the original hyperspectral image, which can be understood as splicing the reconstructed shadow area of ​​the first stage fusion result and the non-shadow area image of the original hyperspectral image. Specifically, the shadow mask can be used to fuse the reconstructed shadow area in the processed image with the illuminated area of ​​the original hyperspectral image. The spectral characteristics of the objects in the shadow area can be reconstructed by fusing the original hyperspectral image and the exposure image sequence using the two-stage image fusion method.

[0060] Step 105: Perform dimensionality reduction processing on the shadow-free hyperspectral image to obtain a shadow-free hyperspectral image after dimensionality reduction processing, and input the shadow-free hyperspectral image after dimensionality reduction processing into a classification model to perform object classification to obtain a classification result.

[0061] For example, when performing dimensionality reduction processing on a shadow-free hyperspectral image, a sparse principal component analysis (PCA) method can be used to reduce the dimensionality, so as to reduce the amount of data for subsequent SVM model training and retain spectral information with obvious features.

[0062] For example, the specific steps of the PCA algorithm are: centering the shadow-free hyperspectral image, subtracting the average value of the spectral feature of each dimension from the spectral feature of each dimension, constructing the spectral features of all samples into a spectral feature matrix, and obtaining the covariance matrix using the singular value decomposition principle; performing eigenvalue decomposition on the covariance matrix, sorting the eigenvalues ​​from large to small, and taking the eigenvector corresponding to the largest specific eigenvalue. The eigenvector is a vector in the spectral feature of the sample after dimensionality reduction, so as to perform dimensionality reduction processing on the shadow-free hyperspectral image.

[0063] For example, the algorithm used for sample classification is the linear inseparable SVM algorithm under the Gaussian kernel function mode. The basic idea of ​​this algorithm is to introduce slack variables, use the Gaussian kernel function to map nonlinear samples to high-dimensional feature space, and construct the lowest possible optimal classification hyperplane to achieve the best classifier performance. The linear inseparable SVM model under the Gaussian kernel function mode is trained with training samples to obtain a prediction model, and the accuracy of the prediction model is evaluated with test samples.

[0064] In an embodiment of the present application, an original hyperspectral image is obtained and converted into a color space image, a shadow feature map is calculated according to the hue component and brightness component of the color space image, and a shadow mask is obtained according to the shadow feature map. The exposure intensity of the original hyperspectral image is calculated according to the shadow mask and the original hyperspectral image, and an exposure image is obtained according to the exposure intensity and the original hyperspectral image, so as to amplify the information of the shadow area in the hyperspectral image to different degrees. The original hyperspectral image and the exposure image are subjected to multi-scale fusion processing to obtain a processed image, and the processed image is fused with the original hyperspectral image to obtain a shadow-free hyperspectral image to reconstruct the spectral characteristics of the object in the shadow area, and the hyperspectral image is de-shadowed. The shadow-free hyperspectral image is subjected to dimensionality reduction processing to obtain a shadow-free hyperspectral image after dimensionality reduction processing, and the shadow-free hyperspectral image after dimensionality reduction processing is input into a classification model for object classification to obtain a classification result. Since the present application performs shadow removal on the hyperspectral image and fuses the processed image obtained by multi-scale fusion with the original hyperspectral image, overexposure of the illuminated area is avoided and the spectral characteristics of the original hyperspectral image are effectively reconstructed, thereby improving the classification accuracy when classifying objects in the hyperspectral image.

[0065] Figure 2This is a flowchart of another method for classifying objects in hyperspectral images provided by this application. Figure 2 , the method may include the following steps:

[0066] Step 201: Acquire an original hyperspectral image, and convert the original hyperspectral image into a color space image; the original hyperspectral image includes the shadow of the object.

[0067] This step may be specifically referred to as the above step 101, and will not be described in detail here.

[0068] For example, Figure 3 is a schematic diagram of a hyperspectral image proposed in this application, see Figure 3 , Figure 3 The black area 31 in the figure is the shadow area. The shadow is generated in the acquired hyperspectral image when the hyperspectral remote sensing imaging device collects data due to the influence of objects such as clouds, forests, and buildings, which blocks the sunlight in some areas. For example, it can be seen that Figure 3 There are shadow areas on the streets 31 and buildings 32 in the figure.

[0069] Optionally, step 201 may specifically include:

[0070] Sub-step 2011, normalizing the original hyperspectral image to obtain a normalized original hyperspectral image;

[0071] Sub-step 2012: obtaining images of different preset bands in the normalized original hyperspectral image, and combining the images of all the preset bands to obtain a color image corresponding to the original hyperspectral image;

[0072] Sub-step 2013: converting the color image into the color space image by a color space conversion method.

[0073] For example, for sub-steps 2011 to 2013, the original hyperspectral image obtained needs to be normalized first, the maximum pixel value and the minimum pixel value of each band of the original hyperspectral image are calculated, and the pixel value of the original hyperspectral image is normalized from 0 to 255 to 0 to 1. This is to facilitate subsequent processing. The normalization formula is as follows:

[0074]

[0075] Among them, I 原始 is the original image, I 归一化 is the normalized image, min_value is the minimum pixel value in the image, and max_value is the maximum pixel value in the image.

[0076] For example, the original hyperspectral image is a three-dimensional data, which is width, height and number of bands. The RGB channels of the original hyperspectral image are selected as 59, 40, and 23, that is, the images of the 59th, 40th, and 23th bands of the original hyperspectral image are selected. The images of these three bands can be combined into a three-band RGB image. Convert the RGB image of the original hyperspectral image into a hue-saturation-intensity (HIS) color space image.

[0077] For example, the formula for converting an RGB image to an HSI image is as follows:

[0078] The formula for the brightness component is:

[0079]

[0080] The formula for the saturation component is:

[0081]

[0082] In the case of B<=G, the formula for the hue component is:

[0083]

[0084] In the case of B>G, the formula for the hue component is:

[0085]

[0086] Among them, R represents the image of the 59th band of the original hyperspectral image, G represents the image of the 40th band of the original hyperspectral image, B represents the image of the 23rd band of the original hyperspectral image, I represents the brightness component of the original hyperspectral image, S represents the saturation component of the original hyperspectral image, and H represents the hue component of the original hyperspectral image.

[0087] Step 202: Calculate a shadow feature map according to the hue component and the brightness component of the color space image, and obtain a shadow mask according to the shadow feature map; the shadow feature map is used to reflect the hue and brightness of the shadow area in the original hyperspectral image.

[0088] This step may be specifically referred to as the above step 102, and will not be described in detail here.

[0089] Optionally, step 202 may specifically include:

[0090] Sub-step 2021: comparing the pixel value of each pixel point in the shadow feature map with a first preset value, and obtaining a shadow detection map according to the comparison result; the shadow detection map is a binary image;

[0091] Sub-step 2022, calculating the number of pixels included in the connected area whose pixel values ​​of the pixels in the shadow detection image are the first value;

[0092] Sub-step 2023: modify the pixel values ​​of the pixels in the connected area according to the number of pixels included in the connected area, and generate the shadow mask according to the modified pixel values ​​of the pixels in the connected area.

[0093] For sub-steps 2021 to 2023, in the shadow feature map, the pixel values ​​of the shadow area are much higher than the pixel values ​​of the illuminated area. Based on this characteristic, an initial shadow detection map can be obtained by an adaptive threshold segmentation method. The adaptive meaning is that corresponding thresholds can be obtained for different shadow detection maps, and each shadow detection map will get a fixed threshold. The first preset value can be an adaptive threshold.

[0094] For example, the calculation formula for obtaining the initial shadow detection map through the adaptive threshold segmentation method is as follows:

[0095]

[0096] Where T represents the adaptive threshold, F i，j represents the pixel value of the (i, j)th pixel in the shadow feature map, Represents the pixel value in the initial shadow detection map corresponding to the pixel value of the (i, j)th pixel in the shadow feature map.

[0097] For example, the first value can be 0, representing a shadow pixel, and the number of pixels included in the connected area whose pixel value is 0 in the shadow detection image is calculated. According to the number of pixels included in the connected area, the pixel values ​​of the pixels in the connected area are modified, and the shadow mask is generated according to the modified pixel values ​​of the pixels in the connected area.

[0098] Optionally, sub-step 2023 may specifically include:

[0099] Sub-step 20231: if the number of pixels included in the connected area is less than a second preset value, determine that the connected area is an illuminated area, and modify the pixel values ​​of the pixels in the connected area to a second value; the second value is a value different from the first value;

[0100] Sub-step 20232: If the number of pixel points included in the connected area is greater than a third preset value, modify the pixel values ​​of the pixel points in the connected area whose pixel values ​​are the second value to the first value.

[0101] For sub-steps 20231 and 20232, the shadow detection image is a binary image, the first value can be 0, representing a shadow pixel, and the second value can be 1, representing an illuminated pixel. The area with a pixel value of 0 represents a shadow area, and the area with a pixel value of 1 represents an illuminated area. The area with a black pixel connected area less than 4000 pixels is identified as an illuminated area, and the pixels in these areas are set to 1. The pixels with individual pixel values ​​of 1 in the area with a black pixel connected area greater than 40,000 pixels are set to 0 to obtain a shadow mask image, remove the noise of the shadow detection image, and obtain the final shadow mask.

[0102] Step 203: Calculate the exposure intensity of the original hyperspectral image according to the shadow mask and the original hyperspectral image, and obtain an exposure image according to the exposure intensity and the original hyperspectral image.

[0103] This step may be specifically referred to as the above step 103, and will not be described in detail here.

[0104] Optionally, step 203 may specifically include:

[0105] Sub-step 2031, performing a dot multiplication operation on the shadow mask and the original hyperspectral image to determine a shadow area in the original hyperspectral image;

[0106] Sub-step 2032: obtaining a first average spectral reflectance and a second average spectral reflectance according to the shadow mask, the shadow area, and the original hyperspectral image; the first average spectral reflectance is used to reflect the spectral reflectance of the shadow area; the second average spectral reflectance is used to reflect the spectral reflectance of the illuminated area;

[0107] Sub-step 2033: Calculate the ratio of the first average spectral reflectance to the second average spectral reflectance, and scale the ratio by different preset scaling factors to obtain a plurality of different exposure intensities.

[0108] For sub-steps 2031 to 2033, a dot multiplication operation is performed between the shadow mask and the original hyperspectral image to determine the shadow area in the original hyperspectral image. The calculation formula is as follows:

[0109] I s =A·I

[0110] Among them, I is the original hyperspectral image, I s is the shadow area of ​​the original hyperspectral image of Houston, and A is the shadow mask.

[0111] For example, according to the shadow mask, the shadow area, and the original hyperspectral image, a first average spectral reflectance of the spectrum reflecting the shadow area and a second average spectral reflectance of the spectrum reflecting the illuminated area are obtained. The calculation formula for the ratio of the first average spectral reflectance to the second average spectral reflectance is as follows:

[0112]

[0113] Among them, d represents the ratio, q t represents the first average spectral reflectance, q r represents the second average spectral reflectance.

[0114] For example, a set of scaling factors, such as [0.5; 1; 1.5], may be set to scale the ratio of the first average spectral reflectance to the second average spectral reflectance to obtain multiple different exposure intensities. The calculation formula is as follows:

[0115] α k =λ k d

[0116] Among them, α k represents the exposure intensity, λ k represents the scaling factor, and d represents the ratio of the first average spectral reflectance to the second average spectral reflectance.

[0117] Optionally, sub-step 2031 may specifically include:

[0118] Sub-step 20311: calculating the first average spectral reflectance according to the shadow mask and the shadow area;

[0119] Sub-step 20312: determining the illumination area of ​​the original hyperspectral image according to the shadow mask and the original hyperspectral image;

[0120] Sub-step 20313: Calculate the second average spectral reflectance according to the shadow mask and the illuminated area.

[0121] For example, for sub-steps 20311 to 20313, an image of the shadow area at the shadow boundary of the original hyperspectral image is obtained through a morphological erosion operation.

[0122]

[0123] Wherein, B is a structural element. The shape and size of the structural element can be designed according to specific application requirements, and common shapes include circular, square, cross, etc. This method uses a cross-shaped structural element. is the morphological erosion operator, T r is the shadow image at the shadow boundary, Is is the shadow area of ​​the original hyperspectral image of Houston, and A is the shadow mask.

[0124] For example, the calculation formula of the first average spectral reflectance is as follows:

[0125]

[0126] Where M represents the total number of pixels, q r represents the average spectral reflectance of the shadow area, A is the shadow mask, B is the structural element, and I r The shadow image at the shadow boundary.

[0127] For example, the image of the illuminated area at the shadow boundary is obtained through the morphological dilation operation. The calculation formula is as follows:

[0128]

[0129] in, is the morphological dilation operator, I t is the illumination image at the shadow boundary, A is the shadow mask, B is the structural element, and I is the original hyperspectral image.

[0130] For example, the calculation formula for the average spectral reflectance of the illuminated area is as follows:

[0131]

[0132] Among them, q t Represents the average spectral reflectance of the illuminated area, I t is the illumination image at the shadow boundary, A is the shadow mask, B is the structural element, and M represents the total number of pixels.

[0133] Step 204: Perform multi-scale fusion processing on the original hyperspectral image and the exposure image to obtain a processed image.

[0134] This step may be specifically referred to as the above step 104, and will not be described in detail here.

[0135] Optionally, step 204 may specifically include:

[0136] Sub-step 2041: obtaining a first weight map corresponding to the original hyperspectral image according to the pixel values ​​of the pixels in the original hyperspectral image and a fourth preset value, and normalizing the first weight map to obtain a normalized first weight map;

[0137] Sub-step 2042: decompose each image in the original hyperspectral image into a first multi-resolution image, and perform weighted averaging on the first multi-resolution image using the normalized first weight map to obtain a processed image corresponding to the original hyperspectral image.

[0138] Sub-step 2043: obtaining a second weight map corresponding to the exposure image according to the pixel values ​​of the pixels in the exposure image and a fifth preset value, and normalizing the second weight map to obtain a normalized second weight map;

[0139] Sub-step 2044: decomposing each image in the exposure image into a second multi-resolution image, and performing weighted averaging on the second multi-resolution image using the normalized second weight map to obtain a processed image corresponding to the exposure image;

[0140] Sub-step 2045: performing a sum operation on the processed image corresponding to the original hyperspectral image and the processed image corresponding to the exposure image to obtain the processed image.

[0141] For example, for sub-steps 2041-2045, the fourth preset value can be 0.5. In hyperspectral images with different exposure intensities, the information of the shadow area is amplified to different degrees. The spectral characteristics of the objects in the shadow area can be reconstructed by fusing the original hyperspectral image and the exposure image sequence using a two-stage image fusion method. In the first stage of image fusion, in order to ensure that the intensity of the pixel is neither close to 0 (underexposure) nor close to 1 (overexposure), the Gaussian curve is used to calculate the proximity of each pixel intensity to 0.5, which is used as the weight of the fusion process. The first weight map is obtained by calculating the proximity of the pixel value of the pixel point in the original hyperspectral image to 0.5 through the Gaussian curve, and the first weight map corresponding to the original hyperspectral image is obtained. The calculation formula is as follows:

[0142]

[0143] Where, σ is a fixed coefficient, which is set to 0.2 by default. represents the fusion weight map of the kth exposure image, represents the kth exposure image.

[0144] For example, in order to ensure the consistency of the fusion result, it is necessary to normalize the first weight map to obtain a normalized first weight map so that the sum of the weights on each pixel is 1. The calculation formula is as follows:

[0145]

[0146] Where k is the total number of weight graphs, represents the fusion weight map of the kth exposure image, Represents the normalized fusion weight map of the kth exposure image.

[0147] For example, in order to avoid the phenomenon that the gradient of the weight value in the weight map changes quickly and causes obvious seams in the fusion result, the multi-scale fusion method is used to perform image fusion at multiple resolutions. The calculation formula is as follows:

[0148]

[0149] in, represents the exposure image at the lth level in the Laplace pyramid decomposition, represents the weight map in the Gaussian pyramid decomposition layer l image, L{P} l Laplacian pyramid representing the fused image.

[0150] The final processed image can be obtained by L{P} l The inverse Laplace transform of is obtained.

[0151] Optionally, step 203 may further include:

[0152] Sub-step 2034, performing a dot multiplication operation on the multiple different exposure intensities and the original hyperspectral image, respectively, to obtain multiple exposure images;

[0153] Step 204 may specifically include:

[0154] Sub-step 2045: performing multi-scale fusion processing on the multiple exposure images respectively to obtain a processed image corresponding to each exposure image.

[0155] For example, multiple exposure images, i.e., a set of multi-exposure image sequences, are generated by multiplying multiple exposure intensities with the original hyperspectral image. The calculation formula is as follows:

[0156]

[0157] in, represents the kth exposure image, α k Indicates exposure intensity.

[0158] Step 205: Perform Gaussian smoothing on the shadow mask to obtain a smoothed shadow mask.

[0159] For example, in order to ensure that the transition of the shadow area in the reconstructed hyperspectral remote sensing image is more natural, the shadow mask needs to be Gaussian smoothed to obtain a soft shadow mask.

[0160] Step 206 : Obtain a weight map corresponding to the smoothed shadow mask, and perform a dot product operation on the weight map corresponding to the smoothed shadow mask and the processed image to obtain a first dot product result.

[0161] For example, after obtaining the weight map corresponding to the smoothed shadow mask, the weight map corresponding to the smoothed shadow mask is multiplied with the processed image to obtain the first dot product result. That is, the weighted average value of the processed image in the lth layer image is calculated using the lth layer image of the weight map corresponding to the smoothed shadow mask as the weight value, and the obtained weighted average value is used as the first dot product result.

[0162] Step 207: Perform a dot product operation on the weight map corresponding to the smoothed shadow mask and the original hyperspectral image to obtain a second dot product result.

[0163] For example, after obtaining the weight map corresponding to the smoothed shadow mask, the weight map corresponding to the smoothed shadow mask is multiplied with the original hyperspectral image to obtain the second dot product result. That is, the weighted average of the original hyperspectral image in the lth layer image is calculated using the lth layer image of the weight map corresponding to the smoothed shadow mask as the weight value, and the obtained weighted average is used as the second dot product result.

[0164] Step 208: Add the first dot product result and the second dot product result, and then subtract a preset unit matrix from the sum to obtain the shadow-free hyperspectral image.

[0165] For example, after obtaining the first dot product result and the second dot product result, the first dot product result and the second dot product result are added, and then the preset unit matrix is ​​subtracted from the sum to obtain a shadow-free hyperspectral image. The calculation formula for fusing the reconstructed shadow area in the processed image with the illuminated area of ​​the original hyperspectral image using the shadow mask is as follows:

[0166] Q=P·G{A}+I·(1-G{A})

[0167] Among them, Q is the shadow-free hyperspectral image, P is the processed image, I is a matrix of all 1s, and G{A} is the weight map corresponding to the smoothed shadow mask.

[0168] For example, Figure 4 is a shadow-free hyperspectral image proposed in this application, see Figure 4 , Figure 4 The shadow-free hyperspectral image is an image obtained by the shadow removal algorithm proposed in this application. For example, it can be seen that the shadow in area 41 is basically removed, and the spectral information of street 42 and building 43 is reconstructed.

[0169] Step 209: perform dimensionality reduction processing on the shadow-free hyperspectral image to obtain a shadow-free hyperspectral image after dimensionality reduction processing, and input the shadow-free hyperspectral image after dimensionality reduction processing into a classification model to perform object classification to obtain a classification result.

[0170] This step may be specifically referred to as the above step 105, and will not be described in detail here.

[0171] Optionally, step 209 may specifically include:

[0172] Sub-step 2091, calculating the average value of the spectral features of each dimension of the shadow-free hyperspectral image;

[0173] Sub-step 2092: subtract the average value of the spectral features of each dimension from the spectral features of each dimension to obtain a spectral feature matrix;

[0174] Sub-step 2093, using singular value decomposition to calculate the eigenvalues ​​of the spectral matrix, determining the eigenvector corresponding to the largest eigenvalue as the vector in the spectral matrix after dimension reduction processing, and obtaining the shadow-free hyperspectral image after dimension reduction processing based on the spectral matrix after dimension reduction processing.

[0175] For sub-steps 2091 to 2093, the PCA algorithm is used to reduce the number of spectral features of all shadow-free hyperspectral images from 144 to 40. The implementation steps of the PCA algorithm are: centering the shadow-free hyperspectral image, subtracting the average value of the spectral features of each dimension from the spectral features of each dimension, and the calculation formula is as follows:

[0176]

[0177] Among them, e i represents the spectral features of all shadow-free hyperspectral images in the i-th dimension, and a is the spectral feature dimension of the sample.

[0178] For example, the spectral features of all shadow-free hyperspectral images are used to construct a spectral feature matrix, and the covariance matrix is ​​obtained using the singular value decomposition principle; the covariance matrix is ​​decomposed by eigenvalues, and the eigenvalues ​​are sorted from large to small, and the eigenvectors corresponding to the largest d eigenvalues ​​are taken. The eigenvectors are the spectral features of the sample after dimensionality reduction, and the spectral feature dimension of the sample is d. The eigenvector corresponding to the largest eigenvalue is determined as the vector in the spectral matrix after dimensionality reduction, and the shadow-free hyperspectral image after dimensionality reduction is obtained according to the spectral matrix after dimensionality reduction.

[0179] Optionally, the method further includes:

[0180] Step 210: dividing the shadow-free hyperspectral image after the dimension reduction process into training samples and test samples;

[0181] Step 211: train a linearly inseparable support vector machine classification model under a Gaussian kernel function mode using the training samples to obtain a trained support vector machine classification model;

[0182] Step 209 may further include:

[0183] Sub-step 2094: input the test sample into the trained support vector machine classification model to obtain the classification result.

[0184] For step 210-step 211, and sub-step 2094, for example, the total number of label samples of the shadow-free hyperspectral image is 15029, the number of label categories of the ground objects is 15, and the number of spectral features of the ground objects is 144. 10% of the total number of label samples of each type of ground object is extracted as training samples, and the rest are used as test samples.

[0185] For example, the linear inseparable SVM model under the Gaussian kernel function mode of the training sample is used to train the prediction model, and the accuracy of the prediction model is evaluated using the test sample. First, a training data set D is constructed:

[0186]

[0187] Among them, x i Represents sample i in d-dimensional space R d A real vector, y i Represents sample x i The corresponding category, n is the total number of samples in the training set D. The discriminant function of the hyperplane d-dimensional space can be expressed as:

[0188] g(x)=w T x+b

[0189] Where w represents the normal vector of the hyperplane, and b is an arbitrary constant. When the linear inseparable SVM algorithm is used, the sample must satisfy the following formula after normalization to achieve the condition of g(x) = 1:

[0190] y i [(w·x i )+b]=1-ε i , i=1,2,...,n

[0191] Among them, ε i is a slack variable, when 0<ε i When <1, sample x i is correctly classified; when ε i >1, sample x iis misclassified. The generalized optimal classification surface can be defined as:

[0192]

[0193] Where w represents the normal vector of the hyperplane and C is a constant. The generalized optimal classification surface problem is transformed into a dual problem using the Lagrangian optimization algorithm. A parameter β is set. i , solve the parameter β under the constraint condition i The maximum value of is subject to the following constraints:

[0194]

[0195] 0≤β i ≤C, i=1,...,n

[0196]

[0197] Map all samples into a high-dimensional feature space, construct the optimal classification hyperplane in the feature space, and the classification function becomes:

[0198]

[0199] Among them, β * is the optimal solution under the constraints. * It is calculated according to the following two formulas.

[0200]

[0201] β i (y i (w·x i )+b]-1)=0

[0202] In this embodiment, the SVM algorithm is set to the C-SVC classification mode, and the penalty slack variable is set to 1.2.

[0203] The kernel function is set to RBF Gaussian kernel function: exp(-gamma|uv| 2 ), and the gamma value is set to 2.8.

[0204] For example, the prediction model is obtained by training the SVM model using the training samples after dimensionality reduction, and the prediction model is used to classify the test samples after dimensionality reduction. By comparing with the sample labels, the overall classification accuracy (OA), average classification accuracy (AA) and Kappa coefficient are calculated to evaluate the performance of the present invention. The classification accuracy of different methods is shown in Table 1 below:

[0205] Table 1

[0206]

[0207] For example, Figure 5 This is a schematic diagram of the classification results of a SVM algorithm provided in an embodiment of the present application for a hyperspectral image. The overall classification accuracy of the algorithm is 75.54%. Figure 5 , for example, you can see that street 51 is misclassified and is marked as dark grey instead of light grey.

[0208] For example, Figure 6 This is a schematic diagram of the classification results of a PCA algorithm and a SVM algorithm for hyperspectral images provided in an embodiment of the present application. The overall classification accuracy of the algorithm is 85.31%. Figure 6 , for example, we can see that street 61 is marked in light grey, indicating that the probability of the street being correctly classified has increased.

[0209] For example, Figure 7 This is a schematic diagram of the classification results of a shadow removal algorithm, a PCA algorithm, and a SVM algorithm for hyperspectral images provided in an embodiment of the present application. The overall classification accuracy of the algorithm is 85.31%. Figure 7 ,For example, we can see that street 71 can be correctly classified, and the outline of street 71 is more obvious, which further illustrates the importance of removing shadows for hyperspectral image classification.

[0210] In an embodiment of the present application, an original hyperspectral image is obtained and converted into a color space image, a shadow feature map is calculated according to the hue component and brightness component of the color space image, and a shadow mask is obtained according to the shadow feature map. The exposure intensity of the original hyperspectral image is calculated according to the shadow mask and the original hyperspectral image, and an exposure image is obtained according to the exposure intensity and the original hyperspectral image, so as to amplify the information of the shadow area in the hyperspectral image to different degrees. The original hyperspectral image and the exposure image are subjected to multi-scale fusion processing to obtain a processed image, and the processed image is fused with the original hyperspectral image to obtain a shadow-free hyperspectral image to reconstruct the spectral characteristics of the object in the shadow area, and the hyperspectral image is de-shadowed. The shadow-free hyperspectral image is subjected to dimensionality reduction processing to obtain a shadow-free hyperspectral image after dimensionality reduction processing, and the shadow-free hyperspectral image after dimensionality reduction processing is input into a classification model for object classification to obtain a classification result. Since the present application performs shadow removal on the hyperspectral image and fuses the processed image obtained by multi-scale fusion with the original hyperspectral image, overexposure of the illuminated area is avoided and the spectral characteristics of the original hyperspectral image are effectively reconstructed, thereby improving the classification accuracy when classifying objects in the hyperspectral image.

[0211] See also Figure 8, which shows a flowchart of the steps of another method for classifying objects in a hyperspectral image provided by an embodiment of the present application, including:

[0212] Step S1: input a hyperspectral image under shadow conditions.

[0213] Step S2: Detect the shadow area of ​​the original hyperspectral image using a color space transformation method.

[0214] Step S3: Reconstruct the spectral information of the objects in the shadow area of ​​the original hyperspectral image using a multi-exposure image fusion algorithm to obtain a shadow-free hyperspectral image.

[0215] Step S4: Use the PCA algorithm to perform dimensionality reduction processing on the shadow-free hyperspectral image.

[0216] Step S5: using the SVM algorithm to classify the objects in the shadow-free hyperspectral remote sensing image after the dimensionality reduction process.

[0217] In an embodiment of the present application, an original hyperspectral image is obtained and converted into a color space image, a shadow feature map is calculated according to the hue component and brightness component of the color space image, and a shadow mask is obtained according to the shadow feature map. The exposure intensity of the original hyperspectral image is calculated according to the shadow mask and the original hyperspectral image, and an exposure image is obtained according to the exposure intensity and the original hyperspectral image, so as to amplify the information of the shadow area in the hyperspectral image to different degrees. The original hyperspectral image and the exposure image are subjected to multi-scale fusion processing to obtain a processed image, and the processed image is fused with the original hyperspectral image to obtain a shadow-free hyperspectral image to reconstruct the spectral characteristics of the object in the shadow area, and the hyperspectral image is de-shadowed. The shadow-free hyperspectral image is subjected to dimensionality reduction processing to obtain a shadow-free hyperspectral image after dimensionality reduction processing, and the shadow-free hyperspectral image after dimensionality reduction processing is input into a classification model for object classification to obtain a classification result. Since the present application performs shadow removal on the hyperspectral image and fuses the processed image obtained by multi-scale fusion with the original hyperspectral image, overexposure of the illuminated area is avoided and the spectral characteristics of the original hyperspectral image are effectively reconstructed, thereby improving the classification accuracy when classifying objects in the hyperspectral image.

[0218] See also Fig. 9 , which shows an object classification device 30 in a hyperspectral image provided by an embodiment of the present application, the object classification device 30 in a hyperspectral image comprises:

[0219] The conversion module 301 is used to obtain an original hyperspectral image and convert the original hyperspectral image into a color space image; the original hyperspectral image includes the shadow of the object;

[0220] A first calculation module 302 is used to calculate a shadow feature map according to the hue component and the brightness component of the color space image, and obtain a shadow mask according to the shadow feature map; the shadow feature map is used to reflect the hue and brightness of the shadow area in the original hyperspectral image;

[0221] A second calculation module 303 is used to calculate the exposure intensity of the original hyperspectral image according to the shadow mask and the original hyperspectral image, and obtain an exposure image according to the exposure intensity and the original hyperspectral image;

[0222] A multi-scale fusion module 304 is used to perform multi-scale fusion processing on the original hyperspectral image and the exposure image to obtain a processed image, and fuse the processed image with the original hyperspectral image to obtain a shadow-free hyperspectral image;

[0223] The classification module 305 is used to perform dimensionality reduction processing on the shadow-free hyperspectral image to obtain the shadow-free hyperspectral image after dimensionality reduction processing, and input the shadow-free hyperspectral image after dimensionality reduction processing into the classification model to perform object classification to obtain a classification result.

[0224] Optionally, the image conversion module includes:

[0225] A normalization processing submodule is used to perform normalization processing on the original hyperspectral image to obtain a normalized original hyperspectral image;

[0226] A combination submodule, used to obtain images of different preset bands in the normalized original hyperspectral image, and combine the images of all the preset bands to obtain a color image corresponding to the original hyperspectral image;

[0227] The image conversion submodule is used to convert the color image into the color space image through a color space conversion method.

[0228] Optionally, the first computing module includes:

[0229] A comparison submodule, used for comparing the pixel value of each pixel point in the shadow feature map with a first preset value, and obtaining a shadow detection map according to the comparison result; the shadow detection map is a binary image;

[0230] A first calculation submodule, used for calculating the number of pixels included in the connected area whose pixel values ​​of the pixels in the shadow detection image are the first value;

[0231] The modification submodule is used to modify the pixel values ​​of the pixels in the connected area according to the number of pixels included in the connected area, and generate the shadow mask according to the pixel values ​​of the modified pixels in the connected area.

[0232] Optionally, the modification submodule includes:

[0233] A first determining unit is configured to determine that the connected area is an illuminated area if the number of pixels included in the connected area is less than a second preset value, and to modify the pixel values ​​of the pixels in the connected area to a second value; the second value is a value different from the first value;

[0234] A modifying unit is used to modify the pixel values ​​of the pixel points in the connected area whose pixel values ​​are the second value to the first value if the number of pixel points included in the connected area is greater than a third preset value.

[0235] Optionally, the second computing module includes:

[0236] A first dot product calculation submodule is used to perform a dot product operation on the shadow mask and the original hyperspectral image to determine a shadow area in the original hyperspectral image;

[0237] An obtaining submodule, configured to obtain a first average spectral reflectance and a second average spectral reflectance according to the shadow mask, the shadow area, and the original hyperspectral image; the first average spectral reflectance is used to reflect the spectral reflectance of the shadow area; the second average spectral reflectance is used to reflect the spectral reflectance of the illuminated area;

[0238] The second calculation submodule is used to calculate the ratio of the first average spectral reflectance to the second average spectral reflectance, and scale the ratio by different preset scaling factors to obtain a plurality of different exposure intensities.

[0239] Optionally, the obtaining submodule includes:

[0240] A first calculation unit, configured to calculate the first average spectral reflectance according to the shadow mask and the shadow area;

[0241] A second determining unit, configured to determine an illuminated area of ​​the original hyperspectral image according to the shadow mask and the original hyperspectral image;

[0242] The second calculation unit is used to calculate the second average spectral reflectance according to the shadow mask and the illuminated area.

[0243] Optionally, the second computing module includes:

[0244] A second dot product calculation submodule is used to perform dot product operations on the multiple different exposure intensities and the original hyperspectral image, respectively, to obtain multiple exposure images;

[0245] The multi-scale fusion module comprises:

[0246] The multi-scale fusion submodule is used to perform multi-scale fusion processing on the multiple exposure images respectively to obtain a processed image corresponding to each exposure image.

[0247] Optionally, the multi-scale fusion module includes:

[0248] A first obtaining submodule is used to obtain a first weight map corresponding to the original hyperspectral image according to the pixel values ​​of the pixels in the original hyperspectral image and a fourth preset value, and normalize the first weight map to obtain a normalized first weight map;

[0249] The first weighted average submodule is used to decompose each image in the original hyperspectral image into a first multi-resolution image, and perform weighted averaging on the first multi-resolution image using the normalized first weight map to obtain a processed image corresponding to the original hyperspectral image.

[0250] A second obtaining submodule, configured to obtain a second weight map corresponding to the exposure image according to the pixel values ​​of the pixels in the exposure image and a fifth preset value, and normalize the second weight map to obtain a normalized second weight map;

[0251] a second weighted averaging submodule, configured to decompose each image in the exposure image into a second multi-resolution image, and perform weighted averaging on the second multi-resolution image using the normalized second weight map to obtain a processed image corresponding to the exposure image;

[0252] The first summing calculation submodule is used to perform a summing operation on the processed image corresponding to the original hyperspectral image and the processed image corresponding to the exposure image to obtain the processed image.

[0253] Optionally, the multi-scale fusion module includes:

[0254] A Gaussian smoothing submodule, used for performing Gaussian smoothing on the shadow mask to obtain a smoothed shadow mask;

[0255] a third dot product calculation submodule, configured to obtain a weight map corresponding to the smoothed shadow mask, and perform a dot product operation on the weight map corresponding to the smoothed shadow mask and the processed image to obtain a first dot product result;

[0256] A fourth dot product calculation submodule is used to perform a dot product operation on the weight map corresponding to the smoothed shadow mask and the original hyperspectral image to obtain a second dot product result;

[0257] The second addition calculation submodule is used to add the first dot product result and the second dot product result, and then subtract a preset unit matrix from the sum result to obtain the shadow-free hyperspectral image.

[0258] Optionally, the classification module includes:

[0259] A third calculation submodule is used to calculate the average value of the spectral features of each dimension of the shadow-free hyperspectral image;

[0260] The third submodule is used to subtract the average value of the spectral features of each dimension from the spectral features of each dimension to obtain a spectral feature matrix;

[0261] A determination submodule is used to calculate the eigenvalues ​​of the spectral matrix by singular value decomposition, determine the eigenvector corresponding to the largest eigenvalue as the vector in the spectral matrix after dimension reduction processing, and obtain the shadow-free hyperspectral image after dimension reduction processing based on the spectral matrix after dimension reduction processing.

[0262] Optionally, the device further comprises:

[0263] A division module, used for dividing the shadow-free hyperspectral image after the dimensionality reduction process into training samples and test samples;

[0264] A training module, used for training a linearly inseparable support vector machine classification model under a Gaussian kernel function mode using the training samples to obtain a trained support vector machine classification model;

[0265] Optionally, the classification module includes:

[0266] The fourth obtaining submodule is used to input the test sample into the trained support vector machine classification model to obtain the classification result.

[0267] In an embodiment of the present application, an original hyperspectral image is obtained and converted into a color space image, a shadow feature map is calculated according to the hue component and brightness component of the color space image, and a shadow mask is obtained according to the shadow feature map. The exposure intensity of the original hyperspectral image is calculated according to the shadow mask and the original hyperspectral image, and an exposure image is obtained according to the exposure intensity and the original hyperspectral image, so as to amplify the information of the shadow area in the hyperspectral image to different degrees. The original hyperspectral image and the exposure image are subjected to multi-scale fusion processing to obtain a processed image, and the processed image is fused with the original hyperspectral image to obtain a shadow-free hyperspectral image to reconstruct the spectral characteristics of the object in the shadow area, and the hyperspectral image is de-shadowed. The shadow-free hyperspectral image is subjected to dimensionality reduction processing to obtain a shadow-free hyperspectral image after dimensionality reduction processing, and the shadow-free hyperspectral image after dimensionality reduction processing is input into a classification model for object classification to obtain a classification result. Since the present application performs shadow removal on the hyperspectral image and fuses the processed image obtained by multi-scale fusion with the original hyperspectral image, overexposure of the illuminated area is avoided and the spectral characteristics of the original hyperspectral image are effectively reconstructed, thereby improving the classification accuracy when classifying objects in the hyperspectral image.

[0268] See also Fig.10 , the electronic device 400 may include one or more of the following components: a processing component 402 , a memory 404 , a power component 406 , a multimedia component 408 , an audio component 410 , an input / output (I / O) interface 412 , a sensor component 414 , and a communication component 416 .

[0269] The processing component 402 generally controls the overall operation of the electronic device 400, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 402 may include one or more modules to facilitate the interaction between the processing component 402 and other components. For example, the processing component 402 may include a multimedia module to facilitate the interaction between the multimedia component 408 and the processing component 402.

[0270] The memory 404 is used to store various types of data to support the operation of the electronic device 400. Examples of such data include instructions for any application or method operating on the electronic device 400, contact data, phone book data, messages, pictures, multimedia, etc. The memory 404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0271] The power supply component 406 provides power to the various components of the electronic device 400. The power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 400.

[0272] The multimedia component 408 includes an interface that provides an output interface between the electronic device 400 and the user. In some embodiments, the interface may include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 408 includes a front camera and / or a rear camera. When the electronic device 400 is in an operating mode, such as a shooting mode or a multimedia mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0273] The audio component 410 is used to output and / or input audio signals. For example, the audio component 410 includes a microphone (MIC), and when the electronic device 400 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 404 or sent via the communication component 416. In some embodiments, the audio component 410 also includes a speaker for outputting audio signals.

[0274] The input / output I / O interface 412 provides an interface between the processing component 402 and the peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0275] The sensor assembly 414 includes one or more sensors for providing various aspects of status assessment for the electronic device 400. For example, the sensor assembly 414 can detect the open / closed state of the electronic device 400, the relative positioning of components, such as the display and keypad of the electronic device 400, and the sensor assembly 414 can also detect the position change of the electronic device 400 or a component of the electronic device 400, the presence or absence of user contact with the electronic device 400, the orientation or acceleration / deceleration of the electronic device 400, and the temperature change of the electronic device 400. The sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 414 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 414 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0276] The communication component 416 is used to facilitate wired or wireless communication between the electronic device 400 and other devices. The electronic device 400 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 416 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 416 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0277] In an exemplary embodiment, the electronic device 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement a method for classifying objects in a hyperspectral image provided in an embodiment of the present application.

[0278] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, and the instructions can be executed by a processor 420 of an electronic device 400 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0279] Fig.11is a block diagram of an electronic device 500 according to another embodiment of the present invention. For example, the electronic device 500 may be provided as a server. Fig.11 The electronic device 500 includes a processing component 522, which further includes one or more processors, and a memory resource represented by a memory 532 for storing instructions executable by the processing component 522, such as an application. The application stored in the memory 532 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 522 is configured to execute instructions to perform a method for classifying objects in a hyperspectral image provided in an embodiment of the present application.

[0280] The electronic device 500 may also include a power supply component 526 configured to perform power management of the electronic device 500, a wired or wireless network interface 550 configured to connect the electronic device 500 to a network, and an input / output (I / O) interface 558. The electronic device 500 may operate based on an operating system stored in the memory 532, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0281] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0282] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for classifying objects in hyperspectral images, characterized in that: The method comprises: Acquire an original hyperspectral image, and convert the original hyperspectral image into a color space image; the original hyperspectral image includes a shadow of an object; Calculating a shadow feature map according to the hue component and the brightness component of the color space image, and obtaining a shadow mask according to the shadow feature map; the shadow feature map is used to reflect the hue and brightness of the shadow area in the original hyperspectral image; Calculating the exposure intensity of the original hyperspectral image according to the shadow mask and the original hyperspectral image, and obtaining an exposure image according to the exposure intensity and the original hyperspectral image; Performing multi-scale fusion processing on the original hyperspectral image and the exposure image to obtain a processed image, and fusing the processed image with the original hyperspectral image to obtain a shadow-free hyperspectral image; The shadow-free hyperspectral image is subjected to dimensionality reduction processing to obtain a shadow-free hyperspectral image after dimensionality reduction processing, and the shadow-free hyperspectral image after dimensionality reduction processing is input into a classification model to perform object classification to obtain a classification result.

2. The method according to claim 1, characterized in that The converting the original hyperspectral image into a color space image comprises: Normalizing the original hyperspectral image to obtain a normalized original hyperspectral image; Obtaining images of different preset bands in the normalized original hyperspectral image, and combining all the images of the preset bands to obtain a color image corresponding to the original hyperspectral image; The color image is converted into the color space image by a color space conversion method.

3. The method according to claim 1, characterized in that The step of obtaining a shadow mask according to the shadow feature map comprises: Compare the pixel value of each pixel point in the shadow feature map with a first preset value, and obtain a shadow detection map according to the comparison result; the shadow detection map is a binary image; Calculate the number of pixels included in the connected area whose pixel values ​​of the pixels in the shadow detection image are the first value; According to the number of pixels included in the connected region, pixel values ​​of the pixels in the connected region are modified, and according to the modified pixel values ​​of the pixels in the connected region, the shadow mask is generated.

4. The method according to claim 3, characterized in that The step of modifying the pixel value of the pixel point according to the number of the pixel points included in the connected area comprises: If the number of pixels included in the connected area is less than a second preset value, the connected area is determined to be an illuminated area, and the pixel values ​​of the pixels in the connected area are modified to a second value; the second value is a value different from the first value; If the number of pixel points included in the connected area is greater than a third preset value, the pixel values ​​of the pixel points in the connected area whose pixel values ​​are the second value are modified to the first value.

5. The method according to claim 1, characterized in that The step of calculating the exposure intensity of the original hyperspectral image according to the shadow mask and the original hyperspectral image comprises: Performing a dot multiplication operation on the shadow mask and the original hyperspectral image to determine a shadow area in the original hyperspectral image; According to the shadow mask, the shadow area, and the original hyperspectral image, a first average spectral reflectance and a second average spectral reflectance are obtained; the first average spectral reflectance is used to reflect the spectral reflectance of the shadow area; the second average spectral reflectance is used to reflect the spectral reflectance of the illuminated area; The ratio of the first average spectral reflectance to the second average spectral reflectance is calculated, and the ratio is scaled by different preset scaling factors to obtain a plurality of different exposure intensities.

6. The method according to claim 5, characterized in that The step of obtaining a first average spectral reflectance and a second average spectral reflectance according to the shadow mask, the shadow area, and the original hyperspectral image comprises: Calculating the first average spectral reflectance according to the shadow mask and the shadow area; Determining an illuminated area of ​​the original hyperspectral image according to the shadow mask and the original hyperspectral image; The second average spectral reflectance is calculated based on the shadow mask and the illuminated area.

7. The method according to claim 5, characterized in that The step of obtaining an exposure image according to the exposure intensity and the original hyperspectral image comprises: Performing a dot multiplication operation on the multiple different exposure intensities and the original hyperspectral image, respectively, to obtain multiple exposure images; Performing multi-scale fusion processing on the exposure image to obtain a processed image includes: Multi-scale fusion processing is performed on the multiple exposure images respectively to obtain a processed image corresponding to each exposure image.

8. The method according to claim 1, characterized in that The performing multi-scale fusion processing on the original hyperspectral image and the exposure image to obtain a processed image includes: According to the pixel values ​​of the pixels in the original hyperspectral image and a fourth preset value, a first weight map corresponding to the original hyperspectral image is obtained, and the first weight map is normalized to obtain a normalized first weight map; Decomposing each image in the original hyperspectral image into a first multi-resolution image, and performing weighted averaging on the first multi-resolution image using the normalized first weight map to obtain a processed image corresponding to the original hyperspectral image; Obtaining a second weight map corresponding to the exposure image according to pixel values ​​of pixels in the exposure image and a fifth preset value, and normalizing the second weight map to obtain a normalized second weight map; Decomposing each image in the exposure image into a second multi-resolution image, and performing weighted averaging on the second multi-resolution image using the normalized second weight map to obtain a processed image corresponding to the exposure image; The processed image corresponding to the original hyperspectral image and the processed image corresponding to the exposure image are added together to obtain the processed image.

9. The method according to claim 1, characterized in that: The step of fusing the processed image with the original hyperspectral image to obtain a shadow-free hyperspectral image includes: Performing Gaussian smoothing on the shadow mask to obtain a smoothed shadow mask; Acquire a weight map corresponding to the smoothed shadow mask, and perform a dot product operation on the weight map corresponding to the smoothed shadow mask and the processed image to obtain a first dot product result; Performing a dot product operation on the weight map corresponding to the smoothed shadow mask and the original hyperspectral image to obtain a second dot product result; The first dot product result and the second dot product result are added together, and a preset unit matrix is ​​subtracted from the sum to obtain the shadow-free hyperspectral image.

10. The method according to claim 1, characterized in that The step of performing dimensionality reduction processing on the shadow-free hyperspectral image to obtain the shadow-free hyperspectral image after dimensionality reduction processing includes: Calculating the average value of the spectral features of each dimension of the shadow-free hyperspectral image; Subtract the average value of the spectral features of each dimension from the spectral features of each dimension to obtain a spectral feature matrix; The eigenvalues ​​of the spectral matrix are calculated using singular value decomposition, the eigenvector corresponding to the largest eigenvalue is determined as the vector in the spectral matrix after dimension reduction processing, and the shadow-free hyperspectral image after dimension reduction processing is obtained according to the spectral matrix after dimension reduction processing.

11. The method according to claim 1, characterized in that: The method further comprises: Dividing the shadow-free hyperspectral image after the dimensionality reduction process into training samples and test samples; Using the training samples to train a linearly inseparable support vector machine classification model under a Gaussian kernel function mode, to obtain a trained support vector machine classification model; The step of inputting the shadow-free hyperspectral image after the dimension reduction process into a classification model to classify objects and obtain a classification result includes: The test sample is input into the trained support vector machine classification model to obtain the classification result.

12. An object classification device in a hyperspectral image, characterized in that: The device comprises: An image conversion module, used for acquiring an original hyperspectral image and converting the original hyperspectral image into a color space image; the original hyperspectral image includes a shadow of an object; A first calculation module is used to calculate a shadow feature map according to the hue component and the brightness component of the color space image, and obtain a shadow mask according to the shadow feature map; the shadow feature map is used to reflect the hue and brightness of the shadow area in the original hyperspectral image; a second calculation module, configured to calculate the exposure intensity of the original hyperspectral image according to the shadow mask and the original hyperspectral image, and obtain an exposure image according to the exposure intensity and the original hyperspectral image; A multi-scale fusion module is used to perform multi-scale fusion processing on the original hyperspectral image and the exposure image to obtain a processed image, and fuse the processed image with the original hyperspectral image to obtain a shadow-free hyperspectral image; The classification module is used to perform dimensionality reduction processing on the shadow-free hyperspectral image to obtain the shadow-free hyperspectral image after dimensionality reduction processing, and input the shadow-free hyperspectral image after dimensionality reduction processing into the classification model to perform object classification to obtain a classification result.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images