A hyperspectral payload on-orbit real-time target detection and identification method and system

Abstract

This invention relates to the field of hyperspectral remote sensing technology, and provides a method and system for real-time on-orbit target detection and recognition of hyperspectral payloads. The method includes receiving a raw hyperspectral image; automatically acquiring spectral correction coefficients, radiometric correction coefficients, and blind pixel locations, and performing spectral and radiometric processing combined with an on-board fundamental coefficient library to generate a hyperspectral image; detecting targets using a lightweight encoder-decoder convolutional neural network with a dual attention mechanism, where a high- and low-frequency coupled adversarial autoencoder network separates and extracts features from high- and low-frequency spectral information to generate a target spectral fingerprint vector; inputting the target spectral fingerprint vector into a relational metric network for matching and identification to obtain the corresponding target type, and outputting a descriptive frame of the extracted target attribute information; adaptively correcting the raw hyperspectral image through spectral and radiometric processing; a focus mechanism aggregating key features of the hyperspectral image; and matching high-dimensional heterogeneous features to achieve high-precision detection at the high-resolution pixel level and target recognition of specific categories.

Description

Technical Field

[0001] This invention relates to the field of hyperspectral remote sensing technology, and in particular to a method and system for real-time on-orbit target detection and identification of hyperspectral payloads. Background Technology

[0002] The continuous narrow bands and high resolution of hyperspectral images enable individual pixels to carry spectral information that approximates the material fingerprint, allowing for differentiation among similar-looking targets. They also maintain strong stability under interference conditions such as complex backgrounds, weak contrast, and changes in illumination, making them suitable for applications in precision agriculture, industrial quality inspection, and sorting.

[0003] However, in existing technologies, due to the large volume of hyperspectral data, the abundance of redundant backgrounds, and the limited downlink time slots, the traditional mode of full on-orbit imaging downlinking and offline processing on the ground often results in a delay of several hours or even days between data acquisition and information output. For applications such as early warning monitoring of natural disasters like forest fires, landslides, and floods, as well as emergency rescue of personnel at disaster accident sites, timely acquisition of target information is crucial. The earlier effective information is obtained, the more economic losses can be minimized. Summary of the Invention

[0004] The purpose of this invention is to address the aforementioned technical problems by proposing a method and system for real-time on-orbit target detection and recognition using a hyperspectral payload. This method enables on-orbit detection and recognition of hyperspectral images under the constraint of limited satellite payload resources, thereby promoting the development and application of hyperspectral satellite remote sensing. The objective of this invention can be achieved through the following technical solutions: This invention provides a method for real-time on-orbit target detection and identification of hyperspectral payloads, comprising: Step S1: Receive the raw hyperspectral image; Step S2: Based on the spectral correction coefficients, radiometric correction coefficients and blind pixel positions automatically obtained from the onboard calibration, and combined with the onboard basic coefficient library, the original hyperspectral image is subjected to spectral and radiometric processing to generate a hyperspectral image. The process of generating the hyperspectral image also includes scene segmentation of the hyperspectral image, and for images of land scenes, key areas are extracted by combining the GIS library and prior information. Step S3: Target spatial region information is obtained by performing target detection on the hyperspectral image using a lightweight encoder-decoder convolutional neural network with a dual attention mechanism. Based on the target spatial region information, the corresponding hyperspectral image is input into a high-low frequency coupled adversarial autoencoder network to separate the high-frequency and low-frequency spectral information and extract high-level features, generating a target spectral fingerprint vector. The high-low frequency coupled adversarial autoencoder network consists of a low-pass filter network, a high-frequency deep autoencoder network, a low-frequency deep autoencoder network, and an adversarial discriminant network. Step S4: Use a hyperspectral multi-objective optimization sparse unmixing model to fuse spectral pixel vectors, generate target spectral fingerprint prototypes, and construct a hybrid pixel spectral database; input the target spectral fingerprint vectors into a relational metric network, match and identify the target spectral fingerprint prototypes in the hybrid pixel spectral database to obtain the corresponding target type, and output the extracted target attribute information description frame.

[0005] Further, in step S2, based on the spectral correction coefficients, radiometric correction coefficients, and blind pixel positions automatically obtained from onboard calibration, and combined with the onboard fundamental coefficient library, the original hyperspectral image is subjected to spectral and radiometric processing to generate a hyperspectral image, including... The spectral correction coefficients, radiometric correction coefficients, and blind cell positions are automatically obtained through onboard calibration, and the spectral payload data pre-stored in the onboard basic coefficient library is called. The spectral payload data includes the dark background time-varying model, pre-launch blind cell positions, Etalon coefficients, relative radiometric calibration coefficients, and spectral calibration coefficients. The hyperspectral raw image is subjected to on-orbit spectral correction based on the spectral correction coefficient and the spectral calibration coefficient to correct the lateral spectral deviation and compensate for the drift when the center wavelength of the band is detected. The original hyperspectral image after on-orbit spectral correction is used to subtract the on-orbit dark background signal using a dark background time-varying model and radiometric correction coefficients to eliminate the detector's dark level background noise. Based on the blind pixel location and the pre-launch blind pixel location, the hyperspectral raw image after deducting the on-orbit dark background signal is repaired in orbit to restore the image response value at the abnormal pixel. The images repaired by in-orbit blind pixels are corrected using Etalon coefficients, and then hyperspectral images with spectral and radiometric processing are output after in-orbit non-uniformity correction based on relative radiometric calibration coefficients.

[0006] The step also includes dividing the hyperspectral image into land, cloud, and sea scenes based on pixel spectral features, and using location information in the GIS database to determine the target area in the land scene; Using a pre-stored map of the hyperspectral payload as prior information, the target area is registered with the pre-stored map to extract the key areas of the land scene.

[0007] Further, in step S3, target spatial region information is obtained by performing target detection on the hyperspectral image using a lightweight encoder-decoder convolutional neural network with a dual attention mechanism, including: The hyperspectral image is input into an encoder composed of Blocks 1 and 2 stacked in a pre-defined residual network structure to extract the corresponding target feature representation. The decoder, composed of Blocks 1 and 3 stacked in a pre-defined residual network structure, performs feature recovery, preserving effective target features and extracting spatial feature information with a stronger receptive field. The decoder is a mirror image of the encoder, and Blocks 1, 2, and 3 are different spatial-channel attention modules. Block 1 is used to keep the input feature map size and number of channels unchanged, Block 2 is used to reduce the width and height of the output feature map to half that of the input and double the number of channels, and Block 3 is used to enlarge the width and height of the output feature map to twice that of the input feature map and halve the number of channels, thereby enhancing the spatial and channel features of the target.

[0008] Furthermore, the encoder and decoder of the lightweight convolutional neural network with dual attention mechanism use pre-defined pointwise convolutional layers to achieve target spatial region information detection in hyperspectral images under satellite payload constraints. The weight parameters in the binary quantized pointwise convolutional layers are constrained to be binary, and each convolutional kernel is assigned a pre-defined floating-point coefficient to compress the number of floating-point parameters and multiplication operations.

[0009] Furthermore, network parameters are optimized during the training of a lightweight encoder-decoder convolutional neural network with a dual attention mechanism by using a loss function consisting of a weighted sum of the exponential logarithmic Dice loss function and the weighted exponential cross-entropy loss function to improve the accuracy of target space detection.

[0010] Further, in step S3, based on the target spatial region information, the corresponding hyperspectral image is input into a high-low frequency coupled adversarial autoencoder network to separate the high-frequency and low-frequency spectral information and extract high-level features, generating a target spectral fingerprint vector, including... The hyperspectral image is filtered by a low-pass filter network to separate the high-frequency and low-frequency components of the spectrum. High-level features of the high-frequency and low-frequency components of the spectrum are extracted by high-frequency deep autoencoder networks and low-frequency deep autoencoder networks respectively, and then fused. The input spectral vector and the fused high-level features are combined and fed into the adversarial discriminant network to generate an adversarial signal to update and optimize the parameters of the high-frequency deep autoencoder network and the low-frequency deep autoencoder network, thereby obtaining the corresponding target spectral high-level features to form the target spectral fingerprint vector.

[0011] Further, in step S4, a hyperspectral multi-objective optimization sparse unmixing model is used to fuse the spectral pixel vectors to generate a target spectral fingerprint prototype and construct a hybrid pixel spectral database, including: Each spectrum in the spectral library is binary-encoded using a hyperspectral multi-objective optimization sparse unmixing model, and the multi-objective optimization problem for each spectrum is decomposed into several single-objective sub-problems. A random flipping strategy is used to generate offspring individuals, and spectral features are added as a regularization term to the Chebyshev decomposition function to determine whether to update the individual and its neighboring individuals. After reaching the preset iteration threshold, the spectrum corresponding to the optimal individual of each subproblem is extracted, and the abundance matrix is ​​solved by non-negative least squares to generate the target spectral fingerprint prototype and store it in the mixed pixel spectral database.

[0012] Further, in step S4, the target spectral fingerprint vector is input into the relation metric network and matched with the target spectral fingerprint prototypes in the mixed pixel spectral database to obtain the corresponding target type. The extracted target attribute information description frame is then output, including... The target spectral fingerprint vector and the target spectral fingerprint prototype in the mixed pixel spectral database are input into the relation measurement network. The corresponding relation score is obtained by comparing the similarity between the target high-dimensional features and the target sample prototype. The target type is determined based on the relation score, and the target category, height, and location information are framed and output.

[0013] Further, in step S4, the relation measurement network includes a first two-dimensional convolutional block, a second two-dimensional convolutional block, a first fully connected layer, and a second fully connected layer that are sequentially connected. The first two-dimensional convolutional block and the second two-dimensional convolutional block each include a convolutional layer, a batch normalization layer, and a ReLU activation layer. The first fully connected layer uses the ReLU activation function, and the second fully connected layer uses the Sigmoid function. The loss function of the relation measurement network is the mean squared error, which outputs a score on the relationship between the target spectral fingerprint vector and the target spectral fingerprint prototype.

[0014] Based on the same inventive concept, this invention provides a method for real-time on-orbit target detection and identification of hyperspectral payloads, employing the aforementioned method, including: The image correction module is used to receive the original hyperspectral image, and based on the spectral correction coefficient, radiometric correction coefficient and blind pixel position automatically obtained by on-board calibration, combined with the on-board basic coefficient library, it performs spectral and radiometric processing on the original hyperspectral image to generate a hyperspectral image. The process of generating the hyperspectral image also includes scene segmentation of the hyperspectral image, and for images of land scenes, it extracts key areas by combining GIS library and prior information. The target detection module is used to perform target detection on hyperspectral images using a lightweight encoder-decoder convolutional neural network with a dual attention mechanism to obtain target spatial region information. Based on the target spatial region information, the corresponding hyperspectral image is input into a high-low frequency coupled adversarial autoencoder network to separate high-frequency and low-frequency spectral information and extract high-level features, generating a target spectral fingerprint vector. The high-low frequency coupled adversarial autoencoder network consists of a low-pass filter network, a high-frequency deep autoencoder network, a low-frequency deep autoencoder network, and an adversarial discriminant network. The target recognition module is used to fuse spectral pixel vectors using a hyperspectral multi-objective optimized sparse unmixing model to generate target spectral fingerprint prototypes and construct a hybrid pixel spectral database. The target spectral fingerprint vectors are input into a relational metric network and matched with the target spectral fingerprint prototypes in the hybrid pixel spectral database to identify the corresponding target type. The extracted target attribute information is then used to describe and output the frame.

[0015] Compared with the prior art, the present invention has at least one of the following technical advantages: This invention adaptively corrects original hyperspectral images through spectral and radiometric processing, aggregates key features of hyperspectral images through an attention-focusing mechanism, and accurately matches high-dimensional heterogeneous features to achieve high-precision detection at the high-resolution pixel level and target recognition for specific categories. Addressing the technical challenges of relatively low spatial resolution, small target size, and the presence of band noise, data redundancy, and spatial domain detection errors in hyperspectral images, this invention effectively focuses target features through the ordered stacking of spatial-channel attention modules. Furthermore, it reduces network model storage space consumption and the computational complexity of multiplication between convolutional and fully connected layers by replacing some point-by-point convolutional layers with binary quantization point-by-point convolutional layers, enabling rapid and accurate target detection. Simultaneously, a high-low frequency coupled adversarial autoencoder deep neural network separates high- and low-frequency spectral information and extracts high-level features, obtaining target spectral high-dimensional features as a spectral fingerprint vector, further separating the target from the background and performing spectral dimensionality reduction. In addition, a relational metric network compares the similarity between the target spectral fingerprint vector and the prototype samples in a mixed pixel spectral library to generate relational scores, ensuring high relational scores for similar targets and low relational scores for dissimilar targets, achieving high-precision target recognition and attribute information extraction in hyperspectral images. Attached Figure Description

[0016] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart illustrating the steps of a real-time on-orbit target detection and recognition method for a hyperspectral payload according to the present invention. Figure 2 This is a flowchart of the on-orbit real-time preprocessing of the original hyperspectral image in the on-orbit real-time target detection and recognition method of the hyperspectral payload of the present invention. Figure 3 This is a structural diagram of a lightweight encoder-decoder convolutional neural network with a dual attention mechanism in an embodiment of the present invention, wherein, Figure 3 (a) in the diagram is the overall block diagram of the encoder-decoder lightweight convolutional neural network. Figure 3 (b) shows the structure diagrams of three different spatial-channel attention modules. Figure 3 (c) in the figure shows the detection results of the encoder-decoder lightweight convolutional neural network; Figure 4 This is a structural diagram of the high- and low-frequency coupled adversarial autoencoder network in an embodiment of the present invention; Figure 5 This is a flowchart illustrating the operation of the multi-objective optimization sparse unmixing model in an embodiment of the present invention. Figure 6 The diagram shows the network structure and recognition results of the relation measurement system in this embodiment of the invention. Figure 6 (a) Relationship metric network structure diagram; Figure 6 (b) in the figure shows the results of the relation measurement network identification. Figure 7 This is a diagram of a hyperspectral payload on-orbit real-time target detection and recognition system according to the present invention. Detailed Implementation

[0017] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] First Embodiment In existing technologies, the typical approach for target detection and recognition in hyperspectral images involves first acquiring raw image data using a hyperspectral payload, then performing spectral correction, radiometric correction, noise suppression, band selection, feature extraction, and target recognition on the transmitted images at the ground. For target detection, existing technologies generally extract target regions using traditional convolutional neural networks or spatial domain feature-based detection algorithms. For spectral feature analysis, methods such as dimensionality reduction, filtering, or sparse unmixing are typically used to extract the spectral domain information of the target, which is then further combined with classifiers or similarity matching methods to achieve target recognition. While these methods can accomplish target detection and recognition in hyperspectral images to a certain extent, the inherent characteristics of hyperspectral images, such as relatively low spatial resolution, small target size, strong band noise, and significant data redundancy, coupled with the reliance on offline ground-based computation in traditional processing workflows, result in a significant time delay from data acquisition to output. This makes it difficult to meet the timeliness requirements of on-orbit real-time processing and disaster emergency scenarios. Furthermore, in the target detection stage, most existing technologies rely solely on spatial domain features or conventional convolutional networks for target localization. This is easily affected by factors such as complex backgrounds, small target scales, and limited spatial resolution, leading to false detections, missed detections, or unclear target boundaries. In the spectral feature extraction stage, existing methods typically employ uniform dimensionality reduction or conventional encoding to process spectral information, making it difficult to simultaneously consider both high-frequency detail information and low-frequency overall trend information, resulting in insufficient spectral discrimination between the target and the background. In the target recognition stage, existing methods often struggle to maintain stable matching accuracy when faced with changes in target features due to factors such as observation distance, observation angle, and atmospheric turbulence, affecting the accuracy of the final recognition results.

[0019] Based on the above analysis, the applicant team recognized that to achieve real-time on-orbit target detection and recognition for hyperspectral payloads, relying solely on existing offline ground processing methods or single spatial or spectral domain processing approaches is insufficient to simultaneously ensure processing timeliness, detection accuracy, and recognition stability. Therefore, the applicant team proposed a real-time on-orbit target detection and recognition method for hyperspectral payloads: Onboard hyperspectral images undergo spectral and radiometric processing, and key regions are extracted using a GIS database and prior information to reduce redundant background interference. Then, a lightweight encoder-decoder convolutional neural network with a dual attention mechanism is used for rapid detection of target spatial regions. Next, a high-low frequency coupled adversarial autoencoder network is used to separate high and low frequencies and extract high-level features from the target's corresponding spectral information, generating a target spectral fingerprint vector. A hyperspectral multi-objective optimized sparse unmixing model is then used to generate a target spectral fingerprint prototype and construct a hybrid pixel spectral database. Finally, a relational metric network is used to match and identify the target spectral fingerprint vector with the target spectral fingerprint prototype, achieving high-precision output of target type and attribute information. The above technical solution can improve the real-time performance, accuracy, and stability of target detection and recognition under conditions of limited spatial resolution and high noise and redundancy in hyperspectral images. Specific implementation methods are as follows: like Figure 1 As shown, this invention provides a method for real-time on-orbit target detection and identification of hyperspectral payloads, including: Step S1: Receive the raw hyperspectral image; Step S2: Based on the spectral correction coefficients, radiometric correction coefficients and blind pixel positions automatically obtained from the onboard calibration, and combined with the onboard basic coefficient library, the original hyperspectral image is subjected to spectral and radiometric processing to generate a hyperspectral image. The process of generating the hyperspectral image also includes scene segmentation of the hyperspectral image, and for images of land scenes, key areas are extracted by combining the GIS library and prior information. Step S3: Target spatial region information is obtained by performing target detection on the hyperspectral image using a lightweight encoder-decoder convolutional neural network with a dual attention mechanism. Based on the target spatial region information, the corresponding hyperspectral image is input into a high-low frequency coupled adversarial autoencoder network to separate the high-frequency and low-frequency spectral information and extract high-level features, generating a target spectral fingerprint vector. The high-low frequency coupled adversarial autoencoder network consists of a low-pass filter network, a high-frequency deep autoencoder network, a low-frequency deep autoencoder network, and an adversarial discriminant network. Step S4: Use a hyperspectral multi-objective optimization sparse unmixing model to fuse spectral pixel vectors, generate target spectral fingerprint prototypes, and construct a hybrid pixel spectral database; input the target spectral fingerprint vectors into a relational metric network, match and identify the target spectral fingerprint prototypes in the hybrid pixel spectral database to obtain the corresponding target type, and output the extracted target attribute information description frame.

[0020] Furthermore, such as Figure 2 As shown, in step S2, based on the spectral correction coefficients, radiometric correction coefficients, and blind pixel positions automatically obtained from onboard calibration, and combined with the onboard fundamental coefficient library, the original hyperspectral image is subjected to spectral and radiometric processing to generate a hyperspectral image, including... The spectral correction coefficients, radiometric correction coefficients, and blind cell positions are automatically obtained through onboard calibration, and the spectral payload data pre-stored in the onboard basic coefficient library is called. The spectral payload data includes the dark background time-varying model, pre-launch blind cell positions, Etalon coefficients, relative radiometric calibration coefficients, and spectral calibration coefficients. The hyperspectral raw image is subjected to on-orbit spectral correction based on the spectral correction coefficient and the spectral calibration coefficient to correct the lateral spectral deviation and compensate for the drift when the center wavelength of the band is detected. The original hyperspectral image after on-orbit spectral correction is used to subtract the on-orbit dark background signal using a dark background time-varying model and radiometric correction coefficients to eliminate the detector's dark level background noise. Based on the blind pixel location and the pre-launch blind pixel location, the hyperspectral raw image after deducting the on-orbit dark background signal is repaired in orbit to restore the image response value at the abnormal pixel. The images repaired by in-orbit blind pixels are corrected using Etalon coefficients, and then hyperspectral images with spectral and radiometric processing are output after in-orbit non-uniformity correction based on relative radiometric calibration coefficients.

[0021] The step also includes dividing the hyperspectral image into land, cloud, and sea scenes based on pixel spectral features, and using location information in the GIS database to determine the target area in the land scene; Using a pre-stored map of the hyperspectral payload as prior information, the target area is registered with the pre-stored map to extract the key areas of the land scene.

[0022] Specifically, the onboard database contains data from pre-stored spectral payloads on the ground, including dark background time-varying models, pre-launch blind cell positions, Etalon coefficients, relative radiometric calibration coefficients, and spectral calibration coefficients. Combined with spectral correction coefficients, radiometric correction coefficients, and blind cell positions automatically obtained from onboard calibration, on-orbit spectral correction, on-orbit dark level background signal subtraction, on-orbit blind cell repair, on-orbit Etalon effect correction, and on-orbit non-uniformity correction are performed.

[0023] On-orbit spectral calibration is based on the stability of typical absorption positions of atmospheric gas molecules and their sensitivity to the rate of change of absorption positions. Combined with the laboratory spectral response measurement results of the instrument, the relative shift of the center wavelength is analyzed. At the same time, the regularity of the spectral profile of the light-emitting diode is utilized to improve the accuracy of spectral calibration.

[0024] In-orbit radiometric correction analysis examines various factors affecting the photoelectric response of the detector. Based on maximum a posteriori probability theory, an in-orbit adaptive system radiometric correction model is constructed by introducing constraints and weighting. Using calibration data or real-time remote sensing data, an intelligent sample selection model is built, and an incremental statistical strategy is adopted to optimize the solution accuracy of radiometric calibration parameters. Finally, based on the ground radiometric calibration results, the calibration lookup table is updated on the remote sensor, and the remote sensor performs in-orbit non-uniformity correction according to the lookup table to obtain the corrected image.

[0025] The on-orbit dark-level background signal is subtracted by establishing a linear model of the dark background by individually constructing each pixel of the detector array: in, This represents the digital quantization value of pixel (i,j) under no-light conditions. Frame number Let be the slope of the dark level of pixel (i,j) as a function of frame number. The bias for the pixel baseline includes the readout bias and the underlying dark current.

[0026] In-orbit blind pixel restoration is primarily a process of predicting and replacing blind pixel information based on the correlation between adjacent pixels or consecutive frames. First, the continuity of blind pixels is assessed by marking their locations on the image. Continuity is determined in both the spatial and spectral dimensions, and these locations are then marked. Based on the marked spatial continuity results, a blind pixel restoration window is set. Blind pixels within this window are restored using inverse distance weights. After restoration using the spatial dimension, new image data is obtained. Then, based on the number and location of continuous blind pixels in both the spatial and spectral dimensions, a new window is established. The numerical response ratio at the restored blind pixel location is calculated using surrounding pixels. If this ratio exceeds a threshold, an anomaly is detected, and blind pixel restoration is performed in the spectral dimension. The distances from the unknown point to all points are calculated, and these distances are converted into weights. The restored pixel DN value is obtained by multiplying each DN value by its weight and summing the results.

[0027] In-orbit Etalon effect correction requires determining the correction coefficient for each pixel so that the pixel response value after correction is as close as possible to the unaffected response value, i.e.: In the formula, This is the correction factor for the Etalon effect. The corrected pixel response value. This is the pixel response value before correction.

[0028] The on-orbit non-uniformity correction uses a linear correction model: in, This is the final output DN value after radiation correction. These are the DN values ​​after Etalon correction. and This is a non-uniform correction factor, which is related to the load operating conditions and needs to be monitored and updated regularly.

[0029] Further, in step S3, target spatial region information is obtained by performing target detection on the hyperspectral image using a lightweight encoder-decoder convolutional neural network with a dual attention mechanism, including: The hyperspectral image is input into an encoder composed of Blocks 1 and 2 stacked in a pre-defined residual network structure to extract the corresponding target feature representation. The decoder, composed of Blocks 1 and 3 stacked in a pre-defined residual network structure, performs feature recovery, preserving effective target features and extracting spatial feature information with a stronger receptive field. The decoder is a mirror image of the encoder, and Blocks 1, 2, and 3 are different spatial-channel attention modules. Block 1 is used to keep the input feature map size and number of channels unchanged, Block 2 is used to reduce the width and height of the output feature map to half that of the input and double the number of channels, and Block 3 is used to enlarge the width and height of the output feature map to twice that of the input feature map and halve the number of channels, thereby enhancing the spatial and channel features of the target.

[0030] Furthermore, the encoder and decoder of the lightweight convolutional neural network with dual attention mechanism use pre-defined pointwise convolutional layers to achieve target spatial region information detection in hyperspectral images under satellite payload constraints. The weight parameters in the binary quantized pointwise convolutional layers are constrained to be binary, and each convolutional kernel is assigned a pre-defined floating-point coefficient to compress the number of floating-point parameters and multiplication operations.

[0031] Furthermore, network parameters are optimized during the training of a lightweight encoder-decoder convolutional neural network with a dual attention mechanism by using a loss function consisting of a weighted sum of the exponential logarithmic Dice loss function and the weighted exponential cross-entropy loss function to improve the accuracy of target space detection.

[0032] Specifically, such as Figure 3 As shown, the encoder-decoder lightweight convolutional neural network with a dual attention mechanism consists of three spatial-channel attention modules stacked in an ordered manner according to the ResNet-18 network structure, completing the extraction of the main features of the target; the decoder is a mirror structure of the encoder, retaining effective target features and extracting feature information with a stronger receptive field, referring to... Figure 3(a) and Table 1, where Table 1 is a lightweight encoding-decoding convolutional neural network structure with dual attention mechanism.

[0033] Table 1 Where Layer represents the current layer number, ELi represents the i-th layer of the current encoding network, and DLi represents the i-th layer of the current decoding network; the Type parameter represents the type of each convolutional module, where Conv represents one of the three operations: a convolutional layer, a BN normalization layer, and a ReLU activation function, and Blocks 1-3 correspond to the three spatial-channel attention modules; Cin represents the number of input channels of the current module, Cout represents the number of output channels, and Output represents the size of the feature map output after convolution.

[0034] For the lightweight convolutional neural network architecture with dual attention mechanism, the loss function adopted is the exponential logarithmic Dice loss function. ) and weighted exponential cross-entropy loss function ( The weighted sum of the components is used to improve the segmentation accuracy of small-structure targets.

[0035] in, These represent the pixel location, the predicted label, and the actual data label, respectively. This indicates that in the loss function It is a smoothed Dice loss function, that is, adding 1 to both the numerator and denominator of the Dice loss function, which avoids the division by zero problem. and Used to control the nonlinearity of the two-part loss function. and represents the weights of the exponential logarithmic loss function and the weighted exponential cross-entropy loss function, respectively.

[0036] The dual attention mechanism of the encoder-decoder lightweight convolutional neural network uses binary quantized pointwise convolutional layers to replace some pointwise convolutional layers to achieve lightweighting, reduce the storage space consumption of the network model and reduce the amount of multiplication operations in the convolutional and fully connected layers.

[0037] Binary quantization of pointwise convolutional layers constrains all weight parameters in the pointwise convolutional layer to... And assign a floating-point coefficient to each convolution kernel. The forward propagation formula for the binary-quantized pointwise convolutional layer weights is as follows: In the formula, The binary weight matrix represents the convolution kernel. This represents the number of parameters in the convolution kernel. This represents the floating-point weight matrix of the convolution kernel. floating-point coefficients .

[0038] The backpropagation formula for the weights of the binary-quantized pointwise convolutional layer is as follows: In the formula, This represents the derivative propagated back through the floating-point weight matrix. This represents the derivative propagated back through the binary weight matrix.

[0039] refer to Figure 3 In (b), three different spatial-channel attention modules are shown, from left to right: Black1, Black2, and Black3. This indicates the size of the input feature map. Taking Block2 as an example, the convolutional layer in the right branch indicates that the kernel size is 1×1 and the number of output channels is 2×1. The convolutional stride used is 2. In the two convolutional layers of the backbone, the second layer uses the same kernel as the first layer, but the stride is reduced to 1, mainly to achieve information fusion between channels. After two convolutional layers, the channel attention mechanism Wc and the spatial attention mechanism Ws are executed sequentially. The final output feature map has its width and height halved compared to the input, and the number of channels doubles. The main difference between the three attention modules lies in the size of the input and output feature maps and the number of channels. These three modules can be flexibly used to design network structures according to different needs. Block 1 is mainly suitable for situations where the input and output sizes and the number of channels do not change; Block 2 is suitable for situations where the width and height of the feature map need to be halved, while the number of channels of the feature map needs to be doubled; the input and output of Block 3 are exactly the opposite of Block 2.

[0040] Finally, Figure 3 In (c), the left side is the input hyperspectral image, and the right side is the result after network processing. The result shows certain contour information, indicating that the network can detect and segment the spatial region to which the target belongs relatively well.

[0041] Further, in step S3, based on the target spatial region information, the corresponding hyperspectral image is input into a high-low frequency coupled adversarial autoencoder network to separate the high-frequency and low-frequency spectral information and extract high-level features, generating a target spectral fingerprint vector, including... The hyperspectral image is filtered by a low-pass filter network to separate the high-frequency and low-frequency components of the spectrum. High-level features of the high-frequency and low-frequency components of the spectrum are extracted by high-frequency deep autoencoder networks and low-frequency deep autoencoder networks respectively, and then fused. The input spectral vector and the fused high-level features are combined and fed into the adversarial discriminant network to generate an adversarial signal to update and optimize the parameters of the high-frequency deep autoencoder network and the low-frequency deep autoencoder network, thereby obtaining the corresponding target spectral high-level features to form the target spectral fingerprint vector.

[0042] Specifically, such as Figure 4 As shown, the high- and low-frequency coupled adversarial autoencoder network structure mainly includes: Low-pass filter networks constrain spectral energy The high-frequency and low-frequency components of the separated spectrum are included. It represents the L1 paradigm for the high-frequency components of the spectrum.

[0043] High-frequency deep autoencoder networks and low-frequency deep autoencoder networks extract high-frequency components of the spectrum, respectively. and low frequency components To ensure the extraction of high-level features, spectral angular distance is added as a reconstruction loss function to the mean squared error (MSE) function, i.e.: In the formula, The spectral characteristic constraint coefficient is used to constrain... The spectral feature distortion caused by excessive size is characterized by the spectral constraint term from the perspective of the spectral feature vectors before and after reconstruction. .

[0044] The inverse cosine function It is a monotonically decreasing function with a maximum value of π. The loss function is normalized to [0,1] by dividing the spectral characteristic angle by π. The smaller the value, the more similar the two spectral feature vectors are, and the more convergent the model becomes.

[0045] The training process of the network is mainly divided into two stages: Phase 1: Update the encoding and decoding network parameters of the two autoencoders respectively, so that the reconstruction loss function... and Minimize the output to obtain an output similar to the high-frequency and low-frequency components of the input. Then fuse the reconstructed high-frequency and low-frequency components.

[0046] The second stage involves updating the parameters of the discriminator network to distinguish between real samples (input spectral vectors) and fake samples (samples fused from high and low frequency components reconstructed by the autoencoder); then updating the parameters of the generator network (two autoencoder networks), and training the generator network and the discriminator network alternately to improve the ability of the discriminator network to confuse the discriminator network.

[0047] After training, the encoding layer will obtain a mapping from the input to high-level features. Combining the high-level features output by the two encoders yields the dimensionality-reduced high-level features. .

[0048] Further, in step S4, a hyperspectral multi-objective optimization sparse unmixing model is used to fuse the spectral pixel vectors to generate a target spectral fingerprint prototype and construct a hybrid pixel spectral database, including: Each spectrum in the spectral library is binary-encoded using a hyperspectral multi-objective optimization sparse unmixing model, and the multi-objective optimization problem for each spectrum is decomposed into several single-objective sub-problems. A random flipping strategy is used to generate offspring individuals, and spectral features are added as a regularization term to the Chebyshev decomposition function to determine whether to update the individual and its neighboring individuals. After reaching the preset iteration threshold, the spectrum corresponding to the optimal individual of each subproblem is extracted, and the abundance matrix is ​​solved by non-negative least squares to generate the target spectral fingerprint prototype and store it in the mixed pixel spectral database.

[0049] Specifically, such as Figure 5 As shown, the execution flow of the hyperspectral multi-objective optimization sparse unmixing model mainly includes: By binary encoding the spectral library, the sparsity of the library can be directly expressed. The multi-objective optimization of each spectrum is decomposed into multiple single-objective sub-problems, each consisting of a corresponding individual and several neighboring individuals. For each individual in a sub-problem, a random flipping strategy is used to generate offspring individuals. An improved Chebyshev decomposition function based on spectral features determines whether to update the individual and its neighboring individuals; if an update is needed, it replaces the current individual with a newly generated one, and this process is repeated. After reaching the maximum number of iterations, the endmembers corresponding to the obtained individuals are extracted. The abundance matrix is ​​then calculated using non-negative least squares as the spectral fingerprint vector of the objective. The improved Chebyshev decomposition function based on spectral features is as follows: The optimal solution of this equation corresponds to a Pareto optimal solution to the original multi-objective optimization problem. Let i be the objective function of the i-th subproblem after decomposition. The original Chebyshev decomposition function, for The corresponding binary code, and Individuals and The corresponding subset of the spectral library, The regularization coefficient is . for and Spectral information divergence.

[0050] Further, in step S4, the target spectral fingerprint vector is input into the relation measurement network and matched with the target spectral fingerprint prototype in the mixed pixel spectral database to obtain the corresponding target type. The extracted target attribute information is then framed and output. This includes inputting the target spectral fingerprint vector and the target spectral fingerprint prototype in the mixed pixel spectral database into the relation measurement network, obtaining the corresponding relation score by comparing the similarity between the target's high-dimensional features and the target library sample prototype, determining the target type based on the relation score, and framing and outputting the target category, height, and location information.

[0051] Further, in step S4, the relation measurement network includes a first two-dimensional convolutional block, a second two-dimensional convolutional block, a first fully connected layer, and a second fully connected layer that are sequentially connected. The first two-dimensional convolutional block and the second two-dimensional convolutional block each include a convolutional layer, a batch normalization layer, and a ReLU activation layer. The first fully connected layer uses the ReLU activation function, and the second fully connected layer uses the Sigmoid function. The loss function of the relation measurement network is the mean squared error, which outputs a score on the relationship between the target spectral fingerprint vector and the target spectral fingerprint prototype.

[0052] Specifically, see Figure 6 (a) The relation metric network structure mainly consists of two two-dimensional convolutional blocks and two fully connected layers. Except for the last fully connected layer which uses the Sigmoid function, the activation functions of the other fully connected layers are all ReLU functions.

[0053] The relation metric network uses a convolutional neural network to compare the similarity between the target spectral fingerprint prototype and the target spectral fingerprint vector to obtain a relation score. Relationships belonging to the same target class have higher scores, while those belonging to different classes have lower scores. Specifically, the target spectral fingerprint vector is a data cube corresponding to each pixel in the hyperspectral remote sensing image data. Samples in the support set are mapped to embedding features in the embedding space using a projection function. The class prototype for each class is the mean vector of the support set samples for that class in the embedding space. Samples in the query set are also mapped to embedding features in the embedding space using a projection function. Then, the embedding features and the class prototype corresponding to each class are concatenated in the depth dimension.

[0054] See Figure 6(b) and Table 2. Table 2 shows the recognition accuracy of the relation metric network. The recognition results are from left to right: pseudo-color image, reference image, and recognition result image. From top to bottom, the recognition results are extracted from four public datasets: Cuprite, Cat Island, San Diego, and BayChampagne. It can be seen that the highlighted parts in the recognition result image are basically consistent with the reference image, indicating that the relation metric network has a good recognition effect.

[0055] Table 2 Second Embodiment Based on the same inventive concept, such as Figure 7 As shown, this invention provides a method for real-time on-orbit target detection and identification of hyperspectral payloads, employing the method described above, including: The image correction module is used to receive the original hyperspectral image, and based on the spectral correction coefficient, radiometric correction coefficient and blind pixel position automatically obtained by on-board calibration, combined with the on-board basic coefficient library, it performs spectral and radiometric processing on the original hyperspectral image to generate a hyperspectral image. The process of generating the hyperspectral image also includes scene segmentation of the hyperspectral image, and for images of land scenes, it extracts key areas by combining GIS library and prior information. The target detection module is used to perform target detection on hyperspectral images using a lightweight encoder-decoder convolutional neural network with a dual attention mechanism to obtain target spatial region information. Based on the target spatial region information, the corresponding hyperspectral image is input into a high-low frequency coupled adversarial autoencoder network to separate high-frequency and low-frequency spectral information and extract high-level features, generating a target spectral fingerprint vector. The high-low frequency coupled adversarial autoencoder network consists of a low-pass filter network, a high-frequency deep autoencoder network, a low-frequency deep autoencoder network, and an adversarial discriminant network. The target recognition module is used to fuse spectral pixel vectors using a hyperspectral multi-objective optimized sparse unmixing model to generate target spectral fingerprint prototypes and construct a hybrid pixel spectral database. The target spectral fingerprint vectors are input into a relational metric network and matched with the target spectral fingerprint prototypes in the hybrid pixel spectral database to identify the corresponding target type. The extracted target attribute information is then used to describe and output the frame.

[0056] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A method for real-time on-orbit target detection and identification of a hyperspectral payload, characterized in that, include, Step S1: Receive the raw hyperspectral image; Step S2: Based on the spectral correction coefficients, radiometric correction coefficients, and blind pixel positions automatically obtained from onboard calibration, and combined with the onboard basic coefficient library, the hyperspectral original image is subjected to spectral and radiometric processing to generate a hyperspectral image. The process of generating the hyperspectral image also includes scene segmentation of the hyperspectral image, and for images of land scenes, key areas are extracted by combining the GIS library and prior information. Step S3: Target spatial region information is obtained by performing target detection on the hyperspectral image using a lightweight encoder-decoder convolutional neural network with a dual attention mechanism. Based on the target spatial region information, the corresponding hyperspectral image is input into a high-low frequency coupled adversarial autoencoder network to separate high-frequency and low-frequency spectral information and extract high-level features, generating a target spectral fingerprint vector. The high-low frequency coupled adversarial autoencoder network consists of a low-pass filter network, a high-frequency deep autoencoder network, a low-frequency deep autoencoder network, and an adversarial discriminant network. Step S4: Use a hyperspectral multi-objective optimization sparse unmixing model to fuse spectral pixel vectors, generate target spectral fingerprint prototypes, and construct a hybrid pixel spectral database; input the target spectral fingerprint vectors into a relational metric network, match and identify the target spectral fingerprint prototypes in the hybrid pixel spectral database to obtain the corresponding target type, and output the extracted target attribute information description frame.

2. The method for real-time on-orbit target detection and identification of hyperspectral payloads according to claim 1, characterized in that, In step S2, based on the spectral correction coefficients, radiometric correction coefficients, and blind pixel positions automatically obtained from onboard calibration, and in conjunction with the onboard fundamental coefficient library, the original hyperspectral image is subjected to spectral and radiometric processing to generate a hyperspectral image. include, The spectral correction coefficients, radiometric correction coefficients, and blind cell positions are automatically obtained through the on-board calibration, and the spectral payload data pre-stored in the on-board basic coefficient library is called. The spectral payload data includes a dark background time-varying model, pre-launch blind cell positions, Etalon coefficients, relative radiometric calibration coefficients, and spectral calibration coefficients. The hyperspectral original image is subjected to on-orbit spectral correction based on the spectral correction coefficient and the spectral calibration coefficient to correct the lateral spectral deviation and compensate for the drift when a drift in the center wavelength of the band is detected. The original hyperspectral image after on-orbit spectral correction is used to perform on-orbit dark background signal subtraction using the dark background time-varying model and the radiometric correction coefficient to eliminate detector dark level background noise. Based on the blind pixel location and the pre-launch blind pixel location, the hyperspectral original image after deducting the on-orbit dark background signal is repaired in orbit to restore the image response value at the abnormal pixel. Based on the Etalon coefficients, the image for in-orbit blind pixel restoration is subjected to in-orbit Etalon correction, and after in-orbit non-uniformity correction based on the relative radiometric calibration coefficients, the hyperspectral image after spectral and radiometric processing is output.

3. The method for real-time on-orbit target detection and identification of hyperspectral payloads according to claim 1, characterized in that, In step S3, target spatial region information is obtained by performing target detection on the hyperspectral image using a lightweight encoder-decoder convolutional neural network with a dual attention mechanism, including: The hyperspectral image is input into an encoder composed of Blocks 1 and 2 stacked in a preset residual network structure to extract the corresponding target feature representation. The decoder, composed of Blocks 1 and 3 stacked in a preset residual network structure, performs feature recovery, retaining effective target features and extracting spatial feature information with a stronger receptive field. The decoder is a mirror structure of the encoder, and Blocks 1, 2, and 3 are different spatial-channel attention modules. Block 1 is used to keep the input feature map size and number of channels unchanged, Block 2 is used to reduce the width and height of the output feature map to half that of the input and double the number of channels, and Block 3 is used to enlarge the width and height of the output feature map to twice that of the input feature map and halve the number of channels, thereby enhancing the spatial and channel features of the target.

4. The method for real-time on-orbit target detection and identification of hyperspectral payloads according to claim 3, characterized in that, The encoder and decoder in the lightweight convolutional neural network with dual attention mechanism achieve target spatial region information detection of the hyperspectral image through the preset pointwise convolutional layers of the encoder and decoder under the condition of satellite payload limitation by using binary quantized pointwise convolutional layers; the weight parameters in the binary quantized pointwise convolutional layers are constrained to be binary, and each convolutional kernel is assigned a preset floating-point coefficient to compress the number of floating-point parameters and multiplication operations.

5. The method for real-time on-orbit target detection and identification of hyperspectral payloads according to claim 3, characterized in that, The network parameters are optimized during the training of the encoder-decoder lightweight convolutional neural network with dual attention mechanism by using a loss function consisting of a weighted sum of the exponential logarithmic Dice loss function and the weighted exponential cross-entropy loss function to improve the accuracy of target space detection.

6. The method for real-time on-orbit target detection and identification of hyperspectral payloads according to claim 1, characterized in that, In step S3, based on the target spatial region information, the corresponding hyperspectral image is input into a high-low frequency coupled adversarial autoencoder network to separate the high-frequency and low-frequency spectral information and extract high-level features, generating a target spectral fingerprint vector, including: The hyperspectral image is filtered by the low-pass filter network to separate the high-frequency and low-frequency components of the spectrum. The high-level features of the spectral high-frequency component and the spectral low-frequency component are extracted by the high-frequency deep autoencoder network and the low-frequency deep autoencoder network respectively, and then fused. The input spectral vector and the fused high-level features are combined and input into the adversarial discriminant network to generate an adversarial signal to update and optimize the parameters of the high-frequency deep autoencoder network and the low-frequency deep autoencoder network, thereby obtaining the corresponding target spectral high-level features to form the target spectral fingerprint vector.

7. The method for real-time on-orbit target detection and identification of hyperspectral payloads according to claim 6, characterized in that, In step S4, a hyperspectral multi-objective optimization sparse unmixing model is used to fuse spectral pixel vectors, generating a target spectral fingerprint prototype and constructing a hybrid pixel spectral database, including... The hyperspectral multi-objective optimization sparse unmixing model is used to encode each spectrum in the spectral library in binary form, and the multi-objective optimization problem for each spectrum is decomposed into several single-objective sub-problems. A random flipping strategy is used to generate offspring individuals, and spectral features are added as a regularization term to the Chebyshev decomposition function to determine whether to update the individual and its neighboring individuals. After reaching the preset iteration threshold, the spectrum corresponding to the optimal individual of each subproblem is extracted, and the abundance matrix is ​​solved by non-negative least squares to generate the target spectral fingerprint prototype and store it in the mixed pixel spectral database.

8. The method for real-time on-orbit target detection and identification of hyperspectral payloads according to claim 7, characterized in that, In step S4, the target spectral fingerprint vector is input into the relational metric network and matched with the target spectral fingerprint prototypes in the mixed pixel spectral database to obtain the corresponding target type. The extracted target attribute information description frame is then output, including... The target spectral fingerprint vector and the target spectral fingerprint prototype in the hybrid pixel spectral database are input into the relation measurement network. The corresponding relation score is obtained by comparing the similarity between the target high-dimensional features and the target library sample prototype. The target type is determined based on the relation score, and the target category, height, and location information are framed and output.

9. The method for real-time on-orbit target detection and identification of hyperspectral payloads according to claim 8, characterized in that, In step S4, the relation measurement network includes a first two-dimensional convolutional block, a second two-dimensional convolutional block, a first fully connected layer, and a second fully connected layer that are sequentially connected. The first two-dimensional convolutional block and the second two-dimensional convolutional block each include a convolutional layer, a batch normalization layer, and a ReLU activation layer. The first fully connected layer uses the ReLU activation function, and the second fully connected layer uses the Sigmoid function. The loss function of the relation measurement network is the mean squared error, which outputs the relation score between the target spectral fingerprint vector and the target spectral fingerprint prototype.

10. A method for real-time on-orbit target detection and recognition of a hyperspectral payload, employing the method for real-time on-orbit target detection and recognition of a hyperspectral payload as described in any one of claims 1 to 9, characterized in that, include, The image correction module is used to receive the original hyperspectral image, and based on the spectral correction coefficient, radiometric correction coefficient and blind pixel position automatically obtained by on-board calibration, and combined with the on-board basic coefficient library, to perform spectral and radiometric processing on the original hyperspectral image to generate a hyperspectral image. The process of generating the hyperspectral image also includes scene segmentation of the hyperspectral image, and for images of land scenes, key areas are extracted by combining GIS library and prior information. The target detection module is used to perform target detection on the hyperspectral image using a lightweight encoder-decoder convolutional neural network with a dual attention mechanism to obtain target spatial region information. Based on the target spatial region information, the corresponding hyperspectral image is input to a high-low frequency coupled adversarial autoencoder network to separate the high-frequency and low-frequency spectral information and extract high-level features, generating a target spectral fingerprint vector. The high-low frequency coupled adversarial autoencoder network consists of a low-pass filter network, a high-frequency deep autoencoder network, a low-frequency deep autoencoder network, and an adversarial discriminant network. The target recognition module is used to fuse spectral pixel vectors using a hyperspectral multi-objective optimized sparse unmixing model to generate target spectral fingerprint prototypes and construct a hybrid pixel spectral database. The target spectral fingerprint vectors are input into a relational metric network and matched with the target spectral fingerprint prototypes in the hybrid pixel spectral database to obtain the corresponding target type. The extracted target attribute information is then used to describe and output the frame.

Images