Hyperspectral remote sensing image registration method based on representation learning

Abstract

This invention relates to a hyperspectral remote sensing image registration method based on representation learning, comprising: acquiring a set of hyperspectral images captured by a hyperspectral remote sensing satellite and preprocessing them; constructing an adversarial generative network and performing multi-band feature learning and feature mapping for different band ranges; constructing a multi-band structured semantic CNN-RNN hybrid neural network deep representation model to extract the structural semantic information of each band of the hyperspectral image; and establishing a network model for cross-band feature alignment and feature association to match images between different bands. This invention can improve the accuracy of hyperspectral remote sensing image registration and significantly save registration efficiency, manpower, and resources.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image technology, and in particular to a hyperspectral remote sensing image registration method based on representation learning. Background Technology

[0002] Hyperspectral imagery is a collection and processing of information across the entire electromagnetic spectrum, with detailed segmentation along the spectral dimensions, resulting in rich spectral information. Hyperspectral remote sensing combines imaging and spectral detection techniques. While imaging spatial features, it disperses each spatial pixel to create dozens or even hundreds of narrow bands for continuous spectral coverage. The images are acquired and synthesized from several different types of sensors, with each band consisting of numerous channels, such as the near-infrared, infrared, visible, and shortwave ranges. Utilizing their different imaging principles, different properties of surface objects can be obtained. Therefore, hyperspectral remote sensing imagery offers significantly richer information than traditional remote sensing imagery. However, while the visible light band aligns with human visual habits, other bands are often difficult for humans to identify. Therefore, a universal registration method is needed to assist in registering different bands of hyperspectral images.

[0003] The basic methods for hyperspectral image registration are mainly divided into two types: gray-level-based methods and feature-based methods. Gray-level-based methods mainly include wavelet transform, cross-correlation, and phase transform methods. However, for hyperspectral remote sensing applications, the resolution scales between bands differ significantly, making these methods difficult to use directly. Feature-based methods mainly include geometric feature methods, edge intensity feature methods, and keypoint feature methods. However, hyperspectral remote sensing images typically have hundreds or thousands of bands, containing a large amount of redundant information. Their features often cannot be represented by a single algorithm. For example, the average band information method, linear discriminant analysis (LDA), and principal component analysis (PCA) are only effective on specific data. In practice, registration often requires manual selection of different methods based on the characteristics of different data, which is inefficient and consumes a lot of manpower and resources.

[0004] Given that hyperspectral remote sensing imagery has unique advantages that other remote sensing imagery does not possess, containing rich spatial, radiometric, and spectral information, covering fields such as the atmosphere, ocean research, vegetation ecology, mineral geology, water body research, and archaeological research, its application is entering a mature stage and has important comprehensive application value. In the future, the amount of data will inevitably increase significantly. However, the existing registration methods for hyperspectral remote sensing imagery are difficult to meet future needs. Summary of the Invention

[0005] To address the technical problems existing in the prior art, the present invention aims to provide a hyperspectral remote sensing image registration method based on representation learning, thereby achieving automated registration of hyperspectral remote sensing images.

[0006] To achieve the above-mentioned objectives, this invention provides a hyperspectral remote sensing image registration method based on representation learning, comprising the following steps:

[0007] Step S1: Acquire a set of hyperspectral images captured by a hyperspectral remote sensing satellite and preprocess them;

[0008] Step S2: Construct an adversarial generative network and perform multi-band feature learning and feature mapping for different band ranges;

[0009] Step S3: Construct a multi-band structured semantic CNN-RNN hybrid neural network deep representation model to extract the structural semantic information of each band of hyperspectral image;

[0010] Step S4: Establish a network model for cross-band feature alignment and feature association to match images between different bands.

[0011] According to one technical solution of the present invention, it further includes:

[0012] Step S5: Perform multi-band feature fusion on hyperspectral data of different bands to verify the registration results and obtain hyperspectral images after fusion of high-level semantic features between different bands.

[0013] According to one technical solution of the present invention, in step S1, a set of hyperspectral images consists of multiple hyperspectral images of different band ranges taken by the same spectral remote sensing satellite at the same time and in the same area;

[0014] The preprocessing steps include at least strip noise removal, data compression, radiometric calibration, atmospheric correction, and geometric correction.

[0015] According to one technical solution of the present invention, in step S2, the adversarial generative network includes a feature extractor, a band extractor, and a discriminator;

[0016] The loss function of the discriminator is a binary cross-classification loss function, as shown in the following formula:

[0017] L(D) = -E xp(i) [logD(x)]-E zp(j) [log(1-D(T(z)))]

[0018] Where i = 0, 1 indicates whether the input image is the original image or the extracted image, E xp(i) This indicates that image x conforms to the band distribution of p(i), E zp(j)Let z represent the original image, T(z) represent the transformed image, x represent the image of another mode corresponding to z, and D(x) represent the output probability distribution of the discriminator.

[0019] According to a technical solution of the present invention, in step S3, the CNN model generates image feature representation with discriminative ability, and the RNN model recursively learns semantic and structured representations at the same time. The feature representation of each semantic category output by the CNN model is used as input to predict the structured combination relationship in the image, generate the image parse tree, perform hierarchical representation of the image content, and predict the interaction relationship between semantic entities, thereby completing the extraction of deep semantic information of each band of the hyperspectral remote sensing image.

[0020] According to one technical solution of the present invention, step S4 specifically includes:

[0021] A network model for cross-band feature alignment and association is established. Images from different bands are segmented according to their features, and the segmented images are fed into the network model to obtain block features. The cross-entropy loss function is used to classify the block features, marking features belonging to the same block as 1 and different features as 0. At the same time, a metric learning method is introduced, and an additional triplet loss is constructed to calculate the loss function for positive and negative samples, as shown in the following expression:

[0022] L band =max(d p -d n +m,0)

[0023] Where, d p ,d n , respectively, are the distances between positive and negative sample pairs, and m is a constant used to increase the distance between negative sample pairs;

[0024] Finally, the positive and negative samples of multiple bands are matched and aligned to achieve image registration between different bands.

[0025] According to one technical solution of the present invention, in step S5, the resolution of different bands is normalized, common feature sub-controls between different bands are found, and a cross-fusion strategy is adopted for mapping and fusion.

[0026] The cross-fusion strategy is expressed as follows:

[0027] For the i-th pixel in the s-th band range, its feature vector in the p-th band is: For the cross-fusion network, the parameters of the two branches are first used to perform calculations on the two input feature maps respectively, and then the results are added together. Therefore:

[0028]

[0029] The final output of the band fusion layer can be expressed as:

[0030]

[0031] According to one aspect of the present invention, an electronic device is provided, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory; when the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform a hyperspectral remote sensing image registration method based on representation learning as described in any of the above technical solutions.

[0032] According to one aspect of the present invention, a computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement a hyperspectral remote sensing image registration method based on representation learning as described in any of the above technical solutions.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] This invention proposes a hyperspectral remote sensing image registration method based on representation learning. When performing inter-band registration of hyperspectral remote sensing images, a representation learning method is used, employing an adversarial generative network and a CNN-RNN network, to extract deep representation information of different bands of the hyperspectral remote sensing image. Then, matching is performed based on this deep representation information to achieve registration of multiple bands of the hyperspectral remote sensing image. The above multiple bands are then fused and mapped together, which not only achieves efficient and accurate inter-band registration of hyperspectral remote sensing images, but also enables more effective utilization of hyperspectral remote sensing images in the later stages, while also greatly saving manpower and resources.

[0035] Compared with traditional methods, this invention improves the representation ability of hyperspectral images and increases the utilization rate of hyperspectral remote sensing data by constructing a multi-band structured semantic CNN-RNN hybrid neural network to extract deep representation information of hyperspectral images, rather than just using geometric information and band frequency information. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0037] Figure 1 The diagram illustrates the overall flowchart of a hyperspectral remote sensing image registration method based on representation learning according to one embodiment of the present invention.

[0038] Figure 2 The diagram illustrates the overall flowchart of a hyperspectral remote sensing image registration method based on representation learning according to another embodiment of the present invention.

[0039] Figure 3 The diagram illustrates the overall flowchart of a hyperspectral remote sensing image registration method based on representation learning according to another embodiment of the present invention. Detailed Implementation

[0040] The description of the embodiments in this specification should be taken in conjunction with the accompanying drawings, which should form part of the complete specification. In the drawings, the shape or thickness of the embodiments may be exaggerated and may be indicated in a simplified or convenient manner. Furthermore, parts of the various structures in the drawings will be described separately; it is worth noting that elements not shown in the figures or not described in words are in a form known to those skilled in the art.

[0041] The descriptions of the embodiments herein, including any references to directions and orientations, are for ease of description only and should not be construed as limiting the scope of the invention. The following description of preferred embodiments involves combinations of features, which may exist independently or in combination; the invention is not particularly limited to the preferred embodiments. The scope of the invention is defined by the claims.

[0042] This invention discloses a hyperspectral remote sensing image registration method based on representation learning. By using a generative adversarial network and a CNN-RNN network, deep representation information of different bands of hyperspectral remote sensing images is extracted. Then, matching is performed based on this deep representation information to achieve registration of multiple bands of hyperspectral remote sensing images. The above multiple bands are then fused and mapped together, which not only achieves high-efficiency and high-accuracy registration of hyperspectral remote sensing images, but also enables more effective utilization of hyperspectral remote sensing images in the later stages.

[0043] like Figures 1 to 3 As shown, a hyperspectral remote sensing image registration method based on representation learning according to the present invention includes the following steps:

[0044] Step S1: Acquire a set of hyperspectral images captured by a hyperspectral remote sensing satellite and preprocess them;

[0045] Step S2: Construct an adversarial generative network and perform multi-band feature learning and feature mapping for different band ranges;

[0046] Step S3: Construct a multi-band structured semantic CNN-RNN hybrid neural network deep representation model to extract the structural semantic information of each band of hyperspectral image;

[0047] Step S4: Establish a network model for cross-band feature alignment and feature association to match images between different bands;

[0048] Step S5: Perform multi-band feature fusion on hyperspectral data of different bands to verify the registration results and obtain hyperspectral images after fusion of high-level semantic features between different bands.

[0049] In some embodiments of the present invention, in step S1, firstly, a set of hyperspectral images of different band ranges (near-infrared, infrared, visible light, shortwave, midwave, and longwave) of the same area at the same time are acquired by a hyperspectral remote sensing satellite, and then the hyperspectral images are preprocessed.

[0050] Preprocessing of hyperspectral remote sensing images mainly includes strip noise removal, data compression, radiometric calibration, atmospheric correction, and geometric correction.

[0051] For example, hyperspectral remote sensing images were acquired in three bands: near-infrared (NIR), visible light (VIS), and mid-infrared (MIR).

[0052] The near-infrared (NIR), visible (VIS), and mid-infrared (MIR) bands of hyperspectral remote sensing images were used as examples. The wavelengths of the visible band are 780–400 nm, the near-infrared band is 780–2526 nm, and the mid-infrared band is 3000–5000 nm. The registration of hyperspectral remote sensing images was then carried out using these three bands as examples.

[0053] In some embodiments of the present invention, in step S2, the adversarial generative network includes a feature extractor, a band extractor, and a discriminator;

[0054] The antibody component is incorporated into the training phase of the discriminator to enable the feature extractor to extract invariant features, i.e., to minimize the discriminator's loss function value. The discriminator's loss function is a binary cross-classification loss function, as shown in the following formula:

[0055] L(D) = -E xp(i) [logD(x)]-E zp(j) [log(1-D(T(z)))]

[0056] Where i = 0, 1 indicates whether the input image is the original image or the extracted image, E xp(i) This indicates that image x conforms to the band distribution of p(i), E zp(j)Let z represent the original image, T(z) represent the transformed image, x represent the image of another mode corresponding to z, and D(x) represent the output probability distribution of the discriminator.

[0057] During discriminator training, the goal is for the discriminator to distinguish between the source domain and the feature domain as accurately as possible, meaning the classification loss of the domain classifier should be minimized. Simultaneously, the loss function of the classifier should be minimized. After network training, given a sample, if the classifier can accurately identify the label value, while the domain classifier cannot distinguish whether the sample belongs to the source domain or the feature domain, then the network has learned invariant features, and the feature samples can achieve good classification accuracy in the classifier. When the domain classifier reaches Nash equilibrium and cannot distinguish whether the label classifier comes from the source domain or the feature domain, it can be considered that the feature distribution difference between the source and feature domains has been eliminated during the feature stage.

[0058] In some embodiments of the present invention, in step S3, a multi-band structured semantic CNN-RNN hybrid neural network deep representation model is constructed to extract the structural semantic information of each band of the hyperspectral image. The CNN model generates image feature representations with discriminative capabilities, while the RNN model recursively learns both semantic and structured representations, ultimately achieving representation learning, i.e., the representation of information for each band.

[0059] The constructed CNN-RNN hybrid neural network image semantic representation model has the following characteristics. This model integrates a series of structured grammatical knowledge into deep representation learning, leveraging the characteristics of recurrent neural networks to structurally combine and represent feature relationships in images. The final model achieves deep, hierarchical semantic parsing of images.

[0060] In constructing a hybrid CNN-RNN neural network for image semantic representation, the CNN and RNN models work collaboratively. The CNN model generates discriminative image feature representations, while the RNN model recursively learns both semantic and structured representations. Taking the feature representations of each semantic category output by the CNN model as input, it predicts the structured combinations in the image, i.e., generates a parse tree of the image, performing hierarchical representation of the image content, and predicting the interaction relationships between semantic entities. This completes the extraction of deep semantic information for each band of the hyperspectral remote sensing image, thereby achieving representation learning—the representation of each hyperspectral band. The CNN model generates discriminative image feature representations, while the RNN model recursively learns both semantic and structured representations, ultimately achieving representation learning—the representation of information for each band.

[0061] In some embodiments of the present invention, step S4 specifically includes:

[0062] A cross-band feature alignment and feature association network model is constructed to match images between different bands. The feature maps after feature extraction of images from different bands are used to calculate and sort the block feature similarity, thereby aligning and registering the block features of different bands.

[0063] Registration is achieved by associating features across three different wavelength bands: visible light, near-infrared, and mid-infrared. First, images of each band are segmented based on their features, and the segmented images are then fed into the network. Since the feature distribution varies significantly across bands, the feature extractor uses three channels with non-shared weights. After obtaining the block features, the cross-entropy loss function is used to classify the features, labeling features belonging to the same block as 1 and features from different blocks as 0. Simultaneously, a metric learning method is introduced, employing a triplet loss to construct a loss function for calculating both positive and negative samples. Its expression is as follows:

[0064] L band =max(d p -d n +m,0)

[0065] Where, d p ,d n , where are the distances between positive and negative sample pairs, and m is a constant used to increase the distance between negative sample pairs. This loss function aims to shorten the distance between positive sample intervals while increasing the distance between negative sample pairs. This allows positive and negative sample pairs to be distinguished using Euclidean distance or the cosine theorem between block features.

[0066] Finally, the positive and negative samples of multiple bands are matched and aligned to achieve image registration between different bands.

[0067] In some embodiments of the present invention, in step S5, multi-band feature fusion is performed on hyperspectral data of different bands to verify the registration result and complete the final image output of the hyperspectral remote sensing image, resulting in a set of hyperspectral images fused in high-level semantic features between different bands.

[0068] Taking the three bands as an example again, the resolutions of the visible light band, near-infrared band, and mid-infrared band are normalized. In order to better represent features among the three bands, common feature sub-controls are found among multiple different bands, and a cross-fusion strategy is adopted for mapping and fusion.

[0069] The cross-fusion strategy is expressed as follows:

[0070] For the i-th pixel in the s-th band range, its feature vector in the p-th band is: For the cross-fusion network, the parameters of the two branches are first used to perform calculations on the two input feature maps respectively, and then the results are added together. Therefore:

[0071]

[0072] The final output of the band fusion layer can be expressed as:

[0073]

[0074] According to one aspect of the present invention, an electronic device is provided, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory; when the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform a hyperspectral remote sensing image registration method based on representation learning as described in any of the above technical solutions.

[0075] According to one aspect of the present invention, a computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement a hyperspectral remote sensing image registration method based on representation learning as described in any of the above technical solutions.

[0076] Computer-readable storage media can include any medium capable of storing or transmitting information. Examples of computer-readable storage media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and so on. Code segments can be downloaded via computer networks such as the Internet and intranets.

[0077] Furthermore, it should be noted that the present invention can be provided as a method, apparatus, or computer program product. Therefore, embodiments of the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0078] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0079] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0080] It should also be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0081] Finally, it should be noted that the above description represents a preferred embodiment of the present invention. It should be pointed out that although preferred embodiments have been described, those skilled in the art, once they understand the basic inventive concept of the present invention, can make various improvements and modifications without departing from the principles described herein. These improvements and modifications should also be considered within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present invention.

Claims

1. A hyperspectral remote sensing image registration method based on representation learning, characterized in that, Includes the following steps: Step S1: Acquire a set of hyperspectral images captured by a hyperspectral remote sensing satellite and preprocess them; Step S2: Construct an adversarial generative network and perform multi-band feature learning and feature mapping for different band ranges; extract invariant features based on the feature extractor of the adversarial generative network. Step S3: Construct a multi-band structured semantic CNN-RNN hybrid neural network deep representation model to extract the structural semantic information of each band of hyperspectral image; The CNN model generates image feature representations with discriminative capabilities, while the RNN model recursively learns both semantic and structured representations. It takes the feature representation of each semantic category output by the CNN model as input, predicts the structured combination relationships in the image, generates the image parse tree, performs hierarchical representation of the image content, and predicts the interaction relationships between semantic entities, thus completing the extraction of deep semantic information for each band of the hyperspectral remote sensing image. Step S4: Establish a network model for cross-band feature alignment and feature association to match images across different bands, specifically including: A network model for cross-band feature alignment and association is established. Images from different bands are segmented according to their features, and the segmented images are fed into the network model to obtain block features. The cross-entropy loss function is used to classify the block features, marking features belonging to the same block as 1 and different features as 0. At the same time, a metric learning method is introduced, and an additional triplet loss is constructed to calculate the loss function for positive and negative samples, as shown in the following expression: in, , respectively, are the distances between positive and negative sample pairs, and m is a constant used to increase the distance between negative sample pairs; Finally, the positive and negative samples of multiple bands are matched and aligned to achieve image registration between different bands; Step S5: Perform multi-band feature fusion on hyperspectral data of different bands to verify the registration results and obtain hyperspectral images after fusion of high-level semantic features between different bands.

2. The hyperspectral remote sensing image registration method based on representation learning according to claim 1, characterized in that, In step S1, a set of hyperspectral images consists of multiple hyperspectral images of different band ranges taken by the same spectral remote sensing satellite at the same time and in the same area. The preprocessing steps include at least strip noise removal, data compression, radiometric calibration, atmospheric correction, and geometric correction.

3. The hyperspectral remote sensing image registration method based on representation learning according to claim 1, characterized in that, In step S2, the adversarial generative network includes a feature extractor, a band extractor, and a discriminator. The loss function of the discriminator is a binary cross-classification loss function, as shown in the following formula: Where i = 0, 1 indicates whether the input image is the original image or the extracted image. This indicates that image x conforms to the band distribution of p(i). Let z represent the original image, T(z) represent the transformed image, x represent the image of another mode corresponding to z, and D(x) represent the output probability distribution of the discriminator.

4. An electronic device, characterized in that, include: One or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the hyperspectral remote sensing image registration method based on representation learning as described in any one of claims 1 to 3.

5. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, implement the hyperspectral remote sensing image registration method based on representation learning as described in any one of claims 1 to 3.

Images