Hyperspectral image feature representation method and device, electronic equipment and storage medium

Abstract

The application provides a hyperspectral image feature representation method and device, electronic equipment and storage medium, and relates to the technical field of image processing.The method comprises the following steps: acquiring multi-view information of a hyperspectral image, wherein the multi-view information comprises first-view information of the hyperspectral image in a spatial dimension and second-view information of the hyperspectral image in a spectral dimension; performing image enhancement processing on the multi-view information to obtain multi-view information after image enhancement processing, wherein the image enhancement processing comprises a random cropping operation on the first-view information and a random sampling operation on the second-view information; inputting the multi-view information after image enhancement processing into a pre-constructed multi-view contrast learning deep feature representation model to obtain deep feature representation output by the multi-view contrast learning deep feature representation model.The application maximizes mutual information between different views specific to the hyperspectral image, and greatly improves the performance of hyperspectral image feature representation.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, electronic device, and storage medium for representing hyperspectral image features. Background Technology

[0002] Hyperspectral image feature representation is one of the key technologies in the field of hyperspectral image processing. It mainly involves extracting or selecting features with discriminative information from hyperspectral images to provide a basis for relevant decisions (such as classification and recognition).

[0003] Currently, contrastive learning is the mainstream method for image feature representation. Existing contrastive learning methods typically augment data samples to construct positive sample pairs. Through an individual discrimination task, each sample instance is treated as a category, and the goal is to learn a feature that can distinguish each sample. By learning an encoder and a multilayer perceptron mapping head, the training data is mapped to a projection space through two nonlinear networks, bringing positive samples closer together and pushing negative samples further apart in the feature space. This pairwise interaction ensures that features are evenly distributed on the hypersphere of the feature space, preserving as much useful information as possible.

[0004] However, due to the large amount of data, high redundancy, and complex spatial spectral structure of hyperspectral images, existing contrastive learning methods cannot be well applied to hyperspectral pixel samples, resulting in poor performance of the final hyperspectral image feature representation. Summary of the Invention

[0005] To address the problems existing in the prior art, the present invention provides a method, apparatus, electronic device, and storage medium for representing hyperspectral image features.

[0006] In a first aspect, the present invention provides a method for representing hyperspectral image features, comprising:

[0007] Acquire multi-view information of a hyperspectral image, wherein the multi-view information includes a first view information of the hyperspectral image in the spatial dimension and a second view information of the hyperspectral image in the spectral dimension;

[0008] The multi-view information is subjected to image enhancement processing to obtain multi-view information after image enhancement processing. The image enhancement processing includes random cropping of the first view information and random sampling of the second view information.

[0009] The multi-view information after image enhancement is input into a pre-constructed multi-view contrast learning deep feature representation model to obtain the deep feature representation output by the multi-view contrast learning deep feature representation model. The deep feature representation includes a first feature representation corresponding to the first view information and a second feature representation corresponding to the second view information.

[0010] Optionally, according to the hyperspectral image feature representation method provided by the present invention, the step of obtaining multi-view information of the hyperspectral image includes:

[0011] The hyperspectral image is represented as X∈R W×H×B W×H represents the data space size of the hyperspectral image, and B represents the spectral dimension of the hyperspectral image;

[0012] A defined image block of size s×s×B represents any target pixel x in the hyperspectral image. i The image block of size s×s×B contains the target pixel x. i The adjacent pixels and spectral information, where s is a pre-set first parameter value;

[0013] For the target pixel x i Perform the first block processing to obtain s 2 A 1×B image patch, and the s 2 A 1×B image block is used as the target pixel x. i First-person perspective information;

[0014] For the target pixel x i Perform the second block processing to obtain B blocks of size 1×s. 2 The image blocks, and the B blocks of size 1×s 2 The image patch is used as the target pixel x i Second-person perspective information;

[0015] The first viewpoint information corresponding to all target pixels in the hyperspectral image is used as the first viewpoint information of the hyperspectral image, and the second viewpoint information corresponding to all target pixels in the hyperspectral image is used as the second viewpoint information of the hyperspectral image.

[0016] The first viewpoint information and the second viewpoint information of the hyperspectral image are used as the multi-viewpoint information of the hyperspectral image.

[0017] Optionally, according to the hyperspectral image feature representation method provided by the present invention, the random cropping operation on the first viewpoint information and the random sampling operation on the second viewpoint information include:

[0018] For the target pixel x in the hyperspectral image i Based on first-person perspective information, the image block of size s×s×B is randomly cropped into an image block of size m×m×B, and the image block of size m×m×B is determined to represent the target pixel x. i and for the target pixel x i Perform the first block processing to obtain m 2 A 1×B image patch, and the m 2 A 1×B image block is used as the target pixel x after image enhancement processing. i The first-person perspective information, where m is the second preset parameter;

[0019] For the target pixel x in the hyperspectral image i Second-view information for the target pixel x i The random sampling operation is performed on the second-view information to obtain B data of size 1×m. 2 The image blocks, and the B blocks of size 1×m 2 The image patch is used as the target pixel x after image enhancement processing. i Second-person perspective information.

[0020] Optionally, according to a hyperspectral image feature representation method provided by the present invention, the multi-view contrastive learning deep feature representation model includes two networks, one network for processing the first view information of the hyperspectral image, and the other network for processing the second view information of the hyperspectral image, each of the two networks including an encoder module and a mapping head.

[0021] Optionally, according to a hyperspectral image feature representation method provided by the present invention, the encoder module of each of the two networks includes a linear projection layer, which is used to adjust the first viewpoint information and the second viewpoint information of the hyperspectral image to viewpoint information of the same dimension.

[0022] Optionally, according to the hyperspectral image feature representation method provided by the present invention, the multi-view contrastive learning deep feature representation model is obtained by minimizing a target loss function during training, the target loss function being shown in the following equation:

[0023]

[0024] in, This represents the loss anchored to the information from the first perspective. V1 represents the loss anchored by the second perspective information, and V2 represents the first perspective information.

[0025] Optionally, in a hyperspectral image feature representation method provided by the present invention, the target loss function is calculated based on a noise contrast estimation method.

[0026] In a second aspect, the present invention also provides a hyperspectral image feature representation device, comprising:

[0027] A multi-view information acquisition module is used to acquire multi-view information of a hyperspectral image, wherein the multi-view information includes a first-view information of the hyperspectral image in the spatial dimension and a second-view information of the hyperspectral image in the spectral dimension.

[0028] An image enhancement module is used to perform image enhancement processing on the multi-view information to obtain multi-view information after image enhancement processing. The image enhancement processing includes random cropping of the first view information and random sampling of the second view information.

[0029] The feature representation acquisition module is used to input the multi-view information after the image enhancement processing into a pre-constructed multi-view contrast learning deep feature representation model to obtain the deep feature representation output by the multi-view contrast learning deep feature representation model. The deep feature representation includes a first feature representation corresponding to the first view information and a second feature representation corresponding to the second view information.

[0030] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the hyperspectral image feature representation method as described in the first aspect.

[0031] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the hyperspectral image feature representation method as described in the first aspect.

[0032] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the hyperspectral image feature representation method as described in the first aspect.

[0033] The hyperspectral image feature representation method, apparatus, electronic device, and storage medium provided by this invention acquire multi-view information of hyperspectral images and perform different image enhancement processes on the first and second view information in the multi-view information, namely, random cropping of the first view information and random sampling of the second view information, which increases the sample diversity and makes the generalization ability of the multi-view contrastive learning deep feature representation model stronger. Then, the multi-view information after image enhancement is input into the multi-view contrastive learning deep feature representation model to extract the deep feature representation of the hyperspectral image. This maximizes the mutual information between different viewpoints unique to hyperspectral images and greatly improves the performance of hyperspectral image feature representation. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating the hyperspectral image feature representation method provided by the present invention;

[0036] Figure 2 This is a schematic diagram of the multi-view contrastive learning deep feature representation model provided by the present invention;

[0037] Figure 3 This is a schematic diagram of the structure of the hyperspectral image feature representation device provided by the present invention;

[0038] Figure 4 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0040] It should be noted that in the description of this invention, the terms "first," "second," etc., are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, the first object can be one or more.

[0041] The hyperspectral image feature representation method, apparatus, electronic device, and storage medium provided by the present invention will be described exemplarily below with reference to the accompanying drawings.

[0042] Figure 1 This is a flowchart illustrating the hyperspectral image feature representation method provided by the present invention, as shown below. Figure 1 As shown, the method includes:

[0043] Step 100: Obtain multi-view information of the hyperspectral image, wherein the multi-view information includes first-view information of the hyperspectral image in the spatial dimension and second-view information of the hyperspectral image in the spectral dimension.

[0044] Step 110: Perform image enhancement processing on the multi-view information to obtain multi-view information after image enhancement processing. The image enhancement processing includes random cropping of the first view information and random sampling of the second view information.

[0045] Step 120: Input the multi-view information after image enhancement processing into a pre-constructed multi-view contrast learning deep feature representation model to obtain the deep feature representation output by the multi-view contrast learning deep feature representation model. The deep feature representation includes a first feature representation corresponding to the first view information and a second feature representation corresponding to the second view information.

[0046] It should be noted that the execution subject of the hyperspectral image feature representation method provided in this embodiment of the invention can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, a mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., while a non-mobile electronic device can be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This embodiment of the invention does not specifically limit the specific implementation of these methods.

[0047] The following describes in detail the technical solution of the embodiments of the present invention, taking the execution of the hyperspectral image feature representation method provided by the present invention by a computer as an example.

[0048] Specifically, to overcome the shortcomings of existing contrastive learning methods that cannot be well applied to hyperspectral pixel samples, resulting in poor performance of hyperspectral image feature representation, this invention acquires multi-view information of hyperspectral images and performs different image enhancement processes on the first and second view information in the multi-view information. Specifically, it performs random cropping on the first view information and random sampling on the second view information, increasing sample diversity and making the generalization ability of the multi-view contrastive learning deep feature representation model stronger. Then, the multi-view information after image enhancement is input into the multi-view contrastive learning deep feature representation model to extract the deep feature representation of the hyperspectral image. This maximizes the mutual information between different viewpoints unique to hyperspectral images and greatly improves the performance of hyperspectral image feature representation.

[0049] It's important to note that multi-view contrastive learning is based on the concept of contrastive representations from different perspectives. Input data of the same sample from different perspectives are positive sample pairs, and these positive sample pairs are close to each other in the feature space. Therefore, if another sample is randomly selected, regardless of the perspective, different samples are negative samples to each other and are far apart in the feature space.

[0050] It is understood that, in the embodiments of the present invention, the multi-view information of the same pixel in the hyperspectral image is a positive sample pair, and the multi-view information of different pixels in the hyperspectral image is a negative sample pair.

[0051] Optionally, the first-view information of the hyperspectral image in the data space dimension can be obtained first, and the second-view information of the hyperspectral image in the spectral dimension can be obtained. Then, image enhancement processing can be performed on the obtained first-view and second-view information, wherein the first-view information is randomly cropped and the second-view information is randomly sampled.

[0052] It should be noted that data augmentation strategies can effectively improve the performance of contrastive learning deep feature representation models, making them more robust and reducing overfitting. Related techniques have shown that a combination of various data augmentations is crucial for generating effective feature representations, with the combination of random cropping and color jitter being particularly prominent. Through data augmentation strategies, the model can learn the differences between data points, not just their statistical features. When data is imbalanced, data augmentation strategies can also help the model better learn minority class features, thereby improving model accuracy. Therefore, this embodiment of the invention performs image augmentation on data from two different perspectives to improve the performance of a multi-view contrastive learning deep feature representation model.

[0053] Optionally, in this embodiment of the invention, image enhancement processing is performed on multi-view information. In addition to random cropping and random sampling, data enhancement strategies such as color dithering, horizontal flipping, masking, and Gaussian blur can also be applied to help comparative learning obtain better representations.

[0054] Optionally, after obtaining the multi-view information after image enhancement processing, the multi-view information is input into a pre-constructed multi-view contrast learning deep feature representation model to obtain the deep feature representation output by the multi-view contrast learning deep feature representation model, wherein the deep feature representation includes a first feature representation corresponding to the first view information and a second feature representation corresponding to the second view information.

[0055] Optionally, the multi-view contrastive learning deep feature representation model includes two networks, one network for processing the first-view information of the hyperspectral image and the other network for processing the second-view information of the hyperspectral image, each of the two networks including an encoder module and a mapping head.

[0056] Specifically, in this embodiment of the invention, after obtaining the first viewpoint information and the second viewpoint information after image enhancement processing, the first viewpoint information is used as the input data of the encoder module of the first network in the multi-view contrast learning deep feature representation model, and the second viewpoint information is used as the input data of the encoder module of the second network in the multi-view contrast learning deep feature representation model. After the first viewpoint information and the second viewpoint information are processed by the encoder module and the mapping head respectively, the first feature representation and the second feature representation output by the multi-view contrast learning deep feature representation model can be obtained.

[0057] Optionally, the encoder module in this embodiment of the invention includes a Transformer encoder.

[0058] Optionally, for multi-view contrastive learning deep feature representation models, after self-supervised pre-training, fine-tuning strategies can be adopted for supervised classification tasks. For example, the learned feature representations can be input into the classifier, and the network parameters can be updated in a supervised manner using label information.

[0059] It should be noted that the embodiments of the present invention take into account that hyperspectral image pixel samples have unique multi-view information in both spectral and spatial dimensions. Each viewpoint of the data is noisy and may be incomplete, but the most important information is shared across these viewpoints. If the learned feature is viewpoint-invariant, that is, if it can capture key information from multiple viewpoints, then it is a powerful feature. Therefore, the embodiments of the present invention fuse the two viewpoint information of hyperspectral images in both spectral and spatial dimensions, and learn a deep feature representation model based on multi-view contrastive learning, thereby maximizing the mutual information between different viewpoints, learning effective feature representations, and greatly improving the performance of feature representation.

[0060] The hyperspectral image feature representation method provided by this invention acquires multi-view information of a hyperspectral image and performs different image enhancement processing on the first and second view information in the multi-view information. Specifically, it performs random cropping operation on the first view information and random sampling operation on the second view information, which increases the sample diversity and makes the generalization ability of the multi-view contrastive learning deep feature representation model stronger. Then, the multi-view information after image enhancement processing is input into the multi-view contrastive learning deep feature representation model to extract the deep feature representation of the hyperspectral image. This maximizes the mutual information between different viewpoints unique to hyperspectral images and greatly improves the performance of hyperspectral image feature representation.

[0061] Optionally, acquiring multi-view information of the hyperspectral image includes:

[0062] The hyperspectral image is represented as X∈R W×H×BW×H represents the data space size of the hyperspectral image, and B represents the spectral dimension of the hyperspectral image;

[0063] A defined image block of size s×s×B represents any target pixel x in the hyperspectral image. i The image block of size s×s×B contains the target pixel x. i The adjacent pixels and spectral information, where s is a pre-set first parameter value;

[0064] For the target pixel x i Perform the first block processing to obtain s 2 A 1×B image patch, and the s 2 A 1×B image block is used as the target pixel x. i First-person perspective information;

[0065] For the target pixel x i Perform the second block processing to obtain B blocks of size 1×s. 2 The image blocks, and the B blocks of size 1×s 2 The image patch is used as the target pixel x i Second-person perspective information;

[0066] The first viewpoint information corresponding to all target pixels in the hyperspectral image is used as the first viewpoint information of the hyperspectral image, and the second viewpoint information corresponding to all target pixels in the hyperspectral image is used as the second viewpoint information of the hyperspectral image.

[0067] The first viewpoint information and the second viewpoint information of the hyperspectral image are used as the multi-viewpoint information of the hyperspectral image.

[0068] Specifically, given a hyperspectral image X∈R W×H×B Let W×H represent the data space size of the hyperspectral image, and B represent the spectral dimension (band dimension) of the hyperspectral image. For each pixel in the hyperspectral image, a three-dimensional image block of size s×s×B is selected to represent that pixel. This three-dimensional image block contains the neighboring pixels of that pixel and their spectral information. Then, for the pixel set {x1, x2, ..., x...} of the hyperspectral image... M}, x i ∈s×s×B, where M=W×H, first construct the multi-view information of the pixels, here for two views. and Different perspectives of the same pixel and For mutually positive sample pairs, multi-view information of any other pixel within the batch can be used as... and Negative samples.

[0069] Alternatively, the pixel x can be analyzed from the spatial dimension. i The first block processing is performed on ∈s×s×B, where each pixel is treated as a 3D image block. This 3D image block is then further divided into blocks, resulting in each image block being a vector of size 1×B. The total number of blocks is s. 2 Image patches, thus obtaining pixel x i V1 perspective information

[0070] Alternatively, pixel x can be analyzed from the spectral dimension. i The second block processing is performed on the ∈s×s×B bands, treating the spatial information of each band as an image block. Each image block is a two-dimensional image of size s×s, which is then stretched into a one-dimensional vector, resulting in a total of B 1×s dimensions. 2 The image patch, thus obtaining pixel x i V2 perspective information

[0071] It should be noted that the first preset parameter s can be adaptively set based on actual applications, and the embodiments of the present invention do not impose specific limitations on it.

[0072] It is understood that the embodiments of the present invention effectively obtain the first-view information and the second-view information of each pixel by dividing each pixel in the hyperspectral image into blocks from the data space dimension and the spectral dimension, thereby effectively obtaining the first-view information of the hyperspectral image in the data space dimension and the second-view information in the spectral dimension.

[0073] Optionally, the random cropping operation on the first viewpoint information and the random sampling operation on the second viewpoint information include:

[0074] For the target pixel x in the hyperspectral image i Based on first-person perspective information, the image block of size s×s×B is randomly cropped into an image block of size m×m×B, and the image block of size m×m×B is determined to represent the target pixel x. i and for the target pixel x i Perform the first block processing to obtain m 2 A 1×B image patch, and the m 2 A 1×B image block is used as the target pixel x after image enhancement processing. i The first-person perspective information, where m is the second preset parameter;

[0075] For the target pixel x in the hyperspectral image iSecond-view information for the target pixel x i The random sampling operation is performed on the second-view information to obtain B data of size 1×m. 2 The image blocks, and the B blocks of size 1×m 2 The image patch is used as the target pixel x after image enhancement processing. i Second-person perspective information.

[0076] Specifically, in this embodiment of the invention, for pixel x in a hyperspectral image i From the V1 perspective, a random cropping operation is performed to obtain an image patch of size m×m×B, which is then divided into blocks to obtain m 2 A 1×B vector For pixel x i The V2 perspective, in order to match the perspective While maintaining a consistent sequence length, perform random sampling, randomly selecting m... 2 One band, resulting in B bands of size 1×m. 2 Image patch, as pixel x i perspective

[0077] It should be noted that the second preset parameter m can be adaptively set based on actual applications, and the embodiments of the present invention do not impose specific limitations on it.

[0078] It is understood that the embodiments of the present invention effectively realize image enhancement processing of the first-view information and the second-view information of each pixel in the hyperspectral image, which can increase the diversity of samples and make the generalization ability of the multi-view contrast learning deep feature representation model stronger.

[0079] Optionally, the encoder module of each of the two networks includes a linear projection layer, which is used to adjust the first viewpoint information and the second viewpoint information of the hyperspectral image to viewpoint information of the same dimension.

[0080] Specifically, in this embodiment of the invention, the encoder module of each of the two networks included in the contrastive learning deep feature representation model includes a linear projection layer, which is used to adjust the first viewpoint information and the second viewpoint information of the hyperspectral image to viewpoint information of the same dimension.

[0081] It should be noted that, since the dimensions of each image block element are inconsistent between the V1 and V2 perspectives, this embodiment of the invention uses two branches of linear projection layers to adjust them to the same dimension as the input of the Transformer encoder.

[0082] Figure 2This is a schematic diagram of the multi-view contrastive learning deep feature representation model provided by the present invention, as shown below. Figure 2 As shown, the multi-view information of the constructed hyperspectral image is input into the encoder module of the multi-view contrast learning deep feature representation model. The encoder module includes image segmentation, linear projection layer and Transformer encoder. After processing by the encoder module, the information is input into the mapping head of the multi-view contrast learning deep feature representation model for further processing.

[0083] It should be noted that, inspired by the Visual Transformer, the multi-view contrastive learning deep feature representation model uses ViT (Vision Transformer) as the encoder module. ViT first performs block processing on the image, targeting hyperspectral pixels x... i It has the unique feature of containing both spatial information of neighboring pixels and rich spectral information, and constructs multi-view information of pixel blocks through different perspective division methods.

[0084] Inputting multi-view information into the Transformer encoder yields... This is then input into the mapping header to obtain... Then, the model is trained to represent multi-view features from the same pixel. and Zoom in, while simultaneously representing the characteristics from different pixel perspectives. and Push away, such as Figure 2 As shown.

[0085] Optionally, the multi-view contrastive learning deep feature representation model is obtained by training by minimizing a target loss function, which is shown in the following equation:

[0086]

[0087] in, This represents the loss anchored to the information from the first perspective. V1 represents the loss anchored by the second perspective information, and V2 represents the first perspective information.

[0088]

[0089] in, It is an anchor point from view V1, which has a positive sample. and k negative samples obtained from view V2; the discriminant function s(·,·) is the cosine similarity with temperature control parameter τ:

[0090]

[0091] Finally, based on the above equation (2), the losses with the two viewpoints as anchor points are calculated respectively. and Then, by adding them together, we obtain the objective function for multi-view contrastive learning, which is the above equation (1).

[0092] Optionally, the target loss function is calculated based on a noise contrast estimation method.

[0093] It should be noted that since k in formula (2) may be very large, the present invention adopts the Noise Contrastive Estimation (NCE) method to approximate the loss function, which can reduce the computational requirements.

[0094] It is understood that the hyperspectral image feature representation method provided in this invention first constructs multi-view information of hyperspectral data from both spatial and spectral dimensions, and then performs data augmentation on the data from both perspectives. Based on this, a deep feature representation model based on multi-view contrastive learning is studied. This model can capture information from multiple perspectives of the data, learn effective feature representations, fully exploit the unique multi-view information of hyperspectral data, and fuse complementary information from multiple perspectives, thereby greatly improving the performance of feature representation.

[0095] The hyperspectral image feature representation method provided by this invention acquires multi-view information of a hyperspectral image and performs different image enhancement processing on the first and second view information in the multi-view information. Specifically, it performs random cropping operation on the first view information and random sampling operation on the second view information, which increases the sample diversity and makes the generalization ability of the multi-view contrastive learning deep feature representation model stronger. Then, the multi-view information after image enhancement processing is input into the multi-view contrastive learning deep feature representation model to extract the deep feature representation of the hyperspectral image. This maximizes the mutual information between different viewpoints unique to hyperspectral images and greatly improves the performance of hyperspectral image feature representation.

[0096] The hyperspectral image feature representation device provided by the present invention is described below. The hyperspectral image feature representation device described below and the hyperspectral image feature representation method described above can be referred to in correspondence.

[0097] Figure 3 This is a schematic diagram of the hyperspectral image feature representation device provided by the present invention, as shown below. Figure 3 As shown, the device includes: a multi-view information acquisition module 310, an image enhancement module 320, and a feature representation acquisition module 330; wherein:

[0098] The multi-view information acquisition module 310 is used to acquire multi-view information of a hyperspectral image, wherein the multi-view information includes a first view information of the hyperspectral image in the spatial dimension and a second view information of the hyperspectral image in the spectral dimension.

[0099] Image enhancement module 320 is used to perform image enhancement processing on the multi-view information to obtain multi-view information after image enhancement processing. The image enhancement processing includes random cropping of the first view information and random sampling of the second view information.

[0100] The feature representation acquisition module 330 is used to input the multi-view information after the image enhancement processing into a pre-constructed multi-view contrast learning deep feature representation model to obtain the deep feature representation output by the multi-view contrast learning deep feature representation model. The deep feature representation includes a first feature representation corresponding to the first view information and a second feature representation corresponding to the second view information.

[0101] The hyperspectral image feature representation device provided by this invention acquires multi-view information of a hyperspectral image and performs different image enhancement processing on the first and second view information in the multi-view information. Specifically, it performs random cropping operation on the first view information and random sampling operation on the second view information, which increases the sample diversity and makes the generalization ability of the multi-view contrastive learning deep feature representation model stronger. Then, the multi-view information after image enhancement processing is input into the multi-view contrastive learning deep feature representation model to extract the deep feature representation of the hyperspectral image. This maximizes the mutual information between different viewpoints unique to hyperspectral images and greatly improves the performance of hyperspectral image feature representation.

[0102] Optionally, the multi-view information acquisition module 310 is specifically used for:

[0103] The hyperspectral image is represented as X∈R W×H×B W×H represents the data space size of the hyperspectral image, and B represents the spectral dimension of the hyperspectral image;

[0104] A defined image block of size s×s×B represents any target pixel x in the hyperspectral image. i The image block of size s×s×B contains the target pixel x. i The adjacent pixels and spectral information, where s is a pre-set first parameter value;

[0105] For the target pixel x i Perform the first block processing to obtain s 2 A 1×B image patch, and the s 2A 1×B image block is used as the target pixel x. i First-person perspective information;

[0106] For the target pixel x i Perform the second block processing to obtain B blocks of size 1×s. 2 The image blocks, and the B blocks of size 1×s 2 The image patch is used as the target pixel x i Second-person perspective information;

[0107] The first viewpoint information corresponding to all target pixels in the hyperspectral image is used as the first viewpoint information of the hyperspectral image, and the second viewpoint information corresponding to all target pixels in the hyperspectral image is used as the second viewpoint information of the hyperspectral image.

[0108] The first viewpoint information and the second viewpoint information of the hyperspectral image are used as the multi-viewpoint information of the hyperspectral image.

[0109] Optionally, the image enhancement module 320 is specifically used for:

[0110] For the target pixel x in the hyperspectral image i Based on first-person perspective information, the image block of size s×s×B is randomly cropped into an image block of size m×m×B, and the image block of size m×m×B is determined to represent the target pixel x. i and for the target pixel x i Perform the first block processing to obtain m 2 A 1×B image patch, and the m 2 A 1×B image block is used as the target pixel x after image enhancement processing. i The first-person perspective information, where m is the second preset parameter;

[0111] For the target pixel x in the hyperspectral image i Second-view information for the target pixel x i The random sampling operation is performed on the second-view information to obtain B data of size 1×m. 2 The image blocks, and the B blocks of size 1×m 2 The image patch is used as the target pixel x after image enhancement processing. i Second-person perspective information.

[0112] Optionally, the multi-view contrastive learning deep feature representation model includes two networks, one network for processing the first-view information of the hyperspectral image and the other network for processing the second-view information of the hyperspectral image, each of the two networks including an encoder module and a mapping head.

[0113] Optionally, the encoder module of each of the two networks includes a linear projection layer, which is used to adjust the first viewpoint information and the second viewpoint information of the hyperspectral image to viewpoint information of the same dimension.

[0114] Optionally, the multi-view contrastive learning deep feature representation model is obtained by training by minimizing a target loss function, which is shown in the following equation:

[0115]

[0116] in, This represents the loss anchored to the information from the first perspective. V1 represents the loss anchored by the second perspective information, and V2 represents the first perspective information.

[0117] Optionally, the target loss function is calculated based on a noise contrast estimation method.

[0118] It should be noted that the hyperspectral image feature representation device provided in this embodiment of the invention can implement all the method steps implemented in the above-mentioned hyperspectral image feature representation method embodiment, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0119] Figure 4 This is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute the hyperspectral image feature representation method provided by the above methods, which includes:

[0120] Acquire multi-view information of a hyperspectral image, wherein the multi-view information includes a first view information of the hyperspectral image in the spatial dimension and a second view information of the hyperspectral image in the spectral dimension;

[0121] The multi-view information is subjected to image enhancement processing to obtain multi-view information after image enhancement processing. The image enhancement processing includes random cropping of the first view information and random sampling of the second view information.

[0122] The multi-view information after image enhancement is input into a pre-constructed multi-view contrast learning deep feature representation model to obtain the deep feature representation output by the multi-view contrast learning deep feature representation model. The deep feature representation includes a first feature representation corresponding to the first view information and a second feature representation corresponding to the second view information.

[0123] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0124] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the hyperspectral image feature representation method provided by the above methods, the method comprising:

[0125] Acquire multi-view information of a hyperspectral image, wherein the multi-view information includes a first view information of the hyperspectral image in the spatial dimension and a second view information of the hyperspectral image in the spectral dimension;

[0126] The multi-view information is subjected to image enhancement processing to obtain multi-view information after image enhancement processing. The image enhancement processing includes random cropping of the first view information and random sampling of the second view information.

[0127] The multi-view information after image enhancement is input into a pre-constructed multi-view contrast learning deep feature representation model to obtain the deep feature representation output by the multi-view contrast learning deep feature representation model. The deep feature representation includes a first feature representation corresponding to the first view information and a second feature representation corresponding to the second view information.

[0128] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the hyperspectral image feature representation methods provided above, the method comprising:

[0129] Acquire multi-view information of a hyperspectral image, wherein the multi-view information includes a first view information of the hyperspectral image in the spatial dimension and a second view information of the hyperspectral image in the spectral dimension;

[0130] The multi-view information is subjected to image enhancement processing to obtain multi-view information after image enhancement processing. The image enhancement processing includes random cropping of the first view information and random sampling of the second view information.

[0131] The multi-view information after image enhancement is input into a pre-constructed multi-view contrast learning deep feature representation model to obtain the deep feature representation output by the multi-view contrast learning deep feature representation model. The deep feature representation includes a first feature representation corresponding to the first view information and a second feature representation corresponding to the second view information.

[0132] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0133] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for representing hyperspectral image features, characterized in that, include: Acquire multi-view information of a hyperspectral image, wherein the multi-view information includes a first view information of the hyperspectral image in the spatial dimension and a second view information of the hyperspectral image in the spectral dimension; The multi-view information is subjected to image enhancement processing to obtain multi-view information after image enhancement processing. The image enhancement processing includes random cropping of the first view information and random sampling of the second view information. The multi-view information after image enhancement processing is input into a pre-constructed multi-view contrast learning deep feature representation model to obtain the deep feature representation output by the multi-view contrast learning deep feature representation model. The deep feature representation includes a first feature representation corresponding to the first view information and a second feature representation corresponding to the second view information. The acquisition of multi-view information from hyperspectral images includes: The hyperspectral image is represented as , This indicates the data space size of the hyperspectral image. This indicates the spectral dimension of the hyperspectral image; Determine the size as An image block represents any target pixel in the hyperspectral image. The size is The image block contains the target pixel Adjacent pixels and spectral information, The first parameter value is preset; For the target pixel Perform the first block processing to obtain The size is Image blocks, and the The size is Image blocks as the target pixels First-person perspective information; For the target pixel Perform the second block processing to obtain The size is Image blocks, and the The size is Image blocks as the target pixels Second-person perspective information; The first viewpoint information corresponding to all target pixels in the hyperspectral image is used as the first viewpoint information of the hyperspectral image, and the second viewpoint information corresponding to all target pixels in the hyperspectral image is used as the second viewpoint information of the hyperspectral image. The first viewpoint information and the second viewpoint information of the hyperspectral image are used as the multi-viewpoint information of the hyperspectral image; The multi-view contrastive learning deep feature representation model includes two networks, one of which processes the first-view information of the hyperspectral image and the other of which processes the second-view information of the hyperspectral image. Each of the two networks includes an encoder module and a mapping head. The encoder module of each of the two networks includes a linear projection layer, which is used to adjust the first and second viewpoint information of the hyperspectral image to viewpoint information of the same dimension.

2. The hyperspectral image feature representation method according to claim 1, characterized in that, The random cropping operation on the first viewpoint information and the random sampling operation on the second viewpoint information include: For the target pixel in the hyperspectral image First-person perspective information, the size of Image patches are randomly cropped to a size of The image block, determining the size as Image blocks represent the target pixels and for the target pixel Perform the first block processing to obtain The size is Image blocks, and the The size is The image patch is used as the target pixel after image enhancement processing. First-person perspective information This is the second preset parameter; For the target pixel in the hyperspectral image Second-view information for the target pixel The random sampling operation is performed on the second-view information to obtain... The size is Image blocks, and the The size is The image patch is used as the target pixel after image enhancement processing. Second-person perspective information.

3. The hyperspectral image feature representation method according to claim 1, characterized in that, The multi-view contrastive learning deep feature representation model is obtained by training by minimizing the target loss function, which is shown in the following formula: ； in, This represents the loss anchored to the information from the first perspective. This represents the loss anchored to the information from the second perspective. This represents the information from the first perspective. This represents the information from the second perspective.

4. The hyperspectral image feature representation method according to claim 3, characterized in that, The target loss function is calculated based on the noise contrast estimation method.

5. A hyperspectral image feature representation device, characterized in that, include: A multi-view information acquisition module is used to acquire multi-view information of a hyperspectral image, wherein the multi-view information includes a first-view information of the hyperspectral image in the spatial dimension and a second-view information of the hyperspectral image in the spectral dimension. An image enhancement module is used to perform image enhancement processing on the multi-view information to obtain multi-view information after image enhancement processing. The image enhancement processing includes random cropping of the first view information and random sampling of the second view information. The feature representation acquisition module is used to input the multi-view information after the image enhancement processing into a pre-constructed multi-view contrast learning deep feature representation model to obtain the deep feature representation output by the multi-view contrast learning deep feature representation model. The deep feature representation includes a first feature representation corresponding to the first view information and a second feature representation corresponding to the second view information. The acquisition of multi-view information from hyperspectral images includes: The hyperspectral image is represented as , This indicates the data space size of the hyperspectral image. This indicates the spectral dimension of the hyperspectral image; Determine the size as An image block represents any target pixel in the hyperspectral image. The size is The image block contains the target pixel Adjacent pixels and spectral information, The first parameter value is preset; For the target pixel Perform the first block processing to obtain The size is Image blocks, and the The size is Image blocks as the target pixels First-person perspective information; For the target pixel Perform the second block processing to obtain The size is Image blocks, and the The size is Image blocks as the target pixels Second-person perspective information; The first viewpoint information corresponding to all target pixels in the hyperspectral image is used as the first viewpoint information of the hyperspectral image, and the second viewpoint information corresponding to all target pixels in the hyperspectral image is used as the second viewpoint information of the hyperspectral image. The first viewpoint information and the second viewpoint information of the hyperspectral image are used as the multi-viewpoint information of the hyperspectral image; The multi-view contrastive learning deep feature representation model includes two networks, one of which processes the first-view information of the hyperspectral image and the other of which processes the second-view information of the hyperspectral image. Each of the two networks includes an encoder module and a mapping head. The encoder module of each of the two networks includes a linear projection layer, which is used to adjust the first and second viewpoint information of the hyperspectral image to viewpoint information of the same dimension.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the hyperspectral image feature representation method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the hyperspectral image feature representation method as described in any one of claims 1 to 4.

Images