Hyperspectral target detection method based on unsupervised momentum contrast learning

Abstract

This invention discloses a hyperspectral image target detection method based on momentum-free contrastive learning and a Transformer network. The method includes: First, a Transformer-based encoder is designed to extract spectral features and a momentum encoder for hyperspectral target detection. To focus on long-range spectral dependencies and self-similarity without neglecting local spectral details, the spectral feature extraction encoder and momentum encoder utilize overlapping spectral block feature mapping and interactive token feedforward layers to focus on local spectral details. Second, unsupervised momentum contrastive learning is used to learn spectral discrimination capabilities. A queue and a momentum encoder that updates slowly with momentum are used to provide a sufficient number of consistent negative sample features to help the model learn better representations. Finally, exponential and normalization operations, and power and normalization operations, are used to perform two nonlinear boosts on the detection results obtained through cosine similarity to suppress background noise.

Description

Technical Field

[0001] This invention relates to the field of hyperspectral image target detection, and in particular to a hyperspectral target detection method based on unsupervised momentum contrastive learning and Transformer networks. Background Technology

[0002] Hyperspectral images are three-dimensional cubic image data containing rich spectral and spatial information, obtained by imaging spectrometers capturing the reflectance of ground surfaces across hundreds of narrow and continuous bands. The fine spectral features can effectively reflect the subtle characteristics of different substances. Thanks to the extremely high spectral resolution of hyperspectral images, target detection can perform target detection based on the spectral differences of different ground objects, and it has important applications in military and security fields such as camouflage target identification, pollution detection, mineral exploration, food safety, and medical diagnosis.

[0003] Traditional hyperspectral image target detection methods, such as Spectral Matched Filtering (SMF) and Adaptive Coherence Estimation (ACE), are classic probabilistic statistical methods that assume the background follows a multivariate Gaussian distribution. Orthogonal Subspace Projection (OSP), based on subspace projection, suppresses the background and highlights the target by projecting the pixel spectrum into an orthogonal subspace of the background subspace, given prior information about the target and background. Target detectors based on Constrained Energy Minimization (CEM) construct finite impulse response filters and constrain the characteristics of the target spectrum with specific gains while minimizing the influence of the background. Traditional hyperspectral target detection methods have weak adaptability to the inherent variability of target spectra, leading to decreased detection accuracy. In recent years, deep learning-based methods have emerged in the field of hyperspectral image target detection. From the perspective of supervised learning, to train deep neural networks and address the issue of insufficient training samples, transfer learning-based methods have emerged. To expand the training samples, the spectra of pixels of the same class in known labeled hyperspectral datasets are subtracted, and the spectra of pixels of different classes are subtracted, thus increasing the number of training samples. However, the target detection performance of this method is largely limited by the adaptability of the transferred knowledge. From the perspective of unsupervised learning, the background learning hyperspectral target detection method based on target inhibition constraints uses the CEM method to coarsely detect the hyperspectral image to be detected to obtain background samples. The background samples are then fed into an adversarial autoencoder (AAE) with target inhibition constraints for training to reconstruct a clean background. Target detection is achieved by comparing the reconstructed background image with the image to be detected. However, the detection results are affected by the quality of the target and background samples. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention discloses a hyperspectral image target detection method based on momentum-free contrastive learning and Transformer networks, specifically including the following steps:

[0005] By constructing feature maps and spatial location information maps for overlapping spectral blocks, the length of the input sequence is reduced while preserving local information between adjacent spectral blocks, thereby providing a higher quality input sequence to improve the performance of the Transformer. First, for the pixel spectrum of band B... The original image is divided into several overlapping spectral blocks using convolution, thereby obtaining the embedded spectral vector sequence. Where N = ((Bk) / s + 1) is the effective input sequence length of the Transformer, and d is the dimension of each embedded spectral vector sequence. Then, in the embedded spectral vector sequence... A learnable embedding map x was previously added. learn During training, it is updated and iterated through a Transformer encoder, which is used in x e The 'x' is considered as a representation of 'x'. Finally, the learnable 1D positional information is mapped and added to 'x' in an additive manner. e The positional information of the spectral patch within the original pixel spectrum is preserved. The final embedded spectral vector sequence z0, which serves as input to the Transformer encoder, can be represented as:

[0006]

[0007] in It is a learnable location information mapping;

[0008] An interactive token feedforward layer is constructed to establish long-range spectral dependencies without neglecting local spectral details. Based on the fully connected feedforward network of the original Transformer encoder, depth-wise convolutions are added between the two fully connected layers of the feedforward layer to supplement local details and achieve information interaction across different spectral channels. This process can be described as follows:

[0009] Z' = FC(Z; ω1)

[0010] Z”=FC(σ(Z’+DWConv(Z’;ω));ω2)

[0011] Where Z is the input of the interactive token feedforward layer, ω1 and ω2 are the parameters of the two fully connected layers, ω is the parameter of the 1D convolutional layer, and σ is the activation function of the Gaussian error linear unit (GELU).

[0012] An encoder and momentum encoder for spectral feature extraction in hyperspectral target detection tasks are constructed based on the Transformer module.

[0013] Spectral instance identification agent task is built using data augmentation;

[0014] The spectral feature extraction encoder and momentum encoder are trained using an unsupervised momentum contrastive learning approach. First, small batches of original samples X are obtained through random sampling. q and augmented sample X k And use a spectral feature extraction encoder and a momentum encoder to extract X respectively. q and X k The representation is then input into each projection head to obtain the feature matrix. and This process can be represented as:

[0015]

[0016]

[0017] θ encoder With θ m_encoder These are encoder f encoder (·) and momentum encoder f m_encoder The parameters of (·). The encoder's projection head MLP encoder With the momentum encoder projection head MLP m_encoder These are respectively having parameters θ MLPencoder and θ MLPm_encoder An MLP containing one hidden layer. encoder With MLP m_encoder Use the ReLU activation function.

[0018] Then the momentum encoder will convert the feature matrix features The samples are fed into a queue and used as a small negative sample set to achieve one-to-many contrastive learning of positive samples. The contrastive loss in the contrastive learning process uses the InfoNCE loss function, as shown below:

[0019]

[0020] Where τ is the temperature hyperparameter. Includes a positive sample embedding feature (assuming) ) and K negative sample embedding features.

[0021] Finally, during training, to ensure feature consistency within the queue, the features should be generated using the same or similar momentum encoders and projectors to help the model avoid learning shortcut solutions. Therefore, the momentum encoder f m encoder(·) and its projection head MLP m_encoder Momentum updates are used. This process can be formalized as follows:

[0022] θ m_encoder ←mθ m_encoder +(1-m)θ encoder

[0023]

[0024] Where m∈[0,1) is the momentum coefficient, which is generally set to a large value. It is important to note the encoder f... encoder (·) and its projection head MLP enc o der Updated via gradient backpropagation;

[0025] The trained spectral feature extraction encoder network is used to extract the representation vectors of the spectra of each pixel in the hyperspectral image to be detected and the prior spectra. The similarity between the spectral representation vector of the pixel to be detected and the prior target spectral representation vector is measured by cosine similarity to obtain the detection result using only spectral information.

[0026] By applying exponential and normalization operations, and power function and normalization operations, the detection results are nonlinearly enhanced twice to suppress the background. This process can be formalized as follows:

[0027] S = α B

[0028]

[0029] R = S β

[0030]

[0031] α and β are positive parameters that adjust the background suppression performance.

[0032] By employing the aforementioned technical solutions, this invention provides a hyperspectral image target detection method based on momentum-free contrastive learning and a Transformer network. This method effectively solves the problems of difficult background suppression and insufficient utilization of spectral information in the field of hyperspectral target detection. First, it designs overlapping spectral block feature mapping and an interactive token feedforward layer to help the Transformer focus on local spectral details. Second, during spectral discriminative learning, pixel spectra at the same location in the original and data-enhanced hyperspectral images are considered positive sample pairs, while negative sample pairs come from a queue. The queue can hold a large number of negative samples and separates the number of negative samples from the mini-batch size. To ensure the consistency of features among negative samples in the queue, a momentum encoder based on momentum updates is used to extract pixel spectral features, and contrastive loss is used to maximize the similarity of positive pairs while minimizing the similarity of negative pairs for spectral discriminative learning. Finally, during target detection, the trained spectral feature extraction encoder network is used to extract features of the target prior and the spectra of the pixels to be detected. Cosine similarity is used to determine the similarity between the features of the pixel to be detected's spectra and the target's prior features to obtain the target detection result. Then, by utilizing the function curve characteristics of exponential and power functions between 0 and 1, the detection results are subjected to two nonlinear enhancements through exponential and normalization operations, and power function and normalization operations, in order to suppress the background. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart of the method disclosed in this invention;

[0035] Figure 2 This is a structural diagram of the CTFFN (Trusted Token Forwarding Layer).

[0036] Figure 3 This is a structural diagram of the self-attention mechanism;

[0037] Figure 4 Diagram of the multi-head attention mechanism;

[0038] Figures 5a-5b Annotated ground truth map and detection results for the SanDiego dataset;

[0039] Figures 6a-6bAnnotated ground truth map and detection results for the PaviaC dataset;

[0040] Figures 7a-7b To propose an object detection method (MCLT) and compare it with other object detection methods on the San Diego dataset (P... D ,P F ) and (P F ,τ)ROC curve diagram.

[0041] Figures 8a-8b To propose an object detection method (MCLT) and compare it with other object detection methods on the PaviaC dataset (P D ,P F ) and (P F ,τ)ROC curve diagram. Detailed Implementation

[0042] To make the technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention:

[0043] like Figure 1 The method for hyperspectral image target detection based on momentum-free contrastive learning and Transformer networks, as shown, specifically includes the following steps:

[0044] S1: Construct feature mapping and spatial location information mapping for overlapping spectral blocks;

[0045] S1 specifically adopts the following method:

[0046] S11: To establish reliable interaction channels between blocks, an overlapping spectral block feature mapping is designed to enable information transfer between adjacent blocks in an overlapping manner. Specifically, the overlapping spectral block feature mapping is implemented through a 1D convolutional layer, where the number of convolutional kernels *d* controls the dimension of each spectral block after feature mapping, the kernel size *k* controls the length of each spectral block, and the stride *s* controls the size of the non-overlapping portion between adjacent spectral blocks. For a pixel spectrum in band B... The embedded spectral vector sequence is obtained by feature mapping of overlapping spectral blocks. Where N = ((Bk) / s + 1) is the effective input sequence length of the Transformer, and d is the dimension of each embedded spectral vector sequence.

[0047] S12: To avoid losing the spatial location information of the image itself during the block segmentation process, embed the spectral vector sequence. A learnable embedding map x was previously added. learnDuring training, it is updated and iterated through a Transformer encoder, which is used in x e The 'x' is considered as a representation of 'x'. Finally, the learnable 1D positional information is mapped and added to 'x' in an additive manner. e The positional information of the spectral patch within the original pixel spectrum is preserved. The final embedded spectral vector sequence z0, which serves as input to the Transformer encoder, can be represented as:

[0048]

[0049] in It is a learnable location information mapping;

[0050] S2: Construct an interactive token feedforward layer to establish long-distance spectral dependencies;

[0051] In S2, the specific approach is as follows: The feedforward layer in the original Transformer encoder is a fully connected feedforward network, consisting of two linear transformations with a ReLU activation function in between. Its fully connected layer is point-wise and cannot learn cross-token information. To supplement the local detail information in the feedforward layer, an interactive token feedforward layer was designed, such as... Figure 2 As shown, local details in the feedforward layer are supplemented by adding depth-wise convolutions between the two fully connected layers. This process can be described as follows:

[0052] Z' = FC(Z; ω1)

[0053] Z”=FC(σ(Z’+DWConv(Z’;ω));ω2)

[0054] Where Z is the input of the interactive token feedforward layer, ω1 and ω2 are the parameters of the two fully connected layers, ω is the parameter of the 1D convolutional layer, and σ is the activation function of the Gaussian error linear unit (GELU).

[0055] S3: Construct an encoder and momentum encoder based on the Transformer module for spectral feature extraction in hyperspectral target detection tasks;

[0056] S31: The structure of the Transformer module is as follows Figure 1 As shown in the Transformer module, it mainly consists of three parts: layer normalization, multi-head self-attention, and an interaction token feedforward layer. The self-attention mechanism is as follows: Figure 3 As shown, firstly for the input sequence The query Q, key K, and value V are obtained by projecting the learnable feature matrix. This process can be described as follows:

[0057] [Q,K,V]=zE QKV

[0058] in It is a learnable feature matrix. Then, by calculating the dot product of query Q and all keys K and scaling it, an attention map is obtained. The attention map is then transformed into weights using a softmax function. Finally, the weights are multiplied by the value V to obtain the self-attention embedded in the spectral vector sequence. This process can be represented as:

[0059]

[0060] Among them, the dot product is... Scaling is used to avoid the vanishing gradient phenomenon after applying the softmax function, d k It is the dimension of querying Q, key K, and value V;

[0061] S32: To enhance the model's expressive power and accelerate network training, h self-attention operations, referred to as "heads," are run in parallel and finally projected into the output in a cascaded manner. This is the main process of constructing multi-head self-attention, as follows: Figure 4 As shown, multi-head self-attention can be represented as:

[0062]

[0063] MSA(z)=[SA1(Q1,K1,V1);SA2(Q2,K2,V2);…;SA h (Q h ,K h V h )]E MSA

[0064] in It is used to project the input sequence into a query Q. i , key K i Sum V i Learnable projection matrix, is the feature matrix of the projection cascade output, and h is the number of self-attention operations used in parallel. To make the overall computational cost of multi-head attention similar to that of full-dimensional single-head attention, d is generally used... k Set to d / h;

[0065] S33: Repeat the Transformer module L times in the encoder to construct an encoder with L layers. The output of multi-head self-attention in each Transformer module can be formalized as:

[0066] z' l =MSA(LN(z) l-1 ))+z l-1,l=1...L

[0067] The output of the interaction token feedforward layer in each Transformer module can be formalized as follows:

[0068] z l =CTFFN(LN(z') l ))+z' l ,l=1...L

[0069] Both the spectral feature extraction encoder and the momentum encoder utilize h = 8 parallel self-attention operations and L = 6 Transformer layers. The encoder output can be represented as:

[0070]

[0071] in It is a learnable embedding x learn The output at the corresponding position obtained after passing through the Transformer module will The output y after layer normalization is used as the representation of the spectrum;

[0072] S4: Construct a spectral instance identification agent task through data augmentation;

[0073] S4 specifically adopts the following method: First, the original hyperspectral image is processed... Data augmentation is achieved using Gaussian blurring, specifically with a random sampling standard deviation δ = [0.1, 2.0] and a kernel size of 1×1. Then, a spectral instance identification proxy task is constructed by pairing the original hyperspectral image with the Gaussian-blurred image using pixel spectra. Finally, the spectra of pixels at the same location are considered positive sample pairs, while the spectra of pixels at different locations are considered negative sample pairs.

[0074] S5: The spectral feature extraction encoder and momentum encoder are trained using an unsupervised momentum contrastive learning method;

[0075] The specific approach used in S5 is as follows:

[0076] S51: Obtain a small batch of original samples X through random sampling. q and augmented sample X k And use a spectral feature extraction encoder and a momentum encoder to extract X respectively. q and X k The representation is then input into each projection head to obtain the corresponding feature matrix. and This process can be represented as:

[0077]

[0078]

[0079] θ encoder With θ m_encoder These are encoder f encoder (·) and momentum encoder f m_encoder The parameters of (·). The encoder's projection head MLP encoder With the momentum encoder projection head MLP m_encoder These are parameters and An MLP containing one hidden layer. encoder With MLP m_encoder Use the ReLU activation function.

[0080] S52: The characteristic output of the momentum encoder through the projection head The negative samples are fed into a queue and used as a small set of negative samples to enable one-to-many learning of positive samples. Therefore, a larger queue can be used to store more negative samples, which helps the model learn more discriminative features. Furthermore, due to the first-in, first-out (FIFO) nature of the queue, the relationships between different contrastive units are preserved when feature samples are gradually replaced. The size of the queue can be flexibly and independently set as the hyperparameter K.

[0081] S53: Encoder f encoder (·) via MLP projection head encoder The output features and features in the queue are fed into the contrastive loss. The contrastive loss maximizes the similarity of positive pairs and minimizes the similarity of negative pairs, thus enabling the encoder to discriminate spectral differences. The similarity between positive and negative pairs is measured by a dot product, and the contrastive loss uses the InfoNCE loss function, as shown below:

[0082]

[0083] Where τ is the temperature hyperparameter. Includes a positive sample embedding feature (assuming) ) and K negative sample embedding features.

[0084] S54: During training, to ensure feature consistency within the queue, features should be generated using the same or similar momentum encoders and projectors to help the model avoid learning shortcut solutions. Therefore, the momentum encoder f m_encoder (·) and its projection head MLP m_encoder Momentum updates are used. This process can be formalized as follows:

[0085] θ m_encoder ←mθm_encoder +(1-m)θ encoder

[0086]

[0087] Where m∈[0,1) is the momentum coefficient, which is generally set to a large value. It is important to note the encoder f... encoder (·) and its projection head MLP encoder Updated via gradient backpropagation;

[0088] S6: Input the known prior target spectrum and the spectrum of each pixel in the hyperspectral image to be detected into the trained spectral feature extraction encoder network, and determine whether the spectrum of the pixel to be detected belongs to the target based on cosine similarity;

[0089] S6 specifically adopts the following approach: target prior Compared with the original hyperspectral image The spectrum of each pixel in the image is transmitted through a trained encoder f. encoder (·) Extract the corresponding representation, and then use cosine similarity to measure the similarity between the representation of each pixel spectrum in the original hyperspectral image and the prior representation of the target, to obtain the target detection result B = [b1; b2; ...; b H×W This process can be formalized as follows:

[0090]

[0091] S7: The detection results in S6 are subjected to two nonlinear boosts through exponential and normalization operations, and power function and normalization operations to suppress the background.

[0092] In S7, the specific approach is as follows: In the target detection result B obtained through cosine similarity, the target pixel value is relatively large, but the distance between the background pixel value and the target pixel value is relatively small. Exponential and normalization operations are used to distance the background pixel value from the target pixel value in the target detection result. Then, power function and normalization operations are used to further distance the background pixel value from the target pixel value, thereby achieving background suppression. Background suppression is achieved through exponential, normalization, power function, and normalization operations. This process can be formalized as follows:

[0093] S = α B

[0094]

[0095] R = S β

[0096]

[0097] α and β are positive parameters that adjust the background suppression performance.

[0098] The present invention will evaluate and assess a hyperspectral image target detection method based on momentum-free contrastive learning and Transformer network on two publicly available real hyperspectral image datasets.

[0099] The first set of experimental data is a real hyperspectral image dataset of San Diego, obtained from images of the San Diego airport area in California, USA, taken by an airborne visible / infrared imaging spectrometer (AVIRIS). The original image size is 400×400, with a wavelength range of 370 to 2510 nm, a spatial resolution of 3.5 m, and a spectral resolution of 10 nm. The experiment selected a 120×120 area, containing 58 target pixels (aircraft). After removing low signal-to-noise ratio and water absorption bands, 189 bands remained.

[0100] The second set of experimental data is the PaviaC real hyperspectral image dataset, captured by the ROSIS-03 imaging spectrometer in the central city of Pavia, Italy. It has 100×120 pixels and consists of 102 bands, with wavelengths ranging from 430 to 860 nm. Its spatial resolution is 1.3 meters, and its spectral resolution is 4 nm. The background in this scene mainly consists of water and a bridge, with vehicles on the bridge considered as targets. A total of 68 pixels were considered as targets for hyperspectral target detection.

[0101] This invention provides a hyperspectral image target detection method based on momentum-free contrastive learning and Transformer networks. The detection results are shown in the figure below, obtained from two sets of real hyperspectral image data. Figure 5b , Figure 6b As shown. The receiver operating characteristic curve (ROC) and area under the curve (AUC) are used to quantitatively analyze the detection results. For the San Diego dataset, (P... D ,P F ) and (P F The ROC curve of τ is as follows Figure 7a and 7b As shown, the area under the curve and The values ​​are 0.99877 and 0.00192, respectively. For the PaviaC dataset, (P D ,P F ) and (P F The ROC curve of τ is as follows Figure 8a and 8b As shown, the area under the curve and The values ​​are 0.99126 and 0.00007, respectively. The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the technical scope disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A hyperspectral target detection method based on unsupervised momentum contrast learning and a Transformer network, characterized in that include: Constructing an overlapping spectral block feature map and a spatial position information map as an embedded spectral vector sequence input to the Transformer encoder ; In the Transformer encoder, an interactive token feedforward layer is constructed and long-range spectral dependencies are established. An encoder and momentum encoder for spectral feature extraction and motion encoder are constructed based on the Transformer network for hyperspectral target detection tasks. A data augmentation approach is used to construct a spectral instance identification agent task for pixel spectral pairing. The spectral feature extraction encoder and momentum encoder were trained using an unsupervised momentum contrastive learning method. The trained spectral feature extraction encoder is used to extract the representation vectors of the spectrum and prior spectrum of each pixel in the hyperspectral image to be detected. The cosine similarity is used to measure the similarity between the spectral representation vector of the pixel to be detected and the spectral representation vector of the prior target, so as to obtain the detection result using only spectral information. Background noise is suppressed by performing two nonlinear enhancements on the detection results through exponential and normalization operations, as well as power function and normalization operations.

2. The method of claim 1, wherein: The construction process of the overlapping spectral block feature map and spatial location information map is as follows: For the band B pixel spectrum The original image is divided into several overlapping spectral blocks by convolution, thereby obtaining the embedded spectral vector sequence. ,in It is the length of the valid input sequence for the Transformer. d It is the dimension of each embedded spectral vector sequence, in the embedded spectral vector sequence Previously added learnable embedding maps ,in This is the output vector of the Transformer encoder. As a pixel spectral representation, learnable 1D positional information is then mapped and added to the embedded spectral vector sequence to preserve the positional information of the spectral patch in the original pixel spectrum. The final embedded spectral vector sequence serves as the input to the Transformer encoder. Represented as: in It is a learnable location information mapping.

3. The method according to claim 1, characterized in that: The construction process of the interaction token feedforward layer is as follows: Depth-wise convolutions are added between the two fully connected layers in the feedforward layer of the original Transformer encoder to supplement local details in the feedforward layer and to enable information exchange in different spectral channels. The process is as follows: Where Z is the input of the interaction token feedforward layer. and These are the parameters of two fully connected layers. These are the parameters of a 1D convolutional layer. It is the activation function of the Gaussian Error Linear Unit (GELU).

4. The method according to claim 1, characterized in that: By analyzing the original hyperspectral image Gaussian blur is used to achieve data augmentation, which is done by convolving each band in the original hyperspectral image with a Gaussian kernel.

5. The method according to claim 4, characterized in that: The method for constructing the spectral instance identification agent task is as follows: the pixel spectra at the same location in the original hyperspectral image and the hyperspectral image after Gaussian blurring are paired, where the pixel spectra at the same location in the original hyperspectral image and the hyperspectral image after data augmentation are regarded as positive sample pairs.

6. The method according to claim 1, characterized in that: When training the spectral feature extraction encoder and momentum encoder using unsupervised momentum contrastive learning: First, a small batch of original samples is obtained through random sampling. and enhanced samples Spectral feature extraction encoder and momentum encoder are used to extract respectively and The representation is input into each projection head to obtain the feature matrix. and The process is represented as: and They are encoders and momentum encoder The parameters of the encoder's projection head Projection head of momentum encoder These are parameters and An MLP containing one hidden layer and Use the ReLU activation function; The momentum encoder will feature the feature matrix. The samples are fed into a queue and used as a small negative sample set to achieve one-to-many contrastive learning of positive samples. The contrastive loss in the contrastive learning process uses the InfoNCE loss function, as shown below: in It's a temperature over-parameter. Includes a positive sample embedding feature (assuming) )and K Embedded features of each negative sample; Momentum encoder during training Its projection head Using momentum updates, this process is formalized as follows: in It's the momentum coefficient, the encoder. Its projection head Updates are performed using gradient backpropagation.

Images