A video retrieval method based on cross-modal perturbation features

Abstract

This paper discloses a video retrieval method based on cross-modal perturbation features, which belongs to the field of video retrieval. The method first preprocesses data to extract video and query features. Then, a candidate segment generation module is constructed to perform feature fusion. Finally, a reconstruction module is constructed to output and evaluate video retrieval results and perform training. This method significantly improves video retrieval performance and enables more efficient video retrieval under weak supervision that does not rely on temporal boundaries.

Description

Technical Field

[0001] The present invention belongs to the field of video retrieval, and in particular relates to a video retrieval method based on cross-modal perturbation features. Background Art

[0002] The task of fully supervised temporal language localization has achieved remarkable results. Starting from the Cross-modal Temporal Regression Localizer (CTRL) and the Temporal Context Network (MCN), the possibility of cross-modal fusion of text and video has been explored, and traditional convolutional neural networks have been introduced to the video feature level. The Cross-modal Temporal Regression Localizer fuses the features of the two modalities, scores each candidate sample, and then locates the temporal boundary through regression, setting a paradigm for subsequent research.

[0003] Subsequent work has focused on better video encoding, using sentence queries as filters embedded in each convolutional step. Graph convolutional networks have also been used to further enhance feature representation. Previous models considered feature construction solely from the temporal dimension. The two-dimensional temporal adjacency network (2D-TAN) constructs a two-dimensional relational feature graph based on temporal relationships: given a video segment of length τ, coordinates (i, j) represent the feature matrix of candidate segments from start time iτ to (j+1)τ. This temporal relation matrix contains neighboring relationships between different time points and lengths. The temporal adjacency network fuses sentence representations with features at each candidate moment, then embeds video context using a convolutional neural network. Finally, a confidence score is predicted for each candidate as the final target segment. This two-dimensional matrix construction offers some scalability. However, full supervision requires laborious manual annotation of temporal boundaries for each query, limiting its scalability and practicality in practical applications. Consequently, weakly supervised learning schemes that require only video and natural language queries during training have attracted increasing attention due to their low annotation costs and reasonable efficiency. Summary of the Invention

[0004] In response to the above problems, the present invention proposes a video retrieval method based on cross-modal perturbation features, focusing on exploring highly confused negative samples to encourage the model to learn fine-grained cross-modal alignment, and proposes a Contrastive Perturbation Network (CPN), which introduces the perturbation mechanism into the proposal generation and reconstruction modules. In the proposal generation module, the KL divergence loss is used to compare the perturbed positive sample mask with the true positive sample mask, so that the network has better robustness to redundant video content and the ability to capture fine-grained cross-modal interactions. In the reconstruction module, a highly confused perturbed candidate feature, a reference sample and a series of negative samples from easy to difficult are generated. Negative samples are used to enhance the supervision of the network through contrastive learning.

[0005] The present invention provides an effective video retrieval method using perturbation features, and the technical solution adopted to solve the technical problem includes the following steps:

[0006] Step (1), video data preprocessing, extracting video features and query features;

[0007] Step (2), constructing a candidate segment generation module and performing feature fusion;

[0008] Step (3): construct a reconstruction module, output the video retrieval results, and construct a loss function for training.

[0009] The proposed Contrastive Perturbation Network (CPN) consists of two modules: a candidate segment generation module and a reconstruction module. The perturbation mechanism is introduced into these two modules of the Contrastive Perturbation Network. The candidate segment generation module predicts potential positive samples based on the original and perturbed video features, and by minimizing the KL divergence between the mask features of the true positive sample and the contrast (perturbed positive sample) mask features, it has robustness to redundant information and better video retrieval capabilities. The reconstruction module generates multiple negative samples, including a highly confused perturbed negative sample, and compares their ability to reconstruct the mask query through contrastive learning, providing a stronger supervision signal for weakly supervised network training.

[0010] Step (1) data preprocessing is specifically implemented as follows:

[0011] Obtain a video dataset, downsample each video to 8 frames, use the expanded 3D convolutional neural network model (I3D) to extract video features, and use the pre-trained word vector model (Glove) to process each word in each input sentence to form query features.

[0012] Encode the above two different modalities into their respective feature vectors: the query feature is represented as Where M represents the number of words, D T Represents the dimension of word features; for video features, it is represented as Among them D V represents the dimension of the video features, and N is the length of the video extracted by the expanded 3D convolutional neural network model.

[0013] Furthermore, the contrastive perturbation network is divided into two parts: a candidate segment generation module and a reconstruction module. The candidate segment generation module primarily performs a perturbation mechanism based on the extracted video and query features, and then generates a hybrid feature through cross-modal fusion. This hybrid feature is then used to generate various mask generation losses and candidate segments, which are used to generate more weak supervisory signals for subsequent modules.

[0014] Step (2) candidate segment generation module:

[0015] First, a perturbation mechanism is introduced before cross-modal fusion: the video features obtained in step (1) are transformed into For which v k Apply the perturbation operation to the segment and get the following formula:

[0016]

[0017] represents the feature after perturbation, ∏(·) represents the perturbation operation, and the perturbation feature of video V is given by Indicates that in the disturbance operation, the disturbance operation ratio γ is introduced to limit the number of disturbances.

[0018] Secondly, the candidate segment generation module constructs a two-dimensional feature map to enhance the expressiveness of features. The two-dimensional feature map contains features from two modalities. In the two-dimensional feature map, each unit length feature from one modality interacts with all features from the other modality: the query feature T is filled by the filling operation to obtain Aligned with the length of the video feature to obtain a two-dimensional feature map representation D V ,D T Represent the video feature dimension and query feature dimension respectively; similarly, by perturbing the video features, the perturbed two-dimensional feature map is obtained The specific calculation is as follows:

[0019] M=V·T N ,M * =V * ·T N

[0020] Then, the visual transformer VIT is used to process the two-dimensional feature maps M and M * , and apply a convolution operation to each element along the diagonal to obtain a mixed feature set of length N.

[0021] Finally, a visual transformer is used to represent the multimodal interaction between video features and text features, and a hybrid semantic feature H = {h1, h2, ..., h N , h cls}, h cls Represents the features that interact with all features and is used to calculate the positions of candidate features; the calculation of H is as follows:

[0022] H=D(patch(M),E(patch(M)))

[0023] Where E(·) and D(·) represent the encoder and decoder of the transformer respectively, and patch(·) represents the improved patch operation mentioned above; similarly, the same operation is used to process the M constructed using the perturbed features. * , get H * ; Use h in the mixed semantic feature H cls and Computing Distribution Center and distribution width K represents the number of generated candidate features, N represents the video length, and the candidate feature probability mask is generated according to the following formula:

[0024]

[0025] k=1,...,K;i=1,...,N

[0026] Among them, σ is a hyperparameter.

[0027] Applying the diversity loss to the candidate feature probability mask m obtained by the above formula gives the diversity loss:

[0028]

[0029] Among them, ||·|| F represents the F-norm of the matrix, and the hyperparameter λ∈[0, 1] controls the degree of overlap between candidate segments.

[0030] Step (3) Reconstruct the module:

[0031] The candidate feature probability mask m is used to generate negative samples, perturbed negative samples, and reference samples for comparative training of the candidate generation module.

[0032] The masked conditional reconstruction module based on the semantic reconstruction network (SCN) and the contrastive candidate network (CPL) measures the semantic relevance between candidate samples and query sentences. It uses word vectors to mask certain words in each query feature, and then fuses the masked word vectors with the obtained positive and negative candidate samples to restore the masked words. Assuming that different candidate samples have different reconstruction capabilities after masking, the following relationship is obtained:

[0033] R(m p ,Q)>R(m pert ,Q)>R(m r ,Q)>R(m n ,Q),

[0034] Among them, m p 、m pert 、m r and m nIn turn, it represents the probability mask of the positive, the probability mask of the perturbed positive sample, the reference probability mask of the entire segment, and the probability mask of the negative sample. Represents the cross entropy loss of reconstructed query statements.

[0035] Finally, the cross entropy loss is used to quantify the similarity between the reconstructed query vector and the original query vector. The cross entropy loss of positive samples, perturbation samples, reference samples and negative samples is expressed as and

[0036] Select the candidate features that are most relevant to the query features and calculate the reconstruction loss in The reference sample and perturbed negative samples Introducing the operation, the reconstruction loss Expressed as:

[0037]

[0038] Given that there is also a semantic relationship between positive samples, negative samples, reference samples and perturbation samples, the intra-video contrast loss is used. Compare four types of samples, where the hyperparameters β1 and β2 satisfy β1 < β2:

[0039]

[0040] KL divergence is introduced as a metric to quantify the similarity between the positive Gaussian probability distribution p and the perturbation probability distribution q. The KL loss is expressed as:

[0041]

[0042] Then the total loss function is:

[0043]

[0044] Among them, α1, α2, and η are hyperparameters set respectively, and are trained in an end-to-end manner.

[0045] The beneficial effects of the present invention are as follows:

[0046] 1. More efficient video retrieval under weak supervision that does not rely on temporal boundaries.

[0047] 2. A new cross-modal fusion strategy, two-dimensional feature map and its operation method are proposed to efficiently fuse different modalities.

[0048] 3. A perturbation mechanism is proposed and KL loss is introduced to evaluate the generated candidate features.

[0049] Results on benchmark datasets show that the proposed contrastive perturbation network significantly improves the performance of video retrieval and demonstrates the effectiveness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is an overview of the overall architecture of this model;

[0051] Figure 2 Schematic diagram of the two-dimensional feature map of the cross-modal fusion strategy of the present invention. DETAILED DESCRIPTION

[0052] The method of the present invention and its detailed parameters are further described in detail below.

[0053] A video retrieval method based on cross-modal perturbation features, such as Figure 1 As shown, the specific steps include:

[0054] Step (1) data preprocessing is specifically implemented as follows:

[0055] The benchmark dataset used is Charades-STA. Each video is downsampled to 8 frames, and the video features of Charades-STA are extracted using an inflated 3D convolutional neural network (I3D). A pre-trained word embedding model (Glove) is used to process each word in each input sentence to form query features.

[0056] Encode the above two different modalities into their respective feature vectors: the query feature can be expressed as Where M represents the number of words, D T Represents the dimension of word features. For video features, it is represented as Among them D V represents the dimension of the video features, and N is the length of the video extracted by the expanded 3D convolutional neural network model.

[0057] Furthermore, the contrastive perturbation network is divided into two parts: a candidate segment generation module and a reconstruction module. The candidate segment generation module primarily performs a perturbation mechanism based on the extracted video and query features, and then generates a hybrid feature through cross-modal fusion. This hybrid feature is then used to generate various mask generation losses and candidate segments, which are used to generate more weak supervisory signals for subsequent modules.

[0058] Step (2) candidate segment generation module:

[0059] In order to make better use of video information, a perturbation mechanism is introduced before cross-modal fusion to generate more adversarial samples, so as to generate more supervisory signals for the following, which can effectively enhance the model's ability to extract valuable information for text query positioning. The video features obtained in step (1) are For a specific vk Apply the perturbation operation to the segment and get the following formula:

[0060]

[0061] represents the perturbation feature, Π(·) represents the perturbation operation, and four types of perturbation operations are tried and compared. For example, v k Some dimensions of V are randomly assigned values ​​between 0 and 1. By studying the effects of different perturbation operations in experiments, we have a comprehensive understanding of their results. Therefore, the perturbation features of video V are given by To prevent excessive information loss due to perturbations, a perturbation operation ratio γ is introduced to limit the number of perturbations. For practical purposes, γ is set to 0.3 as it produces relatively good results.

[0062] Next, in order to input the video features and query features into cross-modal fusion, the candidate segment generation module constructs a two-dimensional feature map to enhance the expressive power of the features. The two-dimensional feature map contains features from both modalities. In this two-dimensional feature map, each unit length feature from one modality interacts with all features from the other modality, so that the features can more effectively collect global semantic information. To achieve this, the query feature T is padded by the padding operation. Aligned with the length of the video feature to obtain a two-dimensional feature map representation D V , D T Represent the video feature dimension and query feature dimension respectively. Similarly, by perturbing the video features, the perturbed two-dimensional feature map is obtained. The specific calculation is as follows:

[0063] M=V·T N , M * =V * ·T N

[0064] Next, instead of using the traditional transformer, which can only process one-dimensional sequence input, we use the visual transformer (VIT) to process the two-dimensional feature maps M and M * In order to achieve cross-modal interaction, the concept of patch operation in the visual transformer is improved. Specifically, a 3×3 convolution operation is applied to each element along the diagonal to obtain a mixed feature set of length N. This ensures that the pre-processed features have a certain semantic interaction ability before being input into the transformer, such as Figure 2 shown.

[0065] Then, a visual transformer is used to represent the multimodal interaction between video features and text features, and the hybrid semantic features H = {h1, h2, ..., h N , h cls Here, h cls Represents the features that interact with all features and is used to calculate the positions of candidate features. The calculation of H can be expressed as follows:

[0066] H=D(patch(M),E(patch(M)))

[0067] Here, E(·) and D(·) denote the encoder and decoder of the transformer, respectively, and patch(·) denotes the improved patch operation mentioned above. Similarly, the same operation is used to process the M constructed using the perturbed features. * , get H * In order to obtain the probability mask required by the next module, h in the mixed semantic feature H is used cls and To calculate the distribution center and distribution width K represents the number of generated candidate features, and N represents the video length. The candidate feature probability mask is generated according to the following formula:

[0068]

[0069] Among them, σ is a hyperparameter.

[0070] In order to generate more differentiated positive samples, the diversity loss is applied to the candidate feature probability mask m obtained by the above formula to obtain the diversity loss:

[0071]

[0072] Among them, ||•|| F represents the F norm of the matrix, and the hyperparameter λ∈[0, 1] controls the degree of overlap between candidate segments. This loss encourages reducing the degree of overlap and prevents them from converging to the same center and width, thereby ensuring the diversity of each generated candidate sample.

[0073] Step (3) Reconstruct the module:

[0074] In this module, the candidate probability mask m obtained in the previous module is used to generate various samples (negative samples, perturbed negative samples) and reference samples for comparison and training of the candidate generation module. A well-established assumption is that different samples have varying degrees of ability to reconstruct the query features, thus establishing a ranking. This assumption allows for the generation of robust supervisory signals to measure the model's output capabilities.

[0075] To compare positive and negative samples, we leverage the masked conditional reconstruction module from the Semantic Reconstruction Network (SCN) and the Contrastive Candidate Network (CPL) to measure the semantic relevance between candidate samples and the query. We use word embeddings to mask a few words from each query feature. The resulting positive and negative candidate samples are then fused with the masked word embeddings to restore the masked words. Assuming that different candidate samples have different reconstruction capabilities after masking, we obtain the following relationship:

[0076] R(m p ,Q)>R(m pert ,Q)>R(m r ,Q)>R(m n , Q), where m p 、m pert 、m r and m n The following represents the probability mask of the positive sample, the probability mask of the perturbed positive sample, the reference probability mask of the entire segment, and the probability mask of the negative sample, respectively. R(·) represents the cross-entropy loss of reconstructing the query. The positive sample should preserve the most complete semantics, so the loss is the lowest. The perturbed features generate the perturbed candidate segment from the most credible positive probability mask. Since only a small amount of random information is lost, the candidate segment retains most of the semantically relevant information. Therefore, compared with the reference sample, the perturbed negative candidate segment should better reconstruct the query. However, the negative sample has the lowest semantics, so the reconstruction loss is large.

[0077] Finally, the cross entropy loss is used to quantify the similarity between the reconstructed query vector and the original query vector. The cross entropy loss of positive samples, perturbation samples, reference samples and negative samples is expressed as and

[0078] Select the candidate features that are most correlated with the query features to calculate the reconstruction loss in In addition, the reference sample and perturbed negative samples operations are introduced because they all contain relevant information matching the query. Therefore, the final reconstruction loss is It can be expressed as:

[0079]

[0080] In order to better provide supervision information, given that there is a semantic similarity relationship between the positive samples, negative samples, reference samples and perturbation samples, the intra-video contrast loss is used. To compare four types of samples, where the hyperparameters β1 and β2 satisfy β1 < β2:

[0081]

[0082] KL divergence is introduced as a metric to quantify the similarity between the positive Gaussian probability distribution p and the perturbation probability distribution q. This loss is used to effectively align two different modalities, which can enhance the inference performance even in the presence of different information losses. KL loss can be expressed as:

[0083]

[0084] In summary, the network of consists of four parts of loss functions, which are used as supervisory signals. The generated video features help the model reconstruct the query vector to enhance the alignment between video and text. Positive and negative samples are compared to encourage the proposal generation module to output more relevant query vectors and semantically accurate temporal boundaries. Generate multiple different positive proposals with different video contents to improve recall. It measures the distance between the true positive mask and the perturbed positive mask and promotes greater fine-grained alignment between text and video modalities. Finally, the total loss is calculated. The network trained in an end-to-end manner can be expressed as:

[0085]

[0086] Among them, α1, α2, and η are the hyperparameters set respectively.

[0087] Example

[0088] To prepare data for training and evaluation, each video is downsampled every 8 frames, and its video features are extracted using the expanded 3D convolutional neural network model (I3D) adopted by the CharadesSTA dataset. Both frame-level representations are 1024-dimensional. For query sentences, a pre-trained word vector model is used to process each word token. The maximum description length is limited to 20, and the vocabulary size is limited to 8000. The visual transformer (VIT) is used as the backbone model in the cross-modal fusion process, and the mixed cross-modal features are pre-processed with two-dimensional feature maps.

[0089] In the Visual Converter, the number of attention heads H in each encoder layer's global multi-head attention mechanism was set to 8. In the local multi-head attention module, the video was divided into four segments, and local attention features were generated for each segment by concatenating four single-head attention mechanisms. Training was performed using the Adam optimizer (learning rate 0.0005, regularization factor 0.0001) in full batch mode for 50 iterations. All experiments were conducted using the PyTorch platform on a machine equipped with a GeForce RTX2080Ti GPU.

[0090] To evaluate the performance of the proposed method, we use "R@n, IoU = m," a performance metric commonly used for such tasks. Here, m is a predefined Intersection over Temporal Union (IoU) threshold, and n represents the recall of the top n predictions. This metric measures the percentage of prediction moments with an IoU greater than m among the top n predictions. Specifically, "R@n, IoU = m," where n ∈ {1, 5} and m ∈ {0.3, 0.5, 0.7}, is used for the Charades-STA dataset.

[0091]

[0092] Experimental results:

[0093] Table 1: Evaluation results on the Charades-STA dataset (where n∈{1,5},m∈{0.3,0.5,0.7}). The bold underlined numbers indicate the best results.

[0094]

[0095] Table 1 compares the performance of the contrast perturbation network with previous work. The results highlight the following: (1) On the Charades-STA dataset, the model significantly improves performance compared to the baseline model CPL and outperforms all previous state-of-the-art methods on most metrics. (2) The model's fusion strategy is more effective than previous methods, achieving better performance under weak supervision.

[0096] To verify the effectiveness of the proposed method, ablation experiments were conducted on the proposed method in the model. Table 2 shows the ablation results using different loss functions; Table 3 shows the performance impact of different perturbation forms on the dataset. Table 2 demonstrates that the various loss functions introduced into the model have a significant impact on the model's performance, and that as the constraints of the loss function increase, the model's generalization ability becomes stronger. Table 3 demonstrates that different perturbation strategies also affect the model's performance.

[0097] Table 2: The impact of different loss functions on the experiment (√ represents adoption, × represents non-adoption)

[0098]

[0099] Table 3: Effects of different perturbations on the experiment

[0100]

[0101] In this paper, a novel contrastive perturbation network is introduced, applying perturbation techniques to enhance candidate feature generation. By minimizing the KL divergence between the true and perturbed normal distribution masks, the candidate generation module is more robust to information redundancy. Furthermore, in the reconstruction module, more confusing perturbed negative samples participate in contrastive learning, providing a stronger supervisory signal and improving video retrieval performance. Experiments demonstrate promising results on the Charades-STA dataset. Furthermore, ablation studies confirm the effectiveness of the proposed additive approach.

[0102] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may be implemented in the form of entirely hardware, entirely software, or a combination of software and hardware.

Claims

1. A video retrieval method based on cross-modal perturbation features, characterized in that: The steps include: Step (1), video data preprocessing, extracting video features and query features; Step (2) constructs a candidate segment generation module and performs feature fusion. The specific process is as follows: First, a perturbation mechanism is introduced before cross-modal fusion: the video features obtained in step (1) are transformed into For which v k Apply the perturbation operation to the segment and get the following formula: represents the feature after perturbation, Π(·) represents the perturbation operation, and the perturbation feature of video V is given by express; Secondly, the candidate segment generation module constructs a two-dimensional feature map to enhance the expressiveness of features. The two-dimensional feature map contains features from two modalities. In the two-dimensional feature map, each unit length feature from one modality interacts with all features from the other modality: the query feature T is filled by the filling operation to obtain Aligned with the length of the video feature to obtain a two-dimensional feature map representation D V , D T Represent the video feature dimension and query feature dimension respectively; Similarly, by perturbing the video features, the perturbed two-dimensional feature map is obtained The specific calculation is as follows: M=V·T N ,M * =V * ·T N Then, the visual transformer VIT is used to process the two-dimensional feature maps M and M * , and apply a convolution operation to each element along the diagonal to obtain a mixed feature set of length N; Finally, a visual transformer is used to represent the multimodal interaction between video features and text features, and a hybrid semantic feature H = {h1, h2, ..., h N , h cls }, h cls Represents the features that interact with all features and is used to calculate the positions of candidate features; the calculation of H is as follows: H=D(patch(M),E(patch(M))) Where E(·) and D(·) represent the encoder and decoder of the converter respectively, and patch(·) represents the improved patch operation mentioned above; similarly, the same operation is used to process the M constructed using the perturbation features. * , get H * ; Use h in the mixed semantic feature H cls and Computing Distribution Center and distribution width K represents the number of generated candidate features, N represents the video length, and the candidate feature probability mask is generated according to the following formula: Among them, σ is a hyperparameter; Applying the diversity loss to the candidate feature probability mask m obtained by the above formula gives the diversity loss: in,‖·‖ F represents the F norm of the matrix, and the hyperparameter λ∈[0,1] controls the degree of overlap between candidate segments; Step (3): construct a reconstruction module, output the video retrieval results, and construct a loss function for training.

2. The video retrieval method based on cross-modal perturbation features according to claim 1, characterized in that: The specific process of step (1) is as follows: Obtain a video dataset, downsample each video to 8 frames, use the expanded 3D convolutional neural network model (I3D) to extract video features, and use the pre-trained word embedding model (Glove) to process each word in each input sentence to form query features. Encode the above two different modalities into their respective feature vectors: the query feature is represented as Where M represents the number of words, D T Represents the dimension of word features; for video features, it is represented as Among them D V represents the dimension of the video features, and N is the length of the video extracted by the expanded 3D convolutional neural network model.

3. The video retrieval method based on cross-modal perturbation features according to claim 2, characterized in that: In the disturbance operation, the disturbance operation ratio γ is introduced to limit the number of disturbances.

4. The video retrieval method based on cross-modal perturbation features according to claim 3 is characterized in that: The specific process of step (3) is as follows: Use the candidate feature probability mask m to generate negative samples, perturbed negative samples, and reference samples for comparative training of the candidate generation module; The masked conditional reconstruction module based on the semantic reconstruction network (SCN) and the contrastive candidate network (CPL) measures the semantic relevance between candidate samples and query sentences. It uses word vectors to mask certain words in each query feature, and then fuses the masked word vectors with the obtained positive and negative candidate samples to restore the masked words. Assuming that different candidate samples have different reconstruction capabilities after masking, the following relationship is obtained: R(m p ,Q)>R(m pert ,Q)>R(m r ,Q)>R(m n ,Q), Among them, m p 、m pert 、m r and m n represents the probability mask of the positive, the probability mask of the perturbed positive sample, the reference probability mask of the entire segment, and the probability mask of the negative sample, respectively. R(·) represents the cross entropy loss of the reconstructed query statement. Finally, the cross entropy loss is used to quantify the similarity between the reconstructed query vector and the original query vector. The cross entropy loss of positive samples, perturbation samples, reference samples and negative samples is expressed as and Select the candidate features that are most relevant to the query features and calculate the reconstruction loss in The reference sample and perturbed negative samples Introducing the operation, the reconstruction loss Expressed as: Given that there is also a semantic relationship between positive samples, negative samples, reference samples and perturbation samples, the intra-video contrast loss is used. Compare four types of samples, where the hyperparameters β1 and β2 satisfy β1<β2: KL divergence is introduced as a metric to quantify the similarity between the positive Gaussian probability distribution p and the perturbation probability distribution q. The KL loss is expressed as: Then the total loss function is: Among them, α1, α2, and η are hyperparameters set respectively, and are trained in an end-to-end manner.

Images