An Adaptive Text Summarization Method with Multimodal Anchors
By employing a multimodal anchor adaptive text summarization method, which utilizes a multi-layered Cross-transformers cross-attention module and a gating mechanism model, the problem of inconsistent semantic representations between different modalities is solved, resulting in more comprehensive and accurate multimodal summaries.
Patent Information
- Application Number
- CN202410228552.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-29
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-02-29
AI Technical Summary
Existing multimodal summarization methods struggle to effectively learn semantic representations across different modalities, leading to redundancy and low summarization quality.
A multimodal anchor adaptive text summarization method is adopted. Through a multi-layered Cross-transformers cross-attention module and a gating mechanism model, positive and negative samples are selected by using the similarity measure between images and text to generate multimodal summaries.
The consistency of multimodal feature representation and the completeness of summary generation have been optimized, providing richer and more readable summary content.
Smart Images

Figure CN118035433B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to an adaptive text summarization method with multimodal anchors, belonging to the field of intelligent summarization technology. Background Technology
[0002] Multimodal summarization aims to condense and compress diverse media data (such as images, audio, and text) and generate summaries or generalizations using various presentation techniques. It represents an intersection of natural language processing and computer graphics, aiming to combine multiple media data to generate more comprehensive, accurate, and readable summaries.
[0003] Existing multimodal summarization methods face the following challenges:
[0004] 1) Different modalities exist in different semantic spaces, and it is crucial to express various modal information in a more similar semantic representation space.
[0005] 2) For the results of text summarization, it is necessary to consider how to learn the similarities and differences between various modalities, so as to reduce redundancy and improve the quality of text summarization.
[0006] To overcome the above challenges, text summarization of various media data is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0007] Objective: To overcome the shortcomings of existing technologies, this invention provides an adaptive text summarization method with multimodal anchors, which takes the source text, source image, and image text title in news content as input, and outputs the generated text summary and the generated image summary.
[0008] Technical solution: To solve the above technical problems, the technical solution adopted by the present invention is as follows:
[0009] An adaptive text summarization method with multimodal anchors includes the following steps:
[0010] Obtain sentence features from source text Image target features , source text sentence features Image target features Inputting the multimodal cross-attention module yields the sentence features of the cross-attention source text. Cross-attention image target features Cross-attention source text sentence features Cross-attention image target features The updated k-th layer source text sentence features are obtained through the feedforward sublayer. Image target features .
[0011] Update the k-th layer source text sentence features Image target features As a text sample Image target samples According to the text sample Image target samples Obtain positive target samples from images negative target image samples Positive sentence samples in text negative sentence samples in text .
[0012] Sample positive sentences from the text Input decoder to obtain positive features of text Sample negative sentences from the text Input decoder to obtain negative features of text .
[0013] The text summarization gating mechanism model is used to calculate the text's positive features. Negative features of the text Core text summarization probability distribution Selecting the probability distribution of core text summarization The largest vocabulary is used as a text summary.
[0014] As a preferred option, it also includes:
[0015] Active target samples of the image Input to decoder to obtain image positive features negative target samples of the image Input to decoder to obtain negative features of the image .
[0016] Image active features are calculated using an image summarization gating mechanism model. With negative features of the image Core image summary probability distribution Selecting the probability distribution of core image summarization The image with the highest probability is used as the image summary.
[0017] As a preferred option, the source text sentence features Image target features The acquisition methods specifically include:
[0018] Get the source text, the corresponding source images, and the text title for each source image.
[0019] Input the source text and text title into the BERT model, and the output will be the source text features. .
[0020]
[0021] in, , Let represent the word embedding vector of the s-th word in the m-th sentence, where m represents the number of sentences in the source text and s represents the maximum length of each sentence. This represents the text title corresponding to the a-th source image.
[0022] The word embedding vectors of s words in each sentence are merged to form the sentence features of the source text. , .
[0023] Each source image is input into a convolutional neural network, and the output is the corresponding image for each source image. Image target features of a target , .
[0024] As a preferred embodiment, the multimodal cross-attention module is composed of k layers of Cross-transformers connected in series. The implementation process of the k-th layer Cross-transformers involves converting the output of the (k-1)-th layer Cross-transformers... , As , ,Will enter The module obtains sentence features from the self-attention source text. ,Will enter The module obtains self-attention image target features. Then , enter The module obtains sentence features from the cross-attention source text. Then , enter The module obtains the target features of the cross-attention image. .
[0025] in, The module calculation formula is as follows:
[0026]
[0027] The module calculation formula is as follows:
[0028]
[0029] The module calculation formula is as follows:
[0030]
[0031] The module calculation formula is as follows:
[0032] .
[0033] As a preferred embodiment, the image active target sample negative target image samples The acquisition methods specifically include:
[0034] Each text sample is calculated using cosine similarity. and each image target sample Similarity measurement between , .
[0035] Based on similarity metrics Calculate the similarity distribution between each image target sample and all text samples. , .
[0036] Select similarity distribution The top O highest-ranking image target samples are used as image target anchors. .
[0037] Calculate the target sample for each image using cosine similarity. Image target anchor point Similarity performance metrics between and based on similar performance metrics Obtain the sorted image target sample set from high to low. .
[0038] Use activation functions to evaluate text samples. and image target samples The similarity between them is used to obtain the number of positive target samples in the image. .
[0039] Based on the sorted image target sample set Before choosing Each image target sample is used as an active target sample in the image. The rest are negative target samples in the image. .
[0040] As a preferred option, the text positive sentence sample negative sentence samples in text The acquisition methods specifically include:
[0041] Calculate positive target samples in images using cosine similarity. With text samples Cosine similarity, to obtain the similarity matrix , , Represents positive target samples in an image The j-th sample.
[0042] Calculate the similarity distribution between each text sample and all image positive target samples. , .
[0043] Negative target samples in images are calculated using cosine similarity. With text samples Cosine similarity, thus obtaining the similarity matrix. , , Indicates negative target samples in the image The j-th sample.
[0044] Calculate the similarity distribution between each text sample and all negative target image samples. , .
[0045] Build a gating mechanism model, considering and To determine the similarity between each text sample and the core semantics of the image. and according to similarity The text samples are sorted from highest to lowest to obtain the sorted set of text samples. ,in, The calculation formula is as follows:
[0046]
[0047] in: Let z represent the activation function, and z represent the gating weights. , These are parameter matrices for text and images, respectively.
[0048] Use an activation function to evaluate the text samples needed for the current text summarization process. and image target samples The similarity between them is used to obtain the number of text-positive target samples. Based on the sorted set of text samples Before choosing Each text target sample is used as a text positive sentence sample. The rest are samples of negative sentences in the text. .
[0049] As a preferred embodiment, the core text summary probability distribution The calculation formula is as follows:
[0050]
[0051] In the formula,
[0052] in, This represents the activation function. Indicates the weights of the positive features of the text in the gating process. This indicates the weight of negative features in the text during the gating process. The text positive weight matrix represents the vocabulary selection process in text summarization. This represents the negative weight matrix of the text in the vocabulary selection process for text summarization.
[0053] As a preferred embodiment, the core image summary probability distribution The calculation formula is as follows:
[0054]
[0055] In the formula,
[0056] in, This represents the activation function. Represents the weights of positive features in the gating process image. Indicates the weight of negative features in the gating process image. This represents the image positive weight matrix in the image summary selection process. This represents the negative weight matrix of images in the image summary selection process.
[0057] Beneficial Effects: This invention provides an adaptive text summarization method with multimodal anchors, which utilize multimodal anchors as bridges for multimodal alignment to mitigate the semantic gap between text and images. Based on the distance between the anchors and the semantic information of each modality, the positive and negative semantic information of each modality is distinguished. Based on the negative semantic information, a counterfactual learning mechanism is constructed to optimize the multimodal summarization results. Finally, multi-anchor selection of image summarization is used to optimize the multimodal feature interaction process. Experimental results show that, compared with existing multimodal summarization methods, the proposed method achieves improvements in the consistency of multimodal feature representation and the completeness of summary generation.
[0058] The advantage of the multimodal text summaries generated by the method of this invention lies in its ability to present information in various media formats, enriching the expression of the summaries and enabling users to gain a more comprehensive understanding of the original content. It can be applied to scenarios such as news summaries, multimedia advertisements, and social media summaries, providing richer and more diverse information presentation methods and enhancing the user's reading and comprehension experience. Attached Figure Description
[0059] Figure 1 This is a flowchart of the method of the present invention.
[0060] Figure 2 This is a schematic diagram of the overall process of the model of the present invention.
[0061] Figure 3 This diagram illustrates the distinction between image anchor points and positive and negative samples in an image.
[0062] Figure 4 A diagram illustrating the distinction between positive and negative samples in the text. Detailed Implementation
[0063] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.
[0064] The present invention will be further described below with reference to specific embodiments.
[0065] like Figure 1 As shown, an adaptive text summarization method with multimodal anchors includes the following steps:
[0066] Step S10: Collect the dataset and construct a large-scale Chinese summary dataset with complete image annotations and fine-grained image target object information. Specifically, select news documents and summaries from TTNEWS as text input. For each news document, search and crawl relevant images from Baidu, and select the three most relevant images to the text document as image input. In addition, each image is accompanied by a text title as text input. The dataset contains 52,656 training data points, 5,154 validation data points, and 5,070 test data points.
[0067] Step S20: For news documents and summaries, in the self-attention module, to obtain the autocorrelation within different modalities, this paper uses a pre-trained BERT model to extract the embedding vector of each word in the news source text, encoding the information of one word in each sentence. Word information encoding absolute position information of words The words are projected onto the same dimension and then summed to obtain the word embedding vector. .
[0068] Detection of each source image using a pre-trained Faster-RCNN Identify targets and extract corresponding target detection features. and location features Finally, the image target features are obtained by aggregating the target detection features and location features of each image. Each image target feature is treated as an image target sample.
[0069] Step S30: The target detection features (ROI Feat) and location features (Pos Feat) are added together to obtain the feature vector (Vision Output) for each target in each image. , Represents the feature of the j-th image target. .
[0070] The source text is divided into sentences, and the text titles of all images are also considered as sentences in the source text to obtain source text features (Caption & Language Output). , ,in, Let represent the word embedding vector of the s-th word in the m-th sentence, where m represents the number of sentences in the source text and s represents the maximum length of each sentence. Let represent the title of the 'a'-th image. The word embedding vectors of the 's' words in each sentence are merged to form the sentence features of the source text. , , Indicates the first in the source text Features of each sentence.
[0071] In short, for image targets, the location information and feature vectors are added to obtain a new image feature representation, which serves as the input for the next step of the model. For text modalities, the embedding vectors of the source text and image title are added to the text location embedding vector to obtain a new text character embedding representation, which serves as the input for the next step of the model.
[0072] Step S40: In the multimodal cross-attention module, to obtain the correlation between different modalities, the multimodal cross-attention model uses multiple layers of cross-transformers, utilizing the sentence features of the source text. Image target features They observe each other to update the feature representations of their respective modalities.
[0073] Through the multimodal cross-attention process, the dependencies between text modalities and image modalities will be deeply explored. In other words, the attention-based semantic relationships between source text sentence features and image target features will be finely aligned.
[0074] The multimodal cross-attention module consists of k layers of cross-transformers connected in series. The implementation process of the k-th layer cross-transformer involves taking the output of the (k-1)-th layer cross-transformer... , As , ,Will enter The module obtains sentence features from the self-attention source text. ,Will enter The module obtains self-attention image target features. Then , enter The module obtains sentence features from the cross-attention source text. Then , enter The module obtains the target features of the cross-attention image. The calculation formulas for each module can be expressed by formulas (1)-(4):
[0075] (1)
[0076] (2)
[0077] (3)
[0078] (4)
[0079] in: This represents the self-attention layer from sentence to sentence in the text. This represents the self-attention layer from image target to target. This represents the cross-attention layer from the sentence to the target. This represents the cross-attention layer from the target to the sentence. It is based on the updated source text sentence features from layer k-1. Image target features The outputs of the self-attention layers within the k-layer sentence features and image target features are obtained using formulas (1) and (2), respectively. , This allows us to obtain the relationships between samples within each modality (text sentence features, image target features). Then, as shown in formulas (3) and (4), the multimodal features updated through the self-attention layer are... , The input is processed through a cross-modal attention layer to calculate cross-modal relevance, thereby obtaining the updated cross-modal attention features between the source text sentence and the image target. , ).
[0080] Cross-attention source text sentence features Cross-attention image target features The updated source text sentence features at layer k are obtained through the feed-forward sub-layer. Image target features .
[0081] At this point, the multimodal encoder has completed the process of acquiring text sentence features (text sentence samples) and image target features (image target samples), and updated the k-th layer source text sentence features. Image target features As a text sample in step S50 Image target samples .
[0082] Step S50: In order to select the image semantics most relevant to the text semantics as the core shared semantics expressed by the image modality, so as to serve the text. Figure 2 The adaptive image semantic information discriminator shown selects the image target sample that is most similar to the text semantics as the image target anchor point by observing global text samples.
[0083] Then, the sample most similar to the image target anchor point in the image target sample is calculated as the positive target sample, and the sample with low similarity to the image target anchor point is the negative sample.
[0084] Each text sample is calculated using cosine similarity. and each image target sample Similarity measurement between them, obtaining the similarity measurement matrix of cross-modal text to image. The specific process is shown in formula (5).
[0085] (5)
[0086] The similarity distribution between each image target sample and all text samples is calculated using a similarity metric matrix. Specifically, as shown in formula (6).
[0087] (6)
[0088] To ensure the similarity between the selected image target samples and the text samples, thereby obtaining positive image target samples that fit the text semantics.
[0089] First, select the similarity distribution. The top O highest-ranking image target samples are used as image target anchors. .like Figure 2 The three images of the traffic accident scene each show multiple targets, such as vehicles and people, highlighted in red.
[0090] like Figure 3 As shown, then, the target sample of each image is calculated using cosine similarity. Image target anchor point Similarity performance metrics between and based on similar performance metrics Obtain the sorted image target sample set from high to low. The specific implementation is shown in formula (7):
[0091] (7)
[0092] Subsequently, image target samples are distinguished into positive target samples and negative target samples using image target anchor points. This invention uses Gumbel-softmax to evaluate text samples. and image target samples The similarity between them is used to measure the current text summarization process. The image serves as a positive target sample for support.
[0093] Finally, using the transformer's masking mechanism, an image mask sequence is set for the target sample in the image. To distinguish between positive and negative samples of image targets, based on the sorted set of image target samples. Before choosing Each image target sample is used as an active target sample in the image. The rest were negative samples. .
[0094] The specific implementation is shown in formulas (8)-(9):
[0095] (8)
[0096] (9)
[0097] in: This represents the parameter matrix.
[0098] In step S60, such as Figure 4 As shown, the calculation of positive target samples in the image With text samples Cosine similarity, thus obtaining the similarity matrix. The calculation is shown in formula (10):
[0099] (10)
[0100] Represents positive target samples in an image The j-th sample.
[0101] Then, the similarity distribution between each text sample and all positive target samples of the image is calculated using formula (11). .
[0102] (11)
[0103] Calculate negative target samples in images With text samples Cosine similarity, thus obtaining the similarity matrix. The calculation is shown in formula (12):
[0104] (12)
[0105] Indicates negative target samples in the image The j-th sample.
[0106] Then, the similarity distribution between each text sample and all negative target image samples is calculated using formula (13). .
[0107] (13)
[0108] To select the most relevant and most dissimilar images from text samples to the images' positive target samples, a gating mechanism model is constructed, taking into account... and To determine the similarity between each text sample and the core semantics of the image. and according to similarity The text samples are sorted from highest to lowest to obtain the sorted set of text samples. The specific implementation is shown in formula (14).
[0109] (14)
[0110] in: The activation function is Sigmoid, and z represents the gating weights used to measure the similarity matrix. and The importance of , All of these are learnable parameter matrices.
[0111] Using Gumbel-softmax to measure the current text summarization process's requirements For each positive sentence sample in the text, a text mask sequence is set using the transformer's masking mechanism to target the sentence samples in this text. To distinguish between positive and negative sentence samples in the text, based on the sorted set of text samples... Before choosing Each text target sample is used as a text positive sentence sample. The rest are samples of negative sentences in the text. .
[0112] The specific implementation is shown in formulas (15)-(16):
[0113] (15)
[0114] (16)
[0115] in: This represents the parameter matrix.
[0116] In step S70, positive information in the text modalities is used as information to be observed in the text summarization process. Positive sentence samples from the text are then used. Input the knowledge-sharing decoder to obtain positive text features. , , sample negative sentences from the text Input the knowledge-sharing decoder to obtain negative features of the text. .
[0117] In step S80, during the text summarization selection stage, the text's positive features are analyzed using a text summarization gating mechanism model. Negative features of the text Calculate the probability distribution of core text summarization Selecting the probability distribution of core text summarization The largest word is used as the text summary for the current generation step.
[0118] Specifically, this is achieved using formulas (17)-(18).
[0119] (17)
[0120] (18)
[0121] in, Indicates the weights of the positive features of the text in the gating process. This indicates the weight of negative features in the text during the gating process. The text positive weight matrix represents the vocabulary selection process in text summarization. This represents the negative weight matrix of the text in the vocabulary selection process for text summarization.
[0122] In step S90, the positive information in the image modalities is used as the information that needs to be observed in the image summarization process. The positive target samples of the image are... Input the knowledge-sharing decoder to obtain the positive features of the image. , to sample negative targets in the image Input the knowledge-sharing decoder to obtain negative features of the image. .
[0123] In step S100, during the image summarization selection stage, the image active features are selected through the image summarization gating mechanism model. With negative features of the image Calculate the probability distribution of core image summarization Selecting the probability distribution of core image summarization The image with the highest probability is used as the image summary.
[0124] Specifically, this is achieved using formulas (19)-(20).
[0125] (19)
[0126] (20)
[0127] in, Represents the weights of positive features in the gating process image. Indicates the weight of negative features in the gating process image. This represents the image positive weight matrix in the image summary selection process. This represents the negative weight matrix of images in the image summary selection process.
[0128] This invention first constructs a multimodal coding mechanism, dividing text information into sentence-level features and image features into target features. The target features are then co-encoded with the sentence features of the source text through a multimodal coding process, achieving cross-modal learning of multimodal feature representations. Notably, the image title is treated as a sentence-level feature in the cross-modal coding process and used as a text anchor for subsequent work.
[0129] Secondly, a multimodal anchor measurement mechanism is used to select the most representative sentence features (target features) from both text sentence features and image target features as anchors. This reduces the semantic gap between text and image modalities by narrowing the semantic distance between text and image anchors. Then, each modality uses its own anchor as a benchmark, with all other information blocks referencing it. The summary generation process adaptively confirms each modality's contribution, distinguishing between positive and negative information from both text and image modalities.
[0130] Finally, a decoder is constructed. The model autoregressively decodes the positive information to obtain the actual word probability distribution and selects the word with the highest probability as the summary word. To widen the gap between positive and negative information and thus highlight the feature representation of positive information, the model observes negative information at each step of decoding to obtain the counterfactual word probability distribution and performs counterfactual learning by minimizing the probability of the current summary word. Simultaneously, an image summary closely aligned with the text summary is generated through text-image anchors.
[0131] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. An adaptive text summarization method with multimodal anchors, characterized in that: Includes the following steps: Obtain sentence features from source text Image target features , source text sentence features Image target features Inputting the multimodal cross-attention module yields the sentence features of the cross-attention source text. Cross-attention image target features Cross-attention source text sentence features Cross-attention image target features The updated k-th layer source text sentence features are obtained through the feedforward sublayer. Image target features ; Update the k-th layer source text sentence features Image target features As a text sample Image target samples According to the text sample Image target samples Obtain positive target samples from images negative target image samples Positive sentence samples in text negative sentence samples in text ; Sample positive sentences from the text Input decoder to obtain positive features of text Sample negative sentences from the text Input decoder to obtain negative features of text ; The text summarization gating mechanism model is used to calculate the text's positive features. Negative features of the text Core text summarization probability distribution Selecting the probability distribution of core text summarization The largest vocabulary is used as a text summary; Active target samples of the image Input to decoder to obtain image positive features negative target samples of the image Input to decoder to obtain negative features of the image ; Image active features are calculated using an image summarization gating mechanism model. With negative features of the image Core image summary probability distribution Selecting the probability distribution of core image summarization The image with the highest probability is used as the image summary; The image is a positive target sample negative target image samples The acquisition methods specifically include: Each text sample is calculated using cosine similarity. and each image target sample Similarity measurement between , ; Based on similarity metrics Calculate the similarity distribution between each image target sample and all text samples. , ; Select similarity distribution The top O highest-ranking image target samples are used as image target anchors. ; Calculate the target sample for each image using cosine similarity. Image target anchor point Similarity performance metrics between and based on similar performance metrics Obtain the sorted image target sample set from high to low. ; Use activation functions to evaluate text samples. and image target samples The similarity between them is used to obtain the number of positive target samples in the image. ; Based on the sorted image target sample set Before choosing Each image target sample is used as an active target sample in the image. The rest are negative target samples in the image. .
2. The adaptive text summarization method with multimodal anchors according to claim 1, characterized in that: The source text sentence features Image target features The acquisition methods specifically include: Obtain the source text, the corresponding source images, and the text title for each source image; Input the source text and text title into the BERT model, and the output will be the source text features. ; ; in, , Let represent the word embedding vector of the s-th word in the m-th sentence, where m represents the number of sentences in the source text and s represents the maximum length of each sentence. This represents the text title corresponding to the a-th source image; The word embedding vectors of s words in each sentence are merged to form the sentence features of the source text. , ; Each source image is input into a convolutional neural network, and the output is the corresponding image for each source image. Image target features of a target , .
3. The adaptive text summarization method with multimodal anchors according to claim 1, characterized in that: The multimodal cross-attention module consists of k layers of cross-transformers connected in series; wherein, the implementation process of the k-th layer of cross-transformers is as follows: the output of the (k-1)-th layer of cross-transformers... , As , ,Will enter The module obtains sentence features from the self-attention source text. ,Will enter The module obtains self-attention image target features. Then , enter The module obtains sentence features from the cross-attention source text. Then , enter The module obtains the target features of the cross-attention image. ; in, The module calculation formula is as follows: ; The module calculation formula is as follows: ; The module calculation formula is as follows: ; The module calculation formula is as follows: 。 4. The adaptive text summarization method with multimodal anchors according to claim 1, characterized in that: The text positive sentence sample negative sentence samples in text The acquisition methods specifically include: Calculate positive target samples in images using cosine similarity. With text samples Cosine similarity, to obtain the similarity matrix , , Represents positive target samples in an image The j-th sample; Calculate the similarity distribution between each text sample and all positive target samples in the images. , ; Negative target samples in images are calculated using cosine similarity. With text samples Cosine similarity, thus obtaining the similarity matrix. , , Indicates negative target samples in the image The j-th sample; Calculate the similarity distribution between each text sample and all negative target image samples. , ; Build a gating mechanism model, considering and To determine the similarity between each text sample and the core semantics of the image. and according to similarity The text samples are sorted from highest to lowest to obtain the sorted set of text samples. ,in, The calculation formula is as follows: ; in: Let z represent the activation function, and z represent the gating weights. , These are parameter matrices for text and images, respectively; Use an activation function to evaluate the text samples needed for the current text summarization process. and image target samples The similarity between them is used to obtain the number of text-positive target samples. Based on the sorted set of text samples Before choosing Each text target sample is used as a text positive sentence sample. The rest are samples of negative sentences in the text. .
5. The adaptive text summarization method with multimodal anchors according to claim 1, characterized in that: The probability distribution of the core text summary The calculation formula is as follows: ; In the formula, ; in, This represents the activation function. Indicates the weights of the positive features of the text in the gating process. Indicates the weight of negative features in the text during the gating process. The text positive weight matrix represents the vocabulary selection process in text summarization. This represents the negative weight matrix of the text in the vocabulary selection process for text summarization.
6. The adaptive text summarization method with multimodal anchors according to claim 1, characterized in that: The probability distribution of the core image summary The calculation formula is as follows: ; In the formula, ; in, This represents the activation function. Represents the weights of positive features in the gating process image. Indicates the weight of negative features in the gating process image. This represents the image positive weight matrix in the image summary selection process. This represents the negative weight matrix of images in the image summary selection process.
