An abstract generation system based on multi-modal key information analysis
By filtering secondary information from text and images using multimodal selection gating networks and visual selection gating networks, and combining multi-head attention mechanisms to model the interdependence between images and text, high-quality multimodal summaries are generated. This solves the problem of information redundancy in existing models and improves the effectiveness of summarization.
Patent Information
- Application Number
- CN202211628261.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-17
- Publication Date
- 2026-02-27
- Estimated Expiration
- 2042-12-17
AI Technical Summary
Existing multimodal summarization generation models fail to fully extract key multimodal information, resulting in secondary and redundant information in the input stream, which increases the pressure on the decoder and affects model performance.
Multimodal selection gating network and visual selection gating network are used to filter secondary and redundant information in text and images. Multi-head attention mechanism is combined to model the interdependence between objects and text sequences in images. Key information is mined through multi-head selection attention module to generate high-quality text summaries.
It effectively filters out secondary and redundant information, highlights key object information in images, and improves the quality of multimodal summarization generation.
Smart Images
Figure CN115934932B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of automatic abstract generation, and particularly relates to a summary generation system based on multi-modal key information analysis. BACKGROUND
[0002] Multi-modal summary aims to mine various modal contents such as text, video, picture and the like, and generate a core summary. In recent years, as multi-modal data is flooding our life, it is increasingly urgent to enable a computer to understand multi-modal information and generate a summary.
[0003] At present, automatic summary generation researches are mainly divided into text summary and multi-modal summary.
[0004] Text summary task. Text summary refers to outputting a short and accurate core summary based on source text information. The task includes extractive text summary and generative text summary. The main idea of extractive summary is to extract some important phrases or sentences from the source text and recombine them to form a summary. Common methods include Lead-3, clustering, TextRank algorithm and the like. Extractive summary is stable in effect, simple to implement and not easy to completely deviate from the main idea of the article, but the number of words in the output summary is not easy to control and the coherence is poor. With the research of deep learning, generative text summary has attracted widespread attention. The model understands the content of the original text and reorganizes the language to summarize it, and the generated summary has higher flexibility and has a certain probability to generate new words or phrases, close to the human thinking mode. At present, the sequence-to-sequence (Seq2Seq) model is widely used in generative summary tasks and has achieved certain results.
[0005] Multi-modal summary task. With the vigorous development of the Internet, the problem of multi-modal information overload is becoming increasingly serious, and it is urgent to obtain a summary based on multi-modal data. The task can be divided into multi-modal input single-modal output (MISO) and multi-modal input multi-modal output (MSMO). The former input is multi-modal information, and the output is single-modal information of text; the latter input is multi-modal information, and the output is also multi-modal information. Researchers propose a multi-modal fusion block (Multimodal fusion block, MFB) to model the complementary and verification relationship between text and image, and obtain image position information through a position perception mechanism. However, non-text information (such as images) is usually integrated into the text decoding stage, and the decoder simultaneously undertakes the tasks of text generation and text compression. At present, the model method does not fully mine the multi-modal key information, and there is relatively much secondary and redundant information in the input stream, which increases the pressure on the decoder to a certain extent and affects the performance of the model. SUMMARY
[0006] The application provides a summary generation system based on multi-modal key information analysis, which can mine key information of text and images through a multi-modal selection gating network and a visual selection gating network, introduce a multi-head attention mechanism to model the mutual dependence between objects in the image and the text sequence, further highlight the key object information of the image, and generate a high-quality text summary.
[0007] The application adopts the following technical solutions.
[0008] A summary generation system based on multi-modal key information analysis comprises the following modules.
[0009] A text encoder module is configured to obtain encoding information of the text.
[0010] An image encoder module is configured to obtain feature information of the image at different angles.
[0011] A selection gating mechanism module is configured to mine key information of the text and image modalities, and comprises a multi-modal selection gating network and a visual selection gating network.
[0012] A multi-head selection attention module is configured to mine key object information in the image.
[0013] A multi-modal feature fusion module is configured to obtain multi-modal context representation.
[0014] A decoder module is configured to generate a summary statement.
[0015] When generating a summary, the system introduces object information of the image, and simultaneously shares the multi-modal key information extraction process of text compression to the encoding end, and the specific method is as follows: the multi-modal selection gating network and the visual selection gating network are used to filter secondary and redundant information of the text and the image, mine key information, and the multi-head attention mechanism of the multi-head selection attention module is used to model the mutual dependence between objects in the image and the text sequence, highlight the key object information in the image, and thus generate a high-quality text summary.
[0016] When the text encoder module works, the Bi-LSTM is used to encode the text sequence, the forward and backward hidden layer vectors are concatenated as the text encoding information, and the encoding information at the last time step is taken as the global feature of the text sequence.
[0017] First, the source text is segmented; then, the text data is converted from the text form into the word embedding vector through Word2Vec; finally, the word embedding vector is encoded through the Bi-LSTM network, and the forward and backward hidden layer vectors are concatenated to obtain the initial text feature h i The specific formula is as follows:
[0018]
[0019]
[0020]
[0021]
[0022] where E[x i ] represents the word vector of the input word x i , is the concatenation symbol, and r is the hidden layer vector at the last time step of the Bi-LSTM.
[0023] The image encoder module introduces a ResNet-152 network to extract image feature information, including local features and global features, and introduces a Faster R-CNN network to extract object features in the image, and then projects the three different angle visual information into the same vector space as the text information; the method for obtaining the initial feature information of different angles of the image is specifically as follows:
[0024] The ResNet-152 pre-training network model is introduced, and the input vector of the last fully connected layer (Fully Connected Layers, FC) of the network is taken as the image global feature, and the output of the last average pooling layer (avgpool) is mapped into the image local feature;
[0025] The Faster R-CNN network includes a Faster R-CNN model pre-trained by Facebook;
[0026] The algorithm of the Faster R-CNN network includes two parts: (1) a candidate region network RPN: used to generate candidate boxes. (2) a Faster R-CNN detector: based on the candidate boxes extracted by the RPN, the target in the candidate box is detected and recognized;
[0027] When the Faster R-CNN network is called, the entire picture is first input into the CNN to generate the convolution feature map feature map of the image, then the RPN network uses an active window to traverse the feature map to generate target object candidate box information, and then the ROIPooling layer is used to resize the feature maps of different sizes to a uniform size, and then the non-maximum suppression algorithm NMS is used to screen 16 object candidate boxes with the highest confidence;
[0028] Finally, through linear transformation, the three kinds of visual features are projected into the same vector space as the text features. The specific formula is as follows:
[0029] g=ResNetfc (I) Formula five;
[0030] (a1, a2,..., a 49 ) = ResNet avgpool (I) Formula six;
[0031] (o1, o2,..., o 16 ) = FasterRCNN ROIPooling (I) Formula seven;
[0032] g = U r g Formula eight;
[0033] A = (a1, a2,..., a 49 ) = W r (a1, a2,..., a 49 ) Formula nine;
[0034] O = (o1, o2,..., o 16 ) = V r (o1, o2,..., o 16 ) Formula ten;
[0035] Wherein, a i (i = 1, 2,..., 49) represents image local features, corresponding to a grid area in the image, o i (i = 1, 2,..., 16) represents an object candidate box feature in the image, g represents the image global feature, W r , U r , V r are model trainable parameter matrices.
[0036] The multi-head selection attention module takes the text key information as the query vector Query, the object information extracted by Faster R-CNN in the image as the key information vector Key and the content vector Value, captures the mutual dependence between the objects in the image and the text sequence, and then splices it with the text key information, while suppressing useless noise, further highlighting the key object information in the image, and the specific method is:
[0037] The text key information H is taken as the query vector Query, the object information O extracted by Faster R-CNN in the image is taken as the key information vector Key and the content vector Value, different projection information is established in multiple different projection spaces through the multi-head attention mechanism, the mutual dependence between the objects in the image and the text sequence is captured, while suppressing useless noise, further highlighting the key object information in the image, and then splicing with the text key information, and the specific formula is as follows:
[0038]
[0039] MATT(H,O)=[ATT1(H,O),...,ATT h (H,O)]W o Equation twelve;
[0040] H'=[H,MATT(H,O)]W h Equation thirteen;
[0041] where d k ∈R d / h , W i Q , W i K , W i V , W o , W h are model trainable parameter matrices.
[0042] The selection gate mechanism module introduces a multimodal selection gate network and a visual selection gate network to filter noise on the text and image modalities respectively, capture key information of each modality, and the method is as follows:
[0043] The selection gate mechanism module includes a multimodal selection gate network and a visual selection gate network; a text hidden layer state vector h i , a text global feature r obtained by the text encoder module, and an image global feature g obtained by the image encoder module are used to generate a multimodal selection gate vector The text secondary information is filtered, and key information is mined, and the specific formula is as follows:
[0044]
[0045]
[0046] H=(h1',h'2,...,h' n ) Equation sixteen;
[0047] where σ is a sigmoid function, W m , V m , U m are model trainable parameter matrices, and represents matrix element point multiplication.
[0048] The visual selection gate network uses the image global feature g and the image local feature a i obtained by the image encoder module to generate a visual selection gate vector The text secondary information is filtered, and key information is mined, and the specific formula is as follows: Filter the image secondary information, and dig out its key information. The specific formula is as follows:
[0049]
[0050]
[0051] Wherein, sigma is sigmoid function, W i , U i is the model trainable parameter matrix, and is the matrix element point multiplication.
[0052] The multi-modal feature fusion module first introduces a text attention mechanism and an image attention mechanism to capture the dependency between the decoder hidden layer and the text and image, and to obtain the context information of each modality. Then, the multi-modal attention mechanism is used to fuse the text and visual context information to obtain a multi-modal feature vector, i.e. to mix the information of the two modalities. The method is as follows:
[0053] The hierarchical attention mechanism introduced by the multi-modal feature fusion module is composed of a text attention layer, an image attention layer and a multi-modal attention layer. First, the text attention and the image attention are used to give different attention weights to different words in the input text or different local regions in the image within the respective modality channel, and to generate a text context vector and a visual context vector, respectively.
[0054] Then, the multi-modal attention gives different weights to the two modalities, and the two modality context vectors are weighted and summed according to the weights to obtain a multi-modal feature vector. The specific formula is as follows:
[0055]
[0056]
[0057]
[0058] Wherein, s t-1 represents the hidden layer state of the decoder at the t-1 time step, represents the attention weight of the decoder at the current time step t to the word x i in the input text, W w , U w is the model trainable parameter matrix;
[0059]
[0060]
[0061]
[0062] wherein, denotes the attention weight of the i-th region in the image at the current time step t by the decoder, W a , U a are trainable parameter matrices of the model.
[0063] The decoder module first initializes the decoder LSTM with the image global feature, and then at each decoding time step, calculates the vocabulary probability distribution through the multi-modal feature vector, and combines the pointer mechanism to select a new word from the vocabulary or copy a word from the source text to obtain the final probability distribution of the predicted word, generates the summary sentence, and the specific method is:
[0064] The decoder module uses LSTM as the decoder, first initializes the decoder LSTM with the image global feature g obtained by the image encoder module to obtain the initial state s0, and then introduces the pointer network on this basis, and calculates the probability p of generating a new word at each decoding time step gen , calculates the probability distribution p gen according to the value of the pointer p w , guides the predicted word to be generated from the vocabulary or copied from the source text, and the specific formula is as follows:
[0065] s0=tanh(W s r+U s g+b s ) formula twenty-five;
[0066]
[0067]
[0068]
[0069] wherein, c mm denotes the multi-modal context vector, s t denotes the hidden layer state of the decoder at the current time step, E[y t ] denotes the word vector of the target word y t , P vocab denotes the probability distribution of the predicted word w in the vocabulary, denotes the attention distribution of the decoder to the text sequence, W s , U s , V, V s , V c are trainable parameter matrices of the model, and b s is a bias parameter.
[0070] The abstract generation system constantly adjusts the model network according to the forward propagation of information and the backward propagation of errors in the model training stage of the decoder module, and gradually optimizes the objective function.
[0071] The application can mine key information of text and images through the multimodal selection gating network and the visual selection gating network, introduce a multi-head attention mechanism to model the mutual dependence between objects in the image and the text sequence, further highlight the key object information of the image, and thus generate a high-quality text abstract. BRIEF DESCRIPTION OF DRAWINGS
[0072] The application will be further described in detail below in combination with the drawings and specific embodiments:
[0073] ATTACHED Figure 1 is a schematic diagram of the application. DETAILED DESCRIPTION
[0074] As shown in the figure, an abstract generation system based on multimodal key information analysis includes the following modules:
[0075] A text encoder module is used to obtain encoding information of the text.
[0076] An image encoder module is used to obtain feature information of the image from different angles.
[0077] A selection gating mechanism module is used to mine key information of text and image modalities, including a multimodal selection gating network and a visual selection gating network.
[0078] A multi-head selection attention module is used to mine key object information in the image.
[0079] A multimodal feature fusion module is used to obtain multimodal context representation.
[0080] A decoder module is used to generate an abstract sentence.
[0081] When the system generates an abstract, the object information of the image is introduced, and the multimodal key information extraction process of text compression is simultaneously shared by the encoding end, and the specific method is as follows: the multimodal selection gating network and the visual selection gating network are used to filter the secondary and redundant information of the text and the image, mine key information, and the multi-head attention mechanism of the multi-head selection attention module is used to model the mutual dependence between objects in the image and the text sequence, highlight the key object information in the image, and thus generate a high-quality text abstract.
[0082] When the text encoder module works, the Bi-LSTM is used to encode the text sequence, the forward and backward hidden layer vectors are concatenated in series as the text encoding information, and the encoding information of the last time step is taken as the global feature of the text sequence; specifically,
[0083] First, the source text is segmented into words. Next, Word2Vec is used to convert the text data from text format into word embedding vectors. Finally, a Bi-LSTM network is used to encode the word embedding vectors, concatenating the forward and backward hidden layer vectors to obtain the initial text features h. i The specific formula is as follows:
[0084]
[0085]
[0086]
[0087]
[0088] Wherein, E[x i ] indicates that the input word is x i Word vectors, is the connection symbol, and r is the hidden layer vector of the last time step of the Bi-LSTM.
[0089] The image encoder module incorporates a ResNet-152 network to extract image feature information, including local and global features, and a Faster R-CNN network to extract object features from the image. These three different visual angles are then projected into the same vector space as the text information. The specific method for obtaining initial feature information from different angles of the image is as follows:
[0090] A ResNet-152 pre-trained network model is introduced, and the input vector of the last fully connected layer (FC) of the network is used as the global feature of the image, while the output of the last average pooling layer is mapped to the local feature of the image.
[0091] The Faster R-CNN network includes a Faster R-CNN model pre-trained by Facebook;
[0092] The Faster R-CNN network algorithm consists of two parts: (1) Region Candidate Network (RPN): used to generate candidate boxes. (2) Faster R-CNN detector: based on the candidate boxes extracted by the RPN, it detects and identifies the targets in the candidate boxes;
[0093] In the FasterR-CNN network, the whole picture is first input into the CNN to generate a feature map of the image, and then the RPN network uses an active window to traverse the feature map to generate target object candidate box information, and then the ROIPooling layer is used to resize the feature maps of different sizes to a uniform size, and then the non-maximum suppression algorithm NMS is used to screen 16 object candidate boxes with the highest confidence;
[0094] Finally, the three visual features are projected into the same vector space as the text feature through linear transformation. The specific formula is as follows:
[0095] g=ResNet fc (I) Formula five;
[0096] (a1,a2,...,a 49 )=ResNet avgpool (I) Formula six;
[0097] (o1,o2,...,o 16 )=FasterRCNN ROIPooling (I) Formula seven;
[0098] g=U r g Formula eight;
[0099] A=(a1,a2,...,a 49 )=W r (a1,a2,...,a 49 ) Formula nine;
[0100] O=(o1,o2,...,o 16 )=V r (o1,o2,...,o 16 ) Formula ten;
[0101] Where, a i (i=1,2,...,49) represents the local feature of the image, corresponding to a grid area in the image, o i (i=1,2,...,16) represents the feature of an object candidate box in the image, g represents the global feature of the image, W r , U r , V r are model trainable parameter matrices.
[0102] The multi-head selection attention module takes the text key information as a query vector Query, the object information extracted by the Faster R-CNN in the image as a key information vector Key and a content vector Value, captures the mutual dependence between the objects in the image and the text sequence, and then splices the text key information, thereby suppressing useless noise and further highlighting the key object information in the image.
[0103] The text key information H is taken as a query vector Query, the object information O extracted by the Faster R-CNN in the image is taken as a key information vector Key and a content vector Value, different projection information is established in multiple different projection spaces through a multi-head attention mechanism, the mutual dependence between the objects in the image and the text sequence is captured, useless noise is suppressed, and the key object information in the image is further highlighted, and then the text key information is spliced, and the specific formula is as follows:
[0104]
[0105] MATT(H,O)=[ATT1(H,O),...,ATT h (H,O)]W o Formula twelve;
[0106] H'=[H,MATT(H,O)]W h Formula thirteen;
[0107] Wherein, d k ∈R d / h , W i Q , W i K , W i V , W o , W h are model trainable parameter matrices.
[0108] The selection gate mechanism module introduces a multi-modal selection gate network and a visual selection gate network, respectively filters the noise on the text and image modalities, and captures the key information of the respective modalities, and the method is as follows:
[0109] The selection gate mechanism module includes a multi-modal selection gate network and a visual selection gate network; a text hidden layer state vector h i , a text global feature r obtained by the text encoder module, and an image global feature g obtained by the image encoder module are used to generate a multi-modal selection gate vector The text secondary information is filtered again through to mine the key information, and the specific formula is as follows:
[0110]
[0111]
[0112] H=(h1',h'2,...,h' n )Formula sixteen;
[0113] where σ is sigmoid function, W m , V m , U m are model trainable parameter matrix, and ⊙ represents matrix element point multiplication.
[0114] The visual selection gating network uses the image global feature g and the image local feature a i obtained by the image encoder module to generate a visual selection gating vector Then, the image secondary information is filtered, and the key information is mined. The specific formula is as follows:
[0115]
[0116]
[0117] where σ is sigmoid function, W i , U i are model trainable parameter matrix, and ⊙ represents matrix element point multiplication.
[0118] The multi-modal feature fusion module first introduces a text attention mechanism and an image attention mechanism to capture the dependency relationship between the decoder hidden layer and the text and image, and to obtain the context information of each modality. Then, a multi-modal attention mechanism is used to fuse the text and visual context information to obtain a multi-modal feature vector, i.e., to mix the information of the two modalities. The method is as follows:
[0119] The hierarchical attention mechanism introduced by the multi-modal feature fusion module is composed of a text attention layer, an image attention layer, and a multi-modal attention layer,
[0120] First, the text attention and the image attention are used to give different attention weights to different words in the input text or different local regions in the image within the respective modality channel, and a text context vector and a visual context vector are generated respectively.
[0121] Then, the multi-modal attention gives different weights to the two modal data, and the two modal context vectors are weighted and summed according to the weights to obtain a multi-modal feature vector. The specific formula is as follows:
[0122]
[0123]
[0124]
[0125] where s t-1 denotes the hidden state of the decoder at time step t-1, denotes the attention weight of the decoder at current time step t on the word x i in the input text, W w , U w are model trainable parameter matrices.
[0126]
[0127]
[0128]
[0129] where, denotes the attention weight of the decoder at current time step t on the i-th region in the image, W a , U a are model trainable parameter matrices.
[0130] The decoder module first initializes the decoder LSTM with the image global feature, and then at each decoding time step, calculates the vocabulary probability distribution through the multi-modal feature vector, and combines the pointer mechanism to select a new word from the vocabulary or copy a word from the source text to obtain the final probability distribution of the predicted word, and generates the summary sentence, the specific method is:
[0131] The decoder module uses LSTM as the decoder, first initializes the decoder LSTM with the image global feature g obtained by the image encoder module to obtain the initial state s0, and then introduces the pointer network on this basis, and calculates the probability p gen of generating a new word at each time step decoding, calculates the probability distribution p gen according to the value of the pointer p w , and guides the predicted word to be generated from the vocabulary or copied from the source text, and the specific formula is as follows:
[0132] s0=tanh(W s r+U s g+b s ) formula twenty-five;
[0133]
[0134]
[0135]
[0136] where c mm represents the multi-modal context vector, s t represents the hidden layer state of the decoder at the current time step, E[y t ] represents the word vector of the target word y t , P vocab represents the probability distribution of the predicted word w in the vocabulary, represents the attention distribution of the decoder to the text sequence, W s , U s , V, V s , V c are parameter matrices trainable by the model, and b s is a bias parameter.
[0137] The summary generation system will constantly adjust the model network according to the forward propagation of information and the backward propagation of errors during the model training phase of the decoder module, gradually optimizing the objective function.
Claims
1. A summary generation system based on multi-modal key information analysis, characterized by: The system comprises the following modules: a text encoder module for obtaining encoding information of the text; an image encoder module for obtaining feature information of the image at different angles; a selection gate mechanism module for mining key information of the text and image modalities, comprising a multi-modal selection gate network and a visual selection gate network; a multi-head selection attention module for mining key object information in the image; a multi-modal feature fusion module for obtaining multi-modal context representation; a decoder module for generating an abstract sentence; When generating an abstract, the system introduces object information of the image, and simultaneously shares the process of extracting multi-modal key information of text compression to the encoding end. Specifically, the multi-modal selection gate network and the visual selection gate network are used to filter secondary and redundant information of the text and the image, and to mine key information. Meanwhile, the multi-head attention mechanism of the multi-head selection attention module is used to model the mutual dependence between objects in the image and the text sequence, and to highlight key object information in the image, so as to generate a high-quality text abstract. The image encoder module introduces a ResNet-152 network to extract image feature information, including local features and global features, and introduces a Faster R-CNN network to extract object features in the image, and then projects the three kinds of visual information at different angles into the same vector space as the text information. The method for obtaining initial feature information of the image at different angles is as follows: A ResNet-152 pre-trained network model is introduced, and the input vector of the last fully connected layer (FC) of the network is taken as the global feature of the image, and the output of the last average pooling layer (avgpool) is mapped into the local feature of the image. The Faster R-CNN network comprises a Faster R-CNN model pre-trained by Facebook; The algorithm of the Faster R-CNN network comprises two parts: (1) a candidate region network RPN for generating a candidate frame; and (2) a Faster R-CNN detector for detecting and identifying a target in the candidate frame based on the candidate frame extracted by the RPN. When the Faster R-CNN network is called, the entire image is first input into the CNN to generate a convolution feature map of the image, then the RPN network uses an active window to traverse the feature map to generate target object candidate frame information, and then the ROIPooling layer is used to resize the feature maps of different sizes to a uniform size, and then the non-maximum suppression algorithm NMS is used to screen 16 object candidate frames with the highest confidence; Finally, the three kinds of visual features are projected into the same vector space as the text features by linear transformation; and the specific formula is as follows: g = ResNet fc (I) Equation Five; (a1,a2,...,a 49 ) = ResNet avgpool (I) Equation Six; (o1, o2,..., o 16 ) = FasterRCNN ROIPooling (I) Equation Seven; g = U r g Formula Eight; A = (a1, a2,..., a 49 ) = W r (a1, a2,..., a 49 ) Equation Nine; O = (ol, o2,..., o 16 ) = V r (o1, o2,..., o 16 ) Equation Ten; wherein, a i (i = 1, 2, …, 49) represents an image local feature, corresponding to a grid area in the image, o i (i = 1, 2, …, 16) represents an image object candidate box feature, g represents an image global feature, W r , U r , V r is a model trainable parameter matrix; The multi-head selection attention module takes the text key information as a query vector Query, the object information in the image extracted by the FasterR-CNN as a key information vector Key and a content vector Value, captures the mutual dependence between the objects in the image and the text sequence, and then splices the text key information, thereby further highlighting the key object information in the image while suppressing useless noise, and the specific method is as follows: The multi-head selection attention module takes the text key information as a query vector Query, the object information in the image extracted by the FasterR-CNN as a key information vector Key and a content vector Value, captures the mutual dependence between the objects in the image and the text sequence, and then splices the text key information, thereby further highlighting the key object information in the image while suppressing useless noise, and the specific method is as follows: MATT(H, O) = [ATT1(H, O),...,ATT h (H, O)]W o Equation Twelve; H' = [H, MATT(H, O)]w h Equation Thirteen; where d k ∈ R d / h , W i Q , W i K , W i V , W o , W h are model trainable parameter matrices; The selection gate mechanism module introduces a multi-modal selection gate network and a visual selection gate network to filter the noise in the text and image modalities respectively and capture the key information of the respective modalities, and the method is as follows: The selection gate mechanism module comprises a multi-modal selection gate network and a visual selection gate network; a text hidden layer state vector h obtained by using a text encoder module, a text global feature r, and an image global feature g obtained by using an image encoder module are used to generate a multi-modal selection gate vector i The multi-modal selection gate vector is used to filter the text secondary information and mine key information of the text secondary information. The multi-modal selection gate vector is used to filter the text secondary information and mine key information of the text secondary information. H = (h1', h'2,..., h'N) n Equation Sixteen; where σ is a sigmoid function, W m , V m , U m are model trainable parameter matrices, and denotes element-wise multiplication of matrices. Visual selection gating networks utilize the global image features g and local image features a obtained from the image encoder module. i Generate visual selection gating vectors Then through Filter out secondary information from an image and extract its key information; the specific formula is as follows: where σ is a sigmoid function, W i , U i are model trainable parameter matrices, and denotes element-wise multiplication of matrices. 2.The summary generation system based on multi-modal key information analysis of claim 1, wherein: When the text encoder module works, the Bi-LSTM is used to encode the text sequence, the forward and backward hidden layer vectors are concatenated in series as the text encoding information, and the encoding information at the last time step is taken as the global feature of the text sequence. Specifically, First, the source text is segmented; then, the text data is converted from text form to word embedding vector through Word2Vec; finally, the word embedding vector is encoded through the Bi-LSTM network, and the initial text features h are obtained by concatenating the forward and backward hidden layer vectors i The specific formula is as follows: where E[x i ] denotes the word vector of input word x i , is the concatenation symbol, and r is the hidden layer vector at the last time step of the Bi-LSTM. 3.The summary generation system based on multi-modal key information analysis of claim 1, wherein: The multi-modal feature fusion module first introduces a text attention mechanism and an image attention mechanism to capture the dependence between the decoder hidden layer and the text and image and obtain the context information of each modality, and then uses a multi-modal attention mechanism to fuse the text and visual context information to obtain a multi-modal feature vector, that is, the information of the two modalities is mixed, and the method is as follows: The hierarchical attention mechanism introduced by the multi-modal feature fusion module is composed of a text attention layer, an image attention layer and a multi-modal attention layer, Firstly, the text attention and the image attention are used to give different attention weights to different words in the input text or different local regions in the image in the respective modality channel, and the text context vector and the visual context vector are generated respectively; Then, the multi-modal attention gives different weights to the two kinds of modality data, and the multi-modal feature vector is obtained by weighting the two kinds of modality context vectors according to the weights, and the specific formula is as follows: where s t-1 denotes the decoder's hidden layer state at time step t - 1, denotes the decoder's attention weight for word x i in the input text at the current time step t, W w , U w are model trainable parameter matrices; wherein, denotes the attention weight of the decoder for the i-th region in the image at the current time step t, W a , U a is a matrix of trainable parameters of the model. 4.The summary generation system based on multi-modal key information analysis of claim 1, wherein: The decoder module first initializes the decoder LSTM using the image global feature, and then at each decoding time step, the multi-modal feature vector is used to calculate the probability distribution of the vocabulary, and then the pointer mechanism is combined to select a new word from the vocabulary or copy a word from the source text, so as to obtain the final probability distribution of the predicted word and generate a summary sentence, and the specific method is as follows: The decoder module uses an LSTM as the decoder. First, it initializes the LSTM decoder using the global image features g obtained from the image encoder module, obtaining an initial state s0. Then, a pointer network is introduced on top of this, calculating the probability p of generating a new word at each time step during decoding. gen According to pointer p gen Calculate the probability distribution p value w The formula guides the generation of predicted words from the vocabulary or copying them from the source text, and is as follows: s0 = tanh(W s r + U s g + b s ) Equation Twenty-Five; where c mm represents the multi-modal context vector, s t represents the decoder's hidden layer state at the current time step, E[y t ] represents the word vector of the target word y t , P vocab represents the probability distribution of the predicted word w over the vocabulary, represents the decoder's attention distribution over the text sequence, W s , U s , V, V s , V c are model trainable parameter matrices, and b s is a bias parameter. 5.The summary generation system based on multi-modal key information analysis of claim 1, wherein: During the model training stage of the decoder module, the model network is constantly adjusted according to the forward propagation of information and the backward propagation of errors, and the objective function is gradually optimized.