News customization-oriented multi-modal abstract generation method

Abstract

The invention discloses a news customization-oriented multi-modal abstract generation method, which comprises the following steps of: firstly, inputting pictures and text data into a BART-based multi-modal code, realizing multi-modal interaction in a coding process, meanwhile, adding an image-text matching module to select a sentence corresponding to each picture, and adding a cross attention layer in a decoder; then, constructing an abstract scoring model, and selecting an optimal abstract by calculating the similarity between the candidate abstract and the news text as a score; and calculating a ROUGE score between the sentence corresponding to the picture and the optimal abstract as a text similarity score, obtaining an image-text similarity score in combination with the picture features after multi-modal interaction, obtaining a picture selection probability, selecting the picture with the highest probability as a model selection picture, and generating a multi-modal news abstract. According to the system, a B / S framework is adopted, a lightweight Web framework Flask is used for building, and a Layui open source framework is used for building an interactive interface. According to the method, the accuracy and richness of news multi-modal abstract generation are effectively improved, and the robustness is high.

Description

Technical Field

[0001] The present invention relates to the technical field of customized news generation, and in particular to a multimodal summary generation method for news customization. Background Art

[0002] With the rapid expansion of the Internet, people are facing unprecedented information challenges. Due to the huge amount of information, it has become difficult to filter out truly useful and relevant information. This situation is particularly prominent in Internet news, where a large number of news reports are intertwined, making it difficult for people to quickly understand the core and key points of the event. Therefore, a method that can help people quickly integrate news information can greatly improve the efficiency and experience of users in obtaining news information on the Internet.

[0003] However, there are still some problems in the current way of helping people quickly obtain news information. The first is the search engine. The search engine finds news related to the keywords entered by the user, and provides a brief summary for the search results. This summary mainly relies on the text in the news web page that matches the search keyword, but ignores the overall content and theme of the news web page. Therefore, this summary may deviate from the actual content of the news, which in turn affects the user's reading experience and judgment. In addition, the search engine usually selects the image in the summary according to the established rules, which is usually the first image in the searched web page. This practice may cause the news content to not match the selected image, which will confuse the user. Another way is RSS (Really Simple Syndication) technology, which allows users to obtain website content in RSS readers by subscribing to RSS links provided by websites, and realize real-time updates of website content, but it also has obvious limitations. First, not all websites support RSS, which means that once a website does not support RSS, users cannot obtain information on the website through RSS readers. Second, the RSS subscription links provided by most news websites usually use channels as the finest granularity, which makes it impossible for users to subscribe to news based on their own interests. Finally, RSS technology simply presents all the news web page content in the subscription link in the RSS reader without providing any form of summary, which undoubtedly increases the user's reading burden.

[0004] Therefore, people are in urgent need of a better way to quickly and accurately obtain the information they need. Among them, dimensionality reduction of news reports is an effective way, that is, by simplifying the complex news content into a concise and clear summary, ensuring that the original meaning is retained while removing redundant and irrelevant details, which greatly reduces people's browsing burden and frees people from cumbersome and redundant information.

[0005] The traditional text summary writing process usually relies on manual operation, which means that the writer needs to invest a lot of time and energy to read the entire article, understand its content, and extract key information from it. This process is not only time-consuming and labor-intensive, but also limited by the writer's personal understanding ability and professional knowledge. Different writers may extract different key information from the same article, resulting in discrepancies in the content of the summary. In the era of big data, news information has shown an explosive growth. If every news needs to be manually summarized, it is obviously unrealistic. In addition, there are potential problems such as inaccurate understanding, omissions and errors in manual summary writing, which may affect the integrity and accuracy of the information. Therefore, it is difficult to meet people's needs by relying solely on manual summary writing. With the development of natural language processing technology (NLP), summary generation is constantly being promoted to achieve automation and intelligence. In supervised methods, the text summary task is regarded as a binary classification task, and the correspondence between sentences and their labels is learned through neural networks. Zhou et al. proposed a new scoring method, which records the situation of extracting sentences in the past and uses sentence benefits as a scoring method, taking into account the relationship between sentences. Liu et al. first applied the pre-trained language model to the field of extractive summarization and proposed BERTSUM. Based on the Bert model, this model adds [CLS] token in front of each sentence to obtain the characteristics of each sentence, and obtains the summary through the summary judgment layer. Zhong et al. took into account the relationship between sentences in the article, extracted the summary by combining the candidate sentences, and used the similarity between the combined sentences of the candidate sentences and the original document to judge the document summary. Bahdanau et al. proposed to apply the attention mechanism to the original Seq2Seq to solve the problem of Seq2Seq's poor ability to handle long sequences. The decoder uses the attention mechanism to dynamically extract the encoded information to generate a summary. Compared with manually written summaries, automatic text summarization technology is more efficient and accurate, greatly improving the efficiency and quality of information acquisition.

[0006] The rapid development of Internet technology has also brought about a large increase in multimedia data. Most of the current news web pages are multimedia documents, among which the most common non-text data are pictures interspersed between texts. These pictures, with their vivid presentation and intuitive visual effects, enable people to better understand the content of the news. However, the current automatic text summarization technology generally has the problem of single modality extraction when processing news data. Most of these technologies are limited to the processing of pure text content, while ignoring the rich image information in the news. This single-modality summary generation method not only limits the comprehensiveness of the summary content, but also fails to make full use of the multiple modal information in multimedia data. At the same time, the current automatic text summary generation technology only outputs text summaries, which gives people too single sensory experience.

[0007] Therefore, researchers in the academic community have begun to turn their attention to the task of multimodal summary generation based on multimedia graphic data, aiming to break the limitations of traditional summary technology and achieve more comprehensive and accurate information extraction and presentation. This task refers to making full use of the text and image information in the data, realizing the organic integration of text features and image features, semantic complementarity, and jointly generating a graphic summary, so that users can better understand the content of the original multimedia document and improve the user's reading experience. Existing studies have shown that compared with pure text summaries, multimodal summaries can improve the quality of generated summaries by using information in the visual modality. And compared with pure text output, multimodal output (text and pictures) increases user satisfaction by 12.4%. Most of the existing multimodal summary generation methods focus on how to interact between the modalities of image data and text data, but in fact, there are significant differences between the two different modal data of pictures and text, which makes the interaction between them extremely challenging. Lin et al. proposed a BART-based multimodal sentence summary generation framework (BART-MMSS). By introducing a cue-guided image encoding module, the framework can effectively extract and utilize key information in the image and fuse it with text information to generate more accurate and rich summaries. In order to further improve the quality of summaries, some researchers have introduced auxiliary tasks in the multimodal summary generation task. Zhu et al. proposed a new task, the multimodal summary task with multimodal output (MSMO), which adds an image selection task in addition to the text summary generation task. However, previous studies have ignored the description text of the image. The description text of the image carries the information of the image to a certain extent, connects the image and the text, and provides a bridge of cross-modal semantics. It can be used as enhanced information for summary generation and image selection in multimodal output. However, in fact, not all images are equipped with description text, which limits the use of this information. Therefore, how to solve this problem is also a new challenge for multimodal summary generation. At the same time, the current summary generation model has the problem of inconsistency between the objective function in the training phase and the evaluation index in the evaluation phase. That is, during training, the difference between the summary generated by the model and the reference summary at each position is measured, while the evaluation index is usually a measure of the overall similarity between the summary generated by the model and the reference summary. This will lead to a problem. Even if the model selects the sequence with the highest generation probability as the summary, it may not be the best summary.

[0008] The present application is different from the prior art as follows:

[0009] Technical comparison with patent CN106844341A "News summary extraction method and device based on artificial intelligence"

[0010] Patent CN106844341A provides a news summary extraction method based on artificial intelligence, including: selecting core news from all news of the same news event, selecting core news from all news of the core news event, splitting all news included in the news cluster according to sentences, obtaining the semantic similarity between every two sentences, and then selecting important sentences to form summaries from the core news according to the importance of the sentences, and splicing them according to the original text order, so as to avoid logical confusion and semantic differences; this patent inputs multimodal image and text data into a multimodal encoder, performs multimodal interaction, and uses an image and text matching module to select the sentence corresponding to each picture, and adds a cross-attention layer in the decoder to construct a summary scoring model, and selects the optimal summary by calculating the similarity between the candidate summary and the news text as a score, and calculates the ROUGE score between the sentence corresponding to the picture and the optimal summary as a text similarity score, and obtains the image and text similarity score by combining the picture features after multimodal interaction, and obtains the picture selection probability, and selects the picture with the highest probability as the model selection picture, thereby generating a multimodal news summary. There is an essential difference between the two in the content of the invention.

[0011] Technical comparison with patent CN114880461A "A Chinese news text summarization method combining contrastive learning and pre-training technology"

[0012] Patent CN114880461A provides a Chinese news text summary method combining contrastive learning and pre-training technology, the method includes: constructing contrastive learning input data, using the BERT pre-training model fine-tuned by the Chinese news corpus to obtain the context vector representation of the news text, classifying and scoring the sentences in the text, extracting candidate sentences containing key information to obtain a candidate sentence set, inputting the candidate sentence set into the MT5 model fine-tuned by the Chinese news corpus, generating summary results, and finally realizing the end-to-end training of the extractive model and the generative model by combining the AECLoss loss function; this patent can input graphic data at the same time, and realize the selection of the picture that best matches the summary while generating the summary, realize the generation of multi-modal news summaries, and build a fully functional system, the system adopts B / S architecture, uses the lightweight Web framework Flask to build, and uses the Layui open source framework to realize the construction of the interactive interface, which can accurately and efficiently complete the multi-modal summary generation task. There are essential differences between the two in the implementation method and technical ideas of the invention.

[0013] Therefore, combined with the above research background, the present invention uses multimodal summary generation technology to alleviate the problem of multimedia news information overload in the current Internet. In view of the limitation of the lack of descriptive text of pictures in the current multimodal summary generation technology, it is proposed to use the most relevant sentences in the original text as a substitute for the descriptive text of the picture, and integrate it into the sequence-to-sequence model as summary generation and enhanced information for picture selection in multimodal output. And when generating summaries, multiple summaries are generated, and these summaries are input into a summary scoring model based on contrastive learning training. The summary scoring model is trained to ensure that candidate summaries with higher evaluation indicators obtain higher scores. Finally, the summary with the highest score is selected as the output of the model. Finally, a news customization system is implemented based on the model. The system not only allows users to customize news topics and sources according to personal interests, but also automatically generates previews containing graphic summaries for retrieval results, helping users to quickly grasp the content of the news and greatly improving the efficiency and convenience of obtaining news information. Summary of the invention

[0014] In order to solve the above technical problems, the present invention proposes a news customized multimodal summary generation method, which can effectively improve the accuracy and richness of news multimodal summary generation and has strong robustness.

[0015] To achieve the above object, the technical solution adopted by the present invention is:

[0016] A news-customized multimodal summary generation method, the method comprising the following steps:

[0017] (1) Multimodal feature extraction of text and images:

[0018] Use ViT-B / 32 to extract features from the original input image to obtain an image feature sequence. The original text is divided into sentences and then input into the embedding layer to be converted into word vectors to obtain a text feature sequence. Finally, the image feature sequence and text feature sequence are input into the Transformer encoder together to realize the extraction of multimodal features through the attention mechanism.

[0019] (2) Text-image matching model:

[0020] According to the image feature sequence and text feature sequence obtained in step (1), they are converted into feature vectors through the CLIP model, and the cosine similarity between the image feature and the text feature is calculated. By using contrastive learning, the text and the image have similar representations in the same feature space. The sentence with the largest cosine similarity will be regarded as the sentence most relevant to the image, thereby achieving cross-modal matching of images and texts.

[0021] (3) Multimodal summary generation integrating image-text matching information:

[0022] According to the output of the image-text matching module in step (2) and the output of the Transformer encoder obtained in step (1), they are used as enhanced information and a summary is generated through a multi-layer Transformer decoder using a multi-layer cross-attention mechanism. The first cross-attention layer is used to input the text feature vector corresponding to the image into the decoder in advance and make the decoder pay more attention to the text vector, while the second cross-attention layer is used to obtain all the text vector features after the multi-modal encoder interaction to obtain complete information and generate a candidate summary;

[0023] (4) Summary scoring model based on contrastive learning:

[0024] According to the candidate summaries generated in step (3), they are converted into vectors together with the news text features through the embedding layer and then input into the encoder of the summary scoring model again. After semantic encoding, the [CLS] token in the last hidden state of the encoder output is taken as the vector representation of the candidate summary. The cosine similarity between the candidate summary vector and the news text vector is calculated to evaluate the quality of the candidate summary. The candidate summary with the highest similarity to the news text is selected as the best summary;

[0025] (5) Image selection module integrating image-text matching information:

[0026] According to the output of step (3) and step (4), the image-text similarity score and the text similarity score are obtained, which are concatenated and input into a linear layer, and the Sigmoid function is applied for normalization, and finally the image selection score is obtained;

[0027] (6) System function demonstration.

[0028] As a further improvement of the present invention, in step (1), feature extraction is performed using ViT-B / 32 based on the original image and the original text, specifically comprising the steps of:

[0029] (1-1) Use the pre-trained ViT-B / 32 model to extract the original image. Specifically, use the ViT-B / 32 model pre-trained on the JFT-3B video dataset to extract the features of the original image and original text: ViT-B / 32 extracts features from the N input images to obtain a picture feature sequence For the text with a sequence length of M, perform word segmentation, add [CLS]token and [SEP]token before and after each sentence, and then input it into the embedding layer to convert it into a word vector to obtain the text feature sequence In the v and e t Before inputting into the Transformer encoder, ev and e t Each vector in is added with a learnable modality type flag vector, and then the image features and text features are concatenated and positionally encoded to obtain e = [e v ;e t ];

[0030] (1-2) According to the obtained image feature sequence e v and text feature sequence e t , in the v and e t Before inputting into the Transformer encoder, e v and e t Each vector in is added with a learnable signature vector e of the modality type. type , e corresponding to the picture and text type Different, type The dimension of is the same as that of the image feature and the text feature, so that the model can better learn the interaction between the modalities. Considering that the position encoding has been added to the image feature when passing through the Vision Transformer, only the position encoding is added to the text vector in the multimodal encoder. The calculation formula is as follows:

[0031]

[0032] where e type_img Represents the type code corresponding to the picture, e type_text Represents the type code corresponding to the text, e pos Represents the position encoding and concatenates the image feature sequence and the text feature sequence:

[0033] e=Concat(e v , e t ) (2)

[0034] Where Concat(·) represents the concatenation operation;

[0035] (1-3) After obtaining e, it is input into the Transformer encoder for processing. The Transformer encoder is composed of N stacked Transformer encoding sublayers. Each encoding sublayer contains a multi-head attention part and a feedforward neural network. If the current one is the first encoding sublayer, then the input of the first layer is H 0 =e={e v ;e t}, otherwise, assuming that the current layer is the j-th coding sublayer, its input is the hidden state sequence of the previous layer output in and e v and e t After the hidden state of the j-1th layer output, for each encoding layer, the model first performs multi-head self-attention calculation on the input sequence H of the current layer. The self-attention output is obtained by the interaction between the query matrix Q, the key matrix K and the value matrix V. When calculating the self-attention, Q, K and V are all input H. On each head, Q and K are dot-producted and then divided by the square root of the dimension of the key matrix K. After scaling, the attention weights are obtained through the Softmax operation, and these weights are multiplied by V to obtain the self-attention output of the head. Finally, the self-attention outputs of all heads are spliced ​​together and may undergo a linear transformation to obtain the final output result;

[0036] The multi-head attention mechanism is expressed as the following formula:

[0037]

[0038] Where W O is the weight matrix to be trained, head i is the result obtained by the i-th self-attention. After the multi-head attention calculation is completed, the result obtained by the multi-head attention needs to be residually connected and layer normalized. Let the output after the multi-head attention be H mul , the input of the encoding sublayer is H, and the result after residual connection and layer normalization is H′, then H′ can be calculated by the following formula:

[0039] H′=LayerNorm(H+H mul ) (4)

[0040] Among them, LayerNorm represents layer normalization, and for input H, it can be expressed as the following formula:

[0041]

[0042] Where μ and σ represent the mean and standard deviation of the normalized implicit representation sequence, ∈ is a very small constant used to avoid the situation where the denominator is 0, α is the scaling factor, and β is the bias used to scale and translate the normalized vector. After obtaining the output result H′, H′ is input into the feedforward neural network and a residual connection and layer normalization are performed again to obtain the implicit representation of the output of the current coding sublayer. Let the implicit representation of the j-th layer output be H j , H j The calculation formula of FFN is as follows: 1 , W 2 、b 1 、b 2 These are all learnable parameters.

[0043]

[0044] Where W 1 , W 2 , b 1 , b 2 are all learnable parameters;

[0045] (1-4) Use the attention mechanism for model training and classification. The encoder continuously inputs the hidden state output by the previous encoding sublayer into the next encoding sublayer until the last encoding sublayer. The hidden state h of the last layer of the encoder is taken as h = [h v ;h t ], h is the visual language representation obtained after multimodal interaction, where For the obtained h=[h v ;h t ], take h v In As the feature vector representing each image, we get Take h t In As the feature vector representing each sentence of the original text, we get Where L is the number of sentences in the original text after segmentation. and is the output of the multimodal encoder, is the image feature after interacting with the text content, let The feature of the i-th picture after multimodal interaction is linearly transformed to reduce the dimension to 1, and then mapped to the [0, 1] interval through the Sigmoid activation function, so that the similarity score between the i-th picture and the text is obtained. Finally get The calculation formula for the image-text similarity score is as follows:

[0046]

[0047] When training the model, the binary cross entropy is used as the loss function, denoted by L 1 , the calculation formula is as follows:

[0048]

[0049] where y i represents the label of the real sample, Represents the score of the model output, and N represents the number of images.

[0050] As a further improvement of the present invention, step (2) specifically comprises:

[0051] Using the CLIP model, by contrastive learning, text and images have similar representations in the same feature space, thereby achieving cross-modal matching. The L input sentences are converted into L feature vectors through the CLIP text encoder, and the N input pictures are converted into N feature vectors through the CLIP picture encoder. For each picture, the cosine similarity is calculated between the feature vector of this picture and the feature vector of the L sentences. The sentence with the largest cosine similarity will be regarded as the sentence most relevant to this picture. Then, the sequence number of this sentence in the input sentence sequence is taken. N pictures get N sentence sequence numbers. Finally, the output of the multimodal encoder in step (1) is used to calculate the cosine similarity between the feature vector of this picture and the feature vector of the L sentences. Take out the feature vector of the corresponding sequence number This is the output of the image-text matching module.

[0052] As a further improvement of the present invention, step (3) specifically comprises:

[0053] According to the output of the multimodal summary encoder in step (1) Output of the image-text matching module It is used as enhanced information to generate the final summary. During training, the input of the summary generation encoder is shifted back one position at each position and a special start marker is inserted at the starting position. After obtaining the text vector features after multimodal interaction, And the image-text matching features of the image corresponding to the text After that, the decoder needs to generate a summary based on the above features. The structure of the decoder is similar to that of the encoder, and is also composed of N decoding layers stacked together. If the first layer of the decoder is currently being processed, then the input of this layer is the result of the initial input of the decoder passing through the embedding layer. Conversely, assuming that the jth decoding layer is currently being processed, then its input will come from the previous layer, that is, the implicit state sequence H output by the j-1th decoding layer j-1 ;

[0054] Unlike the traditional Transformer decoding layer, which only has one cross-attention layer, the decoding layer of the multimodal summary generation model proposed in this invention also needs to receive the sentence vector features of the sentence corresponding to the picture, so a cross-attention layer is added to the decoding layer. The main function of the first cross-attention layer is to pre-attract the sentence vector of the sentence corresponding to the picture. Input to the decoder, so that the decoder pays more attention to these sentence vectors, and then compares them with the encoder output The calculation of cross attention focuses more on these sentence vectors. The second cross attention layer is to obtain all sentence vector features after the multimodal encoder interaction. Used to obtain complete information to generate a summary.

[0055] Assume that the current decoding layer is the jth layer, the input of the current layer is H, and the output hidden state is H j , then the calculation process of the decoding layer can be expressed as follows:

[0056]

[0057] Where Q H =K H =V H =H, Represents the sentence vector of each sentence corresponding to each picture output by the image-text matching module, Represents all sentence vector features output by the multimodal encoder. Unlike the encoder, the decoder has a MaskedMultiHeadAttention layer. This is because when the decoder generates sentences, it must follow the order from left to right, that is, only the words at the current position and its previous position can be considered. In order to achieve this order, a mask mechanism is introduced when calculating self-attention. The specific operation is to calculate the dot product of matrix Q and matrix K and add a specific attention mask matrix when calculating formula 3. The lower left corner of the matrix M is all 0, and the upper right corner is all negative infinity. Since the upper right corner of the matrix M is negative infinity, after adding this mask, the corresponding position values ​​of the result matrix will become extremely small. When performing the Softmax operation, the weight results corresponding to these minimum values ​​will be close to 0. Therefore, when calculating self-attention, the model will actually only calculate the attention of the current position and the previous position. At this time, the attention calculation formula is as follows:

[0058]

[0059] Finally, take the hidden state H output by the last layer of the decoder Decoder ={h 1 ,h 2 , ..., h n}, where h i is the hidden state of the ith position, n is the length of the generated summary, and h i Input to the last linear layer, the output dimension of the linear layer is the size of the vocabulary, and then perform a Softmax calculation on the output of the linear layer to obtain the probability distribution P(y i，p );

[0060] When training, cross entropy is used as the loss function, denoted as L 2 , the loss function calculation formula is as follows:

[0061]

[0062] Among them, y i represents the character distribution in the reference abstract, P(y i，p ) is the character probability distribution of the model output. The total loss formula of the multimodal summary generation model is as follows:

[0063] L=L 1 +L 2 (12)

[0064] It consists of two parts: L 1 The loss function selected for the image, L 2 Loss function for summary generation.

[0065] As a further improvement of the present invention, step (4) specifically comprises:

[0066] Using the n candidate summaries generated in step (3), the candidate summaries and news text are converted into vectors through the embedding layer and then input into the encoder in the summary scoring model. After semantic encoding, the [CLS] token in the last hidden state of the encoder output is taken as the sentence vector of the candidate summary and the document vector of the news text. The cosine similarity between the candidate summary vector and the news text vector is used to evaluate the quality of the candidate summary. The candidate summary with the highest similarity to the news text is the best summary. The calculation formula of this process is as follows:

[0067]

[0068] where s i is the sentence vector representation of the i-th candidate summary 1≤i≤n, D is the document vector representation of the news text, cosine(-) represents the cosine similarity between the inputs, argmax(sim i ) represents the subscript when the cosine similarity is the largest, and the candidate summary corresponding to the subscript is the output summary;

[0069] Map the news text, candidate summaries, and reference summaries to the same semantic space. The similarity between the reference summary and the news text should be greater than or equal to the similarity between each candidate summary and the news text. Then the loss function L ref The calculation formula is as follows:

[0070]

[0071] Where n is the number of candidate summaries, Represents the sentence vector representation obtained by the encoder of the summary scoring model after the reference summary is passed;

[0072] The candidate summaries generated by the multimodal summary generation model are used as samples for comparative learning. The ROUGE score is calculated for each generated candidate summary and the reference summary. The average of ROUGE-1, ROUGE-2 and ROUGE-L is taken as the comprehensive evaluation index of the candidate summary. The candidate summaries are sorted from high to low according to the evaluation index. The candidate summaries ranked higher should have a higher similarity with the news text, and vice versa. At the same time, the difference in similarity between the two candidate summaries with a large ranking difference and the news text should also be large. Therefore, the loss function L cand The calculation formula is as follows:

[0073]

[0074] Where λ is the margin, which is a hyperparameter. For a reference summary ranked at a specific position, the similarity between the reference summary at the next position and the news text should be at least λ lower than the similarity between the reference summary at that position and the news text.

[0075] The loss function calculation formula of the final summary scoring model is as follows:

[0076] Loss = L ref +L cand (16).

[0077] As a further improvement of the present invention, step (5) specifically comprises:

[0078] According to the image-text similarity score s obtained in step (1) 1 The serial number of the sentence corresponding to the picture in step (2) is used to match the picture with the sentence text input by the model one by one. The sentence corresponding to each picture is compared with the optimal summary selected by the summary scoring model in step (4) to calculate the ROUGE score.

[0079] Specifically, the average of ROUGE-1, ROUGE-2, and ROUGE-L is used as the text similarity score s 2 This score reflects the textual similarity between the sentence corresponding to the image and the optimal summary;

[0080] The image-text similarity score s 1 and text similarity score s 2 After the concatenation operation, it is input into a linear layer, which first performs a dimensionality increase operation, then a dimensionality reduction operation, and applies the Sigmoid function for normalization. Finally, the image selection score s is obtained. The dimensionality increase operation can extract more potential features in the input data and help the model capture more complex and subtle information in the data.

[0081] As a further improvement of the present invention, step (6) specifically comprises:

[0082] The system function display in step (6) includes news crawling module, news customization module, today's hot search module, browsing history module, personal center module, news website management module and user management module;

[0083] The news crawling module is used to dynamically crawl the latest information on the news website in real time;

[0084] The news customization module is used to load the pre-selected and trained two-stage multimodal summary generation model to provide users with a service of generating a graphic summary preview;

[0085] The today's hot search module is used to record and display the current search popularity in real time. Users can call the news customization module to generate a multimodal summary by clicking on the hot search keyword;

[0086] The browsing history module is used to record the user's news browsing history to ensure that the user behavior data is completely saved;

[0087] The personal center module is used to provide users with personal information management services, where users can modify personalized information such as nicknames, personal descriptions, and view their personal browsing history;

[0088] The news website management module is used by the system administrator to manage the news websites provided by the system;

[0089] The user management module is used by the system administrator to manage user rights and provide a user password reset function.

[0090] Beneficial effects: Compared with the prior art, the present invention adopts the above technical solution and has the following advantages:

[0091] (1) To address the problem of missing description text for news pictures on web pages, the present invention proposes to use the sentences most relevant to the pictures as substitutes for the description text and implement multimodal interaction during the encoding process to ensure that the information of the sentences closely related to the pictures is fully considered during summary generation to enhance the effect of summary generation;

[0092] (2) In view of the inconsistency between the objective function in the training phase of the summary generation model and the evaluation index in the evaluation phase, the present invention constructs a summary scoring model. The model calculates the similarity between the candidate summary and the news text as the score, ensuring that the summary with high evaluation index has a higher score, thereby selecting the optimal summary;

[0093] (3) The present invention also considers integrating image-text matching information into the image selection task. The image-text matching module obtains the sentence number corresponding to each image, and further selects candidate images based on the optimal summary to obtain the image that best matches the news summary. This allows readers to view matching images while obtaining the best summary, greatly improving the reader's reading experience from multiple modes. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 It is the overall framework diagram of the present invention;

[0095] Figure 2 is a schematic diagram of the structure of a multimodal encoder;

[0096] Figure 3 It is a schematic diagram of the structure of the image-text matching module;

[0097] Figure 4 is a schematic diagram of the structure of the summary generation decoder;

[0098] Figure 5 It is a schematic diagram of the structure of the summary scoring model;

[0099] Figure 6 It is the overall framework diagram of the system of the present invention;

[0100] Figure 7 It is the system news customization interface of the present invention;

[0101] Figure 8 It is the system personal center interface of the present invention;

[0102] Fig. 9 It is the system user management interface of the present invention. DETAILED DESCRIPTION

[0103] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0104] The following is only one embodiment of the present invention. The present invention may have other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention. These corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

[0105] like Figure 1 As shown, the news customized multimodal summary generation method of the present invention comprises the following steps:

[0106] 1. Multimodal feature extraction of text and images

[0107] ViT-B / 32 is used to extract features from the input original image to obtain an image feature sequence. The original text is divided into sentences and then input into the embedding layer to be converted into word vectors to obtain a text feature sequence. Finally, the image feature sequence and the text feature sequence are input into the Transformer encoder together to realize the extraction of multimodal features through the attention mechanism. The structural diagram of the multimodal encoder is shown in the figure. Figure 2 shown.

[0108] (1) Use the pre-trained ViT-B / 32 model to extract the original image. Specifically, use the ViT-B / 32 model pre-trained on the JFT-3B video dataset to extract the features of the original image and original text: ViT-B / 32 extracts features from the N input images to obtain the image feature sequence For the text with a sequence length of M, perform word segmentation, add [CLS]token and [SEP]token before and after each sentence, and then input it into the embedding layer to convert it into a word vector to obtain the text feature sequence In the v and e t Before inputting into the Transformer encoder, e v and e t Each vector in is added with a learnable modality type flag vector, and then the image features and text features are concatenated and positionally encoded to obtain e = [e v ;e t ];

[0109] (2) According to the obtained image feature sequence e v and text feature sequence e t , in the v and e t Before inputting into the Transformer encoder, e v and e t Each vector in is added with a learnable signature vector e of the modality type. type , e corresponding to the picture and text type Different, type The dimension of is the same as that of the image feature and the text feature, so that the model can better learn the interaction between the modalities. Considering that the position encoding has been added to the image feature when passing through the Vision Transformer, only the position encoding is added to the text vector in the multimodal encoder. The calculation formula is as follows:

[0110]

[0111] where e type_img Represents the type code corresponding to the picture, etype_text Represents the type code corresponding to the text, e pos Represents the position encoding and concatenates the image feature sequence and the text feature sequence:

[0112] e=Concat(e v , e t ) (2)

[0113] Concat(-) represents the concatenation operation.

[0114] (3) After obtaining e, it is input into the Transformer encoder for processing. The Transformer encoder is composed of N stacked Transformer encoding sublayers. Each encoding sublayer contains a multi-head attention part and a feedforward neural network. If the current layer is the first encoding sublayer, then the input of the first layer is H 0 =e={e v ;e t Otherwise, if the current layer is the j-th coding sublayer, its input is the hidden state sequence of the previous layer output in and e v and e t The hidden state of the j-1th layer output. For each encoding layer, the model first performs multi-head self-attention calculation on the input sequence H of the current layer. The self-attention output is obtained by the interaction between the query matrix Q, the key matrix K and the value matrix V. When calculating the self-attention, Q, K and V are all input H. On each head, Q and K are dot-producted and then divided by the square root of the dimension of the key matrix K. After scaling, the attention weights are obtained through the Softmax operation, and these weights are multiplied by V to obtain the self-attention output of the head. Finally, the self-attention outputs of all heads are concatenated and may undergo a linear transformation to obtain the final output. The multi-head attention mechanism can be expressed as the following formula:

[0115]

[0116] Where W O is the weight matrix to be trained, head i is the result obtained by the i-th self-attention. After the multi-head attention calculation is completed, the result obtained by the multi-head attention needs to be residually connected and layer normalized. Let the output after the multi-head attention be H mul , the input of the encoding sublayer is H, and the result after residual connection and layer normalization is H′, then H′ can be calculated by the following formula:

[0117] H′=LayerNorm(H+H mul ) (4)

[0118] Among them, LayerNorm represents layer normalization, and for input H, it can be expressed as the following formula:

[0119]

[0120] Where μ and σ represent the mean and standard deviation of the normalized implicit representation sequence, ∈ is a very small constant used to avoid the situation where the denominator is 0, α is the scaling factor, and β is the bias used to scale and translate the normalized vector. After obtaining the output result H′, H′ is input into the Feed-Forward Neural Network (FFN) and a residual connection and layer normalization are performed again to obtain the implicit representation of the output of the current coding sublayer. Let the implicit representation of the j-th layer output be H j , H j The calculation formula of FFN is as follows:

[0121]

[0122] Where W 1 , W 2 , b 1 , b 2 These are all learnable parameters.

[0123] (4) Use the attention mechanism for model training and classification. The encoder continuously inputs the hidden state output by the previous encoding sublayer into the next encoding sublayer until the last encoding sublayer. The hidden state of the last layer of the encoder is taken as h = [h v ;h t ], h is the visual language representation obtained after multimodal interaction, where For the obtained h=[h v ;h t ], take h v In As the feature vector representing each image, we get Take h t In As the feature vector representing each sentence of the original text, we get Where L is the number of sentences in the original text after segmentation. and This is the output of the multimodal encoder. is the image feature after interacting with the text content. The feature of the i-th picture after multimodal interaction is linearly transformed to reduce the dimension to 1, and then mapped to the [0, 1] interval through the Sigmoid activation function, so that the similarity score between the i-th picture and the text is obtained. Finally get The calculation formula for the image-text similarity score is as follows:

[0124]

[0125] When training the model, the binary cross entropy is used as the loss function, denoted by L 1 The calculation formula is as follows:

[0126]

[0127] where y i represents the label of the real sample, Represents the score of the model output, and N represents the number of images.

[0128] 2. Text-image matching model

[0129] like Figure 3 As shown, the CLIP (Contrastive Language-Image Pre-training) model is used to make text and images have similar representations in the same feature space through contrastive learning, thereby achieving cross-modal matching. The L input sentences are converted into L feature vectors through CLIP's text encoder, and the N input pictures are converted into N feature vectors through CLIP's picture encoder. For each picture, the cosine similarity is calculated between the feature vector of this picture and the feature vectors of the L sentences respectively. The sentence with the largest cosine similarity will be regarded as the sentence most relevant to this picture. Then, the sequence number of this sentence in the input sentence sequence is taken, and N sentence sequences are obtained for N pictures. Finally, the output of the multimodal encoder in step (1) is obtained. Take out the feature vector of the corresponding sequence number This is the output of the image-text matching module.

[0130] 3. Multimodal summary generation integrating image and text matching information

[0131] According to the output of the multimodal summary encoder in step (1) Output of the image-text matching module They are used as enhanced information and the final summary is generated. The structure diagram of the summary generation decoder is as follows Figure 4The input of the summary generation encoder is shifted back one position during training and a special start token (decoder_start_token) is inserted at the beginning. After obtaining the text vector features after multimodal interaction, And the image-text matching features of the image corresponding to the text After that, the decoder needs to generate a summary based on the above features. The structure of the decoder is similar to that of the encoder, and is also composed of N decoding layers stacked together. If the first layer of the decoder is currently being processed, then the input of this layer is the result of the decoder's initial input passing through the embedding layer. Conversely, assuming that the jth layer of the decoder is currently being processed, then its input will come from the previous layer, that is, the hidden state sequence H output by the j-1th layer of the decoder. j-1 .

[0132] Unlike the traditional Transformer decoding layer, which only has one cross-attention layer, the decoding layer of the multimodal summary generation model proposed in this invention also needs to receive the sentence vector features of the sentence corresponding to the picture, so a cross-attention layer is added to the decoding layer. The main function of the first cross-attention layer is to pre-attract the sentence vector of the sentence corresponding to the picture. Input to the decoder, so that the decoder pays more attention to these sentence vectors, and then compares them with the encoder output The calculation of cross attention focuses more on these sentence vectors. The second cross attention layer is to obtain all sentence vector features after the multimodal encoder interaction. Used to obtain complete information to generate a summary. Assume that the current decoding layer is the jth layer, the input of the current layer is H, and the output hidden state is H j , then the calculation process of the decoding layer can be expressed as follows:

[0133]

[0134] Where Q H =K H =V H =H, Represents the sentence vector of each sentence corresponding to each picture output by the image-text matching module, Represents all sentence vector features output by the multimodal encoder. Unlike the encoder, the decoder has a MaskedMultiHeadAttention layer. This is because when the decoder generates sentences, it must follow the order from left to right, that is, it can only consider the words in the current position and the previous position. In order to achieve this order, a mask mechanism is introduced when calculating self-attention. The specific operation is to calculate the dot product of matrix Q and matrix K and add a specific attention mask matrix when calculating formula 3 The lower left corner of the matrix M is all 0, and the upper right corner is all negative infinity. Since the upper right corner of the matrix M is negative infinity, after adding this mask, the corresponding position value of the result matrix will become extremely small. When performing the Softmax operation, the weight results corresponding to these minimum values ​​will be close to 0. Therefore, when calculating self-attention, the model will actually only calculate attention for the current position and the previous position. At this time, the calculation formula of attention is as follows:

[0135]

[0136] Finally, take the hidden state H output by the last layer of the decoder Decoder ={h 1 ,h 2 , ..., h n}, where h i is the hidden state of the ith position, and n is the length of the generated summary. i Input to the last linear layer, the output dimension of the linear layer is the size of the vocabulary, and then perform a Softmax calculation on the output of the linear layer to obtain the probability distribution P(y i，p ).

[0137] When training, cross entropy is used as the loss function, denoted as L 2 , the loss function calculation formula is as follows:

[0138]

[0139] Among them, y i represents the character distribution in the reference abstract, P(y i，p ) is the character probability distribution of the model output. The total loss formula of the multimodal summary generation model is as follows:

[0140] L=L 1 +L 2 (12)

[0141] It consists of two parts: L 1 Select the loss function for the image obtained in step (1), L 2 Loss function for summary generation.

[0142] 4. Summary scoring model based on contrastive learning

[0143] Using the n candidate summaries generated in step (3), the candidate summaries and news text are converted into vectors through the embedding layer and then input into the encoder in the summary scoring model. After semantic encoding, the [CLS] token in the last hidden state of the encoder output is taken as the sentence vector of the candidate summary and the document vector representation of the news text. The structure of the summary scoring model is as follows: Figure 5 The cosine similarity between the candidate summary vector and the news text vector is used to evaluate the quality of the candidate summary. The candidate summary with the highest similarity to the news text is the best summary. The calculation formula of this process is as follows:

[0144]

[0145] where s i is the sentence vector representation of the i-th candidate summary (1≤i≤n), D is the document vector representation of the news text, cosine(·) represents the cosine similarity between the inputs, argmax(sim i ) represents the subscript when the cosine similarity is the largest, and the candidate summary corresponding to the subscript is the output summary.

[0146] The present invention maps the news text, candidate abstracts and reference abstracts to the same semantic space, and makes the similarity between the reference abstract and the news text greater than or equal to the similarity between each candidate abstract and the news text. The loss function L ref The calculation formula is as follows:

[0147]

[0148] Where n is the number of candidate summaries, Represents the sentence vector representation obtained by passing the reference summary through the encoder of the summary scoring model.

[0149] The candidate summaries generated by the multimodal summary generation model are used as samples for comparative learning. The ROUGE score is calculated for each generated candidate summary and the reference summary. The average of ROUGE-1, ROUGE-2 and ROUGE-L is taken as the comprehensive evaluation index of the candidate summary. The candidate summaries are sorted from high to low according to the evaluation index. The candidate summaries ranked higher should have a higher similarity with the news text, and vice versa. At the same time, the difference in similarity between the two candidate summaries with a large ranking difference and the news text should also be large, so the loss function L cand The calculation formula is as follows:

[0150]

[0151] Where λ is the margin, which is a hyperparameter. For a reference summary ranked at a specific position, the similarity between the reference summary at the next position and the news text should be at least λ lower than the similarity between the reference summary at that position and the news text.

[0152] The loss function calculation formula of the final summary scoring model is as follows:

[0153] Loss = L ref +L cand (16)

[0154] 5. Image selection module integrating image and text matching information

[0155] According to the image-text similarity score s obtained in step (1) 1 The sequence number of the sentence corresponding to the picture in step (2) is used to match the picture with the sentence text input by the model. The sentence corresponding to each picture is used to calculate the ROUGE score with the best summary selected by the summary scoring model in step (4). Specifically, the average of ROUGE-1, ROUGE-2 and ROUGE-L is used as the text similarity score s 2 This score reflects the textual similarity between the sentence corresponding to the image and the optimal summary.

[0156] The image-text similarity score s 1 and text similarity score s 2 After the concatenation operation, it is input into a linear layer. This linear layer first performs a dimensionality increase operation, then a dimensionality reduction operation, and applies a Sigmoid function for normalization, and finally obtains the image selection score s. The dimensionality increase operation can extract more potential features from the input data, helping the model capture more complex and subtle information in the data.

[0157] 6. System function display

[0158] The news customization system designed by the present invention can be divided into five parts: presentation layer, service layer, algorithm layer, data layer and persistence layer. The overall architecture of the system is shown in the figure. Figure 6As shown. The presentation layer interacts directly with system users, provides users with a system entry point, and displays request processing results to users. In this system, the presentation layer is divided into the user side and the administrator side. The service layer is the service provided by the system to users, completing specific business logic processing. This layer mainly provides seven functional modules, including personal center, browsing history, news crawling, news customization, today's hot searches, news website management, and user management. The algorithm layer provides algorithm support for obtaining customized news results and generates relevant graphic summaries for crawled news. The data layer includes user, browsing history and other data. The persistence layer provides data storage. In this system, there are two methods, MySQL and Redis, which are used in different application scenarios.

[0159] The seven functional modules of the news customization-oriented multimodal summary generation system of the present invention include a news crawling module, a news customization module, a today's hot search module, a browsing history module, a personal center module, a news website management module and a user management module.

[0160] The news crawling module is used to crawl the latest information on the news website in real time and dynamically. In view of the fact that the system needs to crawl the latest information on the news website in real time and dynamically, in order to improve the efficiency of the crawler in processing web pages, the present invention implements a distributed crawler, which uses Docker containerization technology to easily realize the packaging, deployment and management of application programs, so that the environmental configuration and version control between different hosts become consistent and convenient. Through Docker, multiple crawler instances can be quickly deployed, and the scale of the crawler cluster can be easily expanded to accommodate crawling tasks of different scales. Distributed crawlers can integrate the computing power of multiple hosts and work together to complete a single crawling task, thereby significantly improving the overall performance and efficiency of the crawler;

[0161] The news customization module is used to load the pre-selected and trained two-stage multimodal summary generation model to provide users with the service of generating graphic summary previews. Users select the news website they want to browse and enter the keywords they are interested in. Then the system retrieves the query interface, relevant parameters and URL crawling links of the website from the MySQL database according to the news website selected by the user, obtains the text and pictures of the news through the news crawling module, inputs them into the multimodal summary generation model, and then generates a summary, and selects the pictures most relevant to the news content as the graphic summary of the news. The effect of this module is shown as follows Figure 7 As shown;

[0162] The today's hot search module is used to record and display the current search popularity in real time. When a user enters the search page, the system will take out the top ten keywords from hotSearch and sort them in descending order of scores. The user can call the news customization module to generate a multimodal summary by clicking on the hot search keyword;

[0163] The browsing history module is used to record the user's news browsing history, ensuring that the user behavior data is completely saved. Whenever a user clicks on a news item pushed by the news customization module, the system backend will respond immediately, automatically capturing and recording the news' URL, title, text summary generated by the model, image link selected by the model, and the current user's unique ID, and adding these data into a record in the browsing history table;

[0164] The personal center module is used to provide users with personal information management services. Users can modify personalized information such as nicknames and personal descriptions, and view their browsing history to meet their personal display needs. At the same time, the module also provides the function of changing passwords to ensure the security of user accounts. In addition, the personal center also integrates the function of displaying browsing history. When a user enters the personal center, the system will query the browsing history of the user from the browsing history table and present the results to the user in real time, so that the user can access his browsing history at any time and review the previous news browsing situation. The effect of this module is shown as follows: Figure 8 As shown;

[0165] The news website management module is used by system administrators to manage the news websites provided by the system. When the crawling rules of a news website become invalid, the system administrator can modify the crawling rules to ensure the normal operation of the news crawling function and maintain the stability of the system. In addition, in order to further enhance the user experience of administrators, the news website management module has specially added a news website testing function. The administrator only needs to click the test button after a specific news website, and the system will try to crawl the URL, title, text and pictures of the news according to the relevant rules of the website, and feedback the results to the front end in real time. This function greatly facilitates administrators to quickly locate and solve problems, and improves the convenience and efficiency of news website management;

[0166] The user management module is used by system administrators to manage user permissions and provide user password reset functions. Its access rights are limited to system administrator roles. System administrators can enhance permissions for ordinary users according to actual needs and perform administrator duties. The effects of this module are shown as follows: Fig. 9 shown.

[0167] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A multimodal summary generation method for news customization, characterized by: The method comprises the following steps: (1) Multimodal feature extraction of text and images: Use ViT-B / 32 to extract features from the original input image to obtain an image feature sequence. The original text is divided into sentences and then input into the embedding layer to be converted into word vectors to obtain a text feature sequence. Finally, the image feature sequence and text feature sequence are input into the Transformer encoder together to realize the extraction of multimodal features through the attention mechanism. (2) Text-image matching model: According to the image feature sequence and text feature sequence obtained in step (1), they are converted into feature vectors through the CLIP model, and the cosine similarity between the image feature and the text feature is calculated. By using contrastive learning, the text and the image have similar representations in the same feature space. The sentence with the largest cosine similarity will be regarded as the sentence most relevant to the image, thereby achieving cross-modal matching of images and texts. (3) Multimodal summary generation integrating image-text matching information: According to the output of the image-text matching module in step (2) and the output of the Transformer encoder obtained in step (1), they are used as enhanced information and a summary is generated through a multi-layer Transformer decoder using a multi-layer cross-attention mechanism. The first cross-attention layer is used to input the text feature vector corresponding to the image into the decoder in advance and make the decoder pay more attention to the text vector, while the second cross-attention layer is used to obtain all the text vector features after the multi-modal encoder interaction to obtain complete information and generate a candidate summary; (4) Summary scoring model based on contrastive learning: According to the candidate summaries generated in step (3), they are converted into vectors together with the news text features through the embedding layer and then input into the encoder of the summary scoring model again. After semantic encoding, the [CLS] token in the last hidden state of the encoder output is taken as the vector representation of the candidate summary. The cosine similarity between the candidate summary vector and the news text vector is calculated to evaluate the quality of the candidate summary. The candidate summary with the highest similarity to the news text is selected as the best summary; (5) Image selection module integrating image-text matching information: According to the output of step (3) and step (4), the image-text similarity score and the text similarity score are obtained, which are concatenated and input into a linear layer, and the Sigmoid function is applied for normalization, and finally the image selection score is obtained; (6) System function demonstration.

2. The news-customized multimodal summary generation method according to claim 1 is characterized by: In step (1), feature extraction is performed using ViT-B / 32 based on the original image and the original text, specifically including the steps of: (1-1) Use the pre-trained ViT-B / 32 model to extract the original image. Specifically, use the ViT-B / 32 model pre-trained on the JFT-3B video dataset to extract the features of the original image and original text: ViT-B / 32 extracts features from the N input images to obtain a picture feature sequence For the text with a sequence length of M, perform word segmentation, add [CLS]token and [SEP]token before and after each sentence, and then input it into the embedding layer to convert it into a word vector to obtain the text feature sequence In the v and e t Before inputting into the Transformer encoder, e v and e t Each vector in is added with a learnable modality type flag vector, and then the image features and text features are concatenated and positionally encoded to obtain e = [e v ;e t ]; (1-2) According to the obtained image feature sequence e v and text feature sequence e t , in the v and e t Input to Transformer Before the encoder, you need to v and e t Each vector in is added with a learnable signature vector e of the modality type. type , e corresponding to the picture and text type Different, type The dimension of is the same as that of the image features and text features, so that the model can better learn the interaction between modalities. Considering that the position encoding has been added to the image features when passing through VisionTransformer, only the position encoding is added to the text vector in the multimodal encoder. The calculation formula is as follows: where e type_img Represents the type code corresponding to the picture, e type_text Represents the type code corresponding to the text, e pos Represents the position encoding and concatenates the image feature sequence and the text feature sequence: e=Concat(e v ,And t )(2) Where Concat(·) represents the concatenation operation; (1-3) After obtaining e, it is input into the Transformer encoder for processing. The Transformer encoder is composed of N stacked Transformer encoding sublayers. Each encoding sublayer contains a multi-head attention part and a feedforward neural network. If the current one is the first encoding sublayer, then the input of the first layer is H 0 =e={e v ;e t }, otherwise, assuming that the current layer is the j-th coding sublayer, its input is the hidden state sequence of the previous layer output in and e v and e t After the hidden state of the j-1th layer output, for each encoding layer, the model first performs multi-head self-attention calculation on the input sequence H of the current layer. The self-attention output is obtained by the interaction between the query matrix Q, the key matrix K and the value matrix V. When calculating the self-attention, Q, K and V are all input H. On each head, Q and K are dot-producted and then divided by the square root of the dimension of the key matrix K. After scaling, the attention weights are obtained through the Softmax operation, and these weights are multiplied by V to obtain the self-attention output of the head. Finally, the self-attention outputs of all heads are spliced ​​together and may undergo a linear transformation to obtain the final output result; The multi-head attention mechanism is expressed as the following formula: Where W O is the weight matrix to be trained, head i is the result obtained by the i-th self-attention. After the multi-head attention calculation is completed, the result obtained by the multi-head attention needs to be residually connected and layer normalized. Let the output after the multi-head attention be H mul , the input of the encoding sublayer is H, and the result after residual connection and layer normalization is H', then H' can be calculated by the following formula: H'=LayerNorm(H+H mul ) (4) Among them, LayerNorm represents layer normalization, and for input H, it can be expressed as the following formula: Where μ and σ represent the mean and standard deviation of the normalized implicit representation sequence, ∈ is a very small constant used to avoid the situation where the denominator is 0, α is the scaling factor, and β is the bias used to scale and translate the normalized vector. After obtaining the output result H', H' is input into the feedforward neural network and a residual connection and layer normalization are performed again to obtain the implicit representation of the output of the current encoding sublayer. Let the implicit representation of the j-th layer output be H j , H j The calculation formula of FFN is as follows: where W1, W2, b1, and b2 are all learnable parameters. Among them, W1, W2, b1, and b2 are all learnable parameters; (1-4) Use the attention mechanism for model training and classification. The encoder continuously inputs the hidden state output by the previous encoding sublayer into the next encoding sublayer until the last encoding sublayer. The hidden state h of the last layer of the encoder is taken as h = [h v ;h t ], h is the visual language representation obtained after multimodal interaction, where For the obtained h=[h v ;h t ], take h v In As the feature vector representing each image, we get Take h t In As the feature vector representing each sentence of the original text, we get Where L is the number of sentences in the original text after segmentation. and is the output of the multimodal encoder, is the image feature after interacting with the text content, let The feature of the i-th picture after multimodal interaction is linearly transformed to reduce the dimension to 1, and then mapped to the [0,1] interval through the Sigmoid activation function, that is, the similarity score between the i-th picture and the text is obtained. Finally get The calculation formula for the image-text similarity score is as follows: When training the model, the binary cross entropy is used as the loss function, denoted as L1, and the calculation formula is as follows: where y i represents the label of the real sample, Represents the score of the model output, and N represents the number of images.

3. The news customized multimodal summary generation method according to claim 1 is characterized by: Step (2) specifically includes: Using the CLIP model, by contrastive learning, text and images have similar representations in the same feature space, thereby achieving cross-modal matching. The L input sentences are converted into L feature vectors through the CLIP text encoder, and the N input pictures are converted into N feature vectors through the CLIP picture encoder. For each picture, the cosine similarity is calculated between the feature vector of this picture and the feature vector of the L sentences. The sentence with the largest cosine similarity will be regarded as the sentence most relevant to this picture. Then, the sequence number of this sentence in the input sentence sequence is taken. N pictures get N sentence sequence numbers. Finally, the output of the multimodal encoder in step (1) is used to calculate the cosine similarity between the feature vector of this picture and the feature vector of the L sentences. Take out the feature vector of the corresponding sequence number This is the output of the image-text matching module.

4. The news customized multimodal summary generation method according to claim 1 is characterized by: Step (3) specifically includes: According to the output of the multimodal summary encoder in step (1) Output of the image-text matching module It is used as enhanced information to generate the final summary. During training, the input of the summary generation encoder is shifted back one position at each position and a special start marker is inserted at the starting position. After obtaining the text vector features after multimodal interaction, And the image-text matching features of the image corresponding to the text After that, the decoder needs to generate a summary based on the above features. The structure of the decoder is similar to that of the encoder, and is also composed of N decoding layers stacked together. If the first layer of the decoder is currently being processed, then the input of this layer is the result of the initial input of the decoder passing through the embedding layer. Conversely, assuming that the jth decoding layer is currently being processed, then its input will come from the previous layer, that is, the implicit state sequence H output by the j-1th decoding layer j-1 ; Assume that the current decoding layer is the jth layer, the input of the current layer is H, and the output hidden state is H j , then the calculation process of the decoding layer can be expressed as follows: Where Q H =K H =V H =H, Represents the sentence vector of each sentence corresponding to each picture output by the image-text matching module, Represents all sentence vector features output by the multimodal encoder. Unlike the encoder, the decoder has a MaskedMultiHeadAttention layer. This is because when the decoder generates sentences, it must follow the order from left to right, that is, only the words at the current position and its previous position can be considered. In order to achieve this order, a mask mechanism is introduced when calculating self-attention. The specific operation is to calculate the dot product of matrix Q and matrix K and add a specific attention mask matrix when calculating formula 3. The lower left corner of the matrix M is all 0, and the upper right corner is all negative infinity. Since the upper right corner of the matrix M is negative infinity, after adding this mask, the corresponding position values ​​of the result matrix will become extremely small. When performing the Softmax operation, the weight results corresponding to these minimum values ​​will be close to 0. Therefore, when calculating self-attention, the model will actually only calculate the attention of the current position and the previous position. At this time, the attention calculation formula is as follows: Finally, take the hidden state H output by the last layer of the decoder Decoder ={h1,h2,…,h n }, where h i is the hidden state of the ith position, n is the length of the generated summary, and h i Input to the last linear layer, the output dimension of the linear layer is the size of the vocabulary, and then perform a Softmax calculation on the output of the linear layer to obtain the probability distribution P(y i,p ); When training, cross entropy is used as the loss function, denoted as L2. The calculation formula of the loss function is as follows: Among them, y i represents the character distribution in the reference abstract, P(y i,p ) is the character probability distribution of the model output. The total loss formula of the multimodal summary generation model is as follows: L=L1+L2 (12) It includes two parts: L1 is the loss function for image selection, and L2 is the loss function for summary generation.

5. The news-customized multimodal summary generation method according to claim 1 is characterized by: Step (4) specifically includes: Using the n candidate summaries generated in step (3), the candidate summaries and news text are converted into vectors through the embedding layer and then input into the encoder in the summary scoring model. After semantic encoding, the [CLS] token in the last hidden state of the encoder output is taken as the sentence vector of the candidate summary and the document vector of the news text. The cosine similarity between the candidate summary vector and the news text vector is used to evaluate the quality of the candidate summary. The candidate summary with the highest similarity to the news text is the best summary. The calculation formula of this process is as follows: where s i is the sentence vector representation of the i-th candidate summary 1≤i≤n, D is the document vector representation of the news text, cosine(·) represents the cosine similarity between the inputs, argmax(sim i ) represents the subscript when the cosine similarity is the largest, and the candidate summary corresponding to the subscript is the output summary; Map the news text, candidate summaries, and reference summaries to the same semantic space. The similarity between the reference summary and the news text should be greater than or equal to the similarity between each candidate summary and the news text. Then the loss function L ref The calculation formula is as follows: Where n is the number of candidate summaries, Represents the sentence vector representation obtained by the encoder of the summary scoring model after the reference summary is passed; The candidate summaries generated by the multimodal summary generation model are used as samples for comparative learning. The ROUGE score is calculated for each generated candidate summary and the reference summary. The average of ROUGE-1, ROUGE-2 and ROUGE-L is taken as the comprehensive evaluation index of the candidate summary. The candidate summaries are sorted from high to low according to the evaluation index. The candidate summaries ranked higher should have a higher similarity with the news text, and vice versa. At the same time, the difference in similarity between the two candidate summaries with a large ranking difference and the news text should also be large. Therefore, the loss function L cand The calculation formula is as follows: Where λ is the margin, which is a hyperparameter. For a reference summary ranked at a specific position, the similarity between the reference summary at the next position and the news text should be at least λ lower than the similarity between the reference summary at that position and the news text. The loss function calculation formula of the final summary scoring model is as follows: Loss=L ref +L cand (16)。 6. The news-customized multimodal summary generation method according to claim 1 is characterized by: Step (5) specifically includes: According to the image-text similarity score s1 obtained in step (1) and the serial number of the sentence corresponding to the image in step (2), the image is matched with the sentence text input to the model through this serial number, and the sentence corresponding to each image is compared with the optimal summary selected by the summary scoring model in step (4) to calculate the ROUGE score; Specifically, the average of ROUGE-1, ROUGE-2, and ROUGE-L is used as the text similarity score s2. This score reflects the text similarity between the sentence corresponding to the image and the optimal summary; After the image-text similarity score s1 and the text similarity score s2 are concatenated, they are input into a linear layer. This linear layer first performs a dimensionality increase operation, then a dimensionality reduction operation, and applies the Sigmoid function for normalization. Finally, the image selection score s is obtained. The dimensionality increase operation can extract more potential features from the input data and help the model capture more complex and subtle information in the data.

7. The news-customized multimodal summary generation method according to claim 1 is characterized by: Step (6) specifically includes: The system function display in step (6) includes news crawling module, news customization module, today's hot search module, browsing history module, personal center module, news website management module and user management module; The news crawling module is used to dynamically crawl the latest information on the news website in real time; The news customization module is used to load the pre-selected and trained two-stage multimodal summary generation model to provide users with a service of generating a graphic summary preview; The today's hot search module is used to record and display the current search popularity in real time. Users can call the news customization module to generate a multimodal summary by clicking on the hot search keyword; The browsing history module is used to record the user's news browsing history to ensure that the user behavior data is completely preserved; The personal center module is used to provide users with personal information management services, where users can modify personalized information such as nicknames, personal descriptions, and view their personal browsing history; The news website management module is used by the system administrator to manage the news websites provided by the system; The user management module is used by the system administrator to manage user rights and provide a user password reset function.

Images