A Vietnamese-Chinese cross-language news text summarization method integrating image information

Abstract

The present invention relates to a Vietnamese-Chinese cross-language news text summarization method that integrates image information, and belongs to the field of natural language processing technology. The present invention first constructs a Vietnamese-Chinese multimodal cross-language summary dataset; secondly, a text encoder and an image encoder are used to represent Vietnamese news text and images, and then the image-text contrast loss is used to enhance the consistency of image and text representation, forcing the Vietnamese representation space to approach the language-independent image representation space. Then, an image-text fuser is used to effectively fuse images and text, enhancing the ability to extract key information from news texts. Finally, a summary decoder is used to generate a Chinese summary. Experiments conducted on a large-scale Vietnamese-Chinese cross-language news summary dataset show that the generated summary has a higher ROUGE score, information content, conciseness, and fluency.

Description

Technical Field

[0001] The present invention relates to a Vietnamese-Chinese cross-language news text summarization method integrating image information, and belongs to the technical field of natural language processing. Background Art

[0002] With the rapid development of information technology, images are playing an increasingly important role in news dissemination and comprehension. However, cross-lingual news summarization is often limited to summarizing and extracting textual information, failing to consider that news often appears in the form of images and text. This overly simplistic approach to feature extraction across data modalities limits the quality and accuracy of summaries. To address this issue, methods for integrating image information into cross-lingual news summarization have become a research focus. In recent years, many scholars have devoted themselves to the study of multimodal cross-lingual summarization methods. These methods aim to improve the performance of cross-lingual news summarization by combining features from both images and text. Traditional cross-lingual news summarization methods primarily generate summaries based on textual information, while image information is often neglected. However, images can provide rich visual content and contextual information, providing important support for news summary generation. Therefore, this paper employs an image encoder to extract image features and fuses them with a text encoder to obtain more comprehensive and accurate semantic information. This approach addresses the problem of leveraging image information to complement the lack of textual information in cross-lingual news summarization. Summary of the Invention

[0003] The present invention provides a Vietnamese-Chinese cross-language news text summarization method that integrates image information, which is used to solve the problem of using image information to make up for the lack of text information in cross-language news summaries, obtain more comprehensive and accurate semantic information, and generate summaries with higher ROUGE scores, information content, conciseness and fluency.

[0004] The technical solution of the present invention is: a Vietnamese-Chinese cross-language news text summarization method integrating image information, the specific steps of the method are as follows:

[0005] Step 1: Crawl Vietnamese news texts and their corresponding news images from multiple news websites on the Internet, process them, and construct a Vietnamese-Chinese multimodal cross-lingual summary dataset;

[0006] Step 2: The text encoder is responsible for converting the input Vietnamese news text into semantic representations, capturing key information and semantic associations;

[0007] Step 3: The image encoder converts the input news image into an image representation and extracts the visual information from the image;

[0008] Step 4: Image-text contrast loss effectively associates and integrates the information between images and text by maximizing the similarity of positive samples and minimizing the similarity of negative samples.

[0009] Step 5: The image-text fusion unit fuses the text representation and the image representation to obtain a fused image-text representation. By fully capturing the correlation between the text and the image, the key information in the text is extracted.

[0010] Step 6: The summary decoder uses the fused image and text features as input to generate a Chinese summary.

[0011] Furthermore, the specific steps of Step 1 are:

[0012] Step 1.1: We crawled Vietnamese news texts and their corresponding news images from multiple news websites to initially construct a multimodal monolingual summary dataset. We then used Google Translate to translate the Vietnamese summaries into Chinese summaries.

[0013] Step 1.2: Clean the constructed data and use a round-trip translation strategy to filter out low-quality translated summary texts by setting a threshold for the ROUGE score, and only retain the first three news pictures.

[0014] Furthermore, the specific steps of Step 2 are:

[0015] Step 2.1, the encoder input sequence is S A The input is encoded to obtain the vector representation of the news sequence W = {w1,w2,w3,...,w E}, where S represents Vietnamese news articles, E is the length of the input text sequence, Represents the Eth word in Vietnamese news article A. The number of encoder layers is set to L, and the final text output is d t is the feature dimension;

[0016] Step 2.2: On the text encoder side, a self-attention mechanism is used to capture the global dependencies in the input sequence, assigning a weight to each input element. This weight is determined by its relevance to all other elements in the sequence.

[0017]

[0018] Where Q, K, and V are query, key, and value, respectively. k represents the dimension of vector K;

[0019] Step 2.3, then connect each head together to form a multi-head attention mechanism;

[0020] MultiHead(Q,K,V)=Concat(head1,...,head n )

[0021] Among them, head i =Attention(QW i Q ,KW i K ,VW i V ), W i Q , W i K , W i V is the weight parameter matrix and h is the number of attention heads.

[0022] Furthermore, the specific steps of Step 3 are:

[0023] Step 3.1, given the image encoder input sequence is I = {img1, img2, ..., img M}, where M is the sequence of input news images, which are converted into image embeddings using ViT original images and used together with text embeddings as the input of the image encoder;

[0024]

[0025] Among them, ViT divides each image into multiple patches and reshapes the 2D image For flat Where (H, W) is the resolution of the original image, C is the number of channels, (P, P) is the resolution of each image block, and N = HW / P 2 The final number of patches is obtained by taking this series of patches as the input of the image tokenizer. After completing the image patch processing, the image block embedding operation is required to linearly project each flattened patch into D dimensions, E pos is the position code, E is the length of the input text sequence;

[0026] Step 3.2. Use the generated Z0 sequence as the input of the Transformer encoder:

[0027] Z l+1 =Transformer(Z l ), l=1, 2, ..., L

[0028] Where L is the number of encoder layers in the Transformer model; then perform a global maximum pooling operation v on the output vectori =MaxPooling(Z L ),in The final output image sequence length is M visual feature vector sequence

[0029] Furthermore, the specific steps of Step 4 are:

[0030] Step 4.1. Similarity function s = g v (v) T g w (w), where v = 0 v ,w=Linear(O t ), g v (v) and g w (w) is a linear transformation that maps the high-dimensional representation of the image and text to a standardized low-dimensional representation. The matching image-text pairs will have a higher similarity score. v (v) and g w (w) Design two queues to store the different image-text representations of two independent unimodal encoders, and record their features as g′ v (v′) and g′ w (w′), for each image-text pair, calculate the softmax normalized image-text and text-image similarity:

[0031]

[0032] Where τ is the control parameter, which adjusts the probability distribution;

[0033] Step 4.2, set and are the one-hot similarities between image to text and text to image ground-truth, respectively. The probability of a negative sample pair is 0, while the probability of a positive sample pair is 1.

[0034] The loss function between the image-text comparison loss is;

[0035]

[0036] Here, H(,) is the cross entropy function. Positive samples are image-text pairs in the same news article, while negative samples are image-text pairs in different news articles in the same batch. By calculating the similarity score of positive and negative samples, the model can learn more discriminative image-text representations and thus better distinguish similar and dissimilar image-text pairs.

[0037] Furthermore, the specific steps of Step 5 are:

[0038] Step 5.1: Before image-text fusion, use the multi-head attention mechanism and feedforward neural network to further encode the text and obtain the output Then combine the output of the image encoder Perform image-text fusion with S visual feature vector sequences, where the number of layers of the image-text fuser is 6-L; L is the number of layers of the encoder;

[0039] Step 5.2: Use multi-head attention fusion to introduce multiple attention heads and use multiple parallel attention heads to simultaneously learn and capture different feature information;

[0040]

[0041] The query vector Q is obtained from the input text features, the key vector K and the value vector V are obtained from the input visual features, and the vectors Q, K, and V are linearly projected accordingly, denoted as P q 、P k and P v ;Q=O c P q , K=O v P k , V=O v W v , a cross-modal multi-head attention is used to filter and reorganize the text, and finally the visual feature D1 is obtained;

[0042] Finally, the reference dot product attention fusion concatenates the input text features O c Combined with the visual feature D1, we get the visually guided text feature O′ c ,Then, this feature is projected to the required dimension through a linear transformation P;

[0043] O′ c =Concat( c , D1)P.

[0044] Furthermore, the specific steps of Step 6 are:

[0045] Step 6.1, the decoder input sequence is Where T represents the Chinese abstract, and D2 is the output sequence length;

[0046] Step 6.2: Each decoder layer consists of three connected sublayers. Unlike the encoder layer, the decoder layer incorporates a masked multi-head attention sublayer, enabling the decoder to sequentially generate the target language sequence while fully utilizing the information from the source language sequence. The encoder and decoder are jointly trained to maximize the target sequence loss function given the source sequence, yielding the decoding loss.

[0047]

[0048] Step 6.3. Finally, add the image-text comparison loss and the decoding loss to get the final loss function:

[0049] L total =L CL +L cls

[0050] Among them, L CL is the image-text comparison loss, L cls is the decoding loss.

[0051] The beneficial effects of the present invention are as follows: the present invention proposes a Vietnamese-Chinese cross-language news text summarization method that integrates image information. In cross-language scenarios, due to language differences and data scarcity, traditional text summarization methods have certain limitations. To overcome these problems, the present invention uses an image encoder to extract image features and fuses them with a text encoder to obtain more comprehensive and accurate semantic information. In order to solve the problem of using image information to make up for the lack of text information in cross-language news summarization, the present invention conducted experiments on a large-scale Vietnamese-Chinese cross-language news summary dataset and showed that the generated summary has a higher ROUGE score, information content, conciseness, and fluency. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of data processing in the present invention;

[0053] Figure 2 It is a flow chart of the model in the present invention;

[0054] Figure 3 It is a model structure diagram of the present invention. DETAILED DESCRIPTION

[0055] Example 1: Figure 1-3 As shown in FIG, a Vietnamese-Chinese cross-language news text summarization method integrating image information is described, and the specific steps of the method are as follows:

[0056] Step 1: Crawl Vietnamese news texts and their corresponding news images from multiple news websites on the Internet, process them, and construct a Vietnamese-Chinese multimodal cross-lingual summary dataset;

[0057] Step 2: The text encoder is responsible for converting the input Vietnamese news text into semantic representations, capturing key information and semantic associations;

[0058] Step 3: The image encoder converts the input news image into an image representation and extracts the visual information from the image;

[0059] Step 4: Image-text contrast loss effectively associates and integrates the information between images and text by maximizing the similarity of positive samples and minimizing the similarity of negative samples.

[0060] Step 5: The image-text fusion unit fuses the text representation and the image representation to obtain a fused image-text representation. By fully capturing the correlation between the text and the image, the key information in the text is extracted.

[0061] Step 6: The summary decoder uses the fused image and text features as input to generate a Chinese summary.

[0062] Furthermore, the specific steps of Step 1 are:

[0063] Step 1.1: We crawled Vietnamese news texts and their corresponding news images from multiple news websites to initially construct a multimodal monolingual summary dataset. We then used Google Translate to translate the Vietnamese summaries into Chinese summaries.

[0064] Step 1.2: Clean the constructed data. Use a round-trip translation strategy to filter out low-quality translated summary texts by setting a threshold for the ROUGE score, and only retain the first three news pictures. The constructed Vietnamese-Chinese multimodal cross-lingual summary dataset is shown in Table 1:

[0065] Table 1. Statistics of the Vietnamese-Chinese multimodal cross-language abstract corpus

[0066]

[0067] Furthermore, the specific steps of Step 2 are:

[0068] Step 2.1, the encoder input sequence is S A The input is encoded to obtain the vector representation of the news sequence W = {w1, w2, w3, ..., w E}, where S represents Vietnamese news articles, E is the length of the input text sequence, Represents the Eth word in Vietnamese news article A. The number of encoder layers is set to L, and the final text output is d t is the feature dimension;

[0069] Step 2.2: On the text encoder side, a self-attention mechanism is used to capture the global dependencies in the input sequence, assigning each input element a weight determined by its relevance to all other elements in the sequence.

[0070]

[0071] Where Q, K, and V are query, key, and value, respectively. k represents the dimension of vector K;

[0072] Step 2.3, then connect each head together to form a multi-head attention mechanism;

[0073] MultiHead(Q,K,V)=Concat(head1,..,head n )

[0074] Among them, head i =Attention(QW i Q , KW i K , VW i V), W i Q , W i K , W i V is the weight parameter matrix and h is the number of attention heads.

[0075] Furthermore, the specific steps of Step 3 are:

[0076] Step 3.1, given the image encoder input sequence is I = {img1, img2, .., img M}, where M is the sequence of input news images, which are converted into image embeddings using ViT original images and used together with text embeddings as the input of the image encoder;

[0077]

[0078] Among them, ViT divides each image into multiple patches and reshapes the 2D image For flat Where (H, W) is the resolution of the original image, C is the number of channels, (P, P) is the resolution of each image block, and N = HW / P 2 The final number of patches is obtained by taking this series of patches as the input of the image tokenizer. After completing the image patch processing, the image block embedding operation is required to linearly project each flattened patch into D dimensions, E pos is the position code, E is the length of the input text sequence;

[0079] Step 3.2. Use the generated Z0 sequence as the input of the Transformer encoder:

[0080] Z l+1 =Transformer(Z l ), l = 1, 2, .., L

[0081] Where L is the number of encoder layers in the Transformer model; then perform a global maximum pooling operation v on the output vector i =MaxPooling(Z L ),in The final output image sequence length is M visual feature vector sequence

[0082] Furthermore, the specific steps of Step 4 are:

[0083] Step 4.1. Similarity function s = g v (v) T g w (w), where v = 0 v ,w=Linear(O t ), g v (v) and g w (w) is a linear transformation that maps the high-dimensional representation of the image and text to a standardized low-dimensional representation. The matching image-text pairs will have a higher similarity score. v (v) and g w (w) Design two queues to store the different image-text representations of two independent unimodal encoders, and record their features as g′ v (v′) and g′ w (w′), for each image-text pair, calculate the softmax normalized image-text and text-image similarity:

[0084]

[0085] Where τ is the control parameter, which adjusts the probability distribution;

[0086] Step 4.2, set and are the one-hot similarities between image to text and text to image ground-truth, respectively. The probability of a negative sample pair is 0, while the probability of a positive sample pair is 1.

[0087] The loss function between the image-text comparison loss is;

[0088]

[0089] Here, H(,) is the cross entropy function. Positive samples are image-text pairs in the same news article, while negative samples are image-text pairs in different news articles in the same batch. By calculating the similarity score of positive and negative samples, the model can learn more discriminative image-text representations and thus better distinguish similar and dissimilar image-text pairs.

[0090] Furthermore, the specific steps of Step 5 are:

[0091] Step 5.1: Before image-text fusion, use the multi-head attention mechanism and feedforward neural network to further encode the text and obtain the output Then combine the output of the image encoder Perform image-text fusion with S visual feature vector sequences, where the number of layers of the image-text fuser is 6-L; L is the number of layers of the encoder;

[0092] Step 5.2: Use multi-head attention fusion to introduce multiple attention heads and use multiple parallel attention heads to simultaneously learn and capture different feature information;

[0093]

[0094] The query vector Q is obtained from the input text features, the key vector K and the value vector V are obtained from the input visual features, and the vectors Q, K, and V are linearly projected accordingly, denoted as P q 、P k and P v ;Q=O c P q , K=O v P k , V=O v W v , a cross-modal multi-head attention is used to filter and reorganize the text, and finally the visual feature D1 is obtained;

[0095] Finally, the reference dot product attention fusion concatenates the input text features O c Combined with the visual feature D1, we get the visually guided text feature O′ c ,Then, this feature is projected to the required dimension through a linear transformation P;

[0096] O′ c =Concat( c ,D1)P.

[0097] Furthermore, the specific steps of Step 6 are:

[0098] Step 6.1, the decoder input sequence is Where T represents the Chinese abstract, and D2 is the output sequence length;

[0099] Step 6.2: Each decoder layer consists of three connected sublayers. Unlike the encoder layer, the decoder layer incorporates a masked multi-head attention sublayer, enabling the decoder to sequentially generate the target language sequence while fully utilizing the information from the source language sequence. The encoder and decoder are jointly trained to maximize the target sequence loss function given the source sequence, yielding the decoding loss.

[0100]

[0101] Step 6.3. Finally, add the image-text comparison loss and the decoding loss to get the final loss function:

[0102] L total =L CL +L cls Among them, L CL is the image-text comparison loss, L cls is the decoding loss.

[0103] In order to illustrate the effectiveness of the present invention, two sets of experiments are designed for verification. The first set of experiments verifies the baseline model and the present invention for comparison. The second set of experiments verifies the ablation experiment of the present invention.

[0104] The first set of experiments, Table 2 shows the comparative experimental results of the proposed method and the baseline model. In order to verify the effectiveness of the Vietnamese-Chinese cross-language summarization method that integrates image information, this paper trained the following baseline models and made corresponding comparisons;

[0105] Sum-Trans (Summarize First, Translate Later): First, a Transformer model is trained on a long Vietnamese news article to produce a Vietnamese summary. Then, the Vietnamese summary is used as an intermediate text and translated into a Chinese target summary using the Google Translate model.

[0106] Trans-Sum (Translate First, Summarize Later): This method uses machine translation training based on the Transformer model to generate a long Chinese news article as the intermediate text. It then uses HanLP to perform monolingual summarization of the long Chinese news article, employing an unsupervised method based on TextRank. HanLP effectively extracts key information from the text and is optimized for Chinese, resulting in superior summarization results.

[0107] NCLS: It uses an end-to-end neural network Transformer model to generate cross-lingual summaries, uses the attention mechanism of the encoder and decoder to achieve alignment between two different languages, and applies Vietnamese-Chinese cross-lingual summaries to this framework for training.

[0108] NCLS+MS: Based on NCLS, a shared encoder and independent decoder are introduced, and the monolingual summarization task and the cross-lingual summarization task are jointly trained to enhance the encoder. In the multi-task framework, the monolingual summarization task is introduced as an additional constraint on the decoder side to strengthen the training and learning process of the model.

[0109] MCLAS: A unified decoder is used to sequentially generate monolingual and cross-lingual summaries, making the monolingual summarization task a precursor to the cross-lingual summarization task. This method can learn the interactive information between the summaries in the two languages, including word alignment relationships and summary patterns in different languages.

[0110] DP-CLS: This method uses dot product attention as the image-text fusion layer.

[0111] MH-CLS: The image-text fusion layer is the multi-head attention method in this paper.

[0112] Table 2 Cross-language summary model comparison experimental results

[0113]

[0114] Both the Trans-Sum model and the Sum-Trans model performed poorly in the ROUGE score. This low score may be due to the accumulated errors between the two independent subtasks. In these models, the error in the previous task directly affects the performance of the subsequent task, significantly reducing the quality of the summary. Secondly, compared with the mainstream multi-task learning frameworks NCLS, NCLS+MS, and MCLAS, DP-CLS and MH-CLS both achieved significant improvements in the ROUGE score, indicating that the integration of image information can effectively improve cross-language summarization performance. At the same time, the MH-CLS method performed slightly better than the DP-CLS method, which shows that the multi-head attention mechanism is more effective than the dot-product attention mechanism in fusing image information.

[0115] Table 3 shows the comparative experimental results of the present invention and the baseline model in the second set of experiments. It can be seen that the present invention achieves the best performance. Experimental verification was performed using a single module. "Without image" refers to removing the image encoding module and the image-text fusion module, retaining only the text encoding portion. "Without CL" refers to removing the image-text contrast loss portion and performing a simple fusion of the image and text components.

[0116] Table 3 Ablation experiment results

[0117]

[0118] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A Vietnamese-Chinese cross-language news text summarization method integrating image information, characterized by: The specific steps of the method are as follows: Step 1: Crawl Vietnamese news texts and their corresponding news images from multiple news websites on the Internet, process them, and construct a Vietnamese-Chinese multimodal cross-lingual summary dataset; Step 2: The text encoder is responsible for converting the input Vietnamese news text into semantic representations, capturing key information and semantic associations; Step 3: The image encoder converts the input news image into an image representation and extracts the visual information from the image; Step 4: Image-text contrast loss effectively associates and integrates the information between images and text by maximizing the similarity of positive samples and minimizing the similarity of negative samples. Step 5: The image-text fusion unit fuses the text representation and the image representation to obtain a fused image-text representation. By fully capturing the correlation between the text and the image, the key information in the text is extracted. Step 6: The summary decoder uses the fused image and text features as input to generate a Chinese summary; The specific steps of Step 5 are: Step 5.1: Before image-text fusion, use the multi-head attention mechanism and feedforward neural network to further encode the text and obtain the output Then combine the output of the image encoder Perform image-text fusion with S visual feature vector sequences, where the number of layers of the image-text fuser is 6-L; L is the number of layers of the encoder; Step 5.2: Use multi-head attention fusion to introduce multiple attention heads and use multiple parallel attention heads to simultaneously learn and capture different feature information; The query vector Q is obtained from the input text features, the key vector K and the value vector V are obtained from the input visual features, and the vectors Q, K, and V are linearly projected accordingly, denoted as P q 、P k and P v ;Q=O c P q , K=O v P k , V=O v W v , a cross-modal multi-head attention is used to filter and reorganize the text, and finally the visual feature D1 is obtained; Finally, the reference dot product attention fusion concatenates the input text features O c Combined with the visual feature D1, we get the visually guided text feature O c ′, then, this feature is projected to the required dimension through a linear transformation P; A' c =Concat(O c ,D1)P.

2. The Vietnamese-Chinese cross-language news text summarization method integrating image information according to claim 1 is characterized by: The specific steps of Step 1 are: Step 1.1: We crawled Vietnamese news texts and their corresponding news images from multiple news websites to initially construct a multimodal monolingual summary dataset, and then translated the Vietnamese summaries into Chinese summaries. Step 1.2: Clean the constructed data and use a round-trip translation strategy to filter out low-quality translated summary texts by setting a threshold for the ROUGE score, and only retain the first three news pictures.

3. The Vietnamese-Chinese cross-language news text summarization method integrating image information according to claim 1 is characterized by: The specific steps of Step 2 are as follows: Step 2.1, the encoder input sequence is S a The input is encoded to obtain the vector representation of the news sequence W = {w1,w2,w3,…,w E }, where S represents Vietnamese news articles, E is the length of the input text sequence, Represents the Eth word in Vietnamese news article A. The number of encoder layers is set to L, and the final text output is d t is the feature dimension; Step 2.2: On the text encoder side, a self-attention mechanism is used to capture the global dependencies in the input sequence, assigning each input element a weight determined by its relevance to all other elements in the sequence. Where Q, K, and V are query, key, and value, respectively. k represents the dimension of vector K; Step 2.3, then connect each head together to form a multi-head attention mechanism; MultiHead(Q,K,V)=Concat(head1,…,head n ) in, is the weight parameter matrix and h is the number of attention heads.

4. The Vietnamese-Chinese cross-language news text summarization method integrating image information according to claim 1 is characterized by: The specific steps of Step 3 are: Step 3.1, given the image encoder input sequence is I = {img1, img2, ..., img M }, where M is the sequence of input news images, which are converted into image embeddings using ViT original images and used together with text embeddings as the input of the image encoder; Among them, ViT divides each image into multiple patches and reshapes the 2D image For flat Where (H, W) is the resolution of the original image, C is the number of channels, (P, P) is the resolution of each image block, and N = HW / P 2 The final number of patches is obtained by taking this series of patches as the input of the image tokenizer. After completing the image patch processing, the image block embedding operation is required to linearly project each flattened patch into D dimensions, E pos is the position code, E is the length of the input text sequence; Step 3.

2. Use the generated Z0 sequence as the input of the Transformer encoder: WITH l+1 =Transformer(Z l ),l=1,2,…,L Where L is the number of encoder layers in the Transformer model; then perform a global maximum pooling operation v on the output vector i =MaxPooling(Z L ),in The final output image sequence length is M visual feature vector sequence 5. The Vietnamese-Chinese cross-language news text summarization method integrating image information according to claim 1 is characterized by: The specific steps of Step 4 are: Step 4.

1. Similarity function s = g v (v) T g w (w), where v = 0 v ,w=Linear(O t ), g v (v) and g w (w) is a linear transformation that maps the high-dimensional representation of the image and text to a standardized low-dimensional representation. The matching image-text pairs will have a higher similarity score. v (v) and g w (w) Design two queues to store the different image-text representations of two independent unimodal encoders, and record their features as g′ v (v′) and g′ w (w′), for each image-text pair, calculate the softmax normalized image-text and text-image similarity: Where τ is the control parameter, which adjusts the probability distribution; Step 4.2, set and are the one-hot similarities between image to text and text to image ground-truth, respectively. The probability of a negative sample pair is 0, while the probability of a positive sample pair is 1. The loss function between the image-text comparison loss is; Here, H(,) is the cross entropy function. Positive samples are image-text pairs in the same news article, while negative samples are image-text pairs in different news articles in the same batch. By calculating the similarity score of positive and negative samples, the model can learn more discriminative image-text representations and thus better distinguish similar and dissimilar image-text pairs.

6. The Vietnamese-Chinese cross-language news text summarization method integrating image information according to claim 1 is characterized by: The specific steps of Step 6 are: Step 6.1, the decoder input sequence is Where T represents the Chinese abstract, and D2 is the output sequence length; Step 6.

2. Each decoder layer consists of three sub-layers connected together. The difference from the encoder layer is that a masked multi-head attention sub-layer is introduced, which enables the decoder to generate the target language sequence step by step in sequence and make full use of the information of the source language sequence during the generation process. Through the joint training of the encoder and decoder, the loss function of the target sequence is maximized given the source sequence, and the decoding loss L is obtained. cls ; Step 6.

3. Finally, add the image-text comparison loss and the decoding loss to get the final loss function: L total =L CL +L cls Among them, L cL is the image-text comparison loss, L cls is the decoding loss.

Images