A multi-language common attention topic detection method based on mT5 and word frequency information entropy

Abstract

The application discloses a multi-language common attention topic detection method based on mT5 and word frequency information entropy, and belongs to the technical field of social media information mining.The method adopts a statistical method of multi-language word frequency information entropy, fully considers the feature that a common attention topic is evenly distributed in multiple languages, and realizes extraction of theme words commonly attended by different cultures; synonymous words are aggregated by using mBERT, redundancy of output words is reduced by realizing identification of redundant words in the process of constructing a word table; by improving a theme word extraction algorithm in BERTopic, a single document is subjected to theme word extraction by using a fine-tuned mT5, and the method is improved from a traditional single-language topic detection to a multi-language mode, so that the problem of missing potential value theme words is avoided to a certain extent.The application is suitable for the technical field of social media information mining, improves the detection and recognition capability of common attention topics, and improves the output quality of theme words.

Description

Technical Field

[0001] This invention relates to a statistical method based on a large language model and word frequency information entropy, specifically a method for detecting multilingual common interest topics based on mT5 (multilingual-t5) and word frequency information entropy, belonging to the field of social media information mining technology. Background Technology

[0002] Different languages ​​and cultures exhibit unique biases in their focus on daily trending news. Unique topic preferences exist globally at specific times. Identifying shared and individual topics of interest can reduce the impact of cognitive biases from different cultures, thereby revealing cultural differences and possessing significant potential value.

[0003] However, existing topic models fail to effectively highlight topics of common interest. For example, LDA (Latent Dirichlet Allocation), as a probabilistic generative model, can detect topic structures from text using Dirichlet distributions. HDP (Hierarchical Dirichlet Process) introduces an infinite probability process to automatically infer topic structures. The BERTopic model uses SBERT (Sentence BERT) to create sentence embeddings and leverages the c-tf-idf algorithm to extract topic words from clustered text. These models cannot effectively handle multilingual text (BERTopic supports multiple languages, but multilingual input increases output redundancy) and do not adequately emphasize common topics of interest across multiple languages. Furthermore, many keywords with similar meanings often appear simultaneously in the output because their frequencies or distributions in the corpus are comparable, leading to significant redundancy.

[0004] Topic models typically employ methods to rank output keywords by quality. However, popular keywords often occupy the majority of the space at the beginning of the sequence, resulting in potentially valuable keywords being relegated to the back of the output sequence in a more chaotic manner. These factors not only perpetuate the misleading influence of cultural attention bias on topic model output but also lead to low-quality output.

[0005] Based on the aforementioned shortcomings of existing technologies, this paper proposes an improved method based on word frequency information entropy and mT5, which avoids synonym redundancy to a certain extent and can extract topic words of common interest to different languages ​​and cultures while supporting multilingual topic detection. Summary of the Invention

[0006] To address the problem of poor topic quality in existing topic detection technologies due to neglecting multilingual synonym redundancy and cultural attention biases, this invention aims to disclose a multilingual common-interest topic detection method based on mT5 and word frequency information entropy. This method addresses the word frequency distribution characteristics caused by attention biases across different languages ​​by statistically analyzing the frequency of words with specific meanings in different languages ​​and calculating information entropy, thus automatically sorting common-interest words in the output. It constructs a low-redundancy common-interest topic extraction model by reusing the BERTopic text clustering architecture, fine-tuning mT5 single-document topic extraction, and using mBERT (multilingual BERT) to cluster synonyms. The OCTIS framework is used to evaluate classic topic models and assess the limitations of multilingual corpora. This improves the detection capability of common-interest topics and enhances the output quality of topic words.

[0007] This invention discloses a multilingual common interest topic detection method based on mT5 and word frequency information entropy, comprising the following steps:

[0008] A multilingual common interest topic detection method based on mT5 and word frequency information entropy includes the following steps:

[0009] Step 1: Construct a "document-topic term" dataset based on the BERTopic model;

[0010] Specifically, it includes the following steps:

[0011] Step 1.1: Construct a topic word extraction model based on BERTopic;

[0012] Using BERTopic as the baseline model, we optimized the topic word quality using c-tf-idf and maximum marginal relevance algorithms. We replaced the vector clustering algorithm of the BERTopic model with the self-supervised HDBSCAN algorithm or the supervised KMeans algorithm, and set the relevant hyperparameters. We replaced the text embedding model with a task-related language model, and replaced multilingual input with a multilingual model to obtain the topic word extraction model.

[0013] Fine-tuning mT5 using the dataset obtained from the topic word extraction model can unleash mT5's potential for more efficient topic word extraction.

[0014] Step 1.2: Data collection, constructing a "document-topic term" dataset;

[0015] The “Document-Topic” dataset contains a collection of documents in five languages: English, Arabic, Chinese, Korean, and Japanese, denoted as {Text}. en}, {Text ar}, {Text zh}, {Text ko}, {Text ja}

[0016] For the keyword extraction model obtained in step 1.1, the input document set {Text i}, set a specific input batch size to obtain single documents in five languages, as shown in formula (1):

[0017] SingleText i =HDBSCAN(UMAP(SBERT({Text i})))i∈{en,ar,zh,ja,ko} (1)

[0018] Embedding document collection {Text} using SBERT model i The embedding vector is obtained; dimensionality is reduced using the UMAP algorithm; and then clustered using the HDBSCAN algorithm to obtain the aggregated document set SingleText.

[0019] For the documents in the aggregated document set SingleText, the c-tf-idf algorithm is used to obtain the topic words, and the results are optimized by the maximum marginal relevance algorithm to obtain the "document-topic word" dataset, as shown in formula (2):

[0020] TopicWords i =MMR(c-tf-idf(SingleText) i ))i∈{en,ar,zh,ja,ko} (2)

[0021] Among them, TopicWords represents the topic words corresponding to the documents in five languages ​​obtained through BERTopic. Since the c-tf-idf algorithm requires counting word frequencies and inverse document frequencies for multiple documents, it cannot input a single document to obtain the dataset.

[0022] Step 2: Fine-tune mT5 using FineTune or PrefixTune to train mT5's keyword extraction capability;

[0023] Step 2.1: Fine-tune mT5;

[0024] mT5 is an encoder-decoder language model, parameterized as... Where x represents a single text; y represents the generated keyword sequence; z is the concatenation of x and y (Concat(x, y); h <i Y represents the activation function for the first i time steps; idxLet represent the index sequence corresponding to y; Φ is the initialization parameter of the pre-trained model mT5_multilingual_XLSum on the Huggingface large language model ensemble website. Then, the gradient of the target shown is updated according to formula (3):

[0025]

[0026] Step 2.2: Fine-tune mT5 using PrefixTune;

[0027] This implementation uses the peft library to perform prefix tuning. Prefix tuning is one of the most widely used and efficient parameter tuning methods. Here, continuous word embeddings are used to optimize the prompts instead of discrete tokens. The trainable matrix P is initialized. θ To store the prefix parameters. To prevent unstable optimization and mitigate the slight performance degradation, the matrix is ​​reparameterized, as given by equation (4):

[0028] P θ [i,:]=MLP θ (P' θ [i,:]) (4)

[0029] Among them, P' θ Composed of a large feedforward neural network (MLP), the hyperparameters such as the number of model layers need to be set according to the model size, where i represents the i-th time step. Here, only the prefix parameters θ and P need to be trained. θ [i,:] refers to all parameters of the model after fine-tuning.

[0030] The final model fine-tuning method is selected based on the size of the "document-topic term" dataset and the model's fine-tuning effect. The model fine-tuning effect refers to the proportion of topic terms predicted by the fine-tuned model relative to the corresponding topic terms in the test set. If the model's performance after PrefixTune is close to or higher than that after FineTune, PrefixTune, which has lower time costs, is chosen for fine-tuning subsequent datasets of similar size; otherwise, FineTune is used.

[0031] Step 3: Construct a topic vocabulary for the dataset based on word frequency information entropy and mBERT;

[0032] Step 3.1: Perform synonym aggregation based on mBERT;

[0033] First, the input news text dataset is preprocessed, including removing special characters, punctuation marks, stop words, and performing word segmentation to ensure the dataset has a uniform and consistent format. Then, mBERT encoding is performed: the pre-trained mBERT model is used to encode the preprocessed dataset. Each synonym in each language is converted into a BERT vector representation, as shown in formula (5).

[0034]

[0035] in Word i Vectorized representation using mBERT, for the synonyms extracted by mT5 i By calculating its representation in the BERT representation space and other words in the text, j The cosine similarity, as shown in Formula (6), determines the other synonyms that are most similar to it.

[0036]

[0037] Similarity(Word) i Word j ) represents a word i Word j Embedded word vectors The cosine similarity between synonyms is used to group all synonyms together, grouping those with high similarity into one category. Clustering algorithms or tree structures are then used to group the synonyms. For each grouped synonym, one is selected as a representative word and translated into the desired language form for display.

[0038] Step 3.2: Perform word frequency statistics for the topic words based on information entropy;

[0039] Using the Shannon entropy (frequency entropy) of word frequencies in different languages, a common interest vocabulary is constructed and the entries are sorted. The words contained in the nth topic are designated as Topic. n ={w n1 ,w n2 ,…w nj During the process of constructing a vocabulary list for translation and synonym merging, each word w ni Mapped to its synonym W ni The representation of the synonym, which exists in set S. n ={W n1 W n2 ,…W nq In}. For set S n The term W, after being combined with synonyms, represents the vocabulary word W in the language.ni Each synonym w n The frequency of occurrence is statistically analyzed and denoted as F. lw The probability of a synonym of W appearing in language l is shown in formula (7):

[0040]

[0041] Among them, P lw F represents the probability that a synonym of W appears in a specific language l, where L is the total number of languages. iw Let E represent the frequency of synonyms of W in language i. Then, the frequency entropy E of the synonym W in each language is... W As shown in formula (8):

[0042]

[0043] By analyzing the frequency entropy E W Sort the keywords and construct a sorted table T of commonly concerned keywords. co-concerned .

[0044] Step 4: Based on the fine-tuned mT5 and the topic vocabulary, output the predicted multilingual common interest topics;

[0045] For the constructed BERTopic model, the UMAP and HDBSCAN algorithms in BERTopic are reused to cluster the documents.

[0046] For the clustered document clusters, the c-tf-idf algorithm originally used by BERTopic for processing the clustered document clusters is replaced with the mT5 model fine-tuned in step 2 for extracting multilingual single-document topic terms. The number of output terms is customized when fine-tuning mT5. This achieves a multi-document, multilingual topic clustering detection function based on a combination of BERTopic document clustering and single-document topic term extraction mT5 models. For the single-document topic term set output by the softmax function of the last layer of the mT5 model, T... co-concerned The keyword sorting table is used for sorting and filtering to obtain the final output, achieving multilingual, low-redundancy, and common-interest keyword detection. This improves the detection capability of common-interest topics and enhances the output quality of keyword information.

[0047] Beneficial effects

[0048] The method of the present invention has the following advantages compared with the prior art:

[0049] 1. This invention discloses a multilingual common interest topic detection method based on mT5 and word frequency information entropy. It adopts the statistical method of multilingual word frequency information entropy, and fully considers the characteristic that the common interest topics are relatively evenly distributed in multiple languages, so as to realize the extraction of common interest topic words in different cultures.

[0050] 2. The present invention discloses a multilingual common interest topic detection method based on mT5 and word frequency information entropy. It uses mBERT for synonym aggregation and realizes the identification of redundant words in the process of constructing the vocabulary, thereby reducing the redundancy of the output vocabulary.

[0051] 3. This invention discloses a multilingual common interest topic detection method based on mT5 and word frequency information entropy. It improves the topic word extraction algorithm in BERTopic, uses a fine-tuned mT5 to extract topic words from a single document, and improves the traditional monolingual topic detection to a multilingual mode. This avoids the problem of missing potential valuable topic words to a certain extent, improves the detection and recognition ability of common interest topics, and enhances the output quality of topic words. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the architecture of a multilingual common interest topic detection method based on mT5 and word frequency information entropy disclosed in this embodiment.

[0053] Figure 2 This is a flowchart illustrating a multilingual common interest topic detection method based on mT5 and word frequency information entropy disclosed in this invention. Detailed Implementation

[0054] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. The technical problems solved by the present invention and its beneficial effects are also described. It should be noted that the described embodiments are only intended to facilitate understanding of the present invention and do not constitute any limitation thereof.

[0055] This embodiment discloses a multilingual common interest topic detection method based on mT5 and word frequency information entropy, such as Figure 1 As shown, it comprises three parts. First, the adjusted BERTopic text clustering architecture is used to enable the model to cluster texts and then extract topic words sequentially. Second, the fine-tuned mT5 already possesses topic word extraction capabilities on multilingual single texts, which is used here for topic word extraction from clustered texts. Finally, the proposed topic word list based on word frequency information entropy is used to detect and filter topics of common interest across multiple languages. This process still facilitates the expansion of other functions, enabling synonym deduplication and language-specific conversion output.

[0056] Taking a news dataset in five languages ​​obtained from news websites such as Al Jazeera, China News Network, and the BBC as an example, it includes three major themes: sports, technology, and economy. The corresponding common events of interest are: "2022 World Cup Final," "Chatgpt 4.0 Release," and "IMF World Economic Outlook Release." On average, there are one thousand documents in each language, with each document ranging from 600 to 2000 characters. Figure 2 As shown, the specific steps include:

[0057] Step 1: Construct a "document-topic term" dataset based on the BERTopic model;

[0058] Step 1.1: Construct a topic word extraction model based on BERTopic.

[0059] BERTopic is used as the baseline model, employing c-tf-idf and MMR (Maximal Marginal Relevance) algorithms to optimize topic word quality. The mT5 dataset obtained from the BERTopic model is then fine-tuned to unleash its potential for more efficient topic word extraction. The text embedding model is replaced with a multilingual BERT model.

[0060] The “Document-Topic” dataset contains documents in five languages: English, Arabic, Chinese, Korean, and Japanese, corresponding to {Text}. en}, {Text ar}, {Text zh}, {Text ko}, {Text ja}

[0061] Secondly, the model vector clustering algorithm was modified, and relevant hyperparameters were set. After SBERT vectorization, the document yields a set of document vectors {V}. text The original BERTopic model uses the HDBSCAN algorithm to implement unsupervised clustering, and this part needs to be replaced with the KMeans algorithm to specify the number of clusters.

[0062] Step 1.2: Data collection.

[0063] For the constructed model BERTopic, the input document set {Text} i}, set a specific input BatchSize to obtain single documents in five languages, as shown in formula (1):

[0064] SingleText i =HDBSCAN(UMAP(SBERT(Text) i)))i∈{en,ar,zh,ja,ko} (1)

[0065] As shown in formula (2), the topic terms are obtained through the c-tf-idf algorithm, and the results are optimized through MMR (Maximal Marginal Relevance) to obtain the "document-topic term" dataset:

[0066] TopicWords i =MMR(c-tf-idf(SingleText) i ))i∈{en,ar,zh,ja,ko} (2)

[0067] Since the c-tf-idf algorithm requires counting TF and IDF values ​​from multiple documents, it cannot obtain a dataset by inputting a single document. Therefore, the BatchSize value is set to 30 based on the dataset size to improve the performance of the c-tf-idf algorithm.

[0068] Step 2: Fine-tune mT5 using FineTune or PrefixTune to train its keyword extraction capabilities.

[0069] Step 2.1: Fine-tune using FineTune.

[0070] mT5 is an encoder-decoder language model, parameterized as... Where x represents a single text (Text), and y represents the generated topic word sequence (TopicWords). Assume z is the concatenation of x and y (Concat(x, y)). <i Y represents the activation function for the first i time steps. idx Φ represents the index sequence corresponding to y. The initialization of Φ comes from the parameters of the pre-trained model mT5_multilingual_XLSum on the Huggingface website. To handle parts smaller than the maximum input and output lengths, the mask is padded with 0s and the labels are padded with -100s to ensure that attention and cross-entropy loss ignore these parts. Subsequently, the gradient is updated for the objective shown in the following formula (3):

[0071]

[0072] Step 2.2: Fine-tune using the PrefixTune method.

[0073] PrefixTuning is one of the most widely used and efficient parameter tuning methods. Here, continuous word embeddings are used to optimize the prompts instead of discrete tokens. The trainable matrix P is initialized. θTo store the prefix parameters. To prevent unstable optimization and mitigate the slight performance degradation, the matrix is ​​reparameterized, as given by equation (4):

[0074] P θ [i:] = MLP θ (P′ θ [i,:]) (4)

[0075] Among them, P' θ Composed of a large feedforward neural network (MLP), it requires setting appropriate hyperparameters such as the number of model layers (the specific number depends on the model size; a 5-layer fully connected neural network is used here). i represents the i-th time step; here, only the prefix parameters θ and P need to be trained. θ [i,:] refers to all parameters of the model after fine-tuning.

[0076] In terms of dataset size, the dataset in this example is relatively small. Furthermore, the final model fine-tuning method must be selected based on the model's performance. Model fine-tuning performance refers to the proportion of predicted keywords in the test set relative to the corresponding keywords after fine-tuning. If the model performance after PrefixTune is close to or higher than that after FineTune, then PrefixTune, which has lower time consumption, should be selected for fine-tuning subsequent datasets of similar size; otherwise, FineTune should be used.

[0077] Step 3: Construct a topic vocabulary based on word frequency information entropy and mBERT.

[0078] Step 3.1: Synonym aggregation based on mBERT.

[0079] First, the test corpus is preprocessed, including removing special characters, punctuation marks, stop words, etc., and performing word segmentation. The format of the dataset is ensured to be uniform and consistent. Then, mBERT encoding is performed: the pre-trained mBERT model is used to encode the preprocessed thesaurus. For each synonym in each language, it is converted into a BERT vector representation, as shown in formula (5):

[0080]

[0081] in Word i Vectorized representation using mBERT, for the synonyms extracted by mT5 i By calculating its representation in the BERT representation space and other words in the text, j The cosine similarity (as shown in formula (6)) is used to find other synonyms that are most similar to it.

[0082]

[0083] Similarity(Word) i Word j ) represents a word i Word j Embedded word vectors The cosine similarity between synonyms is used to group all synonyms together, grouping those with high similarity into one category. Clustering algorithms (such as k-means) or tree structures are used to group the synonyms. For each grouped synonym, one is selected as a representative word and translated into the desired language form for display.

[0084] Step 3.2: Topic word frequency statistics based on information entropy.

[0085] A common interest vocabulary is constructed using Shannon entropy (hereinafter referred to as frequency entropy) of word frequencies in different languages, and the entries are sorted. The words contained in the nth topic are denoted as Topic. n ={w n1 ,w n2 ,…w nj During the process of constructing a vocabulary list for translation and synonym merging, each word w ni Mapped to its synonym W ni The representation of the synonym, which exists in set S. n ={W n1 W n2 ,…W nq In}. For set S n The element W (representing the synonyms after merging) will be used to represent the language under W. ni Each synonym w n The frequency of occurrence of [something] is statistically analyzed and denoted as F. lw Therefore, the probability of a synonym of W appearing in language l is shown in formula (7):

[0086]

[0087] Among them, P lw F represents the probability that a synonym of W appears in a specific language l, where L is the total number of languages. iw Let E represent the frequency of synonyms of W in language i. Then, the frequency entropy E of the synonym W in each language is... W Given by formula (8):

[0088]

[0089] By analyzing information entropy E WBy sorting, we can construct a sorted table T of commonly concerned keywords. co-concerned .

[0090] Step 4: Multilingual common interest topic prediction output based on mT5 and word frequency information entropy.

[0091] Step 4.1: Document clustering and mT5 topic output based on BERTopic.

[0092] First, the documents are clustered using the UMAP (Uniform, Manifold, Approximation, Projection) algorithm and the HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) algorithm implemented in BERTopic.

[0093] For the clustered document clusters, the c-tf-idf algorithm originally used by BERTopic for processing the clustered document clusters is replaced with the mT5 model fine-tuned in step 2, which is used to extract multilingual single-document topic terms. The number of output terms can be customized when fine-tuning mT5; here, it is defined as 10 based on the requirements of the number of documents and topic terms. This realizes a multi-document, multilingual topic clustering detection function based on a combination of BERTopic document clustering and the mT5 model for single-document topic term extraction.

[0094] Step 4.2: Output the word list filter.

[0095] The output keywords of the softmax function in the last layer of the model need to be derived from T. co-concerned After filtering and sorting the table, the final output is shown below:

[0096] Topic1:Qatar,World Cup,football,Argentina,France,Lussel……

[0097] Topic2:Chatgpt,openai,Intelligence,chat,models,technology,Revolution……

[0098] Topic3:IMF,world,economy,recovery,decline,slowdown,global,US……

[0099] It can be observed that keywords with high word frequency information entropy rank at the top of the output, highly matching the content of events of common interest. Furthermore, due to T co-concernedSynonym aggregation has been achieved. In the process of improving the traditional monolingual topic detection to multilingual patterns, the redundancy of output synonyms has been greatly reduced, the ability to detect and distinguish topics of common interest and topics of individual interest from different languages ​​and cultures has been improved, and the output quality of topic words has been enhanced.

[0100] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is merely a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1.A method for detecting multi-lingual common concern topics based on mT5 and word frequency information entropy, characterized in that, The method comprises the following steps: Step 1: constructing a "document-topic word" data set based on a BERTopic model; The implementation method of step 1 is, Step 1.1: constructing a BERTopic-based topic word extraction model; Taking BERTopic as a benchmark model, the c-tf-idf and maximum marginal correlation algorithm are used to optimize the quality of topic words; the self-supervised HDBSCAN algorithm or the supervised KMeans algorithm is used to replace the BERTopic model vector clustering algorithm, and relevant hyperparameters are set; the text embedding model is replaced by a language model related to the task, and a multi-language input is replaced by a multi-language model, to obtain a topic word extraction model; Step 1.2: data collection and construction of a "document-topic word" data set; In the "document-topic-word" data set, the document contains the document set of "English", "Arabic", "Chinese", "Korean", "Japanese" five languages, and the corresponding records are {Text en}, {Text ar}, {Text zh}, {Text ko}, {Text ja}; For the subject word extraction model obtained in step 1.1, input a document set {Text i}, set a specific input batch size, and obtain single documents in five languages, as shown in formula (1): （1） By model embedding a collection of documents obtaining embedding vectors; by dimensionality reduction algorithm; and by clustering algorithm, obtaining a collection of aggregated documents ; The documents in the aggregated collection of documents after the aggregation The topic words are obtained using the c-tf-idf algorithm and the results are tuned using the maximum marginal relevance algorithm to obtain a "document-topic word" dataset as shown in equation (2): （2） wherein, The five language document corresponding topic words are finally obtained by BERTopic. Since the c-tf-idf algorithm needs to count the word frequency and inverse document frequency value of multiple documents, a single document cannot be input to obtain the data set. Step 2: FineTune or PrefixTune-based mT5 fine-tuning to train the mT5 topic word extraction capability; The implementation method of step 2 is, Step 2.1: FineTune fine-tuning mT5; mT5 is an encoder-decoder language model parameterized as where x denotes the individual texts y denotes the generated topic word sequence z is the concatenation of x and y ; denotes the activation function of the first i time steps; denotes the corresponding index sequence of y; are the initialization parameters of the pre-trained model mT5_multilingual_XLSum on the large language model integration website Huggingface; then the gradient update is performed on the target shown in formula (3): （3） Step 2.2: PrefixTune fine-tuning mT5; PrefixTuning is one of the most widely used parameter-efficient tuning methods, which uses continuous word embeddings to optimize the prompt instead of discrete tokens; the trainable matrix is initialized to store the prefix parameters; to prevent unstable optimization and mitigate slight performance degradation, the matrix is re-parameterized by formula (4): （4） wherein, consists of a large feedforward neural network MLP, which requires setting hyperparameters such as the number of model layers depending on the model size, denotes the th time step, where only the prefix parameters are needed to be trained, denotes the full parameters of the fine-tuned model. According to the size of the "document-topic word" data set and the model fine-tuning effect, the final model fine-tuning method is selected; the model fine-tuning effect refers to the proportion of the predicted topic words of the fine-tuned model to the corresponding topic words in the test set; if the model effect is close to or higher than that of the FineTune model after PrefixTune, then PrefixTune with less time cost is selected to fine-tune the subsequent similar specification data sets, otherwise FineTune is used for fine-tuning; Step 3: constructing a topic word table of the data set based on word frequency information entropy and mBERT; The implementation method of step 3 is, Step 3.1: synonym aggregation based on mBERT; First, the input news text dataset is preprocessed, including deleting special characters, punctuation, stop words, and performing word segmentation processing, to ensure the uniformity and consistency of the dataset format; then Encoding: using a pre-trained model to encode the preprocessed dataset; convert each synonym in each language into a BERT vector representation, as shown in equation (5): （5） wherein representations of words through vectorized representations, for each of the passages synonyms extracted , the most similar other synonyms are determined by computing their cosine similarity in the representation space and in the text , as shown in equation (6). （6） wherein representing the vocabulary embedding word vectors , cosine similarity between the embedding word vectors, all synonyms are aggregated according to the cosine similarity, and synonyms with high similarity are classified into one class; using a clustering algorithm or a tree structure to group the synonyms; for the aggregated synonyms after grouping, taking one as a representative word and translating it into the language form we need to show; Step 3.2: word frequency statistics of topic words based on information entropy; Using the Shannon entropy (frequency entropy) of word frequencies in different languages, a common interest vocabulary is constructed and the entries are sorted; the nth topic contains the following words. In the process of constructing a vocabulary list that combines translation and thesis merging, each word Mapped to its synonyms The representation of the synonym, which exists in the set. In the middle; for sets The representative words after the combination of Chinese and English synonyms , language Down Each synonym The frequency of occurrence of [something] is statistically analyzed and denoted as [something]. Synonyms in The probability of occurrence under a language is shown in formula (7): （7） wherein, denotes the probability of the synonym of appearing in a particular language L, L is the total number of languages, denotes the frequency of the synonym of appearing in a language L; then the frequency entropy of the synonym representative word W in each language L is as shown in equation (8): （8） By sorting the frequency entropy a sorted list of co-occurring topic words is constructed ; Step 4: outputting predicted multi-language common focus topics based on the fine-tuned mT5 and the topic word table; The implementation method of step 4 is, For the constructed BERTopic model, the UMAP algorithm and the HDBSCAN algorithm in BERTopic are reused to cluster the documents; For the clustered document clusters, the c-tf-idf algorithm originally used by BERTopic for processing the clustered document clusters is replaced with the mT5 model fine-tuned in step 2, which is used to extract multilingual single-document topic terms. The number of output terms is customized when fine-tuning mT5. This achieves a multi-document, multilingual topic clustering detection function based on a combination of BERTopic document clustering and single-document topic term extraction mT5 model. For the single-document topic term set output by the softmax function of the last layer of the mT5 model, the following steps are taken: The keyword sorting table is used to sort and filter keywords to obtain the final output, achieving multilingual, low-redundancy, and common-interest keyword detection; improving the detection capability of common-interest topics and enhancing the output quality of keywords.

Images