A method for generating important news information based on a language model

Abstract

The application discloses a kind of based on language model's important information generation method, it is related to natural language processing field, including, retrieve news data from multi-source corpus, constitute original data set, select the field information of title, abstract, body, link, keyword and data source from original data set, empty and duplicate removal processing are carried out to title, body field information, data source is filtered, obtain to be handled data set;Based on to be handled data set, utilize the language model of the pre-training of Transformer to carry out text vectorization processing to title field, obtain sentence vector, and constitute sentence vector set.The application constructs high-quality to be handled data set by weighted Boolean retrieval and MinHash duplicate removal technology, in semantic analysis stage, sentence vector is generated using RoBERTa model, and unsupervised theme discovery is realized by combining UMAP dimension reduction and HDBSCAN clustering, effectively solve the high-dimensional text clustering problem.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and in particular to a method for generating news information based on a language model. Background Technology

[0002] In the field of natural language processing, automatic text summarization and public opinion analysis technologies have made significant progress in recent years. Traditional news information generation systems typically employ rule-based or statistical methods, such as graph ranking algorithms like TextRank combined with TF-IDF feature extraction to extract key information. They first cluster news documents using LDA topic models, then select representative documents from each topic cluster to generate summaries. This approach has, to some extent, solved the problem of automatically classifying massive amounts of news data and can output summaries with a certain degree of readability. Existing technologies can realize a complete process from multi-source data collection and text preprocessing to basic summary generation, providing a basic technical framework for automated news generation. With the development of deep learning technology, sequence-to-sequence models based on neural networks have demonstrated superior performance in text summarization tasks and have become a current research hotspot.

[0003] Despite some progress in existing technologies, significant limitations remain when processing multi-source, heterogeneous news data. This is particularly true when dealing with massive amounts of rapidly updated online news data, where effectively balancing processing efficiency and information quality is difficult. Current technologies lack dynamic evaluation mechanisms for the credibility of data sources and cannot adaptively adjust the weights of different sources. This results in insufficient authority and comprehensiveness in the generated news information. Existing methods typically employ fixed summary lengths and uniform processing workflows, making it difficult to adapt to the diverse characteristics of different types of news content. Especially in sentiment analysis, most systems can only provide simple binary sentiment classification, failing to meet the needs of public opinion analysis for fine-grained sentiment recognition. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a language model-based method for generating news information to solve the technical problems of imbalance between information quality and processing efficiency, insufficient authority, and crude sentiment analysis in the processing of massive heterogeneous news data.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a method for generating news information based on a language model, which includes retrieving news data from a multi-source corpus to form an original dataset, selecting field information of title, abstract, body text, links, keywords and data source from the original dataset, removing empty and duplicate fields from the title and body text fields, filtering the data source, and obtaining a dataset to be processed. Based on the dataset to be processed, the title field is vectorized using a pre-trained Transformer language model to obtain sentence vectors, which are then combined into a sentence vector set. The sentence vectors in the sentence vector set are subjected to UMAP dimensionality reduction to obtain the dimensionality-reduced sentence vector set. The HDBSCAN clustering method is used to cluster the vectors to obtain clusters. The centroid vector of each cluster is calculated to form a centroid vector set. The centroid dataset is formed by calculating the L2 distance between the centroid vector of the cluster and the dimensionality-reduced sentence vector. The main text fields in the centroid dataset are used to extract key sentences to form a summary using an automatic text summarization algorithm; The main idea sentence is extracted from each text field in the centroid dataset. The main idea sentence is then used for sentiment classification using a sentiment prediction algorithm to obtain the sentiment label. The original dataset, key sentences are used to form a summary, centroid dataset, sentiment tags, and integrated into news information.

[0007] As a preferred embodiment of the language model-based news information generation method of the present invention, the method includes: retrieving news data from a multi-source corpus to form an original dataset; selecting fields such as title, abstract, body text, links, keywords, and data source from the original dataset; removing empty and duplicate fields from the title and body text fields; and filtering the data sources to obtain the dataset to be processed. This includes the following steps: Using keyword sets and time ranges, news data is retrieved using weighted Boolean algorithm to form the original dataset; Based on the field mapping rules, select the following fields from the original dataset: title, abstract, body text, links, keywords, and data source information. Null removal is performed on the extracted title and body fields to generate a preliminary cleaned dataset. Duplicate records in the title and body of the preliminary cleaned dataset are removed based on the MinHash algorithm to obtain a deduplicated dataset. Filter the dataset to remove duplicates based on the list of trusted sources, remove low-quality source data, and output the dataset to be processed.

[0008] As a preferred embodiment of the language model-based news information generation method of the present invention, the method includes the following steps: based on the dataset to be processed, the title field is vectorized using a pre-trained Transformer language model to obtain sentence vectors, and a set of sentence vectors is formed. Extract the title field from the dataset to be processed, perform unified encoding conversion and special character filtering to form the original title set, and perform text normalization processing on the original title set to generate a standardized title set; The RoBERTa model is trained based on a standardized title set. The trained RoBERTa-wwm-ext model and WordPiece word segmenter are used to initialize the GPU computing environment and configure the mixed precision mode. Dynamic length analysis is performed on the standardized title set, and the titles are sorted into batch sets in ascending order of title length. The WordPiece word segmenter is used to segment each batch in the batch set into sub-words, generating an ID matrix and an attention mask matrix. The ID matrix and attention mask matrix are fed into the RoBERTa model to obtain the last hidden state, and the sentence vector is obtained by mean pooling. The output sentence vectors are normalized and outlier vectors are filtered out to form a set of sentence vectors.

[0009] As a preferred embodiment of the language model-based news information generation method described in this invention, the method includes: performing UMAP dimensionality reduction on the sentence vectors in the sentence vector set to obtain a dimensionality-reduced sentence vector set; using the HDBSCAN clustering method to cluster the vectors to obtain clusters; calculating the centroid vector of each cluster to form a centroid vector set; and including the following steps. The sentence vector set is Z-score standardized to eliminate dimensional differences and output a standardized sentence vector set. The IsolationForest algorithm is used to detect outliers in the standardized sentence vector set and output a cleaned sentence vector set. The cleaned sentence vector set is input into the UMAP model, and the output is the dimensionality-reduced sentence vector set. Perform HDBSCAN clustering on the dimensionality-reduced sentence vector set and output the cluster label array; Extract non-negative labels from the cluster label array, identify the set of valid clusters, calculate the centroid vector of each cluster, and obtain the set of centroid vectors.

[0010] As a preferred embodiment of the news information generation method based on language model described in this invention, the method involves: forming a centroid dataset by calculating the L2 distance between the centroid vectors of clusters and the reduced-dimensional sentence vectors, including the following steps. A distance matrix is ​​constructed by calculating the L2 distance between the centroid vectors of the clusters and the reduced-dimensional sentence vectors; Extract the distance vectors from the distance matrix, sort the distance vectors in ascending order to obtain the distance vector sequence, and remove duplicates by hashing to obtain the distance vector set; The centroid dataset is formed based on the set of distance vectors and the set of centroid vectors.

[0011] As a preferred embodiment of the language model-based news information generation method of the present invention, the method involves extracting key sentences from the text fields of the centroid dataset using an automatic text summarization algorithm to form a summary, including the following steps. Clean the text fields in the centroid dataset to output a set of standardized sentences. Filter out invalid short sentences from the standardized sentence set to generate a set of valid sentences. The Sentence Transformer model is used to calculate the semantic similarity between each sentence in the effective sentence set and the article title, and the set of sentences with semantic similarity is output. Generate a multidimensional feature matrix by taking the TF-IDF weights and positional weights of each sentence in the set of sentences with semantic similarity. Based on a multidimensional feature matrix, sentences are scored for importance, and a set of sentences with weights is output. The MMR algorithm is used to process a weighted set of sentences to obtain a summary composed of key sentences.

[0012] As a preferred embodiment of the language model-based news information generation method described in this invention, the method includes the following steps: extracting the main idea sentence from each text field in the centroid dataset to obtain the main idea sentence; performing sentiment classification using a sentiment prediction algorithm to obtain sentiment tags. Based on the effective sentence set, the sentiment word density of each sentence is obtained, a preliminary score set is generated, and a comprehensive score set is output by combining sentence position weight and semantic centrality. The main idea sentence is then extracted from the comprehensive score set. The sentiment score of the main idea sentence is predicted using a multi-level sentiment analysis method. The sentiment score is processed to obtain the overall sentiment tendency value; Sentiment labels are assigned based on overall sentiment scores.

[0013] As a preferred embodiment of the language model-based news information generation method described in this invention, the integration of news information from the original dataset, key sentence summary, centroid dataset, and sentiment tags includes the following steps: The original dataset, key sentence summary, centroid dataset, and sentiment tags are standardized and unified to obtain a multi-source unified dataset. Intelligent data fusion technology based on semantic similarity is used to organize a unified dataset from multiple sources to obtain key news information.

[0014] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the news information generation method based on a language model as described in the first aspect of the present invention.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the news information generation method based on a language model as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: A high-quality dataset is constructed using weighted Boolean retrieval and MinHash deduplication techniques. In the semantic analysis stage, the RoBERTa model is used to generate sentence vectors, and unsupervised topic discovery is achieved by combining UMAP dimensionality reduction and HDBSCAN clustering, effectively solving the problem of high-dimensional text clustering. In the information generation stage, the MMR algorithm is used to extract key sentence summaries, and fine-grained sentiment prediction is achieved through a multi-level sentiment analysis method, breaking through the limitations of traditional binary classification. The UMAP-HDBSCAN joint clustering strategy automatically identifies news topic clusters of different densities and extracts representative properties. The weighted average sentiment analysis algorithm integrates features such as sentiment word density, position weight, and semantic centrality to output sentiment tags with intensity distinction. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of a news information generation method based on a language model.

[0019] Figure 2 This is a flowchart of the overall process for generating news information based on language models.

[0020] Figure 3 This is a flowchart illustrating the data retrieval process for a language model-based news information generation method.

[0021] Figure 4 This is a flowchart of the headline clustering method for generating news information based on language models.

[0022] Figure 5This is a flowchart illustrating the text vectorization process for a language model-based news information generation method. Detailed Implementation

[0023] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0024] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0025] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0026] Reference Figures 1-5 This is one embodiment of the present invention, which provides a method for generating news information based on a language model, including the following steps: S1. Retrieve news data from a multi-source corpus to form a raw dataset. Select fields for title, abstract, body text, links, keywords, and data source from the raw dataset. Remove empty and duplicate fields from the title and body text fields and filter the data sources to obtain the dataset to be processed.

[0027] S1.1 Using keyword sets and time ranges, news data is retrieved using weighted Boolean algorithm to form the original dataset.

[0028] Furthermore, based on the core terms provided by users, semantic expansion is performed using the Word2Vec model to ensure comprehensive retrieval. Weighted Boolean queries are executed in the news corpus, and the retrieval results are obtained from multiple sources, including websites, forums, WeChat, mobile apps, Weibo, and digital newspapers, to form the original dataset.

[0029] S1.2. Based on the field mapping rules, select the field information of title, abstract, body text, links, keywords and data source from the original dataset.

[0030] Furthermore, after the initial construction of the original dataset, a field filtering operation is first performed on each structured record in the original dataset. The filtering process is carried out according to the field mapping rules, extracting the title field, summary field, body text field, link field, keyword field, and data source field from each record. The title field is used for subsequent text vectorization processing based on the Transformer-based pre-trained language model. The summary field retains the general description content in the original corpus and is used for comparative analysis with the key sentences extracted by the subsequent automatic text summarization algorithm. The body text field serves as the source of input text content for the subsequent automatic text summarization algorithm and sentiment prediction algorithm. The link field corresponds to the original publishing page address of the news information and is retained for the jumpable access function in the news information output. The keyword field extracts the keyword annotation information from the original dataset and is retained to assist in semantic similarity analysis and topic clustering feature judgment. The data source field is used to mark the publishing platform or source media of the news text and is used to filter low-quality source records in combination with the subsequent data source filtering strategy. During the field filtering process, if some structured records have missing fields, the field value null check logic is executed to skip the records with missing fields and retain the records with complete fields to enter the next step of field null removal and deduplication processing, thus completing the process of extracting field information from the original dataset into a structured normalized dataset.

[0031] S1.3. Perform null removal processing on the extracted title and body fields to generate a preliminary cleaned dataset. Based on the MinHash algorithm, remove duplicate records from the title and body of the preliminary cleaned dataset to obtain a deduplicated dataset.

[0032] Furthermore, for each structured record, the title and body fields are checked for non-emptiness. If either field is empty, the record is removed and not included in subsequent operations. All remaining records that satisfy the field integrity requirement constitute the initial cleaned dataset. After removing empty fields, to avoid semantic interference caused by duplicate title and body fields, duplicate record identification and removal are performed on the title and body fields of the initial cleaned dataset. The MinHash algorithm is used to calculate the similarity of the title and body fields, generating a signature set for hash comparison. The similarity relationship is determined by Jaccard similarity. If the Jaccard similarity between the title and body fields is greater than the similarity threshold, it is considered a duplicate record. Only one record with complete content and higher source priority is retained, and all other duplicate records are removed. After the above processing, the output is a deduplicated dataset containing unique combinations of title and body fields.

[0033] S1.4 Filter the dataset to remove duplicates based on the list of trusted sources, remove low-quality source data, and output the dataset to be processed.

[0034] Furthermore, a trusted source list is first prepared, containing a set of news release channel names verified manually or by rules. Then, the source field of each structured record in the deduplicated dataset is matched against the channel names in the trusted source list. If a content in the source field does not appear in the trusted source list, the record is deemed to have an unacceptable source quality and is discarded. Records with successfully matched source fields are retained, forming the final output dataset. This process ensures that the input data for subsequent title field text vectorization has high information reliability and content accuracy, and logically connects with the deduplicated dataset from the previous stage. Ultimately, this completes the screening and source filtering process for the title, abstract, body, link, keyword, and data source fields.

[0035] S2. Based on the dataset to be processed, use the pre-trained Transformer language model to perform text vectorization on the title field to obtain sentence vectors and form a set of sentence vectors.

[0036] S2.1 Extract the title field from the dataset to be processed, and perform unified encoding conversion and special character filtering to form the original title set. Perform text normalization processing on the original title set to generate a standardized title set.

[0037] Furthermore, the title field content is extracted line by line from the dataset to be processed, forming an original title set. Then, a unified encoding conversion operation is performed on each title field content in the original title set, unifying all characters to UTF-8 encoding format to ensure encoding consistency in subsequent text processing. After encoding conversion, special character filtering is performed on each title field content in the original title set. The filtering targets include invisible control characters, newline characters, tab characters, and special symbols with example values. Valid text information is retained after removal. After special character filtering, a clean original title set is formed. Next, text normalization processing is performed on the original title set. Normalization operations include unifying full-width and half-width characters, unifying Chinese and English punctuation formats, unifying simplified and traditional Chinese character shapes, and unifying uppercase and lowercase formats. All title field content that has undergone text normalization processing forms a standardized title set. S2.2. Train the RoBERTa model based on the standardized title set. Initialize the GPU computing environment and configure the mixed precision mode using the trained RoBERTa-wwm-ext model and WordPiece word segmenter.

[0038] Furthermore, a standardized title set was used as the input training corpus to fine-tune the RoBERTa-wwm-ext Chinese pre-trained language model. During training, the WordPiece word segmenter was used to segment the input text at the sub-word level, converting each title field in the standardized title set into a Token ID sequence and a corresponding attention mask matrix. These were then batch-input into the RoBERTa-wwm-ext Chinese pre-trained language model. To ensure hardware resource efficiency and model stability during training, the GPU computing environment was initialized after the RoBERTa-wwm-ext Chinese pre-trained language model was loaded, and mixed-precision mode was enabled for training computation. Mixed-precision mode employed a hybrid FP16 and FP32 computation method, improving training speed and GPU memory utilization efficiency without affecting the convergence of the RoBERTa-wwm-ext Chinese pre-trained language model. After training, the RoBERTa-wwm-ext Chinese pre-trained language model possessed the ability to efficiently extract semantic vectors from the standardized title set, outputting intermediate hidden state information that could be used for subsequent mean pooling steps.

[0039] It should be noted that the RoBERTa-wwm-ext Chinese pre-trained language model is based on the BERT model structure and uses the Whole Word Masking strategy for Chinese pre-training. The training input uses a standardized set of titles, and the WordPiece word segmenter is used to process sub-words, generating Token IDs and attention mask matrices for input into the BERT model. During training, the Adam optimizer and cross-entropy loss function are used, and the training is executed in a GPU environment. The training batches are divided with dynamic length, and mixed precision acceleration is enabled during training iterations to ensure that the BERT model has higher text semantic expression capabilities after fine-tuning.

[0040] S2.3 Perform dynamic length analysis on the standardized title set, sort the titles into batch sets in ascending order of title length, use the WordPiece word segmenter to segment each batch in the batch set into sub-words, and generate an ID matrix and an attention mask matrix.

[0041] Furthermore, the length of each title field in the standardized title set is statistically analyzed, with the unit of measurement being the number of characters. After length analysis, the standardized title set is sorted in ascending order based on the character length of the title field, forming a title sequence arranged in ascending order of length. The sorted title sequence is then divided into multiple batch sets, with title field lengths in each batch set being similar to reduce computational redundancy caused by padding operations. For each batch set, the WordPiece tokenizer is used to perform sub-word segmentation on each title field, dividing the complete text into several sub-word units and mapping them to corresponding Token IDs. Based on the Token ID sequence generated for each title field, an equal-length ID matrix is ​​constructed. Insufficient parts in the matrix are padded using special padding markers. Simultaneously, an attention mask matrix corresponding to the ID matrix is ​​generated to identify the positional relationship between valid tokens and padding markers. The output consists of the ID matrix and the attention mask matrix.

[0042] S2.4. Input the ID matrix and attention mask matrix into the RoBERTa model to obtain the last hidden state, and obtain the sentence vector through mean pooling.

[0043] Furthermore, the ID matrix and attention mask matrix are used as parallel inputs, corresponding to the token embedding layer and attention calculation mask in the RoBERTa-wwm-ext Chinese pre-trained language model, respectively. During the forward propagation, the model performs deep encoding processing through multiple Transformer structures according to the position and mask information of each token, ultimately outputting the vector representation of each token in the last encoding layer. From the last hidden state output by the RoBERTa-wwm-ext Chinese pre-trained language model, the model extracts the complete set of token vectors corresponding to each title field, converting the token-level semantic representation into a sentence-level overall semantic representation. Mean pooling is then performed on the token vector set corresponding to each title field, averaging all valid token vectors across each dimension to obtain a dimensionally consistent sentence vector representation. This sentence vector representation serves as the compressed semantic representation of the title field. S3. Perform UMAP dimensionality reduction on the sentence vectors in the sentence vector set to obtain the dimensionality-reduced sentence vector set. Then, use the HDBSCAN clustering method to cluster the vectors and obtain clusters. Calculate the centroid vector of each cluster to form a centroid vector set.

[0044] S3.1 Perform Z-score standardization on the sentence vector set to eliminate dimensional differences and output a standardized sentence vector set. Use the IsolationForest algorithm to detect outliers in the standardized sentence vector set and output the cleaned sentence vector set.

[0045] Furthermore, after generating the sentence vector set through mean pooling, in order to eliminate the dimensional differences between different dimensions and improve the effect of subsequent clustering, the sentence vector set needs to be Z-score standardized. The Z-score standardization process is as follows: calculate the mean and standard deviation of each dimension in the sentence vector set, subtract the corresponding mean from the value of each dimension, and then divide by the standard deviation to generate a standardized sentence vector set with a mean of zero and a variance of one. After the standardized sentence vector set is generated, in order to further improve the representativeness of the sentence vector expression, the IsolationForest algorithm needs to be used to detect outliers in the standardized sentence vector set. The IsolationForest algorithm constructs multiple random subsampling tree models and determines the average path length of each sentence vector separated in the tree structure. Sentence vectors with shorter path lengths are identified as outliers and are removed from the standardized sentence vector set. Sentence vectors with path lengths within the normal range are retained as the cleaned sentence vector set.

[0046] S3.2 Input the cleaned sentence vector set into the UMAP model and output the dimensionality-reduced sentence vector set.

[0047] Furthermore, to further reduce the vector dimensionality and enhance subsequent clustering results, the cleaned sentence vector set needs to be input into the UMAP model for nonlinear dimensionality reduction. The UMAP model aims to preserve the local structural relationships of the original sentence vector set. It constructs an adjacency graph of sample points in a high-dimensional space and maps these adjacency relationships to a low-dimensional space, ensuring that similar sentence vectors remain close together in the low-dimensional space. The cleaned sentence vector set serves as input, and after processing by the UMAP model, it outputs a dimensionality-reduced set of sentence vectors. It should be noted that the UMAP model is based on a topology preservation mechanism. During the training phase, it constructs a weighted adjacency graph by calculating the neighborhood relationships between each pair of samples in the cleaned sentence vector set, and then embeds the high-dimensional structure into the low-dimensional space through optimization. In practical use, the UMAP model does not require supervision of the original data labels; the input is the cleaned sentence vector set.

[0048] S3.3 Perform HDBSCAN clustering on the reduced-dimensional sentence vector set and output the cluster label array.

[0049] Furthermore, HDBSCAN clustering is performed on the dimensionality-reduced sentence vector set. Based on the principle of density distribution, the HDBSCAN clustering algorithm constructs a minimum spanning tree structure by considering the relative density and interconnection relationships between vectors in the dimensionality-reduced sentence vector set without pre-setting the number of clusters. It then identifies cluster structures with density stability. First, parameters such as the minimum number of cluster samples are set. Then, the dimensionality-reduced sentence vector set is input into the HDBSCAN clustering algorithm. By calculating the distance and local density index between each pair of vectors, the HDBSCAN clustering algorithm assigns a cluster label to each sentence vector in the dimensionality-reduced sentence vector set. These cluster labels are arranged in an integer array.

[0050] S3.4 Extract non-negative labels from the cluster label array, identify the set of valid clusters, calculate the centroid vector of each cluster, and obtain the set of centroid vectors.

[0051] Specifically, the expression is, ; in, For the first The centroid vectors of the clusters, For clusters The sample set, For the first The feature vector of each sample This is the centroid vector index of the cluster.

[0052] Furthermore, the cluster label array is first traversed, extracting all non-negative integer labels. These labels represent successfully classified sentence vector records, corresponding to valid cluster numbers. Sentence vectors with the same non-negative label value are grouped into a single cluster, thus establishing a set of valid clusters. Then, the arithmetic mean of all sentence vectors in each valid cluster is calculated for each dimension. The averaging operation is performed by summing the elements corresponding to each dimension and then dividing by the number of sentence vectors to obtain the centroid vector of that cluster. This process is repeated for all valid clusters, ultimately yielding a set of centroid vectors. Each vector in the centroid vector set represents the center position of the corresponding cluster in the reduced-dimensional semantic space.

[0053] S4. By calculating the L2 distance between the centroid vectors of the clusters and the reduced sentence vectors, a centroid dataset is formed.

[0054] S4.1 Construct a distance matrix by calculating the L2 distance between the centroid vector of the cluster and the reduced-dimensional sentence vector.

[0055] ; in, It is a distance matrix. The dimension of the centroid vector. For the first A reduced-dimensional sentence vector.

[0056] S4.2 Extract the distance vectors from the distance matrix, sort the distance vectors in ascending order to obtain the distance vector sequence, and remove duplicates by hashing to obtain the distance vector set.

[0057] Furthermore, after constructing the centroid vector set, to identify the most representative sentence vector for each cluster, a distance matrix needs to be constructed based on the L2 distance calculation results between the centroid vector set and the dimensionality-reduced sentence vector set. The distance matrix is ​​a two-dimensional numerical array, where each row represents the Euclidean distance between a centroid vector and all dimensionality-reduced sentence vectors, expressed as the corresponding distance vector. The corresponding distance vector is extracted from each row of the distance matrix to obtain multiple separate distance vector sets. Then, the values ​​in each distance vector are sorted in ascending order to obtain a sequence of distance vectors arranged from near to far. Sentence vectors with smaller distances indicate that they are closer to the centroid vector in the semantic space. To avoid duplicate candidate sentence vectors in different clusters, a hash deduplication operation is performed on all distance vector sequences. The index of the vectors that have appeared is recorded by a hash algorithm, only the first occurrence position is retained, and subsequent duplicates are removed. After deduplication, the distance vector set is output.

[0058] S4.3. Based on the set of distance vectors and the set of centroid vectors, construct the centroid dataset.

[0059] Furthermore, the process first iterates through each centroid vector in the centroid vector set, finding the index of the reduced-dimensional sentence vector with the smallest L2 distance to the centroid vector in the corresponding distance vector set. Using this index, it backtracks to the position of the corresponding original sentence vector in the reduced-dimensional sentence vector set, and then further maps it back to the position of the original structured record in the dataset to be processed. For each cluster, the L2 distance is used to retrieve the n vectors closest to the centroid vector, with n set to 1. The data corresponding to the centroid vector are used to form the centroid dataset. Only the original structured record closest to the current centroid vector is selected as the representative record of the cluster. After completing the corresponding operations for all clusters, all selected original structured records are summarized to form the centroid dataset.

[0060] S5. Extract key sentences from the text fields in the centroid dataset using an automatic text summarization algorithm to form a summary.

[0061] S5.1 Clean the text fields in the centroid dataset, output a set of standardized sentences, filter out invalid short sentences in the set of standardized sentences, and generate a set of valid sentences.

[0062] Furthermore, the text content is extracted line by line from the centroid dataset, and character-level preprocessing is performed on each extracted text segment. This includes removing invisible control characters, HTML tags, emoticons, redundant spaces, line breaks, and special symbols, as well as standardizing full-width and half-width punctuation and English / Chinese symbols. After character cleaning, sentence segmentation methods (such as using periods, question marks, exclamation marks, etc.) are used to divide each text segment into several complete natural language sentences, outputting a structured standardized sentence set. The validity of each sentence in the standardized sentence set is judged, and invalid sentences consisting only of punctuation marks, meaningless phrases, advertising statements, or excessively short character lengths are filtered out. For example, sentences with fewer than 5 valid Chinese characters are considered invalid. All valid natural language sentences are combined into a valid sentence set.

[0063] S5.2 Use the Sentence Transformer model to calculate the semantic similarity between each sentence in the effective sentence set and the article title, and output the sentence set with semantic similarity.

[0064] Furthermore, for each record in the centroid dataset, the title field content and the corresponding set of valid sentences are extracted. The title field content and the text of each valid sentence are then input into the Sentence Transformer model to obtain the corresponding sentence vector representation. For each pair of title field content and valid sentence, after obtaining two sentence vectors, a cosine similarity function is used to calculate their semantic similarity score. A higher similarity score indicates that the valid sentence is semantically closer to the title field content and has higher summarization value. After calculating the semantic similarity between all valid sentences and the title field, the similarity score of each sentence with the title field is recorded, and the set of sentences with semantic similarity is output in record order.

[0065] S5.3. Generate a multidimensional feature matrix by taking the TF-IDF weights and positional weights of each sentence in the semantically similar sentence set.

[0066] Furthermore, for each sentence in the semantically similar sentence set, word segmentation is performed and word frequency is calculated. A term-document frequency statistics table is constructed by combining the content of the text field containing the sentence. The TF-IDF weight score corresponding to the sentence is calculated based on the word frequency and inverse document frequency. The TF-IDF weight reflects the information density of the sentence in the current text. The position index of each sentence in the original text field is recorded. The position weight can be determined by the order of the paragraph in which the sentence is located or its proportion in the overall sentence order in the text; a larger value indicates a higher position. After completing the above two feature calculations, the TF-IDF weight, position weight, and previously obtained semantic similarity score of each sentence are combined as a three-dimensional numerical vector to finally form a structurally unified multidimensional feature matrix.

[0067] S5.4 Based on the multidimensional feature matrix, the importance of sentences is scored, and a set of sentences with weights is output.

[0068] Furthermore, the semantic similarity score, TF-IDF weight, and positional weight in the multidimensional feature vector corresponding to each sentence are normalized to ensure that the scores of each dimension have the same scale. Then, the comprehensive importance score of each sentence is calculated by weighted linear combination. The weight coefficients can be set according to the task objectives. For example, the example values ​​are semantic similarity weight of 0.5, TF-IDF weight of 0.3, and positional weight of 0.2. After the weighted calculation is completed, a set of sentences with numerical weight labels is established in the order of sentences. Each sentence is matched one-to-one with its corresponding importance score to form a weighted set of sentences for output.

[0069] S5.5. Use the MMR algorithm to process the weighted sentence set to obtain a summary composed of key sentences.

[0070] Furthermore, the text content of each data point in the centroid dataset is segmented into sentences to obtain a sentence set, where each sentence has already obtained its corresponding weight value through preprocessing. The MMR algorithm is used to calculate the MMR score of each sentence. An empty summary set is initialized, and the following operations are performed iteratively: the sentence with the highest current MMR score is selected from the sentence set, added to the initialized empty summary set, and removed from the sentence set; the MMR scores of the remaining sentences in the sentence set are updated; the maximum similarity item with the selected sentence is recalculated; the iteration terminates when the number of sentences in the initialized empty summary set reaches a preset value m (3 in the example) or S is empty; the sentences in the initialized empty summary set are reordered according to the order in which they appear in the original document, generating a final summary composed of m key sentences.

[0071] S6. Extract the main idea sentence from each text field in the centroid dataset to obtain the main idea sentence. Use the sentiment prediction algorithm to classify the sentiment and obtain the sentiment label.

[0072] S6.1 Based on the effective sentence set, obtain the sentiment word density of each sentence, generate a preliminary score set, combine sentence position weight and semantic centrality to output a comprehensive score set, and extract the topic sentence from the comprehensive score set.

[0073] Furthermore, the weighted sentence set is initially sorted based on the comprehensive importance score of each sentence, and the sentence with the highest score is selected as the first sentence of the summary. Among the unselected sentences, the semantic similarity between each candidate sentence and the selected summary sentence is calculated in turn. Combined with its original importance score, the marginal score is calculated based on the objective function of the maximum marginal relevance algorithm. The marginal score is jointly determined by the difference in similarity between the sentence importance score and the selected sentence. The similarity between the sentence and the selected sentence is obtained by calculating the cosine similarity between the sentence vectors. In each iteration, the candidate sentence with the highest marginal score is selected and added to the summary set. This process is repeated from the remaining candidate sentences until a set number of key sentences are selected. For example, the set number is 3 sentences. The selected key sentences are reordered according to their order of appearance in the original text field to form the summary text. The generated summary consists of several semantically non-repetitive and representative key sentences.

[0074] S6.2 Predict the sentiment score of the main idea sentence using a multi-level sentiment analysis method.

[0075] Specifically, the expression is, ; in, For the predicted sentiment score, For sentiment word density weight, For sentiment word density, For positional weights, For location feature values, To represent the time decay weight, This is the time decay factor.

[0076] Furthermore, a multi-level sentiment analysis method is applied to predict the sentiment score of the main idea sentence. First, sentiment words, sentiment phrases and their contextual information are extracted for each main idea sentence. Combined with part-of-speech tagging and syntactic dependency relations, the intensity and direction of sentiment expression are identified. Through sentiment dictionary matching and rule calculation, a preliminary sentiment tendency score is obtained. Then, a sentiment classification algorithm based on attention mechanism is used to perform deep semantic understanding in combination with sentence semantic features, and the sentiment score of the main idea sentence is output.

[0077] S6.3 Process the sentiment score to obtain the overall sentiment tendency value.

[0078] Specifically, the expression is, ; in, This represents the overall sentiment tendency score. For the first The sentiment score of each main idea sentence. Number of main idea sentences For the first The weight of each topic sentence Index of main idea sentences.

[0079] S6.4. Divide sentiment labels based on overall sentiment tendency values.

[0080] Furthermore, sentiment tags are categorized based on the overall sentiment tendency score. First, the sentiment scores and corresponding weights of all topic sentences are calculated, and a weighted average is used as the overall sentiment tendency score. Then, based on preset sentiment interval thresholds, the overall sentiment tendency score is divided into different categories of sentiment tags, such as positive, neutral, and negative. The specific categorization process includes comparing the overall sentiment tendency score across intervals to determine the sentiment tag affiliation, ensuring that the sentiment tags accurately reflect the overall emotional tendency of the text.

[0081] S7. Combine the original dataset, key sentences to form a summary, centroid dataset, and sentiment tags to integrate the news information.

[0082] S7.1 Standardize and unify the original dataset, key sentences to form a summary, centroid dataset, and sentiment tags to obtain a multi-source unified dataset.

[0083] Furthermore, the title, abstract, body text, links, keywords, and data source fields contained in the original dataset, along with the key sentence text content in the abstract, the corresponding centroid sentence text and its related attributes in the centroid dataset, and the sentiment classification results in the sentiment tags, are all processed according to a unified data format specification. The processing includes a unified text encoding format, a unified date and time representation, a unified field naming rule, and a unified text cleaning standard, such as removing invalid characters, standardizing punctuation marks, and whitespace. The corresponding fields in each data source are merged according to a predefined mapping relationship to form a multi-source unified data set.

[0084] S7.2. Use semantic similarity-based intelligent data fusion technology to organize multi-source unified data sets to obtain news information.

[0085] Furthermore, text information from different sources is semantically vectorized separately. A pre-trained RoBERTa-wwm-ext model is used to convert the text into high-dimensional semantic vectors. Cosine similarity between these semantic vectors identifies highly semantically related content, and a similarity threshold filters out redundant information. Based on the semantic similarity results, a weighted fusion method is used to integrate the text content. The weights are dynamically adjusted according to the credibility of the text source and the sentiment label's tendency, forming a unified fused expression that generates news information covering the title, summary, body text, keywords, and sentiment information.

[0086] This embodiment also provides a computer device applicable to the news information generation method based on language models, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the news information generation method based on language models as proposed in the above embodiment.

[0087] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0088] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the news information generation method based on a language model as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0089] In summary, this invention constructs a high-quality dataset for processing using weighted Boolean retrieval and MinHash deduplication techniques. In the semantic analysis stage, it employs the RoBERTa model to generate sentence vectors, combining UMAP dimensionality reduction and HDBSCAN clustering to achieve unsupervised topic discovery, effectively solving the problem of high-dimensional text clustering. In the information generation stage, it uses the MMR algorithm to extract key sentence summaries and innovatively achieves fine-grained sentiment prediction through multi-level sentiment analysis, breaking through the limitations of traditional binary classification. The UMAP-HDBSCAN joint clustering strategy automatically identifies news topic clusters of different densities and extracts representative properties. The weighted average sentiment analysis algorithm integrates features such as sentiment word density, positional weight, and semantic centrality to output sentiment tags with intensity distinctions.

[0090] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for generating news information based on a language model, characterized in that: include, News data is retrieved from a multi-source corpus to form a raw dataset. Field information such as title, abstract, body text, links, keywords, and data source is selected from the raw dataset. Empty and duplicate fields are removed from the title and body text fields, and the data source is filtered to obtain the dataset to be processed. Based on the dataset to be processed, the title field is vectorized using a pre-trained Transformer language model to obtain sentence vectors, which are then combined into a sentence vector set. The sentence vectors in the sentence vector set are subjected to UMAP dimensionality reduction to obtain the dimensionality-reduced sentence vector set. The HDBSCAN clustering method is used to cluster the vectors to obtain clusters. The centroid vector of each cluster is calculated to form a centroid vector set. The centroid dataset is formed by calculating the L2 distance between the centroid vector of the cluster and the dimensionality-reduced sentence vector. The main text fields in the centroid dataset are used to extract key sentences to form a summary using an automatic text summarization algorithm; The main idea sentence is extracted from each text field in the centroid dataset. The main idea sentence is then used for sentiment classification using a sentiment prediction algorithm to obtain the sentiment label. The original dataset, key sentences are used to form a summary, centroid dataset, sentiment tags, and integrated into news information. A centroid dataset is constructed by calculating the L2 distance between the centroid vectors of the clusters and the reduced-dimensional sentence vectors. This process includes the following steps. Distance moments are constructed by calculating the L2 distance between the centroid vectors of the clusters and the reduced-dimensional sentence vectors; Extract the distance vectors from the distance matrix, sort the distance vectors in ascending order to obtain the distance vector sequence, and remove duplicates by hashing to obtain the distance vector set; The centroid dataset is formed based on the set of distance vectors and the set of centroid vectors.

2. The news information generation method based on a language model as described in claim 1, characterized in that: News data is retrieved from a multi-source corpus to form a raw dataset. From this raw dataset, fields for title, abstract, body text, links, keywords, and data source information are selected. The title and body text fields are de-emptied and deduplicated. The data sources are then filtered to obtain the dataset to be processed. This process includes the following steps: Using keyword sets and time ranges, news data is retrieved using weighted Boolean algorithm to form the original dataset; Based on the field mapping rules, select the field information of title, abstract, body text, links, keywords and data source from the original dataset; Null removal is performed on the extracted title and body fields to generate a preliminary cleaned dataset. Duplicate records in the title and body of the preliminary cleaned dataset are removed based on the MinHash algorithm to obtain a deduplicated dataset. Filter the dataset to remove duplicates based on the list of trusted sources, remove low-quality source data, and output the dataset to be processed.

3. The news information generation method based on a language model as described in claim 2, characterized in that: Based on the dataset to be processed, the title field is vectorized using a pre-trained Transformer language model to obtain sentence vectors, which are then combined into a sentence vector set. This process includes the following steps: Extract the title field from the dataset to be processed, perform unified encoding conversion and special character filtering to form the original title set, and perform text normalization processing on the original title set to generate a standardized title set; The RoBERTa model is trained based on a standardized title set. The trained RoBERTa-wwm-ext model and WordPiece word segmenter are used to initialize the GPU computing environment and configure the mixed precision mode. Dynamic length analysis is performed on the standardized title set, and the titles are sorted into batch sets in ascending order of title length. The WordPiece word segmenter is used to segment each batch in the batch set into sub-words, generating an ID matrix and an attention mask matrix. The ID matrix and attention mask matrix are fed into the RoBERTa model to obtain the last hidden state, and the sentence vector is obtained by mean pooling. The output sentence vectors are normalized and outlier vectors are filtered out to form a set of sentence vectors.

4. The news information generation method based on a language model as described in claim 3, characterized in that: The sentence vectors in the sentence vector set are subjected to UMAP dimensionality reduction to obtain the dimensionality-reduced sentence vector set. The HDBSCAN clustering method is then used to cluster the vectors, resulting in clusters. The centroid vector of each cluster is calculated, and a centroid vector set is formed. This process includes the following steps: The sentence vector set is Z-score standardized to eliminate dimensional differences and output a standardized sentence vector set. The IsolationForest algorithm is used to detect outliers in the standardized sentence vector set and output a cleaned sentence vector set. The cleaned sentence vector set is input into the UMAP model, and the output is the dimensionality-reduced sentence vector set. Perform HDBSCAN clustering on the dimensionality-reduced sentence vector set and output the cluster label array; Extract non-negative labels from the cluster label array, identify the set of valid clusters, calculate the centroid vector of each cluster, and obtain the set of centroid vectors.

5. The news information generation method based on a language model as described in claim 1, characterized in that: The text fields in the centroid dataset are used to extract key sentences and compose a summary using an automatic text summarization algorithm. This includes the following steps: Clean the text fields in the centroid dataset to output a set of standardized sentences. Filter out invalid short sentences from the standardized sentence set to generate a set of valid sentences. The Sentence Transformer model is used to calculate the semantic similarity between each sentence in the effective sentence set and the article title, and the set of sentences with semantic similarity is output. Generate a multidimensional feature matrix by taking the TF-IDF weights and positional weights of each sentence in the set of sentences with semantic similarity. Based on a multidimensional feature matrix, sentences are scored for importance, and a set of sentences with weights is output. The MMR algorithm is used to process a weighted set of sentences to obtain a summary composed of key sentences.

6. The news information generation method based on a language model as described in claim 5, characterized in that: The main idea sentence is extracted from each text field in the centroid dataset. A sentiment prediction algorithm is then used for sentiment classification to obtain sentiment labels. This process includes the following steps: Based on the effective sentence set, the sentiment word density of each sentence is obtained, a preliminary score set is generated, and a comprehensive score set is output by combining sentence position weight and semantic centrality. The main idea sentence is then extracted from the comprehensive score set. The sentiment score of the main idea sentence is predicted using a multi-level sentiment analysis method. The sentiment score is processed to obtain the overall sentiment tendency value; Sentiment labels are assigned based on overall sentiment scores.

7. The news information generation method based on a language model as described in claim 6, characterized in that: The process of integrating news information from the original dataset, key sentence summaries, centroid datasets, and sentiment tags includes the following steps: The original dataset, key sentence summary, centroid dataset, and sentiment tags are standardized and unified to obtain a multi-source unified dataset. Intelligent data fusion technology based on semantic similarity is used to organize a unified dataset from multiple sources to obtain key news information.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the news information generation method based on any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the news information generation method based on the language model as described in any one of claims 1 to 7.

Images