Semantic selection method based on LongformerBERT model

Abstract

The invention relates to the technical field of text selection, and discloses a semantic selection method based on a LongformerBERT model.According to the semantic selection method based on the LongformerBERT model, systematic preprocessing is conducted on news texts, the structured degree of text data is improved, and clearer and more accurate input is provided for follow-up semantic analysis; a LongformerBERT model is introduced to perform semantic feature extraction, so that potential semantic information in a text can be deeply mined, a context dependency relationship is dynamically captured, and the real meaning of the text is more accurately understood; global attention is introduced into a small number of key tokens, important context semantic information in the long text is captured, the understanding of text depth information is enhanced, the problems of low calculation efficiency and semantic information loss possibly occurring when the long text is processed by a traditional method are avoided, and the accuracy and reliability of text analysis are further improved; according to the method, efficient and accurate text semantic feature extraction of news texts is realized, and powerful technical support is provided for application scenes such as false news detection.

Description

Technical Field

[0001] The present invention relates to the technical field of text selection, and in particular to a semantic selection method based on a Longformer_BERT model. Background Art

[0002] With the continuous development of media technology, the spread of fake news has become increasingly serious around the world. Especially with the popularity of social media and instant messaging tools, the speed and coverage of information have increased significantly, which has become a major challenge in the global information environment. Fake news not only makes it increasingly difficult for users to obtain accurate and reliable information, but may also cause the public to misunderstand real events and trigger many negative social effects. Timely detection and blocking of fake news can effectively slow down its spread and prevent fake news from having adverse effects on the public. It also enhances the reliability and user experience of social media platforms.

[0003] In existing fake news-related judgment and detection research, deep learning technology is often used to conduct relevant analysis and provide accurate and reliable predictions. Fake news detection based on traditional deep learning usually relies on TF-IDF, Bag-of-Words or static word embedding (Word2Vec) to extract features, but these methods can only capture the shallow semantics of the text, cannot perceive subtle changes in the context in a timely manner, and lack the ability to understand deep semantic information, which leads to insufficient performance of the model when dealing with polysemous words and metaphorical expressions. At the same time, the text is less processed in the preprocessing stage, often by simply truncating the text or selecting some key paragraphs as input, and the processing ability of long texts is limited, which not only reduces the computational efficiency, but also may lead to the loss of important semantic features and semantic information, and cannot meet the existing needs. Therefore, a semantic selection method based on the Longformer_BERT model is proposed. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a semantic selection method based on the Longformer_BERT model to solve the background technology problems.

[0005] To achieve the above object, the present invention provides the following technical solution: a semantic selection method based on the Longformer_BERT model, comprising the following steps: S10: Perform structured preprocessing on the input news text; specifically including: S101: Standardizing the input text; obtaining news text, and performing standardization processing on the input text; S102: Deleting stop words in the text; scanning the input text through a predefined stop word list, traversing the text sentence sequence and deleting relevant stop words in the stop word list; S103: extracting stems according to different grammatical forms; simplifying different grammatical forms of words into their root forms, retaining the basic forms of the words, and thus grouping words with the same or similar semantics into one category.

[0006] S20: Use the Longformer_BERT model to perform semantic extraction on the preprocessed text to capture the potential dependencies in the text.

[0007] S30: Obtain news text feature vectors; use the Longformer_BERT model in step S20 to select semantic features of text data to obtain the final representation feature vector of the news text.

[0008] Preferably, the step S20 further includes: S201: Use a sliding window to split long text into subword sequences and add special tags at the head and tail. If the text length exceeds the maximum input limit of the Longformer_BERT model (512 tokens), use a sliding window strategy to efficiently divide the long text into blocks. For the divided subword sequences, add special tags at the head and tail.

[0009] Preferably, the step S20 further includes: S202: Create an input tensor based on subwords; create an input tensor subword index (input_ids) and a mask tensor (attention_mask) based on the subword sequence after segmentation.

[0010] Preferably, the step S20 further includes: S203: forward propagation to obtain text data vectorization; by forward propagation, the input text is mapped into a high-dimensional semantic vector (embedding) to obtain text data vectorization.

[0011] Preferably, the step S20 further includes: S204: Use the attention network to perform weighted summation on the block vectors to generate a global semantic representation; introduce the self-attention mechanism into the Longformer_BERT model, learn the importance weights of the local semantics of each block through a dynamic weighting method, and realize weighted summation to generate a global representation.

[0012] Preferably, in the step S103, the Porter Stemmer algorithm is applied to traverse the input text to remove the suffix or prefix of the word and restore it to a fixed root form.

[0013] Preferably, in S201, the Longformer_BERT model adds a special classification tag [CLS] at the beginning to mark the beginning of the segment, and adds [SEP] at the end to mark the end of the segment.

[0014] Preferably, in step S30, a small number of key tokens in the text data are introduced into global attention to capture important contextual semantic information in the long text.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention improves the structured degree of text data by systematically preprocessing news texts, providing clearer and more accurate input for subsequent semantic analysis; introduces the Longformer_BERT model for semantic feature extraction, so that the present invention can deeply mine the potential semantic information in the text and dynamically capture contextual dependencies, so as to more accurately understand the true meaning of the text; and the present invention introduces global attention to a small number of key tokens to capture important contextual semantic information in long texts, enhance the understanding of deep text information, avoid the problems of low computational efficiency and loss of semantic information that may occur in traditional methods when processing long texts, and further improve the accuracy and reliability of text analysis; realizes efficient and accurate text semantic feature extraction of news texts, and provides strong technical support for application scenarios such as fake news detection.

[0016] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of the text semantic feature selection method of the present invention; Figure 2 Flowchart of the method for preprocessing input text for structure; Figure 3 Flowchart of the method for extracting text semantics for the Longformer_BERT model. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this technical field without creative work are within the scope of protection of the present invention.

[0019] See also Figure 1-3 A semantic selection method based on the Longformer_BERT model in the present invention systematically preprocesses the news style, structures the text data, and introduces the Longformer_BERT model to extract and optimize semantic features, thereby dynamically capturing context dependencies and significantly improving the quality of text extraction. Specifically, the method includes the following steps: S10: Structural preprocessing of input news text The representation of text data greatly affects the accuracy of the results. In the embodiment, the text analysis problem needs to be converted into a representation suitable for the applied method, such as Figure 2 As shown, it specifically includes the following steps: S101: Standardize input text In this embodiment, news text is obtained and the input text is standardized.

[0020] The specific steps are as follows: remove all punctuation marks from the text data, and convert the text data into lowercase to make it consistent in format and convenient for subsequent analysis; use N-chars character filtering to delete words containing less than N characters, remove meaningless or low-frequency parts that have little contribution to semantics, and reduce the sparsity of the data.

[0021] Standardization converts text into a unified format, simplifies subsequent processing, and improves the model's attention to text semantics.

[0022] S102: Remove stop words from text Stop words are words that appear frequently in a language but have limited semantic information, including conjunctions, prepositions and pronouns, such as about, above, after, until, again, etc., and are often used to complete sentence structures and connect expressions. In this embodiment, the input text is scanned through a predefined stop word list, the text sentence sequence is traversed and the relevant stop words in the stop word list are deleted, thereby reducing noise interference, improving the model's ability to focus on key content, and reducing data dimensions, thereby improving model training efficiency.

[0023] S103: Extracting word stems based on different grammatical forms According to the different grammatical forms of words, they are simplified to their root forms, and the basic forms of the words are retained, so that words with the same or similar semantics are classified into one category. In this embodiment, the Porter Stemmer algorithm is applied to traverse the input text, remove the suffix or prefix of the word, and restore it to a fixed root form. For example: words such as connection, connective, connected, connecting and disconnect can be stemmed into the word connect and used directly for subsequent analysis. Through stem extraction, the complexity of data dimensions and feature space can be effectively reduced, providing more concise input data for downstream tasks and improving the efficiency of semantic analysis.

[0024] S20: Use the Longformer_BERT model to extract semantic information from preprocessed text Semantic features play an important role in text analysis and directly affect the subsequent detection rate of fake news. In order to accurately extract semantics, in this embodiment, the Longformer_BERT model is used to extract text features from the preprocessed text to capture the potential dependencies in the text, such as Figure 3 As shown, it includes the following steps: S201: Use sliding windows to split long text into subword sequences and add special tags at the beginning and end When the length of the text exceeds the maximum input limit of the Longformer_BERT model (512 tokens), a sliding window strategy is used to efficiently segment the long text. In this embodiment, the window size is set to 512, and the step size of each step is half of the window size to ensure a certain overlap between segments. The long text is segmented into multiple overlapping segments by step-by-step movement, and each subword sequence contains a maximum of 512 tokens to preserve the continuity and context information of the text.

[0025] For example, assuming that the original text length is 1000 tokens, the window size is 512, and the step size is half of the window size 256, then three blocks [T1, T2, ..., T512], [T257, T258, ..., T768], and [T513, T514,..., T1000] are generated.

[0026] Special tags are added to the head and tail of the subword sequence after segmentation. In this embodiment, the Longformer_BERT model adds a special classification tag [CLS] at the beginning to mark the beginning of the segment for generating global semantic representation; and adds [SEP] at the end to mark the end of the segment for separating input.

[0027] S202: Create input tensor based on subwords The Longformer_BERT model needs to obtain input in a specific format, and creates an input tensor subword index (input_ids) and a mask tensor (attention_mask) based on the subword sequence after segmentation. In this embodiment, input_ids identifies the integer index of the word. After the long text is segmented, each word in the text is converted into a word embedding vector by querying the word vector table; for fragments of insufficient length, 0 is used to fill in to match the maximum length to ensure that the input tensor length is consistent; attention_mask identifies the valid number of digits of the word index, which has the same shape as input_ids, with valid subword bits marked as 1 and padding bits marked as 0.

[0028] S203: Forward propagation to obtain text data vectorization The input text is mapped into a high-dimensional semantic vector (embedding) through forward propagation to obtain text data vectorization.

[0029] In this embodiment, the input tensor subword index (input_ids) and the mask tensor (attention_mask) are used as the input of the Longformer_BERT model, the context embedding of each token and the local semantics of the sequence are calculated, converted into a fixed-length 768-dimensional vector and a result matrix is ​​formed, thereby realizing text feature extraction and data vectorization for each subword sequence block.

[0030] S204: Use the attention network to perform weighted summation on the block vectors to generate a global semantic representation In order to capture the global semantics of the entire text, the Longformer_BERT model introduces a self-attention mechanism, which learns the importance weights of the local semantics of each block through a dynamic weighted method, and realizes weighted summation to generate a global representation.

[0031] In this embodiment, the semantic vector of each block is mapped to a scalar or low-dimensional vector space to generate an attention score sequence [ , , ... , ]. Use the Softmax function for weight normalization:

[0032] represents the normalized attention weight, Ensure the stability of the semantic representation after summation. Perform weighted summation of local semantic vectors according to weights to generate a global semantic representation. Multiply it by its corresponding attention weight , and add up:

[0033] The global semantics is expressed as , integrate the local semantic information of all sub-word blocks, use the dynamic weighting method to highlight the important block semantic information, avoid the semantic weakening problem caused by average pooling, and effectively extract the global semantic representation of long text.

[0034] S30: Obtain news text feature vector The above Longformer_BERT model is used to select semantic features of text data. In this embodiment, news text is imported, and the text format is standardized through structured preprocessing to remove redundant or low-value data, providing concise and high-quality data input for subsequent processing; the pre-trained Longformer_BERT embedding technology is used to efficiently process long text data, introduce global attention to a small number of key tokens in the input, capture important contextual semantic information in the long text, enhance the understanding of the deep information of the text, and obtain the final representation feature vector of the news text.

[0035] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A semantic selection method based on the Longformer_BERT model, characterized in that: The following steps are involved: S10: Perform structured preprocessing on the input news text; specifically including: S101: Standardizing the input text; obtaining news text, and performing standardization processing on the input text; S102: Deleting stop words in the text; scanning the input text through a predefined stop word list, traversing the text sentence sequence and deleting relevant stop words in the stop word list; S103: extracting stems according to different grammatical forms; simplifying different grammatical forms of words into their root forms, retaining the basic forms of the words, and thus grouping words with the same or similar semantics into one category; S20: Use the Longformer_BERT model to perform semantic extraction on the preprocessed text to capture the potential dependencies in the text; S30: Obtain news text feature vectors; use the Longformer_BERT model in step S20 to select semantic features of text data to obtain the final representation feature vector of the news text.

2. The semantic selection method based on the Longformer_BERT model according to claim 1, characterized in that: The step S20 further includes: S201: Use a sliding window to split long text into subword sequences and add special tags at the head and tail. If the text length exceeds the maximum input limit of the Longformer_BERT model (512 tokens), use a sliding window strategy to efficiently divide the long text into blocks. For the divided subword sequences, add special tags at the head and tail.

3. The semantic selection method based on the Longformer_BERT model according to claim 2, characterized in that: The step S20 further includes: S202: Create an input tensor based on subwords; create an input tensor subword index (input_ids) and a mask tensor (attention_mask) based on the subword sequence after segmentation.

4. The semantic selection method based on the Longformer_BERT model according to claim 3, characterized in that: The step S20 further includes: S203: forward propagation to obtain text data vectorization; by forward propagation, the input text is mapped into a high-dimensional semantic vector (embedding) to obtain text data vectorization.

5. The semantic selection method based on the Longformer_BERT model according to claim 4, characterized in that: The step S20 further includes: S204: Use the attention network to perform weighted summation on the block vectors to generate a global semantic representation; introduce the self-attention mechanism into the Longformer_BERT model, learn the importance weights of the local semantics of each block through a dynamic weighting method, and realize weighted summation to generate a global representation.

6. The semantic selection method based on the Longformer_BERT model according to claim 1, characterized in that: In the step S103, the Porter Stemmer algorithm is applied to traverse the input text, remove the suffix or prefix of the word, and restore it to a fixed root form.

7. The semantic selection method based on the Longformer_BERT model according to claim 2, characterized in that: In S201, the Longformer_BERT model adds a special classification tag [CLS] at the beginning to mark the beginning of the segment, and adds [SEP] at the end to mark the end of the segment.

8. The semantic selection method based on the Longformer_BERT model according to claim 1, characterized in that: In step S30, a small number of key tokens in the text data introduce global attention to capture important contextual semantic information in the long text.

Images