A general english tweet preprocessing method and computer equipment

Abstract

The present application relates to a kind of general english tweet preprocessing method and computer equipment, belong to data processing technical field;Solved the problem that a large number of subjective words and non-standard morphemes exist in english tweet, affect tweet preprocessing result and named entity recognition performance;The method of the present application includes: based on multiple field english text, subjective word table is obtained by construction;The non-standard morpheme in the english tweet to be processed is carried out semantic reduction and information extraction, and the tweet text after preprocessing is obtained;Based on the tweet text after preprocessing, double stack structure is constructed to extract clause;Based on subjective word table, using syntax dependency analysis model and tree parent-child level structure, the tweet text after preprocessing is carried out named entity extraction, and the named entity recognition result of english tweet is obtained;The preprocessing result of english tweet is obtained by outputting the tweet text after preprocessing, the clause contained in english tweet and the named entity recognition result of english tweet.

Description

Technical Field

[0001] This invention relates to the field of text processing technology, and in particular to a general method and computer device for preprocessing English tweets. Background Technology

[0002] Text preprocessing is a language- and task-driven pre-processing technique in the field of natural language processing. On the one hand, different languages ​​require different processing methods—for example, in word segmentation, English text can be segmented using spaces, while Chinese text requires greedy matching through a large-scale dictionary, thus necessitating the design of different preprocessing methods for each language. On the other hand, the input text may not meet the requirements of downstream tasks in terms of format and content, and different tasks have different definitions of "text noise"—for example, in emotion classification tasks, subjective emotion words in the text contribute significantly to the task results, so such words cannot be easily removed; in attitude detection tasks, subjective emotion in the text has a weak negative correlation with the target, therefore, the text preprocessing stage needs to emphasize the universality of the methods.

[0003] Currently, the mainstream approach to text preprocessing pipelines for English tweets is to identify morphemes of specific formats (i.e., special rules) and perform operations such as normalization, removal, and restoration. Representative programs include NLTK and Preprocessor. However, due to the informal and immediate nature of tweets, text preprocessing methods based on special rules struggle to reasonably distinguish and select between noise and relevant information in tweet text. Summary of the Invention

[0004] Based on the above analysis, the present invention aims to provide a general method and computer device for preprocessing English tweets; to solve the problem that a large number of non-standard morphemes and subjective words exist in English tweets, which affect the preprocessing results and the performance of subsequent named entity recognition.

[0005] The objective of this invention is mainly achieved through the following technical solutions:

[0006] On the one hand, this invention provides a general method for preprocessing English tweets, which includes the following steps:

[0007] A subjective lexicon was constructed based on English texts from multiple fields.

[0008] The non-standard morphemes in the English tweets to be processed are semantically restored and information extracted to obtain the preprocessed tweet text.

[0009] Based on the preprocessed tweet text, a dual-stack structure is constructed for clause extraction; the clauses contained in the English tweet are obtained.

[0010] Based on the subjective vocabulary, the named entity extraction of the preprocessed tweet text is performed using a grammatical dependency analysis model and a tree-like parent-child structure to obtain the named entity recognition results of the English tweet.

[0011] Output the preprocessed tweet text, the clauses contained in the English tweet, and the named entity recognition results of the English tweet.

[0012] Furthermore, the process of restoring and extracting information from non-standard morphemes in the English tweets to be processed includes:

[0013] Semantic reconstruction of the tags in English tweets yields the semantic information contained in those tags;

[0014] Informal morpheme standardization was performed on the abbreviations included in English tweets to obtain the standard text corresponding to the abbreviations;

[0015] Correcting non-standard capitalized words to obtain standard word forms;

[0016] Sentence-level value information is extracted from English tweets to obtain the semantic information contained in the sentence-level parts of the English tweets.

[0017] Furthermore, the semantic restoration of tags in English tweets includes:

[0018] Extract tags from the English tweets to be processed to obtain a tag set;

[0019] Determine whether each tag in the tag set conforms to the generalized camelCase naming convention; if it does, perform semantic restoration based on regular expressions and output the semantic restoration result of the tag; if it does not conform, perform semantic restoration by a matching method that combines a dictionary and a greedy algorithm.

[0020] The semantic reconstruction method, which combines a fusion dictionary with a greedy algorithm for matching, includes:

[0021] The dictionary is used to search for words in the tweet tags. If a matching word is found in the dictionary, the word is returned.

[0022] If no matching result is found in the dictionary, the label semantics are restored based on a greedy search algorithm.

[0023] Furthermore, the informal morpheme standardization of the abbreviation also includes standardizing the digits 2 and 4 in a free state using a pre-trained 24-reduction model; the training of the 24-reduction model includes:

[0024] Obtain sentences with 2, 4, to, or for states that are in a free state, and construct a 24-fold restored dataset;

[0025] Replace 2, 4, to, and for in the dataset with the [MASK] label to obtain the processed 24 restored training dataset PData;

[0026] Based on the 24-restored training dataset PData, the BERT model was trained using binary cross-entropy to obtain the 24-restored model.

[0027] Furthermore, a pre-trained multilayer perceptron model is used to correct non-standard uppercase words. The training of the multilayer perceptron model includes:

[0028] Obtain formal and informal texts, and embed discrete feature vectors into each morpheme in each text; both formal and informal texts contain non-standard capitalized words; the discrete feature vectors include the morphological and part-of-speech features of the morphemes;

[0029] Radius sliding window downsampling and negative sampling are performed on the text after embedding discrete feature vectors to obtain the perceptron training dataset;

[0030] Based on the perceptron training dataset, a multilayer perceptron model is obtained by training a perceptron containing a binary classification fully connected layer using binary cross-entropy as the loss function.

[0031] Furthermore, the extraction of sentence-level value information from English tweets includes:

[0032] The tweet text to be processed is split into a beginning tag HS, a mixture of natural text and tags in the sentence MI, and an ending tag HE;

[0033] Constructing a one-dimensional conjugate cellular automaton based on word morphology and part of speech;

[0034] Set the forward evolution rule set SR, the backward evolution rule set ER, and the state set SS;

[0035] Based on the constraints of the forward evolution rule set SR and the backward evolution rule set ER, the one-dimensional conjugate cellular automaton evolves from MI to HS and HE, retaining words that satisfy the forward evolution rule set SR and the backward evolution rule set ER, and obtaining the sentence-end value information extraction result.

[0036] Furthermore, the clause includes the text content contained in paired punctuation marks and the content between the two sentence-ending punctuation marks; the clause extraction based on the preprocessed tweet text, constructing a dual-stack structure, includes:

[0037] Initialize the dual-stack structure used to store punctuation marks in sentences, including S pair and S ter , among which, S pair Used to store the paired punctuation marks on the left that have already been traversed, S terUsed to store the punctuation marks at the end of sentences that have been traversed;

[0038] Iterate through all the morphemes obtained by splitting the tweet to be processed according to spaces, and use the dual-stack structure to match them step by step into pairs of punctuation and sentence-end punctuation.

[0039] Extract the text content contained in the successfully matched punctuation pairs and the content between the two sentence-ending punctuation marks to obtain the clauses of the English tweet to be processed.

[0040] Furthermore, the subjective lexicon constructed based on English text from multiple domains includes:

[0041] Collect English texts from multiple fields and build a corpus;

[0042] The English text in the corpus is segmented and the word frequency is counted. Words with a word frequency less than a first preset threshold and a word length of 1 are removed to obtain an objective word list Objws.

[0043] The dictionary is composed of nouns, adverbs, adjectives, numbers, prepositions, conjunctions, and articles whose word frequency ranking is less than the second preset threshold and whose Collins star rating is greater than 1, thus obtaining the word list Trws.

[0044] The subjective vocabulary Subjws is obtained by removing words from the vocabulary Objws from the vocabulary Trws.

[0045] Furthermore, based on the subjective lexicon, the preprocessed tweet text is extracted using a grammatical dependency parsing model and a tree-like parent-child structure, including:

[0046] All noun phrases and noun clauses in the preprocessed tweet text were extracted using a grammatical dependency parsing model.

[0047] Iterate through all noun phrases, removing terminating articles, quantifiers, and stop words; and based on the subjective vocabulary, remove subjective words to obtain a preprocessed named entity set NP consisting of noun phrases. p

[0048] Based on the named entity set NP p The inclusion relationship of noun phrases is defined by using noun phrases containing at least one noun phrase and noun clauses as parent strings, and the noun phrases contained in the parent strings as child strings, thus constructing a tree-like parent-child structure;

[0049] Named entity extraction is performed based on the parent-child structure to obtain the named entity recognition results of the English tweet.

[0050] On the other hand, the present invention also provides a computer device, the computer device including at least one processor and at least one memory communicatively connected to said processor;

[0051] The memory stores instructions that can be executed by the processor to implement the aforementioned general English tweet preprocessing method.

[0052] The beneficial effects of this technical solution are:

[0053] 1. This invention utilizes a set of rules and algorithms summarized from the English social environment to remove, restore, replace, and disassemble information such as abbreviations, emoticons, punctuation, tags, and sets of mentioned users in English tweet text, thereby realizing the standardization of English tweet text from informal text to semi-formal text.

[0054] 2. This invention targets informal English tweet texts. Considering that tweets contain a large number of subjective words, which affect downstream tasks of natural language processing, it constructs two word lists using texts from knowledge websites such as Wikipedia that have been revised by multiple parties and have standardized word usage. These word lists represent subjective words and objective words respectively, which facilitates subsequent content filtering.

[0055] 3. This invention addresses the issue that noun phrases extracted by the Transformer-based grammatical dependency parsing model may contain unnecessary information (such as invalid conjunctions, prepositions, etc.) or various noun phrases and noun clauses. It filters invalid information through a multi-vocabulary and constructs a tree structure on the identified noun phrase set for recursive noun extraction, thereby improving the accuracy of named entity recognition.

[0056] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0057] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0058] Figure 1 This is a flowchart of a general English tweet preprocessing method according to an embodiment of the present invention. Detailed Implementation

[0059] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0060] This embodiment presents a general English tweet preprocessing method, such as... Figure 1As shown, it includes the following steps:

[0061] Step S1: Construct a subjective vocabulary based on English texts from multiple fields;

[0062] Specifically, the first step is to acquire English texts from multiple fields and build a corpus;

[0063] The English text in the corpus is segmented and the word frequency is counted. Words with a word frequency less than a first preset threshold and a word length of 1 are removed to obtain an objective word list Objws.

[0064] The dictionary is composed of nouns, adverbs, adjectives, numbers, prepositions, conjunctions, and articles whose word frequency ranking is less than the second preset threshold and whose Collins star rating is greater than 1, thus obtaining the word list Trws.

[0065] The subjective vocabulary Subjws is obtained by removing words from the vocabulary Objws from the vocabulary Trws.

[0066] Preferably, when constructing the corpus, named entities from multiple domains are first obtained and merged to obtain an initial entity list L. init Specifically, entities from different fields can be specified, including politics, agriculture, information technology, religion, film and entertainment, video games, education, medicine, etc., or entities that have recently received the most public attention can be listed based on Google Index.

[0067] Further obtain the initial entity list L init For each entity, obtain its corresponding entity ID to get a list of entity IDs. Preferably, DBpedia's SPARQL interface can be used to query the corresponding WikiPageID for each entity, forming a list of entity IDs.

[0068] It should be noted that, compared to informal tweets, texts on knowledge websites such as Wikipedia are more formal texts that have undergone multiple revisions and use standardized terminology, which facilitates the subsequent training of deep learning models. At the same time, considering that tweets may use a large number of subjective words, which may affect downstream tasks of natural language processing, two vocabularies will be constructed to represent subjective words and objective words respectively, which will facilitate the subsequent content filtering.

[0069] Further utilize DBpedia's SPARQL interface to obtain the entity ID list. For each entity ID, there are other entity IDs that have a relationship with the entity (e.g., "Capital" and another associated entity "Beijing" that are related to the entity "China"), and a list of entity IDs. The entity IDs corresponding to other entities of the same category as each entity are merged to obtain a list of entity IDs. Specifically, a category can be the Wikipedia category to which each entity is assigned, an attribute of a Wikipedia entity, originally called Category in Wikipedia.

[0070] by replace Get the list of entity IDs For each entity ID, list the entity IDs of other entities that have a relationship with that entity, and a list of entity IDs. The entity IDs corresponding to other entities of the same category as each entity are merged to obtain a list of entity IDs. The entity ID list constructed using the method described in this embodiment The total number of entities involved is approximately 130,000.

[0071] Further access The corpus, denoted as Abs, is constructed based on the summaries of the entity pages corresponding to all entity IDs.

[0072] After constructing the corpus, the abstracts in the corpus Abs are segmented and the word frequency is counted. Words with a word frequency less than a first preset threshold and a word length of 1 are removed to obtain the objective word list Objws. Specifically, in this embodiment, the first threshold is set to 300. Words with a word frequency less than 300 and a word length of 1 are removed to form the objective word list Objws.

[0073] Further, nouns, adverbs, adjectives, numbers, prepositions, conjunctions, and articles in the ECDICT dictionary whose word frequency ranking is less than the second preset threshold and whose Collins star rating is greater than 1 are obtained to form the word list Trws. In this embodiment, the second threshold is set to 10000, that is, the word list Trws contains commonly used words whose word frequency ranking is less than 10000 and whose Collins star rating is greater than 1.

[0074] By removing words from the vocabulary Objws from the vocabulary Trws, the remaining words are mostly words expressing subjective feelings such as nice and beautiful, thus constructing the subjective vocabulary Subjws.

[0075] Step S2: Perform semantic restoration and information extraction on the non-standard morphemes in the English tweets to be processed to obtain the preprocessed tweet text;

[0076] Specifically, the process involves restoring and extracting information from non-standard morphemes in English tweets, including:

[0077] Semantic reconstruction of the tags in English tweets yields the semantic information contained in those tags;

[0078] Informal morpheme standardization was performed on the abbreviations included in English tweets to obtain the standard text corresponding to the abbreviations;

[0079] Correcting non-standard capitalized words to obtain standard word forms;

[0080] Sentence-level value information is extracted from English tweets to obtain the semantic information contained in the sentence-level parts of the English tweets.

[0081] Specifically, existing methods can be used to preprocess tweets, including text standardization, text segmentation, text cleaning, and word standardization. For English tweets, text preprocessing can also involve regularization, removal, and restoration of morphemes with specific formats.

[0082] In this embodiment, considering the characteristics of English tweets, the preprocessing process includes semantic standardization, informal morpheme standardization, correction of non-standard capitalization, and extraction of sentence-level value information to obtain a well-structured standard tweet text. Specifically, the tweet preprocessing can be performed through the following steps:

[0083] Step S301: Perform semantic reconstruction on the tags of the English tweets to obtain the semantic information contained in the tags.

[0084] Specifically, hashtags in tweets refer to text segments starting with "#", such as "#BlackLivesMatter". These hashtags are characterized by words that are typically combined and do not contain spaces or punctuation. To address the issue of poor readability of hashtags in tweets, a "semantic restoration method based on a composite strategy of Generalized Camel Case, dictionary, or greedy search algorithm" is used to restore the semantic meaning of the hashtags. That is, if a hashtag conforms to the Generalized Camel Case (GCC) naming convention, its semantic restoration is performed according to its naming rules; otherwise, a composite strategy of "matching with a dictionary and a greedy algorithm" is used for semantic restoration.

[0085] Preferably, the tags in the tweets are first extracted and a tag set is formed. Each tag is then checked to see if it conforms to generalized camelCase naming convention. If it does, semantic reconstruction is performed using regular expressions, and the semantic reconstruction result is output. If it does not conform, semantic reconstruction is performed using a combination of dictionary and greedy algorithm matching.

[0086] CamelCase is a phrase naming convention that does not include spaces or punctuation marks. Its applicability is extended to phrases containing numbers, ordinal numbers, or consecutive words of different capitalizations. Its syntax is shown in Table 1.

[0087] Table 1. Syntax table of the generalized camel-cup naming convention

[0088]

[0089]

[0090] This embodiment uses the following regular expression to match the syntax rules of all generalized camelCase algorithms and performs semantic restoration on tags that conform to the "generalized camelCase naming convention".

[0091] (?P) <content>[AZ]? [az]+|[AZ]+(?=[AZ][az]+|$|\d)|\d+(?:st|nd|rd|th)? |(?<=[az])[AZ][az]+? (?=[^az]|$)).

[0092] Furthermore, matching methods that combine dictionaries and greedy algorithms include:

[0093] First, a word search is performed using the ECDICT dictionary. If no match is found in the ECDICT dictionary, WordNinja, based on a greedy search algorithm, is used to break down the word. Specifically:

[0094] First, construct a fuzzy query statement, namely "SELECT * FROM stardict WHERE sw Like hashtag", where "hashtag" is the target tag, and query whether the tag exists in the ECDICT dictionary;

[0095] If the current tag cannot match any ECDICT record, then WordNinja is used to restore the tag semantics, which is essentially a maximum greedy matching algorithm on a known vocabulary.

[0096] If the current tag matches a word in the ECDICT dictionary, the algorithm directly queries the ECDICT dictionary to determine if the current tag consists of only one word; if so, it returns that word. Otherwise, it uses a heuristic Gestalt pattern matching algorithm to calculate the similarity between the current tag and all retrieved ECDICT records, and then returns the most similar phrase. Specifically, if the current tag consists of only one word, the corresponding word will be matched; if the current tag is a phrase, there will be no matching results. Also, tweet tags do not contain spaces. For example, if the current tag is "SameAs," although the term "same as" exists in ECDICT, it will not be matched because there is no space between "Same" and "As" in the current tag. Similarly, if the current tag contains only one word, such as "same," the corresponding term will be matched.

[0097] After semantic restoration of the tags, further operations can be performed on the English tweets to be processed, such as extracting secondary morphemes, restoring special semantic punctuation, and separating word-terminal punctuation, to extract semantic information contained in other non-standard format morphemes, including:

[0098] (1) Extracting secondary morphemes:

[0099] Secondary morphemes in tweets refer to morphemes such as emoticons, kaomojis, and interjections that express the author's mood or expression habits, as well as the "@username" syntax that indicates mentioning a user;

[0100] Since "@username" is a tweet syntax, its structure does not provide semantic information; therefore, for the "@username" syntax, a string is randomly generated, starting with an uppercase letter and followed by 5 random lowercase letters, as the nickname of the mentioned user.

[0101] Furthermore, tweets scraped from Twitter require HTML encoding, first being converted to UTF8 or other encoding formats—a process known as decoding or transcoding. However, some HTML text may fail to convert during this process, necessitating handling. For text with decoding errors in the tweets, the replacement content is determined based on the decoding table and then replaced; for example, "&" needs to be replaced with "and". Invalid morphemes such as "RT" indicating a reply, multimedia links, and email addresses are removed using appropriate regular expressions. Invalid morphemes refer to English tweets that do not provide valid information, contain errors, or are non-English language elements.

[0102] (2) Restoring special semantic punctuation marks, including:

[0103] Remove the square brackets "[]"; restore the equals sign "=" to "equals to";

[0104] For a tilde "~" or a hyphen "-" followed by a person's name, the following regular expression can be used to retrieve the tilde "~" and hyphen "-" that may follow a person's name:

[0105] (?<=(?P <ispunc>[^\s])){0,1}(?P <pres>\s{0,})(?P [-\～])(？＝(？P <lasts>\s{0,})(?P <name>([AZ][^\s]+\s?)+));

[0106] Then, a special judgment is made on the hyphen: if the punctuation being judged is a hyphen and there are no spaces before or after it, and the preceding element is not a punctuation mark, then it is regarded as a word-forming hyphen rather than a hyphen with special semantics.

[0107] Finally, if the punctuation mark (referring to the tilde and hyphen) is followed by a person's name, the restored content also needs to take into account the contextual elements of the tilde and hyphen, as shown in Table 2.

[0108] Table 2 shows the restored content based on the relative position of the tilde or hyphen to other morphemes in the sentence.

[0109]

[0110] For a tilde "~" adjacent to a number, use the following regular expression to retrieve the tildes that may be followed by numbers:

[0111] (?P) <prefigure>(?:\d|_NUM)){0,1}\s{0,}(?P <target>\~)\s{0,}(?=(?P <lastfigure>(?:\d|_NUM))).

[0112] Here, _NUM represents possible numeric words.

[0113] If there is no number before the tilde but a number after it, replace it with "approximately"; if both the tilde and the number are numbers, replace it with "to".

[0114] (3) Separate punctuation marks at the beginning of words:

[0115] Word-terminal punctuation refers to punctuation marks at the beginning and end of a word, used to facilitate subsequent clause extraction. Spaces are added on both sides of punctuation marks before and after words (word-terminal punctuation). Furthermore, since possessive plural nouns in English (such as "sisters'" in "Mysisters'hair") are matched by the aforementioned rules, single quotes "'" in words that might represent possessive plural nouns ("z'" and "s'") are replaced with "@#" as placeholders to prevent them from being split. After all punctuation is processed, "@#" is replaced back with the original "'".

[0116] After semantic standardization using the method in this embodiment, the semantic information of tweet tags, minor morphemes, special semantic punctuation, and word-terminal punctuation can be restored, avoiding the problem that non-standard morphemes that may contain semantics are filtered out, resulting in incomplete tweet content obtained in subsequent processing.

[0117] Step S202: Perform informal morpheme standardization on the abbreviations included in the English tweets to obtain the standard text corresponding to the abbreviations; in this embodiment, a multi-source English vocabulary and BERT are used in conjunction to perform informal morpheme standardization.

[0118] This step primarily handles four types of abbreviations: common abbreviations with sentence-ending punctuation (including geographical, date, and physiological or medical categories), Latin abbreviations, prepositional abbreviations, and combination abbreviations of "person (pronoun) - modal verb or auxiliary verb". Except for "common abbreviations with sentence-ending punctuation", which were constructed using the ECDICT dictionary, the other abbreviation types were collected and organized from Wikipedia; furthermore, all matching was completed using "free element regular expressions".

[0119] Specifically, a free element is an element that is in a free state and meets one of the following characteristics:

[0120] a) The element is preceded by a space and followed by the end of the sentence;

[0121] b) The element is preceded by the beginning of a sentence and followed by a space;

[0122] c) The element is preceded and followed by spaces;

[0123] d) This sentence contains only the target element.

[0124] The regular expression for a free element is: (?<=\s)k$|^k(?=\s)|(?<=\s)k(?=\s)|^k$.

[0125] Specifically, in the process of restoring prepositional abbreviations, the free-floating "2" and "4" are restored to "to" and "for," respectively. However, in some contexts, "2" and "4" do not reflect the part of speech of prepositions but simply indicate quantity. To avoid the above-mentioned incorrect restoration problem, this embodiment uses the BERT model to determine whether the free-floating "2" and "4" in the given text need to be restored to the prepositions "to" and "for." That is, the problem is transformed into a binary classification problem of "whether to restore," referred to as the "24 restoration problem."

[0126] Preferably, this embodiment first utilizes a clause extraction algorithm based on a dual-stack structure to segment all summaries in Abs into sentences, retaining only sentences containing the free "2", "4", "to", or "for", and replacing the "2", "4", "to", or "for" with the [MASK] tag. Simultaneously, [CLS] and [SEP] tags are added at the beginning and end of each sentence. The processing result is denoted as PData.

[0127] Based on the "BERT base model (cased)", add a fully connected layer with a binary classification SoftMax structure;

[0128] To fine-tune the pre-trained model, PData is first divided into training, validation, and test sets in a 6:2:2 ratio; BERT weights are frozen; the embedding vector of [MASK] output by BERT is input into a fully connected layer with a binary classification SoftMax structure, and the binary cross-entropy is calculated. After loss iteration, the 24-fold restored model is obtained.

[0129] The English tweet to be processed is input into the trained 24 reconstruction model, which performs judgment and reconstruction on 2 and 4 to obtain the standard representation of 2 and 4.

[0130] Step S203: Correct the non-standard capitalized words to obtain the standard word form;

[0131] Specifically, firstly, formal and informal texts are obtained, and discrete feature vectors are embedded for each morpheme in each text; both formal and informal texts contain non-standard capitalized words; the discrete feature vectors include the morphological and part-of-speech features of the morphemes;

[0132] Then, radius sliding window downsampling and negative sampling are performed on the text after embedding discrete feature vectors to obtain the perceptron training dataset;

[0133] Based on the aforementioned perceptron training dataset, a multilayer perceptron model is obtained by training a perceptron containing a fully connected layer for binary classification using binary cross-entropy as the loss function.

[0134] Preferably, this embodiment extracts a large-scale text dataset from Wikipedia as the formal text dataset; at the same time, in order to increase the model's generalization ability to informal text, the IMDB movie review dataset, which contains a large amount of informal text, is also merged into the formal text dataset to expand the training set's ability to process informal text.

[0135] Furthermore, discrete morpheme embedding features are obtained. Specifically, this embodiment draws on the human approach of judging the correct form of a word in a sentence, which mainly considers factors such as the capitalization of the word and its neighboring words, as well as the word's common parts of speech. This embodiment uses the following features as the embedding vectors for each morpheme in the tweet:

[0136] Dimension 1: The id of this morpheme in ECDICT (starting from 1, 0 indicates an out-of-vocabulary word)

[0137] Dimension 2: The morpheme's form, including: containing only numbers, all lowercase, all uppercase, initial capitalization, partial capitalization within a word, containing only sentence-final punctuation, containing only paired punctuation, containing only connecting punctuation, containing only other punctuation, and various mixed and empty placeholders.

[0138] Dimensions 3, 4, and 5: The three most frequently used parts of speech for this morpheme in ECDICT (0 indicates that the value is empty). When the part of speech of this element is not equal to three, such as apple which only has the noun part of speech, the last two dimensions are empty.

[0139] Among them, "common words" refers to the top 8000 words in the contemporary corpus provided by ECDICT in terms of word frequency or words with a Collins star rating of at least 0. "Morphemes" refers to various linguistic elements such as words (classification criteria may include part of speech, word frequency, etc.), punctuation marks, and numbers. Connecting punctuation marks generally indicate contextual relationships in text, including the ampersand "&", hyphen "-", colon ":", equals sign "=", underscore "_", or sign "|", and tilde "~". Other punctuation marks refer to punctuation marks in the ASCII punctuation set other than "sentence-ending punctuation", "paired punctuation", and "connecting punctuation".

[0140] Furthermore, since lowercase words constitute the majority of normal English text, and the existing training corpus is mostly long formal or semi-formal text, directly using the original corpus data as model input would result in a bias towards classifying the target as normal text, leading to label imbalance. Therefore, this embodiment employs a composite sampling method combining radius sliding window downsampling and negative sampling to obtain the perceptron training dataset, denoted as Sam={s1,s2,…,s…}. n }

[0141] Specifically, the downsampling steps include: using words with irregular capitalization patterns as core morphemes and a radius of WR (set to 1 in this embodiment), introducing their neighboring morphemes as training data. Irregular capitalization patterns include words with full capitalization, initial capitalization, and partial capitalization. For example, in the sentence "He was ORDERED to leave Russia.", with "ordered" as the center and a radius of 1, the resulting training data is "was ORDERED to". If there are not enough morphemes within the WR radius of the core morpheme, an empty string is used instead. Furthermore, if the ranges of clauses truncated from multiple core morphemes overlap, only the first and last endpoints of all overlapping ranges are considered, while all intermediate overlapping ranges are merged. For example, in the sentence "I really love to SWIM in river", the two training corpora should be centered on "LOVE" and "SWIM" with a radius of 1. The two training corpora are "really love to" and "to SWIM in". In this case, since "to" appears repeatedly in both corpora, the two corpora are merged into one corpus: "really love to SWIMin".

[0142] Negative sampling includes constructing negative examples on each clause obtained from downsampling to enhance the model's ability to fit negative examples, specifically including:

[0143] Simply keep the original sentence without modification; or,

[0144] Randomly convert a completely lowercase word in the sentence to completely uppercase according to a certain number (the default value is 30% of the current input sentence's word count); or,

[0145] Convert all lowercase words to all uppercase or initial capitalization. The conversion target depends on the pattern of the adjacent words containing uppercase letters of the current lowercase word, converting them to the opposite or the same pattern with a 50% probability.

[0146] Furthermore, the perceptron training dataset was divided into training, validation, and test sets in a 6:2:2 ratio; then, a classification multilayer perceptron was constructed and trained using binary cross-entropy as the loss function. The multilayer perceptron includes an embedding layer, hidden layers, and an output layer.

[0147] Embedding layer: Used to convert the discrete features of each morpheme in each clause of the perceptron training dataset Sam into continuous vector features. For training clauses s i ∈Sam, for each morpheme {w1,w2,…,w} in the clause n }∈s i Embedding (i.e., v) j =Embedding(w j After concatenating the first and last clauses of the clause s, we get the current clause s. i Hidden representation (in (This represents a vector concatenation operation);

[0148] Hidden layers: Consisting of 5 fully connected layers, each layer (denoted as l) uses ELU as the activation function, and increases the slope of the negative half-axis layer by layer to improve the sensitivity of neurons to negative values. Dropout is added for regularization according to the dropout rate d. The formal definition of the hidden representation of each layer is as follows:

[0149] H l+1 =Dropout(ELU(WH l +b,0.05*l),d)

[0150] Output layer: Used to output binary classification probabilities.

[0151] This step converts non-standard uppercase words in the English tweets to standard lowercase or initial capitalization.

[0152] Step S204: Extract sentence-level value information from the English tweet to obtain the semantic information contained in the sentence-level of the English tweet;

[0153] Specifically, the tweet text to be processed is split into the beginning tag HS, the natural text and tags mixed in the sentence MI, and the ending tag HE;

[0154] Constructing a one-dimensional conjugate cellular automaton based on word morphology and part of speech;

[0155] Set the forward evolution rule set SR, the backward evolution rule set ER, and the state set SS;

[0156] Based on the constraints of the forward evolution rule set SR and the backward evolution rule set ER, the one-dimensional conjugate cellular automaton evolves from MI to HS and HE, retaining words that satisfy the forward evolution rule set SR and the backward evolution rule set ER, and obtaining the sentence-end value information extraction result.

[0157] Preferably, in this embodiment, the tweet text to be processed after steps S201-S203 is divided into three segments: the initial tag set HS, the mixture of natural text and tags in the sentence MI, and the ending tag HE; wherein, the first word of MI is the next non-tag text after HS, and the last word of MI is the previous non-tag text after HE. After obtaining the three segments of data, the process proceeds forward and backward starting from MI.

[0158] Specifically, for a one-dimensional evolution from MI to HS from right to left, let the current cell index be HS. index Each cell's neighbors are the adjacent words in HS. A forward evolution rule set SR is set, and the state set SS = {truncated, retained}. The specific contents of the forward evolution rule set SR are as follows (they are related by "OR"):

[0159] (1) The last word or the last phrase of HS is a continuation word; (Use the AC automaton built based on continuation words to match the longest continuation word)

[0160] (2) The last word in MI is a preposition;

[0161] (3) The first word of MI begins with a lowercase letter;

[0162] (4) According to ECDICT, the two words adjacent to HS and MI are not common words;

[0163] (5) Merge the two words adjacent to HS and MI and query ECDICT to see if there is a corresponding compound word.

[0164] Here, "continuing words" refers to the set of prepositions (phrases), conjunctions (phrases), modal verbs, and auxiliary verbs. The continuing words are obtained by matching the longest continuing word using an AC automaton built upon them. For example, building an AC automaton on the set {as, as well as} ensures that when matching "Mary as well as John", it matches "as well as" instead of "as".

[0165] When the automaton evolves under the constraints of the forward evolution rule set SR, the "truncated" and "preserved" states in the state set SS can be specifically expressed as follows: if the current cell does not satisfy SR, then HS is truncated to HS[HS]. index :] and terminate the automaton operation. If the forward evolution rule set SR is satisfied, then HS in HS will be... index The corresponding element is kept at the beginning of the MI.

[0166] Furthermore, in the left-to-right one-dimensional evolution from MI to HE, the backward evolution rule set ER and the forward evolution rule set SR are conjugate, and their contents are as follows (they are "OR" related to each other):

[0167] (1) The last word of MI is a successor word (using the AC automaton built based on successor words to match the longest successor word);

[0168] (2) The first word in HE is a preposition;

[0169] (3) HE begins with a lowercase letter and MI does not end with a sentence punctuation mark;

[0170] (4) According to ECDICT, the two words adjacent to HE and MI are not common words;

[0171] (5) Merge the two words adjacent to HE and MI and query ECDICT to see if there is a corresponding compound word.

[0172] By evolving from MI to HS and HE using automata, words that satisfy the forward evolution rule set SR and the backward evolution rule set ER are retained to obtain the sentence-end value information extraction results.

[0173] Step S3: Based on the preprocessed tweet text, construct a dual-stack structure to extract clauses; obtain the clauses contained in the English tweet.

[0174] If the last element of the sentence (which can be a letter or a punctuation mark) is not a final punctuation mark, then add a "." at the end of the sentence; initialize two stack structures S to store punctuation marks in the sentence. pair and S ter , respectively representing "the left-side paired punctuation marks that have been visited" and "the sentence-ending punctuation marks that have been visited";

[0175] Then iterate through all the elements obtained by splitting the current sentence according to spaces (including words and punctuation marks, etc.):

[0176] If the current element is a sentence-ending punctuation mark, then compare S. pair and S ter The index of the last element in S is used to determine whether the current clause should be truncated starting from the last punctuation mark of the previous sentence or from the last paired symbol; then, the index of the last element in S is used to determine whether the clause should be truncated starting from the last punctuation mark of the previous sentence or from the last paired symbol. ter Record the current index as a candidate starting index for the next sentence-ending punctuation mark;

[0177] If the current S ter Not empty and the current element is S ter If the last element has a paired punctuation mark to its right, the clause is extracted from the previous paired punctuation mark to the end of the current element; then, it is checked whether the last punctuation mark of the previous clause is within the range of the current paired punctuation mark pair. If so, an S is popped. pair and S ter The last element, otherwise only S is popped. pair The last element;

[0178] If the current element is a left-paired punctuation mark, then move to S. pair Push the index of the current element;

[0179] In this process, each clause is replaced with a random nickname to avoid duplicate matching.

[0180] After the above processing, any unmatched paired punctuation marks in the sentence will be removed directly.

[0181] After the aforementioned preprocessing steps, the standard tweet text is obtained.

[0182] Step S4: Based on the subjective vocabulary, name entity extraction is performed on the preprocessed tweet text using a grammatical dependency parsing model and a tree-like parent-child structure to obtain the name entity recognition results of the English tweet;

[0183] Specifically, the standard tweet text (which may include clauses from the original tweet text and the complete tags before execution 0) is first input into a Transformer-based grammatical dependency parsing model trained on a large corpus to analyze the part-of-speech of each word and its part-of-speech in the sentence. Based on the grammatical dependency parsing results, noun phrases and noun clauses involved in the tweet are extracted and denoted as np. i Let i ∈ {1, 2, ..., n}, where n is the number of noun phrases, and let NP = {Np1, Np2, ..., Np} be the set of all extracted noun phrases and noun clauses. i …,Np n The noun phrases and noun clauses extracted through the grammatical dependency analysis model can include longer noun phrases containing multiple named entities or noun clauses containing multiple noun phrases.

[0184] Furthermore, based on a subjective vocabulary, the noun phrases and noun clauses in set NP are preprocessed to construct set NP. p This includes: traversing all noun phrases, removing terminating articles, quantifiers, and stop words; and based on a subjective vocabulary, removing subjective words to obtain a preprocessed set NP consisting of noun phrases and noun clauses. p .

[0185] Specifically, for terminating articles: match whether the word at the beginning of the phrase is "a", "an", or "the"; if so, remove it.

[0186] For quantifiers: use the following regular expression to match and then remove them: (?i:a|an|\d+|that|this|these|those|_NUM)[az]\w+\s(?i:of)\s;

[0187] Where _NUM represents possible numeric words, such as:

[0188] three|quarter|three|quarters|two|thirds|one|two|first|last|three|next|million|four|five|second|six|third|billion|hundred|thousand|se ven|eight|ten|nine|dozen|fourth|twenty|fifth|thirty|fifteen|fifty|twelve|sixth|forty|seventh|eleven|eighth|zero|twentieth|ninth|nine teenth|trillion|sixteen|eighteen|fourteen|sixty|thirteen|seventeen|eighty|tenth|nineteen|seventy|eighteenth|ninety|seventeenth|sixte enth|fourteenth|twelfth|fifteenth|eleventh|thirteenth|hundredth|fiftieth|thirtieth|fortieth|sixtieth|seventieth|eightieth|ninetieth.

[0189] For subjective words: Based on the subjective vocabulary Subjws, use detached element regular expressions to remove all subjective words present in the phrase;

[0190] (?<=\s)k$|^k(?=\s)|(?<=\s)k(?=\s)|^k$;

[0191] Here, k can be replaced with any piece of free text that you want to match, such as "hello", "#", etc.

[0192] Specifically, a detached element is an element that meets one of the following four characteristics (or the current element is in a "detached state"):

[0193] a) The element is preceded by a space and followed by the end of the sentence;

[0194] b) The element is preceded by the beginning of a sentence and followed by a space;

[0195] c) The element is preceded and followed by spaces;

[0196] d) This sentence contains only the target element (i.e., the "free element" that needs to be matched).

[0197] For stop words: Since the stop words contain multiple words, such as "you" and "yourself", an AC automaton is constructed based on the stop word list, and the longest match in a noun phrase is removed. For example, if both "you" and "yourself" are matched, then "yourself" is removed. Based on the characteristics of English tweets and the task of this embodiment, the stop word list constructed includes:

[0198] it,its,itself,i,me,my,myself,we,us,our,ours,ourselves,ye,u,you,your,ur,yours,urs,yourselves,yourself,thyself,thine,thee,thou,he,him,his,himself,she,her,herself,they,them,t heir,theirselves,diz,this,that,these,those,there,here,thing,other,another,others,some,something,someday,somewhere,somehow,someone,somebody,sometime,sometimes,somewhat,ever y,everything,everywhere,everyone,everybody,everyday,any,anything,anyone,anybody,anyway,anymore,no,nothing,none,null,nowhere,nobody,just,mere,each,many,much,more,most,who,w hom,what,where,which,how,whoever,whomever,whosoever,whomsoever,yes,such,not,do,don,dont,does,doesn,doesnt,did,didn,didnt,true,false,is,are,was,were,be,have,has,had,having.

[0199] Step S5: Based on the set NP p The noun phrases and noun clauses in the text are used to construct a tree-like parent-child structure and perform named entity extraction to obtain the named entity recognition results of the English tweet.

[0200] Because set NP p The output may contain long clauses such as noun clauses or appositive clauses. That is, a noun phrase may contain multiple noun phrases at the same time. Therefore, it is necessary to establish a tree-like parent-child relationship between noun phrases and noun clauses on the set of output noun phrases and noun clauses, and then perform hierarchical structured analysis.

[0201] Specifically, based on the inclusion relationship between noun phrases and noun clauses, a tree-like parent-child structure is constructed, with noun phrases and noun clauses containing at least one noun phrase as the parent string, and the noun phrases contained in the parent string as the child strings; that is, using the set NP... p Construct a tree-like parent-child structure set NP of all noun phrases and noun clauses. t If a noun phrase has multiple parent strings, then the longest parent string is taken as its parent node, and all NP-complete strings are denoteed. t The set of all parent strings in the set is FP. t ={fp1,fp2,…,fp n }

[0202] Based on the possessive structure of nouns in each parent string, extract its core nouns and save them to a named entity set. Specifically, the following regular expression is used to identify NP. t Extract the possessive noun structures from each parent string and then extract their core nouns.

[0203] ([A-Z0-9]\w+\s)*[\w^('s|'z|z'|s')]*(([s|z](?='))|(?='s|'z));

[0204] That is, based on the noun possessive structure, starting from "'", traverse left to the first capitalized word after the previous "'"; for the XX YY's and XX YY'z structures, extract the first capitalized word to YY to obtain the core noun possessive; for the XX YYs' and XX YYz' structures, extract the first capitalized word to YYs or YYz to obtain the core noun possessive, and save it to the named entity set. Where X and Y represent any word.

[0205] Furthermore, using a regular expression for detached elements, all substrings in each parent string are removed, and the remaining content is merged into a new string. The set of all substrings is denoted as cp. In practical applications, this embodiment sorts the substrings by length and removes them in descending order of length, prioritizing the removal of substrings with more information.

[0206] If the string If the corresponding preset conditions are met, the string will be... Save to Named Entity Collection Otherwise, based on the string All noun phrases and noun clauses are extracted using a grammatical dependency parsing model, and the extracted noun phrases and noun clauses are saved to a named entity set. Finally, the substring set cp is merged into the named entity set. middle.

[0207] Named entity collection Replace set NP p The tree-like parent-child structure is reconstructed for named entity extraction; preferably, the tree-like parent-child structure can be reconstructed multiple times for named entity extraction to obtain named entity recognition results; wherein, the set of named entities obtained by reconstructing the tree-like parent-child structure for named entity extraction in the j-th time is denoted as . j is an integer greater than 1; the (j-1)th time the tree-like parent-child structure is reconstructed for named entity extraction, and the resulting set of named entities is denoted as . When set With sets When the difference is an empty set, the set will be... The output is the named entity recognition result of the English tweet.

[0208] Among them, string The preset conditions include: string The string contains only one word besides the included substring, and that word is a preposition or conjunction. The parent string has two substrings, the absolute value of the difference in length between any two substrings is less than or equal to 2, and the length of each substring is less than or equal to 3. If the preset conditions are met, then the string will be... Save to Named Entity Collection Preferably, if the string If the corresponding preset conditions are not met, then the string can be judged. Does it meet the following criteria? It is a non-empty string and the Not all morphemes in the text are invalid information (i.e., prepositions, conjunctions, punctuation, subjective words, stop words, and articles). If they are, then the grammatical dependency parsing model is used again to extract noun phrases and noun clauses, and the extracted noun phrases and noun clauses are saved to the named entity set. The tree-like parent-child structure is reconstructed and named entity extraction is performed; if it does not meet the requirements, the string can be directly removed.

[0209] Step S5: Output the preprocessed tweet text, the clauses contained in the English tweet, and the named entity recognition results of the English tweet as general English tweet preprocessing results.

[0210] In summary, the embodiments of this invention provide a general English tweet preprocessing method. This method utilizes text from knowledge-based websites such as Wikipedia, which have undergone multiple revisions and adhere to standardized terminology, to construct two vocabularies, representing subjective and objective words respectively. This addresses the problem of a large number of subjective words in informal English tweet texts, which negatively impact downstream tasks in natural language processing. Furthermore, it employs a set of rules and algorithms summarized from English social environments to remove, restore, replace, and disassemble information such as abbreviations, emoticons, punctuation, tags, and mentioned user sets in English tweet texts, thus standardizing the text from informal to semi-formal. Finally, to address the issue that named entities extracted by the Transformer-based dependency parsing model may include unnecessary information (such as invalid conjunctions and prepositions) or contain various noun phrases and noun clauses, invalid information is filtered through multiple vocabularies. Simultaneously, a tree-structured approach is constructed on the identified named entity set for recursive noun extraction, improving the accuracy of named entity recognition.

[0211] In another embodiment of the present invention, a computer device is provided, including at least one processor and at least one memory communicatively connected to the processor; the memory stores instructions executable by the processor for implementing the general English tweet preprocessing method of the foregoing embodiments.

[0212] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0213] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.< / lastfigure> < / target> < / prefigure> < / name> < / lasts> < / pres> < / ispunc> < / content>

Claims

1. A general method for preprocessing English tweets, characterized in that, Includes the following steps: A subjective lexicon was constructed based on English texts from multiple fields. The process involves semantic restoration and information extraction of non-standard morphemes in English tweets to obtain preprocessed tweet text. This includes: semantic restoration of tags in English tweets to obtain the semantic information contained in the tags; standardization of informal morphemes in abbreviations in English tweets to obtain the standard text corresponding to the abbreviations; correction of non-standard capitalized words to obtain standard word forms; and extraction of sentence-level value information from English tweets to obtain the semantic information contained in the sentence level of the English tweets. The extraction of sentence-level value information from English tweets includes: splitting the tweet text to be processed into beginning tags. A mixture of natural text and tags located in a sentence and ending tags A one-dimensional conjugate cellular automaton is constructed based on word morphology and part-of-speech; a forward evolution rule set is set. Backward evolution rule set and state set Based on forward evolution rule set Backward evolution rule set The constraints, obtained through the one-dimensional conjugate cellular automaton, are derived from... Towards and Evolution is carried out, preserving the set of rules that satisfy forward evolution. and backward evolution rule set The words are used to extract sentence-level value information. Based on the preprocessed tweet text, a dual-stack structure is constructed to extract clauses, thereby obtaining the clauses contained in the English tweet; Based on the subjective vocabulary, the named entity extraction of the preprocessed tweet text is performed using a grammatical dependency analysis model and a tree-like parent-child structure to obtain the named entity recognition results of the English tweet. The preprocessed text of the tweet, the clauses contained in the English tweet, and the named entity recognition results of the English tweet are output to obtain the preprocessed result of the English tweet.

2. The general English tweet preprocessing method according to claim 1, characterized in that, The semantic restoration of tags in English tweets includes: Extract tags from the English tweets to be processed to obtain a tag set; Determine whether each tag in the tag set conforms to the generalized camelCase naming convention; if it does, perform semantic restoration based on regular expressions and output the semantic restoration result of the tag; if it does not conform, perform semantic restoration by a matching method that combines a dictionary and a greedy algorithm. The semantic reconstruction method, which combines a fusion dictionary with a greedy algorithm for matching, includes: The dictionary is used to search for words in the tweet tags. If a matching word is found in the dictionary, the word is returned. If no matching result is found in the dictionary, the label semantics are restored based on a greedy search algorithm.

3. The general English tweet preprocessing method according to claim 1, characterized in that, The informal morpheme standardization of abbreviations included in English tweets also includes standardizing the digits 2 and 4 that are in a free state using a pre-trained 24-reduction model. The training of the 24-restore model includes: Obtain sentences with 2, 4, to, or for states that are in a free state, and construct a 24-fold restored dataset; Replace 2, 4, to, and for in the dataset with... Labels, and obtain the processed 24-fold restored training dataset. ; Based on 24-fold restoration of the training dataset The BERT model was trained using binary cross-entropy to obtain a 24-restored model.

4. The general English tweet preprocessing method according to claim 1, characterized in that, The non-standard capitalization of words is corrected using a pre-trained multilayer perceptron model. The training of the multilayer perceptron model includes: Obtain formal and informal texts, and embed discrete feature vectors into each morpheme in each text; both formal and informal texts contain non-standard capitalized words; the discrete feature vectors include the morphological and part-of-speech features of the morphemes; The text after embedding discrete feature vectors is subjected to radius sliding window downsampling and negative sampling to obtain the perceptron training dataset; Based on the perceptron training dataset, a multilayer perceptron model is obtained by training a perceptron containing a binary classification fully connected layer using binary cross-entropy as the loss function.

5. The general English tweet preprocessing method according to any one of claims 1-4, characterized in that, The clause includes the text contained in the paired punctuation marks and the content between the two sentence-ending punctuation marks; The process of extracting clauses using a dual-stack structure based on the preprocessed tweet text includes: Initialize the dual-stack structure used to store punctuation marks in sentences, including and ,in, Used to store the paired punctuation marks on the left that have already been traversed. Used to store the punctuation marks at the end of sentences that have already been traversed; Iterate through all the morphemes obtained by splitting the tweet to be processed according to spaces, and use the dual-stack structure to match them step by step into pairs of punctuation and sentence-end punctuation. Extract the text content contained in the successfully matched punctuation pairs and the content between the two sentence-ending punctuation marks to obtain the clauses of the English tweet to be processed.

6. The general English tweet preprocessing method according to any one of claims 1-4, characterized in that, The subjective lexicon, constructed from English texts across multiple domains, includes: Collect English texts from multiple fields and build a corpus; The English text in the corpus is segmented and its word frequency is counted. Words with a word frequency less than a first preset threshold and a word length of 1 are removed to obtain an objective vocabulary. ; The dictionary is compiled by retrieving nouns, adverbs, adjectives, numbers, prepositions, conjunctions, and articles whose frequency ranking is below a second preset threshold and whose Collins star rating is greater than 1. ; From the vocabulary Remove vocabulary from the middle The words in the text are used to obtain a subjective vocabulary. .

7. The general English tweet preprocessing method according to claim 1, characterized in that, The process of extracting named entities from the preprocessed tweet text based on the subjective lexicon, using a syntactic dependency parsing model and a tree-like parent-child structure, includes: All noun phrases and noun clauses in the preprocessed tweet text were extracted using a grammatical dependency parsing model. Iterate through all noun phrases, removing terminating articles, quantifiers, and stop words; and based on the aforementioned subjective vocabulary, remove subjective words, resulting in a preprocessed set of noun phrases and noun clauses. Based on sets The inclusion relationship of noun phrases is defined by taking noun phrases or noun clauses containing at least one noun phrase as the parent string and the noun phrases contained in the parent string as the child strings, thus constructing a tree-like parent-child structure; Named entity extraction is performed based on the parent-child structure to obtain the named entity recognition results of the English tweet.

8. A computer device, characterized in that, It includes at least one processor and at least one memory communicatively connected to the processor; The memory stores instructions that can be executed by the processor to implement the general English tweet preprocessing method according to any one of claims 1-7.

Images