New word extraction method and device, electronic equipment, computer readable storage medium

Abstract

The present disclosure provides a new word extraction method and device, electronic equipment and computer readable medium, the method comprising: performing word segmentation processing on a corpus based on a basic word segmentation rule to obtain a word library, and performing character segmentation processing on the corpus to obtain a character library; performing word group processing based on the character library and the word library respectively to obtain a first candidate word library; screening candidate words according to the cohesion degree of each first candidate word in the first candidate word library to obtain a second candidate word library; performing word segmentation processing on the corpus according to the word group length parameters corresponding to each second candidate word in the second candidate word library, updating the second candidate word library according to the results of the word segmentation processing to obtain a third candidate word library; and screening new words from the third candidate word library according to a screening rule. According to the embodiments of the present disclosure, the accuracy of the extracted new words can be improved, thereby optimizing the extraction effect of new words.

Description

Technical Field

[0001] This disclosure relates to the field of natural language processing, and in particular to a new word extraction method and apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Neologisms are emerging terms that arise with technological advancements and cultural changes, or are old terms used in new ways. For various fields, extracting neologisms is a crucial task to ensure accuracy in word choice and expression. This is especially true in the field of natural language processing, where neologism extraction is a fundamental task. Currently, some conventional neologism extraction techniques rely on the degree of freedom between words or the semantic similarity between words, but both of these methods are relatively ineffective at extracting neologisms. Summary of the Invention

[0003] This disclosure provides a new word extraction method and apparatus, an electronic device, and a computer-readable storage medium.

[0004] Firstly, this disclosure provides a new word extraction method, which includes:

[0005] The corpus is segmented into words based on basic word segmentation rules to obtain a lexicon, and the corpus is segmented into characters to obtain a character library;

[0006] Word grouping is performed based on the character library and the word library respectively to obtain a first candidate word library; the first candidate word library includes multiple first candidate words, and each first candidate word corresponds to a word group length parameter;

[0007] Candidate words are selected based on the cohesion of each first candidate word in the first candidate word library to obtain the second candidate word library. The cohesion of each first candidate word is used to characterize the probability of the corresponding first candidate word as a word group.

[0008] The corpus is segmented using the phrase length parameter corresponding to each second candidate word in the second candidate lexicon, and the second candidate lexicon is updated according to the result of the segmentation to obtain the third candidate lexicon.

[0009] New words are obtained by filtering from the third candidate word library according to the filtering rules.

[0010] Secondly, this disclosure provides a new word extraction device, which includes:

[0011] The segmentation module is used to segment the corpus into words based on basic word segmentation rules to obtain a lexicon, and to segment the corpus into characters to obtain a character library;

[0012] The word grouping module is used to perform word grouping processing based on the character library and the word library respectively to obtain a first candidate word library; the first candidate word library includes multiple first candidate words, and each first candidate word corresponds to a word group length parameter;

[0013] The filtering module is used to filter candidate words based on the cohesion of each first candidate word in the first candidate word library to obtain a second candidate word library. The cohesion of each first candidate word is used to characterize the probability of the corresponding first candidate word as a word group.

[0014] The word segmentation module is used to segment the corpus using the word length parameter corresponding to each second candidate word in the second candidate word library, and update the second candidate word library according to the result of the word segmentation to obtain the third candidate word library;

[0015] The filtering module is also used to filter new words from the third candidate word library according to the filtering rules.

[0016] Thirdly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the above-described new word extraction method.

[0017] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described new word extraction method.

[0018] Fifthly, this disclosure provides a computer program or computer program product, the computer program product including a computer program stored in a computer-readable storage medium, the computer program implementing the above-described new word extraction method when executed by a processor.

[0019] The embodiments provided in this disclosure divide the corpus into a character-based database and a word-based database. Then, words are grouped based on the character and word databases according to a phrase length parameter. These grouped words are then used as candidate words for new words, resulting in a first candidate database. That is, this embodiment of the disclosure groups words based on both characters and words in the corpus to obtain a first candidate database for new words. Since the word selection dimension is not limited to the word level but also includes the character level, the obtained first candidate database provides comprehensive coverage of new words, including even those with relatively few features. This comprehensive first candidate database helps reduce the possibility of missing new words. Furthermore, this embodiment of the disclosure filters candidate words based on the cohesion of each first candidate word in the first candidate database to obtain a second candidate database, where cohesion represents the likelihood of a corresponding first candidate word forming a phrase. In other words, after obtaining a first candidate lexicon with relatively comprehensive coverage of new words, this embodiment first filters out word groups from the first candidate lexicon based on cohesion. This results in a second candidate lexicon with relatively comprehensive coverage and a large number of new words, providing a reliable data foundation for new word selection. Then, the corpus is segmented using the word group length parameters corresponding to each second candidate word in the second candidate lexicon. The second candidate lexicon is updated based on the segmentation results to obtain a third candidate lexicon. New words are then selected from the third candidate lexicon according to the selection rules. Since the second candidate lexicon is a filtered lexicon, the step size of the sliding window corresponding to the second candidate words has a certain reference value. Therefore, the words obtained by segmenting the corpus using the word group length parameters of the second candidate words have a certain corrective effect on the words in the second candidate lexicon, filtering out some inaccurately segmented words. This makes the new words in the third candidate lexicon more accurate than those in the second candidate lexicon, thus improving the accuracy of new words selected from the third candidate lexicon. As can be seen, by using the embodiments provided in this disclosure, a relatively comprehensive lexicon is used as the basic lexicon for extracting new words. Then, by filtering out words with inaccurate word segmentation in the basic lexicon through cohesion and re-segmentation, it is beneficial to extract new words with relatively few features, expand the scope corresponding to the extracted new words, and improve the accuracy of the extracted new words, thereby optimizing the extraction effect of new words.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0022] Figure 1 A flowchart illustrating a novel word extraction method provided in an embodiment of this disclosure;

[0023] Figure 2 A flowchart of a new word extraction method for another implementation of this disclosure embodiment;

[0024] Figure 3 A block diagram of a novel word extraction device provided in an embodiment of this disclosure;

[0025] Figure 4 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0027] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0028] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0030] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0031] This disclosure relates to the field of natural language processing (NLP) technology, specifically to a new word extraction or new word discovery technology based on NLP technology.

[0032] The “new words” involved in this disclosure refer to emerging words that have arisen with the development of technology and the times and cultural changes, such as high-speed rail and QR code scanning, or old words that have new usages, such as “big shot” (new usage refers to a person with a high level of expertise in a certain professional field) and “implementation” (new usage refers to certain systems reaching the level of enforceability, or a certain project reaching the level of usability, etc.).

[0033] New word extraction based on NLP technology can be considered an unsupervised task. Conventional extraction schemes may rely on the degrees of freedom or semantic similarity between words. For example, the degrees of freedom (Information Entropy), also known as left and right entropy, characterize the richness of the surrounding words of a word. Higher richness indicates a greater probability that the word is a fixed word, i.e., a higher probability of it forming a complete word. However, in practice, some new words do not possess the characteristic of rich surrounding words. Therefore, extracting new words based on degrees of freedom has a limited coverage of words with new word features, resulting in some new words not being extracted. On the other hand, relying on semantic similarity for new word extraction is heavily influenced by the initial word segmentation results. Therefore, inaccurate initial word segmentation can lead to new words extracted based on semantic similarity that are not truly new words or do not form complete words, resulting in low accuracy. It is evident that conventional new word extraction methods generally have poor extraction performance.

[0034] To solve the problem of poor extraction effect of conventional new word extraction methods, embodiments of the present disclosure provide a new word extraction method. First, the corpus is grouped into words and characters respectively to obtain a candidate word library with a relatively comprehensive feature coverage of new words as the data basis for new word extraction. Furthermore, through operations such as secondary word segmentation using cohesion and phrase length parameters, words with inaccurate word segmentation in the basic candidate word library are filtered, and a candidate word library with a larger coverage range and relatively high accuracy can be obtained. Then, new words are extracted based on this candidate word library with a larger coverage range and relatively high accuracy, thereby improving the accuracy of the extracted new words and optimizing the new word extraction effect.

[0035] The new word extraction system illustrated in the embodiments of the present disclosure can be deployed in an electronic device, and the electronic device can be a vehicle-mounted device, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some implementation manners, the method for converting text to audio can be implemented by a processor in the electronic device calling computer-readable program instructions stored in a memory.

[0036] Figure 1 It is a flowchart of a new word extraction method provided by an embodiment of the present disclosure. Referring to Figure 1 , the method includes:

[0037] In step S11, the corpus is segmented into a word library based on basic word segmentation rules, and the corpus is segmented into a character library.

[0038] Here, the corpus refers to the text containing new words to be extracted, and this text can be materials, articles, etc. obtained through compliant means.

[0039] In some implementation manners, the electronic device can segment the corpus using some basic word segmentation rules, such as jieba segmentation, yaha segmentation, etc. Exemplarily, the electronic device can segment words by referring to a pre-set dictionary or a domain dictionary, which is beneficial to detecting the accuracy of the obtained words.

[0040] In some implementation manners, word segmentation means splitting the corpus character by character. For example, if the corpus is "I love my hometown ABC City", where "ABC" is used to refer to a place name, taking A, B, and C as an example to represent a Chinese character character respectively. Word segmentation can obtain the word library "I love", "my", "hometown", and "ABC City", and word segmentation can obtain the character library "I", "love", "I", "my", "home", "town", "A", "B", "C", "City".

[0041] In some implementation manners, the sorting of the characters in the font library can be the same as the sorting of the characters in the corpus, and the sorting of the words in the word library can be the same as the sorting of the words in the corpus.

[0042] In step S12, word combination processing is respectively performed based on the font library and the word library to obtain a first candidate word library.

[0043] Among them, the first candidate word library includes multiple first candidate words, and each first candidate word can correspond to a phrase length parameter. The phrase length parameter of each first candidate word is used to represent the number of elements that make up the first candidate word, and the elements that make up the first candidate word are the characters in the font library or the words in the word library involved in step S11.

[0044] In some implementation manners, a sliding window can be used to combine words based on the word library or the font library. The sliding window can correspond to a certain step length, and here the step length is used to represent the number of characters or words taken by the corresponding sliding window each time, that is, the phrase length parameter. Exemplarily, the value of the phrase length parameter can include at least two integers greater than or equal to 2. In some implementation manners, the sliding window can be represented as n-gram, where n is used to represent the value of the step length. Using n-gram to combine words based on the font library or the word library means that the electronic device starts from the first character in the corresponding library, and sequentially combines the 1st to nth characters in order to obtain a new word. After that, the 2nd to n-1th characters are combined to obtain a new word, and then it slides successively according to the word combination rule until the last character.

[0045] For example, taking the sliding window 2-gram as an example, when combining words for the font library obtained in step S11, new words such as "I love", "love me", "my home", "home town", "town A", "AB", "BC", and "C city" can be obtained. Again, taking the sliding window 2-gram as an example, when combining words for the word library obtained in step S11, new words such as "I love my", "my home town", and "home town ABC city" can be obtained. In this example, the phrase length parameter of each new word is 2.

[0046] It should be noted that in actual implementation, the phrase length parameter n can be flexibly set according to the actual implementation scenario, and multiple n-grams can be set. For example, in one implementation scenario, the value of the phrase length parameter n can be set to include each value from 2 to 8.

[0047] In step S13, candidate words are screened according to the cohesion of each first candidate word in the first candidate word library to obtain a second candidate word library.

[0048] Cohesion, also known as Pointwise Mutual Information (PMI), is used to characterize the probability that a first-choice word will form a phrase, or the likelihood of it becoming a word. Higher cohesion indicates a greater probability that the first-choice word will form a phrase; lower cohesion indicates a lower probability.

[0049] Cohesion is typically calculated based on word frequency. Word frequency can refer to the number of times a word appears in the first candidate lexicon, or it can be the ratio of the total number of times the word appears in the first candidate lexicon to the total number of first candidate words. The electronic device can first determine the word frequency of each first candidate word in the first candidate lexicon, and then calculate the cohesion of the first candidate words based on the word frequency of each first candidate word. An exemplary algorithm for characterizing cohesion is as follows:

[0050] or,

[0051]

[0052] Where, x1 to x n This refers to the first candidate word x1x2…x n According to the description of step S12, if the character is the first candidate word obtained based on the character set, x1 to x n "Yes" is the character; if it is the first candidate word obtained based on word grouping from the dictionary, x1 to x n It is a word. s(x1x2…x n ) refers to the first candidate word x1x2…x n The cohesion, p(*) refers to the word frequency in the first candidate vocabulary of string (*), and (*) refers to any string involved in the above cohesion algorithm, such as p(x1x2…x n ) refers to the first candidate word x1x2…x n Word frequency in the first candidate vocabulary.

[0053] As described in step S12, each first candidate word corresponds to a phrase length parameter n. The smaller n is, the more words are formed based on the corresponding n-gram, resulting in fewer characters to be counted when calculating the cohesion of each first candidate word, and a relatively lower cohesion value. Conversely, the larger n is, the fewer words are formed based on the corresponding n-gram, resulting in more characters to be counted when calculating the cohesion of each first candidate word, and a relatively higher cohesion value. Therefore, for each value of n in the n-gram, a subset of candidate words is formed, meaning that first candidate words with the same phrase length parameter in the first candidate word library are considered as a subset of candidate words. A cohesion threshold is set for this phrase length parameter, allowing different phrase length parameters to correspond to different cohesion thresholds. Furthermore, in the process of selecting candidate words from the first candidate word library based on cohesion, the electronic device can compare the cohesion of each first candidate word in each candidate word subset with the cohesion threshold corresponding to the phrase length parameter of each candidate word subset, and add the first candidate word with a cohesion greater than the cohesion threshold to the second candidate word library; otherwise, filter out the first candidate word.

[0054] For example, referring to the example in step S12, the second candidate word library obtained in step S13 includes the candidate words {hometown, ABC city}.

[0055] As can be seen, by adopting this implementation method, the electronic device can select candidate words with relatively high aggregation from the first candidate word library to form a second candidate word library. This can initially filter out candidate words with low word formation probability in the first candidate word library and improve the quality of candidate words in the second candidate word library.

[0056] In step S14, the corpus is segmented using the word length parameter corresponding to each second candidate word in the second candidate lexicon, and the second candidate lexicon is updated according to the segmentation result to obtain the third candidate lexicon.

[0057] The electronic device can segment the corpus using the phrase length parameter of each second candidate word in the second candidate word library. Based on the segmentation result, a secondary segmentation word library can be obtained. Then, words in the secondary segmentation word library with a frequency greater than a preset threshold and not included in the second candidate word library are merged with the second candidate word library to obtain a third candidate word library.

[0058] The preset threshold for word frequency can be set according to the length of the corpus. In some implementations, the preset threshold for word frequency can be greater than or equal to 10 (the total number of times the word appears), for example, the preset threshold is 15.

[0059] It should be noted that the more characters a new word has, the scarcer its surrounding words become. Traditional methods of extracting new words based on degrees of freedom cannot extract new words with a large number of characters. Furthermore, according to other extraction methods, the new word has already been segmented into multiple words with fewer characters during the word segmentation stage. Therefore, traditional extraction methods cannot extract new words with a large number of characters. Based on the embodiments of this disclosure, in the scenario where the phrase length parameter n is set to greater than or equal to 3 during step S12, some candidate words with a large number of characters can be obtained.

[0060] For ease of description, words with a phrase length parameter greater than or equal to 3 in the second candidate lexicon are referred to as long phrases, and words with a phrase length parameter equal to 2 are referred to as short phrases. The corpus includes paragraphs containing long phrases and paragraphs that do not contain long phrases. For paragraphs in the corpus containing long phrases, in step S14, during the process of segmenting the corpus by phrase length parameter, the standardization of the corresponding long phrases can also be verified based on the segmentation results. If the verification result is non-standard, the corresponding long phrase is considered to be just a collection of words and not a standard phrase. In this case, the segmented phrases corresponding to the corresponding paragraph are included in the secondary segmentation lexicon, and the corresponding long phrase is deleted from the second candidate lexicon. If the verification result is standard, the corresponding long phrase is considered to be a standard phrase. In this case, the corresponding long phrase is retained in the second candidate lexicon, and the paragraphs in the corpus containing the corresponding long phrases are no longer segmented.

[0061] For example, for a paragraph in the corpus containing the long phrases, the word segmentation result may include a set of phrases corresponding to each phrase length parameter. If each phrase set includes a word from the second candidate lexicon, then the corresponding long phrase is determined to be a normal phrase; if any phrase set does not include a word from the second candidate lexicon, then the corresponding long phrase is determined to be an abnormal phrase.

[0062] For example, the candidate words in the second candidate word library include {hometown, ABC city, my hometown ABC city}. The phrase length parameters corresponding to the candidate words in the second candidate word library are, for example, {2, 3}. Among them, the phrase length parameter of "my hometown ABC city" is 3, so "my hometown ABC city" is a long phrase. The electronic device can segment the original corpus using 2-gram and 3-gram respectively. When segmenting the paragraph "I love my hometown ABC city", the phrase set obtained by 2-gram segmentation includes "hometown" from the second candidate word library, and the phrase set obtained by 3-gram segmentation includes "ABC city" from the second candidate word library. It can be seen that the long phrase "my hometown ABC city" is a standard word. The long phrase "my hometown ABC city" is retained in the second candidate word library, and the paragraph "I love my hometown ABC city" is not further segmented. For other paragraphs in the corpus, a secondary word segmentation lexicon is obtained by segmenting words using 2-gram and 3-gram. If the word frequency of some words in the secondary word segmentation lexicon is greater than a preset threshold and is not included in the second candidate lexicon, then the corresponding words are merged into the second candidate lexicon to obtain the third candidate lexicon.

[0063] Among the first candidate words filtered out based on cohesion, some may be highly likely to be new words. Furthermore, the long phrases obtained using the aforementioned method may contain non-standard phrases. Therefore, this implementation method, by relying on the phrase length parameters corresponding to each second candidate word in the second candidate lexicon to segment the corpus again, not only obtains some candidate words from a different perspective than cohesion, thus improving the candidate lexicon for new words, but also further corrects the long phrases contained in the second candidate lexicon. This improves the accuracy of each candidate word in the candidate lexicon and provides data support for extracting long phrases.

[0064] In step S15, new words are obtained by filtering from the third candidate word library according to the filtering rules.

[0065] The filtering rules can be preset and can include one filtering rule or at least two filtering rules. When at least two filtering rules are included, candidate words in the third candidate word library can be filtered sequentially using different filtering rules in a certain order, or candidate words in the third candidate word library can be filtered randomly using different filtering rules.

[0066] In some implementations, the electronic device can verify the legality of each long phrase in the third candidate lexicon and delete long phrases that do not meet the legality requirements, thus obtaining a preliminary new lexicon. Then, each word in the preliminary new lexicon is selected according to part-of-speech filtering rules. In this example, the legality of each long phrase is used to characterize the reasonableness of the corresponding long phrase as a new word.

[0067] It is understood that the above screening examples are merely illustrative descriptions and do not constitute a limitation on the execution of step S15 involved in the embodiments of this disclosure. In actual implementation, the execution order of the above two screening rules is not limited. For example, in some other implementations, the electronic device may first screen candidate words in the third candidate word library according to the part-of-speech screening rules to obtain a new word library after initial screening, and then verify the legality of long phrases in the new word library after initial screening.

[0068] For example, verifying the legality of each long phrase in the third candidate lexicon aims to further detect the accuracy of the long phrases. For instance, for each long phrase, the source of the elements constituting the long phrase can be determined, including the character library or lexicon described in step S11. Then, the long phrase is segmented using a phrase length parameter with the segmentation unit determined by the element source of the corresponding long phrase. If the words obtained by segmentation using any phrase length parameter do not include words from the third candidate lexicon, the long phrase is determined to be invalid. That is, for each long phrase, if the long phrase is obtained based on a character library, the long phrase is segmented using any phrase length parameter with characters as the unit. If the words obtained by segmentation using any sliding window of a certain length do not include words from the third candidate lexicon, the long phrase is determined to be invalid and deleted; otherwise, the long phrase is retained. If the long word group is obtained based on the word grouping of the word library, the long word group is segmented into words using any word group length parameter. If the words obtained by the sliding window segmentation of any step length do not include words from the third candidate word library, the long word group is deleted; otherwise, the long word group is retained.

[0069] This approach allows for further verification of whether long phrases are fixed collocations, thereby improving the accuracy of words in the third candidate word library and providing an accurate data foundation for extracting new words with a large number of characters.

[0070] The part-of-speech (POS) filtering rule can be to set various POS categories as auxiliary and non-auxiliary POS categories based on expression experience. The auxiliary POS category can be used as the criterion for filtering out words, while the non-auxiliary POS category can be used as the criterion for selecting words. Based on this, filtering each word in the initial new lexicon according to the POS filtering rule can include: deleting words whose first word in the initial new lexicon has a preset auxiliary POS category to obtain a second-screen lexicon; and then, using words in the second-screen lexicon with a preset selectable POS category as new words.

[0071] For example, preset auxiliary parts of speech may include: conjunctions, classifiers, adverbs, auxiliary words, pronouns, time words, and idioms. Preset optional parts of speech include: nouns, adverbial words, English words, names of organizations or groups, place names, verbs, and personal names.

[0072] As can be seen, the new word extraction method provided in this disclosure, based on character level and word level, and using sliding window word grouping with multiple step sizes to obtain a candidate word library, can obtain a more comprehensive basic word library including words with more characters. Then, by using cohesion and re-segmentation, words with inaccurate segmentation in the basic word library are filtered out, which is conducive to extracting new words with relatively few features and words with more characters, expanding the scope of the extracted new words, and improving the accuracy of the extracted new words, thereby optimizing the extraction effect of new words.

[0073] It is understandable that the adoption Figure 1 The illustrated new word extraction method extracts new words that can be added to a word database. Electronic devices can then use this database to perform one or more functions, such as knowledge extraction, knowledge graph construction, and business detection, to output processing results. Examples include extracting trending internet terms from the word database, and constructing a knowledge graph of classic works. These are just a few examples; further details are omitted here.

[0074] The method for extracting new words according to embodiments of this disclosure will be described below with reference to examples.

[0075] like Figure 2 As shown, Figure 2 A flowchart of an exemplary new word extraction method provided in this disclosure embodiment, the method comprising:

[0076] In step S21, the original corpus is obtained.

[0077] The original corpus includes, for example, the text "Our bank plans to enable small-amount contactless transactions for debit and credit cards through counter and customer service channels, and to enable small-amount contactless transactions for credit cards through SMS channels on the evening of December 22."

[0078] In step S22, the original corpus is segmented into words to obtain a lexicon, and the original corpus is segmented into characters to obtain a character library.

[0079] Examples of the thesaurus and character set are shown in Tables 1 and 2 below.

[0080] Table 1

[0081]

[0082] Table 2

[0083]

[0084] It should be noted that the " / " in Tables 1 and 2 separates the objects before and after the " / ", meaning that the objects before and after the " / " are two independent objects. For example, in Table 1, each word before and after the " / " is a separate word. The same applies to Table 2.

[0085] In step S23, candidate words are formed using an n-gram sliding window for each character and word library, and the word frequency of each candidate word is calculated, where n = 2, 3, 4... 8 represents the word length parameter of each candidate word.

[0086] The first candidate word lexicon is formed by using an n-gram sliding window to generate all candidate words. Examples of candidate words generated when n is 2 and n is 3 are shown in Table 3. For details on the word grouping method using the n-gram sliding window, please refer to the description of the above implementation method; it will not be repeated here.

[0087] Table 3

[0088]

[0089] In step S24, the cohesion of each candidate word is calculated. When n = i, candidate words with a cohesion greater than the cohesion threshold corresponding to i are selected to obtain the word library W.

[0090] In this example, for instance, using an algorithm Calculate the cohesion of each word. Examples will not be provided here. Then, for each word i, select candidate words with a cohesion greater than the cohesion threshold corresponding to i to form a vocabulary W. Vocabulary W can be a second candidate vocabulary.

[0091] In step S25, the original corpus is segmented using the i-gram corresponding to each word in the lexicon W, and the lexicon W is updated based on the segmentation results to obtain lexicon W2.

[0092] The lexicon W2 may include, for example, words with a frequency greater than 15 in the word segmentation results that are not included in W. The lexicon W2 can be a third candidate lexicon.

[0093] Additionally, for paragraphs in the original corpus that include word groups composed of 3-grams or more sliding windows in W, during the segmentation of the paragraph, if the word set obtained by each i-gram corresponding to W includes words from W, then the word group is retained in W; otherwise, the paragraph is segmented. For example, the segment "small amount contactless payment" in the thesaurus W is retained in W.

[0094] In step S26, each long word group in the dictionary W2 is validated using a sliding window, and the dictionary W3 is obtained based on the validation results.

[0095] Among them, the vocabulary library W3 can be used for initial screening of new vocabulary libraries.

[0096] When a long phrase is obtained based on the word grouping shown in Table 2, the segment is divided into characters using i-gram. If i is any one of 2 to 8, and the resulting words do not include words from word group W2, the long phrase is deleted; otherwise, the long phrase is retained.

[0097] When the corresponding segment is obtained based on the vocabulary shown in Table 1, the long word group is segmented into words using i-gram. If i is any one of 2 to 8, the segmented words do not include words from vocabulary W2, and the long word group is deleted; otherwise, the long word group is retained.

[0098] In step S27, candidate words in W3 are filtered according to part-of-speech rules to obtain a new vocabulary.

[0099] For example, words in W3 whose first word is a conjunction, quantifier, adverb, auxiliary word, pronoun, time word, or idiom are deleted to obtain a second-screen word library. Words in the second-screen word library whose part of speech is a noun, adverb, English word, organization or group name, place name, verb, or personal name are used as new words.

[0100] Understandable, Figure 2 The illustrative embodiments are merely descriptive and do not constitute a limitation on the present disclosure. In actual implementation, the new word extraction method of the present disclosure may include more or fewer implementation steps, or the execution order of the steps may differ from that described above.

[0101] As can be seen, the new word extraction method provided in this disclosure, based on character level and word level, and using sliding window word grouping with multiple step sizes to obtain a candidate word library, can obtain a more comprehensive basic word library including words with more characters. Then, by using cohesion and re-segmentation, words with inaccurate segmentation in the basic word library are filtered out, which is conducive to extracting new words with relatively few features and words with more characters, expanding the scope of the extracted new words, and improving the accuracy of the extracted new words, thereby optimizing the extraction effect of new words.

[0102] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0103] In addition, this disclosure also provides a new word extraction device, an electronic device, and a computer-readable storage medium, all of which can be used to implement any of the new word extraction methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding section of the method and will not be repeated here.

[0104] Figure 3A block diagram of a novel word extraction device provided in an embodiment of this disclosure.

[0105] Reference Figure 3 This disclosure provides a new word extraction device, which includes: a segmentation module 31, a word grouping module 32, a filtering module 33, and a word segmentation module 34. Each module, when running, can implement some or all of the functions described in the above-described method. For example:

[0106] The segmentation module 31 can be used to segment the corpus into a word library based on basic word segmentation rules, and to segment the corpus into a character library. The word grouping module 32 can be used to perform word grouping based on the character library and the word library respectively, to obtain a first candidate word library. The first candidate word library includes multiple first candidate words, each first candidate word corresponding to a phrase length parameter. The filtering module 33 can be used to filter candidate words according to the cohesion of each first candidate word in the first candidate word library, to obtain a second candidate word library. The cohesion of each first candidate word is used to characterize the probability that the corresponding first candidate word is a phrase. The word segmentation module 34 can be used to segment the corpus with the phrase length parameter corresponding to each second candidate word in the second candidate word library, and update the second candidate word library according to the result of the word segmentation, to obtain a third candidate word library. The filtering module 33 can also be used to filter new words from the third candidate word library according to the filtering rules.

[0107] In one embodiment, the value of the phrase length parameter includes at least two integers greater than or equal to 2, and the first candidate words in the first candidate word library that have the same phrase length parameter form a subset of candidate words. The first candidate word library includes at least one subset of candidate words, and a phrase length parameter corresponds to a cohesion threshold.

[0108] When the filtering module 33 filters candidate words based on the cohesion of each first candidate word in the first candidate word library to obtain the second candidate word library, it performs the following steps:

[0109] For each candidate word subset, the cohesion of each first candidate word in each candidate word subset is compared with the cohesion threshold corresponding to the phrase length parameter of each candidate word subset, and the first candidate words with a cohesion greater than the cohesion threshold are added to the second candidate word library.

[0110] In one embodiment, the second candidate vocabulary includes long phrases and short phrases, and the corpus includes paragraphs containing the long phrases and paragraphs not containing the long phrases;

[0111] When the word segmentation module 34 performs word segmentation on the corpus using the phrase length parameter corresponding to each second candidate word in the second candidate lexicon, updates the second candidate lexicon according to the word segmentation result, and obtains the third candidate lexicon, it performs the following steps:

[0112] Using the phrase length parameter corresponding to each second candidate word in the second candidate lexicon, each paragraph in the corpus is segmented into words;

[0113] For paragraphs in the corpus that do not contain the long phrases, the resulting segmented phrases are used as words in the secondary segmentation lexicon.

[0114] For paragraphs in the corpus containing the long word groups, the standardization of the corresponding long word groups is verified based on the word segmentation results. The standardization of each long word group is used to characterize whether the corresponding long word group is a standard word group. If the verification result is non-standard, the word group corresponding to the corresponding paragraph is used as a word in the secondary word segmentation lexicon. If the verification result is standard, the corresponding long word group is retained in the second candidate word.

[0115] The words in the secondary word segmentation lexicon with a frequency greater than a preset threshold and not included in the second candidate lexicon are merged with the second candidate lexicon to obtain a third candidate lexicon.

[0116] In one embodiment, for a paragraph in the corpus containing the long phrases, the word segmentation result includes a set of phrases corresponding to each phrase length parameter; when the word segmentation module 34 verifies the standardization of the corresponding long phrases based on the word segmentation result, it performs the following steps:

[0117] If each phrase set includes words from the second candidate vocabulary, then the corresponding long phrase is determined to be a standard word;

[0118] If any word group set does not include a word from the second candidate word library, then the corresponding long word group is determined to be an irregular word.

[0119] In one embodiment, when the filtering module 33 obtains new words from the third candidate word library according to the filtering rules, it performs the following steps:

[0120] Verify the legality of each long phrase in the third candidate word library. The legality of each long phrase is used to characterize the reasonableness of the corresponding long phrase as a new word.

[0121] Long phrases that do not meet the legality requirements are deleted from the third candidate vocabulary to obtain a preliminary new vocabulary;

[0122] Each word in the initial new vocabulary is filtered according to the part-of-speech filtering rules to obtain new words.

[0123] In one embodiment, verifying the legality of each long phrase in the third candidate lexicon includes:

[0124] For each long phrase, determine the source of the elements that make up the long phrase, where the source of the elements includes the character library or the dictionary;

[0125] The long phrase is segmented using a phrase length parameter and a segmentation unit determined by the element source of the corresponding long phrase; the segmentation unit corresponding to the character library is a character, and the segmentation unit corresponding to the word library is a word;

[0126] If the words obtained by segmenting the words using any word group length parameter do not include words from the third candidate word library, then the long word group is determined to be invalid.

[0127] In one embodiment, when the filtering module 33 filters each word in the initial new word library according to the part-of-speech filtering rules, it performs the following steps: delete words in the initial new word library whose first word has a preset auxiliary part-of-speech, to obtain a second word library; and take words in the second word library whose part-of-speech has a preset optional part-of-speech as new words.

[0128] In one embodiment, the preset auxiliary parts of speech include: conjunctions, classifiers, adverbs, auxiliary words, pronouns, time words, and idioms; the preset optional parts of speech include: nouns, adverbial words, English words, names of organizations or groups, place names, verbs, and personal names.

[0129] In one embodiment, the new word extraction device further includes a processing module:

[0130] The processing module is used to add the obtained new words to the word library; and to perform one of the following processing operations based on the word library to output the processing result. The processing operations include any one or more of the following: knowledge extraction, knowledge graph construction, and business detection.

[0131] For details on the specific implementation method, please refer to the above. Figure 1 and Figure 2 The implementation method shown is not described in detail here.

[0132] It is understandable that the above division of modules is only a logical functional division. In actual implementation, each of the above modules can be integrated into the hardware implementation. For example, the functions of acquiring corpus involved in the above implementation can be integrated into the I / O interface, and the functions of segmentation module 31, word grouping module 32, filtering module 33 and word segmentation module 34 can be integrated into the processor implementation.

[0133] Reference Figure 4 , Figure 4This disclosure provides an electronic device comprising: at least one processor 401; at least one memory 402; and one or more I / O interfaces 403 connected between the processor 401 and the memory 402; wherein the memory 402 stores one or more computer programs executable by the at least one processor 401, the one or more computer programs being executed by the at least one processor 401 to enable the at least one processor 401 to perform the above-described new word extraction method.

[0134] This disclosure also provides a computer-readable storage medium, which may be volatile or non-volatile, storing a computer program thereon, wherein the computer program, when executed by processor 401:

[0135] The corpus is segmented into words based on basic word segmentation rules to obtain a word library, and the corpus is segmented into characters to obtain a character library. Word grouping is performed on the character library and the word library respectively to obtain a first candidate word library. The first candidate word library includes multiple first candidate words, each corresponding to a phrase length parameter. Candidate words are selected based on the cohesion of each first candidate word in the first candidate word library to obtain a second candidate word library. The cohesion of each first candidate word is used to characterize the probability that the corresponding first candidate word is a phrase. The corpus is segmented into words using the phrase length parameter corresponding to each second candidate word in the second candidate word library, and the second candidate word library is updated according to the segmentation results to obtain a third candidate word library. New words are selected from the third candidate word library according to the selection rules.

[0136] The value of the phrase length parameter includes at least two integers greater than or equal to 2. First candidate words with the same phrase length parameter in the first candidate word library form a candidate word subset. The first candidate word library includes at least one candidate word subset. One phrase length parameter corresponds to one cohesion threshold. In some embodiments, the processor 401 is further configured to: for each candidate word subset, compare the cohesion of each first candidate word in each candidate word subset with the cohesion threshold corresponding to the phrase length parameter of each candidate word subset, and add the first candidate words with a cohesion greater than the cohesion threshold to the second candidate word library.

[0137] The second candidate lexicon includes long word groups and short word groups. The corpus includes paragraphs containing the long word groups and paragraphs not containing the long word groups. In some embodiments, the processor 401 is further configured to: perform word segmentation on each paragraph in the corpus using the word group length parameter corresponding to each second candidate word in the second candidate lexicon; for paragraphs in the corpus that do not contain the long word groups, use the resulting word groups as words in the secondary word segmentation lexicon; for paragraphs in the corpus that contain the long word groups, verify the standardization of the corresponding long word groups based on the word segmentation results, whereby the standardization of each long word group is used to characterize whether the corresponding long word group is a standard word group; if the verification result is non-standard, use the word groups corresponding to the corresponding paragraph as words in the secondary word segmentation lexicon; if the verification result is standard, retain the corresponding long word groups in the second candidate lexicon; merge words in the secondary word segmentation lexicon that have a word frequency greater than a preset threshold and are not included in the second candidate lexicon with the second candidate lexicon to obtain a third candidate lexicon.

[0138] For a paragraph containing the long phrase in the corpus, the word segmentation result includes a set of phrases corresponding to each phrase length parameter; in some embodiments, the processor 401 is further configured to: if each phrase set includes a word from the second candidate lexicon, then the corresponding long phrase is determined to be a normal word; if any phrase set does not include a word from the second candidate lexicon, then the corresponding long phrase is determined to be an abnormal word.

[0139] In some embodiments, the processor 401 is further configured to: verify the legality of each long phrase in the third candidate word library, wherein the legality of each long phrase is used to characterize the reasonableness of the corresponding long phrase as a new word; delete long phrases in the third candidate word library that do not meet the legality requirements to obtain a preliminary new word library; and perform screening processing on each word in the preliminary new word library according to the part-of-speech filtering rules to obtain new words.

[0140] In some embodiments, the processor 401 is further configured to: for each long word group, determine the source of the elements constituting the long word group, the source of the elements including the character library or the word library; segment the long word group using a word group length parameter with a segmentation unit determined by the element source of the corresponding long word group; if the words obtained by segmenting with any word group length parameter do not include words from the third candidate word library, determine that the long word group is invalid.

[0141] In some embodiments, the processor 401 is further configured to: delete words whose first word in the initial screening of the new word library has a preset auxiliary part of speech, to obtain a second screening word library; and use words in the second screening word library with a preset optional part of speech as new words.

[0142] In some embodiments, the processor 401 is further configured to: add the obtained new words to a word library; and perform one of the following processing operations based on the word library to output a processing result, the processing operations including any one or more of the following: knowledge extraction, knowledge graph construction, and business detection.

[0143] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the above-described new word extraction method.

[0144] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0145] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0146] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0147] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0148] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0149] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0150] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0151] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0153] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A method for extracting new words, characterized in that, include: The corpus is segmented into words based on basic word segmentation rules to obtain a lexicon, and the corpus is segmented into characters to obtain a character library; Word grouping is performed based on the character library and the word library respectively to obtain a first candidate word library; the first candidate word library includes multiple first candidate words, and each first candidate word corresponds to a word group length parameter; Candidate words are selected based on the cohesion of each first candidate word in the first candidate word library to obtain the second candidate word library. The cohesion of each first candidate word is used to characterize the probability of the corresponding first candidate word as a word group. The corpus is segmented using the phrase length parameter corresponding to each second candidate word in the second candidate lexicon, and the second candidate lexicon is updated according to the result of the segmentation to obtain the third candidate lexicon. New words are obtained by filtering from the third candidate word library according to the filtering rules.

2. The new word extraction method according to claim 1, characterized in that, The value of the phrase length parameter includes at least two integers greater than or equal to 2. First candidate words with the same phrase length parameter in the first candidate word library form a subset of candidate words. The first candidate word library includes at least one subset of candidate words. One phrase length parameter corresponds to one cohesion threshold. The step of filtering candidate words based on the cohesion of each first candidate word in the first candidate thesaurus to obtain the second candidate thesaurus includes: For each candidate word subset, the cohesion of each first candidate word in each candidate word subset is compared with the cohesion threshold corresponding to the phrase length parameter of each candidate word subset, and the first candidate words with a cohesion greater than the cohesion threshold are added to the second candidate word library.

3. The new word extraction method according to claim 1, characterized in that, The second candidate lexicon includes long phrases and short phrases. The corpus includes paragraphs containing the long phrases and paragraphs not containing the long phrases. The corpus is segmented using the phrase length parameter corresponding to each second candidate word in the second candidate lexicon, and the second candidate lexicon is updated according to the segmentation result to obtain a third candidate lexicon, including: Using the phrase length parameter corresponding to each second candidate word in the second candidate lexicon, each paragraph in the corpus is segmented into words; For paragraphs in the corpus that do not contain the long phrases, the resulting segmented phrases are used as words in the secondary segmentation lexicon. For paragraphs in the corpus containing the long word groups, the standardization of the corresponding long word groups is verified based on the word segmentation results. The standardization of each long word group is used to characterize whether the corresponding long word group is a standard word group. If the verification result is non-standard, the word group corresponding to the corresponding paragraph is used as a word in the secondary word segmentation lexicon. If the verification result is standard, the corresponding long word group is retained in the second candidate word. The words in the secondary word segmentation lexicon with a frequency greater than a preset threshold and not included in the second candidate lexicon are merged with the second candidate lexicon to obtain a third candidate lexicon.

4. The new word extraction method according to claim 3, characterized in that, For paragraphs in the corpus that contain the long phrases, the word segmentation result includes the set of phrases corresponding to each phrase length parameter; The verification of the standardization of corresponding long phrases based on the word segmentation results includes: If each phrase set includes words from the second candidate vocabulary, then the corresponding long phrase is determined to be a standard word; If any word group set does not include a word from the second candidate word library, then the corresponding long word group is determined to be an irregular word.

5. The new word extraction method according to claim 4, characterized in that, The step of selecting new words from the third candidate word library according to the filtering rules includes: Verify the legality of each long phrase in the third candidate word library. The legality of each long phrase is used to characterize the reasonableness of the corresponding long phrase as a new word. Long phrases that do not meet the legality requirements are deleted from the third candidate thesaurus to obtain a preliminary new thesaurus; Each word in the initial new vocabulary is filtered according to the part-of-speech filtering rules to obtain new words.

6. The new word extraction method according to claim 5, characterized in that, The verification of the legality of each long phrase in the third candidate thesaurus includes: For each long phrase, determine the source of the elements that make up the long phrase, where the source of the elements includes the character library or the dictionary; The long phrase is segmented using a phrase length parameter and a segmentation unit determined by the element source of the corresponding long phrase; the segmentation unit corresponding to the character library is a character, and the segmentation unit corresponding to the word library is a word; If the words obtained by segmenting according to any word group length parameter do not include words from the third candidate word library, then the long word group is determined to be invalid.

7. The new word extraction method according to claim 6, characterized in that, The step of filtering each word in the initial new lexicon according to the part-of-speech filtering rules includes: Delete words whose first word in the initial screening of the new lexicon has a preset auxiliary part of speech, and obtain the second screening lexicon; Words in the second-screen word library that have a preset selectable part of speech are taken as new words.

8. The new word extraction method according to claim 7, characterized in that, The preset auxiliary parts of speech include: conjunctions, classifiers, adverbs, auxiliary words, pronouns, time words, and idioms; The preset selectable parts of speech include: nouns, adverbs, English words, names of organizations or groups, place names, verbs, and personal names.

9. The new word extraction method according to claim 1, characterized in that, Also includes: Add the new words to the vocabulary database; Perform one of the following processing operations based on the vocabulary database to output the processing result. The processing operations include any one or more of the following: knowledge extraction, knowledge graph construction, and business detection.

10. A new word extraction device, characterized in that, include: The segmentation module is used to segment the corpus into words based on basic word segmentation rules to obtain a lexicon, and to segment the corpus into characters to obtain a character library; The word grouping module is used to perform word grouping processing based on the character library and the word library respectively to obtain a first candidate word library; The first candidate word library includes multiple first candidate words, and each first candidate word corresponds to a phrase length parameter; The filtering module is used to filter candidate words based on the cohesion of each first candidate word in the first candidate word library to obtain a second candidate word library. The cohesion of each first candidate word is used to characterize the probability of the corresponding first candidate word as a word group. The word segmentation module is used to segment the corpus using the word length parameter corresponding to each second candidate word in the second candidate word library, and update the second candidate word library according to the result of the word segmentation to obtain the third candidate word library; The filtering module is also used to filter new words from the third candidate word library according to the filtering rules.

11. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the new word extraction method as described in any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the new word extraction method as described in any one of claims 1-9.

Images