A new word discovery method and device

Abstract

The present invention discloses a new word discovery method and device, and relates to the field of computer technology. A specific implementation of the method includes: performing word segmentation processing on a character string to be segmented based on a vocabulary; performing BIO annotation on the result after the word segmentation processing; performing unsupervised training using the result after the BIO annotation to obtain a named entity recognition model; using the named entity recognition model to predict the BIO annotation of the character string to be predicted; synthesizing words based on the BIO annotation of the character string to be predicted; comparing the synthesized words with the word segmentation processing results of the character string to be predicted to obtain new words. This implementation avoids the problem of large-scale labeling when obtaining training samples, and improves the accuracy of word segmentation processing and the efficiency of discovering new words based on the discovered new words.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a new word discovery method and device. Background Art

[0002] With the vigorous development of various industries, more and more new words are created and used. In order to ensure the accuracy of word segmentation results in self-recognized language processing in various fields, discovering new words has become a research focus, especially in the field of e-commerce. With the emergence of a large number of brand words, product words, and attribute words, accurately identifying words is the key to product search and personalized recommendations.

[0003] At present, there are two commonly used new word discovery methods: rule-based new word discovery methods and statistical new word discovery methods. Among them, statistical new word discovery methods include supervised and unsupervised types.

[0004] In the process of realizing the present invention, the inventors found that there are at least the following problems in the prior art: supervision requires a corpus with detailed annotations, but it is often difficult to obtain a corpus with detailed annotations; in the word segmentation process, it is highly dependent on the existing vocabulary, and the existing vocabulary has limited collection of new words, resulting in insufficient accuracy in word segmentation, which in turn leads to the inaccuracy of new word discovery. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a new word discovery method and device, which can not only discover new words based on an existing vocabulary, but also continuously improve the quality and efficiency of discovering new words.

[0006] To achieve the above-mentioned purpose, according to the first aspect of the present invention, a new word discovery method is provided, comprising: performing word segmentation processing on a character string to be segmented based on a vocabulary; performing BIO annotation on the result after the word segmentation processing; performing unsupervised training using the result after the BIO annotation to obtain a named entity recognition model; using the named entity recognition model to predict the BIO annotation of the character string to be predicted; synthesizing words based on the BIO annotation of the character string to be predicted; and comparing the synthesized words with the word segmentation processing result of the character string to be predicted to obtain new words.

[0007] Optionally, the method further includes: performing unsupervised pre-training on the character string to be segmented to generate word vectors; and performing unsupervised training using the word vectors and the results after BIO annotation to obtain the named entity recognition model.

[0008] Optionally, the method further includes: screening the acquired new words according to one or more of the following: the frequency of the new words, the internal coagulation degree of the new words, and the information entropy of the new words.

[0009] Optionally, the method further includes: adding the screened new words to the vocabulary.

[0010] Optionally, a Masked LM method or a Google BERT model is used to perform unsupervised pre-training on the character string to be segmented to generate a word vector.

[0011] To achieve the above-mentioned purpose, according to the second aspect of the present invention, a new word discovery device is provided, including: a word segmentation processing module, a BIO annotation module, a model acquisition module, a BIO annotation prediction module, a word synthesis module, and a new word acquisition module; wherein the word segmentation processing module is used to perform word segmentation processing on the character string to be segmented based on a vocabulary; the BIO annotation module is used to perform BIO annotation on the result after the word segmentation processing; the model acquisition module is used to use the result after BIO annotation to perform unsupervised training to obtain a named entity recognition model; the BIO annotation prediction module is used to use the named entity recognition model to predict the BIO annotation of the character string to be predicted; the word synthesis module is used to synthesize words according to the BIO annotation of the character string to be predicted; the new word acquisition module is used to compare the synthesized words with the word segmentation processing result of the character string to be predicted to obtain new words.

[0012] To achieve the above-mentioned purpose, according to the third aspect of the present invention, a server for new word discovery is provided, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods described in the above-mentioned new word discovery method.

[0013] To achieve the above-mentioned purpose, according to a fourth aspect of the present invention, a computer-readable medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, any method described in the above-mentioned new word discovery method is implemented.

[0014] Due to the adoption of the above-mentioned technical means, the present invention has the following advantages or beneficial effects: based on the word segmentation processing of the vocabulary, a preliminary training sample is obtained, avoiding the problem of a large number of labeling; the word vector generated by powerful pre-training can better extract the semantic information of the word in the text context as well as the part of speech, classification, etc., ensuring the accuracy of the word vector in the training sample; based on the pre-trained word vector and BIO annotation training, the named entity recognition model can effectively predict the classification or BIO annotation of the word, and then can effectively discover new words by comparing with the word segmentation processing results. At the same time, the accuracy of the word segmentation processing results can be further improved based on the discovered new words. In addition, after discovering new words, the quality of the new words added to the vocabulary is guaranteed by screening and evaluating the new words, and the efficiency of new word recognition is further improved by continuously expanding the vocabulary and cyclically discovering new words.

[0015] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0017] Figure 1 is a schematic diagram of the main process of the new word discovery method according to an embodiment of the present invention;

[0018] Figure 2a is a schematic diagram of a BIO annotation result according to an embodiment of the present invention;

[0019] Figure 2b is a schematic diagram of a predicted BIO annotation result according to an embodiment of the present invention;

[0020] Figure 2c is a schematic diagram of a synthetic word result according to an embodiment of the present invention;

[0021] Figure 3 is a schematic diagram of the main process of another new word discovery method according to an embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of main modules of a new word discovery device according to an embodiment of the present invention;

[0023] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0024] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0026] like Figure 1 As shown, the embodiment of the present invention provides a new word discovery method, which may include the following specific steps:

[0027] Step S101, performing word segmentation processing on the character string to be segmented based on the word library.

[0028] The word library is an existing word library in various fields, and the character string to be segmented is a text material library in various fields collected through various channels such as the Internet, books, journals, and magazines, and the text material library is constantly updated and expanded with the accumulation of time, such as the merchant store information, product information, user comments, etc. collected through the e-commerce platform in the field of e-commerce. In the absence of special instructions, the embodiments of this application are all explained by using the word library in the field of e-commerce (abbreviated as e-commerce), and the character string to be segmented is the collected e-commerce field title as an example: the e-commerce field word library includes the relevant words contained in the title and the corresponding types, namely product words (P), brand words (B), attribute words (Q), quantifiers (D), venue words (L), others (O), etc. In the process of segmenting the character string to be segmented based on the word library, the positions of the words obtained after segmentation in the character string can be connected as edges, and the left and right entropies of the words obtained after segmentation can be used as weights to establish a directed graph; finally, the maximum entropy value path in the directed graph is dynamically planned to obtain the best segmentation result. Specifically, taking the segmented title "New Product 2018 Winter New Elderly Men's Wool Sweater Casual Thickened Large Size Bottoming Shirt Upper Green 165 / 105 / m" as an example, the results of the word segmentation processing are as follows: New Product-B, 2018 Winter-D, New-Q, Elderly-O, Men's-Q, Sweater-P, Casual-Q, Thickened-Q, Large Size-Q, Bottoming Shirt-P, Upper Green-O, 165 / 105-D, / -O, mQ.

[0029] Step S102, performing BIO annotation on the result after word segmentation processing.

[0030] BIO annotation is a commonly used sequence annotation method in natural language processing, which refers to annotating labels B, I, and O on each element of the result after word or word segmentation processing. Among them, B (Begin) indicates that the element is located at the beginning of a word or the result of word segmentation processing, I (Inside) indicates that the element is located inside a word or the result of word segmentation processing, and O indicates that the element is located outside the result of a word or word segmentation processing (Outside). Specifically, in an embodiment of the present application, BIO annotation refers to annotating each element contained in each fragment obtained by word segmentation processing as "BX", "IX" or "O". Among them, "BX" indicates that the fragment where this element is located belongs to type X and this element is at the beginning of this fragment, "IX" indicates that the fragment where this element is located belongs to type X and this element is in the middle of this fragment, and "O" means that it does not belong to any type. Among them, type X includes product words (P), brand words (B), attribute words (Q), quantifiers (D), venue words (L), others (O), etc.

[0031] Specifically, see Figure 2a, and provided the results of word segmentation and BIO annotation for the title "Tipsy 2018 New High Waist Harem Pants Autumn and Winter Thickened PU Leather Loose Slimming Skinny Pants Casual Plush Pants Tr0619 Black [Plus Plush] 3xl". After the title word segmentation, multiple segments such as Tipsy-B, 2018-D, and New-Q were obtained. Then, BIO annotation was performed based on the position of the elements in each segment. For example, taking the segment "Tipsy-B" as an example, the element "Tipsy" is located at the beginning of the segment, so it is marked as "BB", and the element "Tipsy" is located in the middle of the segment or at a non-beginning position, so it is marked as "IB".

[0032] Step S103, using the BIO annotated results to perform unsupervised training to obtain a named entity recognition model.

[0033] The named entity recognition model (NER) is essentially a multi-classification model, and its task goal is to identify product words, brand words, attribute words, quantifiers, other words, etc. in the title sentence. Specifically, the word vector corresponding to the fragment after word segmentation processing and the corresponding annotation "BX" are used as training samples for unsupervised training. The word vector represents the semantic information of the corresponding word or fragment in the character string. In this way, the process of labeling a large number of samples is avoided. At the same time, the results of word segmentation processing can be matched or corrected based on the obtained named entity recognition model. It can be understood that the word vector corresponding to the title or the fragment after word segmentation processing can be obtained through various methods such as Word2vector, Glove, FastText, etc., and used as training samples.

[0034] In an optional embodiment, the character string to be segmented is subjected to unsupervised pre-training to generate word vectors; the word vectors and the results after BIO annotation are used to perform unsupervised training to obtain the named entity recognition model. The word vectors generated after pre-training can better represent the semantic information of the fragments in the character string to be segmented, that is, it can improve the accuracy of the results of word segmentation based on the vocabulary.

[0035] In an optional embodiment, the Masked LM method or the Google BERT model is used to perform unsupervised pre-training on the character string to be segmented to generate word vectors. The Masked LM method randomly masks some words in the input sentence and then predicts them in pre-training. The more mature representative is the Google BERT (Bidirectional Encoder Representations from Transformers) model. The advantage of this training method is that it can learn to represent the contextual content in both the left and right directions of the sentence, so that the obtained word vector can more accurately express the semantic information in the context.

[0036] Step S104: using the named entity recognition model to predict the BIO tag of the character string to be predicted.

[0037] As new words and new short texts are continuously generated, in order to timely discover new words that have been generated and expand the vocabulary, the character string to be predicted can be any text material collected that may contain new words, such as the titles of new products obtained through the e-commerce platform, the store information of merchants newly connected to the e-commerce platform, the product information updated by the merchant store, user reviews, etc., which may contain new product names, brand names, store names, etc. It is understandable that in order to improve the accuracy of the prediction of the named entity recognition model and to more accurately discover new words, the character string to be segmented used to train the named entity recognition model and the character string to be predicted used to discover new words should be different text materials.

[0038] See also Figure 2b , providing the BIO annotations corresponding to the title "New 2018 Winter New Elderly Men's Wool Sweater Leisure Thickened Large Size Bottoming Shirt Upper Green 165 / 105 / m" predicted by the named entity recognition model. Among them, [CLS] indicates the beginning of the sentence, and [SEP] indicates the end of the sentence.

[0039] Step S105 , synthesizing words according to the BIO annotation of the character string to be predicted.

[0040] See also Figure 2c , based on the above steps, according to Figure 2b The corresponding BIO annotation of the title "New 2018 Winter New Elderly Men's Wool Sweater Leisure Thickened Large Size Bottoming Shirt Upper Green 165 / 105 / m" provided in the article is synthesized according to the BIO annotation rules. The results are detailed in the following table. Figure 2c or as follows:

[0041] [CLS]-[CLS], new product-B, 2018 winter-D, new style-Q, elderly-O, men's-Q, sweater-P, casual-Q, thickened-Q, large size-Q, base shirt-P, upper green-O, 165 / 105-D, / -O, mQ, [SEP]-[SEP].

[0042] Step S106, comparing the synthesized word with the word segmentation processing result of the character string to be predicted to obtain a new word.

[0043] In getting such Figure 2cBased on the word synthesis result shown, the synthesized result can be compared with the word segmentation result of the title in step S101. If there are differences, the words that only appear in the word synthesis result are considered as the new words to be discovered. Taking the string to be predicted "统领铸钢减速带铸铁道路减速板" as an example, after word segmentation of the string to be predicted based on the existing e-commerce domain word library, the word segmentation result is "统-O,领-O,铸钢-O,减速带-P,铸铁-O,道路-O,减速板-P"; using the named entity recognition model to predict the BIO annotation of the string to be predicted "统领铸钢减速带铸铁道路减速板", the synthesized words are "统领-B,铸钢-O,减速带-P,铸铁-O,道路-O,减速板-P". By comparison, it can be seen that compared with the analysis result, the new word "统领-B" appears in the synthesized words, so "统领-B" is considered as a new brand word.

[0044] Due to the large number of text corpora, the quality of the discovered new words is uneven. To ensure the quality of the discovered new words, it is considered to evaluate or screen the discovered new words through certain rules.

[0045] In an optional implementation manner, the obtained new words are screened according to one or more of the following: the frequency of the new words, the internal cohesion degree of the new words, and the information entropy of the new words.

[0046] The frequency of the new word is the frequency of the new word's appearance. The higher the frequency, the higher the possibility that the new word becomes a high-quality new word. Therefore, according to actual needs, a frequency threshold can be set to remove the new words with a frequency lower than the threshold.

[0047] The internal cohesion degree of the new word is used to measure whether the word collocation is reasonable. For example, assume that the binary word group C is composed of word A and word B, and the probabilities of their individual appearances are P(C), P(A), and P(B) respectively. If C is a real word, then A and B are not independent, and the probability of the two words appearing simultaneously will be greater than P(A)*P(B), that is, P(C)>>P(A)*P(B). If C can be split into two words A and B, then A and B are independent, and the probability of the two words appearing simultaneously is P(A)*P(B). Therefore, an internal cohesion degree threshold can be set, and screening is carried out according to the following formula, and only the new words higher than the internal cohesion degree threshold are retained, which can effectively prevent the extraction of garbage word groups such as "的电影" and "了一".

[0048]

[0049] The information entropy of a new word represents the richness of the word's left neighbor word set and right neighbor word set. The higher the information entropy, the higher the quality of the new word. You can set an information entropy threshold according to actual needs, and filter according to the following formula to only retain new words with information entropy higher than the information entropy threshold.

[0050]

[0051] Wherein, pi is the probability of an element appearing in the left neighbor word set or the right neighbor word set.

[0052] Specifically, take "Eat grapes without spitting out the grape skins, do not eat grapes but spit out the grape skins" as an example to illustrate, the word "grapes" appears four times, where the left neighbor word set is {eat, spit, eat, spit}, and the right neighbor word set is {not, skin, pour, skin}. According to the formula, the information entropy of the left neighbor word of the word "grapes" is -(1 / 2)·log(1 / 2)-(1 / 2)·log(1 / 2)≈0.693, and the information entropy of the right neighbor word is -(1 / 2)·log(1 / 2)-(1 / 4)·log(1 / 4)-(1 / 4)·log(1 / 4)≈1.04, and the information entropy of "grapes" is 1.733 (i.e., the sum of 0.693 and 1.04).

[0053] In an optional implementation, the screened new words are added to the word library.

[0054] After filtering and evaluating new words, some junk words can be effectively screened out. At the same time, in order to ensure the correctness of the results of word segmentation based on the word library, consider adding the newly screened high-quality new words to the word library, so that the results of subsequent word segmentation are more and more accurate, and a new and more optimized named entity recognition model is trained based on the new word segmentation results. In this way, after a certain period of new word accumulation after cyclic model training, the newly trained named entity recognition model can quickly recognize newly appeared words.

[0055] Based on the above embodiments, the new word discovery method provided by the embodiment of the present invention obtains preliminary training samples based on the word segmentation processing of the vocabulary, avoiding the problem of a large number of labeling; the word vectors generated by powerful pre-training can better extract the semantic information of the words in the text context as well as magnetic part of speech, classification, etc., ensuring the accuracy of the word vectors in the training samples; the named entity recognition model obtained based on the pre-trained word vectors and BIO annotation training can effectively predict the classification or BIO annotation of the words, and then effectively discover new words by comparing with the word segmentation processing results. At the same time, the accuracy of the word segmentation processing results can be further improved based on the discovered new words. In addition, after discovering new words, the quality of the new words added to the vocabulary is guaranteed by screening and evaluating the new words, and the efficiency and accuracy of new word recognition are further improved by continuously expanding the vocabulary and cyclically discovering new words.

[0056] See also Figure 3 Based on the above embodiment, the present invention provides a new word discovery method, which specifically includes the following steps:

[0057] Step S201, performing word segmentation processing on the character string to be segmented based on the word library.

[0058] Step S202: perform BIO annotation on the result after word segmentation processing.

[0059] Step S203, performing unsupervised pre-training on the character string to be segmented to generate word vectors.

[0060] Specifically, the Masked LM method or the open source Google BERT model is used to perform unsupervised pre-training on the character string to be segmented to generate word vectors. Specifically, the training of "my dog ​​is hairy" is used as an example for explanation:

[0061] 80% of the time: Mask the previously selected word with a special word (token): [MASK]. For example: replace my dog ​​is hairy with my dog ​​is [Mask].

[0062] 10% of the time, replace with random other words (tokens). For example: replace my dog ​​is hairy with mydog is apple.

[0063] 10% keep the original word unchanged. For example: replace my dog ​​is hairy with my dog ​​is hairy.

[0064] In other words, the approach taken by Masked LM is to randomly select some words to be predicted when inputting a sentence, and then replace them with a special word. Although the model will eventually see the input information at all positions, since the words to be predicted have been replaced by special words, the model cannot know in advance what words are in these positions, so that the model can learn the words to fill in these places based on the given labels. At the same time, since the special word used in the pre-training process will not appear in subsequent tasks, in order to be consistent with subsequent tasks, the original word or a random word is entered in the position of the word to be predicted according to a certain ratio.

[0065] Step S204, performing unsupervised training using the word vector and the BIO annotated results to obtain the named entity recognition model.

[0066] Unsupervised training is performed using word vectors and corresponding annotations “BX” as training samples to obtain a named entity recognition model.

[0067] Step S205: using the named entity recognition model to predict the BIO tag of the character string to be predicted.

[0068] Step S206: synthesizing words according to the BIO annotation of the character string to be predicted.

[0069] Step S207, comparing the synthesized word with the word segmentation result of the character string to be predicted to obtain a new word.

[0070] Step S208, filtering the acquired information according to one or more of the following: frequency of new words, internal coagulation degree of new words, and information entropy of new words.

[0071] Step S209, adding the screened new words to the word library.

[0072] See also Figure 4 The embodiment of the present invention provides a new word discovery device 400, including: a word segmentation processing module 401, a BIO annotation module 402, a model acquisition module 403, a BIO annotation prediction module 404, a word synthesis module 405, and a new word acquisition module 406; wherein,

[0073] The word segmentation processing module 401 is used to perform word segmentation processing on the character string to be segmented based on the word library;

[0074] The BIO annotation module 402 is used to perform BIO annotation on the result after word segmentation processing;

[0075] The model acquisition module 403 is used to perform unsupervised training using the BIO annotated results to obtain a named entity recognition model;

[0076] The BIO annotation prediction module 404 is used to predict the BIO annotation of the character string to be predicted using the named entity recognition model;

[0077] The word synthesis module 405 is used to synthesize words according to the BIO annotation of the character string to be predicted;

[0078] The new word acquisition module 406 is used to compare the synthesized words with the word segmentation processing results of the character string to be predicted to acquire new words.

[0079] In an optional embodiment, it also includes: a pre-training module 407; the pre-training module 407 is used to perform unsupervised pre-training on the character string to be segmented to generate word vectors; the model acquisition module 403 is also used to perform unsupervised training using the word vectors and the results after BIO annotation to obtain the named entity recognition model.

[0080] In an optional implementation, the new word acquisition module 406 is further used to screen the acquired new words according to one or more of the following: the frequency of the new words, the internal coagulation degree of the new words, and the information entropy of the new words.

[0081] In an optional implementation, the new word acquisition module 406 is further configured to add the screened new words to the word library.

[0082] In an optional implementation, the pre-training module 407 is used to perform unsupervised pre-training on the character string to be segmented using a Masked LM method or a Google BERT model to generate word vectors.

[0083] An embodiment of the present invention also provides a server for new word discovery, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the new word discovery method as described in any of the aforementioned embodiments.

[0084] An embodiment of the present invention further provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processor, the new word discovery method as described in any of the aforementioned embodiments is implemented.

[0085] Figure 5 An exemplary system architecture 500 to which the page configuration method or the page configuration apparatus according to the embodiment of the present invention can be applied is shown.

[0086] like Figure 5As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0087] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0088] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0089] Server 505 may be a server that provides various services, such as a backend management server that provides support for shopping websites browsed by users using terminal devices 501, 502, and 503. The backend management server may analyze and process received data such as product information query requests, and feed back processing results (such as new words) to the terminal device.

[0090] It should be noted that the new word discovery method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the new word discovery device is generally disposed in the server 505 .

[0091] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0092] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0093] like Figure 6As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0094] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0095] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0096] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0097] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0098] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be arranged in a processor, for example, they may be described as: a processor including a word segmentation processing module, a BIO annotation module, a model acquisition module, a BIO annotation prediction module, a word synthesis module, and a new word acquisition module. The names of these modules do not constitute a limitation on the modules themselves in certain cases, for example, the word segmentation processing module may also be described as a "module for performing word segmentation processing on the segmented character string based on a vocabulary".

[0099] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes: performing word segmentation processing on the character string to be segmented based on the vocabulary; performing BIO annotation on the result after word segmentation processing; performing unsupervised training using the result after BIO annotation to obtain a named entity recognition model; using the named entity recognition model to predict the BIO annotation of the character string to be predicted; synthesizing words based on the BIO annotation of the character string to be predicted; comparing the synthesized words with the word segmentation processing result of the character string to be predicted to obtain new words.

[0100] According to the technical solution of the embodiment of the present invention, the word segmentation processing based on the vocabulary is used to obtain preliminary training samples, avoiding the problem of large amounts of labeling; the word vectors generated by powerful pre-training can better extract the semantic information of words in the text context, as well as the part of speech, classification, etc., to ensure the accuracy of the word vectors in the training samples; the named entity recognition model obtained based on the pre-trained word vectors and BIO annotation training can effectively predict the classification or BIO annotation of words, and then effectively discover new words by comparing with the word segmentation processing results. At the same time, the accuracy of the word segmentation processing results can be further improved based on the discovered new words. In addition, after discovering new words, the quality of the new words added to the vocabulary is guaranteed by screening and evaluating the new words, and the efficiency of new word recognition is further improved by continuously expanding the vocabulary and cyclically discovering new words.

[0101] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A new word discovery method, characterized in that: include: Perform word segmentation on the segmented string based on the vocabulary; Perform BIO annotation on the results after word segmentation processing; Use the BIO annotated results for unsupervised training to obtain a named entity recognition model; Using the named entity recognition model, predicting the BIO tag of the string to be predicted; Synthesize words according to the BIO annotation of the character string to be predicted; The synthesized words are compared with the word segmentation processing results of the character string to be predicted to obtain new words.

2. The new word discovery method according to claim 1, characterized in that: Also includes: Performing unsupervised pre-training on the character string to be segmented to generate word vectors; Unsupervised training is performed using the word vectors and the results after BIO annotation to obtain the named entity recognition model.

3. The new word discovery method according to claim 1, characterized in that: Also includes: The acquired new words are screened according to one or more of the following: the frequency of the new words, the internal coagulation degree of the new words, and the information entropy of the new words.

4. The new word discovery method according to claim 3, characterized in that: Also includes: The screened new words are added to the word library.

5. The new word discovery method according to claim 1, characterized in that: Use the Masked LM method or the Google BERT model to perform unsupervised pre-training on the string to be segmented to generate word vectors.

6. A new word discovery device, characterized in that: include: Word segmentation processing module, BIO annotation module, model acquisition module, BIO annotation prediction module, word synthesis module, new word acquisition module; among them, The word segmentation processing module is used to perform word segmentation processing on the character string to be segmented based on the word library; The BIO annotation module is used to perform BIO annotation on the result after word segmentation processing; The model acquisition module is used to perform unsupervised training using the results of BIO annotation to obtain a named entity recognition model; The BIO annotation prediction module is used to predict the BIO annotation of the character string to be predicted using the named entity recognition model; The word synthesis module is used to synthesize words according to the BIO annotation of the character string to be predicted; The new word acquisition module is used to compare the synthesized words with the word segmentation processing results of the character string to be predicted to acquire new words.

7. A server for new word discovery, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images