A method and system for discovering new words in literature based on masked language model

Abstract

The present invention discloses a method and system for discovering new words in documents based on a masked language model, belonging to the field of artificial intelligence natural language processing technology. The method includes a masked language training component, a model dependency calculation component, and a maximum probability calculation component. The masked language training component performs data cleaning and sentence segmentation on document data, identifies word embeddings through training component vectors of the training set, establishes an attention mechanism and a feedforward neural network, and combines them into a set of encoders, and builds an encoder training model. The encoded training set is randomly masked, and some input tokens are used as training set inputs, while the masked tokens are used as outputs. This method is used as a data generator to train a deep bidirectional representation network. The present invention can achieve a balance between speed and accuracy, and has a better new word discovery effect for documents in specific scenarios.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence natural language processing technology, and in particular to a method and system for discovering new words in documents based on a masked language model. Background Art

[0002] In the field of new word discovery in natural language processing, the main approaches currently used to address new word discovery and segmentation in specific scenarios include dictionary lookup and character annotation. These approaches each have their own advantages and disadvantages for different scenarios, and domestic and international scholars have made numerous innovations in this area. The most classic example is Bengio's "ANeural Probabilistic Language Model," which has been deeply inspiring for many scholars. In 2017, the Journal of the University of Electronic Science and Technology of China published a paper titled "Probabilistic Chinese Word Segmentation Algorithm Based on the Aho-Corasick Automaton Algorithm (CPACA)." In November 2020, the Journal of Command Information Systems and Technology published "Entity Recognition Method for Military Trends Text Based on LSTM-CRF." That same year, the Journal of Changchun Normal University published "Research on Word2vec Chinese Text Classification Based on Information Entropy Weighting." Based on the research of scholars in recent years, it is concluded that using the cohesion (mutual information) of adjacent words to construct a vocabulary is too rough for word meaning representation and is not conducive to the segmentation of new words in complex scenarios. Unsupervised word segmentation based on language models provides a complete and independent method of other unsupervised word segmentation methods, but it is too complex and contains the bottleneck of viterbi. Summary of the Invention

[0003] The technical task of the present invention is to address the above shortcomings and provide a method and system for discovering new words in documents based on a masked language model. This method can achieve a balance between speed and accuracy and has a better new word discovery effect for documents in specific scenarios.

[0004] The technical solution adopted by the present invention to solve its technical problem is:

[0005] A method for discovering new words in literature based on a masked language model, comprising a masked language training component, a model dependency operation component, and a maximum probability operation component.

[0006] The masked language training component cleans and segments document data. It then constructs a vectorized word embedding based on the training set, builds an attention mechanism and a feedforward neural network, and combines them into an encoder. It then builds an encoder training model. The encoded training set is randomly masked, with some input tokens used as training set input and the masked tokens as output. This method serves as a data generator and trains a deep bidirectional representation network. After iteration and training, a masked language model capable of completing the Cloze task is obtained, and the model is persisted to ensure its incremental computing power and portability.

[0007] The model dependency calculation component is used to obtain the correlation between any two tokens and form a correlation matrix of all tokens. This method obtains a correlation index different from the mutual information measurement and calculates the dependency between words in the sentence.

[0008] The maximum probability operation component uses the inter-word dependency index of the sentence to be tested generated by the model dependency operation component to split the specific threshold, and through the adjustment of multi-word weights, uses the probability product and weight ratio to calculate the maximum probability combination of the sentence to be tested.

[0009] This method optimizes and upgrades the existing inter-word correlation method based on mutual information. It allows the inter-word correlation metric to include inter-word dependencies and word meaning recognition to a certain extent, providing more flexible new word discovery capabilities and even lexical analysis. This method addresses some of the current pain points of new word discovery systems in literature.

[0010] Preferably, the data cleaning and segmentation are to split the Chinese literature data into the smallest units based on punctuation, line breaks and paragraph breaks;

[0011] The training set is trained by encoding Token Embedding in units of words to generate the original word table. Segment Embedding and Position Embedding are then written in sequence according to the smallest unit, and the sum of the two is used to form Word Embedding in units of words.

[0012] Preferably, the Attention mechanism is based on Multi-Head Self-Attention+Add&Normalization, and the feedforward neural network is based on Feed-Forward+Add&Normalization.

[0013] Preferably, a 12-layer Encoder training model is built.

[0014] Preferably, the data generator replaces a word with a [MASK] tag 80% of the time, replaces the word with a random word 10% of the time, and leaves the word unchanged 10% of the time.

[0015] Preferably, the model dependency operation component replaces the sentence to be analyzed with [MASK] one by one in units of words, outputs an initial prediction sequence after passing through the masked language training component, and based on this sequence, replaces the words other than the word segment to be analyzed with [MASK] in turn, repeatedly enters the masked language training model, obtains a matrix sequence composed of a second group of tokens, performs Euclidean distance operation on the two sequences in turn, obtains the degree of dependence of each word in the sentence segment except itself on this word, forms a correlation group, loops until all words to be analyzed are completed, calculates the average value of the positive and negative dependence degrees of two adjacent words, and obtains the dependency between words in the sentence.

[0016] Preferably, the maximum probability operation component uses a tree structure to store new word branch information and matching similarity.

[0017] The present invention also claims protection for an unsupervised document new word discovery system, comprising a masked language training component, a model dependency operation component, and a maximum probability operation component.

[0018] The system implements the above-mentioned unsupervised document new word discovery method through the masked language training component, model dependency operation component and maximum probability operation component.

[0019] The present invention also claims protection for an unsupervised document new word discovery device, comprising: at least one memory and at least one processor;

[0020] The at least one memory is configured to store a machine-readable program;

[0021] The at least one processor is configured to call the machine-readable program to execute the above-mentioned unsupervised document new word discovery method.

[0022] The present invention also claims protection for a computer-readable medium having computer instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned unsupervised document new word discovery method.

[0023] Compared with the prior art, the method and system for discovering new words in documents based on a masked language model of the present invention have the following beneficial effects:

[0024] This method and system optimizes and upgrades the existing inter-word correlation based on mutual information. This allows the inter-word correlation metric to include inter-word dependencies and word meaning recognition to a certain extent, providing more flexible new word discovery capabilities and even lexical analysis. This addresses, to a certain extent, the current pain points of new word discovery systems in literature. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flow chart of a new word discovery system based on a masked language model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described below with reference to specific embodiments.

[0027] Given that currently available new word discovery systems use the degree of cohesion (mutual information) between adjacent characters to construct a vocabulary, their representation of word meaning is too coarse and unfavorable for new word segmentation in complex scenarios, unsupervised word segmentation based on language models offers a completely independent approach, but it is overly complex and suffers from the bottleneck of Viterbi. Therefore, we aim to find a system that strikes a balance between speed and accuracy, and achieves better new word discovery results for documents in specific scenarios.

[0028] The embodiment of the present invention provides a method for discovering new words in literature based on a masked language model, including a masked language training component, a model dependency calculation component and a maximum probability calculation component. Figure 1 The technical implementation scheme is as follows:

[0029] 1) In the masked language model training component, Chinese document data is split into smallest units based on punctuation, line breaks, and paragraph breaks, and token embedding is performed on a character-by-character basis to generate an original character table. Segment embedding and position embedding are then written sequentially based on the smallest units, and the sum of the results forms a word embedding on a character-by-character basis.

[0030] Build an attention mechanism based on Multi-Head Self-Attention + Add & Normalization and a feedforward neural network based on Feed-Forward + Add & Normalization, combining them into a set of encoders;

[0031] A 12-layer encoder training model was built. The encoded training set was used as the training set input by randomly masking some input tokens, and the masked tokens were used as the output to form a data generator. A deep bidirectional representation network was trained. The data generator replaced words with the [MASK] marker 80% of the time, replaced the word with a random word 10% of the time, and kept the word unchanged 10% of the time.

[0032] After iteration and training, a masked language model that can complete the Cloze task is obtained, and the model is persisted to ensure its incremental computing capability and portability.

[0033] 2) In the model dependency operation component, according to the embedding tags of the original text, the sentence to be analyzed can be represented as a sequence of many tokens. Through the model dependency operation component, the correlation between any two tokens can be obtained, and a correlation matrix of all tokens can be constructed to obtain a correlation index different from the mutual information measurement;

[0034] The sentence to be analyzed is replaced with [MASK] one by one in units of words. After passing through the masked language model component, the initial prediction sequence is output. Based on this sequence, the words except the word fragment to be analyzed are replaced with [MASK] in turn. The masked language model is repeatedly entered to obtain a matrix sequence composed of the second group of tokens. The two sequences are sequentially subjected to Euclidean distance operations to obtain the degree of dependence of each word in the sentence fragment except itself on this word, forming a correlation group. The cycle is completed until all the words to be analyzed are completed. The average value of the positive and negative dependency degrees of two adjacent words is calculated to obtain the dependency between the words in the sentence.

[0035] 3) In the maximum probability calculation component, the inter-word dependency index of the sentence under test, generated by the model dependency calculation component, is used to split the sentence at a specific threshold. By adjusting the weights of multiple words, the maximum probability combination of the sentence under test is calculated by multiplying the probability product and the weight ratio. A tree structure is then used to store the branch information of new words and the matching similarity.

[0036] This method aims to match the endless stream of new Internet words, proprietary words in various categories of literature, and related words generated by the development of various industries in natural language texts; the existing inter-word correlation proposed based on mutual information is optimized and upgraded, so that the measurement standard of inter-word correlation can include the recognition of inter-word dependencies and word meanings to a certain extent, with more flexible new word discovery capabilities and even the analysis performance of discovered lexical structure.

[0037] The present invention also claims protection for an unsupervised document new word discovery system, comprising a masked language training component, a model dependency operation component, and a maximum probability operation component.

[0038] The system implements the unsupervised document new word discovery method described in the above embodiment through the masked language training component, model dependency operation component and maximum probability operation component.

[0039] The present invention also claims protection for an unsupervised document new word discovery device, comprising: at least one memory and at least one processor;

[0040] The at least one memory is configured to store a machine-readable program;

[0041] The at least one processor is configured to call the machine-readable program to execute the unsupervised document new word discovery method described in the above embodiment.

[0042] An embodiment of the present invention further provides a computer-readable medium having computer instructions stored thereon. When executed by a processor, the computer instructions cause the processor to perform the unsupervised document new word discovery method described in the above embodiment. Specifically, a system or device equipped with a storage medium can be provided, on which software program code implementing the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device can be caused to read and execute the program code stored in the storage medium.

[0043] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0044] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0045] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0046] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU installed on the expansion board or expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0047] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the scope of protection of the present invention.

Claims

1. A method for discovering new words in literature based on a masked language model, characterized in that: Including mask language training component, model dependency operation component and maximum probability operation component, The masked language training component cleans and segments the literature data. It then uses training data to construct word embedding vectors, builds an attention mechanism and a feedforward neural network, and combines them into an encoder. It then constructs an encoder training model. The encoded training set is randomly masked, with some input tokens used as training set input and the masked tokens used as output. This method serves as a data generator and trains a deep bidirectional representation network. After iteration and training, a masked language model capable of completing the Cloze task is obtained, and the model is persisted to ensure its incremental computing capability and portability. The model dependency calculation component is used to obtain the correlation between any two tokens and form a correlation matrix of all tokens. This method obtains a correlation index different from the mutual information measurement and calculates the dependency between words in the sentence. The model dependency operation component replaces the sentence to be analyzed with [MASK] one by one in units of words, outputs an initial prediction sequence after passing through the mask language training component, and based on this sequence, replaces the words other than the word segment to be analyzed with [MASK] in sequence, repeatedly enters the mask language training model, obtains a matrix sequence composed of a second group of tokens, performs Euclidean distance operation on the two sequences in sequence, obtains the degree of dependence of each word in the sentence segment except itself on this word, forms a correlation group, loops until all words to be analyzed are completed, calculates the average value of the positive and negative dependence degrees of two adjacent words, and obtains the sentence word dependency; The maximum probability operation component uses the inter-word dependency index of the sentence to be tested generated by the model dependency operation component to split the specific threshold, and through the adjustment of multi-word weights, uses the probability product and weight ratio to calculate the maximum probability combination of the sentence to be tested.

2. A method for discovering new words in literature based on a masked language model according to claim 1, characterized in that The data cleaning and segmentation are to split the Chinese literature data into the smallest units based on punctuation, line breaks and paragraph breaks; The training set is trained by encoding Token Embedding in units of words to generate the original word table. Segment Embedding and Position Embedding are then written in sequence according to the smallest unit, and the sum of the two is used to form Word Embedding in units of words.

3. A method for discovering new words in documents based on a masked language model according to claim 1 or 2, characterized in that The Attention mechanism is based on Multi-Head Self-Attention + Add & Normalization, and the feedforward neural network is based on Feed-Forward + Add & Normalization.

4. A method for discovering new words in documents based on a masked language model according to claim 3, characterized in that Build a 12-layer Encoder training model.

5. The method for discovering new words in literature based on a masked language model according to claim 3, characterized in that The data generator replaces the word with a [MASK] token 80% of the time, replaces the word with a random word 10% of the time, and leaves the word unchanged 10% of the time.

6. The method for discovering new words in literature based on a masked language model according to claim 1, characterized in that The maximum probability operation component uses a tree structure to store new word branch information and matching similarity.

7. A new word discovery system based on a masked language model, characterized in that: Including mask language training component, model dependency operation component and maximum probability operation component, The system implements the method for discovering new words in documents based on a masked language model as described in any one of claims 1 to 6 through the masked language training component, the model dependency operation component and the maximum probability operation component.

8. A device for discovering new words in literature based on a masked language model, characterized in that: include: at least one memory and at least one processor; The at least one memory is configured to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method for discovering new words in documents based on a masked language model according to any one of claims 1 to 6.

9. A computer-readable medium, characterized in that The computer-readable medium stores computer instructions, which, when executed by a processor, enable the processor to execute the method for discovering new words in a document based on a masked language model according to any one of claims 1 to 6.

Images