A Chinese self-supervised word meaning understanding method and system

Abstract

The present invention discloses a method and system for self-supervised word meaning understanding in Chinese, which belongs to the field of natural language processing technology. Through generative self-supervision and discriminative self-supervision, the accuracy of the model in identifying the specific meaning of words is improved. The generative method is to design pre-training tasks for word meaning understanding to improve the model's ability to represent ambiguous words in the text; the discriminative method is to construct pseudo-ambiguous word labeling data to guide the model to learn the ability to distinguish the differences between word meanings. Specifically, it includes three modules: model pre-training for word meaning understanding, construction of pseudo-ambiguous word labeling data, and establishment of a word meaning understanding model.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a method and system for self-supervised word meaning understanding in Chinese. Background Art

[0002] Word sense understanding is the basis for machines to understand sentences and passages, and is an important task in natural language processing. In natural language processing, word sense disambiguation is a major word sense understanding task. Overseas, based on the published large-scale English word sense annotated corpus, researchers have mainly proposed two types of methods to solve word sense disambiguation: knowledge-based methods and supervision-based methods. The former mainly uses a meaning dictionary to disambiguate, such as the SREFKB method, which is a vector-based method that uses contextualized word representations and semantic embeddings in the Wordnet meaning dictionary to perform disambiguation. The latter mainly relies on a corpus with word sense annotations to train a meaning classifier, such as the GlossBERT and ESCHER methods trained on the Semcor corpus. These methods map words in the context to the correct meaning of the words by learning parameterized functions.

[0003] In China, due to the lack of large-scale Chinese word sense annotated corpora, Chinese word sense disambiguation primarily relies on the Hownet lexicon, with knowledge-based approaches being the mainstream. However, these methods can only understand the surface meaning of words and cannot discern subtle differences in meaning. Their overall performance is far inferior to supervised English word sense disambiguation methods. Summary of the Invention

[0004] In response to the current problem that Chinese word sense disambiguation lacks large-scale word sense annotated corpus and cannot effectively train machines to understand word meanings, the present invention provides a Chinese self-supervised word sense understanding method and system, which improves the ability of machines to understand word meanings by solving the word sense disambiguation problem.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A Chinese self-supervised word meaning understanding system, comprising a model pre-training module for word meaning understanding, a module for constructing pseudo-ambiguous word labeling data, and a module for establishing a word meaning understanding model;

[0007] The model pre-training module for word meaning understanding obtains initial text containing ambiguous words from an unlabeled Chinese corpus, adds noise to the text based on the different parts of speech characteristics of the ambiguous words, and restores the noisy text to enable the model to learn the semantic representation of ambiguous words in a generative and self-supervised manner. This module aims to pre-train an initial model for ambiguous word understanding, transfer the ambiguous word features learned by the initial model from a larger unlabeled dataset to the word meaning understanding model, and enable the model to train its word meaning understanding capabilities from a better initial state.

[0008] The pseudo-ambiguous word labeling data is constructed as follows: This module selects monosemous words from an unlabeled Chinese corpus as words to be labeled, and uses similarity calculation to add interfering interpretations to them, thereby forming a labeled dataset of pseudo-ambiguous words containing correct interpretations and interfering interpretations. This module aims to construct positive and negative samples for learning a word meaning understanding model, requiring the word meaning understanding model to narrow the distance between the specific meaning of the target word and the correct interpretation in the projection space, and widen the distance between it and the interfering interpretations, thereby learning the difference between word meanings in a discriminative and self-supervised manner.

[0009] Establishment of the word meaning understanding model: This module is based on the above-mentioned pre-trained model and the pseudo-ambiguous word tag dataset, and trains the machine's ability to understand word meaning by constructing multiple-choice questions.

[0010] A Chinese self-supervised word meaning understanding method comprises the following steps:

[0011] Step 1.1, collect original corpus: collect texts required for training from various fields;

[0012] Step 1.2, Design Pre-training Tasks: Specifically, the difficulty in understanding word meaning lies in understanding complex and ambiguous words. In real-world tasks, machines lack grammatical and semantic knowledge compared to humans. Pre-training tasks are required to learn contextual semantic representations to acquire relevant knowledge, and then, through appropriate fine-tuning, to map the semantic representations to specific meanings. However, traditional pre-training tasks mask randomly selected words to understand word meaning. This masking is too broad, and there is no specific learning of ambiguous words and their contextual information, and it does not adequately consider sequence information.

[0013] This step will start with ambiguous words and their contexts, and design pre-training tasks for ambiguous word understanding to help the machine understand word meaning more accurately while learning richer language knowledge.

[0014] Step 1.2.1, ambiguous word masking: replace ambiguous words in the original input text with the special symbol [MASK] to encourage the model to focus on the context in which the ambiguous words appear and train the model to infer individual ambiguous words;

[0015] Step 1.2.2, ambiguous word removal: remove ambiguous words from the original input text to help the model further understand the entire sentence and focus on the location of ambiguous words.

[0016] Step 1.2.3, Dependency Masking of Ambiguous Words: Replace the objects with direct dependencies in the context of the ambiguous word with the special symbol [MASK], so that the model focuses on the collocation relationship between the ambiguous word and the dependency word;

[0017] Step 1.2.4, word order shuffle: Randomly shuffle the order of words in the original input text to encourage the model to pay attention to the context order and train the model to control the overall semantics.

[0018] Step 1.3: Build a pre-trained model. This model uses a standard sequence-to-sequence Transformer architecture, consisting of a 6-layer encoder and a 6-layer decoder. The encoder input is a noisy sequence, while the decoder input consists of two parts: the encoder output and a right-shifted sequence of the original input. The decoder input is the original input sequence. The encoder encodes the noisy input text. The decoder uses a cross-headed multi-head attention mechanism to perform attention calculations on the hidden state results of the encoder's last layer, and then recovers the noisy text using an autoregressive approach.

[0019] Step 2.1, Locating and Screening Pseudo-Ambiguous Words: Extract all words and their meanings from the Chinese dictionary, determine whether the word is monosemous or polysemous based on the number of senses, and generate a corresponding vocabulary. Obtain initial text from the unlabeled corpus and identify monosemous words in the text based on the monosemous vocabulary. Calculate the TF-IDF value of monosemous words and select monosemous words that are greater than a certain threshold and are not proper nouns such as names of people, places, or institutions as pseudo-ambiguous words. Mark them in the corpus.

[0020] Step 2.2: Construct interference interpretations for pseudo-ambiguous words: For the pseudo-ambiguous words located in step 2.1, use the synonym dictionary and polysemous word list to find polysemous words with the same meaning. For each polysemous word, calculate the similarity between its interpretation and the correct interpretation of the pseudo-ambiguous word, eliminate the interpretation that is most similar to the correct interpretation, and treat the remaining interpretations as interference interpretations.

[0021] Step 3: Build a word meaning understanding model: This module treats the word meaning understanding task as an extractive question-answering task. Building on the pre-trained model described above, it adds a nonlinear fully-connected layer to build a word meaning understanding model. The model is then trained using pseudo-word meaning labeled data to improve its ability to discriminate word meanings. Specifically, data containing pseudo-word meaning labels is obtained, with the context of the target word as the question and all of the target word's meanings as the article. The model extracts fragments of the correct meaning from the article based on the question. The model inputs the context of the target word and all of its meanings. The model learns an embedded representation for each token through a pre-trained network. These features are then fed into a fully-connected layer for a linear transformation. The logits scores for the start and end positions of each token are calculated, and the interpretation with the highest probability among all interpretations is selected as the correct interpretation.

[0022] Compared with the prior art, the present invention has the following advantages:

[0023] This paper improves the model's accuracy in identifying the specific meanings of words through both generative and discriminative self-supervision. The generative approach involves designing pre-training tasks for word meaning understanding to improve the model's ability to represent ambiguous words in text; the discriminative approach involves constructing pseudo-ambiguous word labeling data to guide the model's ability to learn to distinguish between word meanings. Specifically, it includes three modules: model pre-training for word meaning understanding, constructing pseudo-ambiguous word labeling data, and establishing a word meaning understanding model.

[0024] (1) In the pre-training task for word meaning understanding, this paper designs four types of noise for ambiguous words and context. Using a large unlabeled corpus, the paper learns the latent semantic feature representation of ambiguous words through self-supervision, and transfers the knowledge learned by the pre-training model to the word meaning understanding model, effectively solving the problem of low performance of the word meaning understanding model due to the lack of large-scale labeled corpus.

[0025] (2) The present invention uses monosemous words to construct pseudo-word sense labeling data, which has high credibility of positive and negative samples, and narrows the gap with the real word sense labeling data through data enhancement, which is more conducive to providing supervision information for the model to acquire the ability to discriminate word meanings.

[0026] (3) The word meaning understanding model proposed in the present invention adopts the form of extractive question-answering tasks. Compared with the binary classification method based on one interpretation, the present invention can input all interpretations in the input sequence at one time. The model will take all interpretations into consideration when training parameters, and is better at distinguishing subtle word meanings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A Chinese self-supervised word meaning understanding method and system flow chart of the present invention;

[0028] Figure 2 This is a specific flow chart of the model pre-training for word meaning understanding described in the present invention;

[0029] Figure 3 This is a specific flow chart for constructing pseudo-ambiguous word tagging data described in the present invention;

[0030] Figure 4 This is a specific flow chart for establishing the word meaning understanding model described in the present invention. DETAILED DESCRIPTION

[0031] The present invention will now be described in further detail with reference to the accompanying drawings. The accompanying drawings are simplified schematic diagrams that only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0032] Example 1

[0033] like Figure 1As shown, this embodiment of the Chinese self-supervised word meaning understanding method and system improves the model's accuracy in identifying the specific meaning of words through both generative and discriminative self-supervision. The generative approach involves designing pre-training tasks for word meaning understanding to improve the model's ability to represent ambiguous words in text; the discriminative approach involves constructing pseudo-ambiguous word labeling data to guide the model's ability to discern the differences between word meanings. Specifically, it includes three modules: model pre-training for word meaning understanding, constructing pseudo-ambiguous word labeling data, and establishing a word meaning understanding model.

[0034] Example 2

[0035] like Figure 2 As shown, the model pre-training for word meaning understanding in this embodiment includes three steps: collecting original corpus, designing pre-training tasks, and building a pre-training model.

[0036] Specifically: We collected original corpus, including nearly 10G of text required for training from news corpus, community interactions, Wikipedia, Baidu Encyclopedia, food reviews, and telecommunications reviews, covering various fields such as news, novels, articles, dialogues, chats, comments, and reviews.

[0037] Design pre-training tasks, including: starting from ambiguous words and their context, designing pre-training tasks for ambiguous word understanding, helping the machine to learn richer language knowledge while more accurately understanding the meaning of words. Specifically, it involves four types of noise: (1) ambiguous word masking, replacing ambiguous words in the original input text with the special symbol [MASK], prompting the model to pay attention to the context in which the ambiguous words are located, and training the model to infer the ability of a single ambiguous word; (2) ambiguous word deletion, deleting ambiguous words in the original input text, prompting the model to further understand the entire sentence and pay attention to the location of the ambiguous words; (3) dependent word masking of ambiguous words, replacing objects with direct dependencies in the context of the ambiguous words with [MASK], prompting the model to pay attention to the collocation relationship between ambiguous words and dependent words; (4) word order scrambling, randomly scrambling the word order in the original input text, prompting the model to pay attention to the context word order, and training the model to control the overall semantics.

[0038] Training the pre-trained model involves: The model uses a standard sequence-to-sequence Transformer architecture, consisting of a 6-layer encoder and a 6-layer decoder. The encoder input is a noisy sequence, while the decoder input consists of two parts: the encoder output and a right-shifted sequence of the original input. The decoder input is the original input sequence. The encoder encodes the noisy input text; the decoder uses a cross-headed multi-head attention mechanism to perform attention calculations on the hidden state of the encoder's last layer, and then recovers the noisy text using an autoregressive approach.

[0039] Example 3

[0040] like Figure 3 As shown, the construction of pseudo-ambiguous word tagging data in this embodiment includes two steps: locating and screening pseudo-ambiguous words, and constructing interfering interpretations of pseudo-ambiguous words.

[0041] Specifically, the location and screening of pseudo-ambiguous words include: extracting all words and their meanings in the Chinese dictionary, judging whether the word is monosemous or polysemous based on the number of its meanings, and generating a corresponding vocabulary; obtaining the initial text in the unlabeled corpus, and identifying monosemous words in the text based on the monosemous vocabulary; calculating the TF-IDF value of monosemous words, selecting monosemous words that are greater than a certain threshold and are not proper nouns such as names of people, places, and institutions as pseudo-ambiguous words, and marking them in the corpus.

[0042] Specifically, constructing the interference interpretation of pseudo-ambiguous words includes: for the pseudo-ambiguous words located in step (1), using the synonym dictionary and the polysemous word list to find polysemous words with the same meaning; for each polysemous word, calculating the similarity between its interpretation and the correct interpretation of the pseudo-ambiguous word, eliminating the interpretation that is most similar to the correct interpretation, and treating the remaining interpretations as interference interpretations.

[0043] Example 4

[0044] like Figure 4 As shown, in this embodiment, a word meaning understanding model is established, and the word meaning understanding task is regarded as an extractive question-answering task. On the basis of the above-mentioned pre-trained model, a word meaning understanding model is established by adding a nonlinear fully connected layer, and the model's ability to discriminate word meanings is trained through pseudo-word meaning labeled data.

[0045] Specifically, data containing pseudo-word sense tags is obtained, with the context of the target word as the question and all of the target word's meanings as the article. The correct meaning fragments are extracted from the article based on the question. The model inputs the context of the target word and all of its meanings. The model learns an embedding representation for each token through a pre-trained network. These features are then fed into a fully connected layer for a linear transformation. The logits score for the start and end positions of each token is calculated, and the interpretation with the highest probability is selected as the correct interpretation among all the interpretations.

[0046] Any matters not described in detail in this specification are prior art known to those skilled in the art. Although the above description of the present invention is based on specific embodiments to facilitate understanding of the present invention by those skilled in the art, it should be understood that the present invention is not limited to the scope of the specific embodiments. As long as various modifications are within the spirit and scope of the present invention as defined and determined by the appended claims, such modifications will be obvious to those skilled in the art, and all inventions and creations utilizing the concepts of the present invention are protected.

Claims

1. A Chinese self-supervised word meaning understanding system, characterized by: It includes a model pre-training module for word meaning understanding, a module for building pseudo-ambiguous word tagging data, and a module for establishing a word meaning understanding model; The model pre-training module for word meaning understanding: This module obtains initial text containing ambiguous words from an unlabeled Chinese corpus, adds noise to the text based on the different parts of speech characteristics of the ambiguous words, and restores the noisy text to allow the model to learn the semantic representation of ambiguous words in a generative and self-supervised manner; The construction of pseudo-ambiguous word marking data: This module selects monosemous words from the unmarked Chinese corpus as words to be marked, and uses similarity calculation to add interference interpretations to them, thereby forming a marked dataset of pseudo-ambiguous words containing correct interpretations and interference interpretations; The establishment of the word meaning understanding model: This module is based on the above pre-trained model and the pseudo-ambiguous word labeling dataset, and trains the machine's ability to understand word meaning by constructing multiple-choice questions; Location and screening of pseudo-ambiguous words: Extract all words and their meanings from the Chinese dictionary, determine whether the word is monosemous or polysemous based on the number of senses, and generate a corresponding vocabulary; obtain the initial text from the unlabeled corpus and identify monosemous words in the text based on the monosemous vocabulary; Calculate the TF-IDF value of monosemous words, select monosemous words with a value greater than the set threshold and not being proper nouns such as names of people, places, or institutions as pseudo-ambiguous words, and mark them in the corpus; Constructing interference interpretations for pseudo-ambiguous words: To locate pseudo-ambiguous words, use the synonym dictionary and polysemous word list to find polysemous words with the same meaning; for each polysemous word, calculate the similarity between its interpretation and the correct interpretation of the pseudo-ambiguous word, eliminate the interpretation that is most similar to the correct interpretation, and treat the remaining interpretations as interference interpretations.

2. A method for self-supervised Chinese word meaning understanding based on the Chinese self-supervised word meaning understanding system according to claim 1, characterized in that: The following steps are involved: Step 1.1, collect original corpus: collect texts required for training from various fields; Step 1.2, design pre-training tasks: Step 1.2.1, ambiguous word masking: replace ambiguous words in the original input text with the special symbol [MASK] to encourage the model to focus on the context in which the ambiguous words appear and train the model to infer individual ambiguous words; Step 1.2.2, ambiguous word removal: remove ambiguous words from the original input text to help the model further understand the entire sentence and focus on the location of ambiguous words. Step 1.2.3, Dependency Masking of Ambiguous Words: Replace the objects with direct dependencies in the context of the ambiguous word with the special symbol [MASK], so that the model focuses on the collocation relationship between the ambiguous word and the dependency word; Step 1.2.4, word order shuffling: Randomly shuffle the order of words in the original input text to force the model to focus on the contextual word order and train the model to grasp the overall semantics; Step 1.3: Build a pre-trained model. This model uses a standard sequence-to-sequence Transformer architecture, consisting of a 6-layer encoder and a 6-layer decoder. The encoder input is a noisy sequence, and the decoder input consists of two parts: the encoder output and a right-shifted version of the original input. The decoder input is the original input sequence. The encoder encodes the noisy input text. The decoder uses a cross-head attention mechanism and the hidden state results of the last layer of the encoder to perform attention calculations, and then restores the noisy text in an autoregressive manner; Step 2.1, Location and Screening of Pseudo-Ambiguous Words: Extract all words and their meanings from the Chinese dictionary, determine whether the word is monosemous or polysemous based on the number of senses, and generate a corresponding vocabulary; obtain the initial text from the unlabeled corpus, and identify the monosemous words in the text based on the monosemous vocabulary; Calculate the TF-IDF value of monosemous words, select monosemous words with a value greater than the set threshold and not being proper nouns such as names of people, places, or institutions as pseudo-ambiguous words, and mark them in the corpus; Step 2.2: Construct interference interpretations for pseudo-ambiguous words: For the pseudo-ambiguous words located in step 2.1, use the synonym dictionary and polysemous word list to find polysemous words with the same meaning. For each polysemous word, calculate the similarity between its interpretation and the correct interpretation of the pseudo-ambiguous word, eliminate the interpretation that is most similar to the correct interpretation, and treat the remaining interpretations as interference interpretations. Step 3: Build a word meaning understanding model: Obtain data containing pseudo-word meaning tags, treat the context of the target word as a question, and all the definitions of the target word as an article. Extract the correct interpretation fragment from the article based on the question. The model input is the context of the target word and all its definitions. The model learns the embedding representation of each token through a pre-trained network, then sends these features to a fully connected layer for linear transformation. The logits scores of the start and end positions of each token are calculated, and the interpretation with the highest probability among all interpretations is selected as the correct interpretation.

Images