Multi-channel mixed hollow convolution combined with residual and attention for chinese word sense disambiguation

Abstract

The present application relates to a kind of Chinese word sense disambiguation method of combining multi-channel hybrid dilated convolution neural network with residual and attention (Combining Multi-Channel Hybrid Dilated Convolution Neural Network with Residual and Attention, MHDCNN-RA). The present application first carries out word segmentation, part-of-speech tagging, semantic class labeling to Chinese sentence containing ambiguous words, obtains processed training corpus and test corpus. Then, the word sense disambiguation model is trained using the training corpus, and the optimized MHDCNN-RA model is obtained. On the optimized MHDCNN-RA model, the test corpus is disambiguated, and the weight of the ambiguous word under each semantic category is obtained. The semantic category with the maximum weight is the semantic category of the ambiguous word. The present application realizes good disambiguation of ambiguous words and more accurately judges the true meaning of ambiguous words.

Description

Technical Field

[0001] This invention relates to a Chinese word sense disambiguation method that combines multi-channel hybrid dilated convolution with residuals and attention, which has good applications in the field of natural language processing technology. Background Technology

[0002] Natural language contains a large number of polysemous words. The problem that word sense disambiguation aims to solve is determining which of the multiple meanings of a word should be selected as the correct meaning in a specific context. Word sense disambiguation plays an important role in machine translation, semantic recognition, information retrieval, and other fields.

[0003] Previously, common algorithms were frequently used for word disambiguation, such as K-means, Naive Bayes, association rule-based classification methods, and artificial neural networks. However, traditional algorithms have some drawbacks. The extracted disambiguation features are limited to local regions, requiring manual feature design, which is labor-intensive and slow, and the training effect of the classifier is not very good. In recent years, deep learning algorithms have been widely applied to the field of natural language processing. In the MHDCNN-RA model, multi-channel convolutional neural networks can fully mine the contextual information and text similarity of the corpus. Hybrid dilated convolution effectively improves the grid effect of ordinary dilated convolution and fully captures multi-scale information. Deep convolutional networks are used to increase the expressive power of the model, and residual structures are used to alleviate the gradient vanishing problem of deep neural networks. Multi-head self-attention mechanisms are used to mine the connections between each disambiguation feature to obtain higher word sense disambiguation accuracy. Summary of the Invention

[0004] To address the lexical ambiguity problem in natural language processing, this invention discloses a Chinese word sense disambiguation method combining multi-channel hybrid dilated convolution with residuals and attention. Natural language contains a large number of polysemous words, and word sense disambiguation aims to determine which of the multiple meanings of a word should be selected as the correct meaning in a specific context. Word sense disambiguation plays an important role in machine translation, semantic recognition, information retrieval, and other fields.

[0005] Therefore, the present invention provides the following technical solution:

[0006] A Chinese word sense disambiguation method combining multi-channel hybrid dilated convolution with residuals and attention is proposed. The ambiguous word m has C semantic categories s1, s2, …, s. C The method is characterized by comprising the following steps:

[0007] Step 1: Perform word segmentation, part-of-speech tagging, and semantic class tagging on the training and testing corpora of SemEval-2007:Task#5. Select the word form, part-of-speech tagging, semantic class, and stroke count of the four adjacent word units to the left and right of the ambiguous word m, as well as the four synonyms with the highest similarity between the two adjacent words to the left and right of the ambiguous word m.

[0008] Step 2: Use three word embedding matrices as training data: a randomly initialized word embedding matrix and a word embedding matrix pre-trained using Word2Vec and FastText, extracted from the training corpus of SemEval-2007:Task#5. Use three word embedding matrices as test data: a randomly initialized word embedding matrix and a word embedding matrix pre-trained using Word2Vec and FastText, extracted from the test corpus of SemEval-2007:Task#5.

[0009] Step 3: Optimize the MHDCNN-RA model using the training data to obtain the optimized MHDCNN-RA model.

[0010] Step 4: The testing process is the semantic classification process. On the optimized MHDCNN-RA model, input the test data and calculate the weight of the ambiguous word m under each semantic category. The semantic category with the largest weight is the semantic category of the ambiguous word m.

[0011] In step 1, the training and testing corpora of SemEval-2007:Task#5 are segmented, labeled with parts of speech, and labeled with semantic categories. The word form, part of speech, semantic category, and number of strokes of the four adjacent word units to the left and right of the ambiguous word m, as well as the four synonyms with the highest similarity between the two adjacent words to the ambiguous word m, are selected as disambiguation features. The specific steps are as follows:

[0012] Step 1-1: Use a Chinese word segmentation tool to segment the Chinese sentence into words;

[0013] Steps 1-2: Use Chinese part-of-speech tagging tools to tag the parts of speech of the vocabulary;

[0014] Steps 1-3: Based on the "Synonyms Dictionary", use Chinese semantic annotation tools to perform semantic category annotation on the words;

[0015] Steps 1-4: Extract the word form, part of speech, semantic class, and number of strokes of the four adjacent word units to the left and right of the ambiguous word m, and use the four synonyms with the highest similarity between the two adjacent words to the left and right of the ambiguous word m as disambiguation features.

[0016] In step 2, three word embedding matrices are used as training data: a randomly initialized word embedding matrix and a word embedding matrix pre-trained using Word2Vec and FastText, representing the disambiguation features extracted from the training corpus of SemEval-2007:Task#5. Similarly, three word embedding matrices are used as test data: a randomly initialized word embedding matrix and a word embedding matrix pre-trained using Word2Vec and FastText, representing the disambiguation features extracted from the test corpus of SemEval-2007:Task#5. The specific steps are as follows:

[0017] Step 2-1: Use random initialization to obtain word embedding matrix V1 from the disambiguation features extracted from the training corpus of SemEval-2007:Task#5;

[0018] Step 2-2: Use Word2Vec to obtain the pre-trained word embedding matrix V2 from the disambiguation features extracted from the training corpus of SemEval-2007:Task#5.

[0019] Steps 2-3: Use FastText to obtain the pre-trained word embedding matrix V3 from the disambiguation features extracted from the training corpus of SemEval-2007:Task#5;

[0020] Step 2-4: Use the three word embedding matrices obtained in steps 2-1, 2-2, and 2-3 as training data;

[0021] Steps 2-5: Randomly initialize the disambiguation features extracted from the test corpus of SemEval-2007:Task#5 to obtain the word embedding matrix V1;

[0022] Steps 2-6: Use Word2Vec to obtain the pre-trained word embedding matrix V2 from the disambiguation features extracted from the test corpus of SemEval-2007:Task#5.

[0023] Steps 2-7: Use FastText to obtain the pre-trained word embedding matrix V3 from the disambiguation features extracted from the test corpus of SemEval-2007:Task#5;

[0024] Step 2-8: Use the three word embedding matrices obtained in steps 2-5, 2-6, and 2-7 as test data.

[0025] In step 3, the MHDCNN-RA model is optimized using the training data to obtain the optimized MHDCNN-RA model. The specific steps are as follows:

[0026] Step 3-1 Load the three word embedding matrices of the training data into the input embedding layer of the initialized MHDCNN-RA model as weights to form a three-channel input matrix [V1,V2,V3];

[0027] Step 3-2: After the feature fusion layer, firstly, two-dimensional convolution is used to fuse the three-channel matrix to obtain output Z1. Then, sine encoding is used for the odd positions of Z1, and cosine encoding is used for the even positions to obtain output P. The obtained positional encoding feature P is added to the original feature Z1 to obtain a new fused feature Z2. Finally, one-dimensional convolution is used to compress and fuse the feature matrix Z2 to obtain output Z3. The feature fusion process is as follows:

[0028]

[0029] Where pos represents the index of a disambiguation feature in a set of disambiguation features, and 2i and 2i+1 represent the parity positions of the word vector dimension. In this paper, the word vector dimension d=256, so 2i=[0,2,4,....,254], 2i+1=[1,3,5,.....,255];

[0030] Step 3-3: After passing through a deep convolutional layer, which consists of 12 stacked one-dimensional convolutional blocks, each block has the same structure except for its dilation rate. The two one-dimensional convolutions within each block have the same form (same number and size of kernels), but their weights are not shared. One uses the sigmoid activation function, while the other does not. They are then multiplied bit-by-bit, and padding is used to ensure the output dimension matches the input dimension. The input is also included to form the residual structure, thus avoiding gradient vanishing and allowing information to be transmitted across multiple channels. The calculation process for the one-dimensional convolutional block is as follows:

[0031]

[0032] in It involves element-wise multiplication, where σ is the sigmoid function;

[0033] The stacking of one-dimensional convolutional blocks employs a hybrid dilated convolution scheme, where the dilation rate [1,2,4] is repeated three times, ensuring the convolutional kernel exactly covers the feature matrix Z3. Then, fine-grained tuning is performed using [1,1,1]. After 12 one-dimensional convolutional blocks, the output Z4 is obtained. The deep convolutional layer process is as follows:

[0034]

[0035] Where Conv1D_Block is a one-dimensional convolutional block;

[0036] Steps 3-4: After the normalization layer, Z4 is normalized;

[0037] Steps 3-5: After passing through the multi-head self-attention layer, the relationships between each disambiguation feature are extracted. The multi-head self-attention calculation process is as follows:

[0038]

[0039] Among them W Q W K W V It is a parameter matrix;

[0040] Steps 3-6: After max pooling, the parameters are reduced while preserving the main features;

[0041] Steps 3-7: After the adaptive average pooling layer, output the ambiguous word m in semantic category s. i The weights assigned under the following conditions w(s) i |m), i=1, 2, …, C;

[0042] Step 3-8: Calculate the error loss between the actual output and the expected output using the cross-entropy loss function. The calculation process is as follows:

[0043]

[0044] loss represents the average error on the training data, where n is the number of training data points, and y is the average error on the training data. k This is the label of the kth training data point. Based on backpropagation of the error loss, the parameters are updated layer by layer. The parameter update process is as follows:

[0045]

[0046] Where θ represents the parameter set, θ' represents the updated parameter set, and a is the learning rate;

[0047] Step 3-9: Iterate through steps 3-1 to 3-8 until the set number of iterations is reached to obtain the optimized MHDCNN-RA model.

[0048] In step 4, the testing process is the semantic classification process. On the optimized MHDCNN-RA model, test data is input, and the weight of the ambiguous word *m* under each semantic category is calculated. The semantic category with the highest weight is the semantic category of the ambiguous word. The specific process is as follows:

[0049] Step 4-1 Load the three word embedding matrices of the test data into the input embedding layer of the initialized MHDCNN-RA model as weights to form a three-channel input matrix [V1,V2,V3];

[0050] Step 4-2: After the feature fusion layer, firstly, two-dimensional convolution is used to fuse the three-channel matrix to obtain output Z1. Then, sine encoding is used for the odd positions of Z1, and cosine encoding is used for the even positions to obtain output P. The obtained positional encoding feature P is added to the original feature Z1 to obtain a new fused feature Z2. Finally, one-dimensional convolution is used to compress and fuse the feature matrix Z2 to obtain output Z3. The feature fusion process is as follows:

[0051]

[0052] Where pos represents the index of a disambiguation feature in a set of disambiguation features, and 2i and 2i+1 represent the parity positions of the word vector dimension. In this paper, the word vector dimension d=256, so 2i=[0,2,4,....,254], 2i+1=[1,3,5,.....,255];

[0053] Step 4-3: After passing through a deep convolutional layer, which consists of 12 stacked one-dimensional convolutional blocks, each block has the same structure except for its dilation rate. The two one-dimensional convolutions within each block have the same form (same number and size of kernels), but their weights are not shared. One uses the sigmoid activation function, while the other does not. They are then multiplied bit-by-bit, and padding is used to ensure the output dimension matches the input dimension. The input is also included to form the residual structure, thus avoiding gradient vanishing and allowing information to be transmitted across multiple channels. The calculation process for the one-dimensional convolutional block is as follows:

[0054]

[0055] in It involves element-wise multiplication, where σ is the sigmoid function;

[0056] The stacking of one-dimensional convolutional blocks employs a hybrid dilated convolution scheme, where the dilation rate [1,2,4] is repeated three times, ensuring the convolutional kernel exactly covers the feature matrix Z3. Then, fine-grained tuning is performed using [1,1,1]. After 12 one-dimensional convolutional blocks, the output Z4 is obtained. The convolutional layer process is as follows:

[0057]

[0058] Where Conv1D_Block is a one-dimensional convolutional block;

[0059] Step 4-4: After the normalization layer, Z4 is normalized;

[0060] Steps 4-5: After passing through the multi-head self-attention layer, the relationships between each disambiguation feature are mined. The multi-head self-attention calculation process is as follows:

[0061]

[0062] Among them W Q W K W V It is a parameter matrix;

[0063] Steps 4-6: After max pooling, the parameters are reduced while preserving the main features;

[0064] Steps 4-7: After the adaptive average pooling layer, output the ambiguous word m in semantic category s. i The weights assigned under the following conditions w(s) i |m), i=1, 2, …, C;

[0065] Steps 4-8 output the semantic category with the highest weight, and the process is as follows:

[0066]

[0067] Where s represents the semantic category of the ambiguous word m.

[0068] Beneficial effects:

[0069] 1. This invention is a Chinese word sense disambiguation method that combines multi-channel hybrid dilated convolution with residuals and attention. The method performs lexical segmentation and part-of-speech tagging on Chinese sentences, extracts the stroke count and synonyms of the left and right adjacent words of ambiguous words, and uses a randomly initialized word embedding matrix, a Word2vec pre-trained word embedding matrix, and a FastText pre-trained word embedding matrix. The resulting three-channel word embedding matrix has high quality.

[0070] 2. The model used in this invention is a multi-channel hybrid dilated convolution combined with residual and attention neural networks. Its most significant feature is combining three word embedding matrices into a three-channel input, which can fully exploit the contextual information and text similarity of the corpus. Positional encoding is performed on the feature matrix to add positional information. A hybrid dilated convolution scheme is adopted to solve the grid effect problem caused by individual dilated convolutions, achieving full coverage scanning of the feature matrix to obtain multi-scale information. Deep neural networks are used to increase the model's expressive power, and residual structures are used to effectively alleviate the gradient vanishing problem of deep neural networks. A multi-head self-attention mechanism is used to mine the connections between each disambiguation feature to improve word sense disambiguation accuracy and achieve good classification results.

[0071] 3. The cross-entropy loss function used in this invention contains a softmax classifier, which can not only solve multi-class data processing, but also has an built-in NLLLoss function to calculate the error loss together.

[0072] 4. During model training, Adam gradient descent is used for parameter updates. The error is calculated and propagated back along the original path, starting from the output layer and proceeding through each intermediate hidden layer, updating the parameters of each layer until the output layer is reached. This forward and backward propagation is continuously performed to reduce the error and update the model parameters until the MHDCNN-RA is successfully trained. As the error is continuously backpropagated and the parameters are updated, the disambiguation accuracy of the entire MHDCNN-RA model on the input data improves. Attached Figure Description

[0073] Figure 1 This is a flowchart of Chinese sentence semantic disambiguation in an embodiment of the present invention;

[0074] Figure 2 This describes the training process of the word sense disambiguation model based on MHDCNN-RA in this embodiment of the invention.

[0075] Figure 3 This describes the testing process of the word sense disambiguation model based on MHDCNN-RA in the embodiments of the present invention. Detailed Implementation

[0076] To clearly and completely describe the technical solutions in the embodiments of the present invention, the Chinese sentence "From January 1, 1999, no department or unit may allocate new houses in kind" from the training corpus and test corpus of SemEval-2007:Task#5, which contains the ambiguous word "unit," will be used as an example, and the present invention will be further described in detail with reference to the accompanying drawings in the embodiments. The training corpus contains 55 sentences, and the test corpus contains 19 sentences. The ambiguous word "unit" has two semantic classes: 0: organization, and 1: unit.

[0077] A flowchart of Chinese word sense disambiguation using multi-channel hybrid dilated convolution combined with residuals and attention, as described in this embodiment of the invention, is as follows: Figure 1 As shown, the process includes the following steps. The training process of the word sense disambiguation model based on MHDCNN-RA in this embodiment of the invention is as follows: Figure 2 As shown. The testing process of the word sense disambiguation model based on MHDCNN-RA in this embodiment of the invention is as follows. Figure 3 As shown.

[0078] Step 1: The process of extracting disambiguation features is as follows:

[0079] The feature extraction steps for the Chinese sentence "From January 1, 1999, no department or unit may allocate new houses in kind." are as follows:

[0080] Step 1-1: Use a Chinese word segmentation tool to segment the Chinese sentence into words. The segmentation result is as follows:

[0081] Starting from January 1, 1999, no department or unit may allocate new housing in kind.

[0082] Step 1-2: Use a Chinese part-of-speech tagging tool to tag the words with their parts of speech. The tagging results are as follows:

[0083] From January 1, 1999, no department or unit may allocate new housing in kind.

[0084] Steps 1-3: Based on the "Synonyms Dictionary," use Chinese semantic annotation tools to perform semantic category annotation on the vocabulary:

[0085] From / p / Hi39 year / t / Ca18 month / t / Bd02 day / t / Di02, / f / Kd02 no / r / Eb02 department / n / Di09 unit / n / Di09 may / v / Gc02 further / d / Ig04 allocate / p / Ae10 new house / n / Bn03 in kind / v / Ig03.

[0086] Steps 1-4: Extract the word form, part of speech, semantic class, and stroke count of the four adjacent word units to the left and right of "unit". Based on the "Synonyms Dictionary", extract the four synonyms with the highest similarity between the two adjacent words to "unit" and merge them into a set of disambiguation features:

[0087] No department or agency may, under any circumstances, violate the provisions of Article 13 of the Civil Code.

[0088] Step 2: Obtain the training and test data for the "unit":

[0089] Step 2-1: Use three word embedding matrices as training data: a randomly initialized word embedding matrix and a word embedding matrix pre-trained using Word2Vec and FastText, to process the disambiguation features extracted from the training corpus of SemEval-2007:Task#5.

[0090] Step 2-2: Using the disambiguation features extracted from the test corpus of SemEval-2007:Task#5, three word embedding matrices are used as test data: a randomly initialized word embedding matrix and a word embedding matrix pre-trained using Word2Vec and Fasttext. The results are shown below:

[0091] No department or agency may, under any circumstances, violate the provisions of Article 13 of the Civil Code.

[0092] The word embedding matrix V1 is obtained using random initialization.

[0093]

[0094] The pre-trained word embedding matrix V2 is obtained using Word2Vec.

[0095]

[0096] Use FastText to obtain the pre-trained word embedding matrix V3

[0097]

[0098] Step 3: Use the training data to optimize the MHDCNN-RA model:

[0099] Step 3-1 Load the three word embedding matrices of the training data into the input embedding layer of the initialized MHDCNN-RA model as weights to form a three-channel input matrix [V1,V2,V3];

[0100] Step 3-2: After the feature fusion layer, firstly, two-dimensional convolution is used to fuse the three-channel matrix to obtain output Z1. Then, sine encoding is used for the odd positions of Z2, and cosine encoding is used for the even positions to obtain output P. The obtained position encoding feature P is added to the original feature Z1 to obtain a new fused feature Z2. Finally, one-dimensional convolution is used to compress and fuse the feature matrix Z2 to obtain output Z3. The feature fusion process is as follows:

[0101]

[0102] Where pos represents the index of a disambiguation feature in a set of disambiguation features, and 2i and 2i+1 represent the parity positions of the word vector dimension. In this paper, the word vector dimension d=256, so 2i=[0,2,4,....,254], 2i+1=[1,3,5,.....,255];

[0103] Step 3-3: After passing through a deep convolutional layer, which consists of 12 stacked one-dimensional convolutional blocks, each block has the same structure except for its dilation rate. The two one-dimensional convolutions within each block have the same form (same number and size of kernels), but their weights are not shared. One uses the sigmoid activation function, while the other does not. They are then multiplied bit-by-bit, and padding is used to ensure the output dimension matches the input dimension. The input is also included to form the residual structure, thus avoiding gradient vanishing and allowing information to be transmitted across multiple channels. The calculation process for the one-dimensional convolutional block is as follows:

[0104]

[0105] in It involves element-wise multiplication, where σ is the sigmoid function;

[0106] The stacking of one-dimensional convolutional blocks employs a hybrid dilated convolution scheme, where the dilation rate [1,2,4] is repeated three times, ensuring the convolutional kernel precisely covers the feature matrix Z3. Then, fine-grained tuning is performed using [1,1,1], resulting in the output Z4 after 12 one-dimensional convolutional blocks. The calculation process for the deep convolutional layer is as follows:

[0107]

[0108] Where Conv1D_Block is a one-dimensional convolutional block;

[0109] Steps 3-4: After the normalization layer, Z4 is normalized;

[0110] Steps 3-5: After passing through the multi-head self-attention layer, the relationships between each disambiguation feature are extracted. The multi-head self-attention calculation process is as follows:

[0111]

[0112] Among them W Q W K W V It is a parameter matrix;

[0113] Steps 3-6: After max pooling, the parameters are reduced while preserving the main features;

[0114] Steps 3-7: After the adaptive average pooling layer, output the ambiguous word "unit" in semantic category s. i The weights assigned under the following conditions w(s) i |m), i=1, 2, 0: s1=organization and 1: s2=unit;

[0115] Step 3-8: Calculate the error loss between the actual output and the expected output using the cross-entropy loss function. 单位 The calculation process is as follows:

[0116]

[0117] Based on backpropagation of the error loss, the parameters are updated layer by layer. The parameter update process is as follows:

[0118]

[0119] Where, θ 单位 Represents the parameter set, θ' 单位 This represents the updated parameter set, where 'a' is the learning rate.

[0120] Step 3-9: Iterate through steps 3-1 to 3-8 until the set number of iterations is reached to obtain the optimized MHDCNN-RA model;

[0121] Step 4: Semantically classify the ambiguous word "unit":

[0122] Step 4-1 Load the three word embedding matrices of the test data into the input embedding layer of the initialized MHDCNN-RA model as weights to form a three-channel input matrix [V1,V2,V3];

[0123] Step 4-2: After the feature fusion layer, firstly, two-dimensional convolution is used to fuse the three-channel matrix to obtain output Z1. Then, sine encoding is used for the odd positions of Z2, and cosine encoding is used for the even positions to obtain output P. The obtained positional encoding feature P is added to the original feature Z1 to obtain a new fused feature Z2. Finally, one-dimensional convolution is used to compress and fuse the feature matrix Z2 to obtain output Z3. The feature fusion process is as follows:

[0124]

[0125] Where pos represents the index of a disambiguation feature in a set of disambiguation features, and 2i and 2i+1 represent the parity positions of the word vector dimension. In this paper, the word vector dimension d=256, so 2i=[0,2,4,....,254], 2i+1=[1,3,5,.....,255];

[0126] Step 4-3: After passing through a deep convolutional layer, which consists of 12 stacked one-dimensional convolutional blocks, each block has the same structure except for its dilation rate. The two one-dimensional convolutions within each block have the same form (same number and size of kernels), but their weights are not shared. One uses the sigmoid activation function, while the other does not. They are then multiplied bit-by-bit, and padding is used to ensure the output dimension matches the input dimension. The input is also included to form the residual structure, thus avoiding gradient vanishing and allowing information to be transmitted across multiple channels. The calculation process for the one-dimensional convolutional block is as follows:

[0127]

[0128] in It involves element-wise multiplication, where σ is the sigmoid function;

[0129] The stacking of one-dimensional convolutional blocks employs a hybrid dilated convolution scheme, where the dilation rate [1,2,4] is repeated three times, ensuring the convolutional kernel precisely covers the feature matrix Z3. Then, fine-grained tuning is performed using [1,1,1], resulting in the output Z4 after 12 one-dimensional convolutional blocks. The calculation process for the deep convolutional layer is as follows:

[0130]

[0131] Where Conv1D_Block is a one-dimensional convolutional block;

[0132] Step 4-4: After the normalization layer, Z4 is normalized;

[0133] Steps 4-5: After passing through the multi-head self-attention layer, the relationships between each disambiguation feature are mined. The multi-head self-attention calculation process is as follows:

[0134]

[0135] Among them W Q W K W V It is a parameter matrix;

[0136] Steps 4-6: After max pooling, the parameters are reduced while preserving the main features;

[0137] Steps 4-7: After the adaptive average pooling layer, output the weights w = [w(s1|unit), w(s2|unit)] = [1.6917, 7.5621] assigned to the ambiguous word "unit" under semantic categories 0: s1=organization and 1: s2=unit.

[0138] Step 4-8 Output the semantic category with the highest weight, as follows:

[0139]

[0140] s2=unit indicates the semantic category corresponding to the ambiguous word "unit".

[0141] The optimized MHDCNN-RA model was used to disambiguate the Chinese sentence "From January 1, 1999, no department or unit may allocate new houses in kind." containing the ambiguous word "unit." The semantic category corresponding to the ambiguous word "unit" was "unit." Experiments verified that the accuracy of the test corpus containing the ambiguous word "unit" on the optimized MHDCNN-RA model reached 84.21%.

[0142] The Chinese word semantic disambiguation method using multi-channel hybrid dilated convolution combined with residuals and attention in the embodiments of the present invention can select precise disambiguation features and use a neural network of multi-channel hybrid dilated convolution combined with residuals and attention to determine the semantic category of ambiguous words.

[0143] The above description, in conjunction with the accompanying drawings, provides a detailed account of the embodiments of the present invention. The specific implementation methods described herein are merely illustrative of the invention. Those skilled in the art will recognize that variations and modifications can be made to the specific implementation methods and applications based on the principles of this invention; therefore, this document should not be construed as limiting the scope of the invention.

Claims

1. A Chinese word sense disambiguation method combining multi-channel hybrid dilated convolution with residuals and attention, where ambiguous words m have C semantic categories s1, s2, …, s C Its characteristics are, The method includes the following steps: Step 1: Perform word segmentation, part-of-speech tagging, and semantic class tagging on the training and testing corpora of SemEval-2007:Task#5. Select the word form, part-of-speech tagging, semantic class, and number of strokes of the four adjacent word units to the left and right of the ambiguous word m, as well as the four synonyms with the highest similarity between the two adjacent words to the left and right of the ambiguous word m. Step 2: Use three word embedding matrices as training data: a randomly initialized word embedding matrix and a word embedding matrix pre-trained using Word2Vec and FastText, extracted from the training corpus of SemEval-2007:Task#5. Use three word embedding matrices as test data: a randomly initialized word embedding matrix and a word embedding matrix pre-trained using Word2Vec and FastText, extracted from the test corpus of SemEval-2007:Task#5. Step 3: Optimize the MHDCNN-RA model using training data to obtain the optimized MHDCNN-RA model. The MHDCNN-RA model is a multi-channel hybrid dilated convolution combined with residual and attention model, which includes an input embedding layer, a feature fusion layer, a deep convolutional layer, a normalization layer, a multi-head self-attention layer, a max pooling layer, and an adaptive average pooling layer. The specific steps for optimizing the MHDCNN-RA model are as follows: Step 3-1 Load the three word embedding matrices of the training data into the input embedding layer of the initialized MHDCNN-RA model as weights to form a three-channel input matrix [V1,V2,V3]; Step 3-2: After the feature fusion layer, firstly, two-dimensional convolution is used to fuse the three-channel matrix to obtain output Z1. Then, sine encoding is used for the odd positions of Z1, and cosine encoding is used for the even positions to obtain output P. The obtained positional encoding feature P is added to the original feature Z1 to obtain a new fused feature Z2. Finally, one-dimensional convolution is used to compress and fuse the feature matrix Z2 to obtain output Z3. The feature fusion process is as follows: Where pos represents the index of a disambiguation feature in a set of disambiguation features, and 2i and 2i+1 represent the parity positions of the word vector dimension. In this paper, the word vector dimension d=256, so 2i=[0,2,4,....,254], 2i+1=[1,3,5,.....,255]; Step 3-3: After passing through a deep convolutional layer, which consists of 12 stacked one-dimensional convolutional blocks, each block has the same structure except for its dilation rate. The two one-dimensional convolutions within each block have the same form (same number and size of kernels), but their weights are not shared. One uses the sigmoid activation function, while the other does not. They are then multiplied bit by bit, and padding is used to ensure the output dimension matches the input dimension. The input is also included to form the residual structure, thus avoiding gradient vanishing and enabling information to be transmitted across multiple channels. The calculation process for the one-dimensional convolutional block is as follows: in It involves element-wise multiplication, where σ is the sigmoid function; The stacking of one-dimensional convolutional blocks adopts a hybrid dilated convolution scheme, that is, the dilation rate [1,2,4] is repeated three times so that the convolutional kernel just covers the feature matrix Z3. Then, fine-grained tuning is performed using [1,1,1]. After 12 one-dimensional convolutional blocks, the output Z4 is obtained. The deep convolutional layer process is as follows: Where Conv1D_Block is a one-dimensional convolutional block; Steps 3-4: After the normalization layer, Z4 is normalized; Steps 3-5 involve using a multi-head self-attention layer to uncover the relationships between each disambiguation feature. The multi-head self-attention calculation process is as follows: Among them W Q W K W V It is a parameter matrix; Steps 3-6: After max pooling, the parameters are reduced while preserving the main features; Steps 3-7: After the adaptive average pooling layer, output the ambiguous word m in semantic category s. i The weights assigned under the following conditions w(s) i |m), i=1, 2, …, C; Step 3-8: Calculate the error loss between the actual output and the expected output using the cross-entropy loss function. The calculation process is as follows: loss represents the average error on the training data, where n is the number of training data points, and y is the average error on the training data. k Here is the label of the kth training data point. Based on backpropagation of the error loss, the parameters are updated layer by layer. The parameter update process is as follows: Where θ represents the parameter set, θ' represents the updated parameter set, and a is the learning rate; Step 3-9: Iterate through steps 3-1 to 3-8 until the set number of iterations is reached to obtain the optimized MHDCNN-RA model; Step 4: The testing process is the semantic classification process. On the optimized MHDCNN-RA model, input the test data and calculate the weight of the ambiguous word m under each semantic category. The semantic category with the largest weight is the semantic category of the ambiguous word m.

2. The Chinese word sense disambiguation method combining multi-channel hybrid dilated convolution with residuals and attention as described in claim 1, characterized in that, In step 1, the training and testing corpora of SemEval-2007:Task#5 are segmented, labeled with parts of speech, and labeled with semantic categories. The word form, part of speech, semantic category, and number of strokes of the four adjacent word units to the left and right of the ambiguous word m, as well as the four synonyms with the highest similarity between the two adjacent words to the ambiguous word m, are selected as disambiguation features. The specific steps are as follows: Step 1-1: Use a Chinese word segmentation tool to segment the Chinese sentence into words; Steps 1-2: Use Chinese part-of-speech tagging tools to tag the parts of speech of the words; Steps 1-3: Based on the "Synonyms Dictionary", use Chinese semantic annotation tools to perform semantic category annotation on the words; Steps 1-4: Extract the word form, part of speech, semantic class, and number of strokes of the four adjacent word units to the left and right of the ambiguous word m, and use the four synonyms with the highest similarity between the two adjacent words to the left and right of the ambiguous word m as disambiguation features.

3. The Chinese word meaning disambiguation method combining multi-channel hybrid dilated convolution with residuals and attention as described in claim 1, characterized in that, In step 2, three word embedding matrices are used as training data: a randomly initialized word embedding matrix and a word embedding matrix pre-trained using Word2Vec and FastText, representing the disambiguation features extracted from the training corpus of SemEval-2007:Task#5. Similarly, three word embedding matrices are used as test data: a randomly initialized word embedding matrix and a word embedding matrix pre-trained using Word2Vec and FastText, representing the disambiguation features extracted from the test corpus of SemEval-2007:Task#5. The specific steps are as follows: Step 2-1: Use random initialization to obtain word embedding matrix V1 from the disambiguation features extracted from the training corpus of SemEval-2007:Task#5; Step 2-2: Use Word2Vec to obtain the pre-trained word embedding matrix V2 from the disambiguation features extracted from the training corpus of SemEval-2007:Task#5. Steps 2-3: Use FastText to obtain the pre-trained word embedding matrix V3 from the disambiguation features extracted from the training corpus of SemEval-2007:Task#5; Step 2-4: Use the three word embedding matrices obtained in steps 2-1, 2-2, and 2-3 as training data; Steps 2-5: Randomly initialize the disambiguation features extracted from the test corpus of SemEval-2007:Task#5 to obtain the word embedding matrix V1; Steps 2-6: Use Word2Vec to obtain the pre-trained word embedding matrix V2 from the disambiguation features extracted from the test corpus of SemEval-2007:Task#5. Steps 2-7: Use FastText to obtain the pre-trained word embedding matrix V3 from the disambiguation features extracted from the test corpus of SemEval-2007:Task#5; Step 2-8: Use the three word embedding matrices obtained in steps 2-5, 2-6, and 2-7 as test data.

4. The Chinese word meaning disambiguation method based on multi-channel hybrid dilated convolution combined with residuals and attention as described in claim 1, characterized in that, In step 4, the testing process is the semantic classification process. On the optimized MHDCNN-RA model, test data is input, and the weight of the ambiguous word *m* under each semantic category is calculated. The semantic category with the highest weight is the semantic category of the ambiguous word. The specific process is as follows: Step 4-1 Load the three word embedding matrices of the test data into the input embedding layer of the initialized MHDCNN-RA model as weights to form a three-channel input matrix [V1,V2,V3]; Step 4-2: After the feature fusion layer, firstly, two-dimensional convolution is used to fuse the three-channel matrix to obtain output Z1. Then, sine encoding is used for the odd positions of Z1, and cosine encoding is used for the even positions to obtain output P. The obtained positional encoding feature P is added to the original feature Z1 to obtain a new fused feature Z2. Finally, one-dimensional convolution is used to compress and fuse the feature matrix Z2 to obtain output Z3. The feature fusion process is as follows: Where pos represents the index of a disambiguation feature in a set of disambiguation features, and 2i and 2i+1 represent the parity positions of the word vector dimension. In this paper, the word vector dimension d=256, so 2i=[0,2,4,....,254], 2i+1=[1,3,5,.....,255]; Step 4-3: After passing through a deep convolutional layer, which consists of 12 stacked one-dimensional convolutional blocks, each block has the same structure except for its dilation rate. The two one-dimensional convolutions within each block have the same form (same number and size of kernels), but their weights are not shared. One uses the sigmoid activation function, while the other does not. They are then multiplied bit by bit, and padding is used to ensure the output dimension matches the input dimension. The input is also included to form the residual structure, thus avoiding gradient vanishing and enabling information to be transmitted across multiple channels. The calculation process for the one-dimensional convolutional block is as follows: in It involves element-wise multiplication, where σ is the sigmoid function; The stacking of one-dimensional convolutional blocks adopts a hybrid dilated convolution scheme, that is, the dilation rate [1,2,4] is repeated three times so that the convolutional kernel just covers the feature matrix Z3. Then, fine-grained tuning is performed using [1,1,1]. After 12 one-dimensional convolutional blocks, the output Z4 is obtained. The convolutional layer process is as follows: Where Conv1D_Block is a one-dimensional convolutional block; Step 4-4: After the normalization layer, Z4 is normalized; Steps 4-5: After passing through the multi-head self-attention layer, the relationships between each disambiguation feature are mined. The multi-head self-attention calculation process is as follows: Among them W Q W K W V It is a parameter matrix; Steps 4-6: After max pooling, the parameters are reduced while preserving the main features; Steps 4-7: After the adaptive average pooling layer, output the ambiguous word m in semantic category s. i The weights assigned under the following conditions w(s) i |m), i=1, 2, …, C; Steps 4-8 output the semantic category with the highest weight, and the process is as follows: Where s represents the semantic category of the ambiguous word m.

Images