Multi-modal dynamic residual error Chinese text spelling error correction method and system
By adopting the multimodal dynamic residual method in the Chinese text spelling correction model, the audio and glyph features of wrong Chinese characters are used to integrate attention mechanisms, which solves the misleading problem of existing models when dealing with shaped characters or pronunciation characters, and achieves higher error correction accuracy and recall rate.
Patent Information
- Application Number
- CN202411847836.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-16
- Publication Date
- 2025-03-28
AI Technical Summary
The existing Chinese text spelling error correction model is easily misleading by the word embedding of wrong Chinese characters when dealing with characters that are close to shape or pronunciation, and fails to effectively utilize the correlation information between the shape and pronunciation of the same Chinese character, resulting in insufficient error correction accuracy.
The multimodal dynamic residual method is adopted to extract the audio features and glyph features of the wrong Chinese characters and perform fusion based on attention mechanism to obtain the multimodal representation features of the wrong Chinese characters, which are used to replace the residual embedding features of the wrong Chinese characters in the initial residual embedding matrix, thereby reducing the risk of misleading error information and improving error correction accuracy.
By using multimodal information to correct the traditional residual structure, the risk of the model being misled by error information is reduced, and the accuracy and recall rate of Chinese text spelling error correction are improved.
Smart Images
Figure CN119721012A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of natural language processing technology, and specifically relates to a multimodal dynamic residual Chinese text spelling correction method and system. Background Art
[0002] Chinese Spelling Check (CSC), as a basic task in natural language processing, aims to detect and correct Chinese character spelling errors in texts due to confusion in pronunciation or glyphs of some Chinese characters. These spelling errors mainly come from writing errors or machine recognition errors. Early Chinese spelling correction mainly relied on rule-based and statistical methods, but these methods have the problem of difficulty in processing complex language structures and understanding their semantics. With the rapid development of artificial intelligence technology, the Chinese text spelling correction model based on pre-training technology has achieved excellent results. It can detect and correct spelling errors in articles more accurately and comprehensively based on context and semantics.
[0003] The current Chinese text spelling correction model uses the convolutional neural network ResNet to promote convergence, but it has shortcomings in the CSC task: Since one of the characteristics of the CSC task is that most wrong characters are similar in shape or pronunciation, there will be no problem with the word embedding superposition of the correct characters in the conventional residual module. However, for wrong characters, simple word embedding superposition will hinder and mislead the model's reasoning, and fusing multimodal information for correct Chinese characters will hinder the model's reasoning. In addition, previous Chinese text spelling correction models have also failed to pay attention to the correlation information between the shape and pronunciation of the same Chinese character. Summary of the invention
[0004] The purpose of this application is to provide a multimodal dynamic residual Chinese text spelling correction method and system. The method and system of this application fully explore and utilize the correlation information between different modalities of the same Chinese character, reduce the misleading of the model by erroneous Chinese characters, and reduce the impact of multimodal information on correct Chinese characters, thereby further improving the accuracy of error correction.
[0005] The first aspect of the present application provides a dynamic residual Chinese text spelling correction method, comprising:
[0006] S100: performing error location and masking processing on the sentence to obtain the erroneous Chinese characters, the embedding features of the erroneous Chinese characters and the mask vector of the sentence, and taking the original embedding features of the sentence as the initial residual embedding matrix;
[0007] S200: extracting audio features and glyph features of the erroneous Chinese characters and fusing them based on the attention mechanism to obtain multimodal representation features of the erroneous Chinese characters, and using the multimodal representation features to replace the residual embedding features of the erroneous Chinese characters in the initial residual embedding matrix;
[0008] S300: Encode using the mask vector of the sentence, classify the encoding result in combination with the current residual embedding feature, and output the corrected sentence.
[0009] Furthermore, the error location and masking processing of the sentence in step S100 includes:
[0010] S110: Obtain the error probability of each Chinese character position in the sentence using an error detection model;
[0011] S120: Obtaining a position index of an erroneous Chinese character according to the error probability, and obtaining the erroneous Chinese character and its corresponding embedded feature according to the position index;
[0012] S130: Masking the original embedding features of the sentence using the error probability of each Chinese character position in the sentence.
[0013] Furthermore, the audio features of the incorrect Chinese characters are extracted in step S200, including:
[0014] Pre-build an audio coding table for Chinese characters, convert the erroneous Chinese characters into corresponding audio codes based on the audio coding table, and then use a convolution module to extract the audio features of the erroneous Chinese characters from the audio codes;
[0015] The audio coding table is constructed by converting Chinese characters into audio, and segmenting the audio into corresponding audio codes.
[0016] Furthermore, the step S200 extracts the glyph features of the erroneous Chinese characters, including:
[0017] A glyph encoding table of Chinese characters is pre-constructed, and the erroneous Chinese characters are converted into corresponding glyph encodings through the glyph encoding table, and then the glyph encodings are input into the Bert encoder to obtain the glyph features of the erroneous Chinese characters.
[0018] Furthermore, the glyph encoding is a concatenation of the character's four-corner encoding and the Wubi sequence.
[0019] Furthermore, the fusion based on the attention mechanism described in step S200 includes:
[0020] The fusion based on the attention mechanism is a fusion based on a parallel cross attention fusion network; the parallel cross attention fusion network includes two parallel feature fusion modules, namely an audio feature fusion module and a glyph feature fusion module; the audio feature fusion module and the glyph feature fusion module each include several layers of attention submodules, each attention submodule includes a multi-head attention layer and a residual normalization layer;
[0021] Firstly, the bilinear pooling and nonlinear activation methods are used to obtain the joint representation features of audio features and glyph features. Then, the audio feature fusion module and the glyph feature fusion module fuse the audio features and glyph features in parallel. Finally, the fused audio features and glyph features are concatenated with the embedded features corresponding to the erroneous Chinese characters, and then the multimodal representation features of the erroneous Chinese characters are obtained through linear transformation and nonlinear fusion.
[0022] Furthermore, the audio feature fusion module and the character feature fusion module fuse the audio features and the character features in parallel, including:
[0023] In the audio feature fusion module, for the first layer of attention submodule, it receives the audio feature F a and encode; for the j-th layer attention submodule, it receives the output and joint representation features of the j-1-th layer attention submodule and encodes them; the N-th layer attention submodule outputs the fused audio feature F a ';
[0024] In the glyph feature fusion module, for the first layer of attention submodule, it receives the glyph feature F f and encode; for the j-th layer attention submodule, it receives the output and joint representation features of the j-1-th layer attention submodule and encodes them; the N-th layer attention submodule outputs the fused glyph feature F f ';
[0025] Among them, j is 2, 3, ... N, N is the number of layers of the attention submodule; Q, K, V represent the query matrix, key matrix and value matrix respectively.
[0026] Specifically, in the audio feature fusion module, the audio feature F a As Q, K and V, it is input into the first layer attention submodule. For the j-th layer attention submodule, the output of the j-1-th layer attention submodule is input into the j-th layer attention submodule as Q, and the joint representation feature C is input into the j-th layer attention submodule as K and V.
[0027] In the glyph feature fusion module, the glyph feature F f The first-layer attention sub-module is input as Q, K and V; for the j-th layer attention sub-module, the output of the j-1-th layer attention sub-module is input as Q into the j-th layer attention sub-module, and the joint representation feature C is input as K and V into the j-th layer attention sub-module.
[0028] The second aspect of the present application provides a dynamic residual Chinese text spelling correction system, comprising:
[0029] The error detection module is used to locate and mask the errors in the sentence, obtain the erroneous Chinese characters, the embedding features of the erroneous Chinese characters and the mask vector of the sentence, and use the original embedding features of the sentence as the initial residual embedding matrix;
[0030] The multimodal dynamic residual module extracts the audio features and glyph features of the incorrect Chinese characters and fuses them based on the attention mechanism to obtain the multimodal representation features of the incorrect Chinese characters. The multimodal representation features are used to replace the residual embedding features of the incorrect Chinese characters in the initial residual embedding matrix.
[0031] The error correction module is used to encode the sentence using the mask vector, classify the encoding result based on the current residual embedding feature, and output the corrected sentence.
[0032] Furthermore, a parallel cross-attention fusion network is used in the multimodal dynamic residual module to perform fusion based on the attention mechanism;
[0033] The parallel cross attention fusion network includes two parallel feature fusion modules, namely an audio feature fusion module and a glyph feature fusion module; the audio feature fusion module and the glyph feature fusion module each include several layers of attention submodules;
[0034] In the audio feature fusion module, the first-layer attention submodule is used to receive and encode audio features, and the j-th-layer attention submodule is used to receive and encode the output and joint representation features of the j-1-th-layer attention submodule;
[0035] In the glyph feature fusion module, the first-layer attention submodule is used to receive and encode glyph features, and the j-th-layer attention submodule is used to receive and encode the output and joint representation features of the j-1-th-layer attention submodule;
[0036] The joint representation feature is a joint representation feature of audio features and glyph features, which is obtained by fusing the audio features and glyph features using a bilinear pooling and nonlinear activation method;
[0037] Among them, j is 2, 3, ... N respectively, and N represents the number of layers of the attention submodule in the audio feature fusion module and the glyph feature fusion module.
[0038] The advantages and beneficial effects of this application are as follows:
[0039] The application uses the multimodal information of the wrong word to correct the traditional residual structure, and by enhancing the model's intervention on the wrong Chinese characters, it reduces the risk of the model being misled by the wrong information, thereby improving the accuracy of error correction. In the preferred scheme, the application uses a parallel cross-attention fusion network to fuse audio features and glyph features, focusing on the correlation between different modalities of phono-semantic characters, which can further improve the accuracy of error correction. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 A schematic diagram of a Chinese text spelling correction model constructed in an embodiment of the present application;
[0041] Figure 2 Schematic diagram of an error detection module in an embodiment of the present application;
[0042] Figure 3 A schematic diagram of a parallel cross-attention fusion module in an embodiment of the present application;
[0043] Figure 4 Schematic diagram of the principle of the multimodal dynamic residual process in the embodiment of the present application. DETAILED DESCRIPTION
[0044] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0045] See also Figure 1 , which shows the Chinese text spelling correction model constructed in the embodiment of the present application, which mainly includes:
[0046] The error detection module is used to locate and mask the errors in the sentence, obtain the erroneous Chinese characters, the embedding features of the erroneous Chinese characters and the mask vector of the sentence, and use the original embedding features of the sentence as the initial residual embedding matrix;
[0047] The multimodal dynamic residual module is used to extract the audio features and glyph features of the incorrect Chinese characters and fuse them based on the attention mechanism to obtain the multimodal representation features of the incorrect Chinese characters. The multimodal representation features are used to replace the residual embedding features of the incorrect Chinese characters in the initial residual embedding matrix.
[0048] The error correction module is used to encode the sentence using the mask vector, classify the encoding result based on the current residual embedding feature, and output the corrected sentence.
[0049] The specific implementation process of the method of the present application is provided below in conjunction with the accompanying drawings, including:
[0050] S100: Perform error location and masking processing on the sentence to obtain the erroneous Chinese characters, the embedding features of the erroneous Chinese characters and the mask vector of the sentence, and use the original embedding features of the sentence as the initial residual embedding matrix.
[0051] Specifically, locating and masking wrong Chinese characters in a sentence further includes:
[0052] S110: Obtain the error probability p of each Chinese character position in the sentence i , p i ∈[0,1];
[0053] In this specific implementation, a sequence labeling model is selected as the error detection model, and the error probability is obtained using the sequence labeling model. The sequence labeling model can select an LSTM model or an ON_LSTM model. Sentences are input into the sequence labeling model. The sequence labeling model performs linear mapping and activation function on the sentences, and then outputs the error probability of each Chinese character position.
[0054] S220: Obtain the position index of the erroneous Chinese character according to the error probability, and obtain the erroneous Chinese character and its corresponding embedded feature according to the position index
[0055] In this specific implementation, the position index Pos of the erroneous Chinese characters is screened using the error threshold η, which is recorded as Pos=[Pos1, Pos2, ..., Pos n ], n represents the number of Chinese characters in the sentence;
[0056] Position indexPos i The calculation of is as follows:
[0057] Pos i =Filter(p i >η) (1)
[0058] Use the following formula to obtain the embedding features corresponding to each incorrect Chinese character
[0059]
[0060] In formula (1) to (2):
[0061] E represents the original embedding features of the sentence;
[0062] η is the error threshold, which is pre-set and generally takes a value in the range of [0.5, 1];
[0063] Filter represents the filtering operation, which is used to select the part exceeding the threshold η;
[0064] Mask represents a mask operation, which is used to extract the embedding features corresponding to the incorrect Chinese characters;
[0065] X represents the Chinese characters in the sentence, Indicates incorrect Chinese characters.
[0066] See also Figure 2, as shown in a specific example, the sentence "Today is a good day", the error probability of each Chinese character position is [0.1, 0.3, 0.8, 0.9, 0.2]. Using formula (1), the position index Pos = [0, 0, 1, 1, 0] of the wrong Chinese character is filtered, where the position corresponding to "1" is the position of the wrong Chinese character. The position index Pos = [0, 0, 1, 1, 0] of the wrong Chinese character is masked with the original embedded feature E of the sentence, that is, the embedded feature corresponding to the wrong Chinese character is extracted.
[0067] S130: Masking the original embedding features of the sentence using the error probability of each Chinese character position in the sentence;
[0068] In this specific implementation, a soft mask is selected for masking. Specifically, the error probability p of each Chinese character position is i The original embedding features E = [e1, e2, ..., e n ] are input into the soft mask module together, and the mask vector is obtained through linear operation. Among them, n represents the number of Chinese characters in the sentence.
[0069] The soft mask processing method is as follows:
[0070]
[0071] In formula (3):
[0072] e mask Represents the mask vector, which is preset;
[0073] e i Represents the original embedding feature of the i-th Chinese character;
[0074] Represents the mask feature of the i-th Chinese character.
[0075] S200: extracting audio features and glyph features of the erroneous Chinese characters and fusing them based on the attention mechanism to obtain multimodal representation features of the erroneous Chinese characters, and using the multimodal representation features to replace the residual embedding features of the erroneous Chinese characters in the initial residual embedding matrix.
[0076] This step is implemented by the multimodal dynamic residual module, which performs two major functions: extracting the multimodal representation features of the erroneous Chinese characters and correcting the initial residual embedding matrix using the multimodal representation features.
[0077] In this specific implementation, the audio features of Chinese characters are generated using the audio spectrum of Chinese characters. Specifically, an audio coding table of Chinese characters is pre-built, and the erroneous Chinese characters are converted into corresponding audio codes based on the audio coding table, and then the audio features of the erroneous Chinese characters are extracted from the audio codes using a convolution module, which is denoted as Fa The audio coding table is constructed by converting Chinese characters into audio, segmenting the audio and converting it into corresponding audio codes, thereby obtaining the audio codes of the Chinese characters.
[0078] In this specific implementation, the four-corner code and / or five-stroke sequence of Chinese characters are used to represent the glyph features of Chinese characters. In the preferred embodiment, the four-corner code and the five-stroke sequence of Chinese characters are spliced together as the glyph code of Chinese characters to improve the glyph feature representation capability. Specifically, a glyph code table of Chinese characters is pre-constructed, and the erroneous Chinese characters are converted into corresponding glyph codes through the glyph code table, and then the glyph codes are input into the Bert encoder to obtain the glyph features of the erroneous Chinese characters, which are recorded as F f .
[0079] In order to further explore the internal correlation between the audio and glyph features of Chinese characters and obtain more complete and deeper Chinese character features, in this specific implementation, a parallel cross attention fusion network based on the staggered attention mechanism is proposed to fuse the audio features and glyph features of incorrect Chinese characters. The structure and process of the parallel cross attention fusion network can be found in Figure 3 , the following will combine Figure 3 A detailed description of the parallel criss-cross attention fusion network is given.
[0080] The parallel cross-attention fusion network proposed in this application includes two parallel feature fusion modules, namely an audio feature fusion module and a glyph feature fusion module; each feature fusion module includes several layers of attention sub-modules, and each attention sub-module constitutes a basic module of the feature fusion module, and each includes a multi-head attention layer and a residual normalization layer.
[0081] The specific workflow of the parallel cross-attention fusion network is as follows:
[0082] First, the audio feature F is obtained using bilinear pooling and nonlinear activation methods. a and glyph feature F f The joint characterization feature C:
[0083] C = tanh(F a WF f ) (4)
[0084] In formula (4):
[0085] Tanh represents the activation function;
[0086] W is a learnable weight matrix, which is first randomly initialized and then continuously learned and updated during the training process.
[0087] Next, the audio feature fusion module and the glyph feature fusion module fuse the audio features and the glyph features in parallel.
[0088] In the audio feature fusion module, for the first layer of attention submodule, it receives the audio feature F a And encode; for the j-th layer attention submodule, it receives the output of the j-1-th layer attention submodule and the joint representation feature C and encodes it, j is 2, 3, ... N in sequence; the N-th layer attention submodule outputs the fused audio feature F a '; Specifically, for the first layer attention submodule, the audio feature F a As Q, K and V input into the first layer of attention submodule; for the j-th layer of attention submodule, the output of the j-1-th layer of attention submodule is used as Q input into the j-th layer of attention submodule, and the joint representation feature C is used as K and V input into the j-th layer of attention submodule, and j is 2, 3,...N respectively.
[0089] In the glyph feature fusion module, for the first layer of attention submodule, it receives the glyph feature F f And encode; for the j-th layer attention submodule, it receives the output of the j-1-th layer attention submodule and the joint representation feature C and encodes it, j is 2, 3, ... N in turn; the N-th layer attention submodule outputs the fused glyph feature F f '; Specifically, for the first layer of attention submodule, the glyph feature F f As Q, K and V input into the first layer of attention submodule; for the j-th layer of attention submodule, the output of the j-1-th layer of attention submodule is used as Q input into the j-th layer of attention submodule, and the joint representation feature C is used as K and V input into the j-th layer of attention submodule, and j is 2, 3,...N respectively.
[0090] It should be noted that Q, K, and V represent the query matrix, key matrix, and value matrix, respectively.
[0091] In the audio feature fusion module and the glyph feature fusion module, the first layer of attention submodule is a self-attention submodule, and the other layers of attention submodules are cross-attention modules.
[0092] Finally, the fused audio feature F a ' and the glyph feature F f 'Embedding features corresponding to incorrect Chinese characters Concatenate, then use linear transformation and nonlinear fusion to obtain the multimodal representation features of incorrect Chinese characters
[0093]
[0094] In formula (5), W' and b' are learnable parameters.
[0095] The specific operation mode of the parallel cross attention fusion network can be seen in the following formula:
[0096]
[0097] In formulas (6) to (9):
[0098] j represents the layer number, and N represents the total number of layers of the audio feature fusion module and the glyph feature fusion module;
[0099] MH a&n Represents the basic module in the Transformer model consisting of a multi-head attention layer and a residual normalization layer;
[0100] They represent the query matrix, key matrix, and value matrix of the jth layer from bottom to top in the audio feature fusion module respectively;
[0101] They represent the query matrix, key matrix, and value matrix of the j-1th layer from bottom to top in the audio feature fusion module respectively;
[0102] They represent the query matrix, key matrix and value matrix of the jth layer from bottom to top in the glyph feature fusion module respectively;
[0103] They respectively represent the query matrix, key matrix and value matrix of the j-1th layer from bottom to top in the glyph feature fusion module.
[0104] The residual module in the deep neural network is used to promote convergence and alleviate the gradient vanishing phenomenon, but the use of traditional residual connections in the CSC task may mislead the model because the embedded feature vectors corresponding to the incorrect Chinese characters in the input sentence will hinder the model's understanding and prediction of the correct characters in the residual connection. To solve this technical problem, the present application proposes a multimodal dynamic residual, that is, using the multimodal representation features of the incorrect Chinese characters to dynamically adjust the residual embedding matrix when the model is updated, so that the model is no longer limited to the features of the incorrect Chinese characters themselves, but can further focus on the features of the shape and audio of the incorrect Chinese characters themselves, further promoting model convergence.
[0105] See also Figure 4 , which shows the principle of the multi-modal dynamic residual process in the embodiment of the present application, and will be combined with Figure 4 Describe in detail the dynamic update process of the residual embedding matrix.
[0106] Figure 4 middle, represents the multimodal representation features of the wrong Chinese characters, E represents the initial residual embedding matrix before the residual connection, and Update E, that is, use the multimodal representation features of the wrong Chinese characters to replace the residual embedding features corresponding to the wrong Chinese characters in E. The blue squares represent the residual embedding features of the correct Chinese characters, and the red squares represent the updated residual embedding features of the wrong Chinese characters.
[0107] The residual embedding matrix is updated using formula (10):
[0108]
[0109] In formula (10):
[0110] E' represents the updated residual embedding matrix;
[0111] Scatter represents a mapping function. In this specific implementation, Scatter is a mapping function in the pytorch deep learning development framework.
[0112] S300: Encode using the mask vector of the sentence, classify the encoding result in combination with the current residual embedding feature, and output the corrected sentence.
[0113] In this specific implementation, the error correction model selects the Bert model. Specifically, the mask vector after masking is Input into the Bert model, the output state of the last layer of the Bert model is recorded as H C , H C represents the feature vector output by the Bert model after encoding the mask vector, which contains the sentence information after the error is corrected; then H C It is superimposed with the current residual embedding matrix E' and then connected through the residual to obtain the hidden state H':
[0114] H'=H C +E' (11)
[0115] H' is classified using a linear layer and a nonlinear activation function, and the word with the highest probability at each Chinese character position is selected to form the final output sentence.
[0116] When training the model, in each round of training, the multimodal dynamic residual module iteratively updates the residual embedding matrix as follows:
[0117] Pre-set the total number of training rounds epochs and batch size batch_size, and traverse the training data sequentially according to the batch size batch_size;
[0118] The sample sentence input error detection module performs error location and mask processing on the sample sentence to obtain the erroneous Chinese characters, the embedding features of the erroneous Chinese characters and the mask vector of the sample sentence, and uses the original embedding features of the sample sentence as the initial residual embedding matrix;
[0119] The multimodal dynamic residual module extracts and fuses the audio features and glyph features of the incorrect Chinese characters to obtain the multimodal representation features of the incorrect Chinese characters. The multimodal representation features are then used to replace the residual embedding features of the incorrect Chinese characters in the initial residual embedding matrix, while the residual embedding features of the correct Chinese characters remain unchanged.
[0120] The pre-trained error correction model is used to predict the mask vector of the sample sentence, and combined with the updated residual embedding matrix for classification, and the corrected sentences of this batch are output.
[0121] Calculate the loss value, backpropagate, and use the gradient descent method to update the weight parameters of the entire model.
[0122] In the early stages of training, the error detection module cannot accurately locate the errors in the sentences, but as training progresses, the error location will gradually converge to the position of the actual erroneous Chinese characters. In each round of training, the correction position of the residual embedding matrix of the same batch of sample sentences may be different, and the multimodal representation features of the erroneous Chinese characters are also constantly learned and updated. As the training progresses, the error position in the sentence can be accurately located, and the correction position of the residual embedding matrix tends to be stable.
[0123] The optimal values of all hyperparameters involved in this application can be determined through simulation experiments. In some specific implementations, it is found that the error threshold η is preferably 0.6 and the total number of layers N is preferably 4.
[0124] Example
[0125] This example uses the bert-base-chinese model on huggingface to initialize the weights of the error correction Bert model, and uses the AdamW optimizer at 1.0e -410 epochs (number of training rounds) were trained at a learning rate of , and the batch size for training and prediction was set to 48. At the same time, PaddleSpeech was used to convert all Chinese characters into audio, and wavTokenizer was used to segment each audio at a sampling rate of 1600Hz and convert it into a Chinese character audio encoding table. The 1600Hz sampling rate can limit the audio encoding length of a single Chinese character to less than 1000, and a smaller sampling rate makes it easier to perceive the similarity between phonemes. The four-corner code and five-stroke sequence of each Chinese character are concatenated and completed into a sequence of length 10, and then converted into a Chinese character glyph encoding table. The convolutional layer of the feature extraction of wav2vec2.0 is used as an audio feature extractor, and a 4-layer Transformer encoder is used as a Chinese character glyph feature extractor. In this embodiment, the error threshold η is set to 0.6, the total number of layers N is set to 4, and the remaining parameters use default parameters. All experiments were completed on an NVIDIAA100 with 40G memory.
[0126] Compared with the best indicator in the baseline model, the model of this application leads by 1.2% and 1.03% in the F1 scores of the sighan14 and sighan15 datasets, respectively, thus proving the effectiveness of this model. Specifically, compared with the multimodal model PHMO of the same type, this application leads by 10.6%, 2.6%, and 0.7% in error correction accuracy, and leads by 0.3%, 5.2%, and 7.1% in recall rate, respectively. Although this model uses an adaptive method to fuse multimodal knowledge, its internal residual structure hinders the reasoning of the model. At the same time, its linear fusion of multimodal features fails to effectively utilize the correlation between speech and glyph features, so its overall effect is not as good as this application.
[0127] This application uses the multimodal representation features of erroneous Chinese characters to correct the classic residual structure, and reduces the risk of the model being misled by erroneous information by enhancing the intervention of erroneous Chinese characters. At the same time, this application proposes a parallel cross-attention fusion mechanism Paca to deeply explore the connection between the glyph features and audio features of the phono-semantic characters, which account for the largest proportion in Chinese, and further improve the model's error correction effect by focusing on the correlation between different modalities of phono-semantic characters.
[0128] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.
Claims
1. A multimodal dynamic residual Chinese text spelling correction method, characterized in that: include: S100: performing error location and masking processing on the sentence to obtain the erroneous Chinese characters, the embedding features of the erroneous Chinese characters and the mask vector of the sentence, and taking the original embedding features of the sentence as the initial residual embedding matrix; S200: extracting audio features and glyph features of the erroneous Chinese characters and fusing them based on the attention mechanism to obtain multimodal representation features of the erroneous Chinese characters, and using the multimodal representation features to replace the residual embedding features of the erroneous Chinese characters in the initial residual embedding matrix; S300: Encode using the mask vector of the sentence, classify the encoding result in combination with the current residual embedding feature, and output the corrected sentence.
2. The multimodal dynamic residual Chinese text spelling correction method according to claim 1, characterized in that: The error location and masking process of the sentence in step S100 further includes: S110: using the error detection model to obtain the error probability of each Chinese character position in the sentence; S120: Obtaining a position index of an erroneous Chinese character according to the error probability, and obtaining the erroneous Chinese character and its corresponding embedded feature according to the position index; S130: Masking the original embedding features of the sentence using the error probability of each Chinese character position in the sentence.
3. The multimodal dynamic residual Chinese text spelling correction method according to claim 1, characterized in that: Step S200 extracts the audio features of the incorrect Chinese characters, including: Pre-build an audio coding table for Chinese characters, convert the erroneous Chinese characters into corresponding audio codes based on the audio coding table, and then use a convolution module to extract the audio features of the erroneous Chinese characters from the audio codes; The audio coding table is constructed by converting Chinese characters into audio, and segmenting the audio into corresponding audio codes.
4. The multimodal dynamic residual Chinese text spelling correction method according to claim 1, characterized in that: Step S200 extracts the glyph features of the erroneous Chinese characters, including: A glyph encoding table of Chinese characters is pre-constructed, and the erroneous Chinese characters are converted into corresponding glyph encodings through the glyph encoding table, and then the glyph encodings are input into the Bert encoder to obtain the glyph features of the erroneous Chinese characters.
5. The multimodal dynamic residual Chinese text spelling correction method as claimed in claim 4, characterized in that: The glyph encoding is a combination of the four-corner encoding of the character and the five-stroke sequence.
6. The multimodal dynamic residual Chinese text spelling correction method according to claim 1, characterized in that: The fusion based on the attention mechanism described in step S200 includes: The fusion based on the attention mechanism is a fusion based on a parallel cross attention fusion network; the parallel cross attention fusion network includes two parallel feature fusion modules, namely an audio feature fusion module and a glyph feature fusion module; the audio feature fusion module and the glyph feature fusion module each include several layers of attention submodules, each attention submodule includes a multi-head attention layer and a residual normalization layer; Firstly, the bilinear pooling and nonlinear activation methods are used to obtain the joint representation features of audio features and glyph features. Then, the audio feature fusion module and the glyph feature fusion module fuse the audio features and glyph features in parallel. Finally, the fused audio features and glyph features are concatenated with the embedded features corresponding to the erroneous Chinese characters, and then the multimodal representation features of the erroneous Chinese characters are obtained through linear transformation and nonlinear fusion.
7. The multimodal dynamic residual Chinese text spelling correction method according to claim 6, characterized in that: The audio feature fusion module and the character feature fusion module fuse the audio features and the character features in parallel, further comprising: In the audio feature fusion module, for the first layer of attention submodule, it receives the audio feature F a and encode; for the j-th layer attention submodule, it receives the output and joint representation features of the j-1-th layer attention submodule and encodes them; the N-th layer attention submodule outputs the fused audio feature F a '; In the glyph feature fusion module, for the first layer of attention submodule, it receives the glyph feature F f and encode; for the j-th layer attention submodule, it receives the output and joint representation features of the j-1-th layer attention submodule and encodes them; the N-th layer attention submodule outputs the fused glyph feature F f '; Among them, j is 2, 3, ... N, N is the number of layers of the attention submodule; Q, K, V represent the query matrix, key matrix and value matrix respectively.
8. The multi-modal dynamic residual Chinese text spelling correction method according to claim 7, characterized in that: In the audio feature fusion module, the audio feature F a As Q, K and V, it is input into the first layer attention submodule. For the j-th layer attention submodule, the output of the j-1-th layer attention submodule is input into the j-th layer attention submodule as Q, and the joint representation feature C is input into the j-th layer attention submodule as K and V. In the glyph feature fusion module, the glyph feature F f The first-layer attention sub-module is input as Q, K and V; for the j-th layer attention sub-module, the output of the j-1-th layer attention sub-module is input as Q into the j-th layer attention sub-module, and the joint representation feature C is input as K and V into the j-th layer attention sub-module.
9. A multimodal dynamic residual Chinese text spelling correction system, characterized by: include: The error detection module is used to locate and mask the errors in the sentence, obtain the erroneous Chinese characters, the embedding features of the erroneous Chinese characters and the mask vector of the sentence, and use the original embedding features of the sentence as the initial residual embedding matrix; The multimodal dynamic residual module extracts the audio features and glyph features of the incorrect Chinese characters and fuses them based on the attention mechanism to obtain the multimodal representation features of the incorrect Chinese characters. The multimodal representation features are used to replace the residual embedding features of the incorrect Chinese characters in the initial residual embedding matrix. The error correction module is used to encode the sentence using the mask vector, classify the encoding result based on the current residual embedding feature, and output the corrected sentence.
10. The multi-modal dynamic residual Chinese text spelling correction system according to claim 9, characterized in that: The multimodal dynamic residual module uses a parallel cross-attention fusion network to perform fusion based on the attention mechanism; The parallel cross attention fusion network includes two parallel feature fusion modules, namely an audio feature fusion module and a glyph feature fusion module; Both the audio feature fusion module and the glyph feature fusion module include several layers of attention submodules; In the audio feature fusion module, the first-layer attention submodule is used to receive and encode audio features, and the j-th-layer attention submodule is used to receive and encode the output and joint representation features of the j-1-th-layer attention submodule; In the glyph feature fusion module, the first-layer attention submodule is used to receive and encode glyph features, and the j-th-layer attention submodule is used to receive and encode the output and joint representation features of the j-1-th-layer attention submodule; The joint representation feature is a joint representation feature of audio features and glyph features, which is obtained by fusing the audio features and glyph features using a bilinear pooling and nonlinear activation method; Among them, j is 2, 3, ... N respectively, and N represents the number of layers of the attention submodule in the audio feature fusion module and the glyph feature fusion module.