Improved Transform Blind-Chinese conversion method based on knowledge distillation and continuous writing pre-training

Abstract

The invention relates to an improved Transform blind-Chinese conversion method based on knowledge distillation and continuous writing pre-training, and belongs to the technical field of natural language processing. The method comprises the following steps: 1, performing data cleaning on a blind Chinese contrast data set by using a Braille reconstruction byte pair segmentation model so as to obtain a pre-training set; 2, constructing an encoder model by using relative distance mask mapping, and transmitting the pre-training set to the encoder model for pre-training by using a knowledge distillation method; 3, constructing a decoder based on a bidirectional screening attention concentration network, and connecting the pre-trained encoder and decoder in series to form an improved Transform network; and carrying out iterative training on the improved Transform network by adopting a self-defined norm loss function and a blind Chinese contrast data set until the precision preset by the user is reached. By adopting the method, the translation accuracy of the blind-Chinese conversion model can be effectively improved, and the complexity of model conversion can be effectively reduced.

Description

Technical Field

[0001] The invention relates to a blind-Chinese conversion method of an improved Transformer based on knowledge distillation and connected writing pre-training, and belongs to the technical field of natural language processing. Background Art

[0002] As an important representative of the disabled population, the blind constitute a large group. Their living and health conditions are one of the focuses that need urgent attention in the "Healthy China" initiative. my country has the largest number of blind people in the world, accounting for 20% of the total number of blind people in the world. The effective development of digital Braille work will greatly improve the quality of life of the blind and enhance their sense of integration in society. Therefore, it is imperative to promote the digitization of Braille. Improving the promotion and social recognition of Braille is the solution, making it a powerful tool for blind learning. Digital Braille provides a more convenient and efficient way of learning for the blind, which helps them to receive better education. In addition, digital technology can also help the blind better integrate into society and participate in various social and cultural activities through channels such as the Internet. In general, devoting ourselves to the digitization of Braille is an important step in promoting the full integration of the blind into society and improving their quality of life. Today, the Braille-Chinese conversion methods are mainly divided into two categories: rule-based and large model-based. The former has the disadvantage of insufficient accuracy, while the latter has too many model parameters, which is not conducive to offline installation on edge devices. Therefore, how to improve the translation accuracy of the Braille-Chinese conversion model and reduce the complexity of model conversion has become an urgent problem to be solved. Summary of the invention

[0003] The purpose of the present invention is to improve the translation accuracy of the braille-Chinese conversion model and reduce the complexity of the model conversion. An improved Transformer braille-Chinese conversion method based on knowledge distillation and continuous writing pre-training is proposed.

[0004] The objective of the present invention is achieved through the following technical solutions:

[0005] The improved Transformer braille-Chinese conversion method based on knowledge distillation and continuous writing pre-training of the present invention is applied to a braille reconstruction byte pair segmentation model, wherein the braille reconstruction byte pair segmentation model includes an encoder based on byte segmentation and a transcoder based on hardened soft output; and includes the following steps:

[0006] Step 1: Use the Braille reconstruction byte pair segmentation model to clean the Braille-Chinese comparison dataset to obtain a pre-training set;

[0007] Step 1.1: The Chinese word segmentation of the original Braille-Chinese comparison dataset is converted into the Braille unit ASCII code with the label Y j =(y1,y2,...,y n), the processed Braille-Chinese comparison data set is used to train the Braille reconstruction byte pair segmentation model, and the Braille sentence X j =(x1,x2,...,x n ) is input into the network, and flag is used as an identifier to distinguish the parts of the network participating in the training, thereby obtaining a Braille reconstruction byte pair segmentation model.

[0008] Step 1.2: Use padding (filling placeholders) to align the data of the maximum length n of the Braille sentences, and use one-hot encoding on the original Braille sentences to form a Braille sentence matrix;

[0009] Step 1.3: Use the encoder-transcoder structure to reconstruct the input χ into the output Y; define the encoder as a BERT consisting of a 6-layer Transformer-Encoder as a TE network. When the flag is 0, input the Braille sentence into the TE network, and use the placeholder index loss function shown in formula (1) to iteratively train the TE network until the user-preset accuracy is reached and the flag is set to 1;

[0010]

[0011] Among them, χ j =TE(X j ) is the output after the input χ passes through the encoder, and λ is the balance parameter; represents the weighted sum of the elements in the first column of the calculation matrix, φ(X j ) represents the search x j Matrix, when the first column element value of a row is the largest, insert the row into X j The corresponding positions of are used to ensure that the dimensions of the two matrices are consistent.

[0012] Step 1.4: Define the transcoder network as a TD network using a 6-layer Transformer-Decoder, and define the network consisting of the TE network and the TD network in series as a TE-TD network; when the flag is 1, input the Braille sentence into the TE-TD network, and use the Hamming loss function based on hard value soft output shown in formula (2) to iteratively train the TE-TD network until the user-preset accuracy is reached and the training is terminated; use the TE network as the final byte pair segmentation model;

[0013]

[0014] Among them, ||·|| h represents the Hamming distance that enhances the decision power of the network, μ and v represent the regularization parameters that balance the learning weights of the network, and the γ(·) function is defined as shown in formula (3) and is used to harden the soft labels;

[0015]

[0016] Step 1.5: Extract Braille sentences from the training set to form X as shown in (4) i , the label obtained by the byte pair segmentation model will be X i Each word in is segmented into byte pairs to construct a pre-training set as shown in formula (5);

[0017]

[0018] Among them, m is the maximum length of the sentence, and the insufficient part is filled with padding;

[0019] Step 2: Use the relative distance mask mapping to build an encoder model, define the encoder model as a series of improved Encoder modules specified by the user, and define each Encoder module as containing a relative distance mask attention network and a learnable parameter residual feedforward network; use the knowledge distillation method to transfer the pre-training set described in step 1 to the encoder model for pre-training.

[0020] Step 2.1: Use the relative distance mask attention module shown in formula (6) to obtain the intermediate vector shown in formula (7);

[0021]

[0022] Where X = {x1, x2, ..., x n}, is the Hadamard product; Attention (Q, K, V) encodes the relationship weights between the row vectors of the matrix, giving the network the ability to pay attention to Braille information in both directions, Ψ(X T X)=[a ij ], Ψ(·) represents the relative distance mapping function It is represented as a punctuation information mask matrix, which is defined as shown in formula (8). Punlist is a punctuation table. The purpose of masking X here is to shield the influence of padding and punctuation on the attention score.

[0023]

[0024] Step 2.2: Connect the intermediate vector shown in formula (7) to the grouped feedforward connection network shown in formula (9) to encode and integrate the attention scores;

[0025]

[0026] Among them, g is the number of custom hyperparameter groups, and m is the number of batches. The split function is used to Split into g vectors, and the concat function is used to merge g vectors into one vector.

[0027] Step 2.3: Use the residual method shown in formula (10) to construct a learnable parameter residual connection module; adjust the upper and lower bounds of the residual through hyperparameters, and dynamically balance the connection weights through iteration to protect the Braille encoding information;

[0028]

[0029] Among them, χ k and are the vectors of steps 2.2 and 2.1 respectively, Φ(input) represents the generalized function represented by the attention layer and the grouped feedforward layer, γ1 and γ2 are learnable parameters, u and s are user-defined hyperparameters, u is the upper bound, and s is the lower bound;

[0030] Step 2.4: Define the Braille reconstruction byte pair segmentation model described in step 1 as the teacher model as TN(·). Define the encoder described in step 2 as the student model as SN(·), and use the knowledge distillation method shown in formula (11) to pre-train the student model using the non-uniform cross distillation loss function shown in formula (12);

[0031]

[0032] Where T is the user-preset distillation temperature. The dimension of W is v×Vcb_size, Vcb_size is the size of the dictionary created based on the training set, and W is used to convert the output of the student model into a probability distribution with a dimension of Vcb_size; α1 and α2 are both balancing factors, α1 represents the hard label loss, and α2 is used to prevent the output probability density from tending to a uniform distribution;

[0033] Step 3: Construct a decoder based on a bidirectional screening attention network, connect the encoder and decoder pre-trained in steps 1 and 2 in series to form an improved Transformer network; and use a custom norm loss function and a blind-Chinese comparison dataset to iteratively train the improved Transformer network until the user's preset accuracy is achieved;

[0034] Step 3.1: Save the encoder student model parameters after the knowledge distillation described in step 2; and define the input Braille sentence as X as shown in formula (13) i , the encoder output is defined as shown in equation (14) Where i represents the i-th sentence in the blind training set;

[0035]

[0036] Step 3.2: Build a decoder to convert the braille sentence X i and encoder output The corresponding correct Chinese translation is as shown in formula (15): Will As the decoder input to the relative distance mask attention module shown in formula (16), the attention score matrix shown in formula (17) is used to capture X i The attention vector of the dependency weight relationship between word vectors;

[0037]

[0038] in, is the Hadamard product, Mask is the lower triangular masking matrix used to mask future information during training. Mask is defined as shown in formula (18), ψ is the relative distance mapping function,

[0039] Step 3.3: Use the grouped linear transformation residual connection module shown in formula (19) Perform integrated coding to form L as shown in formula (20) i ;

[0040]

[0041] Among them, BN γ,β It is the batch normalization operation;

[0042]

[0043] Step 3.4: Output the encoder With L i Through the two-way screening of the attention concentration network for fusion, the attention is further distributed to the sparse matrix using equations (21) and (22) to obtain the maximum weight max(γ1,γ2), thereby enhancing the network's decision-making power, and the tightness value is obtained using equation (23);

[0044]

[0045] Among them, γ1,γ2 are learnable parameters, The screening function is defined as shown in formula (22);

[0046]

[0047] The θ function is defined as shown in (23);

[0048]

[0049] Among them, θ(M) is the tightness function of the matrix element. The larger the calculated tightness value is, the sparser the attention weight distribution of the matrix element will be after the Softmax function is applied.

[0050] Step 3.5: Use the tightness value as input and obtain the prediction vector shown in formula (25) through formula (24)

[0051]

[0052] Where k represents the probability that the kth word in the ith sentence is predicted to be the word with index v in the vocabulary, and the predicted word with the maximum probability is selected as the th word of the Chinese sentence for network translation, thereby achieving translation;

[0053] Step 3.6: Use the custom norm loss function shown in formula (26) to predict the vector and Learning after normalization;

[0054]

[0055] Among them, ||·|| F is the Frobenius norm, ||·|| P for P norm, κ is a regularization hyperparameter used to enhance the network's decisiveness and prevent the output from tending to a uniform distribution;

[0056] Beneficial effects:

[0057] The present invention discloses a Braille-Chinese conversion method based on an improved Transformer with knowledge distillation and continuous writing pre-training. The method is based on a byte pair segmentation model of Braille reconstruction, adopts an inverse information reconstruction network, and does not require a separate data set construction. Since the model is based on a neural network, segmentation can be achieved based on abstract feature extraction, overcoming the problem that the traditional BPE segmentation network is too dependent on corpus distribution and requires manual adjustment of the number of segmentations.

[0058] The present invention discloses a method for converting braille to Chinese using an improved Transformer based on knowledge distillation and pre-training of connected writing. The method is based on a pre-trained network of relative distance mask mapping, and the relative distance mapping module introduces relative distance weights in the calculation of the Attention matrix. Since Braille and Chinese have the same word order, the traditional method of embedding absolute distance vectors in the Attention matrix will distract the network's attention to unnecessary complex semantic relationships, while ignoring the syllable modeling problem within the same word unit. The relative distance weight can force the network to pay attention to more adjacent tokens, thereby making the network pay more attention to words from the same braille connected writing unit. Different from traditional residual connections, residual connections based on learnable parameters with upper and lower bounds can be customized by users to avoid infinite problems, and the network can dynamically balance the residual weights to protect shallow or deep information.

[0059] The present invention discloses a method for converting braille to Chinese based on an improved transformer based on knowledge distillation and pre-training of continuous writing. The method is based on a network pre-training step for restoring continuous writing information. The traditional pre-training step can help the network to quickly model semantic information, but cannot help the network learn the braille continuous writing rules. The pre-training step is more targeted for the braille to Chinese conversion task. The output of the encoder and the decoder are fused by a bidirectional screening attention concentration network. Compared with the general attention score calculation module, a larger weight can be given to the key-value pairs with more concentrated matrix elements, thereby concentrating the network attention. And by using a custom norm loss function, the regularization term of this loss can be customized by the user with a p-norm to avoid the network output from being uniformly distributed, thereby enhancing the network's decisiveness. Thus, the translation accuracy of the braille to Chinese conversion model is effectively improved and the complexity of the model conversion is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 The byte pair segmentation model based on Braille reconstruction of the present invention;

[0061] Figure 2 is the coding layer structure of the present invention;

[0062] Figure 3 The pre-training model based on restoration of continuous writing information of the present invention;

[0063] Figure 4 A decoder based on attention screening of the present invention;

[0064] Figure 5 It is the Braille embedding vector heat map of the present invention;

[0065] Figure 6 This is a visualization diagram of the attention score matrix of the first-layer improved Transformer-Encoder of the present invention;

[0066] Figure 7This is a visualization diagram of the attention score matrix of the sixth layer improved Transformer-Encoder of the present invention;

[0067] Figure 8 Output heat map of the improved Transformer encoder of the present invention;

[0068] Fig. 9 This is the sixth layer of the improved Transformer-Decoder encoding semantic vector heat map of the present invention. DETAILED DESCRIPTION

[0069] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the protection scope of the present invention.

[0070] Example

[0071] The present invention provides a method for converting blind Chinese into Chinese using an improved Transformer based on knowledge distillation and continuous writing pre-training, and the specific implementation steps are as follows:

[0072] Step 1: Use the Braille reconstruction byte pair segmentation model to clean the Braille-Chinese comparison dataset to obtain a pre-training set;

[0073] Step 1.1: The Chinese word segmentation of the original Braille-Chinese comparison dataset is converted into the Braille unit ASCII code with the label Y j =(y1,y2,...,y n ), the processed Braille-Chinese comparison data set is used to train the Braille reconstruction byte pair segmentation model, and the Braille sentence X j =(x1,x2,...,x n ) is input into the network, and flag is used as an identifier to distinguish the parts of the network participating in the training, thereby obtaining a Braille reconstruction byte pair segmentation model.

[0074] Step 1.2: Use padding (filling placeholders) to align the data of the maximum length n of the Braille sentences, and use one-hot encoding on the original Braille sentences to form a Braille sentence matrix;

[0075] In the embodiment, a Braille-Chinese comparison data set is used to build and train a Braille reconstruction segmentation model, which can provide a byte pair splitting function for a pre-trained model based on continuous writing information restoration. The final output dimension of the attention layer and the grouped linear transformation layer participating in the subsequent network calculation is set to 160, the number of attention heads is set to 8, and the Batch_Size is set to 128; the input one-hot vector dimension is set to the number of dictionary characters in the statistical data set, and the dictionary is counted by the unsegmented data results.

[0076] Step 1.3: Use the encoder-transcoder structure to reconstruct the input χ into the output Y; define the encoder as a BERT consisting of a 6-layer Transformer-Encoder as a TE network. When the flag is 0, input the Braille sentence into the TE network, and use the placeholder index loss function shown in formula (1) to iteratively train the TE network until the user-preset accuracy is reached and the flag is set to 1;

[0077]

[0078] Among them, χ j =TE(X j ) is the output after the input χ passes through the encoder, and λ is the balance parameter; represents the weighted sum of the elements in the first column of the calculation matrix, φ(X j ) represents the search x j Matrix, when the first column element value of a row is the largest, insert the row into X j The corresponding positions of are used to ensure that the dimensions of the two matrices are consistent.

[0079] In the embodiment, the encoder adopts a TE network composed of 6 layers of Transformer-Encoder. When the flag is 0, the TE network is first iterated based on the F-norm loss of the all-zero row splicing matrix until the loss is lower than 0.5, and the flag is set to 1.

[0080] Step 1.4: Define the transcoder network as a TD network using a 6-layer Transformer-Decoder, and define the network consisting of the TE network and the TD network in series as a TE-TD network; when the flag is 1, input the Braille sentence into the TE-TD network, and use the Hamming loss function based on hard value soft output shown in formula (2) to iteratively train the TE-TD network until the user-preset accuracy is reached and the training is terminated; use the TE network as the final byte pair segmentation model;

[0081]

[0082] Among them, ||·|| hrepresents the Hamming distance that enhances the decision power of the network, μ and v represent the regularization parameters that balance the learning weights of the network, and the γ(·) function is defined as shown in formula (3) and is used to harden the soft labels;

[0083]

[0084] In the embodiment, the transcoder uses a TD network composed of 6 layers of Transformer-Encoder. In the transcoder, the mutual attention operation between the TE network and the TD network is located in the first layer of Transformer-Encoder. When the flag is 1, the Hamming loss based on hard value soft output is used to update all network parameters. The process is as follows Figure 1 As shown in the figure, the number of training rounds for the entire network is EPOCH=20, the initial learning rate is set to 1, and it is reduced by half every 20 rounds of iterations. The training time is 4243 seconds, and the final loss is 0.343.

[0085] Step 1.5: Extract Braille sentences from the training set to form X as shown in (4) i , the label obtained by the byte pair segmentation model will be X i Each word in is segmented into byte pairs to construct a pre-training set as shown in formula (5);

[0086]

[0087] Among them, m is the maximum length of the sentence, and the insufficient part is filled with padding;

[0088] In the embodiment, in the test part after training, the braille reconstruction and segmentation model is first tested, and the braille ASCII code sentence "tgihw2d9`7`k~"~2hw2 d9`b 8k~"2" is input. In order to avoid ambiguous characters, the re library in Python is used to escape some special characters, "'" is escaped to "`", and " / " is escaped to "~". First, the braille reconstruction and segmentation model is tested, and the ASCII code sentence is segmented into "tg ihw2d9`7`k~"~2hw2d9`b 8k~"2" after the braille reconstruction and segmentation model.

[0089] Step 2: Use the relative distance mask mapping to build an encoder model, define the encoder model as a series of improved Encoder modules specified by the user, and define each Encoder module as containing a relative distance mask attention network and a learnable parameter residual feedforward network; use the knowledge distillation method to transfer the pre-training set described in step 1 to the encoder model for pre-training.

[0090] Step 2.1: Use the relative distance mask attention module shown in formula (6) to obtain the intermediate vector shown in formula (7);

[0091]

[0092] Where X = {x1, x2, ..., x n}, is the Hadamard product; Attention (Q, K, V) encodes the relationship weights between the row vectors of the matrix, giving the network the ability to pay attention to Braille information in both directions, Ψ(X T X)=[a ij ], Ψ(·) represents the relative distance mapping function It is represented as a punctuation information mask matrix, which is defined as shown in formula (8). Punlist is a punctuation table. The purpose of masking X here is to shield the influence of padding and punctuation on the attention score.

[0093]

[0094] In the embodiment, an encoder based on relative distance mask mapping is used for pre-training, the padding placeholder index is 0, and the unregistered word index is 1. The number of encoder layers is set to 6. This module needs to ensure that the input braille embedding dimension and the braille dictionary are consistent with those set in step 1, and the one-hot embedding layer also directly uses the parameters saved in step 1. The output dimension of the conversion matrix is ​​consistent with the length of the Chinese corpus dictionary.

[0095] Step 2.2: Connect the intermediate vector shown in formula (7) to the grouped feedforward connection network shown in formula (9) to encode and integrate the attention scores;

[0096]

[0097] Among them, g is the number of custom hyperparameter groups, and m is the number of batches. The split function is used to Split into g vectors, and the concat function is used to merge g vectors into one vector.

[0098] In the embodiment, the dropout ratio is 0.1, and the maximum sentence length MAX_LENGTH is 40. The optimizer uses NoamOpt, in which the learning rate reduction factor factor is 0.6, the warmup factor warmup is 1000, and when the accuracy reaches 0.9, the training is terminated and the pre-training model parameters are saved.

[0099] Step 2.3: Use the residual method shown in formula (10) to construct a learnable parameter residual connection module; adjust the upper and lower bounds of the residual through hyperparameters, and dynamically balance the connection weights through iteration to protect the Braille encoding information;

[0100]

[0101] Among them, χ k and are the vectors of steps 2.2 and 2.1 respectively, Φ(input) represents the generalized function represented by the attention layer and the grouped feedforward layer, γ1 and γ2 are learnable parameters, u and s are user-defined hyperparameters, u is the upper bound, and s is the lower bound;

[0102] In the embodiment, the byte pair segmentation model obtained in step 1 is used as the teacher network to perform knowledge distillation on the encoder. The embedding vector passes through a total of 6 encoder layers, and the structure of each encoder layer is as follows: Figure 2 As shown in the figure. Ψ(·) is the relative distance mapping function, which can implicitly encode distance information and control the attention weight. The visualization of the attention score matrix obtained by the relative distance mask attention module in the first layer Encoder is shown in the figure Figure 6 The visualization of the attention score matrix obtained by the relative distance mask attention module in the last layer of Encoder is shown in Figure 7 The final encoder output is as follows Figure 8 shown.

[0103] Step 2.4: Define the Braille reconstruction byte pair segmentation model described in step 1 as the teacher model as TN(·). Define the encoder described in step 2 as the student model as SN(·), and use the knowledge distillation method shown in formula (11) to pre-train the student model using the non-uniform cross distillation loss function shown in formula (12);

[0104]

[0105] Where T is the user-preset distillation temperature. The dimension of W is v×Vcb_size, Vcb_size is the size of the dictionary created based on the training set, and W is used to convert the output of the student model into a probability distribution with a dimension of Vcb_size; α1 and α2 are both balancing factors, α1 represents the hard label loss, and α2 is used to prevent the output probability density from tending to a uniform distribution;

[0106] In the embodiment, the Braille dictionary created during the training process is used to convert the Braille ASCII code sentence "tgihw2d9`7`k~"~2hw2d9`b 8k~"2" into the ID number "[2,6,68,24,26,251,32,24,26,252,3]", where the first dimension and the last dimension are the IDs of the start character "BOS" and the end character "EOS". The ID sentence is input into the embedding layer of the trained network, and the embedding layer is completely consistent with the previously trained reconstruction segmentation network parameters to obtain an embedding vector, and its heat map is as follows Figure 5 shown.

[0107] Step 3: Construct a decoder based on a bidirectional screening attention network, connect the encoder and decoder pre-trained in steps 1 and 2 in series to form an improved Transformer network; and use a custom norm loss function and a blind-Chinese comparison dataset to iteratively train the improved Transformer network until the user's preset accuracy is achieved;

[0108] Step 3.1: Save the encoder student model parameters after the knowledge distillation described in step 2; and define the input Braille sentence as X as shown in formula (13) i , the encoder output is defined as shown in equation (14) Where i represents the i-th sentence in the blind training set;

[0109]

[0110] In the embodiment, Figure 3 As shown in the figure, the pre-trained model parameters are loaded, and the decoder based on attention screening is connected after the Encoder layer to start formal training. The optimizer uses NoamOpt, where the learning rate reduction factor is 0.5, the warmup factor is 1000, the EPOCH is set to 100, the Batch_Size is set to 256, the final training time is 15923Sec, and the final loss is 0.0436.

[0111] Step 3.2: Build a decoder to convert the braille sentence X i and encoder output The corresponding correct Chinese translation is as shown in formula (15): Will As the decoder input to the relative distance mask attention module shown in formula (16), the attention score matrix shown in formula (17) is used to capture X i The attention vector of the dependency weight relationship between word vectors;

[0112]

[0113] in, is the Hadamard product, Mask is the lower triangular masking matrix used to mask future information during training. Mask is defined as shown in formula (18), Ψ is the relative distance mapping function,

[0114] In the embodiment, Figure 4 As shown in Figure 1, the decoder first consists of 6 layers of decoders and a bidirectional filtering attention network. Each layer of decoder contains a relative distance mask attention module and a grouped linear transformation residual module. The heat map of the encoded semantic vector obtained after the decoder input passes through the 6 layers of decoders is shown in Figure 1. Fig. 9 shown.

[0115] Step 3.3: Use the grouped linear transformation residual connection module shown in formula (19) Perform integrated coding to form L as shown in formula (20) i ;

[0116]

[0117] Among them, BN γ,β It is the batch normalization operation;

[0118]

[0119] Step 3.4: Output the encoder With L i Through the two-way screening of the attention concentration network for fusion, the attention is further distributed to the sparse matrix using equations (21) and (22) to obtain the maximum weight max(γ1,γ2), thereby enhancing the network's decision-making power, and the tightness value is obtained using equation (23);

[0120]

[0121] Among them, γ1,γ2 are learnable parameters, The screening function is defined as shown in formula (22);

[0122]

[0123] The θ function is defined as shown in (23);

[0124]

[0125] Among them, θ(M) is the tightness function of the matrix element. The larger the calculated tightness value is, the sparser the attention weight distribution of the matrix element will be after the Softmax function is applied.

[0126] Step 3.5: Use the tightness value as input and obtain the prediction vector shown in formula (25) through formula (24)

[0127]

[0128] Where k represents the probability that the kth word in the i-th sentence is predicted to be the word with index v in the vocabulary, and the predicted word with the maximum probability is selected as the kth word of the Chinese sentence for network translation, thereby achieving translation;

[0129] Step 3.6: Use the custom norm loss function shown in formula (26) to predict the vector and Learning after normalization;

[0130]

[0131] Among them, ||·|| F is the Frobenius norm, ||·|| P for P norm, κ is a regularization hyperparameter used to enhance the network's decisiveness and prevent the output from tending to a uniform distribution;

[0132] In the embodiment, the encoded semantic vector and the encoder output are integrated through a bidirectional screening attention network. Different from the traditional mutual attention mechanism, the network can dynamically select the objects of the query value matrix and the key value matrix, and give more weight to the matrix with low tightness to enhance the decisiveness of the final output. The final network output is "He can play both tennis and baseball."

[0133] The results show that the blind-Chinese conversion method of the improved Transformer based on knowledge distillation and continuous writing pre-training of the present invention reduces the computational complexity while having superior performance.

Claims

1. An improved Transformer method for converting braille to Chinese based on knowledge distillation and continuous writing pre-training, characterized by: The following steps are included: Step 1: Use the Braille reconstruction byte pair segmentation model to clean the Braille-Chinese comparison dataset to obtain a pre-training set; Step 2: Construct an encoder model using relative distance mask mapping, define the encoder model as a series of improved Encoder modules specified by the user, and define each Encoder module as a network containing a relative distance mask attention network and a residual feedforward network with learnable parameters; use the knowledge distillation method to transfer the pre-training set described in step 1 to the encoder model for pre-training; Step 3: Construct a decoder based on a bidirectional screening attention network, and connect the encoder and decoder pre-trained in steps 1 and 2 in series to form an improved Transformer network; The improved Transformer network is iteratively trained using a custom norm loss function and a blind-Chinese comparison dataset until the user-preset accuracy is achieved.

2. The improved Transformer method for converting blind Chinese to Chinese based on knowledge distillation and continuous writing pre-training as claimed in claim 1, characterized in that: Step 1 is implemented as follows: Step 1.1: The Chinese word segmentation of the original Braille-Chinese comparison dataset is converted into the Braille unit ASCII code with the label Y j =(y1,y2,...,y n ), the processed Braille-Chinese comparison data set is used to train the Braille reconstruction byte pair segmentation model, and the Braille sentence X j =(x1,x2,...,x n ) is input into the network, and flag is used as an identifier to distinguish the parts of the network participating in the training, thereby obtaining a Braille reconstruction byte pair segmentation model; Step 1.2: Use padding placeholders to align the data of the maximum length n of the Braille sentences, and use one-hot encoding on the original Braille sentences to form a Braille sentence matrix; Step 1.3: Use the encoder-transcoder structure to reconstruct the input χ into the output Y; define the encoder as a BERT consisting of a 6-layer Transformer-Encoder as a TE network. When the flag is 0, input the Braille sentence into the TE network, and use the placeholder index loss function shown in formula (1) to iteratively train the TE network until the user-preset accuracy is reached and the flag is set to 1; Among them, χ j =TE(X j ) is the output after the input χ passes through the encoder, and λ is the balance parameter; represents the weighted sum of the elements in the first column of the calculation matrix, φ(X j ) represents the search x j Matrix, when the first column element value of a row is the largest, insert the row into X j The corresponding positions of are used to ensure that the dimensions of the two matrices are consistent; Step 1.4: Define the transcoder network as a TD network using a 6-layer Transformer-Decoder, and define the network consisting of the TE network and the TD network in series as a TE-TD network; when the flag is 1, input the Braille sentence into the TE-TD network, and use the Hamming loss function based on hard value soft output shown in formula (2) to iteratively train the TE-TD network until the user-preset accuracy is reached and the training is terminated; use the TE network as the final byte pair segmentation model; Among them, ||·|| h represents the Hamming distance that enhances the decision power of the network, μ and v represent the regularization parameters that balance the learning weights of the network, and the Υ(·) function is defined as shown in formula (3) and is used to harden the soft labels; Y(p)=[ρ1,ρ2,…,ρ h ] T ,p=[p1,p2,…,p h ] T Step 1.5: Extract Braille sentences from the training set to form X as shown in (4) i , the label obtained by the byte pair segmentation model will be X i Each word in is segmented into byte pairs to construct a pre-training set as shown in formula (5); Among them, m is the maximum length of the sentence, and the insufficient part is padded with padding.

3. The improved Transformer method for converting blind Chinese to Chinese based on knowledge distillation and continuous writing pre-training as claimed in claim 1, characterized in that: Step 2 is implemented as follows: Step 2.1: Use the relative distance mask attention module shown in formula (6) to obtain the intermediate vector shown in formula (7); Where X = {x1, x2, ..., x n }, is the Hadamard product; Attention (Q, K, V) encodes the relationship weights between the row vectors of the matrix, giving the network the ability to pay attention to Braille information in both directions, ψ(X T X)=[a ij ], Ψ(·) represents the relative distance mapping function It is represented as a punctuation information mask matrix, which is defined as shown in formula (8). Punlist is a punctuation table. The purpose of masking X here is to shield the influence of padding and punctuation on the attention score. M k =(a ij ),i,j∈Z[0,5], Step 2.2: Connect the intermediate vector shown in formula (7) to the grouped feedforward connection network shown in formula (9) to encode and integrate the attention scores; Among them, g is the number of custom hyperparameter groups, m is the number of batches; the split function is used to Split into g vectors, and the concat function is used to merge g vectors into one vector; Step 2.3: Use the residual method shown in formula (10) to construct a learnable parameter residual connection module; adjust the upper and lower bounds of the residual through hyperparameters, and dynamically balance the connection weights through iteration to protect the Braille encoding information; Among them, χ k and are the vectors of steps 2.2 and 2.1 respectively, Φ(input) represents the generalized function represented by the attention layer and the grouped feedforward layer, γ1 and γ2 are learnable parameters, u and s are user-defined hyperparameters, u is the upper bound, and s is the lower bound; Step 2.4: The Braille reconstruction byte pair segmentation model described in step 1 is defined as the teacher model as TN(·); the encoder described in step 2 is defined as the student model as SN(·), and the knowledge distillation method shown in formula (11) is used to perform knowledge distillation pre-training on the student model using the non-uniform cross distillation loss function shown in formula (12); Among them, T is the user-preset distillation temperature; the dimension of W is v×Vcb_size, Vcb_size is the size of the dictionary created according to the training set, and W is used to convert the student model output into a probability distribution with a dimension of Vcb_size; α1 and α2 are both balancing factors, α1 represents the hard label loss, and α2 is used to prevent the output probability density from tending to a uniform distribution.

4. The improved Transformer method for converting blind Chinese to Chinese based on knowledge distillation and continuous writing pre-training as claimed in claim 1, characterized in that: Step 3 is implemented as follows: Step 3.1: Save the encoder student model parameters after the knowledge distillation described in step 2; and define the input Braille sentence as X as shown in formula (13) i , the encoder output is defined as shown in equation (14) Where i represents the i-th sentence in the blind training set; Step 3.2: Build a decoder to convert the braille sentence X i and encoder output The corresponding correct Chinese translation is as shown in formula (15): Will As the decoder input to the relative distance mask attention module shown in formula (16), the attention score matrix shown in formula (17) is used to capture X i The attention vector of the dependency weight relationship between word vectors; in, is the Hadamard product, Mask is the lower triangular masking matrix used to mask future information during training. Mask is defined as shown in formula (18), Ψ is the relative distance mapping function, Step 3.3: Use the grouped linear transformation residual connection module shown in formula (19) Perform integrated coding to form L as shown in formula (20) i ; Among them, BN γ,β It is the batch normalization operation; Step 3.4: Output the encoder With L i Through the two-way screening of the attention concentration network for fusion, the attention is further distributed to the sparse matrix using equations (21) and (22) to obtain the maximum weight max(γ1,γ2), thereby enhancing the network's decision-making power, and the tightness value is obtained using equation (23); Among them, γ1,γ2 are learnable parameters, The screening function is defined as shown in formula (22); The γ function is defined as shown in (23); Among them, θ(M) is the tightness function of the matrix element. The larger the calculated tightness value is, the sparser the attention weight distribution of the matrix element will be after the Softmax function is applied. Step 3.5: Use the tightness value as input and obtain the prediction vector shown in formula (25) through formula (24) Where k represents the probability that the kth word in the i-th sentence is predicted to be the word with index v in the vocabulary, and the predicted word with the maximum probability is selected as the kth word of the Chinese sentence for network translation, thereby achieving translation; Step 3.6: Use the custom norm loss function shown in formula (26) to predict the vector and Learning after normalization; Among them, ||·|| F is the Frobenius norm, ||·|| P for P norm, and κ is a regularization hyperparameter used to enhance the network's decision-making power and prevent the output from tending to a uniform distribution.

Images