A blind Chinese-English translation method, system and device fusing multiple features

Abstract

This application relates to the field of Braille translation, specifically to a method, system, and device for Braille-to-Chinese translation that integrates multiple features. It includes acquiring Braille data to be translated; inputting the Braille data to be translated into a language model for translation to obtain the Chinese translation result; the language model includes a dot matrix encoding layer, a feature extraction layer, a word embedding layer, a feature fusion layer, an encoding layer, and a decoding layer; the Braille data to be translated is input to the word embedding layer to obtain word vectors, and then to the dot matrix encoding layer to obtain Braille dot matrix encoding vectors. The Braille dot matrix encoding vectors are then processed by the feature extraction layer to extract higher-order dot matrix structure features to obtain higher-order dot matrix structure feature vectors. The word vectors and the higher-order dot matrix structure feature vectors are then input to the feature fusion layer for feature fusion to obtain a fused vector. The fused vector is encoded by the encoding layer and then decoded by the decoding layer to obtain the Chinese translation result. This application can perform Braille-to-Chinese translation and has significant application value.

Description

Technical Field

[0001] This application relates to the field of Braille translation, specifically to a Braille-Chinese translation method, system, and device that integrates multiple features. Background Technology

[0002] Braille, as a crucial medium for blind people to access information, has seen some progress in translation technology, particularly in Braille-to-Chinese translation, in recent years, but significant challenges remain. Current research, including using a phonetic knowledge dictionary and statistical language models to handle Braille-to-Pinyin conversion, introducing n-gram language models to directly achieve Braille-to-Chinese translation without Pinyin mediation, and combining NFA with Transformer strategies, all demonstrate the potential and application of deep learning in Braille-to-Chinese translation tasks. However, these methods have limited effectiveness in handling long sentence translation and insufficient consideration of multimodal information fusion, failing to fully leverage the multi-feature fusion advantages of deep learning models.

[0003] Of particular note is CN115238649A, which proposes a Braille-to-Chinese translation method based on CBHG-ECA. This method enhances the model's ability to capture global information through the ECA module, achieving good results in translating Braille sentences of arbitrary length into Chinese. However, this method primarily focuses on enhancing single-modal features and does not fully consider the fusion of multimodal information, especially the integrated use of visual and tactile features. This may limit the model's generalization ability and translation accuracy in practical translation. Furthermore, CN111581985B's Transformer-based Chinese-Braille translation method, while achieving direct conversion at the sentence level and reducing corpus construction costs, still has shortcomings in handling complex language structures and semantic understanding, particularly lacking in-depth exploration of multi-feature fusion. This limits its application in advanced translation tasks. Summary of the Invention

[0004] In view of the limitations of the existing technology, this application aims to develop a multi-feature blind-Chinese translation method. By effectively integrating visual, tactile, and other multimodal information through a deep learning model, it seeks to improve the accuracy and generalization ability of translation, especially when dealing with long sentences and complex semantic structures, with the goal of achieving a breakthrough in the field of blind-Chinese translation. Specifically, it includes: Obtain the Braille data to be translated; The Braille data to be translated is input into a language model for translation to obtain the Chinese character translation result. The language model includes a dot matrix encoding layer, a feature extraction layer, a word embedding layer, a feature fusion layer, an encoding layer, and a decoding layer. The Braille data to be translated is input into the word embedding layer to obtain word vectors, and into the dot matrix encoding layer to obtain Braille dot matrix encoding vectors. The Braille dot matrix encoding vectors are then processed by the feature extraction layer to extract higher-order dot matrix structure features to obtain higher-order dot matrix structure feature vectors. The word vectors and the higher-order dot matrix structure feature vectors are then input into the feature fusion layer for feature fusion to obtain a fused vector. The fused vector is then encoded by the encoding layer and decoded by the decoding layer to obtain the Chinese character translation result.

[0005] The dot matrix encoding layer encodes the dot structure of Braille using the six-dot structure of Braille to obtain a Braille dot matrix encoding vector. The feature extraction layer consists of N linear layers and activation functions alternating between them, where N is a natural number greater than 1. The layer extracts higher-order dot matrix structure features from the Braille dot matrix encoding vector to obtain a higher-order dot matrix structure feature vector.

[0006] The decoding layer of the language model includes a word segmentation layer and a translation layer. The translation layer and the word segmentation layer are parallel network layers. The encoding features of the encoding layer are input to the word segmentation layer to obtain Braille word segmentation prediction labels. The Braille word segmentation prediction labels and the encoding features are fused together and then input to the translation layer to obtain the Chinese character translation results.

[0007] The training of the language model includes: acquiring a Braille dataset and labels; converting the Braille dataset into dot matrix encoded vectors through dot matrix encoding; embedding words into the Braille data to obtain word vectors; extracting high-order dot matrix structural feature vectors from the dot matrix encoded vectors through structural feature extraction; fusing the word vectors and the high-order dot matrix structural feature vectors and inputting them into the encoding layer for encoding to obtain encoded features; inputting the encoded features into the decoding layer for word segmentation and translation to obtain word segmentation prediction labels and Chinese character translation prediction results; comparing the word segmentation prediction labels with the real word segmentation labels; comparing the Chinese character translation prediction results with the real Chinese character labels; calculating the model's loss function; iteratively training the model until the loss function remains unchanged or the preset number of iterations is reached to obtain the language model.

[0008] The labels include Braille word segmentation labels (BMES) and Chinese labels. The Braille word segmentation labels (BMES) are used to mark the start character, middle character, end character, and single character of Braille characters, while the Chinese labels are used for translation comparison. The loss function includes word segmentation loss, translation loss, and dot matrix alignment loss. The translation loss and word segmentation loss are backpropagated bidirectionally using gradients. The dot matrix alignment loss is used to calculate the loss between the high-order structural features of the dot matrix and the semantic matching of Braille.

[0009] The training of the language model also includes rule loss, encoding the Braille word segmentation rules to obtain rule encoding labels, and calculating whether the Braille word segmentation results conform to the word segmentation rules, which is used to constrain the Braille word segmentation results; the rules include interleaving rules, continuous writing rules, and split writing rules.

[0010] The language model's encoder includes S encoders, and the decoding layer includes S decoders. Each encoder or decoder sequentially includes a self-attention layer, a first-batch normalization layer, a forward propagation layer, and a second-batch normalization layer. In the first-batch normalization layer, the output of the previous layer and the initial input are fused through residual connections. In the second-batch normalization layer, the output of the previous layer and the output of the first-batch normalization layer are fused through residual connections. S is a natural number greater than 1.

[0011] The purpose of this invention is to provide a computer program product that includes a computer program or instructions, which are executed by a processor to implement the above-described method for blind-Chinese translation that integrates multiple features.

[0012] The purpose of this invention is to provide a computer device comprising a memory, a processor, and a computer program or instructions stored in the memory, wherein the computer program or instructions are executed by the processor to implement the aforementioned method for blind-Chinese translation incorporating multiple features.

[0013] The purpose of this invention is to provide a computer-readable storage medium storing a computer program or instructions thereon, which is executed by a processor to implement the above-described method for blind-Chinese translation that integrates multiple features.

[0014] Advantages of this invention: 1. This invention starts from the physical essence of Braille, transforming the 6-dot convex structure into learnable digital features. These features are then embedded and fused with the Braille text at the encoder input, providing the model with prior information about the underlying structure of Braille characters. This upgrades the model from "text character recognition" to "dot matrix structure recognition," truly understanding the compositional logic of Braille. It enhances the feature discrimination of easily confused Braille characters, locking in the unique meaning of a character through dot differences. Furthermore, by integrating structural features and textual semantic features, it enriches the encoder's feature expression dimensions, improves semantic understanding accuracy, and exhibits stronger robustness to non-standard and low-resolution Braille inputs. It reduces structural noise interference, alleviates ambiguity in context-free Braille characters, reduces single-character semantic uncertainty by relying on a fixed dot structure, and simultaneously solves the problem of easily confused similar-looking Braille characters (such as misjudging Braille symbols with similar dot positions). It distinguishes easily confused characters from the underlying structure, reducing character-level translation errors.

[0015] 2. Transforming Braille word segmentation into a B / M / E / S sequence labeling task, sharing an encoder and jointly training with the translation task, and incorporating Braille abbreviation / linking / splitting rules as constraints, represents an optimization at the semantic boundary and grammatical rule level. This approach adapts to the mandatory abbreviation, linking, and splitting rules of the "National General Braille Scheme," internalizing the rules into the model modeling process. This enables gradient mutual correction between word segmentation and translation, improving translation accuracy through improved word segmentation quality, and optimizing word segmentation results through translation feedback. Furthermore, it ensures that the semantics of long Braille sentences remain intact, guaranteeing the fluency and completeness of the translation results. It avoids rule violations such as the splitting of Braille abbreviation units, the breaking of linking units, and the sticking together of splitting units, ensuring that the translation complies with national Braille standards. It also solves the problem of semantic fragmentation in long Braille sentences by ensuring the coherence and logical integrity of translated sentences through well-organized word segmentation units. Finally, it addresses the disconnect between Braille grammatical rules and translation modeling by integrating rule constraints into the training process, rather than relying solely on post-processing verification. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the process of a blind-Chinese translation method that integrates multiple features, provided in an embodiment of the present invention. Figure 2 A schematic diagram of a blind-Chinese translation system integrating multiple features provided in an embodiment of the present invention; Figure 3 A schematic diagram of a computer device provided in an embodiment of the present invention; Figure 4 This is a correction example provided in the process of manually constructing a Chinese-Braille parallel gold standard corpus, as an embodiment of the present invention. Figure 5 This is a language model structure diagram provided for an embodiment of the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0019] In some of the processes described in the specification, claims, and above-mentioned drawings of the present invention, a plurality of operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order in which they appear herein or may be executed in parallel. The serial numbers of the operations, such as S101, S102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions such as "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent a sequence, and do not limit that "first" and "second" are of different types.

[0020] Figure 1 The schematic diagram of the blind Chinese translation method integrating multiple features provided by the embodiment of the present invention specifically includes: S1: Obtain the blind text data to be translated; In one embodiment, the blind text data to be translated is vectorized and converted into word vectors.

[0021] In a specific embodiment, the present invention aims to achieve efficient and accurate Braille-Chinese translation through the informationization processing of Braille grammar rules, the construction and verification of a high-quality Chinese-Braille parallel corpus, and the construction of a pre-trained large language model. First, the present invention completes the informationization processing of systematic Braille grammar rules based on the "National General Braille Plan" and develops a Chinese-to-Braille automatic conversion tool. Subsequently, a large-scale Chinese-Braille parallel corpus is constructed using this tool, and its quality is strictly verified. Finally, we train a pre-trained large language model based on the constructed corpus to complete the Braille-Chinese translation task.

[0022] During the construction of the Chinese-Braille corpus, all basic elements including initials, finals, tones, English letters, numbers, and punctuation marks are sorted out, and the corresponding mapping table is constructed to achieve efficient character-Braille mapping. In terms of spelling and tone rules, the specifications in the "National General Braille Plan" are strictly followed, including the rules for initials forming syllables alone, finals forming syllables alone, special spelling rules, and tone abbreviation rules, etc. For example, syllables such as "zhi", "chi", "shi", "zi", "ci", "si" etc. omit the final i and form syllables by themselves with the initials. Finals such as "ai", "en" etc. can form syllables by themselves. "他" and "它" are specially spelled as " ", " ”. In terms of tone abbreviation, for syllables with the initial consonant “f”, the high-level tone symbol is abbreviated; for syllables with initial consonants such as “p, m, t, n, h”, the rising tone symbol is abbreviated; for syllables with initial consonants such as “b, d, l, g, k, j”, the falling tone symbol is abbreviated, and the falling tone symbol is also abbreviated when the final forms a syllable by itself. In addition, there are also some specified syllables where the tone is directly abbreviated. Based on the above rules, we set corresponding restrictive conditions for the output content to ensure that the generated Braille results strictly conform to the specifications of the scheme in terms of form and content. In terms of the abbreviation rules, the present invention has formulated a set of controllable and extensible abbreviation logics for words such as “de”, “me”, “ni”, “ta”, “ta”, “ta” according to the scheme, and dynamically applies the abbreviation rules through a conditional judgment mechanism.

[0023] The present invention has trained a Bert-CRF model specifically for Chinese word segmentation. This model is based on the pre-trained bert-base-chinese and combines CRF (Conditional Random Field) to perform Chinese word segmentation tasks. BERT is a deep pre-trained language model proposed by Google, with the ability of bidirectional context modeling, capable of generating high-quality word vectors and sentence vectors. Its Chinese variant bert-base-chinese is pre-trained on large-scale Chinese corpora (such as Chinese Wikipedia, news texts), supporting Chinese character-level and word-level modeling. CRF is a probabilistic graphical model widely used in sequence labeling tasks, which can effectively model the transition relationships between labels and achieve global optimal prediction. By combining the strong semantic modeling ability of BERT with the structured prediction mechanism of CRF, the Bert-CRF model can improve the consistency of sequence labeling while maintaining semantic understanding ability. Experimental results show that this model performs excellently in Chinese word segmentation tasks, especially having obvious advantages in dealing with ambiguous segmentation and complex contexts.

[0024] The present invention has collected Chinese corpora from the Leipzig Corpus and the Modern Chinese Grammar Information Dictionary. During the preprocessing, it was found that some corpora contain special symbols (such as %, / , etc.) not standardized in the National General Braille Scheme. For these symbols that cannot be represented by Braille norms, we performed data cleaning. Subsequently, we used the Chinese-to-Braille automatic conversion tool to generate Braille texts and initially constructed the alignment data between Chinese and Braille. After further refined processing through the alignment algorithm, a high-quality Chinese-Braille parallel corpus was formed. According to the verification of the accuracy of the conversion tool in the following text, it can also confirm the reliability of this corpus from the side. Figure 4Shows a correction example in the process of manually constructing a Chinese-Braille gold standard corpus. In the Chinese-to-Braille automatic conversion tool, the Chinese text needs to be first converted into pinyin and then mapped to Braille. In this process, due to the existence of polyphonic characters, errors may occur in the pinyin conversion stage. However, the Braille mapping stage is based on a fixed correspondence between initials and Braille, so it does not produce deviations itself. Therefore, only the errors in the pinyin conversion stage need to be corrected to complete the overall correction. Take "refugee" as an example. The correct pinyin of the character "难" should be the fourth tone "nàn", but the tool wrongly converts it to the second tone "nán". Since the Braille corresponding to "nan2" is " ", and the correct "nan4" should be " ", so by correcting " " to " ", the correction can be completed.

[0025] To verify the accuracy of the conversion tool and the quality of the corpus, we aligned the Braille results generated by the tool with the gold standard corpus character by character and calculated the character-level accuracy rate:

[0026] Among them, represents the number of correctly matched characters, represents the total number of characters in the gold standard. Character-level evaluation can accurately measure the technical accuracy of the tool at the Braille coding level.

[0027] At the same time, we perform word-by-word alignment and count the word-level accuracy rate:

[0028] Among them, represents the number of correctly matched words, represents the total number of words in the gold standard. This metric is used to measure the overall accuracy of information transmission.

[0029] Furthermore, the vocabulary of the language model was expanded. The "National General Braille Scheme" defines 62 basic Braille characters as the smallest constituent units of all Braille symbols. However, some Braille characters were not included in the original vocabulary of the mBART-large-50 model (language model). To ensure that the model can accurately recognize these characters, this invention first adds all 62 Braille characters to the tokenizer using the `tokenizer.add_tokens()` method; then, it uses the `model.resize_token_embeddings()` method to adjust the embedding layer dimensions to match the updated vocabulary. This processing effectively improves the model's ability to recognize and express Braille input. In addition, this invention also found that some commonly used Chinese punctuation marks were missing from the bert-base-chinese vocabulary, such as full-width single quotation marks (''), double quotation marks (“”), dashes (——), and ellipses (…). To improve the model's ability to process actual text, we compared and supplemented the missing characters with the People's Republic of China National Standard "Usage of Punctuation Marks" and utilized this information during the training phase, thereby enhancing the model's versatility and accuracy.

[0030] In the task of translating Chinese from Braille, this invention employs the mBART-large-50 model. This model is pre-trained with denoised autoencoders on 50 languages ​​and is specifically optimized for machine translation. Its architecture includes a 12-layer encoder and a 12-layer decoder, with a feedforward layer dimension of 4096. Its "wide and shallow" structure helps model the sparse input features of dot matrix Braille. Furthermore, the denoising pre-training enhances its robustness to Braille input noise.

[0031] In one specific embodiment, this invention uses one type of training dataset (News) and three types of external validation datasets (Web, Wikipedia, and Gold) for the Chinese-Braille translation task. One million Chinese sentences were collected from the "zho_news_2007-2009_1M-sentences" file in the Leipzig Corpus, and combined with approximately 80,000 words from the *Modern Chinese Grammar Information Dictionary* provided by the Institute of Computational Linguistics, Peking University. After data cleaning and processing using an automatic Chinese-Braille translation tool, a parallel Chinese-Braille corpus (i.e., the News dataset) covering both word-level and sentence-level granularity was constructed. This dataset contains a total of 1,080,276 samples, each consisting of a Chinese character string and its corresponding Braille string.

[0032] The Web and Wikipedia datasets were derived from the "zho-simp-tw_web_2014_100K-sentences" and "zho_wikipedia_2014_100K-sentences" files in the Leipzig Corpus, respectively. Each contains 100,000 Chinese sentences and has undergone standard data cleaning procedures. Because the "zho_wikipedia_2014_100K-sentences" file contains a large number of traditional Chinese characters, we simplified the file to reduce interference with the model's recognition capabilities. Ultimately, both the Web and Wikipedia datasets contain 100,000 samples, each consisting of a Chinese character string and its corresponding Braille string.

[0033] The Gold dataset is a manually constructed gold standard corpus of Chinese-Braille. 1000 sentences were selected from the *People's Daily*, and their corresponding Braille texts were generated using an automatic Chinese-to-Braille translation tool. Three annotators then meticulously proofread each sentence to ensure accuracy. The final Gold dataset contains 1000 high-quality samples, each consisting of a Chinese character string and its corresponding Braille string.

[0034] To prevent overfitting, this invention divides the News dataset into training, validation, and test sets in an 8:1:1 ratio.

[0035] In the blind-to-Chinese translation task, the mBART-large-50 model was fine-tuned using the News dataset, and its generalization ability was evaluated on three external validation datasets: Web, Wikipedia, and Gold. The main hyperparameter settings of the model are shown in Table 1. During training, the AdamW optimizer was used for all models, with an initial learning rate of 5e-5. Validation and model checkpoint saving were performed every 5000 steps to prevent overfitting. The number of training epochs was set based on preliminary experimental results: mBART-large-50 exhibits high complexity in the blind-to-Chinese translation task, therefore, it was trained for 5 epochs to fully learn Braille features and avoid underfitting.

[0036] In terms of hardware configuration, the mBART-large-50 uses four NVIDIA A100 GPUs to accelerate training. During the model evaluation phase, BLEU (Bilingual Evaluation Understudy) is used as an automated evaluation metric to measure the degree of matching between the model's output translation and the reference translation at the n-gram level. A higher score indicates better translation quality, ranging from 0 to 100.

[0037] During model training, multiple metrics are used for periodic evaluation on the validation set, including Precision, Recall, F1-score, and Accuracy. The F1-score is the primary performance metric used to select the best model. At the end of each epoch, a complete evaluation is performed on the validation set, and the parameters of the best-performing model are saved.

[0038] Table 1

[0039] In one specific embodiment, during the data preprocessing stage of training, the present invention converts the original Braille word segmentation and annotation data into a (B, I, E, S) labeling system. Specifically, B (Begin) indicates the beginning of a word, I (Inside) indicates the middle of a word, E (End) indicates the end of a word, and S (Single) indicates that a single character constitutes an independent word.

[0040] S2: The Braille data to be translated is input into a language model for translation to obtain the Chinese character translation result; the language model includes a dot matrix encoding layer, a feature extraction layer, a word embedding layer, a feature fusion layer, an encoding layer, and a decoding layer; the Braille data to be translated is input into the word embedding layer to obtain word vectors, and into the dot matrix encoding layer to obtain Braille dot matrix encoding vectors. The Braille dot matrix encoding vectors are processed by the feature extraction layer to extract higher-order dot matrix structure features to obtain higher-order dot matrix structure feature vectors. The word vectors and the higher-order dot matrix structure feature vectors are input into the feature fusion layer for feature fusion to obtain a fusion vector. The fusion vector is encoded by the encoding layer and then decoded by the decoding layer to obtain the Chinese character translation result.

[0041] In one embodiment, the dot matrix encoding layer encodes the dot structure of Braille using the six-dot structure of Braille to obtain a Braille dot matrix encoding vector. The feature extraction layer consists of N linear layers and activation functions alternating between them, where N is a natural number greater than 1. The higher-order dot matrix structure features of the Braille dot matrix encoding vector are extracted to obtain a higher-order dot matrix structure feature vector.

[0042] In one embodiment, the decoding layer of the language model includes a word segmentation layer and a translation layer, which are parallel network layers. The encoding features of the encoding layer are input to the word segmentation layer to obtain Braille word segmentation prediction labels. The Braille word segmentation prediction labels and the encoding features are then fused together and input to the translation layer to obtain the Chinese character translation results.

[0043] In one embodiment, the training of the language model includes: acquiring a Braille dataset and labels; converting the Braille dataset into dot matrix encoded vectors through dot matrix encoding; embedding words into the Braille data to obtain word vectors; extracting high-order dot matrix structural feature vectors from the dot matrix encoded vectors through structural feature extraction; fusing the word vectors and the high-order dot matrix structural feature vectors and inputting them into the encoding layer for encoding to obtain encoded features; inputting the encoded features into the decoding layer for word segmentation and translation to obtain word segmentation prediction labels and Chinese character translation prediction results; comparing the word segmentation prediction labels with the real word segmentation labels; comparing the Chinese character translation prediction results with the real Chinese character labels; calculating the loss function of the model; iteratively training the model until the loss function remains unchanged or the preset number of iterations is reached to obtain the language model.

[0044] Optionally, the labels include Braille word segmentation labels (BMES) and Chinese labels. The Braille word segmentation labels (BMES) are used to mark the start character, middle character, end character, and single character of Braille characters, while the Chinese labels are used for translation comparison. The loss function includes word segmentation loss, translation loss, and dot matrix alignment loss. The translation loss and word segmentation loss are backpropagated bidirectionally using gradients. The dot matrix alignment loss is used to calculate the loss between the high-order structural features of the dot matrix and the semantic matching of Braille.

[0045] Biphasic gradient backpropagation is a translation constrained by word segmentation tags, which optimizes Braille word segmentation by comparing the predicted translation with the actual Chinese characters.

[0046] Optionally, the training of the language model also includes rule loss, encoding the Braille word segmentation rules to obtain rule encoding labels, and calculating whether the Braille word segmentation result conforms to the word segmentation rules, which is used to constrain the Braille word segmentation result; the rules include interleaving rules, continuous writing rules, and split writing rules.

[0047] In one embodiment, the language model's encoder includes S encoders, and the decoding layer includes S decoders. Each encoder or decoder sequentially includes a self-attention layer, a first-batch normalization layer, a forward propagation layer, and a second-batch normalization layer. In the first-batch normalization layer, the output of the previous layer is fused with the initial input through residual connections. In the second-batch normalization layer, the output of the previous layer is fused with the output of the first-batch normalization layer through residual connections. S is a natural number greater than 1.

[0048] In one specific embodiment, the architecture of the encoding and decoding layers is an mBART model. The Braille text to be translated is input into the word embedding layer of the mBART model to obtain word vectors. Simultaneously, the Braille text to be translated is input into the constructed dot matrix encoding layer and feature extraction layer to obtain dot matrix high-order structure feature vectors. The dot matrix high-order structure feature vectors and word vectors are fused and then input into the encoding and decoding layers of the mBART model. The decoding layer includes a word segmentation layer and a translation layer, which are parallel modules. The translation result can be obtained directly through the translation layer, or the word segmentation result can be obtained first using the word segmentation layer, and then the word segmentation result can be fused with the result output from the encoding layer to obtain a second fused result. The second fused result is then input into the translation layer for translation to obtain the translation result. The word segmentation layer obtains the word segmentation result through an MLP layer. The language model can also be other models used for natural language processing, including mT5 and Transformer.

[0049] In one specific embodiment, the core physical feature of Braille is its 6-dot matrix structure. Existing technologies treat Braille merely as ordinary text characters, failing to utilize this structural feature. A module consisting of dot matrix encoding and feature extraction transforms the dot matrix physical structure into digital features, which are then fused with text embedding to improve the accuracy of Braille-to-Chinese translation. Specifically: Step 1: Braille dot matrix standardization coding; Following the 6-dot standard of the National General Braille Scheme (2 rows and 3 columns, dot numbers: 1, 2, 3 on the top row; 4, 5, 6 on the bottom row), each Braille character is converted into a 6-dimensional binary vector (1 = convex dot, 0 = flat dot).

[0050] Step 2: Overall Module Architecture; Dot matrix encoding layer: Braille characters are converted into 6-dimensional binary vectors; Feature extraction layer: includes multiple linear layers and activation functions, used to extract high-order structural features of the dot matrix; (preferably, two linear layers and activation functions are alternately combined).

[0051] Step 3: Feature fusion layer; The high-order structure features of the dot matrix are concatenated / gated with the model text embedding (word vectors of the word embedding layer); (the 6-dimensional feature vector is mapped to 768 dimensions through a linear layer and then fused with the word embedding).

[0052] Step 4: Encoding and Decoding; The fused features are then fed into the encoding and decoding layers of the mBART model for encoding and decoding.

[0053] During the training process, the training data is used to construct a triplet dataset of Braille characters, dot matrix vectors, and Chinese labels to train the mBART model (language model). During the application, after inputting Braille characters, dot matrix vectors are automatically generated. The features of word vectors and dot matrix vectors are fused together, and Chinese text is generated using the fused features. The optimizer is AdamW, with a learning rate of 2e-5 and a weight decay of 1e-8.

[0054] In another embodiment, the language model also includes word segmentation and rule constraints. In step 4, a shared encoding layer is used to obtain encoded features for a unified context. The decoding layer is a dual-task parallel process, splitting the encoded features of the encoding layer into two branches: branch 1 is the word segmentation annotation head (MLP layer), outputting the B / M / E / S classification probabilities for each Braille character; branch 2 is the decoder of the mBART model, outputting the Chinese character sequence, such as... Figure 5 As shown. During training, the losses are calculated, including translation loss and word segmentation loss. These losses are then bidirectionally backpropagated using gradients. If the translation is inaccurate, word segmentation is optimized by correcting the encoder and model parameters; if word segmentation is illegal, translation is optimized by correcting the encoder and model parameters. In application, after encoding, the coding layer first outputs compliant Braille word segments (B / M / E / S). The encoded features, combined with the compliant Braille word segments, are then input into the decoder to obtain the Chinese character translation. This blocks word segmentation errors at the source, thereby reducing translation errors.

[0055] In addition, the word segmentation process in the language model also includes rule constraints, focusing on the core rules affecting word segmentation in the "National General Braille Scheme", which are classified and sorted into modelable constraints.

[0056] 1. Corpus annotation expansion; Based on the existing parallel corpus of blind and Chinese speakers, two layers of annotation were added to support joint training: Word segmentation boundary labeling: Label each Braille character with a B / M / E / S tag; Rule type annotation: Annotate the constraint type (abbreviation / continuous / split / normal) corresponding to each word segmentation unit, which is used for loss function weighting.

[0057] 2. Rule-based encoding mapping; The four types of Braille constraints are converted into digital codes that the model can recognize, which are then used for constraint verification and loss calculation. 0 = Normal unit, 1 = Abbreviated unit, 2 = Forced continuous unit, 3 = Forced separate unit.

[0058] 3. Rule validator construction (no code, rule matching tool); Building a lightweight rule matching library, which takes word segmentation units and rule tags as input and outputs violation judgment results, is the core basis for calculating the loss function.

[0059] During model training, the model predicts the B / M / E / S of each Braille character using the word segmentation header (MLP) and calculates the segmentation loss. The translation decoder predicts translated Chinese characters and calculates the translation generation loss. During word segmentation prediction, rule-based validation checks each segmentation unit to determine if there are any errors, calculating the rule penalty loss. The total loss function is:

[0060] Among them, L 翻译 For Chinese generation loss, L 分词 For B / M / E / S classification loss, L 规则惩罚 For violations of abbreviation / contiguous / split writing rules, L 点阵对齐 The loss is calculated for matching dot matrix features with Braille semantics, where α, β, γ, and δ are manually set weight hyperparameters. Furthermore, the translation loss and word segmentation loss are cross-entropy losses, including:

[0061] N represents the length of the target Chinese character sequence, and X represents the input Braille sequence. Let represent the i-th real Chinese character tag, and p() represent the decoder's prediction probability for the i-th token.

[0062]

[0063] Where T represents the number of input Braille characters, This indicates the one-hot encoding of the true label. This indicates that the model predicts the t-th character as the label of the k-th class.

[0064]

[0065] Where U represents the number of Braille units obtained after word segmentation. () represents a knowledge function, with 1 for violations and 0 for compliance; This indicates the penalty weight for different types of violations.

[0066]

[0067] Where D represents the dimension of the hidden layer of the model (e.g., 1024), F dot F represents the structural features of the output of the lattice feature enhancement. text This represents the word embedding features of Braille text.

[0068] During application, there are no Braille word segmentation labels, and B / M / E / S are not output externally. After the word segmentation header generates word segmentation labels, they are fused with the encoded features to constrain the translation results of the decoder. For example, the feature representation of BME is a single word and cannot be translated separately. Word segmentation is performed on each character for classification and prediction. The rules constrain the word segmentation units. If the word segmentation unit is detected as a contiguous unit (rule 2), the entire unit is forced to be translated. If the word segmentation unit is detected as an abbreviation unit (rule 1), it is forced to be translated according to the abbreviation meaning.

[0069] Example 1; Input: Braille character sequence: ; 1. Dot matrix feature processing; →[1,1,1,0,0,0]、 →[1,0,0,0,0,0]、 →[1,1,0,1,0,0]… → Convert to a dot matrix vector embedding; 2. Shared encoder encoding; 3. Dual-task output; Word segmentation task output (tags: B / M / E / S): BMESB; Translation task output (tag: Chinese): Bank interest rates; Rule constraints: "bank" is a contiguous unit (rule tag 2) → word segmentation must be BME, otherwise a penalty will be imposed.

[0070] In one specific embodiment, to select the most suitable model for fine-tuning to obtain the best translation results, this invention trained and evaluated multiple pre-trained large language models using the News dataset under the same experimental settings. Preliminary experiments show that the mBART-B2C model (blind Chinese translation) also performs stably. It achieved a BLEU score of 92.2430 on the News test set, indicating good adaptability to the blind Chinese translation task. On the Web and Wikipedia test sets, the scores were 81.5924 and 77.4021 respectively; although the scores decreased slightly, the translation quality remained within an acceptable range. On the Gold test set, the score was 85.2554, demonstrating generalization ability in data scenarios with more complex linguistic styles. Comparison results with other models under the same experimental conditions and datasets are shown in Table 2.

[0071] Table 2

[0072] In a specific embodiment, to ensure the quality of the Chinese-Braille parallel gold standard corpus, the present invention has developed a strict manual correction process. Three annotators checked each Braille result generated by the automatic conversion tool based on the People's Daily corpus one by one, and mainly corrected the misjudgment of the polyphonic character "难" in "难民" (should be "nàn" instead of "nán") and other pinyin conversion errors. Table 3 shows the correction results after the comprehensive evaluation by the three annotators. Statistics show that the overall error rate of the automatic conversion tool is only 0.5%, indicating its high stability and reliability.

[0073] Table 3 Manual Correction Results of Chinese-Braille Parallel Gold Standard Corpus

[0074] In addition, the present invention also conducts a comparative analysis of the automatic conversion results and the manual correction results at the character level and the word level. The character-level matching rate reaches 99.66%, and the word-level matching rate is 99.11%. Both indicators show that the data quality of the Chinese-Braille parallel corpus and the Chinese-Braille parallel gold standard corpus is high, providing a solid data foundation for subsequent model training and evaluation.

[0075] [[IDID=11]]In addition, the present invention expands the vocabulary before model training to improve the recognition ability of Braille characters. After expansion, the model can cover all 62 basic Braille characters. Taking mT5-large as an example, Table 4 shows that its BLEU score has increased by 0.5532 after the vocabulary expansion, verifying the positive effect of vocabulary expansion on improving model performance.

[0076] Table 4 BLEU Scores Before and After Expanding the Vocabulary

[0077] (mT5-large-before is the model before expanding the vocabulary, mT5-large-after is the model after expanding the vocabulary).

[0078] The disclosed embodiment of the present invention also provides a computer program product or system, including a computer program, which implements the above method steps when executed by a processor.

[0079] Figure 2 The schematic diagram of the Braille-Chinese translation system integrating multiple features provided by the embodiment of the present invention specifically includes: Acquisition unit: acquire the Braille data to be translated; Translation Unit: The Braille data to be translated is input into a language model for translation to obtain the Chinese character translation result. The language model includes a dot matrix encoding layer, a feature extraction layer, a word embedding layer, a feature fusion layer, an encoding layer, and a decoding layer. The Braille data to be translated is input into the word embedding layer to obtain word vectors, and into the dot matrix encoding layer to obtain Braille dot matrix encoding vectors. The Braille dot matrix encoding vectors are then processed by the feature extraction layer to extract higher-order dot matrix structure features to obtain higher-order dot matrix structure feature vectors. The word vectors and the higher-order dot matrix structure feature vectors are input into the feature fusion layer for feature fusion to obtain a fused vector. The fused vector is then encoded by the encoding layer and decoded by the decoding layer to obtain the Chinese character translation result.

[0080] Figure 3 An embodiment of the present invention provides a schematic diagram of a computer device, specifically including: A memory and a processor; the memory is used to store program instructions; the processor is used to invoke the program instructions, when any of the above-mentioned multi-feature blind-Chinese translation methods are executed.

[0081] The present invention also discloses a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, is any of the above-described methods for blind-Chinese translation that integrates multiple features.

[0082] The verification results of this verification embodiment show that assigning inherent weights to indications can improve the performance of this method compared to the default settings. Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separated; the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of this embodiment. Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0083] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0084] The computer device provided by the present invention has been described in detail above. For those skilled in the art, there will be changes in the specific implementation and application scope based on the ideas of the embodiments of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A blind-Chinese translation method integrating multiple features, characterized in that, include: Obtain the Braille data to be translated; The Braille data to be translated is input into a language model for translation to obtain the Chinese character translation result; The language model includes a dot matrix encoding layer, a feature extraction layer, a word embedding layer, a feature fusion layer, an encoding layer, and a decoding layer; The Braille data to be translated is fed into a word embedding layer to obtain word vectors, and into a dot matrix encoding layer to obtain Braille dot matrix encoding vectors. The Braille dot matrix encoding vectors are then processed by a feature extraction layer to extract higher-order dot matrix structure features, resulting in higher-order dot matrix structure feature vectors. The word vectors and the higher-order dot matrix structure feature vectors are fed into a feature fusion layer to obtain a fusion vector. The fusion vector is then encoded by an encoding layer and decoded by a decoding layer to obtain the Chinese character translation result.

2. The blind-Chinese translation method integrating multiple features according to claim 1, characterized in that, The dot matrix encoding layer encodes the dot structure of Braille using the six-dot structure of Braille to obtain a Braille dot matrix encoding vector. The feature extraction layer consists of N linear layers and activation functions alternating between them, where N is a natural number greater than 1. The layer extracts higher-order dot matrix structure features from the Braille dot matrix encoding vector to obtain a higher-order dot matrix structure feature vector.

3. The blind-Chinese translation method integrating multiple features according to claim 1, characterized in that, The decoding layer of the language model includes a word segmentation layer and a translation layer. The translation layer and the word segmentation layer are parallel network layers. The encoding features of the encoding layer are input to the word segmentation layer to obtain Braille word segmentation prediction labels. The Braille word segmentation prediction labels and the encoding features are fused together and then input to the translation layer to obtain the Chinese character translation results.

4. The blind-Chinese translation method integrating multiple features according to claim 1, characterized in that, The training of the language model includes: acquiring a Braille dataset and labels; converting the Braille dataset into dot matrix encoded vectors through dot matrix encoding; embedding words into the Braille data to obtain word vectors; extracting high-order dot matrix structural feature vectors from the dot matrix encoded vectors through structural feature extraction; fusing the word vectors and the high-order dot matrix structural feature vectors and inputting them into the encoding layer for encoding to obtain encoded features; inputting the encoded features into the decoding layer for word segmentation and translation to obtain word segmentation prediction labels and Chinese character translation prediction results; comparing the word segmentation prediction labels with the real word segmentation labels; comparing the Chinese character translation prediction results with the real Chinese character labels; calculating the model's loss function; iteratively training the model until the loss function remains unchanged or the preset number of iterations is reached to obtain the language model.

5. The blind-Chinese translation method integrating multiple features according to claim 4, characterized in that, The labels include Braille word segmentation labels (BMES) and Chinese labels. The Braille word segmentation labels (BMES) are used to mark the start character, middle character, end character, and single character of Braille characters, while the Chinese labels are used for translation comparison. The loss function includes word segmentation loss, translation loss, and dot matrix alignment loss. The translation loss and word segmentation loss are backpropagated bidirectionally using gradients. The dot matrix alignment loss is used to calculate the loss between the high-order structural features of the dot matrix and the semantic matching of Braille.

6. The blind-Chinese translation method integrating multiple features according to claim 5, characterized in that, The training of the language model also includes rule loss, encoding the Braille word segmentation rules to obtain rule encoding labels, and calculating whether the Braille word segmentation results conform to the word segmentation rules, which is used to constrain the Braille word segmentation results; the rules include interleaving rules, continuous writing rules, and split writing rules.

7. The blind-Chinese translation method integrating multiple features according to claim 1, characterized in that, The language model's encoder includes S encoders, and the decoding layer includes S decoders. Each encoder or decoder sequentially includes a self-attention layer, a first-batch normalization layer, a forward propagation layer, and a second-batch normalization layer. In the first-batch normalization layer, the output of the previous layer and the initial input are fused through residual connections. In the second-batch normalization layer, the output of the previous layer and the output of the first-batch normalization layer are fused through residual connections. S is a natural number greater than 1.

8. A computer program product comprising a computer program or instructions, characterized in that, The computer program or instructions are executed by a processor to implement the multi-feature blind-Chinese translation method according to any one of claims 1-7.

9. A computer device comprising a memory, a processor, and a computer program or instructions stored in the memory, characterized in that, The computer program or instructions are executed by a processor to implement the multi-feature blind-Chinese translation method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, The computer program or instructions are executed by a processor to implement the multi-feature blind-Chinese translation method according to any one of claims 1-7.

Images