A method and system for converting universal Braille to Chinese characters

Abstract

The present invention proposes a method and system for converting universal Braille to Chinese characters, which is characterized by comprising: constructing a deep learning model, generating a pinyin syllable sequence and a word segmentation mark sequence from a universal Braille text, vectorizing the embedding layer of the deep learning model and then splicing the pinyin embedding vector that integrates the Braille word segmentation information, and inputting the local semantic feature extraction layer and the global semantic feature extraction layer of the deep learning model respectively; obtaining local semantic information and contextual semantic information, splicing the two and inputting them into a fully connected layer to obtain the Chinese text corresponding to the universal Braille text. The present invention realizes universal Braille-Chinese character conversion with high accuracy by splicing the pinyin syllable embedding and the word segmentation information embedding into an overall embedding vector, and the local semantic feature extraction layer and the global semantic feature extraction layer share a feature extraction module combined with the embedding layer.

Description

Technical Field

[0001] The present invention relates to the technical field of Braille processing, especially the field of automatic conversion from general Braille to Chinese characters. Background Art

[0002] Braille is an important way for blind people to read and obtain information. It is a tactile symbol system, printed on paper or displayed on a dot display, and read by touch. The basic unit of Braille is called a "cell", and one cell contains 6 dots. By setting whether each dot has a dot or not, a total of 64 combinations can be formed (including the empty cell where none of the 6 dots have a dot), and these combinations constitute the most basic Braille symbols.

[0003] In order to promote the communication between blind people and ordinary people, it is necessary to convert the Braille used by blind people into Chinese characters. For alphabetic languages, there is a direct mapping between letters and Braille symbols, and the conversion is relatively simple. However, due to the characteristics of Chinese Braille itself, the automatic conversion from Braille to Chinese characters is more difficult.

[0004] Since there is no one-to-one correspondence between Braille symbols and Chinese characters, Chinese Braille is defined as a phonetic script. In Chinese Braille, in most cases, 2 - 3 cells are used to represent one Chinese character, where one cell represents the initial consonant, one cell represents the final vowel, and in some cases, one more cell is needed to represent the tone. The biggest difference between Chinese Braille and Chinese character text lies in its "word segmentation and continuous writing" rule, that is, it is required to separate words with empty cells. Braille word segmentation is different from Chinese word segmentation, and many phrases in Chinese need to be written continuously in Braille. Regarding word segmentation and continuous writing, more than 100 rules based on morphology, grammar, and semantics are given in the Chinese Braille Standard, such as "‘不’ should be written continuously with verbs, modal verbs, adjectives, prepositions, and monosyllabic degree adverbs".

[0005] In order to further reduce ambiguity, Braille has also formulated a tone marking rule, that is, it stipulates which syllables need to explicitly mark the tone. Currently, there are mainly two Braille schemes in use, namely "Current Braille" and "General Braille". The word segmentation rules of the two Braille schemes are the same, and the main difference lies in the tone marking rule. In Current Braille, to save reading time and printing costs, it is stipulated that only easily confused words, unfamiliar words, ancient Chinese real words, very rarely used monosyllabic words, etc. are marked with tones. Generally, it is considered that the tone marking rate of Current Braille is about 5%. However, a large number of unmarked syllables are likely to cause confusion and affect reading and understanding. Therefore, the National Language Commission issued the "General Braille" scheme in 2018. Although a certain tone default setting is adopted in General Braille, through the conversion of default rules, the tones of the vast majority of syllables can be determined. Since the release date in 2018, the country has vigorously promoted General Braille. However, due to the lack of tools and systems for automatic conversion from General Braille to Chinese characters currently, and the low efficiency of manual conversion, there are obstacles in the current text communication between blind people and ordinary people.

[0006] Although there are some automatic conversion systems from Chinese characters to Braille, and some have been put into practical use, most of these systems focus on converting Chinese characters into current Braille, and there are few systems that can achieve conversion from universal Braille to Chinese characters. Summary of the invention

[0007] In view of the problems that there are few studies on the conversion from universal Braille to Chinese characters and the Braille word segmentation information is not fully utilized, the present invention adopts universal Braille-Chinese character comparison corpus and pinyin-Chinese character comparison corpus marked with Braille word segmentation information to train a deep learning model to achieve efficient and accurate universal Braille to Chinese character conversion.

[0008] Specifically, the present invention proposes a method for converting universal Braille to Chinese characters, which includes:

[0009] Step 1: construct a deep learning model including a local semantic feature extraction layer and a global semantic feature extraction layer to obtain a universal Braille text to be converted, and the local semantic feature extraction layer and the global semantic feature extraction layer share an embedding layer and a fully connected layer;

[0010] Step 2: Generate a pinyin syllable sequence and a word segmentation mark sequence from the universal Braille text, vectorize them through the embedding layer, and then concatenate them to obtain a pinyin embedding vector that integrates the Braille word segmentation information, and input them into the local semantic feature extraction layer and the global semantic feature extraction layer respectively;

[0011] Step 3: The local semantic feature extraction layer extracts the local semantic information of the pinyin embedding vector, and the global semantic feature extraction layer extracts the contextual semantic information with a temporal relationship in the pinyin embedding vector. The local semantic information and the contextual semantic information are concatenated and input into the fully connected layer to obtain the Chinese character text corresponding to the universal Braille text.

[0012] The method for converting universal Braille to Chinese characters, wherein the training process of the deep learning model is to divide the universal Braille-Chinese character comparison corpus into multiple batches of data, each batch of data includes multiple data, each data includes: a pinyin syllable sequence, a word segmentation mark sequence, and a Chinese character text corresponding to the pinyin; the pinyin syllable sequence and the word segmentation mark sequence of the batch of data are used as input to obtain the predicted Chinese character text of the deep learning model, a loss function is constructed according to the predicted Chinese character text and the Chinese character text corresponding to the pinyin, and the deep learning model is updated by back propagation;

[0013] A deep learning model is fine-tuned and updated using a pinyin-Chinese character comparison corpus, in which each pinyin text has its corresponding Chinese character text; pinyin is converted into pinyin with abbreviated tones according to the universal Braille tone marking standard, including syllables and tones, and the tone markings are: 1 for yinping, 2 for yangping, 3 for shangsheng, 4 for qusheng, and no tone indicates a light tone or the tone is abbreviated; a Braille word segmentation system is used to automatically obtain Braille word segmentation information based on the Chinese character and pinyin information, thereby obtaining the pinyin-Chinese character comparison corpus with marked word segmentation information.

[0014] The method for converting universal Braille to Chinese characters, wherein the word segmentation mark sequence is based on the BMES four-bit sequence annotation to represent the word segmentation information of each pinyin syllable in the pinyin syllable sequence, wherein B represents the first position of the pinyin syllable corresponding to the Braille word; M represents the middle position of the pinyin syllable corresponding to the Braille word; E represents the end position of the pinyin syllable corresponding to the Braille word; and S represents the pinyin syllable corresponding to a separate character.

[0015] The method for converting universal Braille to Chinese characters, wherein the local semantic feature extraction layer includes three convolution kernels of different heights, the heights are 1, 3 and 5 respectively, each height has two convolution kernels respectively, each convolution kernel obtains a feature vector by performing a convolution operation on the pinyin embedding vector, and finally obtains a feature vector that fuses 6 feature spaces to obtain local semantic information; the global semantic feature extraction layer adopts a bidirectional gated recurrent unit network to realize the capture of the contextual semantic information.

[0016] The present invention also proposes a universal Braille to Chinese character conversion system, which includes:

[0017] A model building module, used to build a deep learning model including a local semantic feature extraction layer and a global semantic feature extraction layer, to obtain a universal Braille text to be converted, wherein the local semantic feature extraction layer and the global semantic feature extraction layer share an embedding layer and a fully connected layer;

[0018] A preprocessing module is used to generate a pinyin syllable sequence and a word segmentation mark sequence from the universal Braille text, and concatenate them after vectorization by the embedding layer to obtain a pinyin embedding vector integrating the Braille word segmentation information, and input them into the local semantic feature extraction layer and the global semantic feature extraction layer respectively;

[0019] The feature splicing module is used for the local semantic feature extraction layer to extract the local semantic information of the pinyin embedding vector, and the global semantic feature extraction layer to extract the contextual semantic information with a temporal relationship in the pinyin embedding vector, and the local semantic information and the contextual semantic information are spliced ​​and input into the fully connected layer to obtain the Chinese character text corresponding to the universal Braille text.

[0020] The universal Braille to Chinese character conversion system, wherein the training process of the deep learning model is to divide the universal Braille-Chinese character comparison corpus into multiple batches of data, each batch of data includes multiple data, each data includes: a pinyin syllable sequence, a word segmentation mark sequence, and a Chinese character text corresponding to the pinyin; the pinyin syllable sequence and the word segmentation mark sequence of the batch of data are used as input to obtain the predicted Chinese character text of the deep learning model, a loss function is constructed according to the predicted Chinese character text and the Chinese character text corresponding to the pinyin, and the deep learning model is updated by back propagation;

[0021] A deep learning model is fine-tuned and updated using a pinyin-Chinese character comparison corpus, in which each pinyin text has its corresponding Chinese character text; pinyin is converted into pinyin with abbreviated tones according to the universal Braille tone marking standard, including syllables and tones, and the tone markings are: 1 for yinping, 2 for yangping, 3 for shangsheng, 4 for qusheng, and no tone indicates a light tone or the tone is abbreviated; a Braille word segmentation system is used to automatically obtain Braille word segmentation information based on the Chinese character and pinyin information, thereby obtaining the pinyin-Chinese character comparison corpus with marked word segmentation information.

[0022] The universal Braille to Chinese character conversion system, wherein the word segmentation mark sequence is based on the BMES four-bit sequence annotation to represent the word segmentation information of each pinyin syllable in the pinyin syllable sequence, wherein B represents the first position of the pinyin syllable corresponding to the Braille word; M represents the middle position of the pinyin syllable corresponding to the Braille word; E represents the end position of the pinyin syllable corresponding to the Braille word; and S represents the pinyin syllable corresponding to a separate character.

[0023] The universal Braille to Chinese character conversion system, wherein the local semantic feature extraction layer includes three convolution kernels of different heights, the heights are 1, 3 and 5 respectively, each height has two convolution kernels respectively, each convolution kernel obtains a feature vector by performing a convolution operation on the pinyin embedding vector, and finally obtains a feature vector that fuses 6 feature spaces to obtain local semantic information; the global semantic feature extraction layer adopts a bidirectional gated recurrent unit network to realize the capture of the contextual semantic information.

[0024] The present invention also proposes a storage medium for storing a program for executing any one of the methods for converting universal Braille to Chinese characters.

[0025] The present invention also provides a client, which is used for any of the above-mentioned universal Braille to Chinese character conversion systems.

[0026] It can be seen from the above scheme that the advantages of the present invention are:

[0027] By adopting the universal Braille to Chinese character conversion method and system proposed in the present invention, the word segmentation information of Braille can be effectively utilized, and the universal Braille-Chinese character comparison corpus and the pinyin-Chinese character comparison corpus with marked Braille word segmentation information can be used to train the universal Braille-Chinese character conversion model based on the deep neural network, so as to achieve high-accuracy universal Braille-Chinese character conversion. This technology is conducive to eliminating the barriers for ordinary people to engage in blind education, promoting the development of blind education, facilitating communication between the blind and ordinary people, and thus enabling the blind to better integrate into the information society. At the same time, it has important applications in the fields of Braille publishing, blind terminals, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The following is the overall flow chart;

[0029] Figure 2 It is the model structure diagram;

[0030] Figure 3 This is the structure diagram of TextCNN;

[0031] Figure 4 It is the BiGRU structure diagram;

[0032] Figure 5 Output concatenation of TextCNN and BiGRU;

[0033] Figure 6 This is an example of a Pinyin-Chinese character corpus with Braille word segmentation information marked. DETAILED DESCRIPTION

[0034] The present invention proposes a universal Braille to Chinese character conversion model, the model structure includes a feature extraction module combining pinyin syllable embedding and word segmentation information embedding into an overall embedding vector, and a local semantic feature extraction layer (CtextCNN) and a global semantic feature extraction layer (BiGRU) shared embedding layer. The universal Braille to Chinese character conversion method mainly includes the following steps:

[0035] 1. Build a deep learning model that can effectively utilize the word segmentation information of universal Braille. Preferably, a TextCNN+BiGRU network structure can be used, where TextCNN is responsible for extracting local semantic features, and BiGRU is responsible for capturing global semantic information, and then splicing the corresponding outputs into a feedforward neural network to predict Chinese text.

[0036] 2. The deep learning model constructed in step 1 is trained using the universal Braille-Chinese character comparison corpus.

[0037] When training the model, the dataset compiled from the universal Braille-Chinese character comparison corpus is divided into several batches, each of which contains several pieces of data, and each piece of data also contains: a pinyin syllable sequence, a word segmentation mark sequence, and a Chinese text corresponding to the pinyin. Each batch of data is used for training in turn. During training, the pinyin and word segmentation marks of the batch of data are used as input, and are concatenated after passing through the embedding layer to obtain a pinyin embedding sequence containing word segmentation information. The predicted Chinese characters and Chinese text generated after the model training are sent to calculate the loss function through the cross entropy, and then the parameters of the entire model are updated through the back propagation algorithm.

[0038] Preferably, the model trained in step 2 is fine-tuned using the pinyin-Chinese character comparison corpus. When training the model, a braille segmentation system is used to automatically obtain braille segmentation information based on Chinese characters and pinyin information, and the pinyin-Chinese character comparison corpus marked with braille segmentation information is divided into several batches, each batch containing several data, each data simultaneously containing: pinyin, braille segmentation mark corresponding to the pinyin, and Chinese character text corresponding to the pinyin. Each batch of data is used for training in turn, and the training method is the same as step 2.

[0039] The above-mentioned Pinyin-Chinese character comparison corpus comes from the non-Braille field. For example, the use of various existing Chinese character-Pinyin corpora can make up for the shortcomings of the Braille corpus. Since the original Pinyin-Chinese character comparison corpus comes from the non-Braille field, it does not have Braille word segmentation, and a Braille word segmentation system is required to automatically obtain Braille word segmentation information based on Chinese characters and Pinyin information. Although the automatically obtained Braille word segmentation information has a high accuracy rate, it may still contain a small amount of errors and is different from the completely correct Braille-Chinese character comparison corpus and the corpus containing Pinyin and Braille word segmentation information obtained from the corpus. Therefore, the Pinyin-Chinese character comparison corpus can only be used for fine-tuning and cannot be used as (or converted into) Braille-Chinese character comparison corpus to participate in the model training in the previous step.

[0040] The trained model automatically converts universal Braille to Chinese characters.

[0041] After the model is trained, an input universal Braille ASCII code is converted into Chinese pinyin and word segmentation tags, which are then input into the model to obtain the corresponding prediction results.

[0042] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0043] In order to make the purpose, technical solution and advantages of the present invention clearer, the following is a further detailed description of a method and system for converting general Braille to Chinese characters of the present invention in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0044] The present invention proposes a method and system for converting universal Braille to Chinese characters based on deep learning. The input universal Braille text is converted into a phonetic syllable sequence and a word segmentation mark sequence by using a deep learning method, and semantic feature information is extracted through a deep learning model, and then the corresponding Chinese characters are obtained. The process is as follows: Figure 1 As shown:

[0045] The specific steps include:

[0046] 1. Build a deep learning model that can effectively utilize Braille word segmentation information.

[0047] In one embodiment, a pinyin syllable sequence and a word segmentation mark sequence are generated from a universal Braille text, and then the pinyin embedding vectors that integrate the Braille word segmentation information are obtained after vectorization through an embedding layer, and are input into TextCNN and BiGRU respectively. TextCNN focuses on local semantic information, and BiGRU focuses on contextual semantic information with temporal relationships. The high-level semantic representations generated by the two are then spliced ​​and input into a feedforward neural network to obtain Chinese text output. The two branches share the embedding layer and the feedforward neural network (fully connected layer). The model structure is as follows: Figure 2 shown.

[0048] 2. The deep learning model constructed in step 1 is trained using the universal Braille-Chinese character comparison corpus.

[0049] (1) Universal Braille to Pinyin Conversion

[0050] The universal Braille ASCII code contains both Braille word segmentation information and the pronunciation information of each Chinese character corresponding to the Braille. This pronunciation information can be converted into Chinese Pinyin and word segmentation information through the long-first matching method. The conversion of universal Braille to Pinyin is mainly based on the rule mapping method. Except for a small number of Braille symbol string confusions, the Braille symbol strings basically correspond to the corresponding Pinyin syllables or punctuation marks one by one. The main confusion situations of Braille symbol strings are as follows:

[0051] a) The Braille symbol strings are the same due to the omitting of common Braille tones. For example, the tone omitting rules stipulate that for syllables with the initial consonant m, the yangping symbol is omitted. Therefore, the Braille symbol strings of the pinyin "mo2" and "mo" are the same; the tone symbols of the syllables o1, o2, o3, and o4 are omitted, while the tone symbols of the syllables e1, e2, e3, and e4 are not omitted. There is confusion in that the light tone "e" and the omitted tone "o" have the same Braille symbol string but different syllables.

[0052] b) The same Braille symbol string represents different pinyin syllables or punctuation marks, such as the syllables "yo" and "you", "yo1" and "you1", paired punctuation marks such as double quotation marks, single quotation marks, square brackets, colons and hyphens, as well as the syllable "eng" and the sign symbol "#" for Arabic numerals.

[0053] The above confused Braille symbol strings are preprocessed, and the unambiguous Braille symbol strings are converted into corresponding Pinyin syllables or punctuation marks according to the rule mapping method. For the ambiguous Braille symbol strings but with different corresponding Pinyin, the Braille symbol strings are converted into Pinyin syllables with high occurrence frequency, such as the Braille symbol strings corresponding to the syllables "you" and "yo" are uniformly converted into "you", and the Braille symbol strings corresponding to the light tone "e" and the abbreviated tone "o" are uniformly converted into "o", etc.; for the ambiguous Braille symbol strings but with different corresponding punctuation marks, they are changed to corresponding symbols according to different writing formats and positions in the sentence, such as leaving a space after the colon to distinguish the hyphen, and changing paired punctuation marks to correct punctuation marks in the order of appearance. This method can reduce the ambiguity problem of paired punctuation marks.

[0054] (2) Syllable Embedding Integrating Braille Word Segmentation Information

[0055] The BMES four-digit sequence marking method is used to represent the Braille word segmentation information. B, namely Begin, indicates the first position of the Braille word; M, namely Middle, indicates the middle position of the Braille word, that is, it is neither the first word nor the last word of the Braille word; E, namely End, indicates the end position of the Braille word; S, namely Single, indicates a single word. In one embodiment, as shown in Table 1, the period is marked as a separate word as "S" because the writing rules of Braille stipulate that there are no spaces before and after the period. If the period is marked as part of the Braille word as part of the Braille word, the original relationship of the Braille words is severed, which has a counter-effect on the semantic understanding of the sentence. Therefore, when generating the word segmentation information, the punctuation mark is marked as a separate word as "S".

[0056] Table 1 Braille word segmentation comparison information

[0057]

[0058] (3) Local semantic feature extraction based on TextCNN

[0059] The pinyin embedding integrated with word segmentation information is used as the input of TextCNN. After convolution operations with three convolution kernels of different heights, the semantic representation of the sentence is obtained. Figure 3 As shown, Figure 3It contains three convolution kernels of different heights, 1, 3, and 5, respectively. Each height has two convolution kernels. Each convolution kernel obtains a feature vector by performing a convolution operation on the sentence embedding, and finally obtains a feature vector that integrates 6 feature spaces, that is, the semantic representation of the sentence.

[0060] (3) Global semantic feature extraction based on BiGRU

[0061] In the task of converting universal Braille to Chinese characters, each sentence is a sequence of pinyin with a temporal relationship. To predict the Chinese character corresponding to the pinyin at the current position, it is necessary to judge not only based on the previous text, but also based on the following text, i.e., context information. Therefore, a bidirectional gated recurrent unit network (BiGRU) is used to capture global semantic information. An implementation example is Figure 4 shown.

[0062] The outputs of TextCNN and BiGRU are concatenated to obtain the final semantic vector and input into the feedforward neural network, whose output is passed through the softmax function to obtain the Chinese text. Figure 5 As shown in the figure, the output dimension of each sentence of TextCNN is n×k, and the output dimension of each sentence of BiGRU is n×m. The final sentence vector dimension size is n×(m+k).

[0063] When training the model, the dataset compiled from the universal Braille-Chinese character comparison corpus is divided into several batches, each of which contains several pieces of data, and each piece of data also contains: a pinyin syllable sequence, a word segmentation mark sequence, and a Chinese text corresponding to the pinyin. Each batch of data is used for training in turn. During training, the pinyin and word segmentation marks of the batch of data are used as input, and are concatenated after passing through the embedding layer to obtain the pinyin embedding containing word segmentation information. The predicted Chinese characters and Chinese text generated after the model training are sent to calculate the loss function through the cross entropy, and then the parameters of the entire model are updated through the back propagation algorithm.

[0064] 3. Optionally, use the pinyin-Chinese character comparison corpus to fine-tune the model trained in step 2.

[0065] In this corpus, each pinyin text has its corresponding Chinese text. Pinyin is a pinyin with abbreviated tones according to the universal braille tone marking standard, including syllables and tones. The tone marks are: "1" for the first tone (yinping), "2" for the second tone (yangping), "3" for the third tone (shangsheng), "4" for the fourth tone (qusheng), and no tone for the light tone or the tone is abbreviated. A braille word segmentation system is used to automatically obtain braille word segmentation information based on the Chinese characters and pinyin information.

[0066] The Chinese character corpus marked with Braille word segmentation information is used as the model input. Each piece of data is a pinyin text, the Chinese character text corresponding to each pinyin, and the corresponding word segmentation information. Figure 6 The following is an example of a Pinyin-Chinese character corpus with Braille word segmentation information marked.

[0067] When training the model, the Pinyin-Chinese character comparison corpus is divided into several batches, each batch contains several data items, and each data item contains: Pinyin, the Braille word segmentation mark corresponding to the Pinyin, and the Chinese character text corresponding to the Pinyin. Each batch of data is used for training in turn, and the training method is the same as step 2.

[0068] 4. Use the model trained in step 2 or step 3 to automatically convert universal Braille to Chinese characters.

[0069] After the model is trained, an input universal Braille ASCII code is converted into Chinese pinyin and word segmentation information, and then input into the model to obtain the corresponding prediction result. In one embodiment, the input universal Braille ASCII code is "GV'0A ZU'GO1"2", the word segmentation information obtained after conversion is "BEBES", the pinyin is "gan3 en1 zu3guo2.", and the predicted Chinese text is "Thanks to the motherland."

[0070] At the same time, based on the word segmentation information, a Chinese text containing the word segmentation information can be obtained, and the method is as follows: first, the Chinese text is segmented according to the word segmentation mark, and the Chinese characters between "B" and "E" are connected into one word according to the position correspondence, and the Chinese characters corresponding to the "S" mark are separated into words. As in the above embodiment, the Chinese text "Thanks to the motherland" can be obtained.

[0071] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0072] The present invention also proposes a universal Braille to Chinese character conversion system, which includes:

[0073] A model building module, used to build a deep learning model including a local semantic feature extraction layer and a global semantic feature extraction layer, to obtain a universal Braille text to be converted, wherein the local semantic feature extraction layer and the global semantic feature extraction layer share an embedding layer and a fully connected layer;

[0074] A preprocessing module is used to generate a pinyin syllable sequence and a word segmentation mark sequence from the universal Braille text, and concatenate them after vectorization by the embedding layer to obtain a pinyin embedding vector integrating the Braille word segmentation information, and input them into the local semantic feature extraction layer and the global semantic feature extraction layer respectively;

[0075] The feature splicing module is used for the local semantic feature extraction layer to extract the local semantic information of the pinyin embedding vector, and the global semantic feature extraction layer to extract the contextual semantic information with a temporal relationship in the pinyin embedding vector, and the local semantic information and the contextual semantic information are spliced ​​and input into the fully connected layer to obtain the Chinese character text corresponding to the universal Braille text.

[0076] The universal Braille to Chinese character conversion system, wherein the training process of the deep learning model is to divide the universal Braille-Chinese character comparison corpus into multiple batches of data, each batch of data includes multiple data, each data includes: a pinyin syllable sequence, a word segmentation mark sequence, and a Chinese character text corresponding to the pinyin; the pinyin syllable sequence and the word segmentation mark sequence of the batch of data are used as input to obtain the predicted Chinese character text of the deep learning model, a loss function is constructed according to the predicted Chinese character text and the Chinese character text corresponding to the pinyin, and the deep learning model is updated by back propagation;

[0077] A deep learning model is fine-tuned and updated using a pinyin-Chinese character comparison corpus, in which each pinyin text has its corresponding Chinese character text; pinyin is converted into pinyin with abbreviated tones according to the universal Braille tone marking standard, including syllables and tones, and the tone markings are: 1 for yinping, 2 for yangping, 3 for shangsheng, 4 for qusheng, and no tone indicates a light tone or the tone is abbreviated; a Braille word segmentation system is used to automatically obtain Braille word segmentation information based on the Chinese character and pinyin information, thereby obtaining the pinyin-Chinese character comparison corpus with marked word segmentation information.

[0078] The universal Braille to Chinese character conversion system, wherein the word segmentation mark sequence is based on the BMES four-bit sequence annotation to represent the word segmentation information of each pinyin syllable in the pinyin syllable sequence, wherein B represents the first position of the pinyin syllable corresponding to the Braille word; M represents the middle position of the pinyin syllable corresponding to the Braille word; E represents the end position of the pinyin syllable corresponding to the Braille word; and S represents the pinyin syllable corresponding to a separate character.

[0079] The universal Braille to Chinese character conversion system, wherein the local semantic feature extraction layer includes three convolution kernels of different heights, the heights are 1, 3 and 5 respectively, each height has two convolution kernels respectively, each convolution kernel obtains a feature vector by performing a convolution operation on the pinyin embedding vector, and finally obtains a feature vector that fuses 6 feature spaces to obtain local semantic information; the global semantic feature extraction layer adopts a bidirectional gated recurrent unit network to realize the capture of the contextual semantic information.

[0080] The present invention also proposes a storage medium for storing a program for executing any one of the methods for converting universal Braille to Chinese characters.

[0081] The present invention also provides a client, which is used for any of the above-mentioned universal Braille to Chinese character conversion systems.

Claims

1. A method for converting universal Braille to Chinese characters, characterized in that: include: Step 1: construct a deep learning model including a local semantic feature extraction layer and a global semantic feature extraction layer to obtain a universal Braille text to be converted, and the local semantic feature extraction layer and the global semantic feature extraction layer share an embedding layer and a fully connected layer; Step 2: Generate a pinyin syllable sequence and a word segmentation mark sequence from the universal Braille text, vectorize them through the embedding layer, and then concatenate them to obtain a pinyin embedding vector that integrates the Braille word segmentation information, and input them into the local semantic feature extraction layer and the global semantic feature extraction layer respectively; Step 3, the local semantic feature extraction layer extracts the local semantic information of the pinyin embedding vector, the global semantic feature extraction layer extracts the contextual semantic information with a temporal relationship in the pinyin embedding vector, and the local semantic information and the contextual semantic information are spliced ​​and input into the fully connected layer to obtain the Chinese character text corresponding to the universal Braille text; The local semantic feature extraction layer includes three convolution kernels of different heights, namely 1, 3 and 5, and each height has two convolution kernels. Each convolution kernel obtains a feature vector by performing a convolution operation on the pinyin embedding vector, and finally obtains a feature vector that integrates 6 feature spaces to obtain local semantic information. The global semantic feature extraction layer uses a bidirectional gated recurrent unit network to capture the contextual semantic information.

2. The method for converting universal Braille to Chinese characters as claimed in claim 1, characterized in that: The training process of the deep learning model is to divide the universal Braille-Chinese character comparison corpus into multiple batches of data, each batch of data includes multiple data, and each data includes: a pinyin syllable sequence, a word segmentation mark sequence, and a Chinese character text corresponding to the pinyin; the pinyin syllable sequence and the word segmentation mark sequence of the batch of data are used as input to obtain the predicted Chinese character text of the deep learning model, and a loss function is constructed according to the predicted Chinese character text and the Chinese character text corresponding to the pinyin, and the deep learning model is updated by back propagation; A deep learning model is fine-tuned and updated using a pinyin-Chinese character comparison corpus, in which each pinyin text has its corresponding Chinese character text; pinyin is converted into pinyin with abbreviated tones according to the universal Braille tone marking standard, including syllables and tones, and the tone markings are: 1 for yinping, 2 for yangping, 3 for shangsheng, 4 for qusheng, and no tone indicates a light tone or the tone is abbreviated; a Braille word segmentation system is used to automatically obtain Braille word segmentation information based on the Chinese character and pinyin information, thereby obtaining the pinyin-Chinese character comparison corpus with marked word segmentation information.

3. The method for converting universal Braille to Chinese characters as claimed in claim 1, characterized in that: The word segmentation mark sequence represents the word segmentation information of each pinyin syllable in the pinyin syllable sequence based on the BMES four-bit sequence annotation, where B indicates that the pinyin syllable corresponds to the first position of the Braille word; M indicates that the pinyin syllable corresponds to the middle position of the Braille word; E indicates that the pinyin syllable corresponds to the end position of the Braille word; and S indicates that the pinyin syllable corresponds to a separate character.

4. A universal Braille to Chinese character conversion system, characterized in that: include: A model building module, used to build a deep learning model including a local semantic feature extraction layer and a global semantic feature extraction layer, to obtain a universal Braille text to be converted, wherein the local semantic feature extraction layer and the global semantic feature extraction layer share an embedding layer and a fully connected layer; A preprocessing module is used to generate a pinyin syllable sequence and a word segmentation mark sequence from the universal Braille text, and concatenate them after vectorization by the embedding layer to obtain a pinyin embedding vector integrating the Braille word segmentation information, and input them into the local semantic feature extraction layer and the global semantic feature extraction layer respectively; A feature splicing module, which is used for the local semantic feature extraction layer to extract the local semantic information of the pinyin embedding vector, the global semantic feature extraction layer to extract the contextual semantic information with a temporal relationship in the pinyin embedding vector, and splicing the local semantic information and the contextual semantic information and inputting them into the fully connected layer to obtain the Chinese text corresponding to the universal Braille text; The local semantic feature extraction layer includes three convolution kernels of different heights, namely 1, 3 and 5, and each height has two convolution kernels. Each convolution kernel obtains a feature vector by performing a convolution operation on the pinyin embedding vector, and finally obtains a feature vector that integrates 6 feature spaces to obtain local semantic information. The global semantic feature extraction layer uses a bidirectional gated recurrent unit network to capture the contextual semantic information.

5. The universal Braille to Chinese character conversion system as claimed in claim 4, characterized in that: The training process of the deep learning model is to divide the universal Braille-Chinese character comparison corpus into multiple batches of data, each batch of data includes multiple data, and each data includes: a pinyin syllable sequence, a word segmentation mark sequence, and a Chinese character text corresponding to the pinyin; the pinyin syllable sequence and the word segmentation mark sequence of the batch of data are used as input to obtain the predicted Chinese character text of the deep learning model, and a loss function is constructed according to the predicted Chinese character text and the Chinese character text corresponding to the pinyin, and the deep learning model is updated by back propagation; A deep learning model is fine-tuned and updated using a pinyin-Chinese character comparison corpus, in which each pinyin text has its corresponding Chinese character text; pinyin is converted into pinyin with abbreviated tones according to the universal Braille tone marking standard, including syllables and tones, and the tone markings are: 1 for yinping, 2 for yangping, 3 for shangsheng, 4 for qusheng, and no tone indicates a light tone or the tone is abbreviated; a Braille word segmentation system is used to automatically obtain Braille word segmentation information based on the Chinese character and pinyin information, thereby obtaining the pinyin-Chinese character comparison corpus with marked word segmentation information.

6. The universal Braille to Chinese character conversion system as claimed in claim 4, characterized in that: The word segmentation mark sequence represents the word segmentation information of each pinyin syllable in the pinyin syllable sequence based on the BMES four-bit sequence annotation, where B indicates that the pinyin syllable corresponds to the first position of the Braille word; M indicates that the pinyin syllable corresponds to the middle position of the Braille word; E indicates that the pinyin syllable corresponds to the end position of the Braille word; and S indicates that the pinyin syllable corresponds to a separate character.

7. A storage medium for storing a program for executing the method for converting universal Braille to Chinese characters as claimed in any one of claims 1 to 3.

8. A client, used in the universal Braille to Chinese character conversion system according to any one of claims 4 to 7.

Images