Deep learning-based Chinese braille text intelligent conversion system

Abstract

The invention relates to the technical field of Chinese braille text intelligent conversion, and discloses a deep learning-based Chinese braille text intelligent conversion system, which comprises an input module, an image processing module, a data set construction and preprocessing module, a model training and optimizing module, an output processing module and a text conversion module, handwritten braille alphabets are recognized through mobile phone photographing, and the image processing module comprises image inclination correction and sharpening and braille alphabet image segmentation. The method comprises the following steps of: identifying by using a two-party model, leaving a part with confidence higher than a certain value, identifying the rest part by using a one-party model, leaving a part with confidence higher than a certain value, arranging the two parts according to a pixel sequence to form a format suitable for LLM input, and generating corresponding text representation; the problem that Chinese braille symbols are not in one-to-one correspondence with initial consonants and final consonants is avoided, conversion from Chinese pinyin to Chinese texts is achieved in combination with a large language model, and the error-tolerant rate of large language model translation to original texts is high.

Description

Technical Field

[0001] The present invention relates to Braille recognition technology and translation, which belongs to the field of blind education and assistive technology, and specifically to a Chinese Braille text intelligent conversion system based on deep learning. Background Art

[0002] Braille recognition technology and translation are important research directions in the fields of blind education and assistive technology. In the current context, most Braille users are visually impaired teenagers, and Braille reading is quite difficult for ordinary sighted people and even special education teachers. This has created a huge obstacle to the inclusive education of visually impaired students. If scenarios in life such as homework grading and college entrance examination paper marking can be replaced by automated programs, it will undoubtedly greatly save manpower and material resources, which is of great significance to promoting educational equity.

[0003] Currently, in the process of converting pinyin into text and Braille text, deep learning models LeNet-5 or CNN convolutional neural networks are usually used for Braille image recognition, which can convert Braille images into natural language text and is suitable for multilingual Braille conversion. However, due to the lack of understanding of the overall nature of Chinese Braille, these methods use a one-to-one correspondence between Braille symbols and initials and finals, making it impossible to automatically translate the recognized content into readable Chinese text and thus unable to be applied in actual scenarios. In addition, the conversion accuracy of pinyin is low, which leads to poor performance of the system. Therefore, a Chinese Braille text intelligent conversion system based on deep learning is proposed to solve the above problems. Summary of the Invention

[0004] (1) Technical problems solved

[0005] In response to the shortcomings of the existing technology, the present invention provides a Chinese Braille text intelligent conversion system based on deep learning, which has the advantage of high conversion success rate. It solves the current problem that in the process of converting pinyin into text and Braille text, deep learning models LeNet-5 or CNN convolutional neural networks are usually used for Braille image recognition. It can convert Braille images into natural language text and is suitable for multilingual Braille conversion. However, due to the lack of understanding of the overall nature of Chinese Braille, these methods adopt a one-to-one correspondence between Braille symbols and initials and finals, which makes it impossible to automatically translate the recognized content into readable Chinese text and thus cannot be applied in actual scenarios. In addition, the conversion accuracy of pinyin is low, which leads to the problem that the use effect of the system is poor.

[0006] (2) Technical solution

[0007] To achieve the above-mentioned intelligent text conversion purpose, the present invention provides the following technical solution: a Chinese Braille text intelligent conversion system based on deep learning, including an input module, an image processing module, a data set construction and preprocessing module, a model training and optimization module, an output processing module and a text conversion module. The input module recognizes handwritten Braille by taking a photo with a mobile phone;

[0008] The image processing module includes image tilt correction and sharpening and Braille image segmentation;

[0009] The image preprocessing module includes data collection and annotation, data enhancement and expansion, and image normalization to ensure data consistency and availability;

[0010] The model training and optimization module includes model selection and training, parameter adjustment and optimization, and multi-classification tasks;

[0011] The output processing module organizes the results: first, the two models are used for recognition, and the part with a confidence score higher than a certain value is retained. The remaining part is recognized by the single model, and the part with a confidence score higher than a certain value is retained. The two parts are arranged in pixel order to form a format suitable for LLM input, so as to generate the corresponding text representation;

[0012] The text conversion module, LLM calls: using a large language model (deepseek) to convert Chinese pinyin into Chinese text output. By calling the deepseek API, setting prompt to achieve pinyin to Chinese conversion with certain error correction capabilities.

[0013] Preferably, the data preprocessing module:

[0014] For images: correct image tilt and perform pre-processing such as sharpening and binarization;

[0015] Standardize pinyin: including tone annotation and error correction, and output standardized pinyin sequences as input to the core conversion module;

[0016] For Braille: The deflection angle of the Braille image is detected through the grayscale projection integral image, and the image is rotated to horizontal alignment to eliminate acquisition errors.

[0017] Preferably, the image processing module specifically includes:

[0018] (1) Tilt correction and sharpening: Use image processing technology to perform tilt correction and sharpening on the Braille image and delete the redundant parts to improve the data quality. The grayscale projection method can be used for tilt correction.

[0019] (2) Braille image segmentation: The Braille image is divided into squares for the subsequent construction of the classification dataset. Due to the regular arrangement of Braille, this step can be completed by dividing the pixels proportionally.

[0020] Preferably, the dataset construction and preprocessing includes:

[0021] (1) Data collection and annotation: Collect Chinese Braille image data, use mobile phones to capture images and perform manual annotation;

[0022] (2) Data augmentation and expansion: Data augmentation techniques such as adding random noise are used to expand the data set and improve the generalization ability of the model. The Albumentations library provides a variety of data augmentation algorithms that can effectively improve training results.

[0023] (3) Preprocessing step: Normalize the image to ensure the consistency and usability of the data.

[0024] Preferably, the Chinese-Braille text intelligent conversion system comprises the following steps:

[0025] Step 1: Image processing: correct image tilt and perform preprocessing such as sharpening and binarization;

[0026] Step 2: Dataset construction and preprocessing: Collect and annotate a large amount of Chinese Braille image data, and perform data enhancement and preprocessing;

[0027] Step 3: Feature extraction and model design: Use the Inception model to extract image features and design a model architecture suitable for Braille recognition;

[0028] Step 4: Model training and optimization: Use methods such as transfer learning and adversarial training to optimize model performance and improve recognition accuracy;

[0029] Step 5: Output processing: Rationally organize the recognition results into a format suitable for LLM input;

[0030] Step 6: Text conversion: Call LLM to convert Chinese Pinyin into Chinese text output.

[0031] Preferably, the model training and optimization specifically include:

[0032] (1) Model selection and training: Convolutional neural network (CNN) is used as the basic model, and transfer learning technology is combined to optimize model performance. Transfer learning can extract features through pre-trained models (such as ResNet, VGG, etc.) and fine-tune them on the Braille dataset;

[0033] (2) Parameter adjustment and optimization: Improve the accuracy and stability of the model by adjusting hyperparameters (such as learning rate, batch size, optimizer, etc.) and regularization techniques (such as Dropout, L2 regularization);

[0034] (3) Multi-classification task: For the 360 multi-classification problem, a classifier suitable for Braille recognition is designed, and the model performance is evaluated through cross-validation.

[0035] The steps of the Inception model image extraction are:

[0036] S1 converts the Braille image or other input image into a suitable standardized data format, including resizing: scaling the image to 299 × 299 pixels and normalizing the pixel values to the range [-1, 1];

[0037] S2 Braille image special processing, including image tilt correction through affine transformation, and Braille dot matrix must be strictly aligned;

[0038] Contrast enhancement: Use CLAHE (Contrast Limited Adaptive Histogram Equalization) to improve the distinction between the dots and the background;

[0039] Binarization: Perform Otsu threshold segmentation on the grayscale image to highlight the Braille dots (0 / 1 binarization);

[0040] S3 performs feature extraction, using multi-scale convolution kernels (1×1, 3×3, 5×5) to simultaneously capture local details (such as a single Braille dot) and global structures (such as the arrangement of Braille squares). It then converts the final feature map (8×8×2048) into a 2048-dimensional vector through global average pooling.

[0041] S4 completes multiple tasks, classifying Braille symbols and outputting pinyin labels, and regressing dot coordinates to output Braille square positions.

[0042] Preferably, the loss function of the transfer learning includes: source domain loss: used to retain the knowledge of the source domain and target domain loss: used to optimize the performance of the target task.

[0043] The formula is:

[0044] L=α·Ls(fs,Ds)+β·Lt(ft,Dt)

[0045] Among them, Ls is the loss function of the source domain (such as classification error, regression error, etc.).

[0046] · Lt is the loss function for the target domain.

[0047] α and β are weight coefficients used to balance the losses in the source and target domains.

[0048] Ls is the loss function of the source domain (such as classification error, regression error, etc.).

[0049] Lt is the loss function for the target domain.

[0050] α and β are weight coefficients used to balance the losses in the source and target domains.

[0051] Preferably, the optimization of the large language model further includes the anti-training:

[0052] Generate adversarial examples: Use methods such as gradient ascent to add small perturbations to the input data to cause the model to make incorrect predictions. Add the adversarial examples to the training set so that the model can correctly classify the adversarial examples while minimizing the original loss.

[0053] Preferably, in the large language model, the LLM pre-training corpus contains 1 billion-level pinyin-Chinese character alignment data to achieve priority matching of high-frequency words.

[0054] (3) Beneficial effects

[0055] Compared with the existing technology, the present invention provides a Chinese-to-Braille text intelligent conversion system based on deep learning, which has the following beneficial effects:

[0056] 1. This deep learning-based Chinese Braille text intelligent conversion system first uses two models for recognition, retaining the part with a confidence level higher than a certain value. The remaining part is then recognized by one model, retaining the part with a confidence level higher than a certain value. These two parts are arranged in pixel order to form a format suitable for LLM input, so that the corresponding text representation is generated to avoid the problem that Chinese Braille symbols do not correspond one-to-one with initials and finals. Combined with a large language model, it realizes the conversion of Chinese pinyin to Chinese text. The large language model translation has a high error tolerance rate for the original text.

[0057] 2. This deep learning-based Chinese Braille text intelligent conversion system combines a large language model to achieve the conversion of Chinese pinyin to Chinese text. The large language model translation has a high error tolerance rate for the original text. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Schematic diagram of the intelligent conversion system of the present invention;

[0059] Figure 2 This is a technical flow chart of the present invention;

[0060] Figure 3 It is the blind square structure diagram of the present invention;

[0061] Figure 4 This is an example diagram of Braille of the present invention;

[0062] Figure 5 This is a diagram of Braille segmentation according to the present invention;

[0063] Figure 6 This is a reference structural diagram of the model of the present invention. DETAILED DESCRIPTION

[0064] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0065] See also Figure 1-6 The Chinese Braille text intelligent conversion system based on deep learning includes an input module, an image processing module, a data set construction and preprocessing module, a model training and optimization module, an output processing module and a text conversion module. The input module recognizes handwritten Braille by taking a photo with a mobile phone;

[0066] The image processing module includes image tilt correction and sharpening and Braille image segmentation;

[0067] The image preprocessing module includes data collection and annotation, data enhancement and expansion, and image normalization to ensure data consistency and availability;

[0068] The model training and optimization module includes model selection and training, parameter adjustment and optimization, and multi-classification tasks;

[0069] The output processing module organizes the results: first, the two models are used for recognition, and the part with a confidence score higher than a certain value is retained. The remaining part is recognized by the single model, and the part with a confidence score higher than a certain value is retained. The two parts are arranged in pixel order to form a format suitable for LLM input, so as to generate the corresponding text representation;

[0070] The text conversion module, LLM calls: uses a large language model (deepseek) to convert Chinese pinyin into Chinese text output, and by calling the deepseek API and setting the prompt, it realizes the pinyin to Chinese conversion with certain error correction capabilities.

[0071] Example

[0072] like Figure 3 The following is the structure of the blind square, that is, the structure of a single Braille character. Each blind square consists of 6 dots, and the "convex" and "flat" of these dots form different shapes to correspond to different Chinese pinyin and symbols. These positions are as follows Figure 3The shown is marked as the 123456 position, from which Braille symbols can be represented by Arabic numerals. For example, "Hello" can be represented in Braille as "1345 24125 235".

[0073] The main components of Chinese Braille are the "initial consonants" and "final vowels" in Chinese pinyin, but they do not have a one-to-one correspondence.

[0074] From Figure 4 we can see that: ① The content in the rectangular box is the Braille of "g u", ② The content in the rectangular box is "shi", and ③ The content in the rectangular box is "j iu 3". It is not difficult to find that the first dot group in ① and ③ is both 1245 dots, but the corresponding initial consonants are different. This is the first characteristic of Chinese Braille; In ②, one dot group can represent the character "事", in ①, two dot groups represent "故", and in ③, three dot groups represent "九", where the last dot group is the tone, representing the third tone. Different Chinese characters can be represented by Braille symbols with different numbers of dot groups. This is the second characteristic of Chinese Braille. Due to the existence of these two characteristics, the effect of converting single - dot - group Braille to Chinese has often been poor in the past.

[0075] Therefore, in this study, the pinyin is first divided into "Chinese characters written in one dot group" and "Chinese characters written in two dot groups". Special persons are invited to copy the pinyin table in Braille, five times for each syllable, and images are collected using different mobile phones for making the dataset.

[0076] Table 1 Classification Table of Chinese Pinyin Syllables by the Number of Dot Groups

[0077]

[0078] Pre - process the Braille images, including correcting the image tilt and deleting the redundant parts to improve the image quality. Subsequently, the Braille images are divided into multiple small images according to the grids for subsequent feature extraction and classification recognition. Image pre - processing technology is an important foundation of the Braille recognition system. Through means such as image enhancement and noise removal, the recognition accuracy of the subsequent model can be effectively improved.

[0079] Annotate the Chinese handwritten Braille image data, and construct a high - quality image classification dataset for the data of two dot groups and one dot group. On this basis, use data augmentation technology to expand the dataset. For example, generate new training samples by adding random noise and other means, so as to improve the generalization ability of the model. At the same time, normalize the data to ensure the consistency and usability of the data.

[0080] Train the model with the two datasets respectively, and the reference results are as Figure 6As shown in the figure, during the model training phase, model parameters are adjusted and performance is optimized using methods such as transfer learning. Transfer learning leverages knowledge from pre-trained models to accelerate the training process. Furthermore, loss function optimization strategies (such as minimizing error) are employed to further improve the model's recognition accuracy. Finally, model performance is evaluated using validation and test sets to ensure stability and reliability in different scenarios.

[0081] First, the two models are used for recognition, retaining the parts with a confidence score above a certain value. Then, the remaining parts are recognized by the single model, retaining the parts with a confidence score above a certain value. These two parts are then arranged in pixel order to form a format suitable for input to the Large Language Model (LLM). This process ensures the smooth progress of subsequent text conversion tasks.

[0082] The final step is to call the LLM to convert Pinyin characters into Chinese text output. This study leverages the powerful LLM (DeepSeek) for Chinese processing, using techniques such as word segmentation, word embedding, and BPE encoding to efficiently convert Pinyin characters into Chinese text. Furthermore, the generated text undergoes post-processing, such as spelling correction and grammar optimization, allowing for a certain amount of spelling errors to improve the accuracy and readability of the output.

[0083] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0084] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A deep learning-based intelligent Chinese-to-Braille text conversion system, comprising an input module, an image processing module, a dataset construction and preprocessing module, a model training and optimization module, an output processing module, and a text conversion module, characterized by: The input module recognizes handwritten Braille by taking photos with a mobile phone; The image processing module includes image tilt correction and sharpening and Braille image segmentation; The image preprocessing module includes data collection and annotation, data enhancement and expansion, and image normalization to ensure data consistency and availability; The model training and optimization module includes model selection and training, parameter adjustment and optimization, and multi-classification tasks; The output processing module organizes the results: first, the two models are used for recognition, and the part with a confidence score higher than a certain value is retained. The remaining part is recognized by the single model, and the part with a confidence score higher than a certain value is retained. The two parts are arranged in pixel order to form a format suitable for LLM input, so as to generate the corresponding text representation; The text conversion module, LLM calls: uses a large language model (deepseek) to convert Chinese pinyin into Chinese text output, and by calling the deepseek API and setting the prompt, it realizes the pinyin to Chinese conversion with certain error correction capabilities.

2. The Chinese-to-Braille text intelligent conversion system based on deep learning according to claim 1 is characterized by: The image processing module specifically includes: (1) Tilt correction and sharpening: Use image processing technology to perform tilt correction and sharpening on the Braille image and delete the redundant parts to improve the data quality. The grayscale projection method can be used for tilt correction. (2) Braille image segmentation: The Braille image is divided into squares for the subsequent construction of the classification dataset. Due to the regular arrangement of Braille, this step can be completed by dividing the pixels proportionally.

3. The Chinese-to-Braille text intelligent conversion system based on deep learning according to claim 1 is characterized in that: The dataset construction and preprocessing include: (1) Data collection and annotation: Collect Chinese Braille image data, use mobile phones to capture images and perform manual annotation; (2) Data augmentation and expansion: Data augmentation techniques such as adding random noise are used to expand the data set and improve the generalization ability of the model. The Albumentations library provides a variety of data augmentation algorithms that can effectively improve training results. (3) Preprocessing step: Normalize the image to ensure the consistency and usability of the data.

4. The Chinese-to-Braille text intelligent conversion system based on deep learning according to claim 1, characterized in that: The Chinese Braille text intelligent conversion system includes the following steps: Step 1: Image processing: correct image tilt and perform preprocessing such as sharpening and binarization; Step 2: Dataset construction and preprocessing: Collect and annotate a large amount of Chinese Braille image data, and perform data enhancement and preprocessing; Step 3: Feature extraction and model design: Use the Inception model to extract image features and design a model architecture suitable for Braille recognition; Step 4: Model training and optimization: Use methods such as transfer learning and adversarial training to optimize model performance and improve recognition accuracy; Step 5: Output processing: Rationally organize the recognition results into a format suitable for LLM input; Step 6: Text conversion: Call LLM to convert Chinese Pinyin into Chinese text output.

5. The Chinese-to-Braille text intelligent conversion system based on deep learning according to claim 1, characterized in that: The model training and optimization specifically include: (1) Model selection and training: Convolutional neural network (CNN) is used as the basic model, and transfer learning technology is combined to optimize model performance. Transfer learning can extract features through pre-trained models (such as ResNet, VGG, etc.) and fine-tune them on the Braille dataset; (2) Parameter adjustment and optimization: Improve the accuracy and stability of the model by adjusting hyperparameters (such as learning rate, batch size, optimizer, etc.) and regularization techniques (such as Dropout, L2 regularization); (3) Multi-classification task: For the 360 multi-classification problem, a classifier suitable for Braille recognition is designed, and the model performance is evaluated through cross-validation.

6. The Chinese-to-Braille text intelligent conversion system based on deep learning according to claim 1, characterized in that: The steps of the Inception model image extraction are: S1 converts the Braille image or other input image into a suitable standardized data format, including resizing: scaling the image to 299 × 299 pixels and normalizing the pixel values to the range [-1, 1]; S2 Braille image special processing, including image tilt correction through affine transformation, and Braille dot matrix must be strictly aligned; Contrast enhancement: Use CLAHE (Contrast Limited Adaptive Histogram Equalization) to improve the distinction between the dots and the background; Binarization: Perform Otsu threshold segmentation on the grayscale image to highlight the Braille dots (0 / 1 binarization); S3 performs feature extraction, using multi-scale convolution kernels (1×1, 3×3, 5×5) to simultaneously capture local details (such as a single Braille dot) and global structures (such as the arrangement of Braille squares). It then converts the final feature map (8×8×2048) into a 2048-dimensional vector through global average pooling. S4 completes multiple tasks, classifying Braille symbols and outputting pinyin labels, and regressing dot coordinates to output Braille square positions.

7. The Chinese-to-Braille text intelligent conversion system based on deep learning according to claim 1, characterized in that: The loss function of the transfer learning includes: source domain loss: used to retain the knowledge of the source domain and target domain loss: used to optimize the performance of the target task. The formula is: L=α·Ls(fs,Ds)+β·Lt(ft,Dt) Among them, Ls is the loss function of the source domain (such as classification error, regression error, etc.). · Lt is the loss function for the target domain. α and β are weight coefficients used to balance the losses in the source and target domains. Ls is the loss function of the source domain (such as classification error, regression error, etc.). Lt is the loss function for the target domain. α and β are weight coefficients used to balance the losses in the source and target domains.

8. The Chinese-to-Braille text intelligent conversion system based on deep learning according to claim 1, characterized in that: The optimization of the large language model also includes the anti-training: Generate adversarial examples: Use methods such as gradient ascent to add small perturbations to the input data to cause the model to make incorrect predictions. Add the adversarial examples to the training set so that the model can correctly classify the adversarial examples while minimizing the original loss.

9. The Chinese-to-Braille text intelligent conversion system based on deep learning according to claim 1, characterized in that: In the large language model, the LLM pre-training corpus contains 1 billion pinyin-Chinese character alignment data to achieve priority matching of high-frequency words.

Images