A method for braille recognition, error correction and multi-language translation based on tactile perception

Abstract

The application discloses a Braille recognition, error correction and multi-language translation method based on tactile perception, and relates to the technical field of image recognition, mainly comprising the following steps: collecting Braille touch signals through a flexible tactile sensor array; performing time-space clustering and multi-angle rotation matching on the signals to determine the standard point position layout of each Braille character; judging the reading direction in combination with the touch point cluster movement vector and outputting the standardized Braille point position sequence; calling a multi-standard Braille library to decode and identifying the language type based on semantic rationality to obtain an initial decoding text; sequentially performing rule matching error correction and sequence-to-sequence or Transformer model semantic error correction to obtain a correct text sequence; and finally translating the corrected text into a target language and outputting the text in the form of voice. The application realizes accurate acquisition and barrier-free broadcasting of Braille information through natural touch self-adaptive recognition of direction, multi-standard decoding, hybrid semantic error correction and multi-language voice translation.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, specifically to a method for Braille recognition, error correction, and multilingual translation based on tactile perception. Background Technology

[0002] Braille is an important medium for visually impaired people to obtain information, and it is widely used in public signs, books, equipment operating instructions, and other scenarios. Currently, reading Braille mainly relies on the visually impaired person's own tactile perception and learning accumulation. However, the Braille system is complex, with diverse standards, resulting in high learning costs. Furthermore, there are significant differences in Braille coding rules between different countries and locations (e.g., China's current Braille uses a phonetic-vowel structure, while English Blair Braille is based on a six-dot system of Latin letters). This poses a great challenge to visually impaired people in obtaining information across different scenarios. In addition, in practical applications, some Braille signs have spelling errors or missing, added, or replaced dots, further increasing the difficulty of recognition.

[0003] To assist visually impaired individuals in recognizing Braille, various electronic Braille recognition solutions have emerged in existing technologies. One type is based on optical image recognition, which uses a camera to capture images of Braille dots and then employs computer vision algorithms for character recognition. This type of solution achieves high accuracy under ideal lighting conditions, but it is sensitive to ambient light, with a significant drop in recognition rate under low light or highly reflective surfaces. Furthermore, the devices are bulky and lack portability. Another type is based on capacitive sensing, which locates Braille dots by detecting changes in capacitance caused by touch. It boasts a fast response time, but the capacitance value is easily affected by ambient humidity, and the system's failure probability increases significantly when the user wears gloves or has wet fingers. Additionally, there are wearable Braille recognition systems based on flexible piezoresistive sensor arrays. For example, piezoresistive sensors are integrated into gloves or finger sleeves, collecting resistance signals and using template matching or convolutional neural networks for Braille character recognition. These solutions improve portability and anti-interference capabilities to some extent, but their recognition process usually assumes that the user touches the Braille in a fixed direction and order, and cannot adaptively handle the positional ambiguity caused by finger rotation, tilting, or different reading directions. At the same time, existing systems can only recognize standard, error-free Braille, lack the ability to detect and correct errors in Braille, and do not have the function of automatic recognition and translation of cross-language Braille, making it difficult to meet the usage needs of visually impaired people in complex real-world environments.

[0004] Therefore, there is an urgent need for a portable Braille assistive device and method that can adaptively identify the direction of Braille touch, has error detection and semantic error correction capabilities, and supports multi-standard Braille decoding and multilingual translation, in order to overcome the above-mentioned shortcomings of the existing technology. Summary of the Invention

[0005] To adaptively recognize the touch direction of Braille, identify and correct semantic errors, and support multi-standard Braille decoding and multilingual translation, this invention proposes a tactile perception-based method for Braille recognition, error correction, and multilingual translation, comprising the following steps: S1: By integrating a flexible tactile sensor array into the contact surface of the wearable device, the raw tactile signal generated when the user touches the Braille sign includes the coordinates of the touch point, the touch timestamp, and the pressure information; S2: Perform spatiotemporal clustering processing on the original tactile signal, group multiple touch events that occur within a preset time window into a touch cluster of a Braille character, and perform spatial matching of multiple preset rotation angles on each touch cluster to determine the standard point layout corresponding to the touch cluster. S3: By analyzing the center point movement vectors of multiple consecutive touch point clusters, the global reading direction is determined, and combined with the standardized point layout of each touch point cluster, a standardized Braille point sequence arranged in the correct reading order is output. S4: Call multiple standard Braille coding databases, decode the standardized Braille dot sequence through Braille coding rules of different languages, and identify the language type of the current Braille mark and the initial decoded text based on the semantic rationality of the decoding result; S5: First, perform rule matching and error correction on the initial decoded text, then use a sequence-to-sequence model or Transformer model for semantic correction, and output the corrected text sequence. S6: Take the corrected text sequence as the source language text, translate it into the user-preset target language text, and convert it into a speech signal to output to the user.

[0006] This invention integrates wearable tactile data acquisition, orientation-adaptive recognition, multi-standard Braille decoding, hybrid error correction, and multilingual translation, enabling users to accurately identify Braille symbols of different directions, standards, and even those with errors without learning Braille, and to obtain target language speech output. This significantly improves the convenience and accuracy of information acquisition for visually impaired individuals.

[0007] Furthermore, in step S1, the flexible tactile sensor array is composed of one or more of the following: flexible piezoresistive sensor, flexible piezoelectric sensor, and flexible capacitive touch sensor.

[0008] Furthermore, in step S1, the wearable device is a fingerless glove, and the flexible tactile sensor array is integrated into the contact surface on the inner side of the fingertip of the index finger and / or middle finger of the fingerless glove.

[0009] Furthermore, in step S2, the preset rotation angle includes 0°, 90°, 180° and 270°; the spatial matching uses Hausdorff distance or least squares method to calculate the matching degree between the touch cluster and the standard Braille dot template.

[0010] Furthermore, in step S3, the global reading direction includes a horizontal reading direction and a vertical reading direction; the calculation of the center point movement vector is based on the centroid coordinate difference of the continuous contact point cluster.

[0011] Furthermore, in step S4, the identification of the language type of the current Braille sign based on the semantic rationality of the decoding result specifically includes: The first few characters of the standardized Braille dot sequence are decoded using Braille coding rules for different language types. The reasonableness score of the decoding result in the word fragment or syllable combination rule in the corresponding language type is calculated, and the language type with the highest score is selected as the recognition result.

[0012] Furthermore, in step S5, the rule base used for rule matching and error correction includes point missing rules, point addition rules, and point replacement rules; the point replacement rules include at least adjacent point confusion rules.

[0013] Furthermore, in step S5, the sequence-to-sequence model or Transformer model is a lightweight model pre-trained on parallel corpora of noisy Braille sequences and correct text, and deployed in an embedded device after quantization processing.

[0014] Furthermore, in step S6, the translation of the corrected text sequence into the user-preset target language text is achieved using an embedded statistical machine translation model or a pruned and optimized neural machine translation model.

[0015] Compared with the prior art, the present invention has at least the following beneficial effects: (1) The present invention proposes a Braille recognition, error correction and multilingual translation method based on tactile perception. By integrating a flexible tactile sensor array into the contact surface of a wearable device, the tactile signals of Braille symbols are collected in a natural touch manner. Users can complete the information acquisition without learning any Braille knowledge, which significantly reduces the threshold for use. (2) By performing spatiotemporal clustering and multi-angle rotation space matching on the original tactile signals, combined with the analysis of the movement vector of the center point of the continuous touch cluster, it can automatically adapt to different touch directions and reading order, eliminate the ambiguity of the position caused by finger rotation or tilt, and output a standardized Braille position sequence. (3) By calling multiple standard Braille coding databases and automatically identifying the language type of the current Braille based on the semantic rationality of the decoding results, this solution is compatible with multiple Braille standards such as Chinese, English, and Japanese, without requiring users to pre-set the language; (4) In response to the large number of non-standard or erroneous Braille markings in real-world scenarios, a hybrid error correction strategy combining rule matching and sequence-to-sequence or Transformer models is adopted. This strategy can quickly handle common errors such as missing, added, or replaced points, and can also correct complex or unknown error combinations from a semantic perspective, thereby converting originally unreadable erroneous Braille into correct text that conforms to semantic logic. Attached Figure Description

[0016] Figure 1 This is a step-by-step diagram of a Braille recognition, error correction, and multilingual translation method based on tactile perception. Detailed Implementation

[0017] The present invention will be further described below with reference to specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. It should be noted that the hardware selection, parameter values, and specific algorithm implementations described in this embodiment are merely examples and are not intended to limit the scope of protection of the present invention. Those skilled in the art can use other equivalent substitutions to achieve the same function without departing from the concept of the present invention.

[0018] Example 1: Currently, Braille, as an important medium for visually impaired people to obtain information, is widely used in public signs, books, and device control panels. However, existing Braille recognition and assistive technologies have several inherent limitations. On the one hand, traditional Braille learning requires memorizing complex dot codes and grammatical rules, making it difficult for untrained individuals to quickly recognize Braille information. Furthermore, Braille standards in different countries and regions (such as current Chinese Braille and English Blair Braille) differ significantly in dot combinations and spelling methods, further increasing the barrier to entry. On the other hand, Braille signs in practical applications often suffer from spelling errors, worn or missing dots, and most existing Braille readers and other assistive devices can only recognize standard Braille, lacking the ability to detect and correct errors, and also unable to achieve cross-language translation and voice output. In addition, existing devices often adopt a fixed reader form, are bulky, and lack portability, making it difficult to meet the needs of visually impaired users in outdoor or mobile scenarios. To address the aforementioned pain points, this embodiment provides a tactile perception-based Braille recognition, error correction, and multilingual translation method. By integrating a flexible tactile sensor array into a wearable glove, and combining technologies such as orientation adaptation, hybrid error correction, and embedded translation, it achieves complete information conversion from touch to speech. Figure 1 As shown, the method mainly includes the following steps: S1: By integrating a flexible tactile sensor array into the contact surface of the wearable device, the raw tactile signal generated when the user touches the Braille sign includes the coordinates of the touch point, the touch timestamp, and the pressure information; S2: Perform spatiotemporal clustering processing on the original tactile signal, group multiple touch events that occur within a preset time window into a touch cluster of a Braille character, and perform spatial matching of multiple preset rotation angles on each touch cluster to determine the standard point layout corresponding to the touch cluster. S3: By analyzing the center point movement vectors of multiple consecutive touch point clusters, the global reading direction is determined, and combined with the standardized point layout of each touch point cluster, a standardized Braille point sequence arranged in the correct reading order is output. S4: Call multiple standard Braille coding databases, decode the standardized Braille dot sequence through Braille coding rules of different languages, and identify the language type of the current Braille mark and the initial decoded text based on the semantic rationality of the decoding result; S5: First, perform rule matching and error correction on the initial decoded text, then use a sequence-to-sequence model or Transformer model for semantic correction, and output the corrected text sequence. S6: Take the corrected text sequence as the source language text, translate it into the user-preset target language text, and convert it into a speech signal to output to the user.

[0019] This method is primarily applied to a wearable system. The system uses a fingerless glove as the wearable carrier and is mainly aimed at blind or visually impaired individuals. It helps them obtain translated speech information by naturally touching Braille on public signs, books, or devices without any prior knowledge of Braille. This embodiment illustrates this using a user wearing the glove to recognize Chinese Braille markings on a bus stop sign. The user's preset target language is Chinese speech output, and the system by default supports automatic recognition and switching between multiple Braille standards, including Chinese, English, and Japanese.

[0020] In terms of hardware implementation, the main body of the glove is made of breathable and wear-resistant fabric to ensure comfort during prolonged wear. The inner contact surfaces of the index and middle fingertips integrate a flexible tactile sensor array. In this embodiment, the sensor array uses flexible piezoresistive sensors, with each sensor unit being a square with sides of approximately 3mm, arranged in three rows and two columns, totaling six independent sensing points, precisely covering all the raised dots of a standard Braille character (six-dot system). The characteristic of a piezoresistive sensor is that when it is subjected to pressure, its internal conductive network changes, causing the resistance value to decrease exponentially with increasing pressure. By forming a voltage divider circuit with the sensor and a fixed resistor, the resistance change can be converted into an analog voltage signal. Furthermore, the sensor substrate uses a polyimide film with an overall thickness not exceeding 0.5mm, allowing it to bend freely with the fingers without affecting tactile sensitivity. A waterproof and dustproof protective shell is located on the back of the glove, integrating a main processor, storage module, power management circuitry, and Bluetooth communication module. In this embodiment, the main processor is a microcontroller with an ARM Cortex-M7 core, a clock speed greater than 300MHz, a built-in floating-point unit, and an external 16MB QSPI Flash memory for storing system programs, multi-national standard Braille coding databases, error correction rule bases, neural network model parameters, and user configuration information. The side of the back of the casing features a micro USB-C port for charging and a physical button for power on / off and volume control. An adjustable Velcro strap is designed for the wrist to accommodate different wrist sizes.

[0021] After the user puts on the gloves and powers on the device, the system automatically enters standby mode, listening to sensor signals with low power. When the user approaches a bus stop and naturally touches the Braille markings on the sign with their finger, the flexible sensor array at the fingertip of the glove begins to operate. The user typically slides their index fingertip from left to right across the Braille lines, sequentially touching multiple Braille characters. Each Braille character consists of several raised dots, which press against the corresponding sensing unit in the sensor array. The piezoresistive sensor's resistance decreases when pressed, causing a voltage increase at the output of the voltage divider circuit. This analog voltage signal is converted into a digital value after passing through a locally integrated signal conditioning circuit (including pre-amplifier, low-pass filter, and analog-to-digital converter). The raw tactile signal contains three dimensions of information: the identifier of the triggered sensor channel (i.e., the point coordinates), the absolute timestamp of the signal generation, and the signal strength (proportional to the pressure applied). The system continuously collects this data at a sampling rate of approximately 200Hz, forming a time-series raw data stream.

[0022] Next, the main processor executes the Braille direction recognition and character serialization algorithm. This algorithm first preprocesses the raw signal. Since slight tremors or unintentional touches may occur during the touch process, the system sets a pressure threshold and a de-jittering time window (e.g., 20 milliseconds). Only events with signal strength exceeding the threshold and duration exceeding the de-jittering window are considered valid touch events. Each valid touch event is recorded as a data element containing a timestamp, sensor ID, and pressure value. Then, the algorithm uses a spatiotemporal clustering method to divide consecutive touch events into different character clusters. Specifically, the system sets a maximum time interval (e.g., 500 milliseconds). If the time difference between two adjacent touch events is less than this interval, they are considered to belong to the same Braille character touch action; conversely, if the time difference exceeds the interval, the previous character is considered finished, and the new event belongs to the next character. In this way, the system divides the dozens of touch events generated during the user's entire swipe touch process into several independent touch clusters, each corresponding to a Braille character.

[0023] For each touch cluster, there is a sensor channel ID corresponding to all the pressed bumps in that character. However, when a user touches, their finger may not be perfectly horizontal, but rather rotated at a certain angle (e.g., the wrist naturally tilts outward, causing the fingertip to tilt by 30°). To obtain the correct dot encoding, the system needs to perform orientation correction on each touch cluster. The algorithm internally stores a template layout of standard Braille dots, i.e., the coordinate positions of six dots in three rows and two columns. The system attempts to rotate the dot set in the current touch cluster by 0°, 90°, 180°, and 270° respectively, and then spatially match it with the standard template. The matching metric uses the least squares sum of squared errors, i.e., calculating the sum of the squares of the distances from each measured point to the nearest standard template dot, and selecting the rotation angle with the smallest error as the best match. For example, the top-left corner dot (dot 1) of a standard Braille character, if the user rotates their finger 90° clockwise, will fall to the top-right position of the sensor array (the original position of dot 4). Based on this rotation assumption, the system will find that rotating the measured point set 90° counter-clockwise aligns all points with the corresponding points on the standard template, thus correcting the layout of the standard dots. Finally, the six dots of each character are converted into a six-bit binary array, such as [1,0,0,1,0,0], where 1 indicates that the dot has a dot and 0 indicates that it does not.

[0024] After determining the standard dot layout of each character, the algorithm also needs to determine the global reading direction of the entire Braille label, i.e., whether the user is reading horizontally from left to right or vertically from top to bottom (e.g., some vertically arranged Braille labels). To this end, the system calculates the centroid coordinates of each touch cluster, i.e., the average x and y coordinates of all touches within the cluster. Then, it sequentially calculates the displacement vector between the centroids of two adjacent character clusters. If most displacement vectors are concentrated in the horizontal direction (e.g., the positive x-axis), it is determined to be a horizontal reading order; if concentrated in the vertical direction (e.g., the positive y-axis), it is determined to be a vertical reading order. In this embodiment, the user touches from left to right, and the system detects continuous rightward displacement vectors, thus determining it as horizontal reading from left to right. Therefore, the system concatenates the standard dot layouts of each character according to the order of touch, forming a complete standardized Braille dot sequence, for example: Char1=[1,0,0,1,0,0], Char2=[1,1,0,0,0,0], Char3=[0,0,1,0,1,1]..., corresponding to a single sentence.

[0025] After obtaining the standardized braille dot sequence, the system enters the decoding and language recognition stage. A plurality of standard braille encoding databases are pre-stored in the glove memory, including the current braille encoding table specified by Chinese national standards, English Braille (grade 1 and grade 2) encoding tables, and Japanese braille encoding table. Each database is a mapping table that maps a 6-bit binary array to a corresponding character or syllable. Since the system does not know whether the braille identifier currently touched by the user is in Chinese, English or Japanese, it is necessary to automatically recognize the language type. The algorithm adopts a statistics-based language recognition method: the first three characters of the sequence are taken and decoded respectively with the Chinese braille database, English braille database and Japanese braille database to obtain three groups of decoding results. For Chinese braille, each braille cell is usually decoded into an initial consonant, a final (or a complete syllable). The system checks whether the decoding result conforms to the initial-final combination rules of Chinese Pinyin, for example, an initial consonant must be followed by a final, two consecutive initial consonants cannot occur, etc., and calculates the rationality score. For English braille, the system checks whether the decoding result can form common English word fragments, such as "th", "ing", etc., and calculates the rationality score of word fragments. For Japanese braille, it checks whether it conforms to the composition rules of the Japanese syllabary. Finally, the system selects the language type with the highest score as the language of the current braille identifier. In this embodiment, the first character is decoded to obtain "b", the second character is decoded to obtain "e", and the third character is decoded to obtain "i", which can be combined into a part of the English word "bei", while the initial consonants "b", "e", "i" obtained after Chinese braille decoding cannot form a legal Pinyin (because "e" and "i" cannot be consecutive), so the English score is higher, and the system determines it as English braille. Then, the system calls the English Braille database to perform complete decoding on the entire sequence, and obtains the initial decoded text: "Bus stop No.3".

[0026] However, in actual scenarios, braille identifiers may have dot missing caused by spelling errors or wear, or false triggering caused by uneven force when the user touches, which makes the initial decoded text contain errors. In this embodiment, it is assumed that due to slight wear of the dots of the second character on the braille identifier, the 4th raised dot is missing from the originally intended dot position [1,1,0,1,0,0] (corresponding to the English letter "o"), so the encoding detected by the sensor becomes [1,1,0,0,0,0], which corresponds to the letter "c". Therefore, the initial decoded text becomes "Bcs stop No.3", where "Bcs" is obviously not a correct word. At this time, the system activates the hybrid error correction strategy.

[0027] The hybrid error correction strategy first executes a rule-matching error correction layer. The rule base pre-stores a large number of common Braille error patterns, derived from extensive statistical analysis of actual Braille labeling errors. For example, the dot-missing rule: if the dot pattern of the current character has no corresponding entry in the standard library, or the corresponding entry is unreasonable in the context, but changing one of its digits from 0 to 1 yields a valid character, then correction is triggered. In this embodiment, the system detects that the character "s" after "c" is valid, but "Bcs" as a whole does not conform to English spelling conventions. There is a "neighboring dot confusion rule" in the rule base: due to wear or tactile deviation, the dot (dot 4) at the top right column is easily lost, causing the original letter "o" (dots 1,1,0,1,0,0) to become confused with "c" (dots 1,1,0,0,0,0). The system checks the context of "c" and finds that the preceding character "B" and the following character "s" should have a vowel letter between them to form the word "Bus". According to the rule "If the character 'c' appears after 'B' and before 's', it is recommended to replace it with 'o'", the rule matching layer quickly outputs the corrected text "Bos stopNo.3". Although "Bos" is still not a standard word, the rule layer marks it as suspicious.

[0028] Since the rule layer fails to fully correct the semantics ("Bos" is still incorrect), the system further invokes a deep learning semantic error correction model. The model is a pre-trained sequence-to-sequence (Seq2Seq) neural network, specifically a lightweight model based on the Transformer architecture, which has undergone int8 quantization through the TensorFlow Lite for Microcontrollers framework and is deployed in the microcontroller of the glove. The model is trained on a large amount of parallel corpora consisting of noisy Braille sequences and correct text, where the noise includes random dot addition, deletion and substitution, as well as errors simulating finger rotation and uneven force. In the inference stage, the model receives the text sequence preliminarily corrected by the rule layer (or directly receives the original dot sequence plus context), understands the semantic information at the entire sentence level through an encoder-decoder structure, and then predicts the most likely correct sentence word by word. In this embodiment, the input to the model is "Bos stop No.3". The encoder first converts each word into an embedding vector and adopts a multi-head self-attention mechanism to capture the dependencies between words. For example, there is a subject-predicate relationship between "Bos" and "stop". Based on the frequently occurring phrase "Bus stop" in the training corpus, the model assigns a far higher probability to "Bus" than to "Bos". When generating the output, the decoder combines the language model probability and finally outputs "Bus stop No.3". The model also outputs a confidence score (e.g., 0.96). The system compares the result of the rule matching layer (with low confidence) with the result of the deep learning model (with high confidence), and selects the one with higher confidence as the final corrected correct text sequence. In this example, the final output is "Bus stop No.3".

[0029] After obtaining the corrected source language text (English), the system performs a multilingual translation step. The user's preset target language is Chinese. An embedded neural machine translation (NMT) module is integrated inside the glove. In this embodiment, a small NMT model after pruning and optimization is adopted. The model is a recurrent neural network based on the attention mechanism, which is pre-trained on large-scale Chinese-English parallel corpora, and then reduces the number of parameters through model pruning and weight quantization, enabling it to run in real time on a microcontroller. The translation module takes the English sentence "Bus stop No.3" as input, translates word by word and adjusts the word order, and outputs the Chinese text: "3号公交车站". Since the number of a bus stop is usually placed before the noun in Chinese, the model can correctly output a phrase that conforms to Chinese usage habits.

[0030] Finally, the system calls the embedded text-to-speech (TTS) engine to convert the translated Chinese text into a speech signal. The TTS engine has a built-in lightweight Chinese speech synthesizer that supports fluent natural language reading. The user's glove is equipped with a miniature speaker and also integrates a Bluetooth Low Energy 5.2 module. The system first detects whether there is a paired Bluetooth headset. If the user is wearing a Bluetooth headset, the audio stream is automatically transmitted to the headset via Bluetooth for private listening; otherwise, it is played aloud through the built-in speaker. In this embodiment, the user is wearing a Bluetooth headset, and the system plays the synthesized speech "Bus Stop No. 3" in a clear and natural voice. The user can accurately understand the information on the bus stop sign without seeing or learning Braille.

[0031] Example 2: This example is essentially the same as Example 1, except for the selection and deployment of the flexible tactile sensor array. To reduce hardware costs or cater to different user preferences, flexible piezoelectric sensors or capacitive touch sensors can be used instead of piezoresistive sensors. Piezoelectric sensors utilize the positive piezoelectric effect of piezoelectric materials to directly generate charge signals when deformed, requiring no external excitation power supply and exhibiting extremely fast response speeds (microseconds). However, piezoelectric signals decay over time, making them unsuitable for detecting static pressure; therefore, high-speed sampling and peak hold circuits are required. Capacitive touch sensors utilize the principle that finger pressure changes the dielectric constant or distance between electrodes, offering advantages such as high sensitivity and ease of integration, but are susceptible to environmental humidity. Depending on the application scenario (e.g., humid outdoor or dry indoor environments), users or manufacturers can choose the most suitable sensor type. Furthermore, the sensor array layout is not limited to a single character coverage area of ​​three rows and two columns; it can be expanded to a higher density array (e.g., eight rows and eight columns) to support the simultaneous recognition of multiple characters or the identification of more refined touch gestures. However, regardless of the sensor type used, the core algorithms for signal preprocessing, orientation recognition, error correction, and translation remain unchanged.

[0032] Example 3: This example relates to an alternative wearable device form. Not all users are accustomed to wearing full-finger gloves; for example, in hot summers or when handling delicate objects, full-finger gloves may be inconvenient. Therefore, the wearable device can be simplified to two separate finger sleeves covering only the index and middle fingers, secured at the base of the fingers with elastic straps. The base of the finger sleeves is made of breathable mesh material, with a flexible sensor array integrated on the inside. The two finger sleeves are connected to a main control box on the back of the hand or wrist via a thin, flexible cable. This control box contains a processor, battery, and communication module. This finger sleeve form can still achieve all the functions described in Example 1, but is lighter, more breathable, and suitable for long-term continuous use. In addition, the sensor integration location is not limited to the index and middle fingers; depending on user habits, it can also be integrated on a single index finger, allowing information about each character to be acquired sequentially through multiple touches with a single finger. In this case, the "continuous touch cluster" in the direction recognition algorithm no longer originates from multiple fingers pressing simultaneously, but from the same finger pressing sequentially at different times. The system can still reconstruct character sequences through temporal clustering and moving vector analysis, but the processing speed will be slightly reduced.

[0033] Example 4: This example further illustrates alternative implementations of the rotation angle and matching algorithm in the orientation recognition algorithm. In the previous examples, the preset rotation angles included four discrete directions: 0°, 90°, 180°, and 270°, which is sufficient to cover the natural wrist rotation range of most users. However, to accommodate extreme grip postures of some users (e.g., tilted at 45°), the system can also refine the rotation angle into 15° increments for more precise matching. Besides the least squares method, the matching algorithm can also use Hausdorff distance, which calculates the maximum bidirectional nearest distance between two point sets, is less sensitive to outliers, and has stronger robustness. In the actual code implementation, the system pre-calculates the convex hull of the standard template point set, then extracts features from the measured point set (such as centroid and principal axis direction angle), performs coarse classification first, and then performs fine matching to reduce computational load.

[0034] Example 5: This example details the construction of the rule base in the error correction module and the specific training and deployment process of the deep learning model. The rule base is derived from the statistics of a large number of Braille error samples. For example, the missing dot rule base contains the error pattern and its corresponding correct pattern for each character if a certain dot is missing. The rule base is stored in the form of a decision tree or lookup table to achieve fast matching with O(1) time complexity. For example, when the system decodes two consecutive identical characters "aa" in the Braille sequence, according to the rules of English Braille, unless it is a proper noun abbreviation, consecutive identical letters are very rare. Therefore, the rule base will check whether there is a possibility of dot substitution, such as "an" which should have been "an" but became "am" because dot 5 was misjudged as dot 6, etc.

[0035] The deep learning model employs a bidirectional LSTM Seq2Seq architecture, with each encoder and decoder consisting of two layers, each with 128 hidden units and approximately 150,000 parameters. Model training utilized 500,000 pairs of "noisy Braille sequences - correct text," with noise injection accounting for 20%. After training, the model underwent 8-bit quantization using TensorFlow Lite, converting the weights from 32-bit floating-point numbers to 8-bit integers, compressing the model size from approximately 2MB to approximately 500KB. The inference time on a Cortex-M7 is approximately 80 milliseconds. The model is embedded into the system's firmware and invoked directly at runtime.

[0036] Example 6: This example describes an alternative implementation of the multilingual translation module. On resource-constrained embedded devices, if the user only requires specific language pairs (e.g., Chinese-English translation), a phrase-based statistical machine translation model can be used. This model has a very small memory footprint (<200KB) and does not require a neural network accelerator. The statistical model segments the input source language sentence into phrases by maintaining a phrase translation table and a reordering model, translates each phrase separately, and then recombines them. Although the fluency may not be as good as the neural model, it is sufficient for short texts such as bus stop signs and road signs. Another alternative is to use a cloud translation interface: when the glove is connected to a smartphone via Bluetooth, the source language text can be sent to a cloud translation service (such as the Google Translate API) using the phone's network, and the result can then be retrieved for speech synthesis. This approach can support hundreds of languages, but it relies on a network connection and has higher power consumption.

[0037] In summary, this invention utilizes a flexible tactile sensor array integrated into the contact surface of wearable devices to collect tactile signals from Braille markings through natural touch. Users can acquire information without needing to learn any Braille knowledge, significantly lowering the barrier to entry. Furthermore, by performing spatiotemporal clustering and multi-angle rotational spatial matching on the original tactile signals, combined with the analysis of the movement vector of the center point of continuous touch clusters, it can automatically adapt to different touch directions and reading sequences, eliminating ambiguity caused by finger rotation or tilting, and outputting a standardized Braille dot sequence. Moreover, by calling multiple standard Braille encoding databases and automatically identifying the language type of the current Braille based on the semantic rationality of the decoding results, this solution is compatible with multiple Braille standards, including Chinese, English, and Japanese, without requiring users to pre-set the language. For the numerous non-standard or erroneous Braille markings present in real-world scenarios, this solution employs a hybrid error correction strategy combining rule matching with sequence-to-sequence or Transformer models. This strategy can quickly handle common errors such as missing, added, and replaced dots, and can also correct complex or unknown error combinations at the semantic level, thereby converting previously undecipherable erroneous Braille into semantically correct text. Finally, the corrected text is translated into the user's preset target language and output via voice, achieving barrier-free information conversion from tactile to auditory. Overall, this solution integrates Braille orientation adaptive recognition, multi-standard decoding, semantic error correction, and multilingual translation functions while maintaining the portability of wearable devices, effectively improving the accuracy, robustness, and convenience for visually impaired people to obtain Braille information in different scenarios.

[0038] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a specific posture. If the specific posture changes, the directional indication will also change accordingly.

[0039] Furthermore, in this invention, descriptions involving terms such as "first," "second," and "a" are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0040] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0041] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

Claims

1. A method for Braille recognition, error correction, and multilingual translation based on tactile perception, characterized in that, Including the following steps: S1: By integrating a flexible tactile sensor array into the contact surface of the wearable device, the raw tactile signal generated when the user touches the Braille sign includes the coordinates of the touch point, the touch timestamp, and the pressure information; S2: Perform spatiotemporal clustering processing on the original tactile signal, group multiple touch events that occur within a preset time window into a touch cluster of a Braille character, and perform spatial matching of multiple preset rotation angles on each touch cluster to determine the standard point layout corresponding to the touch cluster. S3: By analyzing the center point movement vectors of multiple consecutive touch point clusters, the global reading direction is determined, and combined with the standardized point layout of each touch point cluster, a standardized Braille point sequence arranged in the correct reading order is output. S4: Call multiple standard Braille coding databases, decode the standardized Braille dot sequence through Braille coding rules of different languages, and identify the language type of the current Braille mark and the initial decoded text based on the semantic rationality of the decoding result; S5: First, perform rule matching and error correction on the initial decoded text, then use a sequence-to-sequence model or Transformer model for semantic correction, and output the corrected text sequence. S6: Take the corrected text sequence as the source language text, translate it into the user-preset target language text, and convert it into a speech signal to output to the user; In step S2, the preset rotation angles include 0°, 90°, 180°, and 270°; the spatial matching uses Hausdorff distance or least squares method to calculate the matching degree between the touch cluster and the standard Braille dot template.

2. The Braille recognition, error correction, and multilingual translation method based on tactile perception as described in claim 1, characterized in that, In step S1, the flexible tactile sensor array is composed of one or more of the following: flexible piezoresistive sensor, flexible piezoelectric sensor, and flexible capacitive touch sensor.

3. The Braille recognition, error correction, and multilingual translation method based on tactile perception as described in claim 1, characterized in that, In step S1, the wearable device is a fingerless glove, and the flexible tactile sensor array is integrated into the contact surface on the inside of the fingertip of the index finger and / or middle finger of the fingerless glove.

4. The Braille recognition, error correction, and multilingual translation method based on tactile perception as described in claim 1, characterized in that, In step S3, the global reading direction includes a horizontal reading direction and a vertical reading direction; The calculation of the center point movement vector is based on the centroid coordinate difference of the continuous contact point cluster.

5. The method for Braille recognition, error correction, and multilingual translation based on tactile perception as described in claim 1, characterized in that, In step S4, identifying the language type of the current Braille sign based on the semantic rationality of the decoding result specifically includes: The first few characters of the standardized Braille dot sequence are decoded using Braille coding rules for different language types. The reasonableness score of the decoding result in the word fragment or syllable combination rule in the corresponding language type is calculated, and the language type with the highest score is selected as the recognition result.

6. The Braille recognition, error correction, and multilingual translation method based on tactile perception as described in claim 1, characterized in that, In step S5, the rule base used for rule matching and error correction includes point missing rules, point addition rules, and point replacement rules; the point replacement rules include at least adjacent point confusion rules.

7. The Braille recognition, error correction, and multilingual translation method based on tactile perception as described in claim 1, characterized in that, In step S5, the sequence-to-sequence model or Transformer model is a lightweight model pre-trained on parallel corpora of noisy Braille sequences and correct text, and then deployed in an embedded device after quantization.

8. The Braille recognition, error correction, and multilingual translation method based on tactile perception as described in claim 1, characterized in that, In step S6, the corrected text sequence is translated into the user-preset target language text, which is achieved using an embedded statistical machine translation model or a pruned and optimized neural machine translation model.

Images