A voice-interactive teaching system integrating visual-tactile sensors and Braille recognition algorithms

Abstract

The present invention discloses a voice-interactive teaching system that integrates a visual-tactile sensor and a Braille recognition algorithm. The system comprises: obtaining a Braille image through a tactile sensing module, obtaining an input instruction through a voice module, and adjusting a mode of a Braille translation module, wherein the adjusted modes include a teaching mode and a translation mode; in the teaching mode, the Braille image is preprocessed by the Braille translation module to obtain dot information, the dot information is recognized by the Braille recognition algorithm to obtain character information, and the character information is broadcasted through the voice module; in the process of recognizing single character information, the use of the Braille recognition algorithm also includes determining whether to use the Braille recognition algorithm for detection through a frame difference algorithm and an inter-frame minimum center distance algorithm; in the translation mode, the character information is detected through four columns of threshold lines according to the dot information to obtain the detected character information, and the detected character information is broadcasted through the voice module.

Description

Technical Field

[0001] The present invention belongs to the field of assistive technology and artificial intelligence technology, and in particular relates to a voice-interactive teaching system integrating visual-tactile sensors and a Braille recognition algorithm. Background Art

[0002] While a variety of teaching tools and systems exist in the field of Braille teaching and translation, they still face several major technical challenges and application scenario limitations: Insufficient interactive teaching: Traditional Braille teaching tools often rely on physical Braille books or non-interactive electronic devices. These methods lack dynamic interaction and cannot provide personalized teaching based on learners' specific needs, which affects learning efficiency and motivation. Accuracy and speed of Braille translation: Existing Braille translation tools still need to improve in terms of accuracy and processing speed, especially in the case of misidentification and duplicate recognition. In practical applications, these issues can prevent visually impaired users from effectively accessing information, affecting their learning and daily lives. Previous Braille recognition systems mostly relied on relatively simple algorithms or limited pattern matching techniques. While these technologies can perform basic text recognition tasks, they have significant shortcomings in processing complex text, multilingual Braille, or understanding context. These traditional methods often have high requirements for data quality and format. If the input data contains noise or is not formatted in a standard way, their recognition accuracy and efficiency will be significantly reduced. Summary of the Invention

[0003] To solve the above technical problems, the present invention proposes a voice-interactive teaching system that integrates visual-tactile sensors and Braille recognition algorithms to improve the accuracy and efficiency of Braille recognition and reduce the repetition rate, thereby solving the problems existing in the above-mentioned prior art.

[0004] To achieve the above objectives, the present invention provides a voice-interactive teaching system that integrates visual-tactile sensors and a Braille recognition algorithm, comprising:

[0005] A voice-interactive teaching system integrating visual-tactile sensors and Braille recognition algorithms, comprising: a tactile sensing module and a Braille translation module connected in sequence, wherein the Braille translation module is further connected to a voice module;

[0006] The braille image is obtained through the tactile sensing module, the input instruction is obtained through the voice module, and the braille translation module is adjusted according to the input instruction. The adjusted mode includes a teaching mode and a translation mode.

[0007] In the teaching mode, the Braille image is preprocessed by the Braille translation module to obtain dot information, the dot information is recognized by the Braille recognition algorithm to obtain character information, and the character information is broadcasted by the voice module. In the process of recognizing single character information, the use of the Braille recognition algorithm also includes using the frame difference algorithm and the minimum center distance algorithm between frames to determine whether to use the Braille recognition algorithm for detection;

[0008] In the translation mode, the Braille image is preprocessed by the Braille translation module to obtain dot information, the dot information is recognized by the Braille recognition algorithm to obtain character information, the character information is detected by four columns of threshold lines based on the dot information to obtain the detected character information, and the detected character information is broadcasted by the voice module.

[0009] Optionally, the tactile sensing module adopts a visual tactile sensor.

[0010] Optionally, the process of recognizing dot information using a Braille recognition algorithm includes:

[0011] Grouping the dots in the dot information, wherein the dots are sorted by their x-axis coordinates in a coordinate system based on the horizontal and vertical directions of the image, and the dots whose x-axis coordinate difference is less than or equal to a preset threshold are traversed from left to right in the sorted result, and are grouped into one group, thereby obtaining dots in different groups;

[0012] For the dots in the same group, determine whether they belong to the same column or two columns, and obtain the column position of the dots. For the dots in each column, use the lowest value on the y-axis as the anchor point. Based on the difference between the dots in each column and the lowest value, obtain the row position of the dots. Based on the row and column positions, obtain the Braille matrix.

[0013] The Braille matrix is ​​queried through a Braille dictionary to obtain a single character corresponding to the Braille matrix, and character information is obtained according to the single character, wherein the character information is a single character or a character string.

[0014] Optionally, the process of judging by the frame difference algorithm and the inter-frame minimum circle center distance algorithm includes:

[0015] The Braille image is judged by a frame difference algorithm, wherein the absolute value of the pixel difference at the same position in the Braille image between adjacent frames is calculated, the average of all the absolute values ​​of the pixel differences is calculated to obtain an average difference, and a threshold judgment is performed on the average difference;

[0016] The dot information is judged by using an inter-frame minimum center distance algorithm, wherein between adjacent frames, the minimum distance between the centers of different dots in the Braille images of adjacent frames is calculated, and a threshold judgment is performed on the minimum matrix;

[0017] When the average difference is less than the corresponding threshold and the minimum distance is greater than the corresponding threshold, the Braille image of the current frame is detected using the Braille recognition algorithm.

[0018] Optionally, the process of detecting character information using four columns of threshold lines includes:

[0019] Before outputting a single character, dot information is judged using four columns of threshold lines, wherein the four columns of threshold lines include a first threshold line, a second threshold line, and a third threshold line (i.e., a fourth threshold line), wherein the spacing between the first threshold line and the second threshold line and between the third threshold line (i.e., the fourth threshold line) is one dot distance; and the spacing between the first threshold line and the third threshold line is one square distance;

[0020] When there are two columns of dots corresponding to the character information, there are dots on both the first threshold line and the second threshold line; when there is only one column of dots corresponding to the character information, there are dots on both the first threshold line and the fourth threshold line and there is no dot on the third threshold line, then a single character is valid.

[0021] Optionally, after detecting the character information using the four columns of threshold lines, the following steps may also be performed:

[0022] The function of outputting a single character is locked through a flag. When the output character recognition is successful and valid, the flag is 0 and no more output is performed. When there is a dot on the third threshold line, the flag is set to 1 and the single character is continuously prepared for output.

[0023] Optionally, it further includes a language processing module, which is connected to the Braille translation module and the speech module;

[0024] The language processing module uses a large language model to identify and interpret character information, and the identified and interpreted content is output and broadcast through the voice module.

[0025] Optionally, in translation mode, the language processing module uses a large language model to correct and amend the detected character information, and obtains standard text that is output and broadcasted through the voice module.

[0026] Compared with the prior art, the present invention has the following advantages and technical effects:

[0027] High-precision and real-time Braille recognition: Utilizing high-resolution visual sensors and sophisticated image processing techniques (including Canny edge detection and Hough circle transform), the system is able to accurately and quickly capture and recognize Braille characters. These technologies improve the recognition accuracy and efficiency of Braille dots, reduce the possibility of misrecognition, and thus ensure the accuracy and real-time nature of translation. For the recognition of single Braille letters in teaching mode, the present invention uses a frame difference algorithm and an inter-frame minimum center distance algorithm to prevent repeated recognition and jitter problems for single letters. For the sliding recognition process in translation mode, the present invention uses four columns of threshold lines, c1, c2, c3, and c4, to manage the output. c1-c2 and c3-c4 are separated by a dot distance, and c1-c3 is separated by a square distance. By limiting the four columns of thresholds, the problems of misrecognition and repeated recognition are effectively solved.

[0028] Dynamic, personalized learning experience: An integrated large-scale language model enables the system to dynamically adjust teaching content based on user input and feedback. In teaching mode, the model not only interprets the words recognized by the user but also answers related questions, providing a highly personalized learning experience and instant feedback.

[0029] Instant translation and enhanced reading support: In translation mode, the system uses a large language model to instantly translate Braille into standard text and output it through voice, effectively solving the problem of ambiguity or errors in recognized text and greatly supporting the reading and comprehension needs of visually impaired users.

[0030] Natural Voice Interaction and Accessible User Interface Design: Leveraging advanced speech-to-text and text-to-speech technologies, the system provides users with natural and accurate voice feedback. The user interface is specifically designed for visually impaired users, supporting tactile feedback and voice prompts to ensure they can easily learn and operate the system. This design reduces the user learning curve and improves overall usability and acceptance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0032] Figure 1 Schematic diagram of the overall architecture of the teaching system according to an embodiment of the present invention;

[0033] Figure 2 A visual-tactile sensor according to an embodiment of the present invention;

[0034] Figure 3 A schematic diagram of Braille characters according to an embodiment of the present invention;

[0035] Figure 4A schematic diagram of the Braille character mechanism and distance relationship according to an embodiment of the present invention;

[0036] Figure 5 The result of recognition in the teaching mode of the embodiment of the present invention;

[0037] Figure 6 This is the result identified in the translation mode of the embodiment of the present invention;

[0038] Figure 7 Schematic diagram of a confusion matrix according to an embodiment of the present invention;

[0039] Figure 8 This is a schematic diagram of detecting a single character with marks in two columns according to an embodiment of the present invention;

[0040] Figure 9 A schematic diagram of detecting a column of marked single characters according to an embodiment of the present invention;

[0041] Figure 10 This is a schematic diagram of a possible error that may occur when detecting two characters with marks in two columns according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0043] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0044] The purpose of the present invention is to provide a voice-interactive teaching system that integrates visual-tactile sensors and Braille recognition algorithms, including a voice module, a tactile sensing module, a Braille translation module, and a language processing module. The tactile sensing module uses visual-tactile sensors, and through the combination of the above different modules, it supports teaching mode and translation mode.

[0045] Among them, in the teaching mode, it includes English letter teaching and English word teaching. In the English letter teaching, instructions are input through the voice module to trigger the English letter teaching in the teaching mode, and the Braille image is obtained through the tactile sensing module. The Braille image is processed through the Braille translation module. The related processing mainly includes Braille image related preprocessing and Braille recognition in sequence, and the recognized Braille is matched to identify the corresponding characters, and the characters are voice output and broadcast through the voice module.

[0046] In the Braille image preprocessing, edge detection algorithm and Hough circle transform are used to identify dots. During the Braille recognition process, dots are identified by the Braille recognition algorithm. The Braille recognition algorithm includes dot grouping and classification, Braille matrix generation, and character recognition and mapping. Before the Braille recognition algorithm is performed, the frame difference algorithm and the minimum center distance algorithm between frames are used to prevent repeated recognition of single letters and jitter problems.

[0047] During the English word teaching process, similar to the teaching mode, the corresponding characters are identified and formed into a string. The output string, i.e. the output word, is analyzed and processed through a large language model (such as Ali Tongyi Qianwen qwen_max), and relevant words and corresponding explanations are given and broadcast through the voice module. At the same time, question information can be obtained through the voice module, and the question information is recognized and answered through the large language model. The recognition and answer results are output and broadcast through the voice model. At the same time, a stop command is input through the voice module to stop the teaching mode.

[0048] In translation mode, a command is input through the voice module to trigger the translation mode, a Braille image is obtained through the tactile sensing module, and the Braille image is preprocessed by the Braille translation module, wherein the preprocessing includes edge detection and Hough circle transform to identify the corresponding dots, and the dots are recognized by the Braille recognition algorithm. The content of the Braille recognition algorithm is the same as the above content. Unlike the above teaching mode, after the corresponding character information is generated in the Braille recognition algorithm, it is not directly output. Before output, sliding recognition is performed through four columns of threshold lines to detect dots in different groups. When the detected content meets the output requirements, it is output. Otherwise, the character is removed. According to the detection result, the relevant characters are output as an output string, that is, output text, and the output string is broadcast through the voice module. The output text is segmented and corrected through the large language model, and the segmented and corrected text is broadcasted through the voice module. At the same time, relevant commands are input through the voice module to refresh and exit.

[0049] The above technical solution of the present invention is described in detail with reference to the relevant drawings:

[0050] like Figure 1 As shown, the present invention provides a voice-interactive Braille recognition and teaching system that integrates visual-tactile sensors and a large language model, including a voice module, a tactile sensing module, a Braille translation module, and a language processing module; the tactile sensing module uses a visual-tactile sensor; the steps corresponding to the key components include:

[0051] 1. Visual and tactile sensor deployment:

[0052] Use Figure 2The high-resolution visual-tactile sensor shown captures the physical features of Braille text using a high-resolution camera with specific camera parameter settings, such as focus and exposure, to optimize the image capture for clarity of Braille characters.

[0053] A live video stream is captured by a camera and subsequently processed by the Braille translation module through rotation and cropping to focus on specific Braille recognition areas. Gaussian blur and grayscale conversion are applied to the captured image to reduce noise and optimize subsequent edge detection. The Canny edge detection algorithm, combined with the Hough circle transform, is used to identify and locate Braille dots (raised dots) in the image. Morphological operations, such as opening operations, further clarify edges to improve the accuracy of dot information detection.

[0054] 2. Braille recognition algorithm deployed in the Braille translation module:

[0055] Dot grouping and classification: The detected dots are grouped according to the x-coordinate of the circle center. Each group of dots represents a potential candidate for a Braille character. The grouping is based on a preset horizontal distance threshold to distinguish dot groups of different characters. The mechanism of Braille characters and the distance relationship are as follows: Figure 3-4 shown.

[0056] Braille matrix generation: Each group of dots is sorted by its vertical position in the image and mapped to a 2x3 Braille grid. The position of each dot is identified and confirmed using set vertical and horizontal thresholds to generate the corresponding Braille matrix.

[0057] Character Recognition and Mapping: The generated Braille matrix is ​​matched against a predefined Braille character library, which contains matrix patterns corresponding to each character. Once a matching matrix pattern is found, the character corresponding to the matrix pattern is added to the output string as the recognized character.

[0058] Based on the above-mentioned Braille recognition algorithm, since different recognition modes face different problems, the present invention provides corresponding processing methods or processing logic for character processing in the teaching mode and translation mode:

[0059] For single Braille letter or character recognition in the teaching mode, the present invention adopts a frame difference algorithm and an inter-frame minimum center distance algorithm to prevent repeated recognition problems and jitter problems for single letters.

[0060] For the sliding recognition process of the translation mode, the present invention uses four columns of threshold lines c1, c2, c3, and c4 to manage the output. The intervals between c1-c2 and c3-c4 are one point distance, and the interval between c1-c3 is one square distance. Through the restriction of the four columns of thresholds, the problems of misrecognition and repeated recognition are effectively solved.

[0061] 3. Integration of large language models deployed in language processing modules:

[0062] Integrate large pre-trained language models (such as Alitongyiqianwenqwen_max) to handle complex natural language processing tasks, including context understanding and text generation, and be able to interact with users to answer user questions.

[0063] In teaching mode, the model dynamically adjusts the teaching content based on user input and feedback, interprets the words recognized by the user, and can answer the user's questions, providing personalized feedback and guidance.

[0064] In translation mode, the system uses this model to instantly translate Braille into standard text and output it through voice to support reading and comprehension for visually impaired users, effectively solving the problem of blurred or erroneous text after recognition.

[0065] 4. Voice module, i.e. voice interaction system:

[0066] Speech recognition and synthesis functions are designed in this module. The speech recognition part uses advanced speech-to-text models, such as Google Speech Recognition API, to provide users with an effective voice input port, while the speech output uses mature TTS technology, such as Google Text-to-Speech, to provide users with natural and accurate voice feedback.

[0067] 5. User interface and interaction design:

[0068] Develop easy-to-use user interfaces, taking into account the operational needs of visually impaired users.

[0069] The interface design supports tactile feedback and voice prompts, making it easy for users to learn and use the system.

[0070] Through these technical means, the present invention effectively addresses the issues of insufficient interactivity, accuracy, and real-time performance in Braille learning and translation, while also optimizing the interactive experience for visually impaired users. Based on advanced sensing technology and artificial intelligence algorithms, the system achieves efficient Braille recognition and interactive teaching, significantly improving learning efficiency and quality of life for visually impaired users.

[0071] The specific method of using the system of the present invention includes: first, the user needs to wear the wearable visual and tactile sensor, and then can select the teaching mode or the translation mode through voice interaction.

[0072] Saying the "teaching" command can select the teaching mode. In this mode, the user can use the visual-tactile sensor to identify words on the Braille board (white board). After identifying the word they want to learn, the user can say the "explain" command to hand over the recognized word text to the large language model for processing and explanation. After the processing is completed, the system will output the processed word and its corresponding explanation through voice. At this time, if the user wants to ask it a question, such as "What is the biggest animals in the world?" The large model will also give a corresponding answer and output it by voice. If the user wants to exit the question mode, he needs to say the "quit" command.

[0073] When the user wants to use the translation function, the user needs to say the "translation" command to switch to the translation mode, and then use the "start" command to start. At this time, the user can use the sensor to slide on the Braille board for recognition. After the text is recognized, the "correction" command is used to let the large model break and correct the recognized text, and finally output the translated result in voice. During the entire interactive process, the user can use the refresh command to refresh the page. Finally, the user can use the stop command to exit the program. The recognition results in teaching mode are as follows: Figure 5 As shown, the results recognized in translation mode are as follows Figure 6 shown.

[0074] About the specific algorithm in the Braille translation module:

[0075] Camera Setup and Image Capture: The camera in the visual-tactile sensor is set up to capture a video stream at regular intervals, and the driver mode is specified via cv2.CAP_DSHOW to ensure device compatibility and performance. The video stream is acquired via cv2.VideoCapture, and the camera settings are adjusted for optimal image quality. The live video stream is rotated and cropped to retain only the area containing the Braille text, optimizing processing speed and reducing unnecessary background noise.

[0076] Image preprocessing and Braille dot detection: Gaussian blur is first applied to the image captured by the camera. This step effectively reduces image noise, smoothes image texture, and creates image conditions that are more conducive to analysis for subsequent edge detection. Next, the image is converted to grayscale format (cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY)) because grayscale image processing is faster and more effective for edge detection.

[0077] The Canny edge detection algorithm (cv2.Canny(binary,50,150)) is used. This algorithm accurately depicts edges in the image and is a key preprocessing step for identifying circular Braille dots. After edge detection, the Hough circle transform (cv2.HoughCircles(edges,cv2.HOUGH_GRADIENT,dp=0.9,minDist=40,param1=120,param2=12,minRadius=11,maxRadius=29)) is applied. This algorithm is specifically designed to identify and locate circular objects from the edge detection results. By precisely setting the circle size and spacing parameters, the algorithm can efficiently identify the dots representing Braille characters.

[0078] Specific content of the Braille recognition algorithm:

[0079] Braille dot grouping and matrix generation: Each Braille character, or letter, is represented by a three-row, two-column matrix of circle-centered dots. Since the distance between each two dots, the distance between each two characters, is fixed. Therefore, they are grouped during processing. The principle of grouping is: first, all circles are sorted according to their x-coordinates, ensuring that the processing order is from left to right. By traversing the sorted circle list, the circles are grouped based on whether the x-coordinate difference is less than or equal to a threshold. This threshold determines whether the circles are close enough to be considered part of the same group. Each new group is assigned a number, which helps subsequent processing identify which groups represent a complete Braille character.

[0080] After that, the grouped circles are converted into a matrix representing Braille dots. Braille characters are usually composed of a 3x2 dot matrix. Only the first group on the left is processed each time. First, the column is determined. According to the x-value range of the circles in the group, it is determined whether these circles belong to the same column or need to be divided into two columns. If the circles are close enough (the x-value difference is less than 15), they are considered to belong to the same column. During the word teaching or translation process, due to the real-time update of images or video frames, each picture only processes the first group on the left. When traversing all pictures, due to the continuous update of the image or sliding recognition in this process, all Braille characters can be processed.

[0081] Otherwise, divide the circles into two columns using the midpoint of their x values. For each column, use the lowest y value in the column as the anchor point. Determine the row position of each circle in the matrix based on the difference between its y coordinate and the anchor point. Update the matrix based on the circle's position in the column, marking the corresponding position with 1.

[0082] Finally, the processed matrix representing the Braille dots is returned, and the corresponding letters are found in the Braille dictionary. Figure 7 As shown in the confusion matrix in , the average recognition accuracy of the present invention for a single letter exceeds 95%.

[0083] For character recognition and output in teaching mode:

[0084] For single Braille letter recognition in the teaching mode, the present invention adopts a frame difference algorithm and an inter-frame minimum center distance algorithm to prevent repeated recognition and jitter problems of single letters.

[0085] Frame Difference Algorithm: Because any motion or the appearance of a new object typically results in significant changes in pixel values, the present invention calculates the pixel-by-pixel absolute difference between the two images, i.e., the absolute value of the difference in pixel intensity at the same location in the two images, frame_diff. By calculating the average value of the difference image frame_diff, a numerical value, diff_mean, is obtained, which represents the average degree of change across the entire image frame. The diff_mean can be used as a quantitative indicator to determine whether the change between the two frames is significant. The present invention compares the diff_mean with a set threshold (for example, FRAME_DIFF_THRESHOLD = 0.25). If the difference is less than the threshold, it indicates that the image change is relatively small and tends to be stable. At this time, text recognition is performed, which can effectively avoid repeated recognition.

[0086] In addition, the present invention also uses an inter-frame minimum circle center distance algorithm to prevent jitter: by calculating the minimum distance circle_difference between all detected circle centers between two frames, small changes in position can be accurately quantified. Then, circle_difference is compared with a set threshold (for example, DISTANCE_THRESHOLD = 14). If the change is less than the threshold, it means that it is in a state of slight jitter, and the system will not perform Braille detection on it. By setting a reasonable distance threshold, the system can distinguish between small, meaningless changes and actual meaningful movements. This reduces misidentification caused by background noise or small-scale vibrations and improves the overall accuracy of the system.

[0087] Through the limitations of these two algorithms, the present invention ensures that Braille detection is performed only at the moment when the image becomes stable, and any subsequent slight jitter will not be detected. This allows each Braille letter to be recognized only once, reducing repetition and meeting the requirements of the teaching mode.

[0088] For output detection in translation mode:

[0089] For the sliding recognition process in translation mode, the present invention adopts a four-threshold line method to limit its output. Since the image is updated in real time, in order to ensure the stability of character detection, reduce repeated recognition and improve accuracy, the present invention adopts four threshold lines: the first threshold line, the second threshold line, the third threshold line, and the fourth threshold line (c1, c2, c3, c4) to manage the output. The selection of thresholds is obtained by integrating the rules and distance relationships of Braille characters and continuously debugging. C1-c2 and c3-c4 are separated by a dot distance, and c1-c3 is separated by a square distance. The dot distance represents the distance between the centers of each adjacent dot under a single Braille character (2*3 dots) in the image, and the square distance represents the distance between the center line between two adjacent Braille characters and the next center line. C1 and c2 are mainly in the left area of ​​the image and are mainly responsible for processing the character to be output currently, while c3 and c4 are mainly used to limit the output conditions and detect the next incoming character. Braille has specific standards on the Braille board, and its actual spacing does not change. At the same time, when using the sensor to scan the above-mentioned Braille characters, the sensor is used in conjunction with the Braille and is in a vertical state at all times. The distance and angle between the sensor and the Braille board will only produce a slight error in the distance between the dots in the image. The dot pitch and square pitch are fixed. The above-mentioned error will only have a slight impact on the detection of the dot pitch and square pitch, but will not affect the use of the detection method. Therefore, the above-mentioned detection scheme of the present invention can effectively detect the array dots corresponding to the characters to be output in real time.

[0090] Regarding the detection of the four threshold lines before the output of Braille characters, the following situations are discussed here:

[0091] 1. When outputting characters with circles in both columns, it is necessary to satisfy the requirement that there are circles on both c1 and c2, that is, Figure 8 the situation in

[0092] 2. When outputting only one column of characters with circles, such as a, b, k, l, it is necessary to satisfy the requirement that c1 and c4 have circles, while c2 and c3 do not have circles. Figure 9 As shown;

[0093] 3. For situations where misrecognition may occur, the algorithm of the present invention can also correctly avoid errors. For example, during the sliding recognition process, it is very likely that the second column of characters with circles in both columns will be in c1, and the current character will be mistakenly identified as a single-column character and output.

[0094] like Figure 10 As shown, b is likely to be misidentified at this time, but since c3 and c4 are used to restrict it, according to case 2, we can know that to output a single column of characters, c3 must not have a circle, so the above algorithm can avoid this error.

[0095] In order to further prevent repeated recognition, the present invention uses a flag to lock the output letter function. After successful recognition and output, the flag is set to 0, and no output is performed thereafter until a new circle is detected in the c3 threshold column. Then the flag is set to 1 and the output is activated. This effectively reduces the repetition rate.

[0096] Finally, the generated braille matrix is ​​matched against a predefined library of braille characters. Once a matching braille pattern is found, the recognized character is added to the output string.

[0097] The recognized characters are converted into speech output through the pyttsx3 library.

[0098] Voice interaction in the voice module:

[0099] The system uses the speech_recognition library to configure the microphone input and continuously monitor the surrounding environment. This library first automatically adjusts the microphone sensitivity through the adjust_for_ambient_noise method to adapt to different noise environments and ensure the accuracy of voice command recognition.

[0100] When the user issues voice commands, the listen method captures these commands and converts the audio into text through the recognize_google method. This process utilizes Google's Cloud Speech Recognition service to provide highly accurate speech-to-text conversion.

[0101] The system can parse and respond to a variety of voice commands, including "teaching" (switch to teaching mode), "explain" (explain the recognized word text), "quit" (exit explanation mode), "translation" (switch to translation mode), "start" (start recognizing text in translation mode), "correction" (correct the recognized text), "refresh" (refresh the page, clear the text box), "stop" (exit the program), and user-defined questions in explanation mode.

[0102] For each recognized command, the system uses built-in logic to determine its type and perform corresponding operations, such as updating the interface display or using speech synthesis technology (such as pyttsx3) to feedback the operation results.

[0103] Application of large language models:

[0104] In teaching mode, the system utilizes the Alitong Yiqianwen qwen_max model to receive Braille input from users, interpret its meaning in real time, and provide relevant learning materials. This model uses deep learning algorithms to understand and generate language, allowing teaching content to go beyond static data and dynamically adjust based on user progress and feedback.

[0105] When users ask questions about a word or sentence, the system can instantly generate educational answers. This feature is achieved through the model's natural language understanding capabilities, enhancing interactivity and personalized learning experience.

[0106] In translation mode, the system also uses the qwen_max model to instantly translate Braille text read from the camera into standard text. This conversion is not only fast and accurate, but can also handle complex texts that contain errors, omissions, or repetitions.

[0107] The translated text is converted into speech output through the pyttsx3 library to ensure that visually impaired users can receive information without obstacles.

[0108] The above technical solution of the present invention has the following effects:

[0109] Improving interactivity and personalization in Braille learning: Traditional Braille teaching tools rely on static physical materials or non-interactive electronic devices, lacking personalization and dynamic interaction. This invention, by incorporating high-precision visual-tactile sensors and real-time voice interaction, provides dynamic learning feedback and personalized instruction, enhancing learner engagement and learning efficiency.

[0110] Improving the accuracy and real-time performance of Braille translation: Existing Braille translation tools are prone to misidentification or repeated recognition of the same letter when processing complex texts, and their accuracy and speed often fall short of actual requirements. This invention utilizes advanced Braille recognition algorithms and large-scale model processing to significantly improve translation accuracy and response speed, enabling real-time translation of more complex texts.

[0111] Optimizing the interactive experience of visually impaired users: Taking into account the special needs of visually impaired users in operation and interaction, the present invention combines the advantages of visual-tactile sensors and large language models to design a multimodal interactive interface. Through voice and tactile feedback, it provides an accessible user interface, allowing visually impaired users to interact with the system more naturally, improving the convenience of learning and use.

[0112] Leveraging large models to improve Braille text processing: Traditional Braille translation tools are often limited in accuracy and speed by the underlying performance of their algorithms when processing semantically complex Braille or multilingual text. This invention significantly improves text processing accuracy and efficiency by integrating advanced large-scale language models and leveraging their deep learning capabilities, ensuring highly accurate translation even for complex or ambiguous Braille input.

[0113] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A voice interactive teaching system integrating visual and tactile sensors and Braille recognition algorithms, characterized in that: include: A tactile sensing module and a Braille translation module connected in sequence, wherein the Braille translation module is further connected to a speech module; The braille image is obtained through the tactile sensing module, the input instruction is obtained through the voice module, and the braille translation module is adjusted according to the input instruction. The adjusted mode includes a teaching mode and a translation mode. In the teaching mode, the Braille image is preprocessed by the Braille translation module to obtain dot information, the dot information is recognized by the Braille recognition algorithm to obtain character information, and the character information is broadcasted by the voice module. In the process of recognizing single character information, the use of the Braille recognition algorithm also includes using the frame difference algorithm and the minimum center distance algorithm between frames to determine whether to use the Braille recognition algorithm for detection; In the translation mode, the Braille image is preprocessed by the Braille translation module to obtain dot information, the dot information is recognized by the Braille recognition algorithm to obtain character information, the character information is detected by four columns of threshold lines based on the dot information to obtain detected character information, and the detected character information is broadcasted by the voice module; The process of judging by the frame difference algorithm and the minimum center distance algorithm between frames includes: The Braille image is judged by a frame difference algorithm, wherein the absolute value of the pixel difference at the same position in the Braille image between adjacent frames is calculated, the average of all the absolute values ​​of the pixel differences is calculated to obtain an average difference, and a threshold judgment is performed on the average difference; The dot information is judged by using an inter-frame minimum center distance algorithm, wherein between adjacent frames, the minimum distance between the centers of different dots in the Braille images of adjacent frames is calculated, and a threshold judgment is performed on the minimum matrix; When the average difference is less than the corresponding threshold and the minimum distance is greater than the corresponding threshold, the Braille image of the current frame is detected using the Braille recognition algorithm; The process of detecting character information through four columns of threshold lines includes: Before outputting a single character, dot information is judged using four columns of threshold lines, wherein the four columns of threshold lines include a first threshold line, a second threshold line, and a third threshold line (i.e., a fourth threshold line), wherein the spacing between the first threshold line and the second threshold line and between the third threshold line (i.e., the fourth threshold line) is one dot distance; and the spacing between the first threshold line and the third threshold line is one square distance; When there are two columns of dots corresponding to the character information, there are dots on both the first threshold line and the second threshold line; when there is only one column of dots corresponding to the character information, there are dots on both the first threshold line and the fourth threshold line, and there is no dot on the third threshold line, then the single character is valid; After detecting the character information through the four-column threshold line, it also includes: The function of outputting a single character is locked through a flag. When the output character recognition is successful and valid, the flag is 0 and no more output is performed. When there is a dot on the third threshold line, the flag is set to 1 and the single character is continuously prepared for output.

2. The system according to claim 1, wherein: The tactile sensing module adopts a visual tactile sensor.

3. The system according to claim 1, wherein: The process of identifying dot information using the Braille recognition algorithm includes: Grouping the dots in the dot information, wherein the dots are sorted by their x-axis coordinates in a coordinate system based on the horizontal and vertical directions of the image, and the dots whose x-axis coordinate difference is less than or equal to a preset threshold are traversed from left to right in the sorted result, and are grouped into one group, thereby obtaining dots in different groups; For the dots in the same group, determine whether they belong to the same column or two columns, and obtain the column position of the dots. For the dots in each column, use the lowest value on the y-axis as the anchor point. Based on the difference between the dots in each column and the lowest value, obtain the row position of the dots. Based on the row and column positions, obtain the Braille matrix. The Braille matrix is ​​queried through a Braille dictionary to obtain a single character corresponding to the Braille matrix, and character information is obtained according to the single character, wherein the character information is a single character or a character string.

4. The system according to claim 1, wherein: It also includes a language processing module, which is connected to the Braille translation module and the speech module; The language processing module uses a large language model to identify and interpret character information, and the identified and interpreted content is output and broadcast through the voice module.

5. The system according to claim 4, characterized in that In translation mode, the language processing module uses a large language model to correct and amend the detected character information, and obtains standard text that is output and broadcast through the voice module.

Images