Voice interaction teaching system integrating visual tactile sensor and braille recognition algorithm

Abstract

The invention discloses a voice interaction teaching system integrated with a visual tactile sensor and a braille recognition algorithm, which comprises the steps that a braille image is acquired through a tactile sensing module, an input instruction is acquired through a voice module to carry out mode adjustment on a braille translation module, and the adjustment modes comprise a teaching mode and a translation mode; in the teaching mode, the braille image is preprocessed through the braille translation module to obtain dot information, the dot information is recognized through the braille recognition algorithm to obtain character information, and the character information is broadcasted through the voice module. Whether the braille recognition algorithm is used for detection or not is judged through a frame difference algorithm and an inter-frame minimum circle center distance algorithm when the braille recognition algorithm is used; in the translation mode, the character information is detected through the four columns of threshold lines according to the dot information, the detected character information is obtained, and the detected character information is broadcasted through the voice module.

Description

Technical Field

[0001] The invention belongs to the technical field of assistive technology and artificial intelligence, and in particular relates to a voice interactive teaching system integrating visual-tactile sensors and a Braille recognition algorithm. Background Art

[0002] In the existing field of Braille teaching and translation, although there are many teaching tools and systems, there are still several major technical problems and limitations in application scenarios: Insufficient teaching interactivity: Traditional Braille teaching tools mostly rely on physical Braille books or non-interactive electronic devices. These methods lack dynamic interaction and cannot provide personalized teaching according to the specific needs of learners, affecting learning efficiency and motivation. Accuracy and speed of Braille translation: Existing Braille translation tools still need to be improved in accuracy and processing speed, especially in the case of misidentification and repeated recognition in recognition. In practical applications, these problems may cause visually impaired users to be unable to effectively obtain information, affecting their learning and daily life. In previous Braille recognition systems, most rely on relatively simple algorithms or limited pattern matching technologies. Although these technologies can complete basic text recognition tasks, they have obvious shortcomings in processing complex texts, multilingual Braille or contextual understanding. These traditional methods usually have high requirements for data quality and format. Once the input data contains noise or non-standard format, its recognition accuracy and efficiency will be greatly reduced. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes a voice interactive teaching system that integrates visual-tactile sensors and Braille recognition algorithms to improve the accuracy and efficiency of Braille recognition and reduce the repetition rate, so as to solve the problems existing in the above-mentioned prior art.

[0004] To achieve the above object, the present invention provides a voice interactive teaching system integrating visual tactile sensors and Braille recognition algorithms, comprising:

[0005] A voice interactive teaching system integrating visual tactile sensors and Braille recognition algorithms, comprising: a tactile sensing module and a Braille translation module connected in sequence, wherein the Braille translation module is also connected to a voice module;

[0006] The braille image is obtained through the tactile sensing module, the input command is obtained through the voice module, and the mode of the braille translation module is adjusted through the input command, and the adjusted mode includes a teaching mode and a translation mode;

[0007] In the teaching mode, the Braille image is preprocessed by the Braille translation module to obtain dot information, the dot information is recognized by the Braille recognition algorithm to obtain character information, and the character information is broadcasted by the voice module. In the process of recognizing single character information, the use of the Braille recognition algorithm also includes judging whether to use the Braille recognition algorithm for detection through the frame difference algorithm and the minimum center distance algorithm between frames;

[0008] In the translation mode, the Braille image is preprocessed by the Braille translation module to obtain dot information, the dot information is recognized by the Braille recognition algorithm to obtain character information, the character information is detected by four columns of threshold lines according to the dot information to obtain the detected character information, and the detected character information is broadcasted by the voice module.

[0009] Optionally, the tactile sensing module adopts a visual tactile sensor.

[0010] Optionally, the process of recognizing dot information by using a Braille recognition algorithm includes:

[0011] The dots in the dot information are grouped, wherein the x-axis coordinates of the dots in a coordinate system based on the horizontal and vertical directions of the image are sorted, and the dots whose x-axis coordinate difference is less than or equal to a preset threshold are traversed from left to right in the sorting result, and the dots are grouped into one group to obtain dots in different groups;

[0012] For the dots in the same group, determine whether they belong to the same column or two columns, and obtain the column position of the dots. For the dots in each column, use the lowest value of the y-axis as the anchor point, and obtain the row position of the dots according to the difference between the dots in each column and the lowest value. According to the row position and column position, obtain the Braille matrix;

[0013] The Braille matrix is ​​queried through a Braille dictionary to obtain a single character corresponding to the Braille matrix, and character information is obtained according to the single character, wherein the character information is a single character or a character string.

[0014] Optionally, the process of judging by using the frame difference algorithm and the minimum center distance algorithm between frames includes:

[0015] The Braille image is judged by a frame difference algorithm, wherein the absolute value of the pixel difference at the same position in the Braille image between adjacent frames is calculated, the average of all the absolute values ​​of the pixel differences is calculated to obtain an average difference, and a threshold judgment is performed on the average difference;

[0016] The dot information is judged by using an inter-frame minimum center distance algorithm, wherein between adjacent frames, the minimum distance between the centers of different dots in the Braille images between adjacent frames is calculated, and a threshold judgment is performed on the minimum matrix;

[0017] When the average difference is less than the corresponding threshold and the minimum distance is greater than the corresponding threshold, the Braille image of the current frame is detected using the Braille recognition algorithm.

[0018] Optionally, the process of detecting character information by using four columns of threshold lines includes:

[0019] Before a single character is output, the dot information is judged by four columns of threshold lines, wherein the four columns of threshold lines include a first threshold line, a second threshold line, and a third threshold line, namely, a fourth threshold line, wherein the spacing between the first threshold line and the second threshold line and between the third threshold line, namely, the fourth threshold line, is one dot distance; the spacing between the first threshold line and the third threshold line is one square distance;

[0020] When there are two columns of dots corresponding to the character information, there are dots on both the first threshold line and the second threshold line; when there is only one column of dots corresponding to the character information, there are dots on both the first threshold line and the fourth threshold line and there are no dots on the third threshold line, then a single character is valid.

[0021] Optionally, after detecting the character information through the four-column threshold line, the following steps are further included:

[0022] The function of outputting a single character is locked through a flag. When the output character recognition is successful and valid, the flag is 0 and no output is performed. When there is a dot on the third threshold line, the flag is set to 1 and the single character is continuously prepared for output.

[0023] Optionally, it also includes a language processing module, which is connected to the Braille translation module and the speech module;

[0024] The language processing module uses a large language model to recognize and interpret character information, and the recognized and interpreted content is output and broadcasted through the voice module.

[0025] Optionally, in the translation mode, the language processing module uses a large language model to correct and amend the detected character information, and obtains standard text which is output and broadcasted through the voice module.

[0026] Compared with the prior art, the present invention has the following advantages and technical effects:

[0027] High-precision and real-time Braille recognition: Utilizing high-resolution visual sensors and sophisticated image processing techniques (including Canny edge detection and Hough circle transform), the system is able to accurately and quickly capture and recognize Braille characters. These technologies improve the recognition accuracy and efficiency of Braille dots and reduce the possibility of misrecognition, thereby ensuring the accuracy and real-time nature of translation. For the recognition of single Braille letters in teaching mode, the present invention uses a frame difference algorithm and an inter-frame minimum center distance algorithm to prevent repeated recognition and jitter problems for single letters. For the sliding recognition process in the translation mode, the present invention uses four-column threshold lines of c1, c2, c3, and c4 to manage the output. c1-c2 and c3-c4 are separated by a dot distance, and c1-c3 is separated by a square distance. Through the limitation of the four-column threshold, the misrecognition and repeated recognition problems are effectively solved.

[0028] Dynamic personalized learning experience: The integrated large-scale language model enables the system to dynamically adjust the teaching content based on the user's input and feedback. In teaching mode, the model not only explains the words recognized by the user, but also answers the user's related questions, thus providing a highly personalized learning experience and instant feedback.

[0029] Instant translation and enhanced reading support: In translation mode, the system uses a large language model to instantly translate Braille into standard text and output it through voice, effectively solving the problem of ambiguity or errors in the recognized text, and greatly supporting the reading and comprehension needs of visually impaired users.

[0030] Natural voice interaction and barrier-free user interface design: Through advanced speech-to-text and text-to-speech technologies, the system provides users with natural and accurate voice feedback. The user interface is specially designed for visually impaired users, supporting tactile feedback and voice prompts to ensure that users can easily learn and operate the system. This design reduces the user's learning curve and improves the overall usability and acceptance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0032] Figure 1 It is a schematic diagram of the overall architecture of the teaching system according to an embodiment of the present invention;

[0033] Figure 2 A visual-tactile sensor according to an embodiment of the present invention;

[0034] Figure 3 A schematic diagram of a Braille character according to an embodiment of the present invention;

[0035] Figure 4A schematic diagram of the Braille character mechanism and distance relationship of an embodiment of the present invention;

[0036] Figure 5 The result of recognition in the teaching mode of the embodiment of the present invention;

[0037] Figure 6 The result identified in the translation mode of the embodiment of the present invention;

[0038] Figure 7 A schematic diagram of a confusion matrix according to an embodiment of the present invention;

[0039] Figure 8 A schematic diagram of detecting a single character with marks in both columns according to an embodiment of the present invention;

[0040] Fig. 9 A schematic diagram of detecting a column of marked single characters according to an embodiment of the present invention;

[0041] Fig.10 This is a schematic diagram of a possible error that may occur when detecting two characters marked in two columns according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0043] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0044] The purpose of the present invention is to provide a voice interactive teaching system integrating visual tactile sensors and Braille recognition algorithms, comprising a voice module, a tactile sensing module, a Braille translation module, and a language processing module; the tactile sensing module adopts a visual tactile sensor, and supports a teaching mode and a translation mode through a combination of the above different modules;

[0045] Among them, in the teaching mode, it includes English letter teaching and English word teaching. In the English letter teaching, instructions are input through the voice module to trigger the English letter teaching in the teaching mode, and the Braille image is obtained through the tactile sensing module, and the Braille image is processed through the Braille translation module. The related processing mainly includes Braille image related preprocessing and Braille recognition in sequence, and the recognized Braille is matched to identify the corresponding characters, and the characters are voice output and broadcast through the voice module.

[0046] In the Braille image related preprocessing, edge detection algorithm and Hough circle transform are used to identify dots. In the Braille recognition process, dots are recognized by the Braille recognition algorithm. The Braille recognition algorithm includes dot grouping and classification, Braille matrix generation, and character recognition and mapping. Before the Braille recognition algorithm is performed, the frame difference algorithm and the minimum center distance algorithm between frames are used to prevent repeated recognition of single letters and jitter problems.

[0047] In the process of teaching English words, similar to the teaching mode, the corresponding characters are identified and then formed into a string. The output string, i.e. the output word, is analyzed and processed through a large language model (such as Ali Tongyi Qianwen qwen_max), and related words and corresponding explanations are given and broadcasted through the voice module. At the same time, question information can be obtained through the voice module, and the question information is identified and answered through the large language model. The identified answer results are output and broadcasted through the voice model. At the same time, a stop command is input through the voice module to stop the teaching mode.

[0048] In the translation mode, a command is input through the voice module to trigger the translation mode, a Braille image is obtained through the tactile sensing module, and the Braille image is preprocessed through the Braille translation module, wherein the preprocessing includes edge detection and Hough circle transform to identify the corresponding dots, and the dots are identified through the Braille recognition algorithm. The content of the Braille recognition algorithm is the same as the above content. Different from the above teaching mode, after the corresponding character information is generated in the Braille recognition algorithm, it is not directly output. Before the output, the dots in different groups are detected by sliding recognition through four columns of threshold lines. When the detected content meets the output requirements, it is output, otherwise the character is removed, and the relevant characters are output as output strings, i.e., output texts, according to the detection results, and the output strings are broadcast through the voice module, and the output texts are segmented and corrected through the large language model, and the segmented and corrected texts are broadcasted through the voice module, and the relevant commands are input through the voice module to refresh and exit.

[0049] The above technical solution of the present invention is described in detail with reference to the relevant drawings:

[0050] like Figure 1 As shown, the present invention provides a voice-interactive Braille recognition teaching system integrating visual-tactile sensors and a large language model, including a voice module, a tactile sensing module, a Braille translation module, and a language processing module; the tactile sensing module adopts a visual-tactile sensor; the steps corresponding to the key components include:

[0051] 1. Visual and tactile sensor deployment:

[0052] Use Figure 2The high-resolution visual-tactile sensor shown captures the physical features of Braille text using a high-resolution camera with specific camera parameter settings, such as focus and exposure, to optimize the image capture for clarity of the Braille characters.

[0053] The live video stream is captured by the camera and processed by the subsequent Braille translation module through rotation and cropping to focus on the specific Braille recognition area. Gaussian blur and grayscale conversion are applied to the captured image to reduce noise and optimize the subsequent edge detection effect. The Canny edge detection algorithm is used in combination with the Hough circle transform to identify and locate the Braille dots (raised dots) in the image. Morphological operations (such as opening operations) are used to further clarify the edges to improve the accuracy of dot information detection.

[0054] 2. Braille recognition algorithm deployed in the Braille translation module:

[0055] Dot grouping and classification: The detected dots are grouped according to the x-coordinate of the center of the circle. Each group of dots represents a potential candidate for a Braille character. The grouping is based on a preset horizontal distance threshold to distinguish dot groups of different characters. The mechanism of Braille characters and distance relationships are as follows: Figure 3-4 shown.

[0056] Braille matrix generation: Sort each group of dots according to their vertical position in the image and map them to a 2x3 Braille dot grid. Identify and confirm the position of each dot through the set vertical and horizontal thresholds to generate the corresponding Braille matrix.

[0057] Character recognition and mapping: The generated Braille matrix is ​​matched with a predefined Braille character library, which contains the matrix patterns corresponding to each character. Once a matching matrix pattern is found, the character corresponding to the matrix pattern is added to the output string as the recognized character.

[0058] Based on the above-mentioned Braille recognition algorithm, since the recognition problems in different modes are different, for the letter teaching in the teaching mode and the translation mode, the present invention provides corresponding processing methods or processing logics to perform relevant processing on the characters:

[0059] For single Braille letter or character recognition in the teaching mode, the present invention adopts a frame difference algorithm and an inter-frame minimum circle center distance algorithm to prevent repeated recognition problems and jitter problems for single letters.

[0060] For the sliding recognition process of the translation mode, the present invention adopts four columns of threshold lines c1, c2, c3, and c4 to manage the output. The intervals of c1-c2 and c3-c4 are one point distance, and the interval of c1-c3 is one square distance. Through the limitation of the four columns of thresholds, the problems of misrecognition and repeated recognition are effectively solved.

[0061] 3. Integration of large language models deployed in language processing modules:

[0062] Integrate large pre-trained language models (such as Alitongyi Qianwen qwen_max) to handle complex natural language processing tasks, including context understanding and text generation, and be able to interact with users to answer user questions.

[0063] In teaching mode, the model dynamically adjusts the teaching content based on the user's input and feedback, interprets the words recognized by the user, and can answer the user's questions, providing personalized feedback and guidance.

[0064] In translation mode, the system uses this model to instantly translate Braille into standard text and output it via voice to support reading and comprehension by visually impaired users, effectively resolving situations where the recognized text is blurry or contains errors.

[0065] 4. Voice module, i.e. voice interaction system:

[0066] Speech recognition and synthesis functions are designed in this module. The speech recognition part uses advanced speech-to-text models, such as Google Speech Recognition API, to provide an effective voice input port for users, while the speech output uses mature TTS technology, such as Google Text-to-Speech, to provide users with natural and accurate voice feedback.

[0067] 5. User interface and interaction design:

[0068] Develop easy-to-use user interfaces, taking into account the needs of visually impaired users.

[0069] The interface design supports tactile feedback and voice prompts, allowing users to easily learn and use the system.

[0070] Through these technical means, the present invention can effectively solve the problems of insufficient interactivity, accuracy and real-time performance in Braille learning and translation, while optimizing the interactive experience of visually impaired users. Based on advanced sensing technology and artificial intelligence algorithms, the system achieves efficient Braille recognition and interactive teaching, significantly improving the learning efficiency and quality of life of visually impaired users.

[0071] The specific method of using the system of the present invention includes: first, the user needs to wear the wearable visual and tactile sensor, and then can select the teaching mode or the translation mode through voice interaction.

[0072] Saying the "teaching" command can select the teaching mode. In this mode, the user can use the visual-tactile sensor to recognize words on the Braille board (white board). After recognizing the words they want to learn, the user can say the "explain" command to hand over the recognized word text to the large language model for processing and explanation. After the processing is completed, the system will output the processed words and their corresponding explanations through voice. At this time, if the user wants to ask questions, such as "What is the biggest animals in the world?" The large model will also give the corresponding answer and output it by voice. If the user wants to exit the question mode, he needs to say the "quit" command.

[0073] When the user wants to use the translation function, the user needs to say the "translation" command to switch to the translation mode, and then use the "start" command to start. At this time, the user can use the sensor to slide on the Braille board for recognition. After the text is recognized, use the "correction" command to let the large model segment and correct the recognized text, and finally output the translated result in voice. During the entire interactive process, the user can use the refresh command to refresh the page. Finally, the user can use the stop command to exit the program. The recognition results in the teaching mode are as follows: Figure 5 As shown, the results recognized in translation mode are as follows Figure 6 shown.

[0074] About the specific algorithm in the Braille translation module:

[0075] Camera setup and image capture: The camera in the visual tactile sensor is set to capture the video stream regularly, and the driver mode is specified through cv2.CAP_DSHOW to ensure compatibility and performance with the device. The video stream is obtained through cv2.VideoCapture, and the camera settings are adjusted to obtain the best image quality. The real-time video stream is rotated and cropped to keep only the area containing the Braille text, thereby optimizing processing speed and reducing unnecessary background interference.

[0076] Image preprocessing and Braille dot detection: Gaussian blur is first applied to the image captured by the camera. This step effectively reduces image noise, smoothes image texture, and creates image conditions that are more conducive to analysis for subsequent edge detection. Next, the image is converted to grayscale format (cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY)), because grayscale image processing is faster and more effective for edge detection.

[0077] The Canny edge detection algorithm (cv2.Canny(binary,50,150)) is used. This algorithm can accurately depict the edges in the image and is a key preprocessing step for identifying circular Braille dots. After edge detection, the Hough circle transform (cv2.HoughCircles(edges,cv2.HOUGH_GRADIENT,dp=0.9,minDist=40,param1=120,param2=12,minRadius=11,maxRadius=29)) is applied. This algorithm is specifically used to identify and locate circular objects from the results of edge detection. By accurately setting the size and spacing parameters of the circle, the algorithm can efficiently identify the dots representing Braille characters.

[0078] Specific content of the Braille recognition algorithm:

[0079] Braille dot grouping and matrix generation: For each Braille character, i.e., letter, a dot matrix with three rows and two columns of circle centers is used to represent it. Since the distance between each two dots, the distance between each two characters is fixed. Therefore, they are grouped first during processing. The principle of grouping is as follows: First, all circles are sorted according to their x-coordinates, ensuring that the processing order is from left to right. By traversing the sorted circle list, the circles are grouped based on whether the x-coordinate difference is less than or equal to a threshold. This threshold determines whether the circles are close enough to be considered part of the same group. Each new group is assigned a number, which helps subsequent processing identify which groups represent a complete Braille character.

[0080] After that, the grouped circles need to be converted into a matrix representing Braille dots. Braille characters are usually composed of a 3x2 dot matrix. Only the first group on the far left is processed each time. First, the column is determined. According to the x value range of the circles in the group, it is determined whether these circles belong to the same column or need to be divided into two columns. If the circles are close enough (the x value difference is less than 15), they are considered to belong to the same column. In the process of word teaching or translation, due to the real-time update of images or video frames, each picture only processes the first group on the far left. When traversing all pictures, due to the continuous update of the image or sliding recognition in this process, all Braille characters can be processed.

[0081] Otherwise, use the midpoint of the x-values ​​to divide the circles into two columns. For each column, use the lowest y-value in the column as the anchor point. Determine the row position of each circle in the matrix based on the difference between the y-coordinate of each circle and the anchor point. Update the matrix based on the position of the circle in the column, marking the corresponding position with 1.

[0082] Finally, the processed matrix representing the Braille dots is returned, and the corresponding letters are found in the Braille dictionary. Figure 7 As shown in the confusion matrix in , the average recognition accuracy of the present invention for a single letter exceeds 95%.

[0083] For character recognition and output in teaching mode:

[0084] For the single Braille letter recognition in the teaching mode, the present invention adopts a frame difference algorithm and an inter-frame minimum circle center distance algorithm to prevent the problem of repeated recognition and jitter of a single letter.

[0085] Frame difference algorithm: Because any movement or the appearance of a new object usually causes a significant change in pixel values, the present invention calculates the absolute pixel-by-pixel difference between the two images, that is, the absolute value frame_diff of the difference in pixel intensity at the same position of the two images, and calculates the average value of the difference image frame_diff to obtain a value, namely, the average difference diff_mean, which represents the average degree of change of the entire image frame. The average difference diff_mean can be used as a quantitative indicator to determine whether the change between the two frames is significant. The present invention compares the average difference diff_mean with a set threshold (exemplarily, FRAME_DIFF_THRESHOLD = 0.25). If it is less than the threshold, it means that the image change is relatively small and tends to be stable. At this time, text recognition will be performed, which can effectively avoid repeated recognition.

[0086] In addition, the present invention also uses the minimum center distance algorithm between frames to prevent jitter: by calculating the minimum distance circle_difference between all detected centers of circles between two frames, the slight changes in position can be accurately quantified. Then compare circle_difference with the set threshold (for example, DISTANCE_THRESHOLD = 14). If the change is less than the threshold, it means that it is in a state of slight jitter, and the system will not perform Braille detection on it. By setting a reasonable distance threshold, the system can distinguish between slight, meaningless changes and actual meaningful movements. This reduces misidentification caused by background noise or small-scale vibrations and improves the overall accuracy of the system.

[0087] Through the limitations of these two algorithms, the present invention ensures that Braille detection is performed only at the moment when the image tends to be stable, and any slight jitter thereafter will not be detected. This allows each Braille letter to be recognized only once, reducing repetitions and meeting the requirements of the teaching mode.

[0088] For output detection in translation mode:

[0089] For the sliding recognition process in the translation mode, the present invention adopts a four-threshold line method to limit its output. Since the image is updated in real time, in order to ensure the stability of character detection, reduce repeated recognition and improve the accuracy, the present invention adopts a first threshold line, a second threshold line, a third threshold line, and a fourth threshold line (c1, c2, c3, c4) to manage the output. The selection of the threshold is obtained by integrating the rules and distance relationships of the Braille characters and continuously debugging. The intervals of c1-c2 and c3-c4 are one dot distance, and the intervals of c1-c3 are one square distance. The dot distance represents the distance between the centers of each adjacent dot under a single Braille character (2*3 dots) in the image, and the square distance represents the distance between the center line between two adjacent Braille characters and the next center line. Among them, c1 and c2 are mainly in the left area of ​​the image, and are mainly used to process the characters to be output currently, while c3 and c4 are mainly used to limit the output conditions and detect the next character to arrive. Braille has specific standards on the Braille board, and its actual spacing does not change. At the same time, when using the sensor to scan the above-mentioned Braille characters, the sensor is used in conjunction with the Braille and is in a vertical state at all times. The distance and angle between the sensor and the Braille board will only produce a slight error in the distance between the dots in the image. The dot pitch and square pitch are fixed. The above-mentioned error will only have a slight impact on the detection of the dot pitch and square pitch, but will not affect the use of the detection method. Therefore, the above-mentioned detection scheme of the present invention can effectively detect the array dots corresponding to the characters ready to be output in real time.

[0090] Regarding the detection of the four threshold lines before the output of the braille characters, the following cases are discussed here:

[0091] 1. When outputting characters with circles in both columns, it is necessary to satisfy the requirement that both c1 and c2 have circles on them, that is, Figure 8 the situation in

[0092] 2. When outputting only one column of characters with circles, such as a, b, k, l, it is necessary to satisfy that c1 and c4 have circles, while c2 and c3 do not have circles. Fig. 9 As shown;

[0093] 3. For situations where misrecognition may occur, the algorithm of the present invention can also correctly avoid errors. For example, during the sliding recognition process, it is very likely that the second column of characters with circles in both columns is in c1, and the current character will be mistakenly regarded as a single-column character and output.

[0094] like Fig.10 As shown, b is likely to be misidentified at this time, but since c3 and c4 are used to restrict it, according to situation 2, we can know that to output a single column of characters, c3 must not have a circle, so the above algorithm can avoid this error.

[0095] In order to further prevent repeated recognition, the present invention uses a flag to lock the output letter function. After successful recognition and output, the flag is set to 0, and no output is performed thereafter until a new circle is detected in the c3 threshold column, and then the flag is set to 1 to activate the output, which effectively reduces the repetition rate.

[0096] Finally, the generated Braille matrix is ​​matched against a predefined library of Braille characters and once a matching Braille pattern is found, the recognized character is added to the output string.

[0097] The recognized characters are converted into speech output through the pyttsx3 library.

[0098] Voice interaction in the voice module:

[0099] The system uses the speech_recognition library to configure the microphone input and constantly monitor the surrounding environment. This library first automatically adjusts the sensitivity of the microphone through the adjust_for_ambient_noise method to adapt to different noise environments and ensure the recognition accuracy of voice commands.

[0100] When the user issues voice commands, the listen method captures these commands and converts the audio into text through the recognize_google method. This process utilizes Google's Cloud Speech Recognition service to provide highly accurate speech-to-text conversion.

[0101] The system can parse and respond to a variety of voice commands, including "teaching" (switch to teaching mode), "explain" (explain the recognized word text), "quit" (exit explanation mode), "translation" (switch to translation mode), "start" (start recognizing text in translation mode), "correction" (correct the recognized text), "refresh" (refresh the page, clear the text box), "stop" (exit the program), and user-defined questions in explanation mode.

[0102] For each recognized command, the system determines its type through built-in logic and performs corresponding operations, such as updating the interface display or using speech synthesis technology (such as pyttsx3) to feedback the operation results.

[0103] Applications of large language models:

[0104] In teaching mode, the system uses the Alitong Yiqianwen qwen_max model to receive the Braille input by the user, interpret its meaning in real time and provide relevant learning materials. This model understands and generates language through deep learning algorithms, so that the teaching content is not limited to static data, but is dynamically adjusted according to the user's progress and feedback.

[0105] When users ask questions about a word or sentence, the system can instantly generate educational answers. This feature is achieved through the model's natural language understanding capabilities, enhancing interactivity and personalized learning experience.

[0106] In translation mode, the system also uses the qwen_max model to instantly translate the braille read from the camera into standard text. This conversion is not only fast and accurate, but also able to handle complex, erroneous, missing or repeated texts.

[0107] The translated text is converted into speech output through the pyttsx3 library to ensure that visually impaired users can receive information without obstacles.

[0108] The above technical solution of the present invention has the following effects:

[0109] Improve the interactivity and personalization of Braille learning: Traditional Braille teaching tools mostly rely on static physical teaching materials or non-interactive electronic devices, lacking personalization and dynamic interaction. This invention introduces high-precision visual-tactile sensors and real-time voice interaction functions to provide dynamic learning feedback and personalized teaching to improve learners' participation and learning efficiency.

[0110] Enhance the accuracy and real-time performance of Braille translation: Existing Braille translation tools are likely to misrecognize or repeatedly recognize the same letter when processing complex texts, and their accuracy and speed often cannot meet actual needs. This invention uses advanced Braille recognition algorithms and large model processing to greatly improve the accuracy and response speed of translation, and supports instant translation of more complex texts.

[0111] Optimizing the interactive experience of visually impaired users: Taking into account the special needs of visually impaired users for operation and interaction, the present invention combines the advantages of visual-tactile sensors and large language models to design a multimodal interactive interface. Through voice and tactile feedback, an accessible user interface is provided, allowing visually impaired users to interact with the system more naturally, thereby improving the convenience of learning and use.

[0112] Using large models to improve the processing capabilities of Braille text: When traditional Braille translation tools process semantically complex Braille or multilingual text, the accuracy and speed are often limited by the basic performance of the algorithm. This invention integrates advanced large-scale language models and uses their deep learning capabilities to significantly improve the accuracy and efficiency of text processing, ensuring that high-accuracy translation can be achieved even when complex or ambiguous Braille is input.

[0113] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A voice interactive teaching system integrating visual tactile sensors and Braille recognition algorithms, characterized in that: include: A tactile sensing module and a Braille translation module connected in sequence, wherein the Braille translation module is also connected to a speech module; The braille image is obtained through the tactile sensing module, the input command is obtained through the voice module, and the mode of the braille translation module is adjusted through the input command, and the adjusted mode includes a teaching mode and a translation mode; In the teaching mode, the Braille image is preprocessed by the Braille translation module to obtain dot information, the dot information is recognized by the Braille recognition algorithm to obtain character information, and the character information is broadcasted by the voice module. In the process of recognizing single character information, the use of the Braille recognition algorithm also includes judging whether to use the Braille recognition algorithm for detection through the frame difference algorithm and the minimum center distance algorithm between frames; In the translation mode, the Braille image is preprocessed by the Braille translation module to obtain dot information, the dot information is recognized by the Braille recognition algorithm to obtain character information, the character information is detected by four columns of threshold lines according to the dot information to obtain the detected character information, and the detected character information is broadcasted by the voice module.

2. The system according to claim 1, characterized in that The tactile sensing module adopts a visual tactile sensor.

3. The system according to claim 1, characterized in that The process of identifying dot information through the Braille recognition algorithm includes: The dots in the dot information are grouped, wherein the x-axis coordinates of the dots in a coordinate system based on the horizontal and vertical directions of the image are sorted, and the dots whose x-axis coordinate difference is less than or equal to a preset threshold are traversed from left to right in the sorting result, and the dots are grouped into one group to obtain dots in different groups; For the dots in the same group, determine whether they belong to the same column or two columns, and obtain the column position of the dots. For the dots in each column, use the lowest value of the y-axis as the anchor point, and obtain the row position of the dots according to the difference between the dots in each column and the lowest value. According to the row position and column position, obtain the Braille matrix; The Braille matrix is ​​queried through a Braille dictionary to obtain a single character corresponding to the Braille matrix, and character information is obtained according to the single character, wherein the character information is a single character or a character string.

4. The system according to claim 1, characterized in that The process of judging by the frame difference algorithm and the minimum center distance algorithm between frames includes: The Braille image is judged by a frame difference algorithm, wherein the absolute value of the pixel difference at the same position in the Braille image between adjacent frames is calculated, the average of all the absolute values ​​of the pixel differences is calculated to obtain an average difference, and a threshold judgment is performed on the average difference; The dot information is judged by using an inter-frame minimum center distance algorithm, wherein between adjacent frames, the minimum distance between the centers of different dots in the Braille images between adjacent frames is calculated, and a threshold judgment is performed on the minimum matrix; When the average difference is less than the corresponding threshold and the minimum distance is greater than the corresponding threshold, the Braille image of the current frame is detected using the Braille recognition algorithm.

5. The system according to claim 1, characterized in that The process of detecting character information through four columns of threshold lines includes: Before a single character is output, the dot information is judged by four columns of threshold lines, wherein the four columns of threshold lines include a first threshold line, a second threshold line, and a third threshold line, namely, a fourth threshold line, wherein the spacing between the first threshold line and the second threshold line and between the third threshold line, namely, the fourth threshold line, is one dot distance; the spacing between the first threshold line and the third threshold line is one square distance; When there are two columns of dots corresponding to the character information, there are dots on both the first threshold line and the second threshold line; when there is only one column of dots corresponding to the character information, there are dots on both the first threshold line and the fourth threshold line and there are no dots on the third threshold line, then a single character is valid.

6. The system according to claim 3, characterized in that After detecting the character information through four columns of threshold lines, it also includes: The function of outputting a single character is locked through a flag. When the output character recognition is successful and valid, the flag is 0 and no output is performed. When there is a dot on the third threshold line, the flag is set to 1 and the single character is continuously prepared for output.

7. The system according to claim 1, characterized in that It also includes a language processing module, which is connected to the Braille translation module and the speech module; The language processing module uses a large language model to recognize and interpret character information, and the recognized and interpreted content is output and broadcasted through the voice module.

8. The system according to claim 7, characterized in that In the translation mode, the language processing module uses a large language model to correct and amend the detected character information, and the standard text is output and broadcasted through the voice module.

Images