Portable braille image interaction sensing device and method

Abstract

According to the portable braille image interaction induction device and method, an electromagnetic pulse drive dot matrix method is adopted to replace a traditional physical convex-concave braille display mode and a point-like touch screen mode, the manufacturing complexity and cost are reduced, the braille feeling precision of blind people is improved, the touch feeling of a user can be changed by adjusting the electromagnetic pulse intensity, and the user experience is improved. The information transmission efficiency and the blind reading interestingness are improved; a planar design is adopted, so that richer information interaction experience is provided for visual impaired and hearing impaired users; due to the modular design, multilingual braille translation display can be compatible, and the requirements of users with different languages are met; the lightweight design can be suitable for visually impaired and hearing impaired people in various information acquisition scenes, and the application scenes are wider; and by adopting the graph conversion module, graph characteristics and contour information can be extracted, matched dot-matrix array codes are generated, high-quality graph conversion and display are realized, and visually impaired and hearing impaired people can conveniently obtain state information of the whole device.

Description

Technical Field

[0001] The present invention relates to the field of Braille reading technology, and more specifically, to a portable Braille image interactive sensing device and method. Background Art

[0002] Most of the existing Braille display devices currently use a physical convex and concave dot matrix display method, which has a complex manufacturing process and high cost. At the same time, this display method has low information transmission efficiency and lacks interest, and cannot meet the needs of visually impaired and hearing-impaired people to obtain information in various scenarios. Recently, Braille readers with dot touch screens have also appeared, but the current Braille readers based on dot touch screens have problems with Braille encoding and high system complexity.

[0003] For example, CN114913745A discloses a braille reader, including a dot touch screen, a communication module, a central processing unit, a braille display module and a speech recognition module. The device uses the communication module to obtain text information on various websites, and the user uses the dot touch screen and then uses the braille display module to display the corresponding braille. However, the patent still has the problem of high complexity of the braille encoding method and system, and the dot touch screen cannot enable disabled people to accurately feel the braille.

[0004] CN113327489A discloses a layered mechanical braille reader based on machine vision, including a machine vision part and a braille display part. The device adopts the working principle of screw transmission, can display one or more lines of braille characters, has a simple and ingenious structural design, and is low in cost, but it can only support the braille translation display of a single language and cannot be compatible with the braille translation requirements of multiple languages.

[0005] In addition, most Braille display devices lack the ability to connect to external information devices, have a single usage scenario, and are difficult to adapt to the needs of visually impaired and hearing-impaired people to obtain information in various scenarios such as shopping malls, classrooms, subways, airports, and homes.

[0006] Therefore, there is an urgent need for a portable Braille image interactive sensing device and method that can reduce manufacturing complexity and cost, improve information transmission efficiency and fun, support multi-language Braille translation display, and be able to connect with various external information devices to meet the needs of visually impaired and hearing-impaired people to obtain information in a variety of scenarios and improve the intuitive perception of Braille. Summary of the invention

[0007] In view of the above problems, the purpose of the present invention is to provide a portable Braille image interactive sensing device to solve the following technical problems existing in the existing Braille display device: 1) The physical convex-concave dot matrix display method commonly used in Braille display devices has a complex manufacturing process and high cost; the dot touch screen method cannot enable disabled people to accurately feel Braille; 2) The transmission efficiency is low and lacks fun, and cannot meet the needs of visually impaired and hearing-impaired people to obtain information in various scenarios; 3) It usually only supports Braille translation display in a single language and is not compatible with Braille translation needs in multiple languages; 4) It lacks the ability to connect to external information equipment and has a single usage scenario; 5) The image recognition accuracy and efficiency are low, making it difficult to achieve high-quality graphic conversion and display.

[0008] The present invention provides a portable Braille image interactive sensing device, comprising a start control module, an image conversion module, a multi-language translation module, an interactive module and a state feedback module; wherein: The startup control module is used to adaptively adjust the frequency and intensity of the interaction module based on user needs to determine the target mode of the interaction module, and is also used to receive the data to be interacted with, perform document judgment on the data to be interacted with to distinguish image files and text files in the data to be interacted with, and transmit the image files to the image conversion module, and transmit the text files to the multi-language translation and conversion module; The image conversion module is used to obtain the graphic features and contour features of the image file based on a preset feature extraction model, and generate image electromagnetic pulse array codes for the graphic features and contour features according to a preset electromagnetic pulse generation rule; The multi-language translation conversion module is used to translate the text file into a target language text according to the target language set by the user, and generate a text electromagnetic pulse array code of the target language text according to a preset electromagnetic pulse matching rule; The interaction module is used to generate an image electromagnetic array in a preset image display area according to the image electromagnetic pulse array code, generate a text electromagnetic array in a preset text display area according to the text electromagnetic pulse array code, and refresh the image electromagnetic array or the text electromagnetic array by the frequency in the target mode, and make the image electromagnetic array or the text electromagnetic array emit a pulse stimulation by the intensity in the target mode so that the user can read the data to be interacted; The status feedback module includes a voice unit and a vibration feedback unit. The voice unit is used to play the interaction process about the user's needs based on a preset speaker or headphone interface module, and the vibration feedback unit is used to make customized reminders according to the user's settings.

[0009] Optionally, the speech unit is connected to the startup control module and the multi-language translation and conversion module; The voice unit includes a self-input button. When the self-input button is turned on, the voice unit is used to receive the user's own voice signal and perform voice control on the start control module according to the user's own voice signal; The voice unit includes a listening button and a preset text extraction module. When the listening button is turned on, the voice unit is used to answer the voice signal of the other party of the user's conversation. The text extraction module extracts text from the other party's voice signal to generate a text file, and the text file is transmitted to the multilingual translation and conversion module to generate a text electromagnetic pulse array code about the text file.

[0010] Optionally, it also includes an input-output module connected to the start control module signal; wherein the input-output module includes a wireless network module, a wireless connection module and a data transmission module; wherein, The wireless network module is used to connect to the wireless network to download documents, and use the downloaded documents as data to be interacted with; The wireless connection module is used to connect to the target device based on Bluetooth and local area network, and transmit the target document of the target device to the startup control module through the file transfer channel based on a preset file transfer protocol, so as to use the target document as the data to be interacted.

[0011] Optionally, the feature extraction model is generated by pre-training; wherein, The feature extraction model obtains the graphic features and contour features of the image file, including: Preprocessing the image file to obtain a standardized image; the preprocessing includes grayscale conversion processing, median filtering denoising processing and histogram averaging processing; Using a preset Canny operator to perform edge detection and extraction on the standardized image to obtain contour data; and, using a preset OpenCV algorithm, extracting graphic feature data of the standardized image; Scaling the outline data and the graphic feature data according to the size of the unused electronic array in the image display area to form outline information and graphic information; The contour information and the graphic information are binarized to generate contour features and graphic features.

[0012] Optionally, the interaction module includes an electronic array panel; The electronic array panel includes four rows of arrays; each row of arrays has 12 square arrays; each square array has 6 electronic dot matrices.

[0013] Optionally, the electromagnetic pulse generation rule is: Corresponding the pixels with binary data of 1 in the outline feature and the graphic feature to the electronic dot matrix of the image display area in the electronic array panel; The electromagnetic pulse matching rule is: The three-square array in each row of the array constitutes a Chinese character. According to the preset Chinese character Braille encoding method, the first square in the three-square array represents the initial consonant, the second square in the three-square array represents the final vowel, and the third square in the three-square array represents the tone; wherein, if the entire electronic array panel only displays the target language text, the electronic array panel refreshes once to sense 16 characters.

[0014] Optionally, it further comprises a Braille input module connected to the start control module signal; The Braille input module is used to input dot Braille through preset dot keys, so as to edit the data to be interacted based on the startup control module, or to control the voice unit to play the voice corresponding to the dot Braille based on the startup control module; wherein, when the Braille input module is enabled, the page-up key used by the user to read the data to be interacted is changed to the editing space key, and the page-down key used by the user to read the data to be interacted is changed to the editing enter key.

[0015] Optionally, the image display area and the text display area are self-generated by the startup control module after receiving the data to be interacted; wherein, if the data to be interacted only includes image files, the entire area of ​​the electronic array panel is the image display area; if the data to be interacted includes text files, the entire area of ​​the electronic array panel is the text display area; if the data to be interacted includes both image files and text files, the electronic array panel adopts a preset ratio of left image and right text or a preset ratio of left text and right image.

[0016] The present invention also provides a portable Braille image interactive sensing method, wherein Braille image interaction is realized based on the portable Braille image interactive sensing device as described above, comprising: receiving data to be interacted with, and performing document determination on the data to be interacted with to distinguish image files and text files in the data to be interacted with; Acquire the graphic features and contour features of the image file based on a preset feature extraction model, and generate image electromagnetic pulse array codes for the graphic features and contour features according to a preset electromagnetic pulse generation rule; translate the text file into a target language text according to a target language set by a user, and generate a text electromagnetic pulse array code of the target language text according to a preset electromagnetic pulse matching rule; An image electromagnetic array is generated in a preset image display area according to the image electromagnetic pulse array code, and a text electromagnetic array is generated in a preset text display area according to the text electromagnetic pulse array code, and the image electromagnetic array or the text electromagnetic array is refreshed by the frequency in the target mode, and the image electromagnetic array or the text electromagnetic array is caused to emit a pulse stimulation by the intensity in the target mode so that the user can read the data to be interacted with; wherein, the target mode is formed by adaptively adjusting the frequency and intensity based on user needs.

[0017] Optionally, it also includes: The dot braille is input through the preset dot key to edit the data to be interacted with or to play the voice corresponding to the dot braille.

[0018] As can be seen from the above technical solutions, the portable Braille image interactive sensing device and method provided by the present invention adopts an electromagnetic pulse driving dot matrix method to replace the traditional physical convex and concave Braille display method and dot touch screen method, which reduces the manufacturing complexity and cost, improves the accuracy of the blind people's perception of Braille, and can change the user's tactile sense by adjusting the intensity of the electromagnetic pulse, thereby improving the efficiency of information transmission and the interest of blind reading; adopts a planar design, and senses Braille and graphics with a planar dot matrix array of 12 squares per row and 4 rows and 48 squares in total, providing visually impaired and hearing-impaired users with a richer information interaction experience; modular design, can be compatible with multi-language Braille translation display, such as foreign languages, to meet the needs of users of different languages; It can use Bluetooth, wireless network, and USB interface to connect to various external information devices. It is suitable for visually impaired and hearing-impaired people to obtain information in various scenarios such as shopping malls, classrooms, subways, airports, and homes, and has a wider range of usage scenarios. The graphic conversion module can extract graphic features and contour information, generate matching dot array codes, and achieve high-quality graphic conversion and display. It supports multiple Braille sensing modes, such as left-picture and right text, left text and right picture, and full-picture sensing, to meet the usage habits of different users. It supports editing by inputting dot Braille through the left and right dot buttons, which improves user interactivity. Voice and vibration feedback prompts various status information, making it convenient for visually impaired and hearing-impaired people to obtain status information of the entire device. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] By referring to the following description in conjunction with the accompanying drawings, and with a more comprehensive understanding of the present invention, other objects and results of the present invention will become more clear and easy to understand. In the accompanying drawings: Figure 1 A logical schematic diagram of a portable Braille image interactive sensing device according to an embodiment of the present invention; Figure 2 A schematic diagram of the working process of a portable Braille image interactive sensing device according to an embodiment of the present invention; Figure 3is a front schematic diagram of a portable Braille image interactive sensing device according to an embodiment of the present invention; Figure 4 is a side schematic diagram of a portable Braille image interactive sensing device according to an embodiment of the present invention; Figure 5 4 is a flow chart of a portable Braille image interactive sensing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The existing Braille display devices have the following technical problems: 1) The physical convex-concave dot matrix display method commonly used in Braille display devices has a complex manufacturing process and high cost; the dot touch screen method cannot enable disabled people to accurately feel Braille; 2) The transmission efficiency is low and lacks fun, and cannot meet the needs of visually impaired and hearing-impaired people to obtain information in various scenarios; 3) It usually only supports Braille translation display in a single language and is not compatible with Braille translation needs in multiple languages; 4) It lacks the ability to connect to external information equipment and has a single usage scenario; 5) The image recognition accuracy and efficiency are low, making it difficult to achieve high-quality graphic conversion and display.

[0021] In view of the above problems, the present invention provides a portable Braille image interactive sensing device and method. The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0022] In order to illustrate the portable Braille image interactive sensing device and method provided by the present invention, Figure 1-Figure 5 The embodiments of the present invention are exemplarily indicated.

[0023] The following description of exemplary embodiments is merely illustrative in nature and is not intended to limit the present invention and its application or use. Technologies and devices known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technologies and devices should be considered as part of the specification.

[0024] like Figure 1-Figure 4 As shown together, the portable braille image interactive sensing device 100 provided by the present invention includes a start control module 110, an image conversion module 120, a multi-language translation module 130, an interactive module 140 and a state feedback module 150; wherein, The startup control module 110 is used to adaptively adjust the frequency and intensity of the interaction module based on user needs to determine the target mode of the interaction module, and is also used to receive data to be interacted with, perform document judgment on the data to be interacted with to distinguish image files and text files in the data to be interacted with, and transmit the image files to the image conversion module, and transmit the text files to the multi-language translation and conversion module; The image conversion module 120 is used to obtain the graphic features and contour features of the image file based on a preset feature extraction model, and generate image electromagnetic pulse array codes for the graphic features and contour features according to a preset electromagnetic pulse generation rule; The multi-language translation conversion module 130 is used to translate the text file into a target language text according to the target language set by the user, and generate a text electromagnetic pulse array code of the target language text according to a preset electromagnetic pulse matching rule; The interaction module 140, which may also be called a voice and vibration feedback module, is used to generate an image electromagnetic array in a preset image display area according to the image electromagnetic pulse array code, generate a text electromagnetic array in a preset text display area according to the text electromagnetic pulse array code, and refresh the image electromagnetic array or the text electromagnetic array at the frequency in the target mode, and make the image electromagnetic array or the text electromagnetic array emit a pulse stimulus at the intensity in the target mode so that the user can read the data to be interacted with; The state feedback 150 module includes a voice unit 151 and a vibration feedback unit 152. The voice unit is used to play the interaction process about the user's needs based on a preset speaker or headphone interface module, and the vibration feedback unit is used to make customized reminders according to the user's settings.

[0025] The speech unit 151 is connected to the startup control module 110 and the multi-language translation and conversion module 130; The voice unit 151 includes a self-input button 1511. When the self-input button is turned on, the voice unit is used to receive the user's own voice signal and perform voice control on the start control module according to the user's own voice signal; The voice unit includes a listening button 1512 and a preset text extraction module 1513. When the listening button is turned on, the voice unit is used to answer the voice signal of the other party of the user's conversation, and the text extraction module is used to extract the text of the other party's voice signal to generate a text file, and the text file is transmitted to the multilingual translation conversion module to generate a text electromagnetic pulse array code about the text file. When a blind person who has lost his hearing communicates with outsiders, the outsiders can convert the sound into a text electromagnetic pulse array code through the voice unit, so that the user can know the communication information of the outsiders based on the electromagnetic array. Then, the user can mark the braille through the subsequent braille input module, and the voice unit will play the voice corresponding to the marked braille, so that people with both hearing impairment and blindness can achieve barrier-free communication with the outside world. In a specific embodiment, the self-input button 1511 can be set to be pressed for more than 10 seconds. Figure 4 The pulse intensity adjustment button shown in FIG. 15 is a button for adjusting the pulse intensity. The listening button 1512 can be set to be pressed for more than 10 seconds. Figure 4 Adjust refresh rate button as shown in .

[0026] The portable braille image interactive sensing device in this embodiment further includes an input-output module 160 connected to the start control module by signal; wherein the input-output module includes a wireless network module 161, a wireless connection module 162 and a data transmission module 163; The wireless network module 161 is used to connect to a wireless network to download documents, and use the downloaded documents as data to be interacted with; The wireless connection module 162 is used to connect to the target device based on Bluetooth or a local area network, and transmit the target document of the target device to the start control module through a file transmission channel based on a preset file transmission protocol, so as to use the target document as data to be interacted with; The data transmission module 163 is used to download documents through an effective transmission channel and use the downloaded documents as data to be interacted with.

[0027] In this embodiment, when Figure 4 After the switch button of the portable braille image interactive sensing device shown is turned on, the user can first press Figure 4 The connection switch button shown enters the connection selection mode, and the user needs to select a connection method to establish a connection with the external device. Here, Bluetooth connection is taken as an example: when the user selects Bluetooth connection, the device initiates a communication request with the external device that has been connected before, and prompts the user by voice and vibration after the Bluetooth connection is successful. The device enters the Braille sensing mode, and the selection information on the screen of the connected external device is transmitted to the Bluetooth interface. If there is no external device that can be connected within the Bluetooth communication range or the attempt to establish a connection fails, the user is prompted by voice and vibration to change the connection mode, which can be achieved by long pressing the connection switch button.

[0028] External device information can be loaded into the device in four ways based on the input and output module 160. The first is the file method. The device supports commonly used text file formats, such as TXT and word; the second is streaming input, mainly including push information and screen reading information, etc.; the third is to connect to screen reading software, such as NVDA (NonVisual Desktop Access), etc. At this time, the input information is controlled by the screen reading software; the fourth is to connect to the device supporting software, which can generate graphics and input graphic information.

[0029] In the image conversion module 120, which may also be called a graphic conversion module, the graphic features and contour features of the image file are obtained based on a preset feature extraction model, and the image electromagnetic pulse array code is generated for the graphic features and contour features according to a preset electromagnetic pulse generation rule. The feature extraction model is generated by pre-training; wherein, The step of the feature extraction model acquiring the graphic features and contour features of the image file comprises: Preprocessing the image file to obtain a standardized image; the preprocessing includes grayscale conversion processing, median filtering denoising processing and histogram averaging processing; Using a preset Canny operator to perform edge detection and extraction on the standardized image to obtain contour data, and using a preset OpenCV algorithm to extract graphic feature data of the standardized image; Scaling the outline data and the graphic feature data according to the size of the unused electronic array in the image display area to form outline information and graphic information; The contour information and the graphic information are binarized to generate contour features and graphic features.

[0030] The specific training process of the feature extraction model will not be described here. You can input data yourself and use the original feature extraction framework for training and generation, or you can directly use a trained mature feature extraction model. As long as it can meet the needs of high-precision feature extraction, there is no restriction here.

[0031] The multi-language translation conversion module 130 may also be called a multi-language Braille translation module. It translates the text file into a target language text according to the target language set by the user, and generates a text electromagnetic pulse array code of the target language text according to a preset electromagnetic pulse matching rule. The translation can support multiple languages. In a specific embodiment, all text information in the data to be interacted received from the external device can be converted into a Braille dot matrix display format by a built-in algorithm in the multi-language translation conversion module 130. In a specific embodiment, Chinese, English, Japanese, French, Korean, German and Latin Braille translation algorithms are built-in. In another specific embodiment, the Android translation software or Microsoft translation software that can be installed on a mobile phone or computer can be directly downloaded on the portable Braille image interactive sensing device of this embodiment. However, in order to ensure the stability of the translation, a translation algorithm for converting a text file into a Braille dot matrix display format is directly built-in in the multi-language translation conversion module 130 in this specific implementation, that is, Chinese, English, Japanese, French, Korean, German and Latin Braille translation algorithms are directly built-in. Other more niche small language algorithms are compatible by downloading and updating algorithm packages. When using the portable Braille image interactive sensing device of this embodiment, the user can load different translation algorithms by long pressing the language switching button. When the switch is successful, voice and vibration prompts are also used, and the currently supported language information is displayed on the Braille and graphic sensing modules. The language switching button is set on the side of the portable Braille image interactive sensing device of this embodiment.

[0032] The interaction module 140 is used to generate an image electromagnetic array in a preset image display area according to the image electromagnetic pulse array code, generate a text electromagnetic array in a preset text display area according to the text electromagnetic pulse array code, and refresh the image electromagnetic array or the text electromagnetic array by the frequency in the target mode, and make the image electromagnetic array or the text electromagnetic array emit a pulse stimulus by the intensity in the target mode so that the user can read the data to be interacted; wherein the interaction module includes an electronic array panel 141, which can also be called a Braille and graphic sensing module; The electronic array panel 141 includes four rows of arrays; each row of arrays has 12 square arrays; each square array has 6 electronic dot arrays; accordingly, the electromagnetic pulse generation rule is: Corresponding the pixels with binary data of 1 in the outline feature and the graphic feature to the electronic dot matrix of the image display area in the electronic array panel; The electromagnetic pulse matching rule is: The three-square array in each row of the array constitutes a Chinese character. According to the preset Chinese character Braille encoding method, the first square in the three-square array represents the initial consonant, the second square in the three-square array represents the final vowel, and the third square in the three-square array represents the tone; wherein, if the entire electronic array panel only displays the target language text, the electronic array panel refreshes once to sense 16 characters.

[0033] In general, that is, for the loaded graphic files, the image conversion module can extract the graphic features and contour features through the algorithm, and automatically generate the matching dot matrix array code according to the number of dot matrices of the graphic sensing module of the device, that is, firstly, the graphic file is converted into grayscale, denoised by median filtering, and histogram equalization, and the Canny operator is used for edge detection to extract the contour data of the graphic. Secondly, after extracting the main graphic data (graphic information) through the OpenCV algorithm, the image is binarized. For example, a threshold can be set, and pixels greater than this threshold are set to 1 (white), and pixels less than this threshold are set to 0 (black). Finally, according to the size of the electronic dot matrix, the size of the graphic body is adjusted to n rows and m columns by scaling and cropping the graphic body, and each pixel value of the adjusted image is mapped to an element of the dot matrix. The Braille and graphic sensing module (electronic array panel 141) receives the code of the supporting software and has three graphic sensing modes. The first is left-picture and right-text, that is, the left-hand area senses the graphic and the right-hand area senses the corresponding Braille; the second is left-text and right-picture, which is the opposite of the first; the third is full-picture sensing, that is, both the left and right areas sense the graphic.

[0034] For loaded text files, all text information received from external devices is converted into a Braille dot matrix display format through the built-in algorithm of the Braille translation module. This device has built-in Braille translation algorithms for Chinese, English, Japanese, French, Korean, German and Latin (you can also use the installation software of the mobile phone or computer). Other small language algorithms can be compatible by downloading and updating the algorithm package. You can load different translation algorithms by long pressing the language switch key. When the switch is successful, voice and vibration prompts are also used, and the currently supported language information is displayed in the Braille and graphic sensing module. The Braille and graphic sensing module (electronic array panel 141) receives the text electromagnetic pulse array code, and the dot matrix can be flexibly configured. For example, 288 electronic dot matrices are realized by electromagnetic pulses, and every 6 electronic dot matrices are one square, with a total of 12 squares from left to right. There are 4 rows from top to bottom, a total of 48 squares. When sensing Chinese Braille, three sides constitute a Chinese character. According to the general encoding method of Chinese Braille, for example, the first side represents the initial consonant, the second side represents the final vowel, and the third side represents the tone. 16 characters are sensed once refreshed. When sensing phonetic characters (English, French, German, etc.), each side represents a letter. Since the number of letters in a word is not uniform, when a word crosses the boundary, the device voice and vibration prompt the user that the word is not completed, and it will continue to be displayed from the left after refreshing. In a specific embodiment, Braille sensing can use automatic refresh mode or manual refresh mode. The information efficiency of Chinese is higher than that of phonetic characters, so the device refresh frequency is set according to the results of the phonetic character survey. According to the English survey, the speed of English word reading by ordinary users is 60 words per minute, and those who have received professional training can reach the fastest 300 words per minute. The average number of letters in an English word is 5 letters, so the automatic refresh is carried out at a speed of 30 to 300 words per minute, that is, 150 to 1500 sides per minute, so the device refresh frequency can be set to 0.5S to 5S. The device refresh rate can be adjusted by the user through the refresh rate adjustment wheel, with a step length of 0.5S. After the setting is successful, voice and vibration prompts will also be given. Manual refresh can be completed by scrolling up and down to refresh the previous or next line, or by turning up and down to refresh the page. The content of a page is exactly 48 squares. Braille can enhance the induction through the intensity of electromagnetic pulses. Users can adjust the intensity of the pulse intensity adjustment wheel to suit their own touch. The pulse intensity is set to 1 to 10 levels.

[0035] In this embodiment, the image display area and the text display area are generated by the startup control module after receiving the data to be interacted; wherein, if the data to be interacted only includes an image file, the entire area of ​​the electronic array panel is the image display area; if the data to be interacted includes a text file, the entire area of ​​the electronic array panel is the text display area; if the data to be interacted includes both an image file and a text file, the electronic array panel is divided into a preset proportion by left image and right text or by left text and right image, that is, after receiving the data to be interacted, the startup control module is divided into a preset proportion by left text and right image according to the data to be interacted (file). ) in the order of the original layout of the data (file) to be interacted with, that is, array codes (image electromagnetic pulse array code, text electromagnetic pulse array code) are generated in the order of the original layout of the data (file) to be interacted with, so as to arrange the array codes in the order in which the Braille and graphic sensing modules (electronic array panel 141) receive the array codes. If there are images and text at the same time, left image and right text can be used, that is, the left hand area senses the graphics and the right hand area senses the corresponding Braille; left text and right image can also be used, which is the opposite of the first method; if there are only graphics or only text, full sensing is used, that is, both the left and right areas sense graphics or both the left and right areas sense text.

[0036] The state feedback 150 module includes a voice unit 151 and a vibration feedback unit 152. The voice unit is used to play the interactive process about the user's needs based on a preset speaker or headphone interface module, and the vibration feedback unit is used to customize reminders according to the user's settings. The existence of the vibration feedback unit allows the visually impaired and hearing-impaired to express different information through different vibration methods. It vibrates once if successful, vibrates twice in a row if failed, and vibrates continuously when the battery is low. Taking Bluetooth connection as an example, if the device is successfully connected, it vibrates once, and if there is no external device to connect or the connection fails, it vibrates twice. Hearing people can turn off vibration and only use voice feedback. In order to protect personal privacy, the device provides a headphone jack.

[0037] In addition, the portable braille image interactive sensing device in this embodiment also includes a braille input module 170 connected to the start control module signal; the braille input module 170 is used to input dot braille through a preset dot key, so as to edit the data to be interacted based on the start control module, or control the voice unit to play the voice corresponding to the dot braille based on the start control module; wherein, when the braille input module is enabled, the page up key used by the user to read the data to be interacted is changed to an editing space key, and the page down key used by the user to read the data to be interacted is changed to an editing enter key; such as Figure 3The dot key is divided into a left-hand dot area and a right-hand dot area. Each dot area has three dots, and a total of six dots are set. The three dots on the left are recorded as left 1 (corresponding to key 1), left 2 (corresponding to key 2), and left 3 (corresponding to key 3). The three dots on the right can be recorded as right 1 (corresponding to key 4), right 2 (corresponding to key 5), and right 3 (corresponding to key 6). The user presses the required keys at the same time to enter a Braille character. For example: - To enter the letter "A" (point 1 is raised): just press key 1.

[0038] - To enter the letter "B" (dots 1 and 2 are raised): Press key 1 and key 2 simultaneously.

[0039] - To enter the letter "C" (dots 1 and 4 are raised): Press key 1 and key 4 simultaneously.

[0040] - The user needs to press the corresponding keys at the same time, and the typewriter will print the corresponding combination of bumps on the paper. The order of pressing the keys does not affect the result, as long as the correct key combination is pressed.

[0041] For example, the letter "D" (dots 1, 4, and 5 are raised): press keys 1, 4, and 5 at the same time; the letter "F" (dots 1, 2, and 4 are raised): press keys 1, 2, and 4 at the same time.

[0042] That is, the six dots correspond to the six dots of Braille, and users input characters by pressing the required key combination at the same time. This method is efficient and intuitive, suitable for visually impaired people, and improves the interaction efficiency of disabled people.

[0043] In addition, if Figure 3 , Figure 4 As shown together, the front of the portable Braille image interactive sensing device of this embodiment is provided with a save page-up button (space button), a "page-down button (enter button)", an exit button, and a scroll wheel for easy document editing and document reading; the side of the portable Braille image interactive sensing device is provided with a switch button, a connection switching button, a language switching button, a USB interface, a headphone interface, a pulse intensity adjustment button, and a refresh frequency adjustment button, and each button has a different form and shape to facilitate the control of the entire portable Braille image interactive sensing device.

[0044] More specifically, the specific implementation parameters of this embodiment are as follows: electromagnetic pulse dot matrix response time: ≤10ms; electromagnetic pulse dot matrix refresh time: 0.5S~5S, adjusted in steps of 0.5S; electromagnetic pulse dot matrix intensity: level 1~10; battery life: ≥15 hours; communication range: stable communication within 10 meters of wireless or Bluetooth connection.

[0045] As described above, the portable Braille image interactive sensing device provided by the present invention adaptively adjusts the frequency and intensity of the interactive module based on user needs through the startup control module 110 to determine the target mode for the interactive module, receives the data to be interacted with, performs document judgment on the data to be interacted with to distinguish image files and text files in the data to be interacted with, and transmits the image files to the image conversion module, and transmits the text files to the multilingual translation conversion module; obtains the graphic features and contour features of the image file based on a preset feature extraction model through the image conversion module 120, and generates an image electromagnetic pulse array code for the graphic features and the contour features according to a preset electromagnetic pulse generation rule; and generates an image electromagnetic pulse array code for the graphic features and the contour features according to a preset electromagnetic pulse generation rule through the multilingual translation conversion module 130 according to the user-set The target language translates the text file into the target language text, and generates the text electromagnetic pulse array code of the target language text according to the preset electromagnetic pulse matching rules; generates an image electromagnetic array in a preset image display area according to the image electromagnetic pulse array code through the interaction module 140, generates a text electromagnetic array in a preset text display area according to the text electromagnetic pulse array code, and refreshes the image electromagnetic array or the text electromagnetic array through the frequency in the target mode, and makes the image electromagnetic array or the text electromagnetic array emit a pulse stimulation through the intensity in the target mode so that the user can read the data to be interacted; plays the interaction process related to the user's needs based on the preset speaker or headphone interface module through the state feedback module 150, and makes customized reminders according to the user's settings.

[0046] In this way, the electromagnetic pulse driven dot matrix method is used to replace the traditional physical convex and concave Braille display method and dot touch screen method, which reduces the manufacturing complexity and cost, improves the accuracy of blind people's perception of Braille, and by adjusting the intensity of the electromagnetic pulse, the user's tactile sense can be changed, which improves the efficiency of information transmission and the interest of blind reading; a planar design is adopted to sense Braille and graphics with a planar dot matrix array of 12 squares per row and 4 rows and 48 squares in total, providing visually impaired and hearing-impaired users with a richer information interaction experience; modular design can be compatible with multi-language Braille translation display, such as foreign languages, to meet the needs of users of different languages; Bluetooth, wireless network, USB port can be used The port can be connected to various external information devices. It is suitable for the visually impaired and hearing-impaired to obtain information in various information scenarios such as shopping malls, classrooms, subways, airports, and homes, and has a wider range of usage scenarios. The graphic conversion module can extract graphic features and contour information, generate matching dot array codes, and achieve high-quality graphic conversion and display. It supports a variety of Braille sensing modes, such as left-picture and right text, left text and right picture, and full-picture sensing, to meet the usage habits of different users. It supports editing by inputting dot Braille through the left and right dot buttons, which improves user interactivity. Voice and vibration feedback prompts various status information, making it convenient for the visually impaired and hearing-impaired to obtain the status information of the entire device.

[0047] In addition, if Figure 4 As shown, the present invention also provides a portable Braille image interactive sensing method, which realizes Braille image interaction based on the portable Braille image interactive sensing device as described above, comprising: S1: receiving data to be interacted with, and performing document determination on the data to be interacted with to distinguish image files and text files in the data to be interacted with; S2: acquiring the graphic features and contour features of the image file based on a preset feature extraction model, and generating image electromagnetic pulse array codes for the graphic features and contour features according to preset electromagnetic pulse generation rules; translating the text file into a target language text according to a target language set by a user, and generating a text electromagnetic pulse array code of the target language text according to preset electromagnetic pulse matching rules; S3: Generate an image electromagnetic array in a preset image display area according to the image electromagnetic pulse array code, generate a text electromagnetic array in a preset text display area according to the text electromagnetic pulse array code, and refresh the image electromagnetic array or the text electromagnetic array at the frequency in the target mode, and make the image electromagnetic array or the text electromagnetic array emit a pulse stimulation through the intensity in the target mode so that the user can read the data to be interacted with; wherein, the target mode is adaptively adjusted in frequency and intensity based on user needs.

[0048] In one embodiment, it also includes: S4: inputting dot braille through a preset dot key to edit the data to be interacted with or play the voice corresponding to the dot braille; thereby enabling the disabled to read, edit, and interact with voice without obstacles.

[0049] The specific embodiments can refer to the specific implementation method of the above-mentioned portable Braille image interactive sensing device, which will not be described in detail again.

[0050] As shown above, the portable Braille image interactive sensing method provided by the present invention adopts an electromagnetic pulse driven dot matrix method, which stimulates the user's finger skin through electromagnetic pulses instead of the traditional physical convex and concave Braille display method, thereby reducing the manufacturing complexity and cost, and providing visually impaired and hearing-impaired users with a richer information interaction experience; it can extract graphic features and contour information, generate matching dot matrix array codes, and achieve high-quality graphic conversion and display, and is suitable for visually impaired and hearing-impaired people in various information acquisition scenarios such as shopping malls, classrooms, subways, airports, and homes. The usage scenarios are more extensive, improving information transmission efficiency and blind reading fun.

[0051] As described above with reference to the accompanying drawings, the portable Braille image interactive sensing device and method proposed in the present invention are described by way of example. However, those skilled in the art should understand that various improvements can be made to the portable Braille image interactive sensing device and method proposed in the present invention without departing from the content of the present invention. Therefore, the protection scope of the present invention should be determined by the content of the attached claims.

Claims

1. A portable Braille image interactive sensing device, characterized in that: It includes a startup control module, an image conversion module, a multi-language translation module, an interaction module and a status feedback module; wherein, The startup control module is used to adaptively adjust the frequency and intensity of the interaction module based on user needs to determine the target mode of the interaction module, and is also used to receive the data to be interacted with, perform document judgment on the data to be interacted with to distinguish image files and text files in the data to be interacted with, and transmit the image files to the image conversion module, and transmit the text files to the multi-language translation and conversion module; The image conversion module is used to obtain the graphic features and contour features of the image file based on a preset feature extraction model, and generate image electromagnetic pulse array codes for the graphic features and contour features according to a preset electromagnetic pulse generation rule; The multi-language translation conversion module is used to translate the text file into a target language text according to the target language set by the user, and generate a text electromagnetic pulse array code of the target language text according to a preset electromagnetic pulse matching rule; The interaction module is used to generate an image electromagnetic array in a preset image display area according to the image electromagnetic pulse array code, generate a text electromagnetic array in a preset text display area according to the text electromagnetic pulse array code, and refresh the image electromagnetic array or the text electromagnetic array by the frequency in the target mode, and make the image electromagnetic array or the text electromagnetic array emit a pulse stimulation by the intensity in the target mode so that the user can read the data to be interacted; The status feedback module includes a voice unit and a vibration feedback unit. The voice unit is used to play the interaction process about the user's needs based on a preset speaker or headphone interface module, and the vibration feedback unit is used to make customized reminders according to the user's settings.

2. The portable Braille image interactive sensing device according to claim 1, characterized in that: The speech unit is connected to the startup control module and the multi-language translation and conversion module; The voice unit includes a self-input button. When the self-input button is turned on, the voice unit is used to receive the user's own voice signal and perform voice control on the start control module according to the user's own voice signal; The voice unit includes a listening button and a preset text extraction module. When the listening button is turned on, the voice unit is used to answer the voice signal of the other party of the user's conversation. The text extraction module extracts text from the other party's voice signal to generate a text file, and the text file is transmitted to the multilingual translation and conversion module to generate a text electromagnetic pulse array code about the text file.

3. The portable Braille image interactive sensing device according to claim 1, characterized in that: It also includes an input-output module connected to the start control module signal; wherein the input-output module includes a wireless network module, a wireless connection module and a data transmission module; wherein, The wireless network module is used to connect to the wireless network to download documents, and use the downloaded documents as data to be interacted with; The wireless connection module is used to connect to the target device based on Bluetooth and local area network, and transmit the target document of the target device to the startup control module through the file transfer channel based on a preset file transfer protocol, so as to use the target document as the data to be interacted.

4. The portable Braille image interactive sensing device according to claim 1, characterized in that: The feature extraction model is generated by pre-training; wherein, The feature extraction model obtains the graphic features and contour features of the image file, including: Preprocessing the image file to obtain a standardized image; the preprocessing includes grayscale conversion processing, median filtering denoising processing and histogram averaging processing; Using a preset Canny operator to perform edge detection and extraction on the standardized image to obtain contour data; and, using a preset OpenCV algorithm, extracting graphic feature data of the standardized image; Scaling the outline data and the graphic feature data according to the size of the unused electronic array in the image display area to form outline information and graphic information; The contour information and the graphic information are binarized to generate contour features and graphic features.

5. The portable Braille image interactive sensing device according to claim 4, characterized in that: The interactive module includes an electronic array panel; The electronic array panel includes four rows of arrays; each row of arrays has 12 square arrays; each square array has 6 electronic dot matrices.

6. The portable Braille image interactive sensing device according to claim 5, characterized in that: The electromagnetic pulse generation rule is: Corresponding the pixels with binary data of 1 in the outline feature and the graphic feature to the electronic dot matrix of the image display area in the electronic array panel; The electromagnetic pulse matching rule is: The three-square array in each row of the array constitutes a Chinese character. According to the preset Chinese character Braille encoding method, the first square in the three-square array represents the initial consonant, the second square in the three-square array represents the final vowel, and the third square in the three-square array represents the tone; wherein, if the entire electronic array panel only displays the target language text, the electronic array panel refreshes once to sense 16 characters.

7. The portable Braille image interactive sensing device according to claim 5, characterized in that: Also included is a Braille input module signal-connected to the startup control module; The Braille input module is used to input dot Braille through preset dot keys, so as to edit the data to be interacted based on the startup control module, or to control the voice unit to play the voice corresponding to the dot Braille based on the startup control module; wherein, when the Braille input module is enabled, the page-up key used by the user to read the data to be interacted is changed to the editing space key, and the page-down key used by the user to read the data to be interacted is changed to the editing enter key.

8. The portable Braille image interactive sensing device according to claim 7, characterized in that: The image display area and the text display area are self-generated by the startup control module after receiving the data to be interacted; wherein, if the data to be interacted only includes image files, the entire area of ​​the electronic array panel is the image display area; if the data to be interacted includes text files, the entire area of ​​the electronic array panel is the text display area; if the data to be interacted includes both image files and text files, the electronic array panel is divided into a preset proportion with the image on the left and the text on the right, or with the text on the left and the image on the right.

9. A portable Braille image interactive sensing method, characterized in that: The portable braille image interaction sensing device according to any one of claims 1 to 8 is used to realize braille image interaction, comprising: receiving data to be interacted with, and performing document determination on the data to be interacted with to distinguish image files and text files in the data to be interacted with; Acquire the graphic features and contour features of the image file based on a preset feature extraction model, and generate image electromagnetic pulse array codes for the graphic features and contour features according to a preset electromagnetic pulse generation rule; translate the text file into a target language text according to a target language set by a user, and generate a text electromagnetic pulse array code of the target language text according to a preset electromagnetic pulse matching rule; An image electromagnetic array is generated in a preset image display area according to the image electromagnetic pulse array code, and a text electromagnetic array is generated in a preset text display area according to the text electromagnetic pulse array code, and the image electromagnetic array or the text electromagnetic array is refreshed by the frequency in the target mode, and the image electromagnetic array or the text electromagnetic array is caused to emit a pulse stimulation by the intensity in the target mode so that the user can read the data to be interacted with; wherein, the target mode is formed by adaptively adjusting the frequency and intensity based on user needs.

10. The portable Braille image interactive sensing method according to claim 9, characterized in that: Also includes: The dot braille is input through the preset dot key to edit the data to be interacted with or to play the voice corresponding to the dot braille.

Images