An information acquisition system and method for the visually impaired

Abstract

The application discloses an information acquisition system and method suitable for visually impaired people, which comprises a hardware layer, a driver layer, an operating system layer, a middleware layer and an application layer connected in sequence through an interface; the application layer comprises a voice interaction module, a large model integration module and a Braille conversion module; the hardware layer is used for collecting voice input instructions, outputting Braille character sequences and voice output results; the voice interaction module is used for converting voice input instructions into text input instructions and converting text output results into voice output results; the large model integration module is used for analyzing text input instructions and generating text output results; and the Braille conversion module is used for converting text output results into Braille character sequences, so that the hardware layer outputs Braille character sequences based on a timestamp alignment technology. Thus, based on powerful semantic understanding and information retrieval of the large model integration module, accurate information acquisition is realized, and a five-layer decoupling architecture is adopted, thereby improving the convenience of later system maintenance.

Description

Technical Field

[0001] This application relates to the field of information processing technology, and in particular to an information acquisition system and method suitable for visually impaired people. Background Technology

[0002] With the rapid development of information technology, the construction of an information-accessible environment is particularly crucial for visually impaired individuals. Currently, access to multimodal information such as text, images, and audio through assistive technologies is an important prerequisite for visually impaired individuals to integrate into various aspects of social life, including education, employment, transportation, and public services.

[0003] Currently, assistive tools for the visually impaired mainly include screen reading software and Braille display devices. However, these tools are limited in function, have low levels of intelligence, and are inefficient in interaction. For example, traditional screen reading software often uses linear voice broadcasting, which can only read interface elements word by word or sentence by sentence. It lacks the ability to understand the semantic content and organize the information in a structured way, making it difficult to handle complex semantic scenarios. Users need to confirm and input commands multiple times, which is cumbersome and has a high error rate, preventing visually impaired people from accurately and quickly obtaining effective information. In addition, the current assistive tools for the visually impaired are integrated into the entire system, making it difficult to maintain, upgrade, and expand them in the future.

[0004] Therefore, how to improve the efficiency and accuracy of information acquisition for visually impaired people, and how to enhance the convenience of subsequent maintenance, are problems that urgently need to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, one aspect of this application provides an information acquisition system suitable for visually impaired people, the system comprising: a hardware layer, a driver layer, an operating system layer, a middleware layer, and an application layer; wherein the application layer includes a voice interaction module, a large model integration module, and a Braille conversion module;

[0006] The hardware layer, the driver layer, the operating system layer, the middleware layer, and the application layer communicate with each other sequentially through standardized interfaces.

[0007] The hardware layer is used to collect the user's voice input commands;

[0008] The voice interaction module is used to convert the voice input command into a text input command;

[0009] The large model integration module is used to parse the text input command to determine the information acquisition task and generate the text output result of the information acquisition task, so that the voice interaction module can convert the text output result into a voice output result.

[0010] The Braille conversion module is used to convert the text output result into a Braille character sequence, so that the hardware layer can output the voice output result and the Braille character sequence simultaneously based on timestamp alignment technology.

[0011] Optionally, the hardware layer includes: a voice input / output device, an embedded processor, a Braille display device, and a network device;

[0012] The voice input / output device is used to collect the voice input command and output the voice output result;

[0013] The Braille display device is used to output the Braille character sequence.

[0014] Optionally, the middleware layer is used to acquire data processing instructions transmitted by the application layer in order to provide data processing services to the application layer; wherein, the middleware layer includes a speech recognition engine, a speech synthesis engine, a network communication library, and a local database; the application layer further includes a data management module for managing the local database; the data processing instructions include speech recognition instructions and speech synthesis instructions;

[0015] The speech recognition engine is used to perform speech recognition on the speech input instruction according to the speech recognition command, and generate the text input instruction;

[0016] The speech synthesis engine is used to convert the text output result into the speech output result according to the speech synthesis instruction;

[0017] The local database is used to store the text output results.

[0018] Optionally, the middleware layer further includes an encoding database for storing preset Braille encoding rules; the Braille conversion module is used for:

[0019] Based on the preset Braille word segmentation rules, the text output result is segmented using a word matching algorithm to obtain multiple words;

[0020] According to the preset Braille encoding rules, a search algorithm is used to convert the words into corresponding Braille characters to obtain an initial Braille character sequence;

[0021] The initial Braille character sequence is rearranged into paragraphs;

[0022] The rearranged Braille character sequence is optimized by point positioning to obtain the Braille character sequence.

[0023] Optionally, the paragraph rearrangement of the initial Braille character sequence includes:

[0024] Determine the first number of blind squares corresponding to each word, and obtain the number of available blind squares for each line of the Braille display device;

[0025] The second blind symbol number corresponding to the initial Braille character sequence is determined based on the first blind symbol number;

[0026] Based on the available number of blind symbols and the second number of blind symbols, the initial Braille character sequence is rearranged into paragraphs.

[0027] Optionally, the rearranged Braille character sequence may be optimized in terms of dot position, including:

[0028] If there is a number in the first Braille square that is greater than the number of available Braille squares, the last Braille character of the Braille display device is set as a Braille connector;

[0029] If the target word exists in the word, the last Braille character is set as the Braille connector; the target word is a word in which at least one of the initial consonant, final vowel and tone is distributed in different numbers of available Braille symbols.

[0030] Optionally, the middleware layer may also include a network communication library;

[0031] The network communication library is used to establish a communication connection between the large model integration module and the target large model; the target large model is a model that can be used for the information acquisition task.

[0032] The large model integration module is used for:

[0033] Determine whether the text output result exists in the local database;

[0034] If it exists, retrieve the text output result from the local database;

[0035] If it does not exist, after establishing a communication connection with the target large model, initiate a request corresponding to the information acquisition task to obtain the text output result.

[0036] Optionally, the driver layer is used to control the hardware layer so that the hardware layer can interact with the operating system layer; wherein, the driver layer includes a voice input / output device driver, a Braille display device driver, and a network device driver.

[0037] Optionally, the operating system layer includes an embedded system, which provides an operating system environment for information acquisition.

[0038] Another aspect of this application provides an information acquisition method suitable for visually impaired people, applied to the aforementioned information acquisition system for visually impaired people, wherein the hardware layer, driver layer, operating system layer, middleware layer, and application layer of the system are sequentially connected for communication through a standardized interface; the method includes:

[0039] Collect user's voice input commands;

[0040] Convert the voice input command into a text input command;

[0041] The text input instruction is parsed to determine the information acquisition task, and the text output result of the information acquisition task is generated;

[0042] The text output is converted into a speech output and a Braille character sequence;

[0043] Based on timestamp alignment technology, the system simultaneously outputs the speech output and the Braille character sequence. The information acquisition system and method provided in this application, suitable for visually impaired individuals, offer the following benefits: Through a hardware layer and a speech interaction module, it enables efficient, accurate, and smooth speech interaction for visually impaired individuals. Simultaneously, based on the powerful semantic understanding and information retrieval capabilities of the large model integration module, it achieves precise information acquisition. Furthermore, the Braille conversion module facilitates information reading for visually impaired individuals. This system breaks through the traditional non-intelligent information presentation methods of assistive devices for the visually impaired, achieving a balance between response speed and information integrity. It solves problems such as low interaction efficiency and difficulty in structured Braille conversion in complex semantic scenarios during information acquisition for visually impaired individuals, thereby improving their information acquisition efficiency.

[0044] In addition, a five-layer decoupled architecture is adopted, that is, the hardware layer and the application layer are relatively independent, and the interaction between the layers is realized through interfaces, thereby achieving decoupling of the system. Each layer has a clear division of labor in terms of function, thereby improving the convenience of later maintenance, upgrades and expansion. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the structure of an information acquisition system suitable for visually impaired people provided in an embodiment of this application;

[0046] Figure 2 This is a schematic diagram of another information acquisition system suitable for visually impaired people provided in an embodiment of this application;

[0047] Figure 3 This is a schematic diagram of a hardware layer structure provided in an embodiment of this application;

[0048] Figure 4 This is a flowchart illustrating an information acquisition method for visually impaired people provided in an embodiment of this application.

[0049] Figure 5 This application provides a data flow diagram of an information acquisition system suitable for visually impaired people, as illustrated in an embodiment of the present application.

[0050] Figure 6 This is a flowchart illustrating an information acquisition method for visually impaired people provided in an embodiment of this application.

[0051] The attached diagram is labeled as follows: Hardware layer 1, Driver layer 2, Operating system layer 3, Middleware layer 4, Application layer 5, Voice input / output device 10, Embedded processor 11, Braille display device 12, Network device 13, Voice input / output device driver 20, Braille display device driver 21, Network device driver 22, Embedded system 30, Speech recognition engine 40, Speech synthesis engine 41, Network communication library 42, Local database 43, Encoding database 44, Voice interaction module 50, Large model integration module 51, Braille conversion module 52, Data management module 53. Detailed Implementation

[0052] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0053] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0054] Figure 1 This is a schematic diagram of the structure of an information acquisition system for visually impaired people provided in an embodiment of this application, as shown below. Figure 1 As shown, the system includes: hardware layer 1, driver layer 2, operating system layer 3, middleware layer 4, and application layer 5. Application layer 5 includes a voice interaction module 50, a large model integration module 51, and a Braille conversion module 52.

[0055] In one optional embodiment, the hardware layer 1, driver layer 2, operating system layer 3, middleware layer 4, and application layer 5 are sequentially connected and communicate with each other through standardized interfaces. Thus, the information acquisition system provided in this application is decoupled into five layers, each with a clearly defined functional division. The hardware layer 1 to the application layer 5 are relatively independent of each other, and interaction between the layers is achieved through standardized interfaces.

[0056] This five-layer decoupled architecture facilitates system modularity, making it easier to maintain, upgrade, and expand later. For example, if you need to replace the voice interaction module 50 in application layer 5 or add a new hardware device in hardware layer 1, you only need to modify the relevant layers without affecting other parts of the entire system, thus improving the system's management convenience.

[0057] Table 1 is a schematic table of a standardized interface provided in the embodiments of this application. It should be noted that this application does not limit the standardized interface between hardware layer 1 and application layer 5. For ease of understanding, in an optional embodiment, examples of standardized interfaces and protocols between different layers will be given below.

[0058] Table 1 is a schematic diagram of a standardized interface.

[0059]

[0060] As shown in Table 1, different layers communicate with each other through different standardized interfaces, and different layers can independently implement different functional applications, supporting independent upgrades and maintenance.

[0061] Based on Table 1, the information acquisition system provided in this application implements a layered communication mechanism. Specifically, as shown in Table 1, hardware layer 1 interacts with driver layer 2 through driver interface (i.e., physical interface), driver layer 2 interacts with operating system layer 3 through system call interface, operating system layer 3 interacts with middleware layer 4 through API interface, and middleware layer 4 interacts with application layer 5 through service call interface.

[0062] Each layer operates independently, summarizing the user information acquisition process. The layered data processing mechanism is as follows: Hardware layer 1 acquires voice input commands → Driver layer 2 processes them → Operating system layer 3 schedules them → Middleware layer 4 parses them → Application layer 5 processes them logically → Middleware layer 4 generates Braille character sequences and voice output results → Operating system layer 3 schedules them → Driver layer 2 controls them → Hardware layer 1 displays the Braille character sequences and plays the voice output results. This achieves a complete process from user voice input to large-scale model information processing, and then to dual-modal output of voice and Braille characters.

[0063] Specifically, based on Figure 1In the information acquisition system shown, in a specific embodiment, the user inputs a voice input command through hardware layer 1. At this time, hardware layer 1 will collect the voice input command based on... Figure 1 The system host shown transmits data downwards. The voice interaction module 50 converts voice input commands into text input commands. In other words, the user inputs voice, which needs to be converted into text that the large model can directly recognize.

[0064] Furthermore, the large model integration module 51 parses the text input command to determine the current user information acquisition task and generates the corresponding text output result. At this time, the large model integration module 51 transmits the text output result to the voice interaction module 50 and the Braille conversion module 52 respectively. The voice interaction module 50 converts the text output result into a voice output result, while the Braille conversion module 52 converts the text output result into a Braille character sequence.

[0065] Hardware layer 1 outputs the Braille character sequence and the voice output result based on timestamp alignment technology, realizing dual-modal output of voice and Braille. Users can obtain the final result through both modes. Among them, the timestamp alignment technology can ensure synchronization between voice feedback and Braille display, improving the user's information acquisition experience.

[0066] Figure 2 This is a schematic diagram of another information acquisition system for visually impaired people provided in an embodiment of this application. In an optional embodiment, based on... Figure 1 The application also provides a software-level layered system in addition to the layered information acquisition system shown. For example... Figure 2 As shown, in a specific embodiment, the user interaction layer is responsible for voice interaction with the user, receiving voice input commands, and outputting Braille character sequences and voice output results. The business logic layer contains core business logic, for example, Figure 1 The system comprises a large model integration module 51, a Braille conversion module 52, and a data management module 53. In the data storage layer, a local database 43 stores user-specific data and text output results. User-specific data may include, but is not limited to, data on blind massage knowledge and data on barrier-free travel for the blind. The system support layer provides the operating system, drivers, and middleware, corresponding to… Figure 1 The system consists of an operating system layer 3, a driver layer 2, and a middleware layer 4, which support system operation. The external interface layer communicates with the large model integration module 51 via an API interface, and is used to call the target large model to generate text output results.

[0067] Therefore, the information acquisition system for visually impaired individuals provided in this application, through hardware layer 1 and voice interaction module 50, enables efficient, accurate, and smooth voice interaction for visually impaired individuals. Simultaneously, based on the powerful semantic understanding and information retrieval capabilities of the large model integration module 51, accurate information acquisition is achieved. Furthermore, the Braille conversion module 52 facilitates information reading for visually impaired individuals. This application breaks through the traditional non-intelligent information presentation methods of assistive devices for the visually impaired, achieving a balance between response speed and information integrity, solving problems such as low interaction efficiency and difficulty in structured Braille conversion in complex semantic scenarios during information acquisition for visually impaired individuals, and improving the efficiency of information acquisition for visually impaired individuals.

[0068] In addition, a five-layer decoupled architecture is adopted, that is, from hardware layer 1 to application layer 5, each layer is relatively independent. The interaction between each layer is realized through interfaces, thereby decoupling the system. Each layer has a clear division of labor in terms of function, thereby improving the convenience of later maintenance, upgrades and expansion.

[0069] In one alternative embodiment, such as Figure 1 As shown, hardware layer 1 includes: a voice input / output device 10, an embedded processor 11, a Braille display device 12, and a network device 13;

[0070] The voice input / output device 10 is used to collect voice input commands and output voice output results. In a specific embodiment, the voice input / output device 10 includes a voice input device and a voice output device. In an optional embodiment, the voice input / output device 10 can be a microphone, such as a high-sensitivity, noise-reduction-performance USB microphone or a patch microphone, like the Audio-Technica AT2020USB+. This microphone effectively reduces environmental noise interference and connects directly to the embedded processor 11 via a USB interface for convenient data transmission. In a specific embodiment, the microphone collects voice input commands, converts them into electrical signals, and then converts them into text commands using a minimum energy spectrum voice noise reduction algorithm and an end-to-end model speech recognition algorithm.

[0071] In another alternative embodiment, the microphone can be a Knowles I2S microphone array, which supports beamforming technology, has strong noise reduction capabilities, and is suitable for application scenarios with extremely high requirements for voice acquisition quality and strict requirements for noise reduction effects. Therefore, in specific embodiments, the voice input device can be selected according to actual business needs, and this application does not limit it.

[0072] The voice output device can be a speaker. In one alternative embodiment, a USB active speaker with clear sound quality, such as the Edifier R10U, can be selected. In a specific embodiment, it is also connected to the embedded processor 11 via a USB interface and can provide good audio output.

[0073] The Braille display device 12 is used to output Braille character sequences. In one optional embodiment, the Braille display device 12 can use an electronic Braille display employing HumanWare Brailliant, such as the Brailleliant BI 40. It connects to the embedded processor 11 via a USB or Bluetooth interface, can display 40 Braille squares, and can dynamically update the displayed content according to Braille data sent by the software. In another optional embodiment, the TouchMind Smart Braille device can be used. This device has lower costs, better maintainability, and is a more economical choice for cost-sensitive projects, while still meeting Braille display requirements. In fact, this application does not specifically limit the selection of the Braille display device 12; it can be selected according to actual business needs.

[0074] In an optional embodiment, network device 13 in hardware layer 1 provides network services to the information acquisition system. Furthermore, hardware layer 1 may also include a power management module and a display screen. The power management module provides energy management for the information acquisition system, while the display screen can be used to directly display text output results.

[0075] Figure 3 This is a schematic diagram of the structure of a hardware layer 1 provided in an embodiment of this application, as shown below. Figure 3 As shown, hardware layer 1 includes an embedded processor 11, a microphone, a speaker, a Braille display device 12, a network device 13, and a power management module. In an optional embodiment, the embedded processor 11 can be a single-board SBC, which may include, but is not limited to, Raspberry Pi 4B, TinkerBoard 2S, etc. This application does not limit the selection of the embedded processor 11, and it can be selected according to actual business needs.

[0076] The Raspberry Pi 4B is based on the ARM Cortex-A72 architecture, featuring a 1.5GHz quad-core CPU and memory options including, but not limited to, 1GB, 2GB, 4GB, and 8GB. This provides ample computing power for system operation with low power consumption. Furthermore, the Raspberry Pi 4B offers a rich array of interfaces, including but not limited to USB, HDMI, Ethernet, and GPIO ports, facilitating connection to various external devices. Therefore, this application does not limit the devices included in hardware layer 1; for example, it may also include displays, keyboards, and mice. In one optional embodiment, middleware layer 4 is used to acquire data processing instructions transmitted from application layer 5 to provide data processing services to application layer 5. Figure 1As shown, the middleware layer 4 includes a speech recognition engine 40, a speech synthesis engine 41, a network communication library 42, and a local database 43; the application layer 5 also includes a data management module 53 for managing the local database 43, and the corresponding data processing instructions include speech recognition instructions and speech synthesis instructions.

[0077] The speech recognition engine 40 is used to perform speech recognition on the speech input command according to the speech recognition instruction, and generate a text input command. The speech synthesis engine 41 is used to convert the text output result into a speech output result.

[0078] In a specific embodiment, after receiving a user's voice input command, the voice interaction module 50 sends the audio data to the voice recognition engine 40. At this time, the voice recognition engine 40 analyzes and recognizes the voice input command using a specified algorithm or model, converts the voice input command into a text input command, and returns the converted text input command to the voice interaction module 50. For example, after completing the conversion, the voice recognition engine 40 returns the recognized text "How is the weather today?" to the voice interaction module 50.

[0079] Local database 43 is used to store user-specific data and the text output results. In an optional embodiment, to further improve the efficiency of user information retrieval, historical data obtained by the user can be stored in local database 43. Furthermore, to save storage resources, the data stored in local database 43 can be updated and optimized at regular intervals (e.g., every week) through database management in application layer 5. Specifically, in an optional embodiment, the access frequency of the data is sorted, and a predetermined percentage of the data at the end (e.g., the last 20%) is deleted.

[0080] In an optional embodiment, the database management module can use an SQLite database management system to manage user query history, favorite information, and other data, supporting operations such as adding, deleting, modifying, and querying data, making it convenient for users to view and manage their information. Of course, the data management module 53 can also periodically back up the data in the local database 43 to prevent data loss. Simultaneously, it provides a data recovery function, enabling quick restoration to the previous backup state when data problems occur.

[0081] In an optional embodiment, the middleware layer 4 further includes an encoding database 44 for storing preset Braille encoding rules, which supports the Braille conversion module 52 in converting text output results into Braille character sequences.

[0082] Specifically, the Braille conversion module is used for:

[0083] Based on the preset Braille word segmentation rules, a word matching algorithm is used to segment the text output to obtain multiple words;

[0084] According to the preset Braille coding rules, a search algorithm is used to convert words into corresponding Braille characters to obtain an initial Braille character sequence;

[0085] Paragraph rearrangement of the initial Braille character sequence;

[0086] The rearranged Braille character sequence is optimized by point placement to obtain the Braille character sequence.

[0087] In a specific embodiment, the text output needs to be converted into Braille characters that visually impaired people can touch. Specifically, based on preset Braille word segmentation rules, the text output is segmented using a word matching algorithm to segment the text output into words. Each word. Furthermore, based on the Braille coding rules stored in the coding database, a lookup algorithm is used to... The words are converted into corresponding Braille characters, that is, words are transformed into Braille characters that visually impaired people can touch and access information from.

[0088] like Figure 2 As shown, in an optional embodiment, when the Braille conversion module 52 converts the text output result into a Braille character sequence, it rearranges the text information of the text output result into paragraphs according to the Braille encoding rules (e.g., GB / T15720-2008) stored in the encoding database 44. The paragraph rearrangement refers to reorganizing the paragraph structure of the initial Braille character text according to Braille reading habits and the number of Braille symbols in the Braille display device 12, so that it presents a more reasonable reading rhythm and information hierarchy in the Braille display device 12.

[0089] In one alternative embodiment, paragraph rearrangement of the initial Braille character sequence includes:

[0090] Determine the first Braille square number corresponding to each word, and obtain the number of available Braille squares for each line of the Braille display device;

[0091] The second blind symbol number corresponding to the initial Braille character sequence is determined based on the first blind symbol number;

[0092] Based on the number of available blind symbols and the second number of blind symbols, the initial Braille character sequence is rearranged into paragraphs.

[0093] In a specific embodiment, the number of first blind symbols corresponding to each word after word segmentation is first determined. Therefore, according to the first blind sign square number The second blind square number of the initial Braille character sequence can be determined as follows: Simultaneously, the number of available Braille squares for each line of the Braille display device 12 is obtained. For example, the number of available Braille squares can be set to... In one alternative embodiment, .

[0094] Furthermore, based on the first Braille square number and the available Braille square number of the Braille display device 12, the initial Braille character text is rearranged into paragraphs to generate a rearranged Braille character sequence, facilitating the Braille display device 12 to output Braille characters sequentially according to the order of the rearranged Braille character sequence. In an optional embodiment, it is assumed that the rearranged Braille character sequence includes the following number of characters: ,in, according to Sure.

[0095] Specifically, in one optional embodiment, if ,but ,in, for The integer part of the calculation result, This represents the remainder. If... ,but In other words, when When the remainder is zero, equal The calculation results, when When the remainder is not zero, equal Add 1 to the integer part of the calculation result.

[0096] In an optional embodiment, the Braille characters obtained after paragraph rearrangement can be optimized in terms of dot position. Dot position optimization refers to determining, through algorithms or rules, whether the last Braille character on the Braille display device 12 should be displayed as a Braille connector, so that it is more in line with tactile recognition habits and reduces misreading and cognitive burden.

[0097] Specifically, as an optional implementation, dot optimization is performed on the rearranged Braille character sequence, including:

[0098] If there is a number in the first Braille square that is greater than the number of available Braille squares, set the last Braille character of the Braille display device as a Braille connector;

[0099] If the target word is present in the word, the last Braille character is set as a Braille connector; the target word is a word in which at least one of the initial consonant, final vowel and tone is distributed in different available Braille symbol squares.

[0100] Specifically, in a specific embodiment, if a certain word occupies the first blind symbol square number Greater than the number of available blind symbols ,Right now, The last Braille symbol displayed by the Braille display device 12 is a Braille connector. If at least one of the initial consonant, final vowel, and tone of a word is distributed across different available Braille symbol squares, the last Braille symbol displayed by the Braille display device 12 is a Braille connector.

[0101] Understandably, paragraph rearrangement focuses on the macro-level structure and reading flow of the text, improving reading pace and structural comprehension, while point optimization focuses on the micro-level of individual characters or symbols, improving tactile recognition efficiency. It's important to note that in this specific implementation, paragraph rearrangement precedes point optimization.

[0102] It should be noted that, in one optional embodiment, the encoding database 44 can use the Pyll library to integrate LibLouis, and the encoding database 44 can include Braille encoding rules for multiple languages, specifically including but not limited to Unicode Braille, Grade 1 (basic Braille, character-letter correspondence), Grade 2 (compressed Braille, including abbreviation rules), Chinese Braille (GB / T 15720-2008), Japanese Braille, French Braille, etc. In a specific embodiment, the information acquisition system can support dynamic switching of Braille encoding rules, and the user can select the desired Braille type through voice commands or buttons on the Braille display.

[0103] In a specific embodiment, when the Braille conversion module 52 performs Braille conversion, it can determine the language currently used by the user through the speech recognition engine 40 and call the Braille encoding rules of the corresponding language to convert the text output result.

[0104] In one alternative embodiment, such as Figure 1 As shown, the middleware layer 4 also includes a network communication library 42;

[0105] Network communication library 42 is used to build a communication connection between the large model integration module 51 and the target large model; the target large model is a model that can be used for information acquisition tasks.

[0106] Large model integration module 51, used for:

[0107] Determine if the text output exists in local database 43;

[0108] If it exists, retrieve the text output result from the local database 43;

[0109] If it does not exist, after establishing a communication connection with the target large model, initiate a request corresponding to the information acquisition task to obtain the text output result.

[0110] Understandably, large models typically require a large amount of storage space and high computing power. If large models are deployed directly on the electronic devices where the information acquisition system is located, it will result in the electronic devices being very large and inconvenient to carry, or it will result in a long time to acquire information and fail to achieve the desired effect for the user.

[0111] Therefore, to solve the aforementioned technical problems, this application provides a technical solution for lightweight deployment of local caching and cloud retrieval on embedded single-board computers, namely, providing a hierarchical processing strategy. Specifically, such as... Figure 1 and 3 As shown, in a specific embodiment, after the large model integration module 51 obtains the JSON-formatted text input instruction transmitted by the voice interaction module 50, it first parses the text input instruction, extracts the key text information, and thus determines the current information acquisition task.

[0112] Furthermore, first determine if identical historical data exists in local database 43, that is, whether the text output result corresponding to the current information retrieval task exists. If it exists, retrieve it directly from local database 43. If it does not exist, then proceed as follows: Figure 3 As shown, a communication connection is established with the target large model through the network communication library 42. Specifically, the network communication library 42 connects to the API interface of the target large model and initiates the corresponding HTTP request. The target large model may include, but is not limited to, Tongyi Qianwen and DeepSeek; this application does not limit the large model deployed.

[0113] In one optional embodiment, the driver layer 2 is responsible for managing and controlling the hardware layer 1, providing a unified device interface for the upper layers, enabling the hardware layer 1 to interact with the operating system layer 3; wherein, as... Figure 1 As shown, the driver layer 2 includes a voice input / output device driver 20, a Braille display device driver 21, and a network device driver 13.

[0114] like Figure 1 As shown, the operating system layer 3 includes an embedded system 30. The operating system layer 3 provides an operating system environment for information acquisition and supports and manages the middleware layer 4 and application layer 5, such as process management, memory management, and file systems. In an optional embodiment, the embedded system 30 may be an embedded Linux system.

[0115] Therefore, the information acquisition system provided in this application adopts a five-layer decoupled architecture, breaks through the traditional non-intelligent information presentation method of visually impaired assistive devices, and achieves a balance between response speed and information integrity through local caching and cloud retrieval. It solves problems such as low efficiency of multimodal interaction, difficulty in Braille structured conversion in complex semantic scenarios, and slow real-time response of large models deployed on embedded devices in the information acquisition process of visually impaired people, thereby improving the information acquisition efficiency of visually impaired people and providing a good user experience.

[0116] Figure 4 This is a schematic diagram of an information acquisition system provided in an embodiment of this application. For ease of understanding, it will be described below in conjunction with... Figure 4 The following example illustrates the information acquisition process provided in this application.

[0117] like Figure 4 As shown, in one optional embodiment, the user initiates a request. Specifically, the user inputs a voice command to the system, such as "What is the weather like today?" This voice input command becomes the trigger point for the entire interaction process, initiating the system's processing flow for the user's request.

[0118] After receiving the user's voice input command, the voice interaction module 50 sends the audio data to the voice recognition engine 40. The voice recognition engine 40 then uses a specified algorithm and model to analyze and recognize the audio data, converting it into text data. After the conversion is complete, the voice recognition engine 40 returns the recognized text "How's the weather today?" to the voice interaction module 50.

[0119] Furthermore, after the voice interaction module 50 receives the text input command, it forwards the command to the large model integration module 51. At this point, the large model integration module 51, using the network communication library 42, initiates an HTTP request according to the format and protocol specified by the target large model. The target large model server receives the request and processes it, such as... Figure 4 As shown, it also returns JSON-formatted information containing weather data. At this point, the network communication library 42 receives this data and passes it to the large model integration module 51.

[0120] The large model integration module 51 parses the returned JSON data, extracts the key weather information text, and passes the text to the Braille conversion module 52. The Braille conversion module 52 calls the encoding rule library and converts the weather information text into a Braille character sequence according to the preset Braille encoding rules.

[0121] Finally, the Braille conversion module 52 sends the converted Braille character sequence to the Braille display driver. Based on the received instructions, the Braille display driver controls the Braille display device 12 to update the displayed content, presenting the corresponding Braille information for blind users to touch and read. Simultaneously, the Braille conversion module 52 transmits the text information for voice feedback to the speech synthesis engine 41. The speech synthesis engine 41 converts the text into natural and fluent speech, which is then played back through a speaker, allowing users to obtain query results via voice.

[0122] It should be noted that, in addition to the examples mentioned above, the information retrieval system provided in this application can also meet the information access needs of visually impaired individuals in various aspects of their daily lives, work, and studies. For example, they can ask questions about various subjects, such as history, science, and culture, to obtain detailed answers. By combining map information, users can search for travel routes and transportation options, facilitating their travel. They can also access entertainment content such as music, stories, and jokes, enriching their leisure time.

[0123] Figure 5 This is a data flow diagram of an information acquisition system for visually impaired people provided in an embodiment of this application. To enable those skilled in the art to further understand the technical solution provided in this application, the following will be combined with... Figure 5 Describe the data flow during the information acquisition process of this application.

[0124] In a specific embodiment, such as Figure 5 As shown, for the input path of voice input commands, i.e., the input stage, the specific data flow is: microphone → voice IO driver. Further, in the processing stage, the information processing path includes: voice recognition engine 40 → voice interaction module 50 → network communication library 42 → large model integration module 51 → Braille conversion module 52.

[0125] After completing the above information processing, dual-mode output of voice and Braille can be performed. Specifically, the Braille output path includes: Braille conversion module 52 → Braille display driver → Braille display device 12. The voice feedback path includes: Braille conversion module 52 → voice synthesis engine 41 → voice IO driver → speaker.

[0126] In the above embodiments, an information acquisition system suitable for visually impaired people has been described in detail. This application also provides an embodiment of an information acquisition method suitable for visually impaired people. The method is applied to the information acquisition system suitable for visually impaired people in any of the above embodiments. The hardware layer, driver layer, operating system layer, middleware layer and application layer in the system are connected to each other in sequence through standardized interfaces.

[0127] Figure 6This is a flowchart illustrating an information acquisition method for visually impaired individuals provided in an embodiment of this application. Figure 6 As shown, the method includes:

[0128] S60: Collects user's voice input commands;

[0129] S61: Converts voice input commands into text input commands;

[0130] S62: Parse the text input instructions to determine the information retrieval task and generate the text output results of the information retrieval task;

[0131] S63: Convert the text output into a speech output and a Braille character sequence;

[0132] S64: Based on timestamp alignment technology, it simultaneously outputs speech output results and Braille character sequences.

[0133] In specific embodiments, the information acquisition method provided in this application corresponds to the information acquisition system provided in the above embodiments. For details, please refer to the description of the above embodiments, which will not be repeated here.

[0134] It should be noted that although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Claims

1. An information acquisition system for the visually impaired, characterized by comprising: The system comprises a hardware layer, a driver layer, an operating system layer, a middleware layer and an application layer; wherein the application layer comprises a voice interaction module, a large model integration module and a Braille conversion module; the hardware layer comprises a Braille display device; The hardware layer, the driver layer, the operating system layer, the middleware layer and the application layer are sequentially connected in communication through standardized interfaces; The hardware layer is configured to collect voice input instructions of a user; The voice interaction module is configured to convert the voice input instructions into text input instructions; The large model integration module is configured to analyze the text input instructions to determine an information acquisition task, and generate a text output result of the information acquisition task, so that the voice interaction module converts the text output result into a voice output result; The Braille conversion module is configured to convert the text output result into a Braille character sequence, so that the hardware layer simultaneously outputs the voice output result and the Braille character sequence based on a timestamp alignment technology; The middleware layer further comprises an encoding database configured to store preset Braille encoding rules; and the Braille conversion module is configured to: perform word segmentation processing on the text output result according to preset Braille word segmentation rules by using a word matching algorithm to obtain a plurality of words; convert the words into corresponding Braille characters by using a lookup algorithm according to the preset Braille encoding rules to obtain an initial Braille character sequence; perform paragraph rearrangement on the initial Braille character sequence; perform point position optimization on the rearranged Braille character sequence to obtain the Braille character sequence; The paragraph rearrangement on the initial Braille character sequence comprises: determining a first number of Braille squares corresponding to each word, and obtaining an available number of Braille squares of each row of the Braille display device; determining a second number of Braille squares corresponding to the initial Braille character sequence according to the first number of Braille squares; performing paragraph rearrangement on the initial Braille character sequence according to the available number of Braille squares and the second number of Braille squares; The point position optimization on the rearranged Braille character sequence comprises: if there is a number of Braille squares in the first number of Braille squares that is greater than the available number of Braille squares, setting a last Braille character of the Braille display device as a Braille connector; if there is a target word in the words, setting the last Braille character as the Braille connector; the target word is a word in which at least one of initials, finals and tones is distributed in different available numbers of Braille squares.

2. The information acquisition system for visually impaired persons according to claim 1, wherein The hardware layer comprises a voice input / output device, an embedded processor and a network device; The voice input / output device is configured to collect the voice input instructions and output the voice output result; The Braille display device is configured to output the Braille character sequence.

3. The information acquisition system for visually impaired persons according to claim 1, wherein The middleware layer is configured to obtain data processing instructions transmitted by the application layer to provide data processing services for the application layer; wherein the middleware layer comprises a voice recognition engine, a voice synthesis engine, a network communication library and a local database; the application layer further comprises a data management module configured to manage the local database; and the data processing instructions comprise voice recognition instructions and voice synthesis instructions. The voice recognition engine is configured to perform voice recognition on the voice input instruction according to the voice recognition instruction, and generate the text input instruction; The voice synthesis engine is configured to convert the text output result into the voice output result according to the voice synthesis instruction; The local database is configured to store the text output result.

4. The information acquisition system for visually impaired persons according to claim 3, wherein The middleware layer further comprises a network communication library; The network communication library is configured to construct a communication connection between the large model integration module and a target large model; the target large model is a model that can be used for the information acquisition task; The large model integration module is configured to: determine whether the text output result exists in the local database; if yes, acquire the text output result from the local database; if no, after establishing a communication connection with the target large model, initiate a request corresponding to the information acquisition task to acquire the text output result.

5. The information acquisition system for visually impaired persons according to claim 1, wherein The driver layer is configured to control the hardware layer to interact with the operating system layer; wherein the driver layer comprises a voice input / output device driver, a Braille display device driver, and a network device driver.

6. The information acquisition system for visually impaired persons according to claim 1, wherein The operating system layer comprises an embedded system, and is configured to provide an operating system environment for information acquisition.

7. An information acquisition method for the visually impaired, characterized by, The information acquisition system for the visually impaired population of any one of claims 1 to 6, wherein the hardware layer, the driver layer, the operating system layer, the middleware layer, and the application layer in the system are sequentially connected in communication through standardized interfaces; the hardware layer comprises a Braille display device; and the method comprises: collecting a voice input instruction of a user; converting the voice input instruction into a text input instruction; parsing the text input instruction to determine an information acquisition task and generate a text output result of the information acquisition task; converting the text output result into a voice output result and a Braille character sequence; simultaneously outputting the voice output result and the Braille character sequence based on a timestamp alignment technique; performing word segmentation processing on the text output result according to a preset Braille word segmentation rule by using a word matching algorithm to obtain a plurality of words; converting the words into corresponding Braille characters by using a lookup algorithm according to the preset Braille encoding rule to obtain an initial Braille character sequence; performing paragraph rearrangement on the initial Braille character sequence; performing point position optimization on the rearranged Braille character sequence to obtain the Braille character sequence; the paragraph rearrangement on the initial Braille character sequence comprises: determining a first number of Braille cells corresponding to each word and acquiring an available number of Braille cells of each row of the Braille display device; determining a second number of Braille cells corresponding to the initial Braille character sequence according to the first number of Braille cells; performing paragraph rearrangement on the initial Braille character sequence according to the available number of Braille cells and the second number of Braille cells; the point position optimization on the rearranged Braille character sequence comprises: if there is a number of Braille cells in the first number of Braille cells that is greater than the available number of Braille cells, setting the last Braille character of the Braille display device as a Braille connector. If there is a target word in the word, set the last braille character as the braille connector; the target word is a word whose initial, final, and tone are distributed in different available braille squares.

Images