Visual assisting system and method for visually impaired person based on multi-mode large model, intelligent glasses and blind assisting application software

Abstract

The invention provides a visual aiding system and method for visually impaired people based on a multi-mode large model, intelligent glasses and application software. The system comprises a multi-mode large model, blind-helping application software and intelligent glasses, enables a visually impaired person to ask questions through simple voice, achieves the recognition of all things such as environment, obstacles, text information, road names / shop names, flowers and plants, and feeds back the questions to the visually impaired person through voice. According to the scheme, the visually impaired person can conveniently and quickly see all information around the person in a hearing mode, then the independence of work and life is greatly improved, and the visual impaired person has wide application prospects and huge social value.

Description

Technical Field

[0001] The present invention relates to the technical field of visually impaired people "seeing" the physical world and the digital world by hearing, and relates to a visual assistance system and method for the visually impaired people based on a multimodal large model, smart glasses and blind-assisting application software. Background Art

[0002] At present, there are two main types of assistive products for the visually impaired at home and abroad: one is navigation devices that rely on ultrasonic recognition systems, radar systems or GPS positioning systems. These devices can help the visually impaired perceive obstacles around them and provide basic navigation information, but they can usually only provide rough environmental perception and cannot recognize complex object features, text content or dynamic scenes; the other is devices that use OCR (optical character recognition) technology to achieve text recognition. These devices can convert printed text into speech to help the visually impaired read books, newspapers or information on product packaging. However, these devices have a single function and cannot meet the visually impaired's needs for obtaining diverse environmental information.

[0003] With the rapid development of artificial intelligence and computer vision technology, especially the multimodal large model has demonstrated strong general capabilities in image understanding and recognition, video understanding and recognition, etc., providing new possibilities for the innovation of assistive technology for the visually impaired. Therefore, it is necessary to develop an assistive device for the visually impaired that combines a multimodal large model, portable smart glasses and blind-assisting application software to better meet the needs of the visually impaired to "see" all the information around them through hearing, and improve their independence and participation in work and life. Summary of the invention

[0004] The present invention discloses a visual assistance system and method for the visually impaired based on a multimodal large model, smart glasses and blind-assisting application software. The visually impaired can take photos or videos through smart glasses (or the smart glasses can take photos or videos automatically), and ask questions on demand in a voice dialogue style. The multimodal large model will understand and infer the voice questions and pictures or video information, and convert them into voice information, so that the visually impaired can "hear" the real world information, and realize "seeing and understanding" the surrounding environment, "seeing and understanding" obstacles, "seeing and understanding" text information, "seeing and understanding" road names / store names, "seeing and understanding" flowers and plants, etc., bringing unprecedented new experience and assistance.

[0005] The visual assistance system and method for the visually impaired of the present invention mainly consists of the following three core modules: 1. Smart glasses: As the user's main interactive device, smart glasses are equipped with high-performance cameras, microphones, speakers, processors and network modules to collect visual and voice data in real time, perform local processing, and transmit data to blind-assistance application software and multimodal large models. (ii) Blind-assisting application software: installed on intelligent terminal devices (such as smart phones or tablets), the blind-assisting application software is responsible for functions such as speech recognition, image preprocessing, data transmission, voice broadcasting and user settings, and serves as a bridge between smart glasses and multimodal large models; (3) Multimodal large model: Deployed on cloud servers or smart terminal devices (such as smartphones or tablets), the multimodal pre-trained model based on the Transformer architecture has the ability to process images, videos and voices, and can perform in-depth understanding and semantic analysis of the environment and content, providing accurate and timely feedback information for the visually impaired.

[0006] (I) Design and implementation of smart glasses Hardware configuration (1) Camera module: High dynamic range (HDR) camera: It uses a CMOS sensor with HDR function, supporting video recording and high-quality image capture with a resolution of 720P or above. HDR technology allows the camera to capture details of bright and dark areas simultaneously under high-contrast lighting conditions to ensure image quality. Autofocus and image stabilization: It has autofocus (AF) and optical image stabilization (OIS) functions to ensure clear images can be obtained even in movement or vibration. The AF function enables the camera to focus on the target object quickly, and the OIS function stabilizes the image by compensating for hand shake. Low-light enhancement: In low-light or nighttime environments, the camera can improve image brightness and clarity by increasing ISO sensitivity and adopting noise reduction algorithms. The low-light enhancement function enables the visually impaired to obtain reliable visual information under various lighting conditions.

[0007] (I) Design and implementation of smart glasses Hardware configuration (2) Microphone array: Omnidirectional microphone array: multiple microphones are configured to use beamforming technology to improve the quality of voice signal collection. The microphone array can collect sound from multiple directions, and the beamforming technology can focus on the user's voice to improve the signal-to-noise ratio; Active noise reduction and wind noise cancellation: Digital signal processing (DSP) technology is used to filter environmental noise and wind noise to ensure accurate recognition of user voice commands in noisy environments. The active noise reduction function offsets noise by generating anti-phase sound, and the wind noise cancellation technology is specifically targeted at low-frequency noise caused by wind;

[0008] (I) Design and implementation of smart glasses Hardware configuration (3) Audio output: Bone conduction headphones or open headphones: Bone conduction or open headphones are used to avoid blocking the user's ear canal, ensure the perception of environmental sounds, and improve the safety of walking and talking. Bone conduction headphones transmit sound through the skull, while open headphones allow environmental sounds and device sounds to enter the ear canal at the same time. : The audio module supports high-quality audio output to ensure the clarity and real-time nature of voice feedback. Low-latency transmission is very important for voice interaction, ensuring that users can quickly get feedback after issuing commands.

[0009] (I) Design and implementation of smart glasses Hardware configuration (4) Network connection and power management: Network module: supports multiple communication protocols such as USB3.0, USB2.0, Wi-Fi 6, Bluetooth 5.2, etc., to ensure connection stability and data transmission speed in different network environments. Wi-Fi 6 provides high-speed data transmission, and Bluetooth 5.2 is used for short-range device connection.

[0010] (I) Design and implementation of smart glasses Hardware configuration (5) Power system: There are two ways. One way is to rely on the USB3.0 or USB2.0 interface to obtain power from the connected smart terminal (smartphone or tablet computer). The other way is to configure a high-energy-density lithium-ion battery that supports fast charging and intelligent power management and provides a battery life of more than 8 hours.

[0011] (I) Functional realization of smart glasses (6) Voice input: A touch panel is designed on the temple or frame to support multiple operation methods such as sliding, clicking and long pressing. It also supports receiving user voice input when the button is pressed, with high accuracy and low latency.

[0012] (I) Functions of smart glasses (7) Image and video acquisition: Users can use buttons, touch operation or automatic mode to acquire images and videos.

[0013] (II) Design and implementation of the blind-assisting application software (1) Software architecture: Cross-platform support: The blind-assisting application software can be installed on mainstream mobile operating systems, such as Android and iOS, and paired and synchronized with smart glasses. This makes the system widely compatible, and users can choose devices they are familiar with. The software adopts a modular architecture, and the functional modules are loosely coupled, which is easy to maintain and upgrade. The main modules include speech recognition module, image and video preprocessing module, data transmission module, voice broadcast module and user personalization setting module.

[0014] (II) Design and implementation of software to assist the blind (2) Speech recognition module of module functions: Speech recognition engine: Integrates local and cloud speech recognition technologies, supports multiple languages ​​and dialects, and has high recognition accuracy. The speech recognition engine can handle changes in the user's accent and speech speed; supports speech-to-text conversion.

[0015] (II) Design and implementation of the software for assisting the blind (3) Image and video preprocessing module of module functions: Image optimization: perform cropping, rotation, color adjustment, noise reduction and other processing on the collected images to ensure that the image quality meets the input requirements of the multimodal large model. Preprocessing can improve the recognition accuracy of the model; Video encoding: compress and encode the video data to reduce the amount of data transmission and improve the transmission efficiency. Using efficient video encoding formats, such as H.264, can significantly reduce the amount of data while ensuring quality.

[0016] (II) Design and implementation of the blind assistance application software (4) Data transmission module of module functions: Network optimization: Use advanced transmission protocols and data compression technology to ensure the immediacy and stability of data. The transmission protocol needs to consider the network's latency, bandwidth and reliability; Security and encryption: Use security protocols such as SSL / TLS to encrypt data transmission and protect user privacy. Data encryption is essential to prevent information leakage and man-in-the-middle attacks.

[0017] (II) Design and implementation of software to assist the blind (5) Voice broadcast module of module functions: Voice synthesis technology: Using TTS (Text-to-Speech) technology, the text data returned by the multimodal large model is converted into natural and fluent voice. Voice synthesis needs to have clarity, naturalness and emotional expression capabilities.

[0018] (II) Design and implementation of software to assist the blind (6) Voice broadcast module of module functions: Voice synthesis technology: Using TTS (Text-to-Speech) technology, the text data returned by the multimodal large model is converted into natural and fluent voice. Voice synthesis needs to have clarity, naturalness and emotional expression capabilities.

[0019] (II) Design and implementation of application software for assisting the blind (7) User personalization module: supports the adjustment of speech speed, volume, timbre and other parameters, and supports the adjustment of image resolution to meet the personalized needs of users. Users can choose different voice styles to enhance the user experience

[0020] (II) Design and implementation of the blind assistance application software (8) Data management module: Provides data caching and synchronization functions. In the case of unstable network, it can temporarily store data and automatically synchronize after the connection is restored. The data management module needs to ensure the consistency and integrity of the data.

[0021] (III) Design and implementation of a large multimodal model (1) Model architecture: a multimodal pre-trained model based on the Transformer architecture; Model foundation: a deep neural network that integrates vision, hearing, and language, with cross-modal feature extraction and association capabilities. The Transformer architecture is suitable for processing multimodal data with its powerful sequence modeling capabilities; Pre-training and fine-tuning: pre-training using large-scale multimodal datasets, and fine-tuning based on the actual needs of the visually impaired to enhance the performance of the model on specific tasks. Pre-training enables the model to have general perception capabilities, while fine-tuning optimizes it for specific applications.

[0022] (III) Design and implementation of a multimodal large model (2) Functional implementation of cross-modal alignment and semantic understanding: Comprehensive analysis of voice, image and video data: Achieve cross-modal information alignment and semantic understanding, locate relevant areas in the image according to the user's voice command, and generate corresponding descriptions. For example, when a user asks "What dangers are ahead?", the system can identify obstacles in the image and give warnings.

[0023] (III) Design and implementation of multimodal large models (3) Functional implementation of environmental understanding and situational adaptation, object recognition: supports the recognition of tens of thousands of common objects and provides detailed attribute information, such as color, shape, location, etc. Object recognition is one of the core functions of the system and is directly related to the visually impaired's understanding of the environment; scene analysis: classifies and semantically segments the environment, such as indoor, outdoor, road, shopping mall, etc., and provides an overall description of the scene. Scene analysis helps users quickly understand the environment they are in; situational adaptation: adaptively adjusts the level of detail and form of feedback content according to the user's usage scenario and preferences to provide personalized services. For example, in a noisy environment, the system can simplify the feedback content to reduce the amount of information; text recognition and translation: high-precision OCR: can recognize text content in complex backgrounds, including handwriting, print, multilingual text, etc. OCR technology needs to process text of various fonts, sizes and layouts; multilingual translation: supports real-time translation between multiple languages ​​to meet the needs of the visually impaired in different language environments. The translation function is very useful for users who travel internationally or in a multilingual environment. Face recognition: It can recognize familiar people and provide their identity information (with user authorization). Face recognition needs to be performed under the premise of protecting privacy and with the consent of the relevant person. Expression and emotion analysis: Identify the expression and emotional state of a person to help the visually impaired understand the emotions of others. For example, the system can prompt the user that "the other person is smiling" or "the other person looks a little angry."

[0024] (III) Design and implementation of multimodal large models (4) Cloud and end-side deployment: Cloud server: The main computing and reasoning tasks of the model are completed in the cloud, using GPU clusters to provide powerful computing capabilities. Cloud deployment can handle complex computing tasks, but network latency and bandwidth need to be considered. End-side computing: Integrate lightweight model versions in smart glasses and blind-assistance application software to ensure that basic functions are still available when the network is unavailable. End-side computing reduces dependence on the cloud and improves the robustness of the system.

[0025] (IV) Collaborative Workflow of the System (1) Data Collection and Preprocessing: Users issue voice commands through smart glasses, such as “Please tell me what is ahead.” Smart glasses collect current image and voice data, and the blind-aiding application software preprocesses the data, including image optimization and voice analysis.

[0026] (IV) Collaborative Workflow of the System (2) Data Transmission and Security: The pre-processed data is transmitted to the multimodal large model in the cloud through a secure and encrypted network connection; the data transmission module ensures the timeliness and integrity of the data and prevents data loss and leakage.

[0027] (IV) Collaborative Workflow of the System (3) Model Reasoning and Feedback Generation: The multimodal large model analyzes the received data, combines voice commands and environmental information, and generates corresponding text descriptions and semantic information; the model considers the user's historical interaction records and preferences to provide personalized feedback content.

[0028] (IV) Collaborative workflow of the system (4) Voice broadcast and user interaction: The blind assistance application software converts the analysis results of the model into natural and fluent voice through the voice broadcast module. The audio output device of the smart glasses plays the feedback content, and the user obtains the required information through hearing. The user can further issue new instructions based on the feedback, forming a continuous interactive process.

[0029] (V) Advantages and innovations of the system (1) Multimodal fusion: The system makes comprehensive use of multimodal data such as vision, hearing and language, providing rich environmental perception capabilities, far exceeding traditional single-modal auxiliary tools;

[0030] (V) Advantages and innovations of the system (2) Multimodal fusion: high precision and real-time performance: By utilizing advanced artificial intelligence technology and high-performance computing resources, the system has high-precision recognition and analysis capabilities, and achieves real-time feedback through optimized data transmission and processing processes.

[0031] (V) Advantages and innovations of the system (3) User-friendliness: Through the friendly interactive design of smart glasses and blind-assistance application software, users can easily obtain the information they need through simple voice commands or touch operations, which lowers the threshold for use.

[0032] (V) Advantages and innovations of the system (4) Environmental adaptability: The system can maintain good performance in different lighting, weather and noise environments, meeting the needs of visually impaired people in various scenarios.

[0033] (V) Advantages and innovations of the system (5) Privacy and security: Using advanced encryption and data protection technologies to protect user privacy and data security and enhance user trust.

[0034] (V) System advantages and innovations (6) Scalability and openness: The system architecture is open, which facilitates the integration of new functional modules and technologies, such as adding navigation functions, social media interfaces and health monitoring, and has good scalability.

[0035] (VI) Application Examples of the System (1) Indoor Object Recognition: At home, the visually impaired can issue the command “Please tell me what is on the table”. The system collects images through the camera, recognizes and gives feedback: “There is a cup and a newspaper on the table. The cup is on the left and the newspaper is on the right.”

[0036] (VI) Application Examples of the System (2) Text Reading: When shopping in a store, a visually impaired person who wants to know product information can pick up the product and issue the command “Please read product information”. The system will recognize the text on the product packaging and provide feedback on the product name, price, production date, and other information.

[0037] (VI) Application Examples of the System (3) Face Recognition: In social situations, if a visually impaired person wishes to know whether there are any acquaintances around him / her, he / she can issue the command “Please tell me who are the acquaintances around me”. The system will then perform face recognition and respond with “Your friend Mr. Zhang is in front of you on the right”. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of the overall information transmission of the present invention, showing the data interaction and information transmission process between smart glasses, blindness assistance application software and multimodal large model.

[0039] Figure 2 This is a schematic diagram of the structure of the smart glasses of the present invention, which details the layout and connection methods of hardware components such as the camera, microphone, speaker, processor, and network module.

[0040] Figure 3 This is a flow chart of the blind-assisting application software of the present invention, which shows the main functional modules and data processing flow of the application software, including speech recognition, image preprocessing, data transmission and speech broadcasting.

[0041] Figure 4This is a design and implementation diagram of the multimodal large model of the present invention, which illustrates the model's architecture, data flow, and functional modules, including pre-training, fine-tuning, and reasoning processes.

Claims

1. A visual assistance system and method for the visually impaired based on a multimodal large model, smart glasses and blind-assisting application software, characterized in that: The system comprises: 1) A large multimodal model, which is used to process and understand pictures or videos taken by visually impaired users and voice questions. It is deployed on cloud servers or smart terminals. It is a pre-trained model based on the Transformer architecture and fine-tuned to meet the needs of the visually impaired. It has the ability to process multimodal data such as text, images, and videos. 2) Blind-assisting application software, installed on the smart terminal, including: a speech recognition module for receiving and recognizing the voice commands of the visually impaired user; an image and video processing module for preprocessing the images or videos collected by the smart glasses terminal; a data transmission module for sending the processed speech and image or video data to the multimodal large model for inference; a speech broadcast module for synthesizing the results returned by the multimodal large model and playing them to the user; 3) Smart glasses, including: a camera for collecting pictures or video information of the environment; a microphone for receiving voice commands from the user; a speaker for playing voice feedback information; and a computing processing unit for running some application software and processing data; The smart glasses are connected in communication with the blind-assisting application software, and the blind-assisting application software is connected in communication with the multimodal large model, forming a complete data processing and feedback closed loop.

2. The system according to claim 1, characterized in that The multimodal large model is a pre-trained model based on the Transformer architecture, which integrates deep neural networks of vision, hearing and language. After fine-tuning to the needs of the visually impaired, it has cross-modal feature extraction, association and semantic understanding capabilities, and can handle complex scenarios and multilingual environments.

3. The system according to claim 1, characterized in that The speech recognition module of the blind-assisting application software supports local and cloud recognition, and can process speech input in multiple languages, multiple dialects and different speaking speeds, thereby improving recognition accuracy and response speed.

4. The system according to claim 1, characterized in that The image and video processing module of the blind assistance application software includes functions such as image optimization, noise reduction, cropping, rotation, color adjustment and video encoding, ensuring that the image and video data meet the input requirements of the multimodal large model and improving the reasoning accuracy and efficiency.

5. The system according to claim 1, characterized in that The data transmission module of the blind-assisting application software adopts security protocols such as SSL / TLS to encrypt data transmission, and has network optimization and data compression functions to ensure the timeliness, stability and security of the data.

6. The system according to claim 1, characterized in that The voice broadcast module of the blind assistance application software adopts high-quality speech synthesis technology, supports personalized settings of parameters such as speech speed, volume, and timbre, and can feed back the inference results of the multimodal large model to the user in the form of natural and fluent speech.

7. The system according to claim 1, characterized in that The camera of the smart glasses uses a high dynamic range (HDR) camera, supports autofocus, optical image stabilization and low-light enhancement functions, ensuring clear images and videos in various lighting conditions.

8. The system according to claim 1, characterized in that The microphone of the smart glasses terminal is an omnidirectional microphone array with active noise reduction and wind noise elimination functions, and can accurately receive the user's voice commands in a noisy environment.

9. The system according to claim 1, characterized in that The speaker of the smart glasses terminal adopts a bone conduction earphone or an open earphone design to avoid blocking the user's ear canal, ensure the perception of environmental sounds, and improve the safety of walking and talking.

10. The system according to claim 1, characterized in that The computing processing unit of the smart glasses terminal integrates a high-performance, low-power embedded AI chip, supports local neural network reasoning and calculation, and meets the requirements of real-time processing and battery life.

11. A visual assistance method for the visually impaired based on the system according to any one of claims 1 to 10, characterized in that: The method comprises the following steps: a) The visually impaired person issues a voice command through the smart glasses terminal, and the smart glasses terminal collects the current image or video and voice data; b) The blind-assisting application software pre-processes the collected data, including speech recognition, image and video optimization; c) The preprocessed data is encrypted and transmitted to the multimodal large model through the data transmission module for reasoning and analysis; d) The multimodal large model performs semantic understanding and cross-modal association on the data and generates corresponding text descriptions or feedback information; e) The speech broadcast module of the blind assistance application software synthesizes the feedback information of the multimodal large model into speech; f) The speaker of the smart glasses terminal plays the voice feedback information to the visually impaired person to complete the information transmission; g) The visually impaired person continues to issue new voice commands based on the feedback information, forming a continuous interaction process.

12. The method according to claim 11, characterized in that The reasoning analysis of the multimodal large model includes functions such as object recognition, scene analysis, text recognition and translation, face recognition and sentiment analysis, which can meet the needs of the visually impaired in different scenarios.

13. The method according to claim 11, characterized in that When the network is unstable or unavailable, the blind-assisting application software can utilize the computing processing unit of the smart glasses terminal to perform local basic reasoning and feedback functions to ensure that the basic functions of the system are available.

14. The method according to claim 11, characterized in that The system supports a multi-language environment and can automatically adjust the language of speech recognition and speech broadcasting according to the language preferences of the visually impaired.

15. The method according to claim 11, characterized in that The system has self-learning and user preference memory functions, and can optimize feedback content and form based on the visually impaired's usage habits and historical records to provide personalized services.

Images