Intelligent blind guiding glasses system based on multi-mode large language model

Abstract

The invention discloses an intelligent blind guiding glasses system based on a multi-mode large language model. The intelligent blind guiding glasses system comprises an image acquisition module, a voice acquisition module, a large language model module, a bone conduction sound transmission module, a navigation path planning module, a central processor module and a wireless communication module, the image acquisition module is used for acquiring a three-dimensional image of a front scene and providing three-dimensional space information; the voice acquisition module is used for acquiring a voice instruction of a user; the big language model module is used for generating character answers and converting the character answers into voice according to the voice instruction of the user by utilizing a big language model; the bone conduction sound transmission module is used for receiving and playing the voice generated by the large language model module; the navigation path planning module is used for planning a navigation path according to the user position and the destination; the functions of navigation positioning, line guiding, voice prompting and the like are provided for the blind, the glasses are adopted as a carrier, carrying is convenient, voice awakening and other instructions are adopted to replace keys, operation is easy, and practicability is high.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent wearable devices, and particularly relates to an intelligent blind guide glasses system based on a multimodal large language model. Background Art

[0002] For blind guide devices or products, there are mainly four categories nowadays: traditional blind guide aids, audio-visual conversion similar products, implantable BMI types, and non-implantable BMI types.

[0003] Traditional blind guide aids mainly include blind canes and guide dogs; blind canes have a low cost, but cannot provide real-time obstacle avoidance information and navigation services, and are limited in use in complex environments; although guide dogs can provide better navigation services, they have high training costs, are difficult to maintain, and not all visually impaired people can adapt to walking with animals. Audio-visual conversion similar products mainly include intelligent blind canes, intelligent guide dogs, and blind guide wearable devices; intelligent blind canes can realize environmental recognition to collect and sense surrounding environmental information and achieve obstacle avoidance functions, but they are easily affected by camera and clothing occlusion, etc., affecting their recognition effect and having a low safety factor; intelligent guide dogs are large in size, high in price cost, complex in control operation, and not conducive to the use of the blind; the recognition accuracy or functional intelligence level of blind guide wearable devices is lacking and cannot meet the navigation and obstacle avoidance needs of the blind for independent travel. Implantable BMI types control external devices, such as artificial eyes or intelligent glasses, by implanting electrodes into the brains of visually impaired people to interpret brain signals. Due to the need to implant electrodes into the brains of visually impaired people, there are surgical risks, difficulties in device maintenance, and possible ethical issues; non-implantable BMI types capture electrical signals of the brain or muscles through external wearable devices, such as head-mounted electrode caps, but such systems usually face problems of large signal noise, low recognition accuracy, and poor real-time performance. Summary of the Invention

[0004] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and propose an intelligent blind guide glasses system based on a multimodal large language model.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions:

[0006] An intelligent blind guide glasses system based on a multimodal large language model includes an image acquisition module, a voice acquisition module, a large language model module, a bone conduction sound transmission module, a navigation path planning module, a central processing unit module, and a wireless communication module;

[0007] The image acquisition module uses a binocular depth camera to obtain a stereoscopic image of the front scene, and calculates the depth of field and the distance of the object through a built-in image processing algorithm to provide three-dimensional space information;

[0008] The voice acquisition module is used to collect the voice commands of the user;

[0009] The large language model module is used to generate text answers using the large language model according to the user's voice instructions and convert the text answers into voices.

[0010] The bone conduction sound transmission module is used to receive and play the voice generated by the large language model module.

[0011] The navigation path planning module is used to plan the navigation path according to the user's location and destination.

[0012] The central processing unit module is connected to each component module of the system and is used to manage the data exchange of the image acquisition module, the voice acquisition module, and the large language model module, and control other components of the system.

[0013] The wireless communication module is used for real-time interaction and processing of data with the cloud server.

[0014] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0015] 1. By providing real-time environment description and navigation services, the present invention helps visually impaired people integrate into society and live independently. At the same time, it realizes real-time environment perception and broadcast, navigation obstacle avoidance voice interaction reminder, enhancing travel safety and autonomy; integrating the large language model to achieve multi-modal interaction and adapting to various scenario services; using bone conduction headphones to meet personalized needs, ensuring comfortable travel and reducing dangerous accidents.

[0016] 2. The present invention provides functions such as navigation positioning, route guidance, and voice prompts for the blind. Using glasses as a carrier is convenient to carry, using voice wake-up and other instructions to replace button operations is simple, and the binocular depth camera has a wide viewing distance and a more accurate ranging method, with strong practicability. Brief Description of the Drawings

[0017] Figure 1 is the organizational structure diagram of the intelligent guide glasses system in the embodiment;

[0018] Figure 2 is the overall schematic diagram of the intelligent guide glasses in the embodiment;

[0019] Figure 3 is the front view of the intelligent guide glasses in the embodiment;

[0020] Figure 4 is the schematic flow diagram of the intelligent guide glasses system for navigation in the embodiment;

[0021] Explanation of the reference numerals in the drawings: 1 - depth camera; 2 - power button; 3 - alarm button; 4 - bone conduction Bluetooth headset; 5 - microphone. Detailed Embodiment

[0022] The present invention will be further described in detail below in conjunction with embodiments and the accompanying drawings, but the implementation manners of the present invention are not limited thereto.

[0023] Embodiment 1

[0024] As Figure 1 、 Figure 2 and Figure 3 shown, an intelligent blind guiding glasses system based on a multimodal large language model includes an image acquisition module, a voice acquisition module, a large language model module, a bone conduction sound transmission module, a navigation path planning module, a central processing unit module, and a wireless communication module.

[0025] The image acquisition module, specifically a binocular depth camera in this embodiment, is used to obtain a stereoscopic image of the front scene, and calculate the depth of field and the distance of the object through built-in image processing algorithms, providing three-dimensional space information.

[0026] The voice acquisition module is used to collect the voice commands of the user, including a microphone and a signal preprocessing unit; the microphone is used to capture the user's voice commands and collect voice signals; the signal preprocessing unit is used to denoise and enhance the collected voice signals, and finally transmit the preprocessed voice signals to the central processing unit module.

[0027] The large language model module includes a large language model and a voice generation unit, and is used to generate a text answer using the large language model according to the user's voice commands, and convert the text answer into voice; in this embodiment, it also supports wake word detection, and the wake word detection is modeled based on the hidden Markov model and compares the scores of sounds to detect the wake word; the voice command recognition includes multiple components of the acoustic model, and each component works together to convert the voice into text; the large language model performs the work of text understanding, which can assist the blind in communication in many aspects; at the decision-making and execution levels, the large language model also provides various supports, and voice interaction can be realized by combining voice synthesis and real-time broadcast.

[0028] The bone conduction sound transmission module is used to receive and play the voice generated by the large language model module. Specifically, it includes bone conduction Bluetooth headphones and an audio processing unit in this embodiment. The bone conduction sound transmission module is wirelessly connected to the central processing unit module; the voice generated by the large language model module is processed by the audio processing unit and then transmitted to the user's auditory nerve through the bone conduction Bluetooth headphones in a cranial conduction manner; the bone conduction method bypasses the outer ear and middle ear, directly stimulates the auditory receptors in the inner ear, achieving the purposes of avoiding noise, creating a unique auditory environment, broadening the frequency domain, assisting hearing, protecting hearing, being convenient to use, and being comfortable to wear.

[0029] The navigation path planning module is used to plan a navigation path according to the user's location and destination.

[0030] The central processing unit module is connected to each component module of the system and is used to manage the data exchange of the image acquisition module, the voice acquisition module, and the large language model module, and control other components of the system; in this embodiment, the central processing unit module includes a microprocessor, a storage unit, and an input / output control unit; the microprocessor is used to execute system control and process various input signals, and execute complex data processing and decision-making algorithms; the storage unit is used to store system programs and user data; the input / output control unit is used to manage the data exchange of the image acquisition module, the voice acquisition module, and the large language model module.

[0031] The wireless communication module is used for real-time interaction and processing of data with the cloud server; in this embodiment, the wireless communication module includes a wireless network interface, a data encryption unit, and a cloud service interface; the wireless network interface supports multiple wireless communication protocols, such as Wi-Fi, Bluetooth, and 4G / 5G, etc.; it is used to establish a connection with the external network to realize data upload and download; the data encryption unit is used to encrypt the transmitted data to ensure the security and privacy protection of the transmitted data; the cloud service interface is used to communicate with the remote cloud server to realize data synchronization and software update.

[0032] In this embodiment, the navigation path planning module includes a GPS module, a path calculation unit, and an electronic map database;

[0033] The GPS module is used to receive satellite signals in real time and locate the geographical location of the user;

[0034] The path calculation unit is used to calculate the best path according to the geographical location of the user located by the GPS module and the destination;

[0035] The electronic map database is established in advance through remote sensing mapping technology and stored in the storage unit of the central processing unit module. It includes electronic maps and detailed geographical information and is used to provide geographical information and navigation information.

[0036] In this embodiment, a power management module and an alarm module are further included;

[0037] The power management module is used to supply power to each component module of the system. In this embodiment, a power switch 2 is provided to control the power supply and shutdown; the alarm module is used to send a distress signal to a preset contact or service in case of an emergency. In this embodiment, it is specifically the alarm key 3.

[0038] Such as Figure 2 And Figure 3As shown, it is the overall schematic diagram and front view of the glasses body of the intelligent blind guide glasses system in this embodiment. In this embodiment, the binocular depth camera 1 is fixedly installed in the middle of the glasses; the bone conduction Bluetooth headset 4 is connected to the glasses leg; the microphone 5 of the voice collection module is fixedly installed on the glasses leg; the power button 2 and the alarm button 3 are set on the side glasses frame. In this embodiment, the central processing unit module specifically uses the Huawei Atlas 200 IDK A2 development board, which has rich hardware interfaces, reference codes and algorithm models, and full-process coverage of tools. It can summarize and analyze the information of each module, regulate the operation of the system, and meet the needs of multiple fields.

[0039] Embodiment 2

[0040] This embodiment provides a navigation method based on the intelligent blind guide glasses system described in Embodiment 1, as Figure 4 shown. The method includes the following steps:

[0041] S1. The binocular depth camera is turned on and images are collected. At the same time, the user's geographical location information is obtained through GPS module positioning, and the corresponding regional electronic map is called from the electronic map database according to the geographical location information; the electronic map database is established in advance through remote sensing mapping technology and stored in the storage unit of the central processing unit module;

[0042] S2. The user conducts voice interaction with the intelligent blind guide glasses system, receives and analyzes the user's voice commands, and obtains the destination information and the corresponding position of the destination in the electronic map;

[0043] S3. The intelligent blind guide glasses perform self-positioning in the electronic map based on their own position in the real environment, and determine the initial pose information and the current pose information of the wearer in the electronic map; both the initial pose and the current pose information include the corresponding position and direction.

[0044] S4. Use the A* algorithm or other path planning algorithms to plan the globally optimal path, and voice broadcast to guide the user to move forward;

[0045] S5. During the movement, the binocular depth camera collects the surrounding environment images, detects local environment changes, performs YOLOV5 recognition on the collected images, and the recognition results are input into the large language model module for language integration; performing YOLOV5 recognition on the collected images requires using a pre-trained model, and the pre-trained model is trained based on the YOLOV5 algorithm; when an obstacle is found, it is necessary to determine the specific distance position of the obstacle, and the obstacle distance is obtained through the binocular ranging algorithm; YOLOv5 combined with edge detection technology significantly improves the target recognition accuracy and can accurately recognize a variety of objects; when recognizing an obstacle, the bone conduction Bluetooth headset broadcasts the voice feedback generated by the large language model module according to the recognition result, such as "There is a car 10 meters ahead", "There is water accumulation on the floor 1m to the left front" and so on.

[0046] S6. Generate a locally optimal path. The intelligent blind guide glasses system generates a locally optimal path based on the current position, the end point of the local path, and a preset path planning algorithm.

[0047] S7. According to the obstacle recognition result, determine whether obstacle avoidance is required. If so, go to step S8; otherwise, jump to step S9.

[0048] S8. Modify the locally optimal path.

[0049] S9. The intelligent blind guide glasses system continues intelligent navigation and guides the user to move forward, and broadcasts navigation voice instructions. Determine whether the destination has been reached. If so, go to step S10; otherwise, jump to step S5.

[0050] S10. Safely reach the destination location. The binocular depth camera captures the surrounding environment images, searches for a safe location in the surrounding environment images, guides the user to reach the safe location, and then the navigation ends. The bone conduction Bluetooth headset broadcasts a voice, such as "You have reached the destination", etc.

[0051] Among them, the voice interaction between the user and the intelligent blind guide glasses system specifically includes:

[0052] S101. Audio acquisition. The voice acquisition module collects the surrounding sounds and transmits the voice signal to the signal preprocessing unit.

[0053] S102. Audio noise reduction. Using the short-time Fourier transform algorithm, the audio signal is decomposed into time-frequency signals, and different gains are applied to each frequency component for noise reduction. Specifically:

[0054] For noise reduction using the short-time Fourier transform, first divide the audio signal into short-time segments by selecting an appropriate window function and perform Fourier transform to obtain time-frequency signals. Analyze the characteristics of each frequency component in the time-frequency domain. Reduce the gain for the frequency components mainly containing noise, and appropriately adjust or maintain the gain for the frequency components containing useful information. Finally, restore the time-domain audio signal through inverse short-time Fourier transform.

[0055] S103. Audio-to-text conversion. Extract features from the audio signal after noise reduction processing, and use a trained speech recognition model to convert the extracted acoustic features into corresponding text information.

[0056] S104. Text processing and analysis. Input the converted text information into the large language model module, understand, analyze, and reason about the input text, perform coherent processing based on the context information, and give logical and accurate answers.

[0057] S105. Convert text to speech. The generated response text is input into the TTS system and converted into speech through the TTS system. Specifically, the conversion into speech through the TTS system is as follows:

[0058] First, preprocess the text, including word segmentation and prosody information annotation, to determine the pronunciation method of each word and the prosody rhythm of the speech. According to the pre-trained speech synthesis model, convert the text into a speech signal. Based on parametric synthesis, concatenative synthesis, or hybrid synthesis technology, generate a natural and fluent speech waveform by synthesizing different speech units and adjusting their acoustic parameters.

[0059] S106. Output the audio. The bone conduction sound transmission module transmits the generated speech to the user's auditory nerve through the skull.

[0060] It should also be noted that in this specification, terms such as "including", "comprising", or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device including a series of elements not only includes those elements but also includes other elements not explicitly listed, or elements inherent to such a process, method, article, or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article, or device including the said element.

[0061] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An intelligent blind guide glasses system based on a multimodal large language model, characterized in that: It includes image acquisition module, voice acquisition module, large language model module, bone conduction transmission module, navigation path planning module, central processing unit module and wireless communication module; The image acquisition module uses a binocular depth camera to obtain a stereo image of the scene in front, and calculates the depth of field and the distance of the object through the built-in image processing algorithm to provide three-dimensional spatial information; Voice collection module, used to collect user's voice commands; A large language model module is used to generate text answers based on the user's voice instructions using a large language model and convert the text answers into voice; A bone conduction sound transmission module is used to receive and play the speech generated by the large language model module; A navigation path planning module is used to plan a navigation path according to the user's location and destination; The central processing unit module is connected with each component module of the system and is used to manage the data exchange between the image acquisition module, the voice acquisition module and the large language model module, and control other components of the system; Wireless communication module, used for real-time interaction and processing of data with cloud servers.

2. The intelligent blind guide glasses system based on a multimodal large language model according to claim 1, characterized in that: The voice acquisition module includes a microphone and a signal preprocessing unit; the microphone is used to capture user voice commands and collect voice signals; the signal preprocessing unit is used to reduce noise and enhance the collected voice signals, and finally transmit the preprocessed voice signals to the central processing unit module.

3. The intelligent blind guide glasses system based on a multimodal large language model according to claim 1, characterized in that: The bone conduction sound transmission module specifically includes a bone conduction Bluetooth headset and an audio processing unit. The bone conduction sound transmission module is wirelessly connected to the central processing unit module. The voice generated by the large language model module is processed by the audio processing unit and then transmitted to the user's auditory nerve through the bone conduction Bluetooth headset using skull conduction.

4. The intelligent blind guide glasses system based on a multimodal large language model according to claim 1, characterized in that: The central processing unit module includes a microprocessor, a storage unit, and an input and output control unit; Microprocessor, used to perform system control and process various input signals, and execute complex data processing and decision-making algorithms; A storage unit for storing system programs and user data; The input and output control unit is used to manage the data exchange between the image acquisition module, the voice acquisition module and the large language model module.

5. The intelligent blind guide glasses system based on a multimodal large language model according to claim 4, characterized in that: The navigation path planning module includes a GPS module, a path calculation unit and an electronic map database; GPS module, used to receive satellite signals in real time and locate the user's geographic location; A path calculation unit, used to calculate the best path according to the user's geographical location and destination located by the GPS module; The electronic map database is pre-established through remote sensing mapping technology and stored in the storage unit of the central processing unit module, including electronic maps and geographic information, and is used to provide geographic information and navigation information.

6. The intelligent blind-guiding glasses system based on a multimodal large language model according to claim 1, characterized in that: The wireless communication module includes a wireless network interface, a data encryption unit, and a cloud service interface; Wireless network interface, supporting multiple wireless communication protocols, used to establish connection with external network to upload and download data; Data encryption unit, used to encrypt the transmitted data to ensure the security and privacy protection of the transmitted data; Cloud service interface, used to communicate with remote cloud servers to achieve data synchronization and software updates.

7. The intelligent blind-guiding glasses system based on a multimodal large language model according to claim 1, characterized in that: Also includes a power management module and an alarm module; The power management module is used to supply power to each component module of the system; the alarm module is used to send a help signal to a preset contact or service in an emergency.

8. The intelligent blind-guiding glasses system based on a multimodal large language model according to claim 5, characterized in that: The smart guide glasses system includes the following steps when navigating: S1. The binocular depth camera is turned on and collects images. At the same time, the user's geographical location information is obtained through the GPS module, and the corresponding regional electronic map is called from the electronic map database according to the geographical location information; S2, the user interacts with the smart blind guide glasses system through voice, receives and analyzes the user's voice command, and obtains the destination information and the corresponding location of the destination in the electronic map; S3, the smart guide glasses perform self-positioning in the electronic map based on the position of the wearer in the real environment, and determine the initial posture information and current posture information of the wearer in the electronic map; S4. Plan the global optimal path using the preset path planning algorithm, and guide the user to move forward through voice broadcast; S5. During the process of traveling, the binocular depth camera collects images of the surrounding environment, detects changes in the local environment, performs yolov5 recognition on the collected images, and inputs the recognition results into the large language model module for language integration; S6, generating a local optimal path. The intelligent guide glasses system generates a local optimal path based on the current position, the end point of the local path and a preset path planning algorithm; S7, determine whether obstacle avoidance is required, if yes, go to step S8, otherwise, jump to step S9; S8, correct the local optimal path; S9, the smart guide glasses system continues to perform smart navigation and voice broadcast to guide the user to move forward; determine whether the destination has been reached, if so, proceed to step S10, otherwise jump to step S5; S10, upon reaching the destination, the binocular depth camera collects images of the surrounding environment, searches for a safe location in the images of the surrounding environment, and guides the user to the safe location, and then the navigation ends.

9. The intelligent blind-guiding glasses system based on a multimodal large language model according to claim 8, characterized in that: The voice interaction between the user and the smart guide glasses system includes: Audio acquisition: The voice acquisition module collects surrounding sounds and transmits the voice signal to the signal preprocessing unit; Audio noise reduction uses the short-time Fourier transform algorithm to decompose the audio signal into time-frequency signals and apply different gains to each frequency component to reduce noise; Audio to text conversion: extract features from the audio signal after noise reduction processing, and use the trained speech recognition model to convert the extracted acoustic features into corresponding text information; Text processing and analysis: input the converted text information into the large language model module, understand, analyze and reason the input text, process it coherently based on the context information, and give a logical and accurate answer; Text-to-speech: the generated answer text is input into the TTS system and converted into speech through the TTS system; Output audio, the bone conduction sound transmission module transmits the generated voice to the user's auditory nerve through the skull.

10. The intelligent blind-guiding glasses system based on a multimodal large language model according to claim 8, characterized in that: Audio noise reduction is specifically: Using short-time Fourier transform to reduce noise, first divide the audio signal into short time segments by selecting a suitable window function and perform Fourier transform to obtain a time-frequency signal. Then analyze the characteristics of each frequency component in the time-frequency domain, reduce the gain of the frequency component containing noise, adjust or maintain the gain of the frequency component containing useful information, and finally restore the time-domain audio signal through inverse short-time Fourier transform. Converted into speech through the TTS system, specifically: First, the text is preprocessed, including word segmentation and prosodic information annotation to determine the pronunciation of each word and the rhythmic rhythm of the speech; the text is converted into a speech signal based on a pre-trained speech synthesis model; based on parameter synthesis, splicing synthesis or hybrid synthesis technology, a speech waveform is generated by synthesizing different speech units and adjusting their acoustic parameters.

Images