Intelligent blind guiding device and method based on terminal epitaxy

Abstract

The invention discloses an intelligent blind guiding device and method based on terminal epitaxy, the intelligent blind guiding device comprises a wearing module and an intelligent mobile phone, the wearing module comprises an earphone, a microphone, a camera, a vibration unit and a connection assembly, and the intelligent mobile phone comprises a GNSS sensor, a network communication assembly, a calculation chip and a blind person navigation APP; the intelligent blind guiding method comprises the following steps: acquiring environment video data when a blind person walks through a camera, receiving and processing the environment video data through a smart phone, carrying out environment perception and path planning, and playing navigation information and feedback information to the blind person through an earphone; according to the invention, the idle computing power of the smart phone is utilized, so that the dependence of the traditional blind-assisting equipment on high-cost independent integrated equipment and multiple sensors is reduced, the cost of the blind-person travel-assisting navigation system is greatly reduced, and the blind-person travel-assisting navigation system is easier to popularize and use.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and barrier-free assistance technology, and particularly relates to an intelligent blind guiding device and method based on terminal extension. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and sensor technology, wearable visual blind assistance devices have gradually become an important tool to assist visually impaired people in traveling independently. Many manufacturers at home and abroad have invested in research and development and launched a series of integrated products. For example, abroad, there are products such as OrCam MyEye of OrCam Company and SUNU BAND of Ustraap Company. Domestically, there are blind visual assistance glasses of Hangzhou Shike Technology and "Angel Eye" blind guiding glasses of Shanghai Zhaoguan Electronics. Most of these products have functions such as obstacle recognition, travel navigation, and text recognition, providing varying degrees of help to visually impaired people.

[0003] The existing technology mainly relies on multi-sensor fusion technologies such as GNSS positioning, IMU sensors, and lidar sensors, as well as indoor microwave radio frequency identification devices for environmental perception and positioning. For example, in the "Wearable Blind Person Assistance Navigation Device and Method Based on Semantic VISLAM and GNSS Positioning" disclosed in CN114527951A, a SLAM method that fuses multiple sensors is proposed. By fusing the data of multiple sensors, the accuracy of indoor and outdoor positioning is ensured, and semantic SLAM is combined to help blind people perceive the surrounding environment and feedback the information to blind people.

[0004] Although the existing technology has made certain progress in assisting visually impaired people in traveling, due to the dependence on precision sensors such as lidar sensors, millimeter-wave radar sensors, and inertial navigation units, as well as the high requirements for the computing power of terminal devices, the existing blind guiding devices are costly, which limits the acquisition and use of such devices by visually impaired people. Summary of the Invention

[0005] Object of the Invention: The object of the present invention is to provide an intelligent blind guiding device based on terminal extension that reduces the cost of the blind person travel assistance navigation system. On the other hand, an intelligent blind guiding method based on terminal extension is provided.

[0006] Technical solution: The intelligent blind guiding device based on terminal extension according to the present invention includes a wearable module and a smart phone. The wearable module includes a camera for collecting the environmental status when a blind person walks, a headset for playing navigation information and feedback information to the blind person, a vibration unit for providing vibration feedback to the blind person, and a microphone for collecting the blind person's instructions. The smart phone includes a GNSS sensor, a network communication component, a computing chip, and a blind person navigation APP. The wearable module is connected to the smart phone through a connection component to achieve data transmission and power supply. The smart phone exchanges data with a cloud server through the network communication component to process complex user instructions and requests.

[0007] Through the above technical solution, the combination of the wearable module and the smart phone realizes a low-cost and efficient blind person travel assistance navigation system. The wearable module is connected to the smart phone to achieve data transmission and power supply. The smart phone uses its idle computing power to process tasks such as environmental perception and path planning, and exchanges data with the cloud server through the network communication component to process complex instructions. This solution significantly reduces the system cost and makes full use of the hardware resources of the smart phone, realizing real-time data transmission and processing, and providing a convenient travel assistance function for the blind.

[0008] Preferably, the connection component is used to connect each component in the wearable module, and according to the different numbers of interfaces of the smart phone, one of the following two connection methods is adopted:

[0009] For a smart phone with multiple interfaces, the headset, microphone, vibration unit, and camera in the wearable module are directly connected to the smart phone respectively;

[0010] For a smart phone with only one data interface, the headset, microphone, vibration unit, and camera in the wearable module are centrally connected to the smart phone through the connection component.

[0011] Through the above technical solution, the flexible design of the connection component adapts to smart phones with different numbers of interfaces. This design significantly improves the compatibility and versatility of the system, ensures that the device can adapt to most smart phones on the market, simplifies the hardware connection method, reduces the usage complexity, and provides a more convenient and flexible travel assistance solution for the blind.

[0012] The intelligent blind guiding method based on terminal extension according to the present invention includes the following steps:

[0013] (1) Collect environmental video data when a blind person walks through the camera;

[0014] (2) The smart phone receives and processes the environmental video data for environmental perception and path planning;

[0015] (3) Provide navigation information and feedback information to the blind through the earphone and the vibration unit, and the feedback information includes pulse sound feedback, voice feedback and vibration feedback.

[0016] Through the above technical solution, the real-time perception and navigation of the walking environment of the blind are realized, an efficient and accurate travel assistance function is provided, and the safety and convenience of the blind travel are significantly improved.

[0017] Preferably, the processing of the environmental video data in step 2 includes:

[0018] (21) Use image processing algorithms to perform semantic segmentation, object detection and monocular depth estimation on the video frames of the environmental video data. Considering the particularity of the blind path, an independent segmentation operation needs to be performed on the blind path during semantic segmentation;

[0019] (22) Perform OCR processing on the detected text blocks to extract the text information in the environment;

[0020] (23) Synthesize a bird's-eye view based on the semantic segmentation result and the monocular depth estimation result, and the bird's-eye view contains the environmental information within 30-50 meters in front of the blind.

[0021] Through the above technical solution, semantic segmentation, object detection and monocular depth estimation, especially an independent segmentation operation for the blind path, ensure the accurate extraction of blind path information; at the same time, OCR processing further enriches the details of environmental perception; the generation of the bird's-eye view provides comprehensive and accurate environmental data support for path planning and navigation. This technical solution significantly improves the accuracy and practicality of environmental perception and provides a more reliable and efficient assistance function for the blind to travel.

[0022] Preferably, the semantic segmentation and object detection use a shared backbone network and adopt independent decoding output heads to give semantic segmentation and object detection information; the monocular depth estimation adopts a cross-domain training method to obtain a depth estimation result with high stability in the real environment.

[0023] Through the above technical solution, the commonality of the underlying features is fully utilized, the repeated consumption of computing resources is reduced, and at the same time, the accuracy and independence of the semantic segmentation and object detection tasks are ensured through independent decoding heads. This technical solution significantly improves the efficiency and performance of the algorithm; the cross-domain training method effectively improves the robustness and generalization ability of depth estimation, ensuring that the system can provide accurate environmental depth information in different scenarios and providing more reliable support for the blind travel assistance navigation.

[0024] Preferably, the path planning in step 2 includes:

[0025] (24) Obtain the real-time positioning information of the blind through the GNSS sensor built in the smartphone module;

[0026] (25) Combine the road information at the street level provided by the navigation map platform as prior knowledge to minimally constrain the bird's-eye view result to correct the errors in visual perception, and use the navigation map for path planning at the road level to generate a global route from the current location to the target location;

[0027] (26) Generate a passable local path based on the obstacle distribution, blind path information, and environmental semantic information in the bird's-eye view;

[0028] (27) Dynamically adjust the path to avoid obstacles detected in real time, and combine the walking speed and direction of the blind person to generate specific walking guidance information, including turning angles, step suggestions, and obstacle avoidance tips.

[0029] Through the above technical solutions, the multi-level and dynamic path planning mechanism not only improves the accuracy and adaptability of path planning, but also significantly enhances the safety and navigation efficiency of the blind person's travel. By adjusting the path in real time and providing detailed walking guidance, the system can effectively cope with the complex and changeable travel environment and provide more reliable and intelligent travel assistance support for the blind.

[0030] Preferably, the pulse sound feedback described in step 3 includes:

[0031] (31) Play a synthesized pulse sound wave to the blind person through headphones. The pulse sound wave contains the moving direction and speed information of the current scene. The pulse sound wave is a two-channel stereo audio, and the left and right channels are respectively composed of ADSR envelope pulse sounds with variable periods, which are used to indicate the walking direction of the blind person and the position of obstacles;

[0032] (32) Dynamically adjust the frequency and intensity of the pulse sound wave according to the relative distance and movement trend between the blind person and the obstacle.

[0033] Through the above technical solutions, it can clearly and intuitively indicate the walking direction and changes in the surrounding environment; in addition, dynamically adjusting the frequency and intensity of the pulse sound wave makes the feedback information more accurate and real-time. This pulse sound feedback mechanism not only provides an efficient information transmission method, but also enhances the blind person's perception ability of the environment, enabling them to quickly and accurately understand the changes in the surrounding environment, thus significantly improving the safety and interaction experience of the blind person's travel; at the same time, the pulse sound feedback does not require additional hardware support, further reducing the system cost and improving the portability and practicality of the device.

[0034] Preferably, the voice feedback described in step 3 includes playing synthesized voice information to the blind through headphones, and the voice information includes real-time voice and non-real-time voice. The real-time voice is used for collision reminders in emergency situations, road-level direction broadcasts, and prompts at traffic light intersections. The non-real-time voice initiates a request to the cloud computing platform according to user instructions to obtain and play the synthesized voice.

[0035] Preferably, the vibration feedback described in step 3 includes, when obstacle detection is triggered, causing the vibration unit in the corresponding direction to generate a vibration prompt with a variable frequency according to the distance, direction and type of the obstacle.

[0036] Through the above technical solutions, voice feedback and vibration feedback mechanisms provide multi-level interactive support for the blind; real-time feedback can timely broadcast key information in emergency situations (such as collision risks, road turns or traffic light intersections), and further remind the blind of road conditions through vibration feedback, ensuring that the blind can quickly respond to environmental changes, significantly improving travel safety. Non-real-time feedback processes complex requests (such as environmental descriptions or answers to specific questions) through the cloud computing platform according to user instructions, and provides personalized voice feedback, enhancing the intelligence and practicality of the system; this hierarchical feedback mechanism not only meets the diverse needs of the blind in different scenarios, but also optimizes the efficiency and accuracy of information transmission through a combination of real-time and offline interaction, further improving the convenience of blind travel and user experience; at the same time, the combination of voice feedback, pulse sound feedback and vibration feedback forms a multi-modal interactive system, providing more comprehensive and intuitive environmental perception support for the blind.

[0037] Preferably, the intelligent blind guiding method also includes the user issuing voice commands through a microphone, the smart phone parsing the commands and executing corresponding operations, and when the commands need to be processed by the cloud platform, the smart phone uploads relevant data to the cloud server and obtains response information to play through headphones.

[0038] Through the above technical solutions, not only the intelligence level of the system is enhanced, but also efficient processing of complex commands is achieved, providing users with more personalized and accurate services; at the same time, the voice command interaction function simplifies the user operation process, improves the system's usability and interactive experience, and enables blind people to obtain the required information or perform specific tasks more conveniently, further improving the practicality of the travel assistance system and user satisfaction.

[0039] Beneficial effects: Compared with the prior art, the present invention has the following remarkable advantages: 1. By utilizing the idle computing power of a smart phone, it reduces the dependence on high-cost independent integrated devices and multiple sensors in traditional blind assistance devices, significantly reducing the cost of the blind travel assistance navigation system and making it easier to popularize and use; 2. It realizes high-precision and high-stability visual perception, ensuring the safety and reliability of blind people's travel and breaking through the limitations of traditional blind assistance devices with limited scenarios; 3. It realizes the applicability of the full-life travel scenarios outdoors, covering diverse travel needs from indoor to outdoor and from short distances to long distances; 4. Vibration feedback, voice feedback, and pulsed sound waves are used as real-time interaction means, which not only reduces the device cost but also improves safety. Brief Description of the Drawings

[0040] Figure 1 It is a schematic structural diagram of the intelligent blind guiding device of the present invention;

[0041] Figure 2 It is a schematic connection diagram of a smart phone with multiple interfaces and a wearable module of the present invention;

[0042] Figure 3 It is a schematic connection diagram of a smart phone with only one data interface and a wearable module of the present invention;

[0043] Figure 4 It is a flowchart of the intelligent blind guiding method of the present invention. Detailed Embodiments

[0044] The technical solutions of the present invention will be further described below with reference to the accompanying drawings.

[0045] An intelligent blind guiding device and method based on terminal extension provided by the present invention aims to reduce the cost of the blind travel assistance navigation system and at the same time provide efficient and accurate navigation services. As Figure 1 shown, the device includes a wearable module and a smart phone. The wearable module integrates a camera, earphones, a microphone, a vibration unit, and a connection component. The camera is used to collect the environmental state when the blind person walks, which can be a monocular camera or a binocular camera. The earphones are used to play feedback and navigation information to the blind person and interact with the blind person. The microphone is used to collect the blind person's instructions. The vibration unit is used to provide high-timeliness feedback to the blind person in a vibrating manner. The connection component is used to connect each module in the wearable module and connect to the smart phone, such as a USB hub, a Type-C hub, or a wired + Bluetooth hybrid, etc.; The smart phone is built-in with a GNSS sensor, a network communication component, a computing chip, and a blind navigation APP, which is responsible for receiving the data transmitted back by the sensors in the wearable module and driving the earphone module to play relevant feedback audio; The wearable module is connected to the smart phone through USB to achieve data transmission and power supply. The smart phone exchanges data with the cloud server through the network communication component to process complex user instructions and requests.

[0046] According to the different numbers of interfaces of smart phones, the connection methods of the wearable module are divided into two types. As Figure 2 shown, for smart phones with multiple interfaces, the earphone 1, microphone 2, vibration unit 3 and camera 4 in the wearable module are directly connected to the smart phone 5 respectively; as Figure 3 shown, for smart phones with only one data interface, the earphone 1, microphone 2, vibration unit 3 and camera 4 in the wearable module are centrally connected to the smart phone 5 through the connection component 6. This design ensures that the device can be adapted to most smart phones on the market, improving the versatility and flexibility of the system.

[0047] As Figure 4 shown, the intelligent blind guidance method of the present invention includes the following steps: after obtaining the data transmitted back by the sensor, first collect the environmental video data of the blind person walking through the camera; then the smart phone receives and processes the environmental video data for environmental perception and path planning; finally, play the navigation information and feedback information to the blind person through the earphone and vibration unit, including pulse sound feedback, vibration feedback and voice feedback. This process ensures that the blind person can obtain the information of the surrounding environment and navigation guidance in real time.

[0048] During the processing of the environmental video data, the smart phone uses image processing algorithms (such as neural network models) to perform semantic segmentation, object detection and monocular depth estimation on the video frames of the environmental video data. Considering the particularity of the blind path, an independent segmentation operation needs to be performed on the blind path during semantic segmentation; in addition, the system will also perform OCR processing on the detected text blocks to extract the text information in the environment; based on the semantic segmentation results and monocular depth estimation results, the system synthesizes a bird's-eye view, which contains the environmental information within 30-50 meters in front of the blind person. The specific range depends on the camera's shooting angle, visible distance and environmental complexity.

[0049] Path planning is an important part of the present invention. The GNSS sensor built into the smart phone module can obtain the positioning information of the blind person in real time. Combining the road information at the street level provided by the navigation map platform, the system minimally constrains the bird's-eye view results to correct the errors of visual perception, and uses the navigation map to perform path planning at the road level to generate a macroscopic route plan from the current position to the target location; in addition, the system will also perform fine-grained path planning according to the obstacle distribution, blind path information and environmental semantic information in the bird's-eye view to generate a passable local path; during the walking process of the blind person, the system will dynamically adjust the path to avoid the obstacles detected in real time, and combine the walking speed and direction of the blind person to generate a specific walking path, including steering angle, step suggestion and obstacle avoidance prompt.

[0050] Feedback information is the key to the interaction between the blind and the system. The present invention adopts three modes: pulse sound feedback, vibration feedback and voice feedback. Pulse sound feedback plays a synthesized pulse sound wave to the blind through headphones. The sound wave contains the moving direction and speed information of the current scene. The stereo audio is dual-channel, and the left and right channels are respectively composed of ADSR envelope pulse sounds with variable periods, which are used to indicate the blind's direction of travel and the location of obstacles. Vibration feedback is based on pulse sound feedback. According to the direction, distance and specific type of the obstacle, the vibration unit in the corresponding direction generates a vibration prompt with variable frequency. Voice feedback is divided into real-time voice and non-real-time voice. Real-time voice is used for collision reminders in emergency situations, road-level direction broadcasts and prompts at traffic light intersections. Non-real-time voice initiates a request to the cloud computing platform according to user instructions to obtain and play synthesized voice.

[0051] Users can issue voice commands through the microphone, and the smartphone will interpret the commands and perform corresponding operations. When the command needs to be processed by the cloud platform, the smartphone will upload the relevant data to the cloud server and obtain the response information and play it through the headset. This design enables the system to handle complex user requests and provide more personalized and intelligent navigation services.

[0052] In order to better illustrate the implementation mode of the present invention, a specific example is described below.

[0053] This embodiment uses a Xiaomi 14 mobile phone, an 88° field of view distortion-free UVC camera and headphones to form a system. The UVC camera and headphones are directly connected to the mobile phone through a Type-C splitter, and the mobile phone directly powers and drives the camera and headphones; among them, the total cost of the UVC camera and headphones is RMB 202; after the deployment is completed, the user can wear it directly. During use, the user will stably clamp the camera at the collar of the top to ensure that the camera has no large shake and is not blocked by hair or clothing.

[0054] In this embodiment, the algorithm deployed in the mobile phone performed an image segmentation accuracy test on the ADE20K dataset, and obtained an mIoU (mean intersection over union) of 42.1, with an inference speed of 15FPS (frames per second). The accuracy of the target detection algorithm was tested on the ms-COCO dataset, and an mAP (average precision) of 44.9 was obtained, with an inference speed of 128 milliseconds. In addition, in the 150-meter reporting test within the preset road section, the test route included 2 route turns and 5 rising / falling edges of the road surface. In three tests, the algorithm successfully reported all 6 turns, 14 out of 15 road edges, and 1 in advance. Combining the field experimental data with the dataset test data, the method in this embodiment demonstrates sufficient robustness and accuracy, and can provide users with safe travel navigation in the case of visual impairment.

[0055] Through the detailed description of the above specific embodiments, it can be seen that the present invention is innovative and practical in terms of hardware deployment, software implementation, algorithm processing, etc., and can effectively solve the problem of blind people's travel navigation, improving the safety and convenience of blind people's travel.

Claims

1. An intelligent blind guide device based on terminal extension, comprising a wearable module, wherein the wearable module comprises an earphone for playing navigation information and feedback information to the blind, a microphone for collecting instructions of the blind, a vibration unit for providing vibration feedback to the blind, and a camera for collecting the environmental status of the blind when walking, characterized in that: It also includes a smart phone, which includes a GNSS sensor, a network communication component, a computing chip and a blind navigation APP. The wearable module is connected to the smart phone through a connecting component to achieve data transmission and power supply. The smart phone exchanges data with the cloud server through the network communication component to process user instructions and requests.

2. The intelligent blind guiding device according to claim 1, characterized in that: The connection component is used to connect the various components in the wearable module, and adopts one of the following two connection methods according to the number of smartphone interfaces: For a smartphone with multiple interfaces, the earphone, microphone, vibration unit and camera in the wearable module are directly connected to the smartphone respectively; For a smartphone with only one data interface, the earphone, microphone, vibration unit and camera in the wearable module are centrally connected to the smartphone via a connection component.

3. An intelligent blind-guiding method based on terminal extension, characterized in that: The following steps are involved: (1) Collecting video data of the environment when the blind person is walking through the camera; (2) The smartphone receives and processes environmental video data to perform environmental perception and path planning; (3) Providing navigation information and feedback information to the blind through headphones and a vibration unit, wherein the feedback information includes pulse sound feedback, voice feedback, and vibration feedback.

4. The intelligent blind guiding method according to claim 3, characterized in that: The processing environment video data described in step 2 includes: (21) Use image processing algorithms to perform semantic segmentation, target detection, and monocular depth estimation on the video frames of the environmental video data. Considering the particularity of the blind path, an independent segmentation operation of the blind path is also required during semantic segmentation; (22) performing OCR processing on the detected text block to extract text information in the environment; (23) A bird's-eye view image is synthesized based on the semantic segmentation results and the monocular depth estimation results, where the bird's-eye view image contains environmental information within a range of 30-50 meters in front of the blind person.

5. The intelligent blind guiding method according to claim 4, characterized in that: The semantic segmentation and target detection use a shared backbone network and adopt an independent decoding output head to provide semantic segmentation and target detection information; the monocular depth estimation adopts a cross-domain training method to obtain a highly stable depth estimation result in a real environment.

6. The intelligent blind guiding method according to claim 3, characterized in that: The path planning described in step 2 includes: (24) Obtaining real-time positioning information of the blind through the built-in GNSS sensor of the smartphone module; (25) Combined with the street-level road information provided by the navigation map platform, as prior knowledge, the bird's-eye view results are constrained to a minimum extent to correct the errors of visual perception, and the navigation map is used for road-level path planning to generate a global route from the current location to the target location; (26) Generate a navigable local path based on the obstacle distribution, blind path information, and environmental semantic information in the bird's-eye view; (27) The path is dynamically adjusted to avoid obstacles detected in real time, and specific walking guidance information is generated based on the blind person’s walking speed and direction, including turning angle, step count recommendations, and obstacle avoidance prompts.

7. The intelligent blind guiding method according to claim 3, characterized in that: The pulse sound feedback described in step 3 includes: (31) playing a synthesized pulse sound wave to the blind person through headphones, wherein the pulse sound wave contains the moving direction and speed information of the current scene, and the pulse sound wave is a two-channel stereo audio, wherein the left and right channels are respectively composed of ADSR envelope pulse sounds with variable periods, and are used to indicate the moving direction of the blind person and the location of obstacles; (32) Dynamically adjust the frequency and intensity of the pulse sound waves according to the relative distance and movement trend between the blind person and the obstacle.

8. The intelligent blind guiding method according to claim 3, characterized in that: The voice feedback described in step 3 includes playing synthesized voice information to the blind through headphones, and the voice information includes real-time voice and non-real-time voice. The real-time voice is used for collision reminders in emergency situations, road-level direction broadcasts, and prompts at traffic light intersections. The non-real-time voice initiates a request to the cloud computing platform according to user instructions to obtain and play the synthesized voice.

9. The intelligent blind guiding method according to claim 3, characterized in that: The vibration feedback described in step 3 includes, when obstacle detection is triggered, causing the vibration unit in the corresponding direction to generate a vibration prompt with a variable frequency according to the distance, direction and type of the obstacle.

10. The intelligent blind guiding method according to claim 3, characterized in that: It also includes the user issuing voice commands through a microphone, the smartphone parsing the commands and executing corresponding operations. When the commands need to be processed by the cloud platform, the smartphone uploads the relevant data to the cloud server and obtains the response information and plays it through the headphones.

Images