Blind person intelligent travel auxiliary system and method based on multi-mode perception

Abstract

The invention discloses a multi-modal perception-based intelligent travel auxiliary system and method for blind people, and the system and method take a smart phone as a carrier, and shoot the surrounding environment and obstacles in real time through a camera shooting function. And the AI multi-target sensing module identifies and analyzes the image, and carries out dynamic path planning by fusing GPS / Beidou positioning and a high-precision map. And finally, voice graded broadcasting and vibration reminding are carried out through the multi-mode interaction module. Therefore, the purpose of avoiding obstacles is achieved, and the travel difficulty of the blind is reduced. On the basis, the functions of volunteers and the guiding function of related traffic facilities such as bus stations, subway stations, railway stations, high-speed railway stations and airports can be expanded. In addition, inertial navigation is automatically switched to when no network exists, so that the blind person is continuously assisted to go out.

Description

Technical Field

[0001] This patent belongs to the field of artificial intelligence and barrier-free assistance technology, and specifically involves a travel assistance system for the blind that combines computer vision, voice interaction and satellite positioning technology. Technical Background

[0002] Blind people rely on canes, guide dogs, or simple electronic devices (such as ultrasonic obstacle avoiders) for daily travel, but existing technologies have the following drawbacks: 1. Weak environmental perception capabilities: Traditional devices are unable to recognize complex road conditions (such as steps, potholes, and moving obstacles) and semantic information (such as traffic lights and traffic signs).

[0003] 2. Lack of real-time performance: Unable to dynamically analyze obstacle distance, direction and movement trend.

[0004] 3. Navigation and obstacle avoidance are separated: Existing navigation tools (such as GPS) are not integrated with real-time obstacle avoidance, making it difficult to cope with sudden obstacles. Summary of the Invention

[0005] 1. Technical issues: This solution reduces hardware costs through the ubiquity of smartphones, and combines cutting-edge AI algorithms with humanized interactive design to effectively solve the core pain points of independent travel for the blind.

[0006] 2 technical solutions: 1. Hardware carrier Smartphone (built-in camera, GPS / Beidou module, gyroscope, accelerometer, microphone and speaker).

[0007] 2. Software functional modules Image acquisition and preprocessing module Use the mobile phone camera to shoot the road conditions ahead in real time (video stream).

[0008] Image enhancement: low-light enhancement, motion blur correction.

[0009] AI multi-target perception module Lightweight deep learning models (such as the MobileNet-YOLO hybrid architecture): Using the improved YOLOv5s architecture, the SPPF layer is replaced with a depth-wise separable convolution. After migration training on the COCO dataset, the number of model parameters is reduced from 7.2M to 4.8M, and the mAP remains at 62.1.

[0010] By using an extended Kalman filter to achieve tight coupling between the IMU and visual SLAM, the positioning error is controlled within 1.2 meters in 30 seconds of dead reckoning (a 40% reduction compared to loose coupling).

[0011] Obstacle detection (pedestrians, vehicles, steps, potholes, railings, etc.).

[0012] Semantic segmentation: Identify ground types (blind path, grass, gravel, etc.).

[0013] Depth estimation: Obstacle distance is calculated by fusing a monocular camera and an IMU sensor (error < 0.5 meters).

[0014] Dynamic path planning module Positioning mechanism: Dual redundant positioning is used, and when the satellite signal strength is lower than -130dBm, it automatically switches to WiFi fingerprint positioning, with a success rate of ≥92%. Path optimization: Based on the potential field method, set the obstacle repulsion coefficient α∈[0.5,1.2] and the target point attraction coefficient β∈[0.8,1.5] Integrate GPS / Beidou positioning with high-precision maps (such as OpenStreetMap blind path data).

[0015] Real-time obstacle avoidance algorithm: dynamically adjusts the navigation path according to the location of obstacles.

[0016] Multimodal interaction module Text-to-speech (TTS): Prioritized announcements (e.g., "There are stairs 2 meters ahead, please turn right").

[0017] Automatic speech recognition (ASR): Supports natural language commands (such as "find the nearest subway station").

[0018] Vibration feedback: Early warning of approaching urgent obstacles (such as moving vehicles).

[0019] 4 Implementation Process 1. Startup phase The user wakes up the device with voice: "Start navigation."

[0020] The system automatically loads preset destinations or confirms new destinations through dialogue.

[0021] 2. On the move The camera samples at 10fps, and the AI model processes each frame in ≤50ms.

[0022] Obstacles are classified by danger level: Level 1 (Emergency): Moving vehicles, hanging objects → immediate voice alarm + continuous vibration.

[0023] Level 2 (static): Stairs, roadblocks → voice prompt 3 meters in advance.

[0024] Navigation instructions and obstacle avoidance instructions are broadcast in layers (e.g., "Go straight for 20 meters and then turn left. Be careful, there is water on the right.").

[0025] 3. Exception handling When the signal is lost, it switches to inertial navigation (dead reckoning) and a voice prompt "Recovering positioning" is given. WiFi fingerprint positioning is added as a backup solution, and floor identification is achieved through RSSI matching. DETAILED DESCRIPTION Example

[0026] On the MediaTek Dimensity 9000 platform, when inputting a 640×480 resolution video stream: The model quantization module automatically switches to INT8 mode, reducing memory usage from 78MB to 37MB.

[0027] The inference speed is increased from 15fps to 28fps, and the end-to-end latency is ≤105ms.

[0028] The power consumption test shows that the depth estimation module consumes 1.3W of power on the Snapdragon 888 platform (2.1W for the unoptimized version). Example

[0029] Field test data in Jing'an District, Shanghai: Moving obstacle warning distance: 6.5 meters (compared to 3.2 meters for SonicGuide ultrasonic equipment) False alarm rate: 7% (compared to 18% for similar devices) Reduce user cognitive load by 37% (NASA-TLX scale evaluation).

Claims

1. A travel assistance system for the blind, characterized in that include: Smartphone carrier (including camera, satellite positioning module, IMU sensor). Multi-sensor synchronization module: aligns IMU and camera data timestamps through hardware clock interrupts.

2. The system according to claim 1, wherein The satellite positioning module includes GPS, BeiDou, and GLONASS multi-system solution capabilities. A method for assisting blind people with travel based on the system of claim 1, characterized by comprising: implementing obstacle detection by improving the YOLOv5s model (parameter count ≤ 5M); dynamically planning a path using the potential field method, setting the repulsion coefficient α∈[0.5,1.2] and the attraction coefficient β∈[0.8,1.5].

3. The method according to claim 2, wherein: By optimizing matrix operations through the ARM NEON instruction set, the power consumption of depth estimation tasks is reduced to ≤1.5W.

4. The system according to claim 1, characterized in that Contains a computer-readable storage medium storing program code for executing obstacle avoidance path generation.