Intelligent navigation system for blind people based on multi-modal sensor fusion

Abstract

The application discloses a blind person intelligent navigation system based on multi-modal sensor fusion and relates to the field of blind person intelligent navigation systems, which comprises a navigation system of initialization modules, data acquisition modules, data processing modules, first judgment modules, second judgment modules, path correction modules and navigation prompt modules. The initialization modules generate an initial path according to a starting position and a target position and start a path guidance mode; the data acquisition modules synchronously acquire image, radar and inertial measurement data; the data processing modules calculate image continuity indexes and generate auxiliary judgment marks; the first judgment modules determine whether to switch an obstacle avoidance mode according to index continuity and auxiliary marks; the second judgment modules evaluate the conditions for returning path guidance in the obstacle avoidance mode; the path correction modules judge path correction modes through spatial relations and deviation coefficients; and the navigation prompt modules output guidance information according to modes. The application can improve the accuracy and stability of navigation mode switching.

Description

Technical Field

[0001] This invention relates to the field of intelligent navigation for the blind, specifically to an intelligent navigation system for the blind based on multimodal sensor fusion. Background Technology

[0002] With the rapid development of multimodal sensor technology, path planning algorithms, and human-computer interaction methods, intelligent navigation technology to assist visually impaired individuals in their travel has become a research hotspot. Current navigation systems for the blind are mainly built upon functional modules such as voice recognition, path guidance, and obstacle avoidance prompts, aiming to provide blind users with autonomous and safe travel assistance. Some advanced systems have attempted to integrate multiple sensors, including electronic maps, cameras, radar, and inertial measurement units, to achieve comprehensive functions such as environmental perception, path navigation, and dynamic obstacle avoidance.

[0003] However, existing navigation systems for the blind still face numerous technical bottlenecks in practical applications. First, regarding path guidance, most systems employ static path planning or rule-based navigation strategies. When encountering sudden obstacles or when the user unexpectedly deviates from the path, the system lacks an effective judgment mechanism to decide whether to switch navigation modes, leading to response delays or frequent error messages. Second, although some systems have adopted multimodal sensing data, an intelligent switching mechanism based on image coherence and multi-source sensor state collaborative analysis has not yet been established, making the system's transition between path guidance and active obstacle avoidance modes unstable and unreliable. Furthermore, when the user deviates from the preset path, existing solutions often simply replan the path, lacking precise quantitative assessment of the degree of deviation and a gradual path correction strategy, easily causing frequent changes in navigation instructions or guidance failures.

[0004] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an intelligent navigation system for the blind based on multimodal sensor fusion.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] In a first aspect, the present invention discloses an intelligent navigation system for the blind based on multimodal sensor fusion, comprising:

[0008] An initialization module is used to generate an initial path based on the user's starting and target locations, and to perform navigation using a path-guided navigation mode based on the initial path.

[0009] The data acquisition module is used to acquire image frame data, radar reflection data, and inertial measurement data of the user's current location;

[0010] The data processing module is used to calculate the image coherence index based on the image frame data, and generate auxiliary judgment flags based on the radar reflection data and inertial measurement data respectively.

[0011] The first judgment module is used to determine whether the image continuity index is within a preset continuous range. If it is, the path guidance navigation mode is maintained; otherwise, it further determines whether the auxiliary judgment flag meets the first switching condition. If it is, the mode is switched to active obstacle avoidance navigation mode; otherwise, the path guidance navigation mode is maintained.

[0012] The second judgment module is used to determine whether the image continuity index and auxiliary judgment flag meet the second switching condition when the system is in active obstacle avoidance navigation mode. If yes, it switches back to path guidance navigation mode; otherwise, it maintains active obstacle avoidance navigation mode.

[0013] The path correction module is used to determine whether the user has deviated from the initial path based on the spatial relationship between the user's current location and the initial path when the system switches back to the path guidance navigation mode. If so, the path guidance navigation mode based on the initial path is continued. Otherwise, the minimum deviation coefficient from the initial path is calculated, and it is determined whether the deviation coefficient exceeds a preset deviation threshold. If so, a new path is generated based on the current location and the target location and the initial path is replaced. Otherwise, a transition path is generated between the current location and the initial path.

[0014] The navigation prompt module is used to generate corresponding navigation prompts based on the navigation mode to guide the user's movement.

[0015] Secondly, this invention discloses an intelligent navigation method for the blind based on multimodal sensor fusion, comprising the following steps:

[0016] An initial path is generated based on the user's starting and destination locations, and navigation is performed using a path-guided navigation mode based on the initial path.

[0017] Acquire image frame data, radar reflection data, and inertial measurement data of the user's current location;

[0018] The image coherence index is calculated based on the image frame data, and auxiliary judgment indicators are generated based on the radar reflection data and inertial measurement data, respectively.

[0019] Determine whether the image continuity index is within a preset continuous range. If yes, maintain the path guidance navigation mode; otherwise, further determine whether the auxiliary judgment flag meets the first switching condition. If yes, switch to active obstacle avoidance navigation mode; otherwise, maintain the path guidance navigation mode.

[0020] When in active obstacle avoidance navigation mode, determine whether the image continuity index and auxiliary judgment flag meet the second switching condition. If yes, switch back to path guidance navigation mode; otherwise, maintain active obstacle avoidance navigation mode.

[0021] When switching back to the path guidance navigation mode, the system determines whether the user has deviated from the initial path based on the spatial relationship between the user's current location and the initial path. If so, the path guidance navigation mode based on the initial path continues to be used. Otherwise, the minimum deviation coefficient from the initial path is further calculated, and it is determined whether the deviation coefficient exceeds the preset deviation threshold. If so, a new path is generated based on the current location and the target location and the initial path is replaced. Otherwise, a transition path is generated between the current location and the initial path.

[0022] Based on the navigation mode, corresponding navigation prompts are generated to guide the user's movement.

[0023] The beneficial effects of this invention are as follows:

[0024] 1. By fusing image coherence indicators with auxiliary judgment markers generated based on millimeter-wave radar and inertial measurement units, a dual verification mechanism is established to achieve dynamic switching between path guidance navigation mode and active obstacle avoidance navigation mode. The system only performs switching when multimodal perception data simultaneously meet preset switching conditions and continue for more than a time threshold, avoiding false triggering caused by misjudgment from a single sensor or instantaneous disturbances, improving the accuracy and stability of navigation mode switching, and significantly reducing the risk of navigation response delay caused by sudden environmental changes;

[0025] 2. A deviation coefficient model based on the minimum vertical distance between the user's current location and the initial path is introduced. The ratio of this coefficient to a preset safety width is used to determine whether the user has seriously deviated from the path. When the deviation is small, the system generates a transitional path to guide the user back gradually; a new path is only replanned when there is a serious deviation. This mechanism avoids the frequent navigation jumps caused by over-reliance on global path replanning in existing technologies, improving the continuity of path guidance and the flexibility of correction control.

[0026] 3. A multi-modal data acquisition system is formed by integrating binocular cameras, millimeter-wave radar, and an inertial measurement unit. This system can acquire image frame sequences, obstacle reflection information, and user action status, achieving visual, spatial, and dynamic triple perception. Especially in scenarios with dynamic obstacles, the three types of sensors can work complementaryly to improve the robustness of environmental perception, providing multi-dimensional data support for obstacle avoidance navigation modes, thereby enhancing the system's navigation practicality and safety in complex and unstructured environments. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is an overall block diagram of the system according to Embodiment 1 of the present invention;

[0029] Figure 2 This is a flowchart of the intelligent navigation system according to Embodiment 1 of the present invention;

[0030] Figure 3 This is an overall block diagram of the method in Embodiment 2 of the present invention. Detailed Implementation

[0031] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Application Overview: In existing technologies, intelligent navigation systems assisting visually impaired individuals in their travel typically rely on static path planning and single sensor data to achieve navigation functionality. When users are in complex and dynamic environments, existing systems struggle to accurately determine when to switch navigation modes, leading to delays in path correction or obstacle avoidance responses. For example, when a moving obstacle temporarily appears in a shopping mall corridor, traditional systems may fail to trigger obstacle avoidance modes due to a lack of multimodal data joint judgment mechanisms, causing obstruction or safety hazards for the user.

[0033] To address the aforementioned issues, the inventors discovered that existing systems suffer from insufficient data fusion in environmental perception and path correction. By analyzing the complementary characteristics of multimodal sensor data, they proposed combining image coherence indicators with auxiliary judgment markers generated from radar and inertial data to establish a dual verification mechanism for accurately determining navigation mode switching conditions. Simultaneously, to address path deviation, a minimum deviation coefficient calculation model based on geometric relationships was introduced to achieve quantitative assessment of deviation levels and a flexible path correction strategy.

[0034] Example 1:

[0035] like Figure 1-2 As shown, the intelligent navigation system for the blind based on multimodal sensor fusion includes:

[0036] An initialization module is used to generate an initial path based on the user's starting and target locations, and to perform navigation using a path-guided navigation mode based on the initial path.

[0037] The data acquisition module is used to acquire image frame data, radar reflection data, and inertial measurement data of the user's current location;

[0038] The data processing module is used to calculate the image coherence index based on the image frame data, and generate auxiliary judgment flags based on the radar reflection data and inertial measurement data respectively.

[0039] The first judgment module is used to determine whether the image continuity index is within a preset continuous range. If it is, the path guidance navigation mode is maintained; otherwise, it further determines whether the auxiliary judgment flag meets the first switching condition. If it is, the mode is switched to active obstacle avoidance navigation mode; otherwise, the path guidance navigation mode is maintained.

[0040] The second judgment module is used to determine whether the image continuity index and auxiliary judgment flag meet the second switching condition when the system is in active obstacle avoidance navigation mode. If yes, it switches back to path guidance navigation mode; otherwise, it maintains active obstacle avoidance navigation mode.

[0041] The path correction module is used to determine whether the user has deviated from the initial path based on the spatial relationship between the user's current location and the initial path when the system switches back to the path guidance navigation mode. If so, the path guidance navigation mode based on the initial path is continued. Otherwise, the minimum deviation coefficient from the initial path is calculated, and it is determined whether the deviation coefficient exceeds a preset deviation threshold. If so, a new path is generated based on the current location and the target location and the initial path is replaced. Otherwise, a transition path is generated between the current location and the initial path.

[0042] The navigation prompt module is used to generate corresponding navigation prompts based on the navigation mode to guide the user's movement.

[0043] The image coherence index refers to an environmental stability assessment parameter calculated by weighting the success rate of feature point matching between adjacent image frames with the standard deviation of the displacement vector. Specifically, it can be implemented using SIFT feature extraction and optical flow methods to reflect the continuity of environmental changes in the user's direction of travel. Auxiliary judgment markers include obstacle reflection markers generated based on the difference in radar reflection intensity and the rate of change of target distance, and action markers determined based on the rate of change of inertial data. Specifically, it can be implemented using sliding window statistics and threshold comparison methods to verify abnormal sensor data states. The first switching condition refers to the combination of the image coherence index exceeding a preset range and the auxiliary markers remaining abnormal. Specifically, it can be implemented using logical AND operations combined with a time window counter to ensure the reliability of mode switching. The second switching condition includes two sub-conditions: the auxiliary markers returning to normal and the image index remaining stable. Specifically, it can be implemented using a parallel state monitoring mechanism to confirm that the environment has returned to a safe state. The deviation coefficient is the ratio of the minimum vertical distance from the user's current position to the initial path to the safe width. Specifically, it can be implemented using vector projection algorithms and proportional calculations to quantify the degree of path deviation.

[0044] Specifically, the system first generates a discrete initial path using an electronic map and initiates path guidance. A binocular camera continuously acquires image frames, a millimeter-wave radar scans for obstacles ahead, and an inertial unit monitors the user's movement. The data processing module calculates the image feature matching rate and displacement standard deviation in real time, generating obstacle reflection markers and action markers. When image indicators exceed a preset range, the system further checks whether the auxiliary markers meet the first switching condition. If the radar detects a sudden change in reflection intensity and the inertial data fluctuation continues to exceed limits, it switches to active obstacle avoidance mode. In obstacle avoidance mode, the system continuously monitors environmental recovery conditions. When the auxiliary markers return to normal and image indicators stabilize continuously, it switches back to path guidance. The path correction module determines the deviation state by calculating the projected distance between the user's position and the initial path. If the deviation coefficient exceeds a threshold, a new path is generated; otherwise, a transitional path is created to gradually guide the user back to the initial route. The navigation prompt module outputs a combination of voice and vibration commands based on the current mode, providing directional prompts during path guidance and sending deceleration warnings during obstacle avoidance.

[0045] Existing systems typically rely on a single sensor to trigger mode switching, such as using radar to detect obstacles or inertial data to determine user actions. This solution, through joint judgment of image coherence indicators and multi-source auxiliary signs, effectively distinguishes between dynamic environmental changes and interference from user actions, avoiding erroneous switching or response delays. Traditional path correction methods directly replan the path when the user deviates, easily leading to frequent changes in navigation commands. This solution quantifies the degree of deviation using a deviation coefficient and employs a flexible transition path correction strategy, ensuring path continuity while reducing the frequency of user orientation adjustments.

[0046] Through the above technical solutions, this application achieves precise switching of navigation modes and intelligent correction of path deviations in complex environments. The multimodal data fusion mechanism effectively improves the reliability of environmental perception, dual judgment conditions ensure the accuracy of mode switching decisions, and the deviation coefficient model provides a quantitative basis for path correction. The system can promptly initiate obstacle avoidance guidance when the user encounters sudden obstacles, smoothly switch back to path navigation after the environment recovers, and dynamically adjust the correction strategy according to the degree of deviation, significantly improving the safety of blind users and the smoothness of their navigation experience.

[0047] This application further proposes a process for generating an initial path that includes receiving the user's voice input of the starting and target positions, calling a preset electronic map database to generate an accessible path between the starting and target positions, and discretizing the accessible path into a set of continuous polyline segments as the initial path.

[0048] User voice input refers to acquiring the user's location information through voice recognition technology. This can be achieved using an offline voice recognition engine combined with a pre-set place name database, avoiding manual input and improving ease of use for blind users. The electronic map database is a structured dataset storing geospatial information and obstacle distribution data. This can be implemented using open-source map data combined with real-time updated obstacle markers to generate accessible paths that meet user safety requirements. An accessible path is a walkable route between the start and end points that avoids fixed obstacles. This can be achieved using a path search method based on the A* algorithm, ensuring that the path planning results meet the safety requirements of blind users. Discretization involves transforming a continuous path into a set of multiple straight line segments. This can be achieved using equidistant sampling or curvature adaptive segmentation algorithms, facilitating segmented management and real-time tracking of the path by the navigation module.

[0049] Specifically, the system receives the starting and destination point information input by the user through the voice interaction module, calls obstacle distribution data from the electronic map database, and uses a path search algorithm to generate the optimal path to avoid fixed obstacles. Subsequently, the path is discretized into a set of multiple polyline segments, enabling the navigation module to provide directional guidance and position matching based on the coordinates of the endpoints of the polyline segments, thereby improving the accuracy and real-time performance of path tracking.

[0050] Existing systems typically rely on manual input or preset paths, lacking flexible path generation capabilities based on voice interaction. Furthermore, path planning results are often continuous curves, limiting navigation accuracy. This solution lowers the operational barrier through voice input and combines dynamic, barrier-free path generation with discretization processing to achieve dual optimization of path planning and navigation guidance.

[0051] Through the above technical solution, this application solves the problems of existing blind navigation systems relying on static data for path generation and low user interaction efficiency. It realizes dynamic path planning based on voice input and improves the accuracy of path guidance through discretization processing, ensuring that users can travel along the initial path efficiently and safely.

[0052] This application further proposes a data acquisition module including a binocular camera, a millimeter-wave radar, and an inertial measurement unit. The binocular camera is used to acquire a continuous sequence of image frames at a fixed sampling frequency; the millimeter-wave radar is used to acquire radar reflection data in a 180° plane in front, including obstacle distance and reflection intensity reflecting the changing state of obstacle reflection; the inertial measurement unit is used to acquire inertial measurement data, including triaxial acceleration and angular velocity.

[0053] Among them, a binocular camera refers to a device that simultaneously acquires image data using two parallel cameras. Specifically, it can use a sampling frequency of 30 frames per second to acquire continuous image frame sequences for subsequent calculation of image coherence indicators. A millimeter-wave radar refers to a radar sensor operating in the millimeter-wave frequency band. Specifically, it can use the 24GHz band to dynamically detect the distance and reflection intensity of obstacles within a 180° plane in front, used to generate obstacle reflection markers. An inertial measurement unit (IMU) is a sensor module integrating an accelerometer and a gyroscope. Specifically, it can use MEMS technology to achieve real-time measurement of three-axis acceleration and angular velocity, used to generate motion markers.

[0054] Specifically, the binocular camera acquires a continuous sequence of image frames at a fixed sampling frequency, such as 30 frames per second, ensuring a constant time interval between adjacent image frames and providing a stable time reference for feature point matching calculations. The millimeter-wave radar transmits millimeter-wave signals and receives reflected waves to calculate obstacle distances and reflection intensity. For example, it performs a scan every 50 milliseconds within a 180° plane in front of the user, generating radar reflection data containing spatial distribution information of obstacles. The inertial measurement unit (IMU) synchronously acquires the user's motion state using a three-axis accelerometer and a three-axis gyroscope, such as recording real-time changes in acceleration and angular velocity during walking, providing a data basis for assessing the stability of the user's movements. The data from these three types of sensors are synchronously transmitted to the data processing module, forming the basis for multimodal perception data fusion.

[0055] Existing navigation systems for the blind typically employ a single camera or low-frequency image acquisition device, making it difficult to guarantee the accuracy of image continuity analysis. This application, however, uses a combination of binocular cameras and a fixed sampling frequency, improving the temporal consistency of image frame sequences. Current radar sensors often employ ultrasonic or lidar, which suffers from limited detection angles or excessive costs. This application uses millimeter-wave radar to achieve 180° planar scanning, ensuring detection range while reducing hardware costs. Existing inertial measurement units often only acquire single parameters such as acceleration or angular velocity, while this application, through simultaneous acquisition of triaxial acceleration and angular velocity, can more comprehensively reflect the user's motion state.

[0056] Through the above technical solution, this application achieves collaborative acquisition of multimodal sensor data. High-frequency image data from the binocular camera provides visual basis for environmental continuity judgment, wide-angle obstacle detection data from the millimeter-wave radar provides spatial information for obstacle avoidance decision-making, and multidimensional motion data from the inertial measurement unit provides physical quantity support for user action stability analysis. The fusion of these three data sources effectively solves the navigation mode switching delay problem caused by incomplete data from a single sensor in existing systems, and improves the reliability of navigation decisions in complex environments.

[0057] This application further proposes a calculation process for the image coherence index, including extracting feature point sets from adjacent image frames, calculating the feature point matching success rate as a first coherence factor, calculating the standard deviation of the feature point displacement vector as a second coherence factor, and using the weighted sum of the first and second coherence factors as the image coherence index; the process of generating auxiliary judgment flags includes auxiliary judgment flags containing obstacle reflection flags and action flags; the obstacle reflection flag is generated based on the difference in reflection intensity of millimeter-wave radar in two consecutive cycles and the rate of change of target distance, and when both indicators exceed the set range, it is judged as an abnormal state, otherwise it is judged as a normal state; the action flag is calculated based on the rate of change of triaxial acceleration and angular velocity recorded by the inertial measurement unit in the current cycle, and if its rate of change exceeds the preset stable range, it is judged as an unstable state, otherwise it is judged as a stable state.

[0058] The feature point matching success rate refers to the proportion of successfully matched feature points between adjacent image frames out of the total number of feature points. This can be achieved by extracting feature points using the ORB algorithm and matching them using Hamming distance, reflecting the coherence of the image sequence. The standard deviation of the feature point displacement vector refers to the dispersion of the displacement vectors of all matched feature points. This can be achieved by calculating the square root of the variance of the coordinate differences of the displacement vectors, measuring the stability of the user's motion state. The abnormal state judgment of obstacle reflection markers refers to the determination of the presence of a sudden obstacle when the difference in reflection intensity detected by the millimeter-wave radar in two consecutive cycles exceeds a threshold and the rate of change of the target distance exceeds a preset range. This can be achieved by using a sliding window statistical method to analyze radar data in real time. The unstable state judgment of action markers refers to the determination of drastic changes in user action when the rate of change of triaxial acceleration or angular velocity of the inertial measurement unit exceeds a preset stable range. This can be achieved by calculating the first derivatives of acceleration and angular velocity and combining them with low-pass filtering.

[0059] Specifically, the image coherence index, by combining the feature point matching success rate and the standard deviation of the displacement vector, can comprehensively evaluate the continuity of environmental visual information and the stability of the user's movement state. For example, when the user is walking smoothly, the feature point matching success rate is high and the standard deviation of the displacement vector is low. In this case, the image coherence index remains within the preset range, and the system maintains the path guidance mode. When the ambient light changes abruptly or the user turns rapidly, causing the image feature point matching to fail or the displacement to fluctuate drastically, the image coherence index exceeds the preset range. At this time, a secondary judgment needs to be made by combining obstacle reflection markers and action markers. Millimeter-wave radar can identify suddenly appearing moving obstacles by analyzing the continuous periodic difference in reflection intensity and the rate of change of target distance; the inertial measurement unit can capture the user's sudden stop or turning actions by monitoring the rate of change of acceleration and angular velocity. If both trigger abnormal states simultaneously, the system switches to active obstacle avoidance mode.

[0060] Existing navigation systems for the blind typically rely on data from a single sensor (e.g., using radar to detect obstacles or using cameras to analyze images) for navigation mode switching. This is prone to frequent mode switching or response delays due to sensor misjudgments. This solution, however, integrates image coherence indicators, obstacle reflection markers, and action markers to construct a joint judgment mechanism based on multimodal data. This avoids interference from single-sensor noise and accurately identifies sudden environmental changes and abnormal user actions, thereby improving the reliability and timeliness of navigation mode switching.

[0061] Through the above technical solution, this application effectively solves the problem of unstable navigation mode switching caused by insufficient multimodal data fusion in existing technologies. By collaboratively analyzing image coherence indicators and auxiliary judgment markers, the system can quickly distinguish between environmental interference and real obstacles, reducing the number of erroneous switching events. Simultaneously, based on a joint judgment mechanism using multi-source data, obstacle avoidance mode can be triggered promptly when user actions are abnormal or sudden obstacles appear, avoiding safety hazards caused by response delays. Furthermore, this solution provides accurate status input for the subsequent path correction module, ensuring that the navigation system maintains stable guidance and obstacle avoidance functions in complex dynamic environments.

[0062] This application further proposes that the first switching condition is that the obstacle reflection flag is in an abnormal state and the action flag is in an unstable state, and both the abnormal state and the unstable state continue for more than a set first time window.

[0063] The obstacle reflection marker refers to the judgment result generated based on the difference in reflection intensity between two consecutive cycles of millimeter-wave radar and the rate of change of target distance. Specifically, it can be implemented by comparing a threshold of reflection intensity difference and judging the range of distance change rate, used to identify dynamic abnormal changes of obstacles ahead. The action marker refers to the judgment result generated based on the three-axis acceleration and angular velocity change rate of the inertial measurement unit. Specifically, it can be implemented by comparing an acceleration change rate threshold and an angular velocity change rate threshold, used to detect the stability of the user's motion state. The first time window refers to a preset time length threshold, such as a fixed duration within the range of 0.5 seconds to 2 seconds, used to filter instantaneous interference signals.

[0064] Specifically, when the millimeter-wave radar detects a difference in reflection intensity exceeding a preset range and an abnormal rate of change in target distance, the obstacle reflection marker is marked as abnormal. Simultaneously, when the inertial measurement unit detects a rate of change in three-axis acceleration or angular velocity exceeding a stable range, the action marker is marked as unstable. The system only triggers navigation mode switching when both markers are simultaneously in abnormal states for more than a first time window. For example, if the first time window is set to 1 second, the system must continuously detect both markers in abnormal states for more than 1 second to determine that the first switching condition is met.

[0065] Existing technologies typically rely on instantaneous data from a single sensor to determine mode switching, such as depending solely on sudden changes in radar reflection intensity or instantaneous fluctuations in acceleration. This is prone to erroneous switching due to environmental noise or brief user movements. This solution, by combining the dual judgment of obstacle reflection markers and action markers with a duration filtering mechanism, effectively distinguishes between genuine obstacle avoidance needs and transient interference, thus preventing erroneous navigation mode switching.

[0066] Through the above technical solution, this application solves the problem of frequent erroneous switching of navigation modes caused by instantaneous data fluctuations in the prior art, improves the accuracy of triggering active obstacle avoidance mode, ensures that users receive timely navigation prompts in real obstacle avoidance scenarios, and reduces operational interference caused by unnecessary mode switching.

[0067] This application further proposes a second switching condition, which specifically includes a first sub-condition and a second sub-condition. The first sub-condition is that the obstacle reflection sign is in a normal state and the action sign is in a stable state. The second sub-condition is that the image continuity index is within a preset continuous interval. Both the first sub-condition and the second sub-condition continue to exceed a set second time window.

[0068] The obstacle reflection flag is an obstacle status indicator generated based on the difference in reflection intensity and the rate of change of target distance between two consecutive millimeter-wave radar cycles. Specifically, it can be determined by whether the reflection intensity difference collected by the millimeter-wave radar is within a preset range and whether the rate of change of target distance is below a threshold, reflecting the presence of abnormal obstacles ahead. The action flag is a user action status indicator generated based on the rate of change of triaxial acceleration and angular velocity recorded by the inertial measurement unit in the current cycle. Specifically, it can be determined by whether the rate of change of triaxial acceleration and angular velocity exceeds a preset stable interval, judging whether the user is in a stable walking state. The image continuity index is an image stability parameter calculated by weighting the feature point matching success rate of adjacent image frames with the standard deviation of the displacement vector. Specifically, it can be achieved by extracting feature points of adjacent frames using a feature point matching algorithm and calculating the matching rate weighted by the displacement standard deviation, evaluating the continuity and stability of the visual environment. The second time window is a time threshold used to verify the persistence of the first and second sub-conditions. Specifically, it can be implemented using a fixed-duration timer or a sliding time window statistical mechanism to avoid erroneous switching caused by instantaneous state fluctuations.

[0069] Specifically, when the system is in active obstacle avoidance navigation mode, it needs to simultaneously meet three conditions: obstacle reflection markers remain in a normal state, action markers remain in a stable state, and image continuity indicators remain within a preset range, and these states must remain unchanged within a second time window. For example, when the millimeter-wave radar detects that the reflection intensity difference has returned to the normal range and the target distance change rate is stable, combined with the inertial measurement unit detecting that the user's acceleration change rate is within a stable range, and the image feature matching rate and displacement standard deviation weighted value collected by the binocular cameras continuously meet the standards, the system will trigger mode switching after this condition has been maintained for more than the second time window. This dual-condition verification mechanism ensures that mode switching is only performed after the environmental state and user actions have both returned to stability through the time series consistency judgment of multimodal data.

[0070] Traditional methods typically rely on a single sensor's instantaneous state or a simple combination of logic for mode switching. For example, they might switch back to path guidance mode simply by detecting the disappearance of an obstacle using radar, without considering continuous verification of user action stability or visual consistency. This solution, by introducing a multimodal state joint continuous monitoring and time window verification mechanism, effectively avoids the problem of frequent mode switching caused by instantaneous sensor false alarms or brief environmental interference.

[0071] Through the above technical solution, this application solves the problem of erroneous or premature switching caused by the lack of continuous multimodal state verification when the existing blind navigation system switches from active obstacle avoidance mode back to path guidance mode. It significantly improves the reliability and environmental adaptability of navigation mode switching and ensures that users can smoothly transition from obstacle avoidance mode to path guidance mode.

[0072] This application further proposes a process for determining whether a user has deviated from the initial path, which includes calculating the vertical distance from the user's current position to each polyline segment based on the geometric relationship between the user's current position and each polyline segment in the initial path, obtaining the path point corresponding to the minimum vertical distance as the projection point, and determining that the user has deviated from the initial path when the minimum vertical distance is greater than a preset path tolerance threshold; otherwise, determining that the user has not deviated.

[0073] The vertical distance refers to the shortest perpendicular distance from the user's current location to a polyline segment in the initial path. This can be calculated using vector projection algorithms or geometric coordinate systems. By calculating the orthogonal distance between the user's location and the path segment, the actual degree of deviation can be accurately reflected. The path tolerance threshold refers to the maximum acceptable deviation range between the user's location and the initial path. This can be set to a fixed value or dynamically adjusted based on the safety requirements of the navigation scenario, used to determine whether to trigger path correction operations. The projection point is the nearest mapped point of the user's current location on the initial path. This can be determined by traversing the perpendicular points of all polyline segments and selecting the minimum value. This point serves as the benchmark reference position for path deviation analysis.

[0074] Specifically, during the user's movement, the system obtains their current position coordinates in real time and traverses all polyline segments in the initial path, calculating the vertical distance from the current position to each segment. By comparing the minimum value of all vertical distances, the corresponding path projection point is determined. If this minimum distance exceeds a preset path tolerance threshold, the user is determined to have deviated from the initial path, and the system needs to initiate a path correction process; if it does not exceed the threshold, the user is determined to still be within the safe range of the initial path, and no correction is required. For example, when the path tolerance threshold is set to 0.5 meters, if the vertical distance from the user's current position to the nearest polyline segment is 0.6 meters, a deviation determination is triggered.

[0075] Existing systems typically use straight-line distance or Euclidean distance of path point sequences for deviation judgment, without considering the geometric structure of the path's polygonal segments, which can easily lead to misjudgments. This solution, however, calculates the vertical distance and locates the projection point, enabling a more accurate reflection of the user's actual offset relative to the path and avoiding misjudgments caused by path curvature or user detours.

[0076] Through the above technical solution, this application can improve the accuracy of path deviation detection, ensure that the navigation system maintains the original path guidance when the user deviates slightly, and triggers correction only when there is a significant deviation, thereby reducing unnecessary path switching operations and enhancing navigation continuity and user travel stability.

[0077] This application further proposes a smart navigation system for the blind based on multimodal sensor fusion, including the calculation process of the minimum deviation coefficient, which includes calculating the vertical distance from the user's current position to each segment of the initial path, taking the minimum vertical distance as the projection distance, and using the ratio of the projection distance to the preset safety width as the deviation coefficient.

[0078] The minimum deviation coefficient is a quantitative indicator obtained by geometrically calculating the minimum vertical distance between the user's current position and the initial path, and then normalizing it with a preset safety width. Specifically, it can be calculated using an Euclidean distance algorithm combined with path polyline segment projection. This indicator is used to dynamically assess the degree to which the user deviates from the initial path. The preset safety width refers to the allowable deviation range of the path, dynamically adjusted according to the navigation environment. Specifically, it can use road width data from an electronic map or user walking habit data as a benchmark value. This parameter constrains the threshold for calculating the deviation coefficient, ensuring the rationality of path correction decisions.

[0079] Specifically, when a user deviates from the initial path, the system calculates the minimum projection distance from the current position to each polyline segment, then calculates a deviation coefficient by comparing this minimum distance with a preset safety width. For example, if the preset safety width is set to 1.5 meters, and the projection distance is 0.6 meters, the deviation coefficient would be 0.4. This coefficient is compared with a preset deviation threshold. If the deviation exceeds the threshold, global path replanning is triggered; otherwise, a transitional path is generated to guide the user back to the initial path. Thus, the system can implement a tiered correction strategy based on the degree of deviation, avoiding frequent path switching due to minor offsets.

[0080] Traditional methods simply use binary judgment to directly replan the path when the user deviates from it, which is prone to frequent changes in the navigation path due to environmental interference or temporary obstacle avoidance actions. In contrast, this solution introduces a deviation coefficient quantitative evaluation mechanism, combined with a preset safety width dynamic adjustment correction strategy, which not only preserves the continuity of path guidance, but also reduces the probability of misjudgment caused by temporary deviations.

[0081] Through the above technical solutions, this application achieves refined measurement of deviation. By dynamically comparing the deviation coefficient with a threshold, it effectively distinguishes between slight and significant deviation scenarios, avoiding navigation path fluctuations caused by over-correction. Simultaneously, the transition path generation mechanism provides flexible guidance when the user has not deviated significantly, reducing the computational load of global path replanning and improving system response efficiency and user experience.

[0082] This application further proposes a transition path generation method, which includes locating the path point closest to the user's current location on the initial path and generating a straight path from the current location to that path point as the transition path. The navigation prompts include direction prompts and turning prompts in the path guidance navigation mode, and emergency obstacle avoidance prompts and deceleration prompts in the active obstacle avoidance navigation mode. The different prompts are output through a combination of voice and vibration.

[0083] Transition path generation refers to generating temporary connecting paths based on the spatial relationship between the user's current location and the initial path. Specifically, this can be achieved by using geometric calculations to locate the nearest path point in the initial path and generating the connecting path through a straight-line interpolation algorithm. For example, the nearest point can be determined by calculating the Euclidean distance between the current location and the set of path points, and then a straight-line trajectory between the two points can be generated. Navigation prompts refer to dynamically adjusting the output content and format according to the navigation mode. Specifically, this can be achieved by using a speech synthesis module and a vibration motor working together. For example, in path guidance mode, a voice prompt of "Turn right ahead" accompanied by a short vibration can be output, while in obstacle avoidance mode, a warning voice of "Obstacle on the left" can be output and triggered with continuous vibration.

[0084] Specifically, when a user deviates from the initial path but the deviation does not exceed a threshold, the system uses geometric calculations to determine the nearest path point within the initial path and generates a straight transition path to guide the user back to the original path. For example, if the projected distance between the user's current position and the initial path is 1.2 meters and the safety width is set to 1 meter, the deviation coefficient is 1.2. In this case, a transition path is generated instead of completely replanning the path. The navigation prompt module switches its output strategy according to the current mode; for example, in path guidance mode, it broadcasts directional instructions via voice, while in obstacle avoidance mode, it alerts the user to slow down via high-frequency vibration.

[0085] In some specific implementations, locating the nearest path point can be achieved by traversing the endpoint coordinates of the initial path polyline segment, for example, by using a KD-tree data structure to accelerate nearest neighbor search; straight path generation can be achieved using the Bresenham algorithm for discretized trajectory calculation. Voice prompts can combine pre-recorded instructions with real-time synthesis technology, for example, triggering a prompt 200 meters before a turn, and the vibration intensity can be dynamically adjusted according to the distance to obstacles.

[0086] Existing systems typically replan the entire route directly when a user deviates from it, leading to frequent changes in navigation instructions. This solution, however, achieves gradual correction by generating transitional routes, reducing the number of route changes. Furthermore, existing technologies often rely on single voice prompts, which are easily disrupted in noisy environments. This solution combines vibration feedback to create multimodal prompts, improving the reliability of information delivery.

[0087] Through the above technical solutions, this application can avoid global path replanning when the user deviates slightly from the initial path, achieve smooth return through transition paths, and reduce the volatility of navigation logic; at the same time, the combined voice and vibration prompts enhance the user's perception efficiency of environmental status and navigation commands, especially in dynamic obstacle avoidance scenarios, shortening user response delay and improving the practicality and safety of the navigation system.

[0088] Example 2:

[0089] like Figure 3 As shown, the intelligent navigation method for the blind based on multimodal sensor fusion includes the following steps:

[0090] An initial path is generated based on the user's starting and destination locations, and navigation is performed using a path-guided navigation mode based on the initial path.

[0091] Acquire image frame data, radar reflection data, and inertial measurement data of the user's current location;

[0092] The image coherence index is calculated based on the image frame data, and auxiliary judgment indicators are generated based on the radar reflection data and inertial measurement data, respectively.

[0093] Determine whether the image continuity index is within a preset continuous range. If yes, maintain the path guidance navigation mode; otherwise, further determine whether the auxiliary judgment flag meets the first switching condition. If yes, switch to active obstacle avoidance navigation mode; otherwise, maintain the path guidance navigation mode.

[0094] When in active obstacle avoidance navigation mode, determine whether the image continuity index and auxiliary judgment flag meet the second switching condition. If yes, switch back to path guidance navigation mode; otherwise, maintain active obstacle avoidance navigation mode.

[0095] When switching back to the path guidance navigation mode, the system determines whether the user has deviated from the initial path based on the spatial relationship between the user's current location and the initial path. If so, the path guidance navigation mode based on the initial path continues to be used. Otherwise, the minimum deviation coefficient from the initial path is further calculated, and it is determined whether the deviation coefficient exceeds the preset deviation threshold. If so, a new path is generated based on the current location and the target location and the initial path is replaced. Otherwise, a transition path is generated between the current location and the initial path.

[0096] Based on the navigation mode, corresponding navigation prompts are generated to guide the user's movement.

[0097] The above description is merely an example and illustration of the structure of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the structure of the invention or exceed the scope defined in the claims, all of which should fall within the protection scope of the present invention.

[0098] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0099] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A smart navigation system for the blind based on multimodal sensor fusion, characterized in that, include: An initialization module is used to generate an initial path based on the user's starting and target locations, and to perform navigation using a path-guided navigation mode based on the initial path. The data acquisition module is used to acquire image frame data, radar reflection data, and inertial measurement data of the user's current location; The data processing module is used to calculate the image coherence index based on the image frame data, and generate auxiliary judgment flags based on the radar reflection data and inertial measurement data respectively. The first judgment module is used to determine whether the image continuity index is within a preset continuous range. If it is, the path guidance navigation mode is maintained; otherwise, it further determines whether the auxiliary judgment flag meets the first switching condition. If it is, the mode is switched to active obstacle avoidance navigation mode; otherwise, the path guidance navigation mode is maintained. The second judgment module is used to determine whether the image continuity index and auxiliary judgment flag meet the second switching condition when the system is in active obstacle avoidance navigation mode. If yes, it switches back to path guidance navigation mode; otherwise, it maintains active obstacle avoidance navigation mode. The path correction module is used to determine whether the user has not deviated from the initial path based on the spatial relationship between the user's current position and the initial path when the system switches back to the path guidance navigation mode. If so, the path guidance navigation mode based on the initial path is continued. Otherwise, the minimum deviation coefficient of the deviation from the initial path is further calculated, and it is determined whether the minimum deviation coefficient exceeds the preset deviation threshold. If so, a new path is generated based on the current position and the target position and the initial path is replaced. Otherwise, a transition path is generated between the current position and the initial path. The process of determining whether the user has not deviated from the initial path includes: based on the geometric relationship between the user's current position and each polyline segment in the initial path, calculating the vertical distance from the user's current position to each polyline segment, obtaining the path point corresponding to the minimum vertical distance as the projection point, and determining that the user has deviated from the initial path when the minimum vertical distance is greater than a preset path tolerance threshold; otherwise, determining that the user has not deviated. The calculation process of the minimum deviation coefficient includes: calculating the vertical distance from the user's current position to each polyline segment of the initial path, taking the minimum vertical distance as the projection distance, and taking the ratio of the projection distance to the preset safety width as the minimum deviation coefficient. The generation of the transition path includes: locating the path point closest to the user's current position on the initial path, and generating a straight path from the current position to that path point as the transition path; The navigation prompt module is used to generate corresponding navigation prompts based on the navigation mode to guide the user's movement.

2. The intelligent navigation system for the blind based on multimodal sensor fusion according to claim 1, characterized in that: The process of generating the initial path includes: The system receives the user's voice input of the starting and target locations, calls a preset electronic map database to generate an accessible path between the starting and target locations, and discretizes the accessible path into a set of continuous polyline segments as the initial path.

3. The intelligent navigation system for the blind based on multimodal sensor fusion according to claim 2, characterized in that: The data acquisition module includes: A binocular camera is used to acquire a continuous sequence of image frames at a fixed sampling frequency; Millimeter-wave radar is used to acquire radar reflection data within a 180° plane in front, and the radar reflection data includes obstacle distance and reflection intensity, which reflect the changing state of obstacle reflection. An inertial measurement unit is used to collect inertial measurement data, which includes triaxial acceleration and angular velocity.

4. The intelligent navigation system for the blind based on multimodal sensor fusion according to claim 3, characterized in that: The calculation process of the image coherence index includes: extracting the feature point set of adjacent image frames, calculating the feature point matching success rate as the first coherence factor, calculating the standard deviation of the feature point displacement vector as the second coherence factor, and using the weighted sum of the first coherence factor and the second coherence factor as the image coherence index. The process of generating the auxiliary judgment flag includes: The auxiliary judgment markers include obstacle reflex markers and action markers; The obstacle reflection indicator is generated based on the difference in reflection intensity of millimeter-wave radar in two consecutive cycles and the rate of change of target distance. When both indicators exceed the set range, it is judged as an abnormal state; otherwise, it is judged as a normal state. The action flag is calculated based on the rate of change of triaxial acceleration and angular velocity recorded by the inertial measurement unit in the current cycle. If the rate of change exceeds the preset stable range, it is judged as an unstable state; otherwise, it is judged as a stable state.

5. The intelligent navigation system for the blind based on multimodal sensor fusion according to claim 4, characterized in that: The first switching condition is specifically: the obstacle reflection flag is in an abnormal state and the action flag is in an unstable state, and both the abnormal state and the unstable state last for more than a set first time window.

6. The intelligent navigation system for the blind based on multimodal sensor fusion according to claim 5, characterized in that: The second switching condition specifically includes: First sub-condition: The obstacle reflex sign is in a normal state and the action sign is in a stable state; Second sub-condition: The image coherence index is within a preset continuous range; Both the first and second sub-conditions continue to exceed the set second time window.

7. The intelligent navigation system for the blind based on multimodal sensor fusion according to claim 6, characterized in that: The navigation prompts include direction and turn prompts in the path guidance navigation mode, and emergency obstacle avoidance and deceleration prompts in the active obstacle avoidance navigation mode. Different prompts are output through a combination of voice and vibration.

8. A method for intelligent navigation for the blind based on multimodal sensor fusion according to any one of claims 1-7, characterized in that, Includes the following steps: An initial path is generated based on the user's starting and destination locations, and navigation is performed using a path-guided navigation mode based on the initial path. Acquire image frame data, radar reflection data, and inertial measurement data of the user's current location; The image coherence index is calculated based on the image frame data, and auxiliary judgment indicators are generated based on the radar reflection data and inertial measurement data, respectively. Determine whether the image continuity index is within a preset continuous range. If yes, maintain the path guidance navigation mode; otherwise, further determine whether the auxiliary judgment flag meets the first switching condition. If yes, switch to active obstacle avoidance navigation mode; otherwise, maintain the path guidance navigation mode. When in active obstacle avoidance navigation mode, determine whether the image continuity index and auxiliary judgment flag meet the second switching condition. If yes, switch back to path guidance navigation mode; otherwise, maintain active obstacle avoidance navigation mode. When switching back to the path guidance navigation mode, the system determines whether the user has not deviated from the initial path based on the spatial relationship between the user's current location and the initial path. If so, the path guidance navigation mode based on the initial path continues. Otherwise, the system further calculates the minimum deviation coefficient from the initial path and determines whether the minimum deviation coefficient exceeds a preset deviation threshold. If so, a new path is generated based on the current location and the target location and the initial path is replaced. Otherwise, a transition path is generated between the current location and the initial path. Based on the navigation mode, corresponding navigation prompts are generated to guide the user's movement.

Images