Blind person intelligent navigation system based on multi-mode sensor fusion

Abstract

The invention discloses a blind person intelligent navigation system based on multi-modal sensor fusion, and relates to the field of blind person intelligent navigation systems, and the blind person intelligent navigation system comprises an initialization module, a data acquisition module, a data processing module, a first judgment module, a second judgment module, a path correction module and a navigation prompt module. The initialization module generates an initial path according to the initial position and the target position and starts a path guiding mode; the data acquisition module synchronously acquires image, radar and inertial measurement data; the data processing module calculates an image coherence index and generates an auxiliary judgment mark; the first judgment module determines whether to switch an obstacle avoidance mode or not according to the index continuity and the auxiliary mark; a second judgment module evaluates a condition of return path guidance in an obstacle avoidance mode; the path correction module judges a path correction mode through the spatial relationship and the deviation coefficient; the navigation prompting module outputs guiding information according to the mode; according to the invention, the accuracy and stability of navigation mode switching can be improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent navigation for the blind, and in particular to an intelligent navigation system for the blind based on multimodal sensor fusion. Background Art

[0002] With the rapid development of multimodal sensor technology, path planning algorithms, and human-computer interaction methods, intelligent navigation technologies to assist the visually impaired have become a research hotspot. Current navigation systems for the blind are primarily based on functional modules such as voice recognition, path guidance, and obstacle avoidance, aiming to provide autonomous and safe travel assistance for blind users. Some advanced systems have attempted to integrate multiple sensors, including electronic maps, cameras, radar, and inertial measurement units, to achieve comprehensive functions for environmental perception, path navigation, and dynamic obstacle avoidance.

[0003] However, existing navigation systems for the blind still face many technical bottlenecks in practical applications. First, in terms of path guidance, most systems use static path planning or rule-based navigation strategies. When encountering sudden obstacles or users accidentally deviate from the path, the system lacks an effective judgment mechanism to decide whether to switch navigation modes, resulting in response delays or frequent error prompts. Secondly, although some systems have adopted multimodal perception data, an intelligent switching mechanism based on image coherence and collaborative analysis of multi-source sensor states has not yet been established, making the system's transition between path guidance and active obstacle avoidance modes not stable and reliable enough. Furthermore, when the user deviates from the preset path, existing solutions often simply re-plan the path, lacking an accurate quantitative assessment of the degree of deviation and a progressive path correction strategy, which can easily lead to problems such as frequent changes in navigation instructions or guidance failure.

[0004] In view of the above problems, the existing technology is in urgent need of improvement. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the present invention provides an intelligent navigation system for the blind based on multimodal sensor fusion.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows: In a first aspect, the present invention discloses an intelligent navigation system for the blind based on multimodal sensor fusion, comprising: An initialization module, configured to generate an initial path according to a user's starting location and target location, and perform navigation in a path-guided navigation mode based on the initial path; Data acquisition module, used to obtain image frame data, radar reflection data and inertial measurement data of the user's current position; a data processing module, configured to calculate an image coherence index based on the image frame data, and simultaneously generate auxiliary judgment marks based on the radar reflection data and the inertial measurement data; a first judgment module, configured to judge whether the image coherence index is within a preset continuous interval, and if so, maintain the path guidance navigation mode; otherwise, further judge whether the auxiliary judgment flag satisfies a first switching condition, and if so, switch to the active obstacle avoidance navigation mode, and otherwise maintain the path guidance navigation mode; a second judgment module, configured to judge, when the system is in the active obstacle avoidance navigation mode, whether the image coherence index and the auxiliary judgment flag meet a second switching condition, and if so, switch back to the path guidance navigation mode; otherwise, maintain the active obstacle avoidance navigation mode; a path correction module, configured to determine, when the system switches back to the path guidance navigation mode, whether the user has deviated from the initial path based on the spatial relationship between the user's current position and the initial path; if so, continue to use the path guidance navigation mode based on the initial path; otherwise, further calculate the minimum deviation coefficient from the initial path, determine whether the deviation coefficient exceeds a preset deviation threshold, and if so, generate a new path based on the current position and the target position and replace the initial path; otherwise, generate a transition path between the current position and the initial path; The navigation prompt module is used to generate corresponding navigation prompt information according to the navigation mode to guide the user.

[0007] In a second aspect, the present invention discloses an intelligent navigation method for the blind based on multimodal sensor fusion, comprising the following steps: Generate an initial path based on the user's starting location and target location, and navigate using a path-guided navigation mode based on the initial path; Obtain image frame data, radar reflection data, and inertial measurement data of the user's current location; Calculating an image coherence index based on the image frame data, and generating auxiliary judgment marks based on the radar reflection data and the inertial measurement data; determining whether the image coherence index is within a preset continuous interval, if so, maintaining the path guidance navigation mode; otherwise, further determining whether the auxiliary judgment flag satisfies a first switching condition, if so, switching to the active obstacle avoidance navigation mode, otherwise maintaining the path guidance navigation mode; When in the active obstacle avoidance navigation mode, determining whether the image coherence index and the auxiliary determination flag meet a second switching condition, and if so, switching back to the path guidance navigation mode; otherwise, maintaining the active obstacle avoidance navigation mode; When switching back to the path guidance navigation mode, the user is judged to have deviated from the initial path based on the spatial relationship between the user's current position and the initial path. If so, the path guidance navigation mode based on the initial path is continued. Otherwise, the minimum deviation coefficient from the initial path is further calculated to determine whether the deviation coefficient exceeds a preset deviation threshold. If so, a new path is generated based on the current position and the target position and replaces the initial path. Otherwise, a transition path is generated between the current position and the initial path. Generate corresponding navigation prompt information according to the navigation mode to guide the user.

[0008] The beneficial effects of the present invention are: 1. By integrating image coherence indicators with auxiliary judgment flags generated by millimeter-wave radar and inertial measurement units, a dual verification mechanism is established to achieve dynamic switching between path guidance navigation mode and active obstacle avoidance navigation mode. The system only switches when multimodal perception data simultaneously meets the preset switching conditions and lasts for more than a time threshold. This avoids false triggering caused by single sensor misjudgment or transient disturbances, improves the accuracy and stability of navigation mode switching, and significantly reduces the risk of navigation response delays caused by sudden environmental changes. 2. A deviation coefficient model based on the minimum vertical distance between the user's current location and the initial path is introduced. The ratio of this coefficient to the preset safety width is used to determine whether the user has seriously deviated from the path. When the deviation is small, the system generates a transition path to guide the user back gradually; a new path is only replanned in case of serious deviation. This mechanism avoids the frequent navigation jumps caused by over-reliance on global path replanning in existing technologies, improving the continuity of path guidance and the flexible control capability of correction. 3. A multimodal data acquisition system integrating binocular cameras, millimeter-wave radar, and an inertial measurement unit (IMU) captures image frame sequences, obstacle reflections, and user motion status, enabling visual, spatial, and dynamic perception. Especially in scenarios with dynamic obstacles, these three sensors complement each other, enhancing the robustness of environmental perception and providing multidimensional data support for obstacle avoidance navigation, thereby improving the system's practicality and safety in complex, unstructured environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 This is an overall block diagram of a system according to an embodiment of the present invention; Figure 2 This is a flowchart of the intelligent navigation system of Example 1 of the present invention; Figure 3 This is an overall block diagram of the method according to the second embodiment of the present invention. DETAILED DESCRIPTION

[0011] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0012] Application Overview: Prior art intelligent navigation systems designed to assist the visually impaired typically rely on static path planning and single-sensor data to achieve navigation. When users navigate complex, dynamic environments, existing systems struggle to accurately determine when to switch navigation modes, resulting in inability to promptly correct path deviations or delayed obstacle avoidance responses. For example, when a moving obstacle temporarily appears in a shopping mall corridor, conventional systems may be unable to trigger obstacle avoidance mode due to a lack of a multimodal data-based joint judgment mechanism, hindering user progress or posing safety risks.

[0013] To address these issues, the inventors discovered that existing systems lack data fusion for environmental perception and path correction. By analyzing the complementary nature of multimodal sensor data, they proposed combining image coherence metrics with auxiliary judgment markers generated by radar and inertial data to establish a dual verification mechanism to accurately determine navigation mode switching conditions. Furthermore, to address path deviation, they introduced a minimum deviation coefficient calculation model based on geometric relationships, enabling quantitative assessment of the degree of deviation and implementing a flexible path correction strategy.

[0014] Example 1:

[0015] like Figure 1-2 As shown in FIG, the intelligent navigation system for the blind based on multimodal sensor fusion includes: An initialization module, configured to generate an initial path according to a user's starting location and target location, and perform navigation in a path-guided navigation mode based on the initial path; Data acquisition module, used to obtain image frame data, radar reflection data and inertial measurement data of the user's current position; a data processing module, configured to calculate an image coherence index based on the image frame data, and simultaneously generate auxiliary judgment marks based on the radar reflection data and the inertial measurement data; a first judgment module, configured to judge whether the image coherence index is within a preset continuous interval, and if so, maintain the path guidance navigation mode; otherwise, further judge whether the auxiliary judgment flag satisfies a first switching condition, and if so, switch to the active obstacle avoidance navigation mode, and otherwise maintain the path guidance navigation mode; a second judgment module, configured to judge, when the system is in the active obstacle avoidance navigation mode, whether the image coherence index and the auxiliary judgment flag meet a second switching condition, and if so, switch back to the path guidance navigation mode; otherwise, maintain the active obstacle avoidance navigation mode; a path correction module, configured to determine, when the system switches back to the path guidance navigation mode, whether the user has deviated from the initial path based on the spatial relationship between the user's current position and the initial path; if so, continue to use the path guidance navigation mode based on the initial path; otherwise, further calculate the minimum deviation coefficient from the initial path, determine whether the deviation coefficient exceeds a preset deviation threshold, and if so, generate a new path based on the current position and the target position and replace the initial path; otherwise, generate a transition path between the current position and the initial path; The navigation prompt module is used to generate corresponding navigation prompt information according to the navigation mode to guide the user.

[0016] The image coherence index is an environmental stability assessment parameter calculated by weighting the success rate of feature point matching between adjacent image frames and the standard deviation of the displacement vector. It can be implemented using SIFT feature extraction and optical flow methods, reflecting the continuity of environmental changes in the user's direction of travel. Auxiliary judgment indicators include obstacle reflection indicators generated based on the difference in radar reflection intensity and the rate of change of target distance, and action indicators based on the rate of change of inertial data. These indicators can be implemented using sliding window statistics and threshold comparison methods to verify abnormal sensor data. The first switching condition is the combination of the image coherence index exceeding a preset range and the auxiliary indicator remaining abnormal. This can be implemented using a logical AND operation combined with a time window counter to ensure the reliability of mode switching. The second switching condition consists of two sub-conditions: the auxiliary indicator returning to normal and the image indicator remaining stable. This can be implemented using a parallel state monitoring mechanism to confirm that the environment has returned to a safe state. The deviation coefficient is the ratio of the minimum vertical distance from the user's current location to the initial path to the safe width. This can be implemented using a vector projection algorithm and proportional calculation to quantify the degree of path deviation.

[0017] Specifically, the system first generates a discretized initial path using an electronic map and initiates path guidance. A binocular camera continuously captures image frames, a millimeter-wave radar scans for obstacles ahead, and an inertial unit monitors the user's motion. The data processing module calculates the image feature matching rate and displacement standard deviation in real time, generating obstacle reflection markers and motion markers. If image metrics exceed a preset range, the system further checks whether auxiliary markers meet the first switching condition. If the radar detects a sudden change in reflection intensity and the inertial data fluctuations continuously exceed the limit, the system switches to active obstacle avoidance mode. In obstacle avoidance mode, the system continuously monitors environmental recovery conditions and switches back to path guidance when the auxiliary markers return to normal and the image metrics remain stable. The path correction module determines deviation by calculating the projected distance between the user's position and the initial path. If the deviation coefficient exceeds a threshold, a new path is generated. Otherwise, a transition path is created to gradually guide the user back to the initial route. The navigation prompt module outputs a combination of voice and vibration commands based on the current mode, providing directional guidance during path guidance and a deceleration warning during obstacle avoidance.

[0018] Existing systems typically rely on a single sensor to trigger mode switching, such as detecting obstacles solely through radar or determining user movements solely through inertial data. This solution effectively distinguishes between dynamic environmental changes and interference from the user's own movements through the combined judgment of image coherence indicators and multi-source auxiliary signs, avoiding false switches or response delays. Traditional path correction methods directly replan the path when the user deviates, which can easily lead to frequent changes in navigation instructions. This solution quantitatively assesses the degree of deviation through a deviation coefficient and adopts a flexible transition path correction strategy to ensure path continuity while reducing the frequency of user direction adjustments.

[0019] Through the above technical solutions, this application realizes the precise switching of navigation modes and intelligent correction of path deviation in complex environments. The multimodal data fusion mechanism effectively improves the reliability of environmental perception, the dual judgment conditions ensure the accuracy of mode switching decisions, and the deviation coefficient model provides a quantitative basis for path correction. The system can promptly initiate obstacle avoidance guidance when the user encounters a sudden obstacle, and smoothly switch back to path navigation after the environment recovers. At the same time, it dynamically adjusts the correction strategy according to the degree of deviation, significantly improving the travel safety and smoothness of the navigation experience of blind users.

[0020] The present application further proposes that the process of generating the initial path includes receiving the starting position and target position input by the user's voice, calling a preset electronic map database to generate an obstacle-free path between the starting position and the target position, and discretizing the obstacle-free path into a set of continuous broken line segments as the initial path.

[0021] Among them, user voice input refers to obtaining the location information sent by the user through voice recognition technology. Specifically, it can be implemented by an offline voice recognition engine combined with a preset place name library, which can avoid manual input operations and improve the operational convenience of blind users. The electronic map database refers to a structured data set that stores geographic spatial information and obstacle distribution data. Specifically, it can be implemented by open source map data combined with real-time updated obstacle markers to generate barrier-free paths that meet user safety needs. An barrier-free path refers to a walkable route between the starting point and the end point that avoids fixed obstacles. Specifically, it can be implemented by a path search method based on the A* algorithm to ensure that the path planning results meet the safe passage needs of blind users. Discretization refers to converting a continuous path into a set of multiple straight line segments. Specifically, it can be implemented by using equal-interval sampling or curvature adaptive segmentation algorithms to facilitate the navigation module to segment and track the path in real time.

[0022] Specifically, the system receives user input of the starting and destination points through a voice interaction module, accesses obstacle distribution data from an electronic map database, and employs a path search algorithm to generate an optimal path that avoids fixed obstacles. Subsequently, through discretization, the path is converted into a collection of multiple polyline segments. This allows the navigation module to provide directional guidance and position matching based on the coordinates of the polyline segment endpoints, improving the accuracy and real-time performance of path tracking.

[0023] Existing systems typically rely on manual input or preset paths, lacking the ability to generate flexible paths through voice interaction, and often produce continuous curves, limiting navigation accuracy. This solution lowers the barrier to entry through voice input and combines dynamic, barrier-free path generation with discretization processing to achieve dual optimization of path planning and navigation guidance.

[0024] Through the above technical solution, this application solves the problem that path generation in existing blind navigation systems relies on static data and has low user interaction efficiency. It realizes dynamic path planning based on voice input and improves the accuracy of path guidance through discretization processing, ensuring that users can travel along the initial path efficiently and safely.

[0025] This application further proposes that the data acquisition module includes a binocular camera, a millimeter-wave radar and an inertial measurement unit. The binocular camera is used to collect a continuous image frame sequence at a fixed sampling frequency; the millimeter-wave radar is used to obtain radar reflection data within a 180° plane in front, and the radar reflection data includes the obstacle distance and reflection intensity reflecting the changing state of obstacle reflection; the inertial measurement unit is used to collect inertial measurement data, and the inertial measurement data includes three-axis acceleration and angular velocity.

[0026] Among them, a binocular camera refers to a device that synchronously captures image data through two parallel cameras. Specifically, a sampling frequency of 30 frames per second can be used to acquire a continuous sequence of image frames for subsequent calculation of image continuity indicators. A millimeter-wave radar refers to a radar sensor operating in the millimeter-wave frequency band. Specifically, a 24GHz frequency band can be used to dynamically detect the distance and reflection intensity of obstacles in the 180° plane ahead, used to generate obstacle reflection marks. An inertial measurement unit refers to a sensor module that integrates an accelerometer and a gyroscope. Specifically, MEMS technology can be used to achieve real-time measurement of three-axis acceleration and angular velocity, used to generate motion marks.

[0027] Specifically, the binocular camera captures a continuous sequence of image frames at a fixed sampling frequency. For example, a capture frequency of 30 frames per second ensures a constant time interval between adjacent image frames, providing a stable time reference for feature point matching calculations. The millimeter-wave radar transmits millimeter-wave signals and receives reflected waves to calculate the distance and reflection intensity of obstacles. For example, it completes a scan every 50 milliseconds within a 180-degree plane in front of the user, generating radar reflection data containing information on the spatial distribution of obstacles. The inertial measurement unit synchronously collects the user's motion status using a three-axis accelerometer and a three-axis gyroscope. For example, during walking, changes in acceleration and angular velocity are recorded in real time, providing a data basis for determining the stability of the user's movements. Data from these three types of sensors are synchronously transmitted to the data processing module, forming the basis for multimodal perception data fusion.

[0028] Existing navigation systems for the blind usually use a single camera or a low-frequency image acquisition device, which makes it difficult to ensure the accuracy of image continuity analysis; however, this application uses a combination of binocular cameras and a fixed sampling frequency to improve the temporal consistency of image frame sequences. In the existing technology, radar sensors mostly use ultrasonic or laser radars, which have problems such as limited detection angles or high costs; this application uses millimeter-wave radar to achieve 180° plane scanning, reducing hardware costs while ensuring the detection range. Existing inertial measurement units mostly only collect a single parameter of acceleration or angular velocity, but this application can more comprehensively reflect the user's motion status through the synchronous collection of three-axis acceleration and angular velocity.

[0029] Through the above technical solution, this application achieves the coordinated collection of multimodal sensor data. The high-frequency image data from the binocular camera provides a visual basis for environmental continuity judgment, the wide-angle obstacle detection data from the millimeter-wave radar provides spatial information for obstacle avoidance decisions, and the multi-dimensional motion data from the inertial measurement unit provides physical quantity support for user motion stability analysis. The data fusion of these three effectively solves the navigation mode switching delay problem caused by incomplete single sensor data in existing systems, and improves the reliability of navigation decisions in complex environments.

[0030] The present application further proposes that the calculation process of the image coherence index includes extracting a set of feature points of adjacent image frames, calculating the feature point matching success rate as a first coherence factor, calculating the standard deviation of the feature point displacement vector as a second coherence factor, and taking the weighted sum of the first coherence factor and the second coherence factor as the image coherence index; the process of generating an auxiliary judgment mark includes the auxiliary judgment mark including an obstacle reflection mark and an action mark; the obstacle reflection mark is generated based on the difference in reflection intensity of the millimeter wave radar in two consecutive cycles and the target distance change rate. When both indicators exceed the set range, it is judged to be an abnormal state, otherwise it is judged to be a normal state; the action mark is calculated based on the change rate of the three-axis acceleration and angular velocity recorded by the inertial measurement unit in the current cycle. If its change rate exceeds the preset stable range, it is judged to be an unstable state, otherwise it is judged to be a stable state.

[0031] The feature point matching success rate refers to the ratio of the number of successfully matched feature points between adjacent image frames to the total number of feature points. This can be achieved by extracting feature points using the ORB algorithm and matching them using the Hamming distance. It reflects the coherence of the image sequence. The standard deviation of the feature point displacement vector refers to the degree of dispersion of the displacement vectors of all matched feature points. This can be achieved by calculating the square root of the variance of the coordinate difference of the displacement vectors. It measures the stability of the user's motion state. The abnormal state judgment of the obstacle reflection mark refers to the determination of the presence of a sudden obstacle when the difference in reflection intensity detected by the millimeter-wave radar in two consecutive cycles exceeds a threshold and the rate of change of the target distance exceeds a preset range. This can be achieved by real-time analysis of radar data using a sliding window statistical method. The unstable state judgment of the action mark refers to the determination of a drastic change in the user's motion when the rate of change of the three-axis acceleration or angular velocity of the inertial measurement unit exceeds a preset stability range. This can be achieved by calculating the first-order derivatives of the acceleration and angular velocity and combining them with low-pass filtering.

[0032] Specifically, the image consistency index, by combining the feature point matching success rate and the standard deviation of the displacement vector, can comprehensively evaluate the continuity of the environmental visual information and the smoothness of the user's motion state. For example, when the user is in a steady walking state, the feature point matching success rate is high and the standard deviation of the displacement vector is low. At this time, the image consistency index remains within the preset range, and the system maintains the path guidance mode. When the ambient light suddenly changes or the user turns quickly, causing the image feature point matching to fail or the displacement to fluctuate violently, the image consistency index exceeds the preset range. At this time, a secondary judgment is required based on the obstacle reflection mark and the action mark. The millimeter wave radar can identify suddenly appearing moving obstacles by analyzing the reflection intensity difference and the target distance change rate in consecutive cycles; the inertial measurement unit can capture the user's sudden stop or turn action by monitoring the rate of change of acceleration and angular velocity. If both trigger an abnormal state at the same time, the system switches to active obstacle avoidance mode.

[0033] Existing navigation systems for the blind typically rely on data from a single sensor (for example, radar alone for obstacle detection or camera alone for image analysis) to switch navigation modes. This can lead to frequent mode switching or delayed responses due to sensor misjudgments. This solution, however, integrates image coherence indicators, obstacle reflection markers, and motion markers to create a multimodal data-based joint judgment mechanism. This approach not only avoids noise interference from a single sensor but also accurately identifies sudden environmental changes and abnormal user movements, thereby improving the reliability and timeliness of navigation mode switching.

[0034] Through the above technical solution, this application can effectively solve the problem of unstable navigation mode switching caused by insufficient multimodal data fusion in the existing technology. Through the collaborative analysis of image coherence indicators and auxiliary judgment signs, the system can quickly distinguish between environmental interference and real obstacles, reducing the number of false switches; at the same time, based on the joint judgment mechanism of multi-source data, the obstacle avoidance mode can be triggered in time when the user's action is abnormal or a sudden obstacle appears, avoiding safety hazards caused by delayed response. In addition, this solution provides accurate state input for the subsequent path correction module, ensuring that the navigation system maintains stable guidance and obstacle avoidance functions in complex dynamic environments.

[0035] The present application further proposes that the first switching condition is specifically that the obstacle reflection flag is in an abnormal state and the action flag is in an unstable state, and both the abnormal state and the unstable state last for more than a set first time window.

[0036] Among them, the obstacle reflection mark refers to the judgment result generated based on the difference in reflection intensity of the millimeter-wave radar in two consecutive cycles and the target distance change rate. Specifically, it can be implemented by comparing the reflection intensity difference threshold with the distance change rate interval judgment, and is used to identify dynamic abnormal changes in obstacles ahead. The action mark refers to the judgment result generated based on the three-axis acceleration and angular velocity change rate of the inertial measurement unit. Specifically, it can be implemented by comparing the acceleration change rate threshold with the angular velocity change rate threshold, and is used to detect the stability of the user's motion state. The first time window refers to a preset time length threshold, for example, it can be a fixed time length in the range of 0.5 seconds to 2 seconds, which is used to filter instantaneous interference signals.

[0037] Specifically, when the millimeter-wave radar detects that the difference in reflection intensity exceeds a preset range and the target distance change rate is abnormal, the obstacle reflection flag is marked as abnormal. At the same time, when the inertial measurement unit detects that the rate of change of the three-axis acceleration or angular velocity exceeds the stable range, the action flag is marked as unstable. The system only triggers the navigation mode switch when both flags are in an abnormal state at the same time and persist for more than a first time window. For example, if the first time window is set to 1 second, the system must continuously detect that the abnormal state of both flags persists for more than 1 second before determining that the first switching condition has been met.

[0038] Existing technologies typically rely on a single sensor's instantaneous data to determine mode switching, such as sudden changes in radar reflection intensity or instantaneous fluctuations in acceleration. This can easily lead to false switching due to environmental noise or brief user actions. This solution combines both obstacle reflection and action indicators, combined with a duration filtering mechanism, to effectively distinguish true obstacle avoidance needs from momentary interference, preventing false triggering of navigation mode switching.

[0039] Through the above technical solution, the present application solves the problem of frequent erroneous switching of navigation modes due to instantaneous data fluctuations in the prior art, improves the accuracy of triggering the active obstacle avoidance mode, ensures that users receive navigation prompts in a timely manner in real obstacle avoidance scenarios, and reduces operational interference caused by unnecessary mode switching.

[0040] The present application further proposes that the second switching condition specifically includes a first sub-condition and a second sub-condition. The first sub-condition is that the obstacle reflection mark is in a normal state and the action mark is in a stable state. The second sub-condition is that the image continuity index is within a preset continuous interval, and both the first sub-condition and the second sub-condition continue to exceed the set second time window.

[0041] The obstacle reflection flag refers to an obstacle status indicator generated based on the difference in reflection intensity from the millimeter-wave radar over two consecutive cycles and the rate of change of the target distance. Specifically, this can be achieved by checking whether the difference in reflection intensity collected by the millimeter-wave radar is within a preset range and whether the rate of change of the target distance is below a threshold. This indicator is used to indicate whether there is an abnormal obstacle ahead. The action flag refers to a user action status indicator generated based on the rate of change of the three-axis acceleration and angular velocity recorded by the inertial measurement unit during the current cycle. Specifically, this can be achieved by calculating whether the rate of change of the three-axis acceleration and angular velocity exceeds a preset stability range. This indicator is used to determine whether the user is in a stable moving state. The image coherence index refers to an image stability parameter calculated by weighting the success rate of feature point matching between adjacent image frames and the standard deviation of the displacement vector. Specifically, this index can be achieved by extracting feature points from adjacent frames using a feature point matching algorithm and calculating the weighted value of the matching rate and the displacement standard deviation. This index is used to assess whether the visual environment is continuous and stable. The second time window refers to the time threshold used to verify the continuity of the first and second subconditions. Specifically, this index can be achieved using a fixed-duration timer or a sliding time window statistical mechanism to avoid false switching caused by transient state fluctuations.

[0042] Specifically, when the system is in active obstacle avoidance navigation mode, it needs to simultaneously meet the following three conditions: the obstacle reflection mark is continuously in a normal state, the action mark is continuously in a stable state, and the image consistency index is continuously in a preset range, and the above states must remain unchanged within the second time window. For example, when the millimeter-wave radar detects that the reflection intensity difference has returned to the normal range and the target distance change rate is stable, combined with the inertial measurement unit detecting that the user's acceleration change rate is in a stable range, and at the same time, the image feature matching rate and displacement standard deviation weighted value collected by the binocular camera continuously meet the standards, the system will trigger a mode switch after the second time window is continuously exceeded. This dual-condition verification mechanism ensures that the mode switch is executed only after the environmental state and user action have returned to stability through time series consistency judgment of multimodal data.

[0043] Traditional approaches typically switch modes based solely on the instantaneous state of a single sensor or simple logic combinations. For example, they rely solely on radar detection of the disappearance of an obstacle to switch back to path guidance mode, without considering the continuous verification of user movement stability or visual coherence. This solution effectively avoids frequent mode switching caused by transient sensor false alarms or brief environmental interference by introducing a mechanism for continuous multimodal state monitoring and time window verification.

[0044] Through the above technical solution, this application solves the problem of incorrect switching or premature switching caused by the lack of continuous verification of multimodal state when the existing blind navigation system switches from active obstacle avoidance mode to path guidance mode, significantly improves the reliability and environmental adaptability of navigation mode switching, and ensures that users can smoothly transition from obstacle avoidance state to path guidance state.

[0045] The present application further proposes a process for determining whether the user has deviated from the initial path, including calculating the vertical distance from the user's current position to each broken line segment based on the geometric relationship between the user's current position and each broken line segment in the initial path, obtaining the path point corresponding to the minimum vertical distance as the projection point, and when the minimum vertical distance is greater than a preset path tolerance threshold, it is determined that the user has deviated from the initial path, otherwise it is determined that the user has not deviated.

[0046] Among them, the vertical distance refers to the shortest vertical distance from the user's current position to a broken line segment in the initial path. Specifically, it can be achieved by using a vector projection algorithm or a geometric coordinate system calculation. By calculating the orthogonal distance between the user's position and the path segment, the actual degree of deviation can be accurately reflected. Among them, the path tolerance threshold refers to the maximum acceptable deviation range between the user's position and the initial path. Specifically, it can be set to a fixed value or a dynamically adjusted value based on the safety requirements of the navigation scenario to determine whether to trigger the path correction operation. Among them, the projection point refers to the nearest mapping point of the user's current position on the initial path. Specifically, it can be determined by traversing the perpendicular points of all broken line segments and screening the minimum value. This point serves as the benchmark reference position for path deviation analysis.

[0047] Specifically, while the user is moving, the system obtains the coordinates of their current position in real time, traverses all the broken line segments in the initial path, and calculates the vertical distance from the current position to each segment in turn. By comparing the minimum value of all vertical distances, the corresponding path projection point is determined. If the minimum distance exceeds the preset path tolerance threshold, it is determined that the user has deviated from the initial path. At this time, the system needs to start the path correction process; if it does not exceed the threshold, it is determined that the user is still within the safe range of the initial path and no correction is required. For example, when the path tolerance threshold is set to 0.5 meters, if the vertical distance from the user's current position to the nearest broken line segment is 0.6 meters, the deviation judgment is triggered.

[0048] Existing systems typically use straight-line distance or Euclidean distance of a sequence of path points to determine deviation, failing to consider the geometric structure of path segments and prone to misjudgment. However, this solution calculates vertical distance and locates the projection point, more accurately reflecting the user's actual deviation from the path and avoiding misjudgments caused by curved paths or user detours.

[0049] Through the above technical solution, the present application can improve the accuracy of path deviation detection, ensure that the navigation system maintains the original path guidance when the user deviates slightly, and triggers correction only when there is a significant deviation, thereby reducing unnecessary path switching operations and enhancing navigation continuity and user travel stability.

[0050] This application further proposes an intelligent navigation system for the blind based on multimodal sensor fusion, including a process for calculating the minimum deviation coefficient, including calculating the vertical distance from the user's current position to each broken line segment of the initial path, taking the minimum vertical distance as the projection distance, and taking the ratio of the projection distance to the preset safety width as the deviation coefficient.

[0051] The minimum deviation coefficient is a quantitative indicator derived by geometrically calculating the minimum vertical distance between the user's current location and the initial path and normalizing it with a preset safety width. This can be achieved using a Euclidean distance algorithm combined with path segment projection. This metric dynamically assesses the degree to which the user deviates from the initial path. The preset safety width refers to the allowable path deviation range, dynamically adjusted based on the navigation environment. Specifically, it can use road width data from electronic maps or user walking habits as a baseline value. This parameter constrains the calculation threshold of the deviation coefficient to ensure the rationality of path correction decisions.

[0052] Specifically, when the user deviates from the initial path, the system calculates the vertical distance from the current position to each broken line segment, determines the minimum projected distance, and then compares this distance with the preset safe width to generate a deviation coefficient. For example, the preset safe width can be set to 1.5 meters. If the projected distance is 0.6 meters, the deviation coefficient is 0.4. This coefficient is compared with the preset deviation threshold. If the threshold is exceeded, global path replanning is triggered. Otherwise, a transition path is generated to guide the user back to the initial path. In this way, the system can implement a graded correction strategy based on the degree of deviation, avoiding frequent path switching due to minor deviations.

[0053] Traditional methods rely solely on binary judgment to directly replan the route when the user deviates from the path. This can easily cause frequent changes in the navigation path due to environmental interference or temporary obstacle avoidance. This solution, however, introduces a quantitative deviation coefficient assessment mechanism, combined with a dynamic adjustment and correction strategy using a preset safety margin. This approach maintains the continuity of path guidance while reducing the probability of misjudgment caused by temporary deviations.

[0054] Through the above technical solution, this application achieves a refined measurement of the degree of deviation. By dynamically comparing the deviation coefficient with the threshold, it effectively distinguishes between slight and significant deviation scenarios, avoiding navigation path fluctuations caused by over-correction. At the same time, the transition path generation mechanism provides flexible guidance when the user does not deviate significantly, reducing the computational load of global path replanning, improving system response efficiency and user experience.

[0055] The present application further proposes that transition path generation includes locating the path point closest to the user's current position on the initial path, and generating a straight line path from the current position to the path point as the transition path; the navigation prompt information includes direction prompts and turning prompts in the path guidance navigation mode, and includes emergency obstacle avoidance prompts and deceleration prompts in the active obstacle avoidance navigation mode, and different prompt information is output jointly through voice and vibration.

[0056] Transition path generation refers to the generation of a temporary connection path based on the spatial relationship between the user's current location and the initial path. Specifically, geometric calculations can be used to locate the nearest path point in the initial path, and a linear interpolation algorithm can be used to generate a connection path. For example, the Euclidean distance between the current location and a set of path points is calculated to determine the nearest point, and then a straight line trajectory is generated between the two points. Navigation prompt information refers to the dynamic adjustment of output content and form based on the navigation mode. Specifically, a speech synthesis module and a vibration motor can be used to work together. For example, in path guidance mode, a voice prompt of "turn right ahead" is output, accompanied by a short vibration. In obstacle avoidance mode, a warning voice of "obstacle on the left" is output and continuous vibration is triggered.

[0057] Specifically, when the user deviates from the initial path but the deviation does not exceed a threshold, the system uses geometric calculations to determine the nearest path point on the initial path and generates a straight transition path to guide the user back to the original path. For example, if the projected distance between the user's current location and the initial path is 1.2 meters and the safety width is set to 1 meter, the deviation coefficient is 1.2, and a transition path is generated instead of a complete re-routing. The navigation prompt module switches its output strategy based on the current mode, for example, providing voice instructions for directions in path guidance mode and high-frequency vibration to warn the user to slow down in obstacle avoidance mode.

[0058] In some specific embodiments, locating the nearest path point can be achieved by traversing the coordinates of the endpoints of the initial path polyline segments, for example, using a KD tree data structure to accelerate the nearest neighbor search. Linear paths can be generated using the Bresenham algorithm to calculate discretized trajectories. Voice prompts can combine pre-recorded instructions with real-time synthesis technology, for example, triggering a prompt 200 meters before a turn, and the vibration intensity can be dynamically adjusted based on the distance to the obstacle.

[0059] Existing systems typically replan the entire route when the user strays, resulting in frequent navigation instruction changes. This solution, however, generates transition paths for gradual correction, reducing the number of route switches. Furthermore, existing technologies often rely on single voice prompts, which are easily interrupted in noisy environments. This solution, however, incorporates vibration feedback to create multimodal prompts, improving the reliability of information transmission.

[0060] Through the above technical solution, the present application can avoid global path replanning when the user deviates slightly from the initial path, achieve smooth regression through the transition path, and reduce the volatility of the navigation logic; at the same time, through the joint prompts of voice and vibration, the user's perception efficiency of environmental status and navigation instructions is enhanced, especially in dynamic obstacle avoidance scenarios, the user response delay is shortened, and the practicality and safety of the navigation system are improved.

[0061] Example 2:

[0062] like Figure 3 As shown, the intelligent navigation method for the blind based on multimodal sensor fusion includes the following steps: Generate an initial path based on the user's starting location and target location, and navigate using a path-guided navigation mode based on the initial path; Obtain image frame data, radar reflection data, and inertial measurement data of the user's current location; Calculating an image coherence index based on the image frame data, and generating auxiliary judgment marks based on the radar reflection data and the inertial measurement data; determining whether the image coherence index is within a preset continuous interval, if so, maintaining the path guidance navigation mode; otherwise, further determining whether the auxiliary judgment flag satisfies a first switching condition, if so, switching to the active obstacle avoidance navigation mode, otherwise maintaining the path guidance navigation mode; When in the active obstacle avoidance navigation mode, determining whether the image coherence index and the auxiliary determination flag meet a second switching condition, and if so, switching back to the path guidance navigation mode; otherwise, maintaining the active obstacle avoidance navigation mode; When switching back to the path guidance navigation mode, the user is judged to have deviated from the initial path based on the spatial relationship between the user's current position and the initial path. If so, the path guidance navigation mode based on the initial path is continued. Otherwise, the minimum deviation coefficient from the initial path is further calculated to determine whether the deviation coefficient exceeds a preset deviation threshold. If so, a new path is generated based on the current position and the target position and replaces the initial path. Otherwise, a transition path is generated between the current position and the initial path. Generate corresponding navigation prompt information according to the navigation mode to guide the user.

[0063] The above content is merely an example and explanation of the structure of the present invention. Those skilled in the art may make various modifications or additions to the described specific embodiments or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the scope of protection of the present invention.

[0064] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0065] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to specific embodiments. Obviously, many modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. Intelligent navigation system for the blind based on multimodal sensor fusion, characterized by: include: An initialization module, configured to generate an initial path according to a user's starting location and target location, and perform navigation in a path-guided navigation mode based on the initial path; Data acquisition module, used to obtain image frame data, radar reflection data and inertial measurement data of the user's current position; a data processing module, configured to calculate an image coherence index based on the image frame data, and simultaneously generate auxiliary judgment marks based on the radar reflection data and the inertial measurement data; a first judgment module, configured to judge whether the image coherence index is within a preset continuous interval, and if so, maintain the path guidance navigation mode; otherwise, further judge whether the auxiliary judgment flag satisfies a first switching condition, and if so, switch to the active obstacle avoidance navigation mode, and otherwise maintain the path guidance navigation mode; a second judgment module, configured to judge, when the system is in the active obstacle avoidance navigation mode, whether the image coherence index and the auxiliary judgment flag meet a second switching condition, and if so, switch back to the path guidance navigation mode; otherwise, maintain the active obstacle avoidance navigation mode; a path correction module, configured to determine, when the system switches back to the path guidance navigation mode, whether the user has deviated from the initial path based on the spatial relationship between the user's current position and the initial path; if so, continue to use the path guidance navigation mode based on the initial path; otherwise, further calculate the minimum deviation coefficient from the initial path, determine whether the deviation coefficient exceeds a preset deviation threshold, and if so, generate a new path based on the current position and the target position and replace the initial path; otherwise, generate a transition path between the current position and the initial path; The navigation prompt module is used to generate corresponding navigation prompt information according to the navigation mode to guide the user.

2. The intelligent navigation system for the blind based on multimodal sensor fusion according to claim 1, characterized in that: The process of generating the initial path includes: The starting position and target position input by the user's voice are received, a preset electronic map database is called to generate an obstacle-free path between the starting position and the target position, and the obstacle-free path is discretized into a set of continuous broken line segments as an initial path.

3. The intelligent navigation system for the blind based on multimodal sensor fusion according to claim 2, characterized in that: The data acquisition module includes: A binocular camera for acquiring a continuous sequence of image frames at a fixed sampling frequency; Millimeter-wave radar, used to obtain radar reflection data within a 180-degree plane ahead, including obstacle distance and reflection intensity, which reflect the changing state of obstacle reflection; The inertial measurement unit is used to collect inertial measurement data, wherein the inertial measurement data includes three-axis acceleration and angular velocity.

4. The intelligent navigation system for the blind based on multimodal sensor fusion according to claim 3, characterized in that: The image coherence index calculation process includes: extracting a set of feature points of adjacent image frames, calculating a feature point matching success rate as a first coherence factor, calculating a standard deviation of a feature point displacement vector as a second coherence factor, and taking a weighted sum of the first coherence factor and the second coherence factor as the image coherence index; The process of generating the auxiliary judgment mark includes: The auxiliary judgment mark includes an obstacle reflection mark and an action mark; The obstacle reflection mark is generated based on the difference in reflection intensity of the millimeter wave radar in two consecutive cycles and the target distance change rate. When both indicators exceed the set range, it is judged as an abnormal state, otherwise it is judged as a normal state; The action flag is calculated based on the change rate of the three-axis acceleration and angular velocity recorded by the inertial measurement unit in the current cycle. If the change rate exceeds the preset stability range, it is judged as an unstable state, otherwise it is judged as a stable state.

5. The intelligent navigation system for the blind based on multimodal sensor fusion according to claim 4, characterized in that: The first switching condition is specifically: the obstacle reflection flag is in an abnormal state and the action flag is in an unstable state, and both the abnormal state and the unstable state last longer than a set first time window.

6. The intelligent navigation system for the blind based on multimodal sensor fusion according to claim 5, characterized in that: The second switching condition specifically includes: The first sub-condition: the obstacle reflection flag is in the normal state and the action flag is in the stable state; Second sub-condition: the image coherence index is within the preset continuous interval; Both the first sub-condition and the second sub-condition continue to exceed the set second time window.

7. The intelligent navigation system for the blind based on multimodal sensor fusion according to claim 6, characterized in that: The process of determining whether the user deviates from the initial path includes: Based on the geometric relationship between the user's current position and each broken line segment in the initial path, the vertical distance from the user's current position to each broken line segment is calculated, and the path point corresponding to the minimum vertical distance is obtained as the projection point. When the minimum vertical distance is greater than the preset path tolerance threshold, it is judged that the user has deviated from the initial path, otherwise it is judged that the user has not deviated.

8. The intelligent navigation system for the blind based on multimodal sensor fusion according to claim 7, characterized in that: The calculation process of the minimum deviation coefficient includes: Calculate the vertical distance from the user's current position to each polyline segment of the initial path, take the minimum vertical distance as the projection distance, and use the ratio of the projection distance to the preset safety width as the deviation coefficient.

9. The intelligent navigation system for the blind based on multimodal sensor fusion according to claim 8, characterized in that: The transition path generation includes: locating a path point closest to the user's current position on the initial path, and generating a straight line path from the current position to the path point as the transition path; The navigation prompt information includes direction prompts and turn prompts in the path guidance navigation mode, and includes emergency obstacle avoidance prompts and deceleration prompts in the active obstacle avoidance navigation mode. Different prompt information is output through voice and vibration.

10. An intelligent navigation method for the blind based on multimodal sensor fusion based on the system according to any one of claims 1 to 9, characterized in that: The following steps are involved: Generate an initial path based on the user's starting location and target location, and navigate using a path-guided navigation mode based on the initial path; Obtain image frame data, radar reflection data, and inertial measurement data of the user's current location; Calculating an image coherence index based on the image frame data, and generating auxiliary judgment marks based on the radar reflection data and the inertial measurement data; determining whether the image coherence index is within a preset continuous interval, if so, maintaining the path guidance navigation mode; otherwise, further determining whether the auxiliary judgment flag satisfies a first switching condition, if so, switching to the active obstacle avoidance navigation mode, otherwise maintaining the path guidance navigation mode; When in the active obstacle avoidance navigation mode, determining whether the image coherence index and the auxiliary determination flag meet a second switching condition, and if so, switching back to the path guidance navigation mode; otherwise, maintaining the active obstacle avoidance navigation mode; When switching back to the path guidance navigation mode, the user is judged to have deviated from the initial path based on the spatial relationship between the user's current position and the initial path. If so, the path guidance navigation mode based on the initial path is continued. Otherwise, the minimum deviation coefficient from the initial path is further calculated to determine whether the deviation coefficient exceeds a preset deviation threshold. If so, a new path is generated based on the current position and the target position and replaces the initial path. Otherwise, a transition path is generated between the current position and the initial path. Generate corresponding navigation prompt information according to the navigation mode to guide the user.

Images