Radar-vision-IMU (Inertial Measurement Unit) fused navigation system for visually impaired people

Abstract

The invention discloses a radar-vision-I MU fused visual impaired person navigation system, which relates to the technical field of multi-source data processing, and is characterized in that a multi-source data collection module is arranged to collect obstacle distance information, ground feature image data and user motion state data in an environment; a time synchronization module is arranged to carry out space-time alignment on obstacle distance information and ground feature image data to generate multi-modal sensing data, a dynamic map generation module and a fusion model based on a particle filter algorithm are arranged, a dynamic environment map is constructed in combination with motion state data, and the spatial relationship between the current position of a user and surrounding obstacles is output. Setting a trajectory prediction module to perform path planning through a reinforcement learning algorithm to generate a prediction trajectory of user motion, and setting a multi-mode interaction module to output a voice navigation instruction to the visually impaired person through a bone conduction earphone based on the prediction trajectory and a dynamic environment map; the problem of intelligent navigation of a dynamic environment faced by visually impaired people is solved.

Description

Technical Field

[0001] This invention relates to the field of multi-source data processing technology, specifically a navigation system for visually impaired individuals that integrates radar, vision, and IMU. Background Technology

[0002] Visually impaired individuals face significant challenges in their daily lives, especially in complex and ever-changing urban environments. While traditional navigation aids such as canes and guide dogs provide basic obstacle detection, their perception range in complex environments is limited, failing to offer proactive path planning and dynamic obstacle warnings. As urban environments become increasingly complex and population mobility increases, the navigation challenges faced by visually impaired individuals also intensify. Ground features such as steps, potholes, and slopes already pose fundamental navigational difficulties, while dynamic obstacles such as pedestrians, bicycles, and electric vehicles further increase the complexity and risk of navigation.

[0003] Existing navigation technologies for the visually impaired have several limitations. Traditional electronic navigation devices mainly rely on GPS positioning and pre-stored map data, and cannot perceive or respond to real-time environmental changes, especially temporary and dynamic obstacles.

[0004] Therefore, this invention proposes a navigation system for visually impaired individuals that integrates radar, vision, and IMU. Summary of the Invention

[0005] This invention aims to address at least one of the technical problems existing in the prior art. To this end, this invention proposes a radar-vision-IMU fusion navigation system for visually impaired individuals, solving the problem of intelligent navigation for visually impaired individuals in dynamic environments.

[0006] To achieve the above objectives, a radar-vision-IMU fusion navigation system for visually impaired individuals is proposed, comprising a multi-source data collection module, a time synchronization module, a dynamic map generation module, a trajectory prediction module, and a multimodal interaction module; wherein, the various modules are connected electrically.

[0007] Among them, the multi-source data collection module collects obstacle distance information in the environment in real time through millimeter-wave radar, combines it with ground feature image data obtained by the camera and user motion state data provided by the IMU, and sends the obstacle distance information and ground feature image data to the time synchronization module and the user motion state data to the dynamic map generation module.

[0008] The time synchronization module aligns the obstacle distance information of the millimeter-wave radar with the ground feature image data in time and space to generate spatiotemporally synchronized multimodal perception data, and sends the spatiotemporally synchronized multimodal perception data to the dynamic map generation module.

[0009] The dynamic map generation module inputs the spatio-temporally synchronized multi-modal perception data into a fusion model based on a particle filtering algorithm, combines the motion state data of the IMU, constructs a dynamic environment map, outputs the spatial relationship between the current position of the user and the surrounding obstacles, and sends the dynamic environment map and the spatial relationship to the trajectory prediction module;

[0010] The trajectory prediction module performs path planning on the dynamic environment map and the spatial relationship through a reinforcement learning algorithm, generates a predicted trajectory of the user's motion, and sends the predicted trajectory to the multi-modal interaction module.

[0011] The multi-modal interaction module outputs voice navigation instructions to the visually impaired through the bone conduction earphone based on the predicted trajectory and the dynamic environment map.

[0012] The collection method of the obstacle distance information is:

[0013] The millimeter wave radar realizes real-time collection of obstacle distance information in the environment by emitting millimeter wave signals of a predetermined frequency and receiving reflected signals.

[0014] The collection method of the ground feature image data is:

[0015] The camera captures the ground image in the environment through an optical lens, and performs image segmentation and feature extraction on the ground image to generate ground feature image data.

[0016] The collection method of the user motion state data is:

[0017] The IMU measures the user's motion state in real time through the combination of an accelerometer, a gyroscope, and a magnetometer.

[0018] The generation of spatio-temporally synchronized multi-modal perception data includes the following steps:

[0019] Step 21: Align the time stamp of the obstacle distance information and the ground feature image data to generate time-synchronized obstacle distance information and ground feature image data.

[0020] Step 22: Align the spatial coordinates of the time-synchronized obstacle distance information and the ground feature image data to generate spatio-temporally synchronized multi-modal perception data.

[0021] The construction of the dynamic environment map and the output of the spatial relationship between the current position of the user and the surrounding obstacles include the following steps:

[0022] Step 31: Input the spatio-temporally synchronized multi-modal perception data as input data, model the probability distribution of the multi-modal perception data through the particle filtering algorithm, and generate obstacle probability distribution data.

[0023] Step 32: The motion state data of the IMU is taken as input data, combined with the obstacle probability distribution data, and the association analysis of the motion state and the obstacle is performed through a particle filtering algorithm to generate dynamic obstacle state data;

[0024] Step 33: The dynamic obstacle state data is taken as input data, and the dynamic update of the environment map is performed through a particle filtering algorithm to generate dynamic environment map data;

[0025] Step 34: The dynamic environment map data is taken as input data, combined with the motion state data of the IMU, and the spatial relationship between the current position of the user and the surrounding obstacles is calculated through a particle filtering algorithm to output user position and obstacle relationship data.

[0026] The dynamic environment map is planned through a reinforcement learning algorithm to generate a predicted trajectory of user motion, comprising the following steps:

[0027] Step 41: The dynamic environment map data and the relationship data are taken as input to construct a reinforcement learning environment state representation and a reward function;

[0028] Step 42: Based on the constructed environment state representation and reward function, an optimal path planning strategy is learned through a deep reinforcement learning algorithm to generate a strategy network model;

[0029] Step 43: The learned strategy network model is applied to the current environment state to generate an initial path trajectory, and fine adjustment is performed through a local optimization algorithm to form a smooth and continuous predicted trajectory of user motion;

[0030] The collection method of the current environment state is:

[0031] First, the current obstacle distance information in the environment is collected in real time through a millimeter wave radar, the current ground feature image data is obtained through a camera, and the current user motion state data is provided by an IMU, and then the spatial relationship between the current position of the user and the surrounding obstacles is generated through a time synchronization module and a dynamic map generation module;

[0032] The way of outputting voice navigation instructions to the visually impaired through bone conduction earphones is:

[0033] The generated predicted trajectory and dynamic environment map are semantically processed and converted into voice navigation instructions executable by the visually impaired, and the bone conduction earphones are taken as output devices to convey voice navigation instructions to the visually impaired.

[0034] Compared with the prior art, the beneficial effects of the present application are:

[0035] This invention addresses the challenges of multi-source heterogeneous data fusion by employing multimodal data acquisition and preprocessing, spatiotemporal synchronization and fusion, achieving deep semantic fusion of radar, visual, and IMU data. It then models obstacle probability distributions to quantify the reliability of perception using probabilistic representation methods. Furthermore, it generates safe and efficient navigation paths through dynamic obstacle state estimation, dynamic environment map construction, calculation of the relationship between user location and obstacles, and reinforcement learning path planning. Finally, it generates voice navigation commands, thus solving the problem of intelligent navigation for visually impaired individuals in dynamic environments. Attached Figure Description

[0036] Figure 1 This is a flowchart of the radar-vision-IMU fusion navigation system for visually impaired individuals in Embodiment 1 of the present invention; Detailed Implementation

[0037] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0038] Example 1

[0039] like Figure 1 As shown, the radar-vision-IMU fusion navigation system for visually impaired individuals includes a multi-source data collection module, a time synchronization module, a dynamic map generation module, a trajectory prediction module, and a multimodal interaction module; the modules are connected electrically.

[0040] Among them, the multi-source data collection module collects obstacle distance information in the environment in real time through millimeter-wave radar, combines it with ground feature image data obtained by the camera and user motion state data provided by the IMU, and sends the obstacle distance information and ground feature image data to the time synchronization module and the user motion state data to the dynamic map generation module.

[0041] The time synchronization module aligns the obstacle distance information of the millimeter-wave radar with the ground feature image data in time and space to generate spatiotemporally synchronized multimodal perception data, and sends the spatiotemporally synchronized multimodal perception data to the dynamic map generation module.

[0042] The dynamic map generation module inputs the spatiotemporally synchronized multimodal perception data into the fusion model based on the particle filter algorithm, combines the motion state data of the IMU to construct a dynamic environment map, outputs the spatial relationship between the user's current position and surrounding obstacles, and sends the dynamic environment map and spatial relationship to the trajectory prediction module.

[0043] The trajectory prediction module plans a path through a reinforcement learning algorithm based on the dynamic environment map and spatial relationship, generates a predicted trajectory of the user's movement, and sends the predicted trajectory to the multi-modal interaction module;

[0044] The multi-modal interaction module outputs voice navigation instructions to the visually impaired person through the bone conduction earphone based on the predicted trajectory and the dynamic environment map.

[0045] In an embodiment of the present application, the obstacle distance information is collected in the following manner:

[0046] The millimeter wave radar realizes real-time collection of obstacle distance information in the environment by emitting millimeter wave signals of a predetermined frequency and receiving reflected signals.

[0047] Specifically, the transmitting port of the millimeter wave radar generates continuous millimeter wave signals in a preset sweep frequency mode. The millimeter wave signals are reflected after encountering an obstacle and are captured by the receiving port of the millimeter wave radar.

[0048] The receiving port mixes the reflected signal with the transmitted signal to generate an intermediate frequency signal. By performing spectral analysis on the intermediate frequency signal, the distance information of the obstacle is extracted. Specifically, the spectral analysis converts the time domain signal into a frequency domain signal through fast Fourier transform (FFT), and the peak value in the frequency domain corresponds to the distance of the obstacle. The output module of the millimeter wave radar outputs the extracted distance information in the form of a data packet to form obstacle distance data.

[0049] It should be noted that the sweep frequency mode of the millimeter wave radar uses the frequency modulated continuous wave (FMCW) technology, and the sweep frequency period and bandwidth are dynamically adjusted according to the environment. The sweep frequency period determines the frequency of data update, and the bandwidth affects the distance resolution. The receiving port of the millimeter wave radar uses a multi-channel design, and each channel independently processes reflected signals in different directions to simultaneously detect multiple target obstacles. The data processing unit of the millimeter wave radar fuses the multi-channel signals to eliminate noise and interference and improve the accuracy of distance measurement.

[0050] In an embodiment of the present application, the obstacle distance data is output in a structured format, including a timestamp, an obstacle distance value, an obstacle direction angle, and a confidence level. The timestamp is used for subsequent synchronization processing of multi-source data, the obstacle distance value represents the straight-line distance between the obstacle and the millimeter wave radar, and the confidence level reflects the reliability of the measurement result.

[0051] Further, the ground feature image data is collected in the following manner:

[0052] The camera captures the ground image in the environment through an optical lens and performs image segmentation and feature extraction on the ground image to generate ground feature image data.

[0053] Specifically, the camera employs a high-resolution sensor to record the texture, color, and geometric features of the ground. The exposure time and aperture size of the camera are automatically adjusted according to the ambient lighting conditions to ensure image quality. The image processing unit of the camera pre-processes the raw images, including denoising, white balance, and distortion correction, to improve the accuracy of image data.

[0054] In embodiments of the present application, the image segmentation separates the ground region from other regions through an edge detection algorithm that employs the Canny operator combined with Gaussian filtering to eliminate noise. The feature extraction stage extracts local feature points of the ground region through the Scale-Invariant Feature Transform (SIFT) or Speeded-Up Robust Features (SURF) algorithm. These feature points include key points and descriptors, which are used for subsequent multi-source data fusion and matching to achieve matching of ground obstacle types such as steps, pits, slopes, etc.

[0055] The output module of the camera encodes the processed image data into a specific format, including timestamps, image frames, and feature point descriptions. The timestamps are synchronized with the data of the millimeter wave radar and IMU, the image frames are compressed grayscale or color images, and the feature point descriptions are vector-form ground feature representations.

[0056] Further, the collection method of the user motion state data is as follows:

[0057] The IMU measures the user's motion state in real time through the combination of an accelerometer, a gyroscope, and a magnetometer.

[0058] Specifically, the accelerometer detects the user's three-axis linear acceleration, the gyroscope measures the user's three-axis angular velocity, and the magnetometer provides the user's direction information. The data fusion unit of the IMU fuses multi-sensor data through the Kalman filtering algorithm to generate the user's attitude, speed, and displacement information.

[0059] The motion state data of the IMU includes the user's current position, motion direction, and speed. The current position is obtained by integrating the acceleration data, the motion direction is determined by combining the magnetometer and gyroscope data, and the speed is obtained by integrating and filtering the acceleration data. The output module of the IMU outputs the motion state data in a structured format, including timestamps, attitude angles, speeds, and displacements. The timestamps are used for multi-source data synchronization, the attitude angles represent the user's pitch, roll, and yaw angles, and the speeds and displacements reflect the user's motion trend.

[0060] In embodiments of the present application, the generation of spatio-temporally synchronized multi-modal perception data includes the following steps:

[0061] Step 21: time stamp alignment of the obstacle distance information and the ground feature image data to generate time-synchronized obstacle distance information and ground feature image data;

[0062] Specifically, the hardware clocks of the millimeter wave radar and the camera are synchronized through a unified network time protocol to ensure that the time bases of the two are consistent. Since the millimeter wave radar and the camera record the time stamps of each frame of data during data acquisition, the time stamps of the two are compared to find the obstacle distance information and the ground feature image data that are closest in time and pair them as data at the same time.

[0063] In further embodiments of the application, if the time stamps have a small deviation less than a preset time difference threshold, a linear interpolation method is used to fine-tune the obstacle distance information to align it with the time stamps of the ground feature image data.

[0064] Step 22: spatial coordinate alignment of the time-synchronized obstacle distance information and ground feature image data to generate spatio-temporally synchronized multi-modal perception data;

[0065] Specifically, the spatial coordinate alignment process includes the following methods:

[0066] Since the coordinate system of the millimeter wave radar is centered on the radar and uses a polar coordinate system to represent the distance and angle of the obstacle, the coordinate system of the camera is centered on the camera and uses an image coordinate system to represent the pixel position of the ground feature. Therefore, to achieve spatial alignment, the polar coordinate data of the millimeter wave radar is first converted to a Cartesian coordinate system, and then the Cartesian coordinates of the millimeter wave radar are mapped to the image coordinate system of the camera through a pre-calibrated extrinsic matrix.

[0067] Specifically, the extrinsic matrix is obtained through an offline calibration process. During calibration, the millimeter wave radar and the camera are fixed at a known position, and multiple sets of calibration data are collected to calculate the rotation and translation relationship between the two. In real-time processing, the extrinsic matrix is used to map the obstacle distance information to the coordinate system of the ground feature image data to ensure that the spatial positions of the two are consistent. If the installation positions of the millimeter wave radar and the camera change, the extrinsic matrix can be updated through an online calibration method.

[0068] It can be understood that the multi-modal perception data includes the obstacle distance information and the ground feature image data after time alignment and spatial alignment. The obstacle distance information provides the accurate distance and position of the obstacles in the environment, and the ground feature image data provides the texture and geometric features of the ground. The combination of the two can more comprehensively describe the environmental state around the user.

[0069] In further embodiments of the present application, the spatio-temporal synchronization module also performs data fusion in the process of generating the multi-modal perception data. For the same spatial position of the obstacle, if both the millimeter wave radar and the camera detect it, the spatio-temporal synchronization module will perform weighted fusion of the data of both, improving the accuracy of detection. If only one sensor (millimeter wave radar or camera) detects the obstacle, it is decided according to the confidence of the sensor whether to retain the data.

[0070] Further, the constructing a dynamic environment map, outputting the spatial relationship between the current position of the user and the surrounding obstacles comprises the following steps:

[0071] Step 31: taking the spatio-temporally synchronized multi-modal perception data as input data, modeling the probability distribution of the multi-modal perception data through a particle filter algorithm, and generating obstacle probability distribution data;

[0072] It should be noted that the core of the particle filter algorithm is to approximate the posterior probability distribution of the state through a set of randomly sampled particles. In embodiments of the present application, each particle represents a possible obstacle state hypothesis, including the position, velocity and shape of the obstacle, etc. The obstacle distance information in the multi-modal perception data is collected by the millimeter wave radar, providing the distance and azimuth angle of the obstacle; the ground feature image data is collected by the camera, providing the visual features and texture information of the obstacle. The particle filter algorithm fuses these multi-modal data, gradually optimizes the distribution of particles through the steps of weight update and resampling, and thus approximates the true state of the obstacle.

[0073] In the specific implementation of the present step, the particle filter algorithm first initializes a set of particles, whose state space covers the possible obstacle position and motion range. Each particle performs state prediction and observation update according to the multi-modal perception data. In the state prediction stage, the position and velocity of the particle are predicted using the motion model; in the observation update stage, the weight of the particle is updated by calculating the matching degree of the particle and the multi-modal perception data. The calculation of the matching degree combines the geometric constraints of the obstacle distance information and the visual similarity of the ground feature image data. Finally, the particle filter algorithm retains high-weight particles and eliminates low-weight particles through the resampling step by a pre-set weight threshold or a pre-determined number of particles, generating obstacle probability distribution data. The obstacle probability distribution data represents the distribution of the obstacle in the environment in the form of a probability density function.

[0074] Step 32: taking the motion state data of the IMU as input data, combining the obstacle probability distribution data, and performing correlation analysis of the motion state and the obstacle through a particle filter algorithm, to generate dynamic obstacle state data;

[0075] Specifically, the particle filter algorithm adopts a double-layer structure design in this step. The outer layer particles represent the user state, and the inner layer particles represent the obstacle state. First, according to the acceleration, angular velocity and attitude data provided by the IMU, the user state is predicted by the state transition equation. The state transition equation adopts an adaptive combination of constant velocity model and constant acceleration model, and dynamically adjusts the weight proportion of the two models according to the stability of the IMU data.

[0076] In the state prediction stage, the following processing is applied to each user state particle: first, extract the linear acceleration component and angular velocity component in the IMU data; second, calculate the displacement and rotation change of the user in three-dimensional space by time integration; then, apply the calculated displacement and rotation change to the current state of the particle to generate a predicted state; finally, add random noise conforming to Gaussian distribution to simulate IMU measurement error and environmental interference.

[0077] The observation update stage adopts a hierarchical likelihood calculation strategy. First, calculate the spatial relationship between the user state particle and the obstacle probability distribution, including the distance to the nearest obstacle, the relative motion direction and the velocity difference. Then, according to the preset safety distance threshold and motion consistency index, calculate the observation likelihood of each user state particle. The likelihood function design considers three key factors: first, the spatial distance between the user and the obstacle, the closer the distance, the higher the likelihood value; second, the consistency of the relative motion direction, the more consistent the motion direction, the higher the likelihood value; third, the velocity matching degree, the smaller the velocity difference, the higher the likelihood value.

[0078] In further embodiments of the present application, an adaptive weight adjustment mechanism is also introduced. When the IMU data quality is high, the weight of the state prediction result is increased; when the obstacle probability distribution data quality is high, the weight of the observation update result is increased. Data quality evaluation is based on variance analysis of IMU data and entropy calculation of obstacle probability distribution.

[0079] The resampling stage of the particle filter algorithm adopts a system residual resampling algorithm, which maintains particle diversity while ensuring that high-weight particles are retained. Specifically, the system first calculates the number of effective particles, and triggers resampling when the number of effective particles is below a preset threshold; for example, during the resampling process, the top 20% of particles in terms of weight are retained, and the remaining 80% of particles are randomly sampled, with the sampling probability being proportional to the particle weight.

[0080] The final generated dynamic obstacle state data at least contains the following information: the three-dimensional position coordinates and their uncertainty ellipsoid of the obstacle; the three-dimensional velocity vector and its covariance matrix of the obstacle; the geometric size estimation of the obstacle; the motion trend prediction of the obstacle, including the possible future trajectory and its probability distribution; and the relative position relationship and collision risk assessment of the obstacle and the user.

[0081] Step 33: input the dynamic obstacle state data into the particle filter algorithm to update the environment map dynamically and generate dynamic environment map data;

[0082] Specifically, in the embodiments of the present application, a hybrid representation method combining grid maps and vector maps is adopted. The grid map represents the environment with fixed resolution grids, and each grid cell stores the obstacle existence probability. The vector map represents the shape, position and motion characteristics of the obstacle with geometric primitives. The particle filter algorithm establishes a mapping relationship between the two representations to realize the bidirectional flow and update of information.

[0083] Specifically, in the initialization phase of map updating, the pre-constructed static environment map is first loaded as prior knowledge, and the active map area is determined according to the current perception range. Each map particle represents a possible state hypothesis of the environment, including grid occupancy probability and dynamic object list. Each map particle is assigned an initial weight, and the weight assignment is based on the consistency score of the prior map and the current perception data.

[0084] The state prediction phase adopts a hierarchical prediction strategy. For the static environment part, the grid occupancy state is kept unchanged; for the dynamic obstacle part, the position and attitude of the obstacle at the next time are predicted according to the speed and acceleration information in the dynamic obstacle state data.

[0085] In the observation update phase, the latest dynamic obstacle state data is compared with the predicted state of the map particle to calculate the observation likelihood. The design of the likelihood function of this observation likelihood needs to consider factors including but not limited to the following: 1) obstacle position matching degree, evaluated by calculating the Mahalanobis distance between the observed position and the predicted position; 2) obstacle shape consistency, evaluated by calculating the similarity of the shape descriptor; 3) motion pattern matching degree, evaluated by comparing the consistency of the actual motion and the predicted motion. The weight of each map particle is updated according to the calculated likelihood value.

[0086] In the resampling phase, an adaptive resampling strategy is adopted to determine whether to perform resampling by calculating the normalized effective sample size. When the effective sample size is below the threshold, the resampling operation is performed to retain high-weight particles and eliminate low-weight particles. To avoid particle degeneracy, a small random disturbance is added to the particle state after resampling to maintain particle diversity.

[0087] In the map fusion phase, the states of all map particles are weighted and averaged to generate the final dynamic environment map. For the grid part, the weighted occupancy probability of each grid is calculated; for the vector part, similar dynamic objects are merged, and the weighted average values of their position, speed and shape are calculated.

[0088] Thus, the finally generated dynamic environment map data contains at least the following information: grid occupancy probability distribution of the environment; current position, velocity and shape of dynamic obstacles; predicted trajectory and uncertainty of dynamic obstacles; passable and dangerous area identification in the environment; and map reliability evaluation index indicating the update time and credibility of each part of the map

[0089] Step 34: input the dynamic environment map data as input data, combine the motion state data of the IMU, calculate the spatial relationship between the user's current position and the surrounding obstacles by a particle filtering algorithm, and output the user position and obstacle relationship data.

[0090] Specifically, a multi-scale user positioning strategy is adopted, and the user is modeled as an entity with a certain spatial range, rather than a single point. On this basis, a user-environment interaction model is constructed, and each particle represents a user position, attitude and motion state hypothesis.

[0091] In the initialization stage, a set of user state particles is generated in the possible state space according to the initial position and attitude information provided by the IMU. Each particle contains the user's three-dimensional position coordinates, attitude represented by Euler angles, linear velocity and angular velocity. Equal initial weights are assigned to each particle, and the uncertainty range of the initial state is set according to the reliability of the IMU data.

[0092] The adaptive motion model is used in the state prediction stage. First, the acceleration, angular velocity and attitude information is extracted from the IMU data, and the user's position change is predicted by the inertial navigation algorithm. The prediction model takes into account the differences in characteristics under different motion modes, such as stationary state, uniform motion and accelerated motion, and automatically selects the most suitable prediction model through a motion state recognition algorithm.

[0093] In the observation update stage, the user state particles are matched with the dynamic environment map, and the observation likelihood is calculated. In the embodiment of the present application, the likelihood function of the observation likelihood is based on at least the following key factors: first, spatial consistency, evaluating whether the user's position is consistent with the passable area in the environment map; second, sensor observation consistency, comparing the difference between the actual perception data and the data expected to be observed based on the current position hypothesis; third, motion constraint consistency, checking whether the user's motion conforms to the physical constraints and environmental restrictions. The weight of each user state particle is updated according to the calculated likelihood value.

[0094] The low-variance resampling algorithm is used in the sampling step, which maintains the diversity of particle distribution while preserving high-weight particles. The system sets a dynamic resampling threshold, and determines the resampling frequency according to the entropy value of the particle weight distribution, avoiding the problem of particle degradation caused by excessive resampling. After resampling, the system performs local perturbation on the particles to simulate the randomness of user motion and the uncertainty of IMU measurement.

[0095] The user-obstacle relationship calculation stage estimates the optimal position and posture of the user based on the weighted particle set. In the specific implementation of the present application, at least the following relationship indicators are calculated for each obstacle: first, the closest distance, representing the shortest Euclidean distance between the user and the surface of the obstacle; second, the relative orientation, describing the directional angle of the obstacle with the user as the reference system; third, the relative velocity, calculating the velocity difference vector between the user and the obstacle; fourth, the collision time prediction, estimating the time of potential collision based on the current motion state; and fifth, the risk level assessment, evaluating the risk level by integrating the distance, velocity, and environmental complexity.

[0096] Therefore, the final output of the user position and obstacle relationship data at least contains the following information: the precise three-dimensional position and uncertainty ellipsoid of the user; the motion state of the user, including velocity, acceleration, and motion trend; the relative position, distance, and azimuth angle of the surrounding obstacles; the motion state of the obstacles and the relative motion relationship with the user; the potential collision risk assessment and warning information; and the passable area;

[0097] Further, the dynamic environment map is subjected to path planning through a reinforcement learning algorithm to generate a predicted trajectory of user motion, which includes the following steps:

[0098] Step 41: The dynamic environment map data and the relationship data are taken as inputs to construct a reinforcement learning environment state representation and a reward function;

[0099] Specifically, the dynamic environment map data and the user position obstacle relationship data are converted into a state space representation that can be processed by the reinforcement learning algorithm, and a reward function is designed to guide the agent to learn the optimal path planning strategy. The environment state representation not only contains static spatial information, but also integrates dynamic obstacle motion characteristics and user behavior patterns.

[0100] In specific implementation, a multi-level state representation structure is adopted. The basic layer adopts a grid representation, dividing the environment into uniform grids, and each grid cell stores static attributes such as passability, obstacle probability, and terrain characteristics; the middle layer represents dynamic information, including the current position, velocity, and predicted trajectory of dynamic obstacles; and the top layer represents user-related information, including the user's current position, motion state, and historical trajectory features.

[0101] The state space construction stage first determines an appropriate spatial resolution to balance the representation accuracy and computational complexity. For indoor environments, a high-resolution representation is adopted, such as 0.2 meters per grid; for outdoor open environments, the system adopts a lower resolution representation, such as 0.5 meters per grid. The continuous environment map is discretized into a state grid, and each grid cell is assigned a pass value. The pass value considers obstacle probability, terrain complexity, and pass difficulty.

[0102] And the reward function in the practical implementation of the application, using a multi-objective combination strategy, at least contains the following reward factors: one is the safety reward, which is proportional to the distance between the user and the obstacle, the farther the reward is higher; two is the efficiency reward, which is inversely proportional to the path length and travel time, the shorter the path the higher the reward; three is the smooth reward, which encourages continuous smooth motion trajectory and avoids frequent turning and speed change; four is the goal-oriented reward, which is proportional to the progress of the user towards the target direction; five is the environmental adaptation reward, which encourages path selection in good traffic conditions. The design of the specific reward function is set according to the actual demand, and the application does not limit it;

[0103] In further embodiments, an adaptive reward weight adjustment mechanism can also be designed, for example, in dangerous areas, increase the weight of safety reward; in time-critical situations, increase the weight of efficiency reward; in complex environment, increase the weight of smooth reward. Reward weight adjustment is based on environmental complexity evaluation, user preference settings and current task requirements;

[0104] Step 42: based on the constructed environment state representation and reward function, the optimal path planning strategy is learned by deep reinforcement learning algorithm to generate a policy network model;

[0105] Specifically, the deep reinforcement learning algorithm is a double network architecture, including policy network and value network. The policy network is responsible for selecting the optimal action according to the current state, using a fully connected neural network structure, the input layer receives the environment state representation, the hidden layer uses ReLU activation function for feature extraction, and the output layer generates the probability distribution of each possible action; the value network is responsible for evaluating the value of state-action pairs, guiding the optimization direction of policy;

[0106] A convolutional layer is introduced in front of the policy network to extract spatial features. The convolutional layer uses a 3x3 convolutional kernel to extract local features and spatial patterns of the environment map; add a pooling layer to reduce the feature dimension and retain key information; add a fully connected layer to integrate spatial features and user state information to generate a comprehensive state representation. The network structure of the policy network is optimized for path planning tasks, and the number of neurons in the middle layer is dynamically adjusted according to the complexity of the environment.

[0107] The deep reinforcement learning algorithm uses a priority experience replay mechanism. By maintaining an experience pool, it stores historical experience tuples (state, action, reward, next state) of the agent interacting with the environment.

[0108] The final learned policy network model can generate optimal path planning decisions in real time according to the current environment state and user location.

[0109] Step 43: Apply the learned policy network model to the current environment state to generate an initial path trajectory, and fine-tune it through a local optimization algorithm to form a smooth and continuous predicted trajectory of user motion;

[0110] Specifically, the current environment state is collected in the following way:

[0111] First, real-time obstacle distance information in the environment is collected by a millimeter wave radar, current ground feature image data is obtained by a camera, and current user motion state data is provided by an IMU. Then, the user's current position and the spatial relationship with surrounding obstacles are generated through a time synchronization module and a dynamic map generation module;

[0112] Further, the way to form a smooth and continuous predicted trajectory of user motion is:

[0113] The current environment state is input into the policy network to obtain an initial path planning from the current position to the target position. The initial path is represented by a series of path points, each containing expected position coordinates and arrival time. It can be understood that the path output by the policy network has considered obstacle avoidance and efficiency optimization;

[0114] In a further preferred embodiment, since the generated predicted trajectory of user motion may have local non-smoothness or redundancy problems, a cubic spline interpolation algorithm can also be used to smooth the predicted trajectory of user motion;

[0115] Specifically, discrete path points are used as control points to generate a continuous parametric curve representation. Spline interpolation ensures the continuity of position, first-order derivative (velocity continuity) and second-order derivative (acceleration continuity) at the control points, generating a smooth trajectory that meets kinematic constraints.

[0116] Further, to adapt to dynamic environmental changes, the environment state is also continuously monitored. When significant changes are detected, such as the appearance of new obstacles or changes in the trajectory of existing obstacles, local trajectory re-planning is triggered. Local re-planning only modifies the affected trajectory segment, leaving the rest unchanged, reducing computational overhead and ensuring trajectory continuity.

[0117] Thus, the final generated predicted trajectory of user motion is represented by a time-parameterized three-dimensional curve, containing the following key information: the spatial geometry of the trajectory, describing the user's expected path in the environment; the velocity curve, describing the user's motion speed variation on the trajectory; key time points, marking important events such as turning points and obstacle avoidance points on the trajectory; and uncertainty estimates of the trajectory, indicating the credibility of the prediction over time.

[0118] Further, the way to output voice navigation instructions to visually impaired people through bone conduction headphones is:

[0119] The generated prediction trajectory is semantically processed with the dynamic environment map, converted into voice navigation instructions executable by the visually impaired, and the bone conduction earphone is used as the output device to convey the voice navigation instructions to the visually impaired.

[0120] Specifically, the prediction trajectory is decomposed into a series of key navigation events, including straight sections, turning points, obstacle avoidance actions, and target arrival; secondly, combined with the dynamic environment map information, the environment context description is added for each navigation event; for example, a series of navigation instruction templates are preset, such as "turn {left / right} at X meters ahead", "be careful, there is {obstacle type} ahead", etc.; finally, the system converts the navigation events into structured voice instruction templates according to the user's cognitive characteristics and navigation needs.

[0121] The above embodiments are only used to illustrate the technical method of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical method of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical method of the present application.

Claims

1. A radar-vision-IMU fused navigation system for visually impaired people, characterized in that, The system comprises a multi-source data collection module, a time synchronization module, a dynamic map generation module, a trajectory prediction module, and a multi-modal interaction module. The multi-source data collection module collects obstacle distance information in real time through a millimeter wave radar, combines ground feature image data obtained by a camera, and user motion state data provided by an IMU, and sends the obstacle distance information and the ground feature image data to the time synchronization module, and sends the user motion state data to the dynamic map generation module. The time synchronization module performs space-time alignment on the obstacle distance information and the ground feature image data of the millimeter wave radar, generates space-time synchronized multi-modal perception data, and sends the space-time synchronized multi-modal perception data to the dynamic map generation module. The dynamic map generation module inputs the space-time synchronized multi-modal perception data into a fusion model based on a particle filtering algorithm, combines the motion state data of the IMU, constructs a dynamic environment map, outputs the spatial relationship between the current position of the user and the surrounding obstacles, and sends the dynamic environment map and the spatial relationship to the trajectory prediction module. The trajectory prediction module performs path planning on the dynamic environment map and the spatial relationship through a reinforcement learning algorithm, generates a predicted trajectory of the user's motion, and sends the predicted trajectory to the multi-modal interaction module. The multi-modal interaction module outputs voice navigation instructions to the visually impaired through bone conduction earphones based on the predicted trajectory and the dynamic environment map.

2. The radar-vision-IMU fused visual impaired navigation system of claim 1, wherein, The collection method of the obstacle distance information is as follows: The millimeter wave radar transmits millimeter wave signals of a predetermined frequency and receives reflected signals to realize real-time collection of obstacle distance information in the environment.

3. The radar-vision-IMU fused visual impaired navigation system of claim 2, wherein, The collection method of the ground feature image data is as follows: The camera captures ground images in the environment through an optical lens, and generates ground feature image data by performing image segmentation and feature extraction on the ground images.

4. The radar-vision-IMU fused visual impaired navigation system of claim 3, wherein, The collection method of the user motion state data is as follows: The IMU measures the motion state of the user in real time through the combination of an accelerometer, a gyroscope, and a magnetometer.

5. The radar-vision-IMU fused visual impaired navigation system of claim 4, wherein, The generation of space-time synchronized multi-modal perception data includes the following steps: Step 21: Align the time stamps of the obstacle distance information and the ground feature image data to generate time-synchronized obstacle distance information and ground feature image data; Step 22: Align the spatial coordinates of the time-synchronized obstacle distance information and the ground feature image data to generate space-time synchronized multi-modal perception data.

6. The radar-vision-IMU fused visual impaired navigation system of claim 5, wherein, The construction of the dynamic environment map and the output of the spatial relationship between the current position of the user and the surrounding obstacles include the following steps: Step 31: Input the space-time synchronized multi-modal perception data as input data, model the probability distribution of the multi-modal perception data through a particle filtering algorithm, and generate obstacle probability distribution data; Step 32: Input the motion state data of the IMU as input data, combine the obstacle probability distribution data, and perform correlation analysis of the motion state and the obstacles through a particle filtering algorithm to generate dynamic obstacle state data; Step 33: input the dynamic obstacle state data into the particle filter algorithm to dynamically update the environment map and generate dynamic environment map data; Step 34: input the dynamic environment map data into the particle filter algorithm to calculate the spatial relationship between the user's current position and the surrounding obstacles based on the IMU motion state data, and output the user position and obstacle relationship data.

7. The radar-vision-IMU fused visual impaired navigation system of claim 6, wherein, The dynamic environment map is updated through the particle filter algorithm, and the dynamic environment map data is input into the particle filter algorithm to calculate the spatial relationship between the user's current position and the surrounding obstacles based on the IMU motion state data, and output the user position and obstacle relationship data. The dynamic environment map is updated through the particle filter algorithm, and the dynamic environment map data is input into the particle filter algorithm to calculate the spatial relationship between the user's current position and the surrounding obstacles based on the IMU motion state data, and output the user position and obstacle relationship data. The dynamic environment map is updated through the particle filter algorithm, and the dynamic environment map data is input into the particle filter algorithm to calculate the spatial relationship between the user's current position and the surrounding obstacles based on the IMU motion state data, and output the user position and obstacle relationship data. The dynamic environment map is updated through the particle filter algorithm, and the dynamic environment map data is input into the particle filter algorithm to calculate the spatial relationship between the user's current position and the surrounding obstacles based on the IMU motion state data, and output the user position and obstacle relationship data.

8. The radar-vision-IMU fused visual impaired navigation system of claim 7, wherein, The dynamic environment map is updated through the particle filter algorithm, and the dynamic environment map data is input into the particle filter algorithm to calculate the spatial relationship between the user's current position and the surrounding obstacles based on the IMU motion state data, and output the user position and obstacle relationship data. The dynamic environment map is updated through the particle filter algorithm, and the dynamic environment map data is input into the particle filter algorithm to calculate the spatial relationship between the user's current position and the surrounding obstacles based on the IMU motion state data, and output the user position and obstacle relationship data.

9. The radar-vision-IMU fused visual impaired navigation system of claim 8, wherein, The dynamic environment map is updated through the particle filter algorithm, and the dynamic environment map data is input into the particle filter algorithm to calculate the spatial relationship between the user's current position and the surrounding obstacles based on the IMU motion state data, and output the user position and obstacle relationship data. The dynamic environment map is updated through the particle filter algorithm, and the dynamic environment map data is input into the particle filter algorithm to calculate the spatial relationship between the user's current position and the surrounding obstacles based on the IMU motion state data, and output the user position and obstacle relationship data.

Images