Visually impaired people barrier-free travel multi-source perception aid decision-making method and system

Abstract

The invention discloses a visual impaired people barrier-free travel multi-source perception aid decision-making method and system, and belongs to the technical field of barrier-free travel. The system comprises an intelligent wearable device end and a public carrier end. The wearable device end adopts a visual SLAM and UWB-IMU tight coupling algorithm to realize hybrid positioning; and the public carrier end generates a high-precision local point cloud by adopting LiDAR-SLAM, and broadcasts the high-precision local point cloud to the outside after carrying out lightweight processing on the high-precision local point cloud. And when the user approaches, the wearable device end receives and matches the point cloud, and calculates the accurate relative pose with the vehicle door of the carrier. And finally, the system generates a three-dimensional audio signal with clear spatial directivity by using a head-related transfer function model based on the pose, so as to realize visual guidance for the user. According to the invention, through human-vehicle cooperative sensing and positioning, full-scene seamless navigation and centimeter-level accurate docking are realized, and through visual spatial audio guidance, the travel safety and convenience of visually impaired people are significantly improved.

Description

TECHNICAL FIELD

[0002] The present application relates to the technical field of intelligent wearable devices and barrier-free travel, and in particular to a multi-source perception assisted decision-making method for barrier-free travel of visually impaired people, which integrates multi-sensor environment perception, high-precision spatial positioning, human-computer collaboration and decision-making assistance technology. BACKGROUND

[0003] Visually impaired people face great challenges in their daily travel. Existing assistive technologies, such as guide canes, guide dogs or GPS-based navigation applications, although provide some help, still have many limitations.

[0004] Firstly, in terms of positioning accuracy and continuity, GPS signals will be severely attenuated or even interrupted in indoor, tunnel or "urban canyon" with high-rise buildings, leading to navigation failure. Inertial navigation relying only on inertial measurement units (IMU) will produce rapidly accumulated errors due to sensor drift, and cannot maintain accurate positioning for a long time. This makes visually impaired people often face the "breakpoint" of positioning service interruption when switching between indoor and outdoor scenes, increasing the uncertainty and risk of travel.

[0005] Secondly, in terms of environmental perception ability, the detection range of guide cane is limited to a very small area in front of the user, and cannot perceive distant obstacles or high-speed approaching dynamic hazards (such as vehicles). Traditional navigation applications can only provide route guidance, and cannot perceive and warn the complex dynamic environment around the user in real time.

[0006] Thirdly, in terms of interaction with public transportation, the "last mile" problem faced by visually impaired people is particularly prominent. Even if the user arrives at the bus stop through navigation, it is still a great challenge to accurately and quickly find the correct bus that is approaching the stop and locate the position of the bus door. Existing technologies lack effective means to achieve centimeter-level precise docking between the user and the moving vehicle.

[0007] Finally, in terms of human-computer interaction, traditional voice or vibration prompts are often abstract and offline, such as "turn right 50 meters ahead", which requires the user to perform mental calculation and spatial imagination, increasing cognitive load. How to provide a more intuitive and more in line with human perception habits non-visual guidance method is the key to improving navigation efficiency and safety.

[0008] Therefore, there is an urgent need for a new integrated travel assistance solution that can solve the above problems and provide visually impaired people with truly seamless, high-precision and high-safety travel assistance in all scenarios. SUMMARY

[0009] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multi-source perception-assisted decision-making method and system for barrier-free travel for visually impaired individuals, aiming to solve one or more of the following core technical problems: 1. How to achieve seamless, continuous, and high-precision positioning for visually impaired individuals in complex environments, including indoors, outdoors, and transitional zones between the two.

[0010] 2. How to integrate multi-source heterogeneous data from the user's micro-perspective (first-person perception) and the public vehicle's macro-perspective (external environment perception) to construct a unified and comprehensive environmental model and improve the breadth and depth of environmental perception.

[0011] 3. How to solve the "last mile" problem when taking public transportation and achieve centimeter-level precise connection between users and moving bus doors.

[0012] 4. How to provide an efficient, intuitive, and visually independent human-computer interaction channel that transforms complex navigation and environmental information into the user's intuitive perception, thereby reducing cognitive load and improving decision-making and navigation efficiency.

[0013] To address the above problems, this invention provides a multi-source perception-assisted decision-making method for barrier-free travel for visually impaired individuals, comprising the following steps: S1: Hybrid Localization and Pose Estimation: The processor deployed on the user's smart wearable device adaptively executes at least one of the following localization modes based on the user's environment to obtain the user's pose in real time: a1) Indoor positioning mode: When the smart wearable device is in an indoor environment, it initiates a monocular vision-based simultaneous localization and mapping (Mono-SLAM) algorithm. This algorithm fuses image data collected by the smart wearable device with inertial measurement unit (IMU) data, and estimates the user's pose state vector x by iteratively executing the following state prediction and observation update steps: State prediction: The prediction model of the algorithm is based on a continuous-time state equation x that describes the dynamics of the system. · =f(x,u)+w; where x is a state vector containing the user's 3D position p, attitude quaternion q, and IMU zero bias, x · Let x be the first derivative of the state vector x with respect to time, u be the IMU measurement input, and w be the continuous process noise. The processor discretizes this continuous-time state equation to determine the state x from the previous time step. k-1 and the current IMU input u k The predicted state x at the current time is calculated. k ; Observation update: z = h(x) + v; where z is the pixel coordinate of the image feature point, h(x) is the reprojection function that projects the map point onto the image plane, and v is the observation noise; a2) Outdoor positioning mode: when the smart wearable device end is in an outdoor environment, an ultra-wideband (UWB) and IMU tightly coupled positioning algorithm based on extended Kalman filtering is started, which is used to process nonlinear problems in system state transition and observation, and through the following prediction-update cycle, UWB ranging data and IMU data are fused to output high-precision user pose estimation: State prediction: based on IMU data u k The described nonlinear kinematic model is used to perform integration on the posterior optimal state estimate x' k-1 of the previous time to obtain the prior state estimate x' k - , that is, x' k - = g(x' k-1 , u k ); wherein the state transition function g is obtained by numerically integrating a continuous-time model describing the physical dynamics of the system; Measurement update: when receiving UWB ranging data z k , the prior state estimate x' k - is mapped to the observation space through a nonlinear observation model h, and the measurement residual is calculated; then, the nonlinear observation model h is linearized through the Jacobian matrix H k , the Kalman gain K k is calculated, and the prior estimate is corrected to obtain the posterior optimal state estimate x' k , that is, x' k = x' k - + K k (z k - h(x' k - )); S2: Public vehicle cooperative positioning and precise docking: when the user approaches the target public vehicle, the following sub-steps are executed to achieve centimeter-level precise docking: S21: Receive broadcast data: the smart wearable device end receives the local environment point cloud q j broadcasted by the public vehicle end, which is generated by laser radar simultaneous localization and mapping (LiDAR-SLAM); S22: Perform point cloud scan matching: the processor of the smart wearable device end matches the current frame local point cloud p i perceived by itself through visual SLAM with the received local environment point cloud q jThe registration is performed by iteratively optimizing a following objective function to calculate the accurate relative pose transformation matrix T' of the smart wearable device end relative to the public carrier end: ; Wherein, argmin is an optimization operation, that is, to find the variable that can make the following objective function reach the minimum value, and the variable here is T, T is a pose transformation matrix, and ° is a rigid body transformation operator; S23: solving guidance parameters: based on the calculated accurate relative pose transformation matrix T', the guidance parameter vector describing the accurate relative position and direction of the user relative to the public carrier door is solved; The scheme adopts a UWB (Ultra Wide Band) and IMU (Inertial Measurement Unit) tightly coupled fusion positioning scheme based on Kalman filter, fully utilizes the high frequency dynamic response characteristics of IMU and the absolute position constraint advantage of UWB, effectively suppresses the integral drift error of IMU and the non-line-of-sight (NLOS) error of UWB in complex environment through data complementation, and thus provides stable and high-precision pose estimation for path planning and safety zone guidance; S3: spatial audio decision guidance: the processor of the smart wearable device end receives the guidance parameter vector solved by step S23, converts the vector into azimuth angle θ and pitch angle φ in spherical coordinate system, and generates three-dimensional audio signals with clear spatial directionality by using a head-related transfer function (HRTF) model, and outputs the signals to the user through earphones, and the core formula of the HRTF model is: ; Wherein, H L is the filter function of the left ear, f is the frequency, A n and τ n are the amplitude and delay of the sound wave path n respectively.

[0014] By simulating a three-dimensional sound field, key navigation instructions and environmental information are converted into spatial orientation prompts that can be directly perceived from hearing, thereby liberating the user's vision and reducing cognitive load.

[0015] As preferred, in the hybrid positioning and pose estimation step, an indoor and outdoor positioning mode switching sub-step is further included: The processor monitors the GPS (Global Positioning System) signal strength, UWB anchor point signal strength, and the positioning confidence of the simultaneous localization and mapping algorithm; when it is detected that the GPS signal is stably enhanced to be greater than the GPS signal threshold and the positioning confidence is decreased to be lower than the confidence threshold due to insufficient environmental texture, it is smoothly switched from the indoor positioning mode to the outdoor positioning mode.

[0016] As preferred, the common vehicle end executes the process of LiDAR-SLAM, which includes: aligning consecutive LiDAR frames to compute odometry constraints T ij ; and further globally optimizes all keyframe poses X = {T1, T2,..., T n} through pose graph optimization to eliminate accumulated errors, the objective function of the pose graph optimization being: wherein argmin is an optimization operation, i.e., finding a variable that can make the following objective function minimum, the variable here being X, i.e., the pose set of all keyframes, X opt is the final result of this optimization process, i.e., the X that can make the objective function minimum, Σ ij is the covariance matrix of the constraint, T i represents the global pose of the i-th keyframe (or node), T j represents the global pose of the j-th keyframe (or node), T ij represents the relative pose transformation measurement from T i to T j . By globally optimizing the poses through graph optimization (such as g2o or Ceres), accumulated errors are eliminated.

[0017] As preferred, the common vehicle end further executes a point cloud lightening processing step before broadcasting the local environment point cloud, which includes spatial range clipping and / or density down-sampling, the spatial range clipping being to retain only the point cloud data within a preset radius range centered on the common vehicle; the density down-sampling being to reduce the point cloud density to below a preset threshold to adapt to the bandwidth limitation of wireless broadcasting.

[0018] As preferred, the multi-source perception assisted decision-making method for barrier-free travel of visually impaired people further includes a multi-modal perception fusion and environment modeling step, which is located after step S1 and before steps S2 and S3, specifically: fusing the user's first perspective visual information collected by the intelligent wearable device end and the external perspective LiDAR point cloud information collected by the common vehicle end, constructing a unified environment model containing static obstacle and dynamic obstacle information in a unified global coordinate system, and performing path planning and danger warning based on the model.

[0019] A multi-source perception assisted decision-making system for barrier-free travel of visually impaired people, which runs the multi-source perception assisted decision-making method as described above, comprising: an intelligent wearable device end, including a camera, an inertial measurement unit (IMU), an ultra-wideband (UWB) module, a processor, and an audio output device, the processor being configured to: b1) performing the hybrid localization and pose estimation step as claimed in claim 1; b2) receiving the local environment point cloud broadcasted by the public vehicle end and performing the point cloud scan matching as claimed in claim 1 to calculate the accurate relative pose with the public vehicle; b3) performing the spatial audio decision guidance step as claimed in claim 1 to generate a three-dimensional audio guidance signal; a public vehicle end mounted on a public transportation vehicle, comprising a LiDAR sensor, a UWB anchor point and a processor, the processor being configured to: c1) running a LiDAR-SLAM algorithm to generate and update a high-precision local environment point cloud map in real time; c2) periodically broadcasting the local environment point cloud map to the outside through a wireless communication module.

[0020] Preferably, the processor of the smart wearable device end is internally configured to include: a monocular vision SLAM module for estimating user pose in indoor localization mode by solving the state prediction equation x = f(x, u) + w and the observation update equation z = h(x) + v; a UWB-IMU tightly coupled localization module for performing Kalman filtering algorithm based on the prediction equation x' k - = f(x k-1 , u k ) and the update equation x k = x k - + K k (z k - h(x k - )) in outdoor localization mode.

[0021] Preferably, the processor of the smart wearable device end is internally further configured to include: a point cloud matching module for realizing accurate docking with the public vehicle by solving an optimization objective function; a spatial audio engine module for generating guidance audio based on head-related transfer functions.

[0022] The substantial effects brought by the present application are: 1. Achieving centimeter-level accurate docking: through the unique human-vehicle collaborative localization and point cloud matching mechanism, the present application can calculate the centimeter-level accurate relative pose between the user and the bus door, effectively solving the "last meter" problem of visually impaired people getting on the bus, greatly improving the convenience and safety of taking public transportation.

[0023] 2. Provides seamless navigation in full scene: The application ensures the continuity and high precision of positioning service in the whole process from indoor to outdoor walking and getting on the bus by intelligent indoor and outdoor positioning mode switching mechanism and robust UWB-IMU tight coupling fusion algorithm, avoiding the "break point" problem of traditional navigation mode when switching scenes.

[0024] 3. Improve the safety of environmental perception: The application not only relies on the user's personal wearable device, but also creatively uses the more powerful LiDAR sensor on the public carrier to build a unified environmental model that integrates the first and external perspectives. This "all-seeing perspective" supplement allows users to perceive dangers that their sensors cannot detect, such as far away or blocked, achieving double safety redundancy.

[0025] 4. Reduce the user's cognitive load: The application uses HRTF-based spatial audio technology to convert abstract direction and distance instructions into sounds that users can intuitively perceive, as if from the real-world target location. Users do not need to think, but can follow the guidance by auditory instinct, greatly reducing cognitive burden and making the navigation process more natural and efficient.

[0026] 5. Ensure the engineering feasibility of the scheme: By lightening the point cloud broadcast on the public carrier side, the application effectively reduces the requirement for wireless communication bandwidth while ensuring the docking accuracy, ensuring that the entire cooperative positioning system can operate stably in high dynamic scenes with low latency and high frequency. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a multi-source perception auxiliary decision system structure diagram for barrier-free travel of visually impaired people of the application; Figure 2 is a multi-source perception auxiliary decision method flow chart for barrier-free travel of visually impaired people of the application; In the figure: 100 - intelligent wearable device side; 101 - camera; 102 - inertial measurement unit; 103 - ultra-wideband module; 104 - processor; 105 - audio output device; 200 - public carrier side; 201 - laser radar sensor; 202 - UWB anchor point; 203 - carrier side processor; 204 - wireless communication module. DETAILED DESCRIPTION

[0028] The technical solutions of the application will be further specifically described below through examples and in combination with the drawings.

[0029] Example: Refer to Figure 1The visual impaired person barrier-free travel multi-source perception assisted decision system of the embodiment mainly consists of two parts: an intelligent wearable device end 100 and a public vehicle end 200, and can further include a cloud service platform storing a high-precision map, which updates the map of the public vehicle end and the intelligent wearable device end in a timely manner or on demand.

[0030] The intelligent wearable device end 100 can be a pair of smart glasses in the embodiment. The smart glasses are integrated with a camera 101 for collecting image data, an inertial measurement unit (IMU) 102 containing an accelerometer and a gyroscope for measuring the attitude and motion of the device, an ultra-wideband (UWB) module 103 for high-precision ranging, a processor 104 serving as the brain of the device end and being responsible for running algorithms, and an audio output device 105 (such as a bone conduction earphone) for providing audio guidance to the user.

[0031] The public vehicle end 200 is installed on a public transportation vehicle such as a bus. The public vehicle end 200 is deployed with a LiDAR sensor 201 for scanning the surrounding environment to generate a high-precision point cloud map, a UWB anchor point 202 for cooperative positioning with the user device, a vehicle end processor 203 responsible for running LiDAR-SLAM and data broadcasting tasks, and a wireless communication module 204 (such as a V2X communication unit) for broadcasting data externally.

[0032] Reference Figure 2 The visual impaired person barrier-free travel multi-source perception assisted decision method of the embodiment mainly includes three steps of S1, S2 and S3.

[0033] S1: Hybrid positioning and pose estimation This step is performed by the processor 104 of the intelligent wearable device end 100, and the core is to switch the positioning mode adaptively according to the scene.

[0034] a1) Indoor positioning mode: when the system determines that the user is indoors (for example, by judging through the absence of GPS signals or connecting a specified indoor Wi-Fi), a monocular vision-based SLAM algorithm is started. In the state prediction stage of the algorithm, the physical model basis is a continuous-time state equation x · =f(x,u)+w describing the dynamics of the system. x is the state vector of the system, and in the embodiment, x=[pᵀ,qᵀ,v T ,b g ᵀ,b a ᵀ]ᵀ, which contains the three-dimensional position p of the user, the attitude quaternion q, the three-dimensional velocity v, the gyroscope zero bias b g and the accelerometer zero bias b a . x ·is the first order derivative of state vector x with respect to time, i.e., the rate of change of state, which contains information of velocity, angular velocity, etc. u is the measurement input of IMU 102, u = [ω m ᵀ,a m ᵀ]ᵀ, which contains measured angular velocity ω m and acceleration a m . w is the continuous process noise, representing the uncertainty of the model.

[0035] The function f maps the current state x and input u to the rate of change of state x · , which is calculated as follows: x · = [(p · )ᵀ,(q · )ᵀ,(v · ) T ,(b g · )ᵀ,(b a · )ᵀ]ᵀ=f(x,u); where the rate of change of position p · = v; the rate of change of attitude q · = (1 / 2) × q [0,( ω m - b g ) T , where is the quaternion multiplication; the rate of change of velocity v · = R(q)(a m - b a )-G, where R(q) is the rotation matrix corresponding to the attitude quaternion q, and G is the gravity vector in the world coordinate system; and the deterministic rate of change of bias b g · and b a · are both modeled as zero, and their random walk characteristics are represented by the process noise w.

[0036] p, q, v, b g and b a are calculated from IMU data.

[0037] Since the processor 104 can only perform discrete-time calculations, the processor 104 performs numerical integration and other discretization processing on the continuous-time state equation to calculate the predicted state x k-1 at the current time from the previous state x k and the current IMU input u k .

[0038] In the observation update phase, the algorithm utilizes the images captured by the camera 101. Its mathematical model is z = h(x) + v. z is the observation vector, i.e., the pixel coordinates (u i, v i ) of the feature points being stably tracked in the image i h(x) is a non-linear re-projection function that projects a known 3D map point X i onto the image plane to obtain a theoretical pixel coordinate, given the current estimated state x (which contains the camera's position and pose). For example, h(x) = π(R(q)(X i − p)), where X k-1 is the 3D position of the map point, R(q) is the rotation matrix derived from the quaternion q, π(·) is the camera projection model, and v is the observation noise representing the uncertainty in image measurement.

[0039] By comparing the difference between the actual observed pixel coordinates z and the theoretical projected coordinates h(x) (i.e., the re-projection error), the algorithm can inversely optimize the state vector x to make it closer to the true value.

[0040] a2) Outdoor positioning mode: when the user moves outdoors, the system will smoothly switch to the outdoor positioning mode. Since both the pedestrian's motion and the UWB ranging model are nonlinear, the embodiment preferably uses an Extended Kalman Filter (EKF) algorithm to tightly couple and fuse the data from the UWB module 103 and the IMU 102. The loop process is as follows: State prediction: this step is based on the high-frequency data of the IMU 102. The processor 104 updates the posterior optimal state estimate x' k-1 at the last time by a non-linear discrete-time state transition function g to obtain the prior state estimate x' k - at the current time, i.e., x' k - = g(x' k-1 , u k ).

[0041] The state transition function g here is not produced out of thin air, but is a discretized form obtained by numerically integrating (such as Euler integration or Runge-Kutta integration) the continuous-time model (i.e., x · = f(x, u)) that describes the physical dynamics of the system. This ensures that the algorithm's prediction is consistent with the real physical process.

[0042] Measurement update: when the UWB module 103 receives ranging data z k from surrounding UWB anchor points (such as streetlights, bus stops, or UWB anchor points 202 on the target bus), the measurement update step is executed, which can be further divided into: (i) Linearized observation model: Due to the distance from the state vector x (including position) to the UWB ranging value z k The transformation is a nonlinear process (involving the calculation of Euclidean distance, including squares and square roots), and needs to be linearized. This is achieved by calculating the nonlinear observation model h to estimate x' in the current prior state. k - The Jacobian matrix at the given location yields a locally efficient linearized observation matrix H. k ,in , x=x' k - .

[0043] (ii) Calculate the Kalman gain: using the observation matrix H obtained in step (i) k And combined with the system's prediction error covariance matrix P k ⁻ With UWB measurement noise covariance matrix R k Calculate the optimal Kalman gain K. k K k Its role is to balance the reliability of the system's predictions with the reliability of UWB measurements.

[0044] (iii) Correcting the state estimate: Finally, the calculated Kalman gain K is used. k For the prior state estimate x' k - After correction, a more accurate posterior state-optimal estimate x' is obtained after UWB data correction. k Its correction formula is: x' k =x' k - +K k (z k -h(x' k - )).

[0045] Through the continuous cycle of prediction and update described above, the system can output a stable pose with centimeter-level accuracy and insensitivity to the brief loss of UWB signals.

[0046] S2: Collaborative Positioning and Precise Docking of Public Vehicles When a user approaches the target bus stop according to the navigation and is ready to board the bus, the system activates the core collaborative positioning mechanism of this invention.

[0047] S21: Receiving Broadcast Data: The vehicle-side processor 203 runs the LiDAR-SLAM algorithm in real time to generate a high-precision local environment point cloud map. To achieve high-frequency (e.g., 10Hz) broadcasting with limited V2X communication bandwidth, the vehicle-side processor 203 performs lightweight processing on the point cloud before broadcasting. For example, it performs spatial range pruning, retaining only point cloud data within a 5-meter radius centered on the bus; and performs density downsampling, reducing the point cloud density to below 500 points / frame. The processed lightweight point cloud q j The data is broadcast externally via the wireless communication module 204. The processor 104 of the smart wearable device 100 is responsible for receiving this data.

[0048] S22: Perform point cloud scanning and matching: Processor 104 will use the local point cloud p of the current frame perceived by visual SLAM to perform point cloud scanning and matching. i (For example, point clouds recovered from depth cameras or stereo cameras), compared with authoritative, high-precision point clouds received from buses. j Registration is performed. The objective function is solved using an iterative optimization algorithm (such as the Iterative Closest Point (ICP) algorithm or the Normal Distribution Transform (NDT) algorithm). The precise relative pose transformation matrix T' of the smart wearable device 100 relative to the public vehicle 200 is calculated.

[0049] S23: Solving guidance parameters: Based on the calculated precise relative pose transformation matrix T', the processor 104 can accurately solve the guidance parameter vector (e.g., a vector containing [distance d, horizontal azimuth α, vertical pitch β]) of the user relative to the bus door (the door position is known in the bus coordinate system).

[0050] S3: Spatial Audio Decision Guidance This invention uses a Head-Related Transfer Function (HRTF) model to generate three-dimensional audio signals. HRTF technology itself is existing technology. However, the inventiveness of this invention does not lie in the HRTF model itself, but in providing the model with unprecedented driving parameters derived from real-time, high-precision collaborative positioning solutions in the physical world, and applying them to solve the specific technical problem of precise docking for visually impaired individuals.

[0051] Processor 104 receives the guidance parameter vector calculated in step S23 and converts it into azimuth θ and pitch φ in the spherical coordinate system required by the HRTF model. Subsequently, the spatial audio engine calls the HRTF database to render a prompt sound source (such as a "beep" sound or the voice "door here") in real time, generating a binaural audio signal with clear spatial directionality. This signal is played to the user through audio output device 105 (such as headphones).

[0052] The effect is that users will perceive the prompt sound as originating from the actual physical location of the bus door. If the door is to the user's right front, the sound will be heard from that direction as well. Users don't need to understand abstract instructions; they can simply turn their bodies and walk towards the direction of the sound to accurately reach the door.

[0053] The present invention may also include a multimodal perception fusion and environmental modeling step, which integrates the first-view visual information from the wearable device and the external-view lidar point cloud information from the public vehicle to build a more comprehensive environmental model in a unified coordinate system for path planning and hazard warning.

[0054] In the system embodiments of the present invention, the processor 104 of the smart wearable device 100 may be configured to include: a monocular visual SLAM module, a UWB-IMU tightly coupled positioning module, a point cloud matching module, and a spatial audio engine module, which are respectively used to execute the corresponding functions in the above method.

[0055] The core advantage of this solution lies in: 1. 10cm-level precision connection: Millimeter-level docking between the car door and the glasses is achieved through tight coupling of UWB-IMU; 2. Seamless navigation across scenarios: Indoor / outdoor positioning switching time ≤ 0.5s; 3. Dual safety redundancy: false alarm rate for risk identification <0.01%.

[0056] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

[0057] Although this document uses various terms extensively, the possibility of using other terms is not excluded. These terms are used merely for the convenience of describing and explaining the essence of the invention; interpreting them as any additional limitation would contradict the spirit of the invention.

Claims

1. A method for multi-source perception assisted decision-making for barrier-free travel of visually impaired people, characterized in that, Comprising the following steps: S1: Hybrid positioning and pose estimation: by the processor deployed on the smart wearable device worn by the user, according to the scene where the user is located, adaptively execute at least one of the following positioning modes to obtain the user's pose in real time: a1) Indoor positioning mode: when the smart wearable device is in an indoor environment, start the monocular vision-based simultaneous localization and mapping algorithm, which fuses the image data and inertial measurement unit (IMU) data collected by the smart wearable device, and estimates the user's pose state vector x by repeatedly executing the following state prediction and observation update steps: State prediction: the prediction model of the algorithm is based on a continuous-time state equation x · =f(x, u) + w; where x is a state vector containing user's three-dimensional position p, attitude quaternion q and IMU bias, x · is the first-order derivative of the state vector x with respect to time, u is the IMU measurement input, and w is the continuous process noise; the processor calculates the predicted state at the current time according to the state at the previous time and the current IMU input by discretizing the continuous-time state equation; Observation update: z = h(x) + v; where z is the pixel coordinates of the image feature points, h(x) is the re-projection function that projects the map points to the image plane, and v is the observation noise; a2) Outdoor positioning mode: when the smart wearable device is in an outdoor environment, start the extended Kalman filter-based ultra-wideband (UWB) and IMU tightly coupled positioning algorithm, which fuses UWB ranging data and IMU data through the following prediction-update loop to output high-precision user pose estimation: State prediction: based on IMU data u k The described nonlinear kinematic model, the posterior optimal state estimate x k-1 Integrating, the prior state estimate x k - That is, x k - = g(x k-1 , u k ); where the state transition function g is obtained by numerically integrating a continuous-time model describing the physical dynamics of the system; Measurement update: upon receiving the UWB ranging data z k , the prior state estimate x' k is mapped to the observation space through a nonlinear observation model h - , and the measurement residual is computed; subsequently, the nonlinear observation model h k is linearized through the Jacobian matrix H k , the Kalman gain K k is computed, and the prior estimate is corrected to obtain the posterior state optimal estimate x' k , i.e., x' k = x' - + K k (z k - h(x' k - )). S2: Public vehicle cooperative positioning and precise docking: when the user approaches the target public vehicle, the following sub-steps are executed to achieve centimeter-level precise docking: S21: receiving broadcast data: the smart wearable device end receives the local environment point cloud q generated by the public carrier end through laser radar simultaneous localization and mapping (LiDAR-SLAM) j ; S22: Perform point cloud scan matching: the processor of the smart wearable device end, through the current frame local point cloud p i received from the public carrier end, performs registration, and solves the following objective function through iterative optimization to calculate the accurate relative pose transformation matrix T' of the smart wearable device end relative to the public carrier end: j received from the public carrier end, performs registration, and solves the following objective function through iterative optimization to calculate the accurate relative pose transformation matrix T' of the smart wearable device end relative to the public carrier end: ； Where argmin is an optimization operation that finds the variable that minimizes the target function, in this case T, which is the pose transformation matrix, and ° is the rigid body transformation operator; S23: Solve guidance parameters: based on the calculated accurate relative pose transformation matrix T', solve the guidance parameter vector that describes the accurate relative position and direction of the user relative to the public vehicle door; S3: Spatial audio decision guidance: the processor of the smart wearable device receives the guidance parameter vector calculated in step S23, converts the vector to azimuth angle θ and elevation angle φ in the spherical coordinate system, and generates a three-dimensional audio signal with a clear spatial direction using the head-related transfer function model, which is output to the user through the earphone. 2.The method of claim 1, wherein, In the hybrid positioning and pose estimation step, there is also an indoor and outdoor positioning mode switching sub-step: The processor monitors the GPS signal strength, UWB anchor signal strength, and the positioning confidence of the simultaneous localization and mapping algorithm; when the GPS signal is detected to be stable and enhanced to be greater than the GPS signal threshold and the positioning confidence is detected to be decreased to be lower than the confidence threshold due to insufficient environmental texture, smoothly switch from the indoor positioning mode to the outdoor positioning mode. 3.The method of claim 1, wherein, The process of the public vehicle end executing LiDAR-SLAM includes: Aligning consecutive lidar frames by scan matching algorithm to compute odometry constraints T ij ; and further globally optimize all keyframe poses X = {T1, T2,..., T n} by pose graph optimization to eliminate accumulated errors, whose objective function is: Where argmin is an optimization operation, i.e. finding the variable that makes the following objective function minimum, in this case the variable is X, i.e. the set of poses of all keyframes, X opt is the final result of this optimization process, i.e. the X that makes the objective function minimum, Σ ij is the covariance matrix of the constraints, T i represents the global pose of the i-th keyframe, T j represents the global pose of the j-th keyframe, T ij represents the relative pose transformation measurement from T i to T j .

4. The method of claim 3, wherein, Before broadcasting the local environment point cloud, the public vehicle end also performs a point cloud lightweight processing step, which includes spatial range clipping and / or density downsampling. Spatial range clipping is to retain only the point cloud data within a preset radius range centered on the public vehicle; density downsampling is to reduce the point cloud density to below a preset threshold to adapt to the bandwidth limitation of wireless broadcasting.

5. The method of claim 1, wherein, It also includes a multi-modal perception fusion and environment modeling step, which is located after step S1 and before steps S2 and S3, specifically: Fusing the first visual information of the user collected by the smart wearable device and the external visual information of the public carrier collected by the laser radar point cloud information, in a unified global coordinate system, a unified environment model containing static obstacle and dynamic obstacle information is constructed, and path planning and danger warning are carried out based on the model.

6. A multi-source perception assisted decision system for barrier-free travel of visually impaired people, running the multi-source perception assisted decision method according to claim 1, characterized in that, Comprise: The smart wearable device end comprises a camera, an inertial measurement unit, an ultra-wideband module, a processor and an audio output device, and the processor is configured to: b1) perform the hybrid positioning and pose estimation step as claimed in claim 1; b2) receive the local environment point cloud broadcast by the public carrier end, and perform the point cloud scan matching as claimed in claim 1 to calculate the accurate relative pose with the public carrier; b3) perform the spatial audio decision guidance step as claimed in claim 1 to generate a three-dimensional audio guidance signal; The public carrier end is installed on a public transport carrier and comprises a laser radar sensor, a UWB anchor point and a processor, and the processor is configured to: c1) run the LiDAR-SLAM algorithm to generate and update a high-precision local environment point cloud map in real time; c2) periodically broadcast the local environment point cloud map to the outside through the wireless communication module.

7. The multi-source perception assisted decision system for barrier-free travel of visually impaired people according to claim 6, characterized in that, The processor of the smart wearable device end is internally configured to comprise: A monocular vision SLAM module for estimating the user's pose by solving the state prediction equation x=f(x,u)+w and the observation update equation z=h(x)+v in indoor positioning mode; UWB-IMU tightly coupled positioning module for performing Kalman filter algorithm based on the prediction equation x' k - = g(x' k-1 , u k ) and the update equation x' k = x' k - + K k (z k - h(x' k - )) in outdoor positioning mode. 8.The system according to claim 6, wherein, The processor of the smart wearable device end is internally further configured to comprise: A point cloud matching module for realizing accurate docking with the public carrier by solving an optimization objective function; A spatial audio engine module for generating guidance audio based on a head-related transfer function.

Images