Intelligent obstacle avoidance navigation glasses with voice broadcast function

Abstract

The invention relates to the technical field of intelligent glasses, and particularly discloses intelligent obstacle avoidance navigation glasses capable of voice broadcasting, which comprise a multi-modal data acquisition module, a dynamic obstacle recognition and tracking module, a real-time navigation path planning module, a self-adaptive voice broadcasting module and a system master control module. The system collects environmental data through stereoscopic vision, depth perception, an IMU and a microphone array, processes the data by using a multi-target tracking algorithm based on an attention mechanism, accurately identifies and predicts a three-dimensional motion track of a dynamic obstacle, and then generates a pace-level navigation instruction according to obstacle information by using an improved dynamic window method. According to the method, through multi-data processing, the three technical problems of obstacle tracking, natural gait navigation and intelligent voice interaction in a complex dynamic environment are effectively solved, and the safety, comfort and navigation efficiency of blind people going out are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of smart glasses technology, specifically to a smart obstacle avoidance navigation glasses with voice broadcast capability. Background Technology

[0002] Visually impaired people face enormous challenges and safety risks in their daily travels. Currently, the assistive tools available on the market mainly include traditional white canes, guide dogs, and electronic assistive devices based on specific technologies. Although traditional white canes and guide dogs are widely used, they have obvious limitations: the detection range of white canes is extremely limited, usually only covering ground obstacles one step in front of the user, and cannot detect suspended obstacles (such as low branches or open windows), let alone perceive dynamically moving threats. On the other hand, the training cost of guide dogs is high, their numbers are scarce, and they cannot perform complex path planning and obstacle avoidance decisions in actual guidance.

[0003] With the development of technology, some electronic obstacle avoidance devices based on single or multiple sensors have emerged. For example, some devices use ultrasonic sensors to detect nearby obstacles and provide feedback to the user through vibration. However, such systems can usually only provide simple, non-specific obstacle presence warnings and cannot identify the specific attributes, motion state, and precise three-dimensional structure of obstacles. In complex environments, they are prone to false alarms or missed alarms due to incomplete information. Other studies have attempted to combine GPS navigation with obstacle avoidance, but GPS signals are easily lost in urban canyons, and their positioning accuracy is far from sufficient to achieve safe navigation, let alone handle dynamic obstacles that appear in real time.

[0004] In recent years, although a few studies have attempted to introduce computer vision technology into assistive devices, most solutions remain at the level of basic obstacle detection. These solutions may work in static, structured environments, but their performance drops sharply when placed in real-world scenarios such as crowded train stations or bustling market streets. They generally lack the ability to accurately track and predict the trajectories of multiple moving obstacles in the environment, and cannot effectively handle common occlusion problems and complex lighting changes, causing the system to fail at critical moments. In addition, even if obstacles can be detected, the navigation instructions are often rigid and lack fine-grained step-level guidance that integrates with the user's natural gait, making the walking process stumbling and disjointed. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] This invention provides a smart obstacle avoidance navigation glasses with voice broadcast capability, which solves the problems mentioned in the background art above.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention provides the following technical solution: a smart obstacle avoidance navigation glasses with voice broadcast capability, comprising:

[0009] A multimodal data acquisition module is used to acquire multimodal sensor data of the environment in real time. The multimodal sensor data includes stereo vision data, depth perception data, self-motion and posture data, and environmental acoustic data.

[0010] The dynamic obstacle recognition and tracking module is used to separate and track the 3D motion trajectories of multiple moving obstacles in real time based on the multimodal sensor data and a multi-object tracking deep learning model, predict their short-term future positions, and output a list of moving obstacles to be tracked.

[0011] The real-time navigation path planning module is used to calculate a real-time safe navigation path based on the moving obstacle tracking list, using an improved dynamic window method combined with a real-time risk map, and to generate a sequence of gait control instructions.

[0012] An adaptive voice broadcast generation module is used to generate adaptive voice prompts based on the step control command sequence using a rule-based natural language generation system, and to adaptively adjust the level of detail and frequency of the voice prompts according to the environmental complexity derived from the step control command sequence.

[0013] The system's main control module is used to coordinate operation, control data flow, and manage system resources, and to enable hierarchical referencing and data transfer between modules.

[0014] Furthermore, the multimodal data acquisition module specifically includes:

[0015] A pair of miniature high-definition cameras, mounted on the left and right sides and the front of the glasses respectively, acquire synchronized binocular RGB video streams at a rate of 30 frames per second as stereo vision data. This data is used to calculate disparity maps and derive 3D point clouds through a stereo matching algorithm. Next, a miniature depth sensor based on the time-of-flight principle, integrated in the center of the glasses, acquires depth images or point cloud data at the same frame rate as depth perception data. The point cloud data contains the three-dimensional coordinate information of each pixel, which is used to directly obtain the environmental distance. Then, a miniature inertial measurement unit (IMU) embedded in the temple of the glasses acquires three-axis acceleration and three-axis angular velocity at a frequency of 100Hz as its own motion and attitude data. The real-time three-axis attitude angle is calculated using a Kalman filter algorithm to compensate for the user's head movement. Finally, a microphone array consisting of four miniature microphones mounted on the left and right temples of the glasses acquires multi-channel audio streams at a sampling rate of 48kHz as environmental acoustic data, which is used for sound source localization and detection of specific sound events.

[0016] Furthermore, the dynamic obstacle recognition and tracking module specifically includes:

[0017] First, stereo vision and depth perception data from the multimodal data acquisition module are fused. A point cloud registration algorithm is used to generate an enhanced 3D environment map. Simultaneously, the user's own motion and posture data are used to eliminate scene offset caused by head movement through a motion compensation algorithm to accurately distinguish between environmental motion and the user's own motion. Then, a multi-object tracking deep learning model based on the Transformer architecture is used. This model integrates an attention mechanism to analyze the fused 3D data. Obstacle features are extracted through a convolutional neural network, and attention weights are used to focus on key obstacles, identifying multiple independent obstacles and assigning them unique IDs. At the same time, a long short-term memory network is used to predict the short-term future position of each obstacle, handling partial occlusion and illumination changes. Finally, sound source localization information from the environmental acoustic data is combined, and a beamforming algorithm is used to calculate the sound source direction to assist in verifying the visual tracking results. In particular, acoustic consistency checks are used to maintain tracking stability under occlusion conditions. The output is a moving obstacle tracking list including each obstacle ID, 3D position, velocity vector, predicted position, and confidence level.

[0018] Furthermore, the real-time navigation path planning module specifically includes:

[0019] The system receives a list of moving obstacles from the dynamic obstacle recognition and tracking module. Based on the predicted position and speed information of obstacles in the list, and combined with the user's current position and movement state, it calculates a real-time risk map using an improved dynamic window method. The risk map assesses the collision risk at each location based on a time-based collision probability algorithm and incorporates an obstacle motion uncertainty model. Then, based on the risk map and the user's current speed and direction, it optimizes path selection through a reinforcement learning algorithm and generates a series of gait control commands, including stride length, turning angle, and execution timestamp, ensuring a smooth path that conforms to the natural human walking pattern while avoiding collisions with dynamic obstacles. In addition, the module integrates the user's historical walking data and adaptively adjusts path parameters through a machine learning model to optimize walking efficiency, outputting a sequence of gait control commands.

[0020] Furthermore, the adaptive voice broadcasting generation module specifically includes:

[0021] The system receives a sequence of gait control commands from the real-time navigation path planning module. First, it maps the gait commands into natural language text, including turning commands and straight-line prompts, using a rule-based natural language generation system. Then, it optimizes the sentence structure using a context-aware algorithm. Next, it automatically calculates the environmental complexity based on the density of the gait control command sequence, where density is defined as the number of commands per unit time. Based on the density value, it adaptively adjusts the interval and simplification of the voice prompts. In high-density environments, it only broadcasts key turning commands to reduce information overload, while in low-density environments, it provides more detailed navigation information. Finally, it converts the text into speech output using a speech synthesis engine, adjusting the speech rate and volume parameters to ensure the timeliness and clarity of the prompts. Simultaneously, it uses an emotion computing algorithm to assess the user's potential information anxiety state and further optimize the voice broadcast strategy.

[0022] Furthermore, the system main control module specifically includes:

[0023] The central processing unit is used to perform scheduling and data exchange, and ensures real-time performance through multi-threaded management.

[0024] The data bus is used to transmit multimodal sensor data, a list of moving obstacles, a sequence of gait control commands, and voice broadcast scripts, and to enable high-speed communication between modules.

[0025] The power management unit is used to optimize system power consumption to extend battery life by dynamically adjusting voltage and frequency according to module load.

[0026] The real-time operating system ensures the real-time processing and priority control of each module. The main control module also monitors the operating status of each module and performs error handling or system reset when a fault occurs through heartbeat detection and anomaly handling mechanism to ensure stable system operation.

[0027] Furthermore, the stereo vision data and processing in the multimodal data acquisition module also includes:

[0028] A stereo matching algorithm is used to calculate a disparity map, thereby generating 3D point cloud data, which is then fused with point cloud data output from a depth sensor. The accuracy and robustness of depth information are improved by iterative nearest-point algorithm, especially in environments with weak texture or changing lighting. At the same time, inertial sensor data is used to correct motion distortion of the point cloud to enhance data quality.

[0029] Furthermore, the multi-object tracking deep learning model in the dynamic obstacle recognition and tracking module also includes an online learning mechanism that can adjust model parameters according to real-time environmental changes, adapt to new obstacle types or movement patterns through incremental learning algorithms, improve tracking accuracy, and enhance the model's robustness in extreme scenarios through adversarial training.

[0030] Furthermore, the improved dynamic window method in the real-time navigation path planning module also considers the user's personal walking preferences, learns user habits through historical path data, analyzes commonly used paths using clustering algorithms, thereby generating more personalized navigation paths, and incorporates a social force model to simulate crowd interaction in order to optimize obstacle avoidance behavior in crowded environments.

[0031] A smart obstacle avoidance navigation glasses includes a frame and temples, wherein all modules are integrated within the frame, the sensors of the multimodal data acquisition module are mounted on the front and sides of the glasses, the hardware of the system main control module and the voice broadcast generation module is embedded in the temples, and the glasses also include a bone conduction speaker for voice broadcast.

[0032] (III) Beneficial Effects

[0033] This invention provides a smart obstacle avoidance navigation glasses with voice broadcast capability. It has the following beneficial effects:

[0034] (i) The intelligent obstacle avoidance navigation glasses with voice broadcast function deeply integrate conventionally collected stereo vision, depth point cloud and IMU data. It uses the attention mechanism of the Transformer model to accurately focus on key moving targets. Even if the target is partially obscured or in poor lighting conditions such as shade or strong light, it can continuously predict its movement trajectory through the long short-term memory network. This creates a stable and forward-looking dynamic environmental model for blind users that far exceeds the perception range of guide canes or traditional obstacle avoidance systems. It significantly reduces the risk of collision caused by missed or false detection of dynamic obstacles and greatly improves the user's sense of security when traveling in the most challenging public scenarios.

[0035] (ii) The intelligent obstacle avoidance navigation glasses with voice broadcast function construct a risk field by calculating the collision time of each obstacle and sampling trajectories that conform to human kinematics in the velocity space. Finally, they output a smooth sequence of gait control commands. This means that blind users no longer receive general instructions, but rather subtle gait guidance that matches their own gait rhythm, as if they had an experienced guide. This allows them to walk calmly through crowds without frequent sudden stops or sharp turns, thus greatly improving the naturalness, comfort, and overall navigation efficiency of walking while ensuring safe obstacle avoidance.

[0036] (III) The intelligent obstacle avoidance navigation glasses with voice broadcast automatically judge the complexity of the environment based on the density of the instruction sequence and dynamically adjust the broadcast strategy. When the density is high, only key turning instructions are broadcast and the speech speed is faster. When the density is low, more detailed navigation information is provided. This intelligent voice interaction ensures that users can receive clear and concise emergency obstacle avoidance instructions in critical moments, and can obtain relaxed and natural situational perception in a relaxed environment. It effectively avoids anxiety and distraction caused by too many prompts or navigation blind spots caused by insufficient prompts. Attached Figure Description

[0037] Figure 1 This is the overall system flowchart of the present invention;

[0038] Figure 2 This is a schematic diagram of the structure of the glasses of the present invention.

[0039] In the image: 1. Frame; 2. Temple; 3. Bone conduction speaker; 4. Sensor. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] First embodiment: as follows Figure 1 As shown, the present invention provides a technical solution: a smart obstacle avoidance navigation glasses with voice broadcast capability, comprising:

[0042] The multimodal data acquisition module is used to acquire multimodal sensor data of the environment in real time. The multimodal sensor data includes stereo vision data, depth perception data, self-motion and posture data, and environmental acoustic data.

[0043] The dynamic obstacle recognition and tracking module is used to separate and track the 3D motion trajectories of multiple moving obstacles in real time based on multimodal sensor data and a multi-object tracking deep learning model, predict their short-term future positions, and output a list of moving obstacles to be tracked.

[0044] The real-time navigation path planning module is used to calculate a real-time safe navigation path based on a moving obstacle tracking list, using an improved dynamic window method combined with a real-time risk map, and to generate a sequence of gait control instructions.

[0045] An adaptive voice prompt generation module is used to generate adaptive voice prompts based on a rule-based natural language generation system using a sequence of gait control commands. The module also adaptively adjusts the level of detail and frequency of the voice prompts according to the environmental complexity derived from the sequence of gait control commands.

[0046] The system's main control module is used to coordinate operation, control data flow, and manage system resources, and to enable hierarchical referencing and data transfer between modules.

[0047] Second embodiment: as follows Figure 1 As shown, the multimodal data acquisition module is implemented through the following steps:

[0048] First, a pair of miniature high-definition cameras, mounted on the left and right sides of the eyeglasses frame respectively, capture timestamp-synchronized binocular RGB video streams at a rate of 30 frames per second, with a resolution of 1280×720 pixels, as the data source for stereo vision. Next, distortion correction and epipolar alignment preprocessing are performed on the left and right images. Then, a miniature depth sensor based on the time-of-flight principle, integrated in the center of the glasses, captures depth images at the same frame rate and converts them into environmental point cloud data containing the three-dimensional coordinates of each point using a built-in processor. The coordinate transformation formula for each point in the point cloud is:

[0049]

[0050] in( , ) represents pixel coordinates, ( , ) are the coordinates of the main point, ( , Let Z be the focal length and Z be the distance value directly measured by the depth sensor. Then, a miniature inertial measurement unit embedded in the temple of the glasses is used to acquire triaxial accelerometer data at a frequency of 100Hz. and three-axis gyroscope data The body pose change is calculated in real time using a Kalman filter data fusion algorithm, and its quaternion differential equation is expressed as:

[0051]

[0052] in The current attitude quaternion, This is a quaternion multiplication operator, and it performs zero-bias correction using accelerometer data in a stationary state, ultimately outputting the compensated roll angle. Pitch angle and yaw angle Finally, a microphone array consisting of four miniature microphones mounted on the left and right temples of the glasses was used to acquire multi-channel audio streams at a sampling rate of 48kHz, and the time difference of sound source arrival was calculated using a generalized cross-correlation function.

[0053]

[0054] in and The system uses signals received from different microphones and a hardware synchronization circuit to add a global timestamp to all sensor data, achieving microsecond-level time alignment. This provides a precise and synchronized multimodal data foundation for subsequent modules.

[0055] Third embodiment: as follows Figure 1 As shown, the multimodal data acquisition module is implemented through the following steps:

[0056] First, synchronous data is received from the multimodal data acquisition module. Then, using the stereo vision data, a disparity map is calculated through a semi-global stereo matching algorithm, and its pixel count... The matching cost is calculated jointly by the Census transform and the Hamming distance, as shown in the formula:

[0057]

[0058] in Candidate disparities are identified and aggregated using the cost function:

[0059]

[0060] Optimization was performed, and the optimal parallax was ultimately determined using a winner-takes-all strategy:

[0061]

[0062] Next, according to the perspective projection formula:

[0063]

[0064] Convert parallax to depth Generate dense 3D point clouds:

[0065]

[0066] Then, this point cloud is compared with the point cloud directly acquired by the depth sensor. Registration is performed using an iterative nearest-point algorithm by minimizing the objective function:

[0067]

[0068] Solving for the optimal rotation matrix Translation vector To achieve point cloud fusion and generate enhanced 3D environment maps:

[0069]

[0070] Then, using the body pose data calculated by the IMU, through the rigid body transformation matrix:

[0071]

[0072] Motion compensation is performed on the point cloud across consecutive frames, transforming the point cloud of the current frame to the world coordinate system:

[0073]

[0074] To eliminate scene drift caused by user head movements and accurately separate real moving objects in the environment, the fused 3D point cloud and RGB image are then input into a Transformer-based multi-object tracking deep learning model. This model first extracts point cloud features through a 3D sparse convolutional network, whose convolution operation is defined as:

[0075]

[0076] in As a feature mask, a visual Transformer is used to encode the features of the RGB image, and a multi-head self-attention mechanism is used to calculate the correlation between features:

[0077]

[0078] And the two features are fused using a cross-attention mechanism:

[0079]

[0080] This allows the model to focus on key obstacle areas and maintain target identification (ID) even under occlusion. Then, a long short-term memory network is used based on the object's historical trajectory sequence.

[0081]

[0082] Predict its future Location within a given time period:

[0083]

[0084] in For network weights, finally, the microphone array is combined using the generalized cross-correlation function:

[0085]

[0086] Calculate the direction of arrival of the sound source and estimate the azimuth angle of the sound source:

[0087]

[0088] Then, Kalman filtering is performed to fuse the results with the visual tracking results.

[0089]

[0090] in The Kalman gain is used to output a list of moving obstacles that includes obstacle IDs, 3D bounding boxes, instantaneous velocity vectors, predicted position sequences, and overall confidence scores. This list serves as input for the subsequent path planning module.

[0091] Fourth embodiment: as Figure 1 As shown, the real-time navigation path planning module is implemented through the following steps:

[0092] First, a list of moving obstacles is received from the dynamic obstacle recognition and tracking module. This list contains each obstacle. Predicted location and velocity vector Next, the user's current location is calculated from IMU data obtained from the system's main control module. and speed Calculate the relative velocity of each obstacle k with respect to the user. And then calculate the precise collision time. This calculation is performed only when the obstacle is within the potential conflict cone of the user's path. Then, based on the TTC values ​​of all potential threatening obstacles, a user-centric two-dimensional polar coordinate grid risk map is constructed, which discretizes the space in front of the user into radial distances. and angle The grid cells on the surface, each grid cell risk value The TTC of all obstacles covering the unit is determined by the following formula: ,in The system's preset time constant is used to adjust risk sensitivity. The weighting factors are based on obstacle type and confidence level. Then, an improved dynamic window method is used in the velocity space. In performing local path planning, the dynamic window is first determined based on the robot's kinematic model and physical constraints. Then, a large number of velocity pairs are uniformly sampled within the dynamic window. And for each set of velocity pairs, simulate the generation of future short-term times. Predicted trajectory within Subsequently, for each simulated trajectory Calculate a comprehensive cost function The function is defined as follows: The first item is the trajectory. Risk Map The path integral represents the collision risk, and the second term is the trajectory endpoint. With global sub-target points The distance, the third term is the trajectory smoothness penalty (sum of squared angular velocities), and the fourth term is the deviation of the trajectory endpoint's orientation from the target direction. These are weighting coefficients, which are then selected through an optimization process to ensure... Minimum optimal speed pair Finally, the optimal linear velocity is... and angular velocity This is converted into specific, user-executable step control instructions, and the conversion formula is step size. and steering angle ,in It is a preset step cycle, which outputs a sequence of step control instructions with precise execution timestamps to the adaptive voice broadcast generation module.

[0093] Fifth embodiment: as follows Figure 1 As shown, the adaptive voice broadcast generation module is implemented through the following steps:

[0094] First, receive a sequence of pace control commands from the real-time navigation path planning module. ,in Indicates the step size. Indicates the steering angle. Representing the timestamp, the abstract instruction parameters are then mapped to natural language text through a rule-based natural language generation system based on a context-free grammar. The core mapping rule is defined as: when... Generate steering instructions in time ,in The steering angle threshold. To determine the minimum effective step size, the current environmental complexity index is then calculated. For sliding time windows, The Dirac function is used, which is essentially the number of instructions per unit time. Then, based on the complexity index... Adaptive adjustment of voice broadcasting strategy: when When entering simplified mode, the voice broadcast interval is... Simplified broadcast content At the same time, the speaking speed is increased to a normal speaking speed. Enter detailed mode at this time, and the broadcast interval is set accordingly. The broadcast content should be kept in its complete form. The speaking speed returned to normal.

[0095] when In standard mode, all parameters are linearly interpolated. , Finally, a WaveNet-based deep neural network speech synthesizer is used to convert the text sequence into speech waveforms. Before output, the real-time incoming instructions are processed by a priority-based instruction management module, whose priority function is defined as follows: To ensure that high-priority commands can interrupt the current broadcast in a timely manner, among which... These are weighting coefficients, which respectively control the degree of influence of timeliness, turning importance, and distance importance.

[0096] Sixth embodiment: as follows Figure 1 As shown, the system's main control module achieves system coordination and control through the following complete technical solution:

[0097] First, multi-task scheduling is implemented through a hard real-time operating system kernel. A rate-monotonic scheduling algorithm is used to assign fixed priorities to each module. The feasibility condition for this scheduling is given by the formula:

[0098]

[0099] in Indicates task Worst execution time Indicates the task cycle. The total number of tasks, with dynamic obstacle recognition and tracking tasks assigned the highest priority. To ensure completion before the deadline, data acquisition tasks are prioritized second, followed by path planning and speech generation tasks, with the latter having decreasing priority. Then, high-speed data transfer between modules is achieved via a parallel data bus based on a hardware DMA controller, with a bus bandwidth of [missing information]. The following inequalities must be satisfied:

[0100]

[0101] in and These represent the sampling frequency and data packet size for each sensor, respectively. and To track list data size and update frequency, and The size and generation cycle of path instruction data are set to ensure unblocked transmission in the data pipeline. Then, the system main control module uses a cycle of... The heartbeat detection mechanism monitors the health status of each submodule, and when continuous... When a heartbeat signal from a module is received at the end of a cycle, the module is determined to be faulty and a graded degradation strategy is initiated. The strategy function is defined as follows:

[0102]

[0103] Finally, the integrated power management unit optimizes power consumption in real time through dynamic voltage and frequency adjustment technology. Its core control algorithm is based on the real-time calculated load of each module. Dynamically adjust the power supply voltage and clock frequency Following the power consumption model:

[0104]

[0105] While ensuring functional integrity, the system automatically adjusts its parameters proportionally when it detects a decrease in environmental complexity. This achieves an optimal balance between power consumption and performance, thereby ensuring that the system can work continuously for no less than 8 hours after a single charge.

[0106] Seventh embodiment: as follows Figure 1 As shown, the fusion process of stereo vision data and depth perception data in the multimodal data acquisition module is specifically achieved through the following steps:

[0107] First, the left and right images captured by the stereo camera are compared. and Distortion and epipolar corrections are performed to ensure that matching points lie on the same horizontal scan line. Next, a Census transform is used to calculate the disparity map. This transform first defines a local neighborhood window for each pixel, comparing the brightness value of the center pixel within the window with the brightness values ​​of other pixels in the neighborhood to generate a bit string describing the difference. Its mathematical expression is Indicates cascading. For the neighborhood, the function The result is 1 if the condition is met, and 0 otherwise. Then, the result is calculated using Hamming distance. As a similarity metric, the right image finds the optimal disparity for each pixel in the left image. Thus, the initial disparity map is formed. Then, based on the principle of triangulation, the disparity map is converted into a 3D point cloud from stereo vision. Its spatial coordinates are calculated using the following formula:

[0108]

[0109] in The binocular baseline distance. For camera focal length, The primary point coordinates are used, and simultaneously, a point cloud with calibrated intrinsic parameters is acquired from the depth sensor. The coordinates of its point are given by the following formula:

[0110]

[0111] in These are the pixel coordinates of the depth map. For the depth sensor intrinsic parameters, then, the iterative nearest-point algorithm is used to... and For fine registration, the algorithm iteratively performs the following two steps until convergence: First, for... Each point in Find the point with the closest Euclidean distance. The first step is to establish the correspondence. The second step is to solve for the optimal rigid body transformation through singular value decomposition. To minimize the mean square error between all corresponding point pairs:

[0112]

[0113] The solution process involves calculating the centroids of the two point clouds. Calculate the centroid-free coordinates and construct the covariance matrix. ,right Perform SVD decomposition Then the optimal rotation Optimal translation Finally, the obtained transformation Applied to This enables precise alignment and data fusion of two point clouds in a unified coordinate system, generating an enhanced 3D environment map that outperforms a single data source in terms of accuracy, completeness, and robustness.

[0114] Eighth embodiment: as follows Figure 1 As shown, the online learning mechanism in the dynamic obstacle recognition and tracking module is implemented through the following steps:

[0115] First, during the inference process, the model calculates a real-time confidence score for each tracked target by comprehensively evaluating the bounding box stability, feature matching consistency, and motion smoothness of the tracking trajectory. Its calculation formula is: The weighting coefficients are used when the confidence score for detecting a target satisfies the following conditions within N consecutive frames. When the online learning process is triggered, the system automatically extracts a multimodal data segment of M frames before and after the moment when the confidence level of the target begins to decay. This segment includes RGB image patches, point cloud fragments, and corresponding tracking trajectory data, forming a sample set to be augmented. Then, for A rigorous data augmentation process was implemented, starting with geometric transformation enhancement, applying random rotations to both image patches and point clouds. Peaceful relocation Next, photometric enhancement is performed by applying random brightness, contrast, and color jitter to the RGB image. The parameters are sampled from a uniform distribution. Finally, more diverse synthetic samples are generated using elastic deformation techniques, forming the enhanced training sample set. Then, an incremental learning algorithm based on elastic weight consolidation is used to update the model parameters. First, the current parameters are calculated. Importance matrix of historical tasks Its elements By calculating the loss function For parameters and The second derivative is obtained by diagonal approximation, i.e. Next, a regularized loss function that incorporates both new and old knowledge protection is constructed. ,in For the new sample set Standard tracking loss on To balance the hyperparameters, the gradient descent algorithm is used to minimize them. Obtain updated model parameters Finally, after ensuring that the accuracy of the model validation set is not lower than a preset threshold, the system seamlessly and hot-swaps the runtime tracking model parameters to... This enables the model to dynamically adapt to and continuously optimize novel obstacle types or motion patterns without disrupting existing knowledge.

[0116] Ninth embodiment: as follows Figure 1 As shown, personalized route optimization in the real-time navigation route planning module is implemented through the following steps:

[0117] First, the system establishes and updates a historical path database based on the gait control command sequence generated by the real-time navigation path planning module and the actual user trajectory after its implementation. Each trajectory data in this database originates from the path planning results calculated from the moving obstacle tracking list in previous cycles. Next, the density-based DBSCAN clustering algorithm is used to mine and analyze these historical trajectory data, with its neighborhood distance function defined as:

[0118]

[0119] Density accessibility analysis was used to identify user preference path pattern clusters in different environmental scenarios. Then, during the execution of the dynamic window method in the real-time navigation path planning module, the identified user preference path patterns are incorporated into the cost function calculation, and the complete optimization objective function is constructed as follows:

[0120]

[0121] Among them, risk score item The risk map, generated directly from the moving obstacle tracking list from the dynamic obstacle recognition and tracking module, has a curvature term. This ensures path smoothness, while personalized preferences... The current simulated trajectory is measured using Hausdorff distance. With user preference path cluster center The degree of agreement is calculated using the following formula:

[0122]

[0123] Finally, the system dynamically adjusts the personalized weight coefficients in the formula through an online learning mechanism. This adjustment is based on a real-time assessment of user compliance: when the deviation rate between the user's actual walking path and the recommended path is detected... At that time, according to Reduce the weight of personalization, otherwise... Enhance personalization to form a closed-loop optimization from historical path data collection to personalized path generation, ensuring that the final output of the gait control command sequence is both safe and efficient and conforms to the user's individual walking habits.

[0124] Tenth embodiment: as Figure 2 As shown, a smart obstacle avoidance navigation glasses includes a frame 1 and temples 2, wherein all modules are integrated in the frame 1, the sensor 4 of the multimodal data acquisition module is installed on the front and sides of the frame 1, the hardware of the system main control module and the voice broadcast generation module is embedded in the temples 2, and the glasses also include a bone conduction speaker 3 for voice broadcast.

[0125] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0126] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A smart obstacle avoidance navigation glasses with voice broadcast capability, characterized in that: include: A multimodal data acquisition module is used to acquire multimodal sensor data of the environment in real time. The multimodal sensor data includes stereo vision data, depth perception data, self-motion and posture data, and environmental acoustic data. The dynamic obstacle recognition and tracking module is used to separate and track the 3D motion trajectories of multiple moving obstacles in real time based on the multimodal sensor data and a multi-object tracking deep learning model, predict their short-term future positions, and output a list of moving obstacles to be tracked. The real-time navigation path planning module is used to calculate a real-time safe navigation path based on the moving obstacle tracking list, using an improved dynamic window method combined with a real-time risk map, and to generate a sequence of gait control instructions. An adaptive voice broadcast generation module is used to generate adaptive voice prompts based on the step control command sequence using a rule-based natural language generation system, and to adaptively adjust the level of detail and frequency of the voice prompts according to the environmental complexity derived from the step control command sequence. The system main control module is used to coordinate operation, control data flow and manage system resources, and realize hierarchical referencing and data transfer between modules.

2. The intelligent obstacle avoidance navigation glasses with voice broadcast capability according to claim 1, characterized in that: The multimodal data acquisition module specifically includes: A pair of miniature high-definition cameras, mounted on the left and right sides and the front of the glasses respectively, acquire synchronized binocular RGB video streams at a rate of 30 frames per second as stereo vision data. This data is used to calculate disparity maps and derive 3D point clouds through a stereo matching algorithm. Next, a miniature depth sensor based on the time-of-flight principle, integrated in the center of the glasses, acquires depth images or point cloud data at the same frame rate as depth perception data. The point cloud data contains the three-dimensional coordinate information of each pixel, which is used to directly obtain the environmental distance. Then, a miniature inertial measurement unit (IMU) embedded in the temple of the glasses acquires three-axis acceleration and three-axis angular velocity at a frequency of 100Hz as its own motion and attitude data. The real-time three-axis attitude angle is calculated using a Kalman filter algorithm to compensate for the user's head movement. Finally, a microphone array consisting of four miniature microphones mounted on the left and right temples of the glasses acquires multi-channel audio streams at a sampling rate of 48kHz as environmental acoustic data, which is used for sound source localization and detection of specific sound events.

3. The intelligent obstacle avoidance navigation glasses with voice broadcast capability according to claim 2, characterized in that: The dynamic obstacle recognition and tracking module specifically includes: First, stereo vision and depth perception data from the multimodal data acquisition module are fused. A point cloud registration algorithm is used to generate an enhanced 3D environment map. Simultaneously, the user's own motion and posture data are used to eliminate scene offset caused by head movement through a motion compensation algorithm to accurately distinguish between environmental motion and the user's own motion. Then, a multi-object tracking deep learning model based on the Transformer architecture is used. This model integrates an attention mechanism to analyze the fused 3D data. Obstacle features are extracted through a convolutional neural network, and attention weights are used to focus on key obstacles, identifying multiple independent obstacles and assigning them unique IDs. At the same time, a long short-term memory network is used to predict the short-term future position of each obstacle, handling partial occlusion and illumination changes. Finally, sound source localization information from the environmental acoustic data is combined, and a beamforming algorithm is used to calculate the sound source direction to assist in verifying the visual tracking results. In particular, acoustic consistency checks are used to maintain tracking stability under occlusion conditions. The output is a moving obstacle tracking list including each obstacle ID, 3D position, velocity vector, predicted position, and confidence level.

4. The intelligent obstacle avoidance navigation glasses with voice broadcast capability according to claim 3, characterized in that: The real-time navigation path planning module specifically includes: The system receives a list of moving obstacles from the dynamic obstacle recognition and tracking module. Based on the predicted position and speed information of obstacles in the list, and combined with the user's current position and movement state, it calculates a real-time risk map using an improved dynamic window method. The risk map assesses the collision risk at each location based on a time-based collision probability algorithm and incorporates an obstacle motion uncertainty model. Then, based on the risk map and the user's current speed and direction, it optimizes path selection through a reinforcement learning algorithm and generates a series of gait control commands, including stride length, turning angle, and execution timestamp, ensuring a smooth path that conforms to the natural human walking pattern while avoiding collisions with dynamic obstacles. In addition, the module integrates the user's historical walking data and adaptively adjusts path parameters through a machine learning model to optimize walking efficiency, outputting a sequence of gait control commands.

5. The intelligent obstacle avoidance navigation glasses with voice broadcast capability according to claim 4, characterized in that: The adaptive voice broadcast generation module specifically includes: The system receives a sequence of gait control commands from the real-time navigation path planning module. First, it maps the gait commands into natural language text, including turning commands and straight-line prompts, using a rule-based natural language generation system. Then, it optimizes the sentence structure using a context-aware algorithm. Next, it automatically calculates the environmental complexity based on the density of the gait control command sequence, where density is defined as the number of commands per unit time. Based on the density value, it adaptively adjusts the interval and simplification of the voice prompts. In high-density environments, it only broadcasts key turning commands to reduce information overload, while in low-density environments, it provides more detailed navigation information. Finally, it converts the text into speech output using a speech synthesis engine, adjusting the speech rate and volume parameters to ensure the timeliness and clarity of the prompts. Simultaneously, it uses an emotion computing algorithm to assess the user's potential information anxiety state and further optimize the voice broadcast strategy.

6. The intelligent obstacle avoidance navigation glasses with voice broadcast capability according to claim 5, characterized in that: The system main control module specifically includes: The central processing unit is used to perform scheduling and data exchange, and ensures real-time performance through multi-threaded management. The data bus is used to transmit multimodal sensor data, a list of moving obstacles, a sequence of gait control commands, and voice broadcast scripts, and to enable high-speed communication between modules. The power management unit is used to optimize system power consumption to extend battery life by dynamically adjusting voltage and frequency according to module load. The real-time operating system ensures the real-time processing and priority control of each module. The main control module also monitors the operating status of each module and performs error handling or system reset when a fault occurs through heartbeat detection and anomaly handling mechanism to ensure stable system operation.

7. The intelligent obstacle avoidance navigation glasses with voice broadcast capability according to claim 6, characterized in that: The stereo vision data and processing in the multimodal data acquisition module also includes: A stereo matching algorithm is used to calculate a disparity map, thereby generating 3D point cloud data, which is then fused with point cloud data output from a depth sensor. The accuracy and robustness of depth information are improved by iterative nearest-point algorithm, especially in environments with weak texture or changing lighting. At the same time, inertial sensor data is used to correct motion distortion of the point cloud to enhance data quality.

8. The intelligent obstacle avoidance navigation glasses with voice broadcast capability according to claim 7, characterized in that: The multi-object tracking deep learning model in the dynamic obstacle recognition and tracking module also includes an online learning mechanism that can adjust model parameters according to real-time environmental changes, adapt to new obstacle types or movement patterns through incremental learning algorithms, improve tracking accuracy, and enhance the model's robustness in extreme scenarios through adversarial training.

9. The intelligent obstacle avoidance navigation glasses with voice broadcast capability according to claim 8, characterized in that: The improved dynamic window method in the real-time navigation path planning module also considers the user's personal walking preferences, learns user habits through historical path data, analyzes commonly used paths using clustering algorithms, thereby generating more personalized navigation paths, and incorporates a social force model to simulate crowd interaction in order to optimize obstacle avoidance behavior in crowded environments.

10. A smart obstacle avoidance navigation glasses system comprising a voice-enabled smart obstacle avoidance navigation glasses system as described in any one of claims 1 to 9, characterized in that, The glasses include a frame (1) and temples (2), wherein all modules are integrated within the frame (1), the sensors (4) of the multimodal data acquisition module are mounted on the front and sides of the frame (1), the hardware of the system main control module and the voice broadcast generation module is embedded in the temples (2), and the glasses also include a bone conduction speaker (3) for voice broadcast.

Images