Wearable intelligent navigation equipment and system for blind people going out

Abstract

The invention discloses wearable intelligent navigation equipment and system for blind people going out, and belongs to the technical field of auxiliary navigation and positioning. According to the method, an auditory search behavior discrimination model is established by analyzing head angular velocity features and visual optical flow features, non-advancing directional head rotation of the blind caused by side ear listening is recognized and eliminated, and head-body course decoupling is achieved; the method comprises the following steps: capturing an environment image by using a camera, extracting visual semantic features of blind sidewalk navigation, constructing a lateral position strong constraint mechanism based on environmental semantics, correcting multipath deviation of satellite positioning, and inhibiting accumulative errors of inertial calculation; and designing a full-scene adaptive filter, evaluating the confidence coefficient according to the noise characteristics of the camera, the inertial measurement unit and the satellite observation data, and outputting an optimal position and attitude estimation result. The portable advantage of the glasses form is utilized, the problem of course angle calculation errors caused by frequent auditory path-finding actions of the head when the blind walk is solved, and high-precision and robust full-scene navigation is realized.

Description

Technical Field

[0001] This invention belongs to the field of assisted navigation and positioning technology, and relates to a method and device for adaptive positioning of blind people in all scenarios based on multi-sensor fusion of smart glasses, designed for the walking characteristics of visually impaired people. Background Technology

[0002] With the development of wearable technology, personal positioning using inertial measurement units (IMUs) and global navigation satellite systems (GNSS) has become the mainstream solution for assisting blind people in their travel. Traditional assistive devices are mostly handheld or head-mounted (such as helmets), but they still face many technical bottlenecks in practical applications.

[0003] Blind people frequently sway their heads left and right or tilt their ears to listen while walking, especially when wearing lightweight smart glasses, which allows for more flexible head movements. Existing navigation algorithms typically assume the head's orientation is the direction of movement, leading to directional errors in the positioning trajectory and failing to accurately reflect the blind person's intention to walk in a straight line. Secondly, GNSS positioning is prone to multipath effects in urban areas with tall buildings, causing random drifts of several meters. While IMUs offer high short-term accuracy, their positional errors accumulate rapidly over time. Blind people have extremely high requirements for positional accuracy; if GNSS drift causes the positioning point to deviate from the roadway, it can lead to serious safety accidents. Furthermore, most existing pedestrian dead reckoning algorithms are "blind" and lack the ability to perceive the surrounding environment, failing to utilize environmental features to calibrate sensor errors. Therefore, there is an urgent need for a multi-sensor fusion method that can adapt to the wearing form of smart glasses, effectively decouple head-body movements, and utilize environmental visual semantics for accurate positioning. Summary of the Invention

[0004] To address the issues of heading angle calculation errors caused by frequent head probing movements of blind individuals in existing technologies, and the insufficient positioning accuracy of single sensors in complex scenarios, the present invention aims to provide a wearable intelligent navigation device and system for blind individuals. This device and system can decouple head-body movements and utilize environmental visual semantics for precise positioning, ensuring continuous accuracy of navigation heading in complex sound field environments.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] This invention discloses a wearable smart navigation system for blind people's travel, comprising the following steps:

[0007] Step S1, Data Acquisition and Preprocessing: Angular velocity is acquired using smart glasses. acceleration Environmental images and GNSS positioning data Using the intrinsic parameter matrix Will Feature points extracted Back projection onto the camera coordinate system yields Then use the extrinsic parameter matrix to Transform to IMU coordinate system; simultaneously utilize lever arm vectors satellite positioning data Transferred to the IMU center to ensure and Consistency in physical space.

[0008] Step S2, Decoupling of Auditory Search Behavior and Heading: Analyze the variance of head yaw angular velocity and the average modulus of visual optical flow to identify whether the user is currently in an auditory search state. If not, return to perform search behavior determination; if so, decouple the locking relationship between head orientation and body direction of travel to obtain the body's true heading.

[0009] Step S3, Visual Semantic Feature Extraction and Constraint Construction: Perform semantic segmentation on the environmental image, identify tactile paving features, and construct a strong lateral position constraint equation based on the geometric relationship between the user and the features;

[0010] Step S4, Full-Scene Adaptive Fusion Localization: Establish an extended Kalman filter model, using the real travel heading, strong lateral position constraints, and feature points and localization data from step S1 as observation vectors. Evaluate the confidence level of each observation based on its noise characteristics, and output the optimal position and attitude estimation results.

[0011] Furthermore, the implementation method of step S1 is as follows:

[0012] For feature points extracted from the environmental image, let their pixel coordinates be... Using the intrinsic parameter matrix By back-projecting it onto the camera coordinate system, we can obtain the three-dimensional coordinate points in the camera coordinate system. :

[0013]

[0014] In the formula, It is the depth information of the feature points;

[0015] Subsequently, using the extrinsic parameter matrix Will Transform to IMU coordinate system:

[0016]

[0017] In the formula, The rotation matrix from the camera to the IMU. It is a translation vector. These are the three-dimensional coordinates of the feature point in the IMU coordinate system.

[0018] Because there is a physical deviation between the satellite antenna mounting frame and the center of the IMU, the lever arm vector is used. Transmit satellite positioning data to the IMU center:

[0019]

[0020] In the formula, It is the rotation matrix calculated from the IMU attitude. It is the lever arm vector. This refers to the satellite positioning coordinates of the satellite positioning data in the IMU coordinate system.

[0021] Furthermore, the variance of the yaw angular velocity mentioned in step S2 for:

[0022]

[0023] In the formula, The instantaneous yaw rate within the window. is the mean; N is the number of IMU data points within the sliding window.

[0024] Furthermore, the average modulus of the visual optical flow described in step S2 for:

[0025]

[0026] In the formula, This represents the horizontal optical flow component of the j-th feature point in the image coordinate system. Let K represent the optical flow component in the vertical direction of the j-th feature point in the image coordinate system, and K represent the total number of feature points extracted from the image.

[0027] Furthermore, the method for identifying whether the user is currently in an auditory search state in step S2: when the condition is met... At that time, it is determined that the user is in an auditory search state.

[0028] Furthermore, the method for obtaining the body's true heading in step S2 is as follows: when not in an auditory search state, at any given time... The attitude quaternion predicted by the extended Kalman filter of -1 is In normal conditions, attitude updates rely on IMU integration:

[0029]

[0030] In the formula, Sampling interval time The equivalent rotation vector within, , Given the quaternion of the posture at time k, we can ultimately obtain the body's actual heading.

[0031] ;

[0032] When in auditory search mode, the head moves significantly, but the environmental background does not shift significantly, indicating that the user is stationary or exploring at low speed. In this case, the yaw angle observation update should be forcibly isolated to reflect the actual body's heading. Locked to the estimated value from the previous time step, i.e.:

[0033] ;

[0034] Furthermore, step S3 is implemented as follows:

[0035] The system utilizes lightweight convolutional neural networks to process environmental images. Perform pixel-level semantic segmentation to identify the central skeleton line of the tactile paving area; utilize the camera intrinsic parameter matrix Given the current pitch angle, the center skeleton line on the image plane is projected onto the local navigation coordinate system, and the equation of the straight line is obtained by fitting the line. Based on this, the lateral position observation equation is constructed:

[0036]

[0037] in, The system predicts the current position coordinates. The visual observation noise follows a Gaussian distribution. ABC are the coefficients of the fitted straight line equation;

[0038] When tactile paving features are detected in multiple consecutive frames, this lateral position strong constraint This will be used as a strong observation information input filter, forcing the estimated location to converge toward the center of the tactile paving.

[0039] Furthermore, step S4 is implemented as follows:

[0040] Define the state vector of the system :

[0041]

[0042] in, , , These represent the three-dimensional position, velocity, and attitude quaternions, respectively. , θ represents the zero bias of the accelerometer and gyroscope, respectively, and k is time.

[0043] The state prediction phase of the extended Kalman filter utilizes the IMU kinematic equations for time updates:

[0044]

[0045]

[0046]

[0047] In the formula, This is the rotation matrix corresponding to the quaternion. It is the gravity vector. The equivalent rotation vector within the sampling interval. This is the sampling interval time;

[0048] In the measurement update phase, the extended Kalman filter defines the combined observation vector of the system at time k. for:

[0049]

[0050] In the formula, The coordinates of the three-dimensional visual features transformed to the IMU coordinate system in step S1. These are the satellite positioning coordinates converted to the IMU coordinate system in step S1. The actual heading of the body obtained in step S2. The lateral position distance obtained in step S3;

[0051] 3D point coordinates for visual features Satellite positioning coordinates in the IMU coordinate system The actual course of the body and lateral position distance , and Sub-observation equations The corresponding noise components are expanded as follows:

[0052] Feature point reprojection observation equation:

[0053]

[0054] In the formula, For attitude quaternions The determined rotation matrix from the navigation coordinate system to the IMU coordinate system. The coordinates of the feature point in the navigation coordinate system. Extract noise from visual features;

[0055] Equation for absolute position observation of satellite positioning data in the IMU coordinate system:

[0056]

[0057] In the formula, The three-dimensional position component in the state vector. It is the identity matrix. Noise measurement for satellite positioning;

[0058] Actual travel heading observation equation:

[0059]

[0060] In the formula, For heading observation noise;

[0061] Lateral position strong constraint equations:

[0062]

[0063] In the formula, A, B, and C are the coefficients of the tactile paving centerline equation fitted in step S3. This is for lateral constraint observation noise;

[0064] Using Jacobi matrix Calculate Kalman gain And update the state vector:

[0065]

[0066]

[0067] In the formula, the measurement noise covariance matrix Each element is dynamically adjusted according to the signal quality to achieve adaptive fusion positioning; finally, the optimal position and attitude estimation results are output.

[0068] This invention also discloses a smart glasses system for implementing a wearable smart navigation system for blind people. The smart glasses system includes a main body, a sensor module, a computing processing unit, and a feedback model. The main body includes a frame and left and right temples. The sensor module is integrated into the main body and includes a camera located at the front of the frame for acquiring environmental image data, an inertial measurement unit located inside the temples for acquiring head angular velocity and acceleration data, and a miniature antenna and satellite positioning module located above the temples for receiving satellite signals. The computing processing unit is embedded inside the temples and is equipped with a processor and memory for executing the positioning method described in steps S1 to S4. The feedback model is used to feed back the positioning result output in step S4 to the user via Bluetooth audio signals.

[0069] Beneficial effects:

[0070] 1. This invention addresses the issue of course misjudgment caused by the unique behaviors of blind individuals, improving navigation stability. Existing navigation technologies typically assume that the direction of travel is the direction of head movement. However, blind individuals often engage in "auditory search" behavior while walking (i.e., turning their heads to locate ambient sound sources through binaural effects), leading traditional algorithms to misinterpret this as body turning. This invention discloses a full-scene adaptive positioning method and smart glasses for blind individuals that integrates visual semantics and head movement features. By analyzing the variance of head yaw angular velocity and optical flow modulus, it accurately identifies the auditory search state and decouples the course, effectively eliminating the interference of random head movements on path planning and ensuring continuous accuracy of navigation course in complex sound field environments.

[0071] 2. The introduction of strong visual semantic constraints significantly improves lateral positioning accuracy and safety. Addressing the issue that single-satellite positioning is prone to multipath effects and drift in urban canyons, and cannot identify road boundaries, this invention discloses a full-scene adaptive positioning method and smart glasses for the blind that integrates visual semantics and head movement features. It utilizes environmental images to extract semantic features for tactile paving navigation and constructs a strong lateral position constraint equation. This relative positioning method firmly "locks" the user within a safe walking area, effectively preventing blind people from accidentally entering traffic lanes or falling off curbs due to positioning drift, thus greatly improving travel safety.

[0072] 3. Adaptive fusion based on noise characteristics enhances environmental robustness across all scenarios. This invention discloses an adaptive localization method and smart glasses for blind people that integrates visual semantics and head movement features. The established extended Kalman filter model does not rely on fixed empirical parameters but dynamically evaluates confidence based on the noise characteristics of each observation. This invention enables the system to seamlessly adapt to various complex scenarios, such as indoor and outdoor environments and varying light levels, ensuring the continuity and optimality of the localization results.

[0073] 4. The smart glasses disclosed in this invention, based on achieving the three beneficial effects mentioned above, integrate the sensor module and computing unit inside the temples and frame, realizing the lightweight and portability of the device and significantly improving the user's wearing comfort. Simultaneously, as the hardware carrier of the aforementioned positioning method, these smart glasses, through multi-source data fusion and real-time calculation, possess the advantages of high positioning accuracy, good environmental robustness, and high safety and reliability. Attached Figure Description

[0074] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0075] Figure 1 This is an overall flowchart of a wearable intelligent navigation system for blind people to travel according to the present invention.

[0076] Figure 2 This is a schematic diagram of the hardware structure of the smart glasses provided by the present invention.

[0077] Among them, 1—camera, 2—inertial measurement unit, 3—miniature antenna, and 4—satellite positioning module.

[0078] Figure 3 This is a simulation diagram comparing the planar positioning trajectory of the auditory search behavior of blind people in Embodiment 1 of the present invention.

[0079] Figure 4 This is a simulation comparison of lateral positioning errors against GNSS drift under visual semantic constraints in Embodiment 1 of the present invention. Detailed Implementation

[0080] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0081] Example 1: A positioning method to solve the positioning drift problem in the special body postures of blind people.

[0082] like Figure 1 As shown in the figure, this embodiment discloses a wearable smart navigation system for blind people's travel. Its core lies in the deep fusion of multi-source sensing data. The specific implementation method is as follows:

[0083] Step S1, Data Acquisition and Preprocessing: Angular velocity data is acquired using the inertial measurement unit integrated into the smart glasses. and acceleration data Use a camera to capture environmental images It uses a miniature antenna to receive satellite signals and then uses a satellite positioning module to analyze them to obtain GNSS positioning data. .

[0084] For feature points extracted from the environmental image, let their pixel coordinates be... Using the intrinsic parameter matrix By back-projecting it onto the camera coordinate system {C}, we obtain the three-dimensional coordinate points in the camera coordinate system {C}. :

[0085]

[0086] In the formula, It is the depth information of the feature points;

[0087] Subsequently, using the extrinsic parameter matrix Will Transform to IMU coordinate system {I}:

[0088]

[0089] In the formula, The rotation matrix from the camera to the IMU. It is a translation vector. These are the three-dimensional coordinates of the feature point in the IMU coordinate system.

[0090] Because there is a physical deviation between the satellite antenna mounting frame and the center of the IMU, the lever arm vector is used. Transmit satellite positioning data to the IMU center:

[0091]

[0092] In the formula, It is the rotation matrix calculated from the IMU attitude. It is the lever arm vector. This refers to the satellite positioning coordinates of the satellite positioning data in the IMU coordinate system.

[0093] Step S2, Decoupling of Auditory Search Behavior Judgment and Heading: To address the heading misjudgment problem caused by blind individuals "turning their heads to listen for sound to determine location," this step introduces a model based on motion statistics. The system maintains a sliding time window W and calculates the variance of the head yaw rate within this window. and visual optical flow The average modulus length is used to identify whether the user is currently in an auditory search state. If so, the locking relationship between head orientation and body direction of travel is decoupled to obtain the body's actual heading.

[0094] Variance of yaw rate for:

[0095]

[0096] In the formula, The instantaneous yaw rate within window W. is the mean; N is the number of IMU data points within the sliding window.

[0097] Average modulus of visual optical flow for:

[0098]

[0099] In the formula, This represents the horizontal optical flow component of the j-th feature point in the image coordinate system. Let K represent the optical flow component in the vertical direction of the j-th feature point in the image coordinate system, and K represent the total number of feature points extracted from the image.

[0100] A method to identify whether a user is currently in an auditory search state: when the condition is met. At that time, it is determined that the user is in an auditory search state.

[0101] When not in an auditory search state, at any time The attitude quaternion predicted by the extended Kalman filter of -1 is In normal conditions, attitude updates rely on IMU integration:

[0102]

[0103] In the formula, Sampling interval time The equivalent rotation vector within, , Given the quaternion of the posture at time k, we can ultimately obtain the body's actual heading.

[0104]

[0105] When in auditory search mode, the head moves significantly, but the environmental background does not shift significantly, indicating that the user is stationary or exploring at low speed. In this case, the yaw angle observation update should be forcibly isolated to reflect the actual body's heading. Locked to the estimated value from the previous time step, i.e.:

[0106]

[0107] To verify the effectiveness of this step, a scenario was simulated where a blind person walks along a straight line and tilts their head to locate sounds (the head is turned approximately 30 degrees to the right). For example... Figure 3 As shown, this is a comparison of planar positioning trajectories in this scenario. Existing technology (dashed line) cannot distinguish between head movement and body movement, misinterpreting head deflection as body turning, resulting in a positioning trajectory that deviates significantly to the right. In contrast, this invention (solid line) successfully identifies and decouples the heading using step S2, ensuring the positioning trajectory closely follows the actual straight path, effectively avoiding route misjudgment caused by exploration actions.

[0108] Step S3, Visual Semantic Feature Extraction and Constraint Construction: The system utilizes a lightweight convolutional neural network to extract environmental features from images. Perform pixel-level semantic segmentation to identify the center skeleton lines of tactile paving, curbs, or corner areas; utilize the camera intrinsic parameter matrix. Given the current pitch angle, the center skeleton line on the image plane is projected onto the local navigation coordinate system, and the equation of the straight line is obtained by fitting the line. Based on this, the lateral position observation equation is constructed:

[0109]

[0110] in, The system predicts the current position coordinates. The visual observation noise follows a Gaussian distribution. A, B, and C are the coefficients of the fitted straight line equation;

[0111] When tactile paving features are detected in multiple consecutive frames, this lateral position strong constraint This will be used as a strong observation information input filter, forcing the estimated location to converge toward the center of the tactile paving.

[0112] To verify the robustness of this step under weak satellite signal conditions, a scenario was simulated where GNSS signals experienced a 3-meter multipath drift in an urban canyon environment. For example... Figure 4 As shown, this is a comparison of lateral positioning errors. In the prior art (dashed line), lateral errors accumulate rapidly and exceed 3 meters due to GNSS drift, meaning a blind person may have mistakenly entered the vehicle lane. In contrast, this invention (solid line), by introducing strong visual constraints in step S3, forcibly corrects the lateral position by detecting the characteristics of the tactile paving, consistently keeping the lateral positioning error within 0.3 meters (i.e., within the width of the tactile paving), significantly improving the safety of blind people walking.

[0113] Step S4, Full-Scene Adaptive Fusion Localization: This step uses an adaptive extended Kalman filter to achieve the final state estimation. Define the system's state vector. :

[0114]

[0115] in, , , These represent the three-dimensional position, velocity, and attitude quaternions, respectively. , θ represents the zero bias of the accelerometer and gyroscope, respectively, and k is time.

[0116] The state prediction phase of the extended Kalman filter utilizes the IMU kinematic equations for time updates:

[0117]

[0118]

[0119]

[0120] In the formula, This is the rotation matrix corresponding to the quaternion. It is the gravity vector. The equivalent rotation vector within the sampling interval. This is the sampling interval time;

[0121] In the measurement update phase, the extended Kalman filter defines the combined observation vector of the system at time k. for:

[0122]

[0123] In the formula, The coordinates of the three-dimensional visual features transformed to the IMU coordinate system in step S1. These are the satellite positioning coordinates converted to the IMU coordinate system in step S1. The actual heading of the body obtained in step S2. The lateral position distance obtained in step S3;

[0124] 3D point coordinates for visual features Satellite positioning coordinates in the IMU coordinate system The actual course of the body and lateral position distance , and Sub-observation equations The corresponding noise components are expanded as follows:

[0125] Feature point reprojection observation equation:

[0126]

[0127] In the formula, For attitude quaternions The determined rotation matrix from the navigation coordinate system to the IMU coordinate system. The coordinates of the feature point in the navigation coordinate system. Extract noise from visual features;

[0128] Equation for absolute position observation of satellite positioning data in the IMU coordinate system:

[0129]

[0130] In the formula, The three-dimensional position component in the state vector. It is the identity matrix. Noise measurement for satellite positioning;

[0131] Actual travel heading observation equation:

[0132]

[0133] In the formula, For heading observation noise;

[0134] Lateral position strong constraint equations:

[0135]

[0136] In the formula, A, B, and C are the coefficients of the tactile paving centerline equation fitted in step S3. This is for lateral constraint observation noise;

[0137] Using Jacobi matrix Calculate Kalman gain And update the state vector:

[0138]

[0139]

[0140] In the formula, the measurement noise covariance matrix Each element is dynamically adjusted according to the signal quality to achieve adaptive fusion positioning; finally, the optimal position and attitude estimation results are output.

[0141] Example 2: Smart Glasses Device

[0142] like Figure 2 As shown, this embodiment discloses a smart glasses, including a glasses body, a sensor module, a computing processing unit, and a feedback model. The glasses body includes a frame and left and right temples. The sensor module is integrated into the glasses body and includes a camera 1 located at the front of the frame for collecting environmental image data, an inertial measurement unit 2 located inside the temples for collecting head angular velocity and acceleration data, and a miniature antenna 3 and a satellite positioning module 4 located above the temples for receiving satellite signals. The computing processing unit is embedded inside the temples and is equipped with a processor and a memory for the positioning method in Embodiment 1. The feedback model is used to feed back the positioning result output in step S4 of Embodiment 1 to the user via Bluetooth audio signal.

[0143] This embodiment discloses a method for operating a smart glasses device: First, the smart glasses activate the sensor module to collect multi-source data in real time: a camera 1 located at the front of the frame captures environmental image data containing tactile paving features; an inertial measurement unit 2 located inside the temple collects the angular velocity and acceleration data of the user's head; and a miniature antenna 3 and a satellite positioning module 4 located above the temple receive and parse GNSS positioning data. Second, the computing and processing unit receives the above data and executes the positioning algorithm as described in Embodiment 1: first, the multi-source data is preprocessed and a coordinate system is established; then, by analyzing the variance of the head yaw angular velocity and the visual optical flow modulus, it identifies whether the user is in an auditory search state and decouples the head-body heading; simultaneously, semantic segmentation is performed on the environmental image, and the center line of the tactile paving is extracted to construct a strong lateral position constraint; then, the actual heading, lateral constraint, and positioning data are input into an extended Kalman filter for adaptive fusion calculation, outputting the optimal position and attitude estimation results. Finally, the feedback model obtains the positioning results, converts them into voice navigation commands, and feeds them back to the user via Bluetooth audio signals to achieve precise assisted navigation for the blind.

[0144] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A wearable intelligent navigation system for blind people's travel, characterized in that: Includes the following steps: Step S1, Data Acquisition and Preprocessing: Angular velocity is acquired using smart glasses. acceleration Environmental images and GNSS positioning data Using the intrinsic parameter matrix Will Feature points extracted Back projection onto the camera coordinate system yields Then use the extrinsic parameter matrix to Transform to IMU coordinate system; simultaneously utilize lever arm vectors satellite positioning data Transferred to the IMU center to ensure and Consistency in physical space; Step S2, Decoupling of Auditory Search Behavior Judgment and Heading: Analyze the variance of head yaw angular velocity and the average modulus of visual optical flow to identify whether the user is currently in an auditory search state; if not, return to perform search behavior judgment; if so, decouple the locking relationship between head orientation and body direction of travel to obtain the true heading of the body. Step S3, Visual Semantic Feature Extraction and Constraint Construction: Perform semantic segmentation on the environmental image, identify tactile paving features, and construct a strong lateral position constraint equation based on the geometric relationship between the user and the features; Step S4, Full-Scene Adaptive Fusion Localization: Establish an extended Kalman filter model, using the real travel heading, strong lateral position constraints, and feature points and localization data from step S1 as observation vectors. Evaluate the confidence level of each observation based on its noise characteristics, and output the optimal position and attitude estimation results.

2. The method as described in claim 1, characterized in that: The implementation method for step S1 is as follows: For feature points extracted from the environmental image, let their pixel coordinates be... Using the intrinsic parameter matrix By back-projecting it onto the camera coordinate system, we can obtain the three-dimensional coordinate points in the camera coordinate system. : In the formula, It is the depth information of the feature points; Subsequently, using the extrinsic parameter matrix Will Transform to IMU coordinate system: In the formula, The rotation matrix from the camera to the IMU. It is a translation vector. These are the three-dimensional coordinates of the feature point in the IMU coordinate system. Because there is a physical deviation between the satellite antenna mounting frame and the center of the IMU, the lever arm vector is used. Transmit satellite positioning data to the IMU center: In the formula, It is the rotation matrix calculated from the IMU attitude. It is the lever arm vector. The satellite positioning coordinates of the satellite positioning data in the IMU coordinate system.

3. The method as described in claim 1, characterized in that: The variance of the yaw rate mentioned in step S2 for: In the formula, The instantaneous yaw rate within the window. is the mean; N is the number of IMU data points within the sliding window.

4. The method as described in claim 1, characterized in that: The average modulus of the visual optical flow described in step S2 for: In the formula, This represents the horizontal optical flow component of the j-th feature point in the image coordinate system. Let K represent the optical flow component in the vertical direction of the j-th feature point in the image coordinate system, and K represent the total number of feature points extracted from the image.

5. The method as described in claim 1, characterized in that: The method for identifying whether the user is currently in an auditory search state as described in step S2: when the condition is met... At that time, it is determined that the user is in an auditory search state.

6. The method as described in claim 1, characterized in that: The method for obtaining the body's true heading in step S2 is as follows: when not in an auditory search state, at any time... The attitude quaternion predicted by the extended Kalman filter of -1 is In normal conditions, attitude updates rely on IMU integration: In the formula, Sampling interval time The equivalent rotation vector within, , Given the quaternion of the posture at time k, we can ultimately obtain the body's actual heading. ； When in auditory search mode, the head moves significantly, but the environmental background does not shift significantly, indicating that the user is stationary or exploring at low speed. In this case, the yaw angle observation update should be forcibly isolated to reflect the actual body's heading. Locked to the estimated value from the previous time step, i.e.: 。 7. The method as described in claim 1, characterized in that: The implementation method for step S3 is as follows: Using lightweight convolutional neural networks to analyze environmental images Perform pixel-level semantic segmentation to identify the central skeleton line of the tactile paving area; utilize the camera intrinsic parameter matrix Given the current pitch angle, the center skeleton line on the image plane is projected onto the local navigation coordinate system, and the equation of the straight line is obtained by fitting the line. Based on this, the lateral position observation equation is constructed: in, The system predicts the current position coordinates. The visual observation noise follows a Gaussian distribution. ABC are the coefficients of the fitted straight line equation; When tactile paving features are detected in multiple consecutive frames, this lateral position strong constraint This will be used as a strong observation information input filter, forcing the estimated location to converge toward the center of the tactile paving.

8. The method as described in claim 1, characterized in that: The implementation method for step S4 is as follows: Define the state vector of the system : in, , , These represent the three-dimensional position, velocity, and attitude quaternions, respectively. , θ represents the zero bias of the accelerometer and gyroscope, respectively, and k is time. The state prediction phase of the extended Kalman filter utilizes the IMU kinematic equations for time updates: In the formula, This is the rotation matrix corresponding to the quaternion. It is the gravity vector. The equivalent rotation vector within the sampling interval. This is the sampling interval time; In the measurement update phase, the extended Kalman filter defines the combined observation vector of the system at time k. for: In the formula, The coordinates of the three-dimensional visual features transformed to the IMU coordinate system in step S1. These are the satellite positioning coordinates converted to the IMU coordinate system in step S1. The actual heading of the body obtained in step S2. The lateral position distance obtained in step S3; 3D point coordinates for visual features Satellite positioning coordinates in the IMU coordinate system The actual course of the body and lateral position distance , and Sub-observation equations The corresponding noise components are expanded as follows: Feature point reprojection observation equation: In the formula, For attitude quaternions The determined rotation matrix from the navigation coordinate system to the IMU coordinate system. The coordinates of the feature point in the navigation coordinate system. Extract noise from visual features; Equation for absolute position observation of satellite positioning data in the IMU coordinate system: In the formula, The three-dimensional position component in the state vector. It is the identity matrix. Noise measurement for satellite positioning; Actual travel heading observation equation: In the formula, For heading observation noise; Lateral position strong constraint equations: In the formula, A, B, and C are the coefficients of the tactile paving centerline equation fitted in step S3. This is for lateral constraint observation noise; Using Jacobi matrix Calculate Kalman gain And update the state vector: In the formula, the measurement noise covariance matrix Each element is dynamically adjusted according to the signal quality to achieve adaptive fusion positioning and output the optimal position and attitude estimation results.

9. Smart glasses that implement the method as described in claim 1, 2, 3, 4, 5, 6, 7 or 8, characterized in that: The glasses include a main body, a sensor module, a computing processing unit, and a feedback model. The main body includes a frame and left and right temples. The sensor module is integrated into the main body and includes a camera located at the front of the frame for acquiring environmental image data, an inertial measurement unit located inside the temples for acquiring head angular velocity and acceleration data, and a miniature antenna and satellite positioning module located above the temples for receiving satellite signals. The computing processing unit is embedded inside the temples and is equipped with a processor and a memory for executing the positioning method as described in any one of claims 1 to 5. The feedback model is used to feed back the positioning result output in step S4 to the user via a Bluetooth audio signal.

Images