Blind sidewalk identification method and system based on intelligent glasses

Abstract

The invention relates to the technical field of intelligent glasses, and discloses a blind sidewalk recognition method and system based on intelligent glasses, and the method comprises the steps: collecting acceleration data, determining the advancing direction of a user, collecting image data in front of the user in real time, inputting the image data into a target detection model, recognizing an obstacle and a blind sidewalk identifier in an image, performing spatial distribution modeling on the obstacle, and judging whether the obstacle is located in a user travel path or a blind sidewalk area; generating an optimized route based on an improved heuristic search route planning method when the blind sidewalk is located in the advancing route or the blind sidewalk area; the real-time movement direction is obtained and compared with the optimized route, when it is judged that the advancing direction of the user deviates from the optimized route, feedback prompt is conducted, and the feedback prompt comprises voice prompt and vibration feedback. According to the method, the perception precision, the path recognition capability and the navigation safety of visually impaired people on the blind sidewalk in a complex dynamic environment are improved, and the intelligence, the man-machine interaction and the actual use reliability of blind sidewalk recognition are enhanced.

Description

Technical Field

[0001] This invention relates to the field of smart glasses technology, and more specifically, to a method and system for identifying tactile paving based on smart glasses. Background Technology

[0002] With the continuous improvement of accessibility facilities on urban roads, tactile paving, as an important infrastructure providing travel assistance for visually impaired people, is widely laid in municipal roads, stations, schools, and public places. Traditional methods of identifying and navigating tactile paving mainly rely on visually impaired people perceiving tactile paving information through canes, tactile feedback from the soles of their feet, or auditory feedback. However, in the complex and dynamic urban environment, tactile paving is often obstructed or occupied by factors such as temporary placement of items, pedestrian stops, and non-motorized vehicle parking, making it difficult for visually impaired users to accurately obtain passage routes using simple physical perception methods.

[0003] In recent years, some wearable assistive devices have attempted to acquire visual information through cameras and combine it with image recognition algorithms to detect and provide prompts for obstacles or tactile paving. However, existing devices have some shortcomings, such as: most can only perform static image analysis, lacking registration and interaction with the user's real-time direction of travel; lacking dynamic modeling of the spatial relationship between tactile paving structure and obstacles, making it difficult to perform path-level obstacle avoidance planning; and unable to determine whether the user has deviated from a feasible route based on their actual posture, resulting in a lagging or inaccurate feedback mechanism that cannot effectively achieve closed-loop guidance.

[0004] Therefore, it is necessary to design a method and system for identifying tactile paving based on smart glasses to solve the problems existing in the current technology. Summary of the Invention

[0005] In view of this, the present invention proposes a method and system for tactile paving based on smart glasses, aiming to solve the current problems of lack of registration and interaction with the user's real-time direction of travel, difficulty in path-level obstacle avoidance planning, and inability to effectively achieve closed-loop guidance.

[0006] In one aspect, the present invention proposes a method for identifying tactile paving based on smart glasses, comprising: Acceleration data is collected, and the user's direction of travel is determined based on the acceleration data. Image data in front of the user is collected in real time by a camera set on the side of the temple of the glasses. The image data is input into the target detection model to identify obstacles in the image and output the location information, size information and category label of the obstacles. The image data is processed based on edge detection to identify tactile paving signs, and the spatial distribution of the obstacles is modeled. Based on the tactile paving signs and the user's direction of travel, it is determined whether the obstacles are located in the user's travel path or within the tactile paving area. When it is determined that the obstacle is located in the user's walking path or the blind path area, an optimized route is generated based on an improved heuristic search path planning method; The user's real-time posture information is obtained based on the gyroscope sensor, and the real-time movement direction is obtained. The real-time movement direction is compared with the optimized route. When it is determined that the user's movement direction deviates from the optimized route, feedback prompts are given, including voice prompts and vibration feedback.

[0007] Furthermore, when collecting acceleration data and determining the user's direction of travel based on the acceleration data, the process includes: Acceleration data sequences of the user during movement are collected using a triaxial accelerometer; noise filtering is performed on the acceleration data sequences, including the use of sliding window mean filtering; The processed acceleration data sequence is segmented according to time windows, and the principal direction vectors in each time period are extracted. The system identifies continuous walking gait rhythms by detecting gait cycles and obtains the user's walking direction based on the vector angle between the main direction vector and the direction of ground gravitational acceleration.

[0008] Furthermore, when inputting the image data into the target detection model to identify obstacles in the image and outputting the location information, size information, and category label of the obstacles, the process includes: The image data is preprocessed to obtain the image to be processed. The preprocessing includes image size normalization, brightness enhancement, and color space conversion. The image to be processed is input into the target detection model, which includes YOLOv5 built based on a convolutional neural network; The multi-scale spatial feature map of the image to be processed is extracted by the feature extraction layer of the model, and candidate regions of potential obstacle areas are determined. The candidate region is input into the classification branch and the regression branch. The classification branch is used to determine the target category label. The target categories include pedestrians, vehicles, roadblocks, boxes, and traffic cones. The regression branch is used to predict the bounding box position coordinates and size parameters of the corresponding target. Confidence scores and non-maximum suppression are applied to all detection results, and the final obstacle recognition result set is output. The result set includes the two-dimensional image coordinates, bounding box width and height, category label and confidence score of each obstacle.

[0009] Furthermore, when processing the image data based on edge detection to identify tactile paving markings, the process includes: The image to be processed is subjected to image enhancement processing, which includes histogram equalization and grayscale transformation. An edge detection algorithm is applied to the enhanced image to be processed to extract boundary features, including the Canny edge detection algorithm. Hough transform is used in edge images to identify candidate regions of tactile paving with continuous line structures, and noisy edges are filtered out based on line segment length and arrangement regularity. A secondary verification is performed by combining color features and tactile paving texture patterns to determine whether it is a valid tactile paving marking area; when it is determined to be a valid tactile paving marking area, the spatial location, extension direction and boundary structure information of the tactile paving marking area are extracted.

[0010] Furthermore, when determining whether an obstacle is located within the user's walking path or the tactile paving area based on the tactile paving markings and the user's direction of travel, the process includes: Extract the extension direction of the tactile paving marking area and extract the center line of the tactile paving area; The cosine similarity between the centerline direction vector and the user's walking direction is calculated to obtain the user's walking trend relative to the tactile paving. Construct a spatial relationship model based on three-dimensional reconstructed coordinates, and determine the relative spatial distance between the obstacle and the center line of the tactile paving according to the bounding box position and size of the obstacle; When the straight-line distance between the center projection of the boundary box of the obstacle and the center line of the tactile paving is less than a distance threshold, the obstacle is determined to be located in the user's walking path or within the tactile paving area. Obstacles that meet the obstacle determination criteria are marked as blocking objects.

[0011] Furthermore, when generating optimized routes based on the improved heuristic search path planning method, the following are included: A two-dimensional grid map is constructed that includes the tactile paving marking area and the obstructing object. The grid map encodes the tactile paving area as a zero-cost passage area, defines the area outside the tactile paving as a first-level cost passage area, and encodes the area where the obstructing object is located as an impassable area. Set the start and end points based on the user's current location and the target direction of the tactile paving; Define an open set and a closed set and initialize the cost function value. The open set stores the nodes to be explored, and initially only contains the starting point. The closed set stores the nodes that have been explored, and is initially empty. From the open set, the path of the node with the lowest cost is gradually selected and extended towards the key location coordinates; Explore all possible path directions for the current node, update the path cost, and if the neighbor node is not in the open set, add the neighbor node to the open set and record the parent node of the neighbor node; if the neighbor node is in the open set, but the new path cost is smaller, update the parent node and the path cost. Repeat the exploration until the current node coincides with the endpoint, then backtrack from the endpoint to the parent node until the starting point, and use this path as the optimized route.

[0012] Furthermore, when acquiring the user's real-time attitude information and real-time motion direction based on the gyroscope sensor, this includes: The angular velocity data of the user's head around the X, Y and Z axes are collected in real time based on a three-axis gyroscope sensor. The collected angular velocity data is fused with the simultaneously collected triaxial accelerometer data, and complementary filtering is used to perform stable estimation of attitude angles, outputting triaxial attitude information including pitch angle, roll angle and yaw angle. Estimate the change in the user's orientation based on the time-varying trend of the yaw angle, and correct the current orientation based on the user's direction of travel. The corrected motion direction vector is used as the real-time motion direction.

[0013] Furthermore, when comparing the real-time motion direction with the optimized route, the process includes: The direction vector of the current position is obtained based on all optimized routes. The direction vector is compared with the real-time movement direction, and the deviation is determined based on the comparison result. When the angle between the direction vector and the vector of the real-time motion direction is greater than the angle threshold, a deviation is determined to have occurred; When the angle between the direction vector and the vector of the real-time motion direction is less than or equal to the angle threshold, it is determined that no deviation has occurred.

[0014] Furthermore, when it is determined that the user's direction of travel deviates from the optimized route, the feedback prompt includes: If the user's deviation direction is to the left, control the left temple vibration motor to start the vibration mode and activate the voice prompt; If the user's deviation direction is to the right, control the vibration motor of the right temple to start the vibration mode and activate the voice prompt; The feedback frequency is adjusted according to the deviation angle. When the deviation angle continues to increase or the duration exceeds the time threshold, the vibration frequency and voice prompt interval density are increased. When the user's direction of travel is realigned with the optimized route and the deviation angle is less than the included angle threshold, the feedback prompt is terminated.

[0015] Compared with existing technologies, the advantages of this invention are as follows: It utilizes an accelerometer to acquire the user's direction of travel in real time, combines this with images captured by a camera mounted on the side of the temples of the glasses, and accurately identifies various obstacles, including pedestrians, vehicles, and roadblocks, through a target detection model. Furthermore, it identifies tactile paving markings through edge detection and texture analysis, and determines whether an obstacle is located on the user's path based on spatial relationship modeling. When a path is blocked, an improved heuristic search algorithm is introduced to reconstruct the optimal obstacle avoidance route. Simultaneously, it integrates real-time attitude information obtained from a gyroscope to dynamically estimate the user's actual direction of movement and compares it with the planned route, enabling immediate judgment of path deviation. By providing navigation information through multimodal feedback methods such as voice and vibration, it effectively improves the perception accuracy, path recognition ability, and navigation safety of visually impaired individuals in complex dynamic environments, enhancing the intelligence, human-computer interaction, and practical reliability of tactile paving identification.

[0016] On the other hand, this application also provides a tactile paving recognition system based on smart glasses, for applying the above-mentioned tactile paving recognition method based on smart glasses, including: The acquisition unit is configured to acquire acceleration data, determine the user's direction of travel based on the acceleration data, acquire image data in front of the user in real time through a camera set on the side of the temple of the glasses, input the image data into the target detection model, identify obstacles in the image, and output the location information, size information and category label of the obstacles; The judgment unit is configured to process the image data based on edge detection to identify tactile paving signs, and to perform spatial distribution modeling of the obstacles, and to determine whether the obstacles are located in the user's walking path or the tactile paving area based on the tactile paving signs and the user's walking direction. The processing unit is configured to generate an optimized route based on an improved heuristic search path planning method when it is determined that the obstacle is located in the user's walking path or the blind path area. The prompting unit is configured to acquire the user's real-time posture information and real-time movement direction based on a gyroscope sensor, compare the real-time movement direction with the optimized route, and provide feedback prompts when it is determined that the user's movement direction deviates from the optimized route. The feedback prompts include voice prompts and vibration feedback.

[0017] It is understandable that the above-mentioned methods and systems for identifying tactile paving based on smart glasses have the same beneficial effects, and will not be elaborated further here. Attached Figure Description

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a method for identifying tactile paving based on smart glasses, provided in an embodiment of the present invention; Figure 2 This is a functional block diagram of a tactile paving recognition system based on smart glasses, provided in an embodiment of the present invention. Detailed Implementation

[0019] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Traditional tactile paving identification suffers from deficiencies in obstacle spatial relationship modeling and real-time path planning in dynamic environments. The lack of a real-time registration mechanism between the user's direction of travel and the tactile paving's extension direction leads to increased errors in obstacle location determination. Furthermore, the absence of a spatial correlation model between the tactile paving area and the projected coordinates of obstacles prevents the path planning algorithm from accurately distinguishing between passable and obstructed areas. These issues directly cause navigation paths to deviate from the tactile paving centerline, increasing the risk of user collisions, and the feedback mechanism cannot promptly correct these trajectory deviations.

[0021] For example, when a user is walking along a tactile paving path, a temporarily parked bicycle may partially obstruct the edge of the paving. Traditional systems only identify the obstacle's bounding box using static images, without combining the user's current direction of movement with the direction of the tactile paving's centerline for spatial projection calculation. When the distance between the obstacle's center of gravity projection and the tactile paving's centerline is at a critical threshold, it cannot accurately determine whether the obstacle has encroached on the user's path. In this case, the path planning module, lacking precise blocking area encoding, may generate a route that bypasses the obstacle but deviates from the tactile paving, or incorrectly retain path nodes that intrude into the tactile paving area. The lack of a real-time comparison mechanism between the head posture angle data collected by the gyroscope sensor and the path direction vector further prevents immediate feedback when the user's actual direction of movement deviates from the planned route.

[0022] If the above problems are not addressed, navigation will be unable to maintain consistency between path planning and user movement in complex and dynamic environments, leading to conflicts between obstacle avoidance decisions and the tactile paving's following target. Errors in calculating the projected position of obstacles can cause the path planning module to generate suboptimal routes, increasing the user's detour distance and energy consumption. The lack of a direction deviation detection mechanism allows the deviation between the user's actual trajectory and the planned path to accumulate over time, ultimately causing navigation failure. These issues will severely reduce the environmental adaptability and navigation reliability of tactile paving recognition, threatening user safety.

[0023] For this, please refer to Figure 1 As shown, this application proposes a method for identifying tactile paving based on smart glasses, comprising: S100: Collects acceleration data, determines the user's direction of travel based on the acceleration data, and collects image data in front of the user in real time through a camera set on the side of the temple of the glasses. The image data is then input into the target detection model to identify obstacles in the image and output the location information, size information and category label of the obstacles.

[0024] S200: Based on edge detection, image data is processed to identify tactile paving signs and spatial distribution modeling of obstacles is performed. Based on the tactile paving signs and the user's direction of travel, it is determined whether the obstacle is located in the user's travel path or within the tactile paving area.

[0025] S300: When it is determined that an obstacle is located in the user's travel path or the blind path area, an optimized route is generated based on an improved heuristic search path planning method.

[0026] S400: Based on the gyroscope sensor, it obtains the user's real-time posture information and real-time movement direction. It compares the real-time movement direction with the optimized route. When it is determined that the user's movement direction deviates from the optimized route, it provides feedback prompts, including voice prompts and vibration feedback.

[0027] The acceleration data refers to the dynamic acceleration information acquired by sensors during the user's movement. Specifically, the STK8321 can be used to extract the main direction vector of the user's walking direction. Real-time image data acquisition by the camera refers to the continuous acquisition of visual information about the foreground environment through a camera device mounted on the side of the temples of the glasses. This can be achieved using a wide-angle lens with an autofocus module to capture real-time images of the tactile paving area and obstacles. The object detection model refers to an object recognition algorithm based on a convolutional neural network. Specifically, the YOLOv5 architecture can be used to achieve multi-scale feature extraction and candidate region classification regression, outputting the location, size, and category information of obstacles. Edge detection processing refers to extracting the contour features of tactile paving markings through image algorithms. Specifically, Canny edge detection combined with Hough transform can be used to identify continuous line segment structures, distinguishing tactile paving areas from non-tactile paving areas. Spatial distribution modeling refers to calculating the relative distance relationship between the obstacle position and the center line of the tactile paving. Specifically, a three-dimensional coordinate projection and distance threshold determination method can be used to determine whether an obstacle obstructs the walking path. The improved heuristic search path planning refers to a path generation algorithm based on a grid map. Specifically, it can employ the A* algorithm to optimize the cost function and dynamically update the node expansion strategy to avoid obstacles and plan feasible routes. The real-time attitude information from the gyroscope sensor refers to estimating the user's head orientation by fusing angular velocity data with accelerometer readings. This can be achieved using a complementary filtering algorithm combined with yaw angle trend analysis to correct the user's actual direction of movement. The feedback prompts guide the user to adjust their direction of travel through tactile and auditory signals. This can be achieved by using a temple vibration motor and a voice broadcast module working together to provide real-time correction instructions when deviating from the optimized route.

[0028] This application integrates multi-sensor data with real-time image processing to construct a dynamic spatial relationship model, enabling collaborative identification of blind path areas and obstacles. Combined with a closed-loop feedback mechanism for user posture and path planning, it forms a complete technical chain from environmental perception to behavior guidance.

[0029] The working process and principle of this application are as follows: First, an accelerometer collects the user's acceleration data during walking, and determines the user's direction of travel based on this data. Simultaneously, a camera on the side of the smart glasses' temple collects real-time image data in front of the user. This image data is input into a target detection model to identify obstacles in the image and outputs the obstacle's location information, size information, and category label.

[0030] Next, edge detection processing is performed on the acquired image data to identify tactile paving markings. Simultaneously, spatial distribution modeling is performed on the identified obstacles. Based on the identified tactile paving markings and the user's direction of travel, it is determined whether an obstacle is located within the user's travel path or the tactile paving area.

[0031] When an obstacle is detected in the user's path or within the tactile paving area, an improved heuristic path planning method is activated to generate an optimized route. This route aims to avoid the obstacle while staying on the tactile paving as much as possible.

[0032] It also uses a gyroscope sensor to acquire the user's real-time posture information, thereby determining the user's real-time direction of movement. This real-time direction of movement is compared with the previously generated optimized route. If it is determined that the user's direction of movement has deviated from the optimized route, feedback will be provided. This feedback includes voice prompts and vibration feedback to guide the user back to the correct route.

[0033] This working principle makes full use of data from multiple sensors and advanced algorithms to achieve real-time identification of tactile paving and obstacles, path planning, and user guidance, thereby providing visually impaired users with safer and more reliable navigation assistance.

[0034] As a preferred embodiment, the solution of this application is specifically implemented as follows: The smart glasses are equipped with a three-axis accelerometer to collect acceleration data during the user's movement. After filtering, the acceleration data is segmented into time windows to extract the principal direction vector. By detecting the continuous walking gait rhythm and combining it with the vector angle between the principal direction vector and the direction of ground gravitational acceleration, the user's walking direction is determined.

[0035] A high-resolution camera is mounted on the side of the temple of the glasses to capture images of the front in real time. After preprocessing, the images are input into the YOLOv5 object detection model. The model extracts multi-scale spatial feature maps to identify potential obstacle regions. The classification branch determines the target category, and the regression branch predicts the bounding box position and size. After non-maximum suppression, the final obstacle recognition result is output.

[0036] Edge detection is performed on the image, and Hough transform is applied to identify candidate regions for tactile paving with continuous line structures. Verification is then performed by combining color features and texture patterns to extract the spatial location, extension direction, and boundary structure information of valid tactile paving marking regions.

[0037] A 3D spatial relationship model is constructed, and the distance between the centroid projection of the obstacle's bounding box and the centerline of the tactile paving is calculated. When the distance is less than a threshold, the obstacle is determined to be located within the user's walking path or the tactile paving area.

[0038] A two-dimensional grid map is constructed based on the locations of tactile paving areas and obstacles. Tactile paving areas are encoded as zero-cost traversable areas, and obstacle areas are encoded as impassable areas. An improved A* algorithm is used for path search to generate optimized routes that avoid obstacles and follow the centerline of the tactile paving as closely as possible.

[0039] The gyroscope sensor collects three-axis angular velocity data of the user's head, which is then fused with acceleration data to estimate the user's attitude angles. The change in motion orientation is estimated based on the yaw angle trend, and combined with the direction of travel to obtain the real-time motion direction vector.

[0040] The real-time motion direction is compared with the optimized route direction vector. When the vector angle exceeds a threshold, a deviation is determined. Based on the deviation direction, the corresponding side mirror arm vibration motor is activated to start the vibration mode and a voice prompt is initiated. The feedback frequency increases with the deviation angle, and feedback terminates when the alignment is restored.

[0041] Through the above-described scheme, this application achieves dynamic spatial relationship modeling between tactile paving and obstacles, ensuring real-time registration of the user's travel direction with the planned path. The multi-sensor fusion mechanism improves the accuracy of obstacle location determination, and the improved path planning algorithm ensures that the generated optimized route extends along the centerline of the tactile paving while avoiding obstacles. Real-time direction deviation detection and dynamic feedback mechanisms effectively prevent the accumulation of navigation errors. These technological innovations improve the adaptability and reliability of tactile paving recognition in complex dynamic environments, reduce user detour distance and energy consumption, and enhance traffic safety through timely direction correction.

[0042] In some of the solutions described above in this application, when collecting acceleration data to determine the user's direction of travel, the original acceleration data may contain noise interference, resulting in inaccurate extraction of the main direction vector. At the same time, the influence of gait period on the continuity of direction is not considered, causing deviations in the judgment of the user's direction of travel.

[0043] This application further proposes a method for acquiring acceleration data sequences of a user during walking using a triaxial accelerometer. The acceleration data sequences are subjected to noise filtering, including sliding window mean filtering. The processed acceleration data sequences are then segmented by time windows, and the principal direction vector for each time period is extracted. Continuous walking gait rhythms are identified through gait period detection, and the user's walking direction is obtained based on the vector angle between the principal direction vector and the direction of ground gravitational acceleration.

[0044] The sliding window mean filtering uses a window length of 5 to 10 sampling points, with time window segments ranging from 0.5 to 1 second. The principal direction vector is extracted using principal component analysis to determine the direction of maximum variance in the acceleration data sequence. Gait cycle detection employs a peak detection algorithm to identify the peaks and troughs of periodic fluctuations in the acceleration sequence. The gait cycle duration is calculated based on the interval between adjacent peaks, and the user's walking direction is dynamically corrected by combining the changing trend of the angle between the principal direction vector and the direction of gravitational acceleration.

[0045] Specifically, the triaxial accelerometer collects raw acceleration data at a frequency of 100Hz. After sliding window mean filtering, high-frequency noise and transient jitter interference are removed. The processed data is divided into continuous segments according to fixed time windows. Principal component analysis is used to extract the principal direction vector for each segment, reflecting the user's main movement trend within that time period. The gait cycle detection module analyzes the changes in acceleration amplitude, identifies the peak corresponding to the foot's contact with the ground, and the interval between adjacent peaks constitutes the gait cycle, ensuring that the direction vector update is synchronized with the user's actual steps. By calculating the vector angle between the principal direction vector and the direction of gravitational acceleration, the directional error introduced by sensor tilt is eliminated. Combined with gait rhythm, the direction output is dynamically adjusted to improve the continuity and accuracy of the walking direction judgment.

[0046] As a preferred embodiment, the solution of this application is specifically implemented as follows: Acceleration data sequences of the user during movement are collected using a triaxial accelerometer. Noise filtering is applied to the acceleration data sequences, including sliding window averaging. Specifically, a sliding window of 20 sampling points is used to calculate the average of the acceleration data for each axis to smooth short-term noise fluctuations.

[0047] The processed acceleration data sequence is segmented according to time windows, and the principal direction vectors for each time period are extracted. For example, a time window is divided into 2-second intervals, and the average acceleration values ​​along the x, y, and z axes within the window are calculated to form the principal direction vectors.

[0048] The system identifies continuous gait rhythms by detecting gait cycles and obtains the user's walking direction based on the vector angle between the principal direction vector and the direction of gravitational acceleration on the ground. Specifically, each complete gait cycle is detected using the periodic characteristics of the acceleration waveform. Within each gait cycle, the angle between the principal direction vector and the direction of gravitational acceleration is calculated, and this angle represents the user's walking direction relative to the ground.

[0049] Through the above technical solution, this application can accurately obtain the user's real-time walking direction, providing important basic data for subsequent tactile paving identification and path planning. By employing sliding window filtering and gait period detection, this method can effectively suppress noise interference in the acceleration signal, improving the stability and accuracy of direction estimation. Simultaneously, by extracting the principal direction vector in segments, this method can adapt to changes in the user's turning direction during walking, achieving dynamic tracking of the walking direction. This provides a reliable direction reference for visually impaired users navigating in complex environments.

[0050] In some of the solutions described above in this application, image data is directly input into the target detection model for obstacle recognition. However, in complex lighting or low-contrast scenes, the original image may have uneven brightness, color distortion, or size differences, which makes it difficult to extract model features, reduces the accuracy of candidate region localization, and affects the accuracy of obstacle category determination and position regression.

[0051] This application further proposes preprocessing the image data to obtain the image to be processed. Preprocessing includes image size normalization, brightness enhancement, and color space conversion. The image to be processed is input into an object detection model, which includes YOLOv5 based on a convolutional neural network. The model's feature extraction layer extracts multi-scale spatial feature maps from the image to be processed and determines candidate regions for potential obstacles. These candidate regions are input into a classification branch and a regression branch. The classification branch determines the target category label, which includes pedestrians, vehicles, roadblocks, boxes, and traffic cones. The regression branch predicts the bounding box coordinates and size parameters of the corresponding target. Confidence scoring and non-maximum suppression are applied to all detection results, outputting the final obstacle recognition result set. The result set includes the two-dimensional image coordinates, bounding box width and height, category label, and confidence score for each obstacle.

[0052] Image size normalization scales the input image to a fixed resolution using bilinear interpolation, eliminating the impact of obstacle size differences at different distances on the model input. Brightness enhancement employs a contrast-limited adaptive histogram equalization algorithm, segmenting the image into local sub-blocks and independently adjusting the grayscale distribution to improve the visibility of details in dark areas. Color space conversion converts the RGB image to YUV format, separating the luminance and chrominance components to enhance color contrast. The YOLOv5 backbone network uses a CSPDarknet53 structure, reducing computational redundancy through cross-stage local connections. Multi-scale spatial feature maps fuse shallow high-resolution features with deep semantic features through a path aggregation network, enhancing the detection capability of small obstacles. The classification branch uses the Softmax function to calculate the class probability distribution, while the regression branch uses the sigmoid function to constrain the predicted range of coordinate offsets. Confidence scoring is calculated based on the product of the classification probability and the intersection-union ratio of the bounding boxes, and non-maximum suppression eliminates redundant detection by iteratively selecting the highest-scoring bounding boxes and suppressing overlapping boxes.

[0053] Specifically, in the preprocessing stage, the original images captured by the camera are normalized to a uniform 640×640 pixels to avoid fluctuations in model input due to inconsistent image resolution. The brightness enhancement module divides the image into blocks, with each 8×8 pixel sub-block undergoing histogram equalization independently, and a contrast limit threshold is used to prevent noise amplification. Color space conversion decomposes the RGB channels into YUV channels, and gamma correction is performed separately on the Y channel to enhance brightness consistency. After the processed image is input into the YOLOv5 model, the CSPDarknet53 backbone network extracts features at different levels through convolutional layers and residual modules. The path aggregation network upsamples and stitches the feature maps of layers 17, 20, and 23 to generate feature pyramids with three scales: 80×80, 40×40, and 20×20. Candidate regions are matched on the feature maps using preset anchor boxes to select initial predicted boxes that are close to the size of real obstacles. The classification branch calculates the probability values ​​for five categories, including pedestrians and vehicles, for each anchor box. The regression branch predicts the center point offset and width / height scaling factors of the bounding box, which are then decoded to obtain normalized coordinates. Detection boxes with a confidence score exceeding 0.5 enter a non-maximum suppression process, where overlapping boxes are removed using an intersection-over-union (IoU) threshold of 0.45. The final output includes the obstacle category, bounding box coordinates, and confidence score. This process improves image quality through standardized preprocessing and combines multi-scale feature fusion with an efficient post-processing mechanism to achieve rapid and accurate obstacle recognition in dynamic environments.

[0054] As a preferred embodiment, the solution of this application is specifically implemented as follows: Image data undergoes preprocessing to obtain the image to be processed. Preprocessing includes image size normalization, brightness enhancement, and color space conversion. Specifically, the acquired raw images are resized to a uniform size, for example, 416x416 pixels. Histogram equalization is used to improve image contrast. The RGB color space is converted to the YUV color space.

[0055] The image to be processed is input into the object detection model. The object detection model includes YOLOv5, which is built based on a convolutional neural network. The YOLOv5 model consists of a backbone network, a feature pyramid network, and a detection head. The backbone network adopts the CSPDarknet53 structure and is used to extract image features.

[0056] The model's feature extraction layer extracts multi-scale spatial feature maps from the image to be processed and identifies candidate regions for potential obstacles. The feature extraction layer includes feature maps at five scales, corresponding to 1 / 8, 1 / 16, 1 / 32, 1 / 64, and 1 / 128 of the original image size, respectively. Each feature map is divided into a grid, and each grid predicts three candidate boxes at different scales.

[0057] The candidate region input consists of a classification branch and a regression branch. The classification branch determines the target category label, which includes pedestrians, vehicles, road barriers, boxes, and traffic cones. The regression branch predicts the bounding box coordinates and dimensions of the corresponding target. The classification branch uses the softmax activation function to output the category probability, while the regression branch directly outputs the center coordinates, width, and height of the bounding box.

[0058] All detection results are evaluated for confidence and non-maximum suppression (NMS), resulting in a final obstacle recognition set. The set includes the 2D image coordinates, bounding box width and height, class label, and confidence score for each obstacle. The confidence threshold is set to 0.5, and the IoU threshold for NMS is set to 0.4. The top 20 detection results with the highest confidence scores are then retained.

[0059] Through the above technical solutions, this application can accurately detect and locate various types of obstacles in images. Using the YOLOv5 model for real-time target detection, single-frame image analysis can be completed within milliseconds. Multi-scale feature extraction and bounding box generation strategies improve the detection capability for targets of different sizes. Non-maximum suppression effectively removes duplicate detections, improving the reliability of the results. The output obstacle location, size, and category information provide important information for subsequent path planning and obstacle avoidance.

[0060] In some of the solutions described above in this application, although the target detection model can identify obstacles, the texture features of the tactile paving markings may be interfered with by background noise in complex environments or low-contrast scenes, resulting in insufficient recognition accuracy of the tactile paving area and difficulty in accurately extracting the boundaries and extension direction of the tactile paving.

[0061] This application further proposes image enhancement processing for the image to be processed, including histogram equalization and grayscale transformation. Edge detection algorithms, including the Canny edge detection algorithm, are applied to the enhanced image to extract boundary features. Hough transform is used in the edge image to identify candidate regions for tactile paving with continuous line structures, and noisy edges are filtered based on line segment length and arrangement regularity. Secondary verification is performed by combining color features and tactile paving texture patterns to determine whether it is a valid tactile paving marking region. When determined to be a valid tactile paving marking region, the spatial location, extension direction, and boundary structure information of the tactile paving marking region are extracted.

[0062] Image enhancement processing adjusts the pixel distribution range through histogram equalization, expanding the dynamic range of image grayscale levels and improving the contrast difference between the tactile paving and the background. Grayscale transformation adjusts pixel brightness through linear or nonlinear mapping functions to compensate for the impact of uneven illumination on edge detection. The Canny edge detection algorithm uses double threshold filtering and gradient direction non-maximum suppression to preserve the true boundaries of the tactile paving while suppressing isolated noise points. Hough transform detects straight line segments in the image through a parameter space voting mechanism, selecting continuous line segments with lengths exceeding a preset threshold as candidate regions for the tactile paving. Further filtering of interfering edges that do not conform to the tactile paving laying rules is performed based on the line segment spacing and parallelism. In the secondary verification stage, the color histogram features of the candidate regions are extracted and similarity matching is performed with the pre-stored standard color distribution of the tactile paving. At the same time, local binary mode analysis is used to analyze the consistency of texture direction and eliminate false detection regions with abnormal colors or messy textures.

[0063] Specifically, the image to be processed first undergoes histogram equalization to evenly distribute pixel values ​​across the entire grayscale range, enhancing the contrast between the raised stripes of the tactile paving and the road surface. Grayscale transformation further adjusts the overall brightness of the image, for example, by using gamma correction to reduce the brightness of overexposed areas. The enhanced image is then input into the Canny edge detector, where Gaussian filtering eliminates high-frequency noise, pixel gradient magnitude and direction are calculated, non-maximum suppression is applied to refine edge width, and strong and weak edges are filtered based on high and low thresholds to form a coherent edge contour. The edge image is then input into the Hough transform module, which converts straight lines in the Cartesian coordinate system to polar coordinate parameter space, calculates the equations of the lines corresponding to peak points, and filters parallel line clusters with a length exceeding 30 pixels and an adjacent line segment spacing within the range of 5-15 cm as candidate regions for the tactile paving. In the color verification stage, the RGB color mean of the candidate regions is calculated; if the color difference with the standard yellow or gray of the tactile paving exceeds a threshold, the region is discarded. In the texture verification stage, the candidate regions are divided into multiple sub-blocks, and the local binary mode histogram of each sub-block is calculated; if the directional consistency is lower than a preset ratio, the region is deemed invalid. The final retained tactile paving marking area outputs its centerline coordinates, boundary endpoint coordinates, and extension direction angles for use by the subsequent path planning module.

[0064] As a preferred embodiment, the solution of this application is specifically implemented as follows: Image enhancement processing is performed on the image to be processed, which includes histogram equalization and grayscale transformation. Specifically, histogram equalization is first performed on the image to stretch the pixel value distribution across the entire grayscale range, improving image contrast. Then, grayscale transformation is performed using a gamma correction method, setting the gamma value to 1.2 to enhance image brightness.

[0065] The Canny edge detection algorithm is applied to the enhanced image to extract boundary features. The Canny algorithm includes steps such as Gaussian filtering, calculating gradient magnitude and direction, non-maximum suppression, double threshold detection, and edge connection. The Gaussian filter kernel size is set to 5x5, the low threshold is 50, and the high threshold is 150.

[0066] Hough transform was used to identify candidate regions for tactile paving with continuous line structures in the edge image. The Hough transform parameters were set as follows: ρ resolution 1 pixel, θ resolution π / 180 radians, and accumulator threshold 100. Noisy edges were filtered based on line segment length and regularity of arrangement, retaining line segments with a length greater than 50 pixels and an angle between 80° and 100°.

[0067] A secondary verification process is performed by combining color features and tactile paving texture patterns. The S channel in the HSV color space is extracted, and the average saturation value within the candidate region is calculated to determine if it conforms to the yellow characteristics of tactile paving. Simultaneously, the texture pattern within the candidate region is analyzed to detect the presence of regular stripes or dotted structures. When both color and texture characteristics are satisfied, the area is determined to be a valid tactile paving marking area.

[0068] When a tactile paving area is determined to be valid, its spatial location, extension direction, and boundary structure information are extracted. The spatial location is obtained by calculating the centroid coordinates of the area, the extension direction is calculated using principal component analysis, and the boundary structure information is obtained by fitting the area's contour curve.

[0069] Through the above technical solution, this application can effectively identify tactile paving markings in images, improving the accuracy and robustness of tactile paving detection. The use of multiple feature verification reduces false detections. Simultaneously, by extracting the spatial location and structural information of the tactile paving, reliable data support is provided for subsequent navigation and obstacle avoidance. Furthermore, this method can adapt to the needs of tactile paving identification under different lighting and environmental conditions, improving its practicality in complex scenarios.

[0070] In some of the solutions mentioned above in this application, edge detection and color feature verification are used to identify the tactile paving marking area. However, when judging the spatial relationship between obstacles and the user's walking path, relying solely on two-dimensional image coordinates may lead to projection errors and fail to accurately reflect the actual distance relationship between obstacles and the center line of the tactile paving, thereby affecting the accuracy of path obstruction judgment.

[0071] This application further proposes extracting the extension direction of the tactile paving marking area and extracting the centerline of the tactile paving area. The cosine similarity between the centerline direction vector and the user's walking direction is calculated to obtain the user's walking trend relative to the tactile paving. A spatial relationship model based on 3D reconstructed coordinates is constructed, and the relative spatial distance between the obstacle and the tactile paving centerline is determined based on the location and size of the obstacle's bounding box. When the straight-line distance between the centroid projection of the obstacle's bounding box and the tactile paving centerline is less than a distance threshold, the obstacle is determined to be located within the user's walking path or the tactile paving area. Obstacles that meet the obstacle determination criteria are marked as obstruction objects.

[0072] The extraction of the center line of the tactile paving is achieved by identifying continuous line segments using Hough transform and then fitting the center axis. The spatial relationship model of the 3D reconstructed coordinates obtains the 3D coordinates of obstacles through binocular visual ranging or monocular depth estimation. The distance threshold is dynamically adjusted according to the width of the tactile paving, specifically set to 0.6 times the standard width of the tactile paving. The centroid projection maps the coordinates of the center point of the bounding box to the 3D reconstructed coordinate system and projects them perpendicularly to the tactile paving plane along the direction of gravity.

[0073] Specifically, the extension direction of the tactile paving area is determined by extracting the mean angle of the line segments after edge detection, and the centerline is generated by connecting the perpendicular bisectors of multiple continuous line segments. The cosine similarity between the user's walking direction and the centerline direction is calculated using a normalized vector dot product method; when the similarity is below 0.85, the user is considered to have a deviation trend. In the 3D spatial relationship model, the size parameters of the obstacle bounding box are converted into actual physical dimensions through depth information, and the coordinates of the centroid projection of the bounding box are mapped to the tactile paving plane coordinate system through a coordinate transformation matrix. When the Euclidean distance between the projection point and the centerline is less than a dynamically adjusted distance threshold, an obstruction judgment is triggered. For example, when the standard tactile paving width is 0.3 meters, the distance threshold is set to 0.18 meters to ensure accurate identification when obstacles intrude into the effective area of ​​the tactile paving. The coordinate information of obstacles marked as obstructions is passed to the path planning module as input parameters for impassable areas.

[0074] As a preferred embodiment, the solution of this application is specifically implemented as follows: The extension direction of the tactile paving marking area is extracted, and the centerline of the tactile paving area is also extracted. Specifically, the boundary contour of the tactile paving is obtained through an edge detection algorithm, and then the principal axis direction of the tactile paving is fitted using the least squares method. This principal axis direction is the extension direction of the tactile paving. Furthermore, multiple parallel lines are sampled at equal intervals along the width direction of the tactile paving, and the line connecting the midpoints of these parallel lines is taken as the centerline of the tactile paving area.

[0075] The cosine similarity between the centerline direction vector and the user's walking direction is calculated to obtain the user's walking trend relative to the tactile paving. The user's walking direction is obtained through analysis of accelerometer data, and the centerline direction vector is determined by the two endpoints of the tactile paving centerline. The cosine similarity formula is: cosθ=(a·b) / (|a||b|), where a and b are the centerline direction vector and the user's walking direction vector, respectively.

[0076] A spatial relationship model based on 3D reconstructed coordinates is constructed, and the relative spatial distance between the obstacle and the center line of the tactile paving is determined according to the location and size of the obstacle's bounding box. Specifically, the intrinsic and extrinsic parameter matrices of a monocular camera are used to convert the 2D image coordinates into 3D world coordinates. Then, the vertical distance from the centroid of the obstacle's bounding box to the center line of the tactile paving is calculated.

[0077] When the straight-line distance between the center of gravity projection of an obstacle's bounding box and the center line of the tactile paving is less than a distance threshold, the obstacle is determined to be within the user's walking path or the tactile paving area. For example, if the distance threshold is set to half the width of the tactile paving, and the distance from the obstacle's center of gravity to the center line is less than this threshold, the obstacle is considered to be within the tactile paving area.

[0078] Obstacles that meet the obstacle determination criteria are marked as blocking objects. Specifically, for obstacles located within the tactile paving area, a blocking object list is created, recording the location, size, and category information of each blocking object for subsequent path planning and obstacle avoidance processing.

[0079] Through the above technical solution, this application can accurately determine whether an obstacle is located within the user's walking path or the tactile paving area, providing a reliable basis for subsequent path planning and obstacle avoidance. This allows for the timely detection of potential obstacle risks, improving the walking safety of visually impaired users in complex environments. Furthermore, by constructing a three-dimensional spatial relationship model, this solution can more accurately assess the relative position of obstacles and tactile paving, effectively reducing misjudgments and omissions compared to relying solely on two-dimensional image analysis. In addition, marking obstacles as obstructions facilitates dynamic path planning and adjustment, providing visually impaired users with a more intelligent and safer navigation service.

[0080] In some of the solutions described above in this application, when obstacles are located in blind paths or on the user's travel path, traditional path planning methods struggle to quickly generate optimized routes that avoid the obstacles, resulting in obstructed travel or the need for frequent direction adjustments, which affects traffic efficiency and safety.

[0081] This application further proposes a method for generating optimized routes based on an improved heuristic search path planning approach, including: constructing a two-dimensional grid map containing tactile paving areas and obstructing objects; encoding tactile paving areas as zero-cost traversable areas, defining areas outside tactile paving as first-cost traversable areas, and encoding areas containing obstructing objects as impassable areas; setting a start and end point based on the user's current location and the target direction of the tactile paving; defining open and closed sets and initializing cost function values; the open set stores nodes to be explored, initially containing only the start point, and the closed set stores explored nodes, initially empty; progressively selecting the node with the lowest cost from the open set to expand the path, gradually approaching the end point coordinates; exploring all possible path directions for the current node, updating the path cost; if a neighboring node is not in the open set, adding the neighboring node to the open set and recording its parent node; if a neighboring node is in the open set, but the new path cost is lower, updating the parent node and the path cost; repeating the exploration until the current node coincides with the end point, backtracking from the end point back to the start point, and using this path as the optimized route.

[0082] In constructing the 2D grid map, the tactile paving area is designated as a zero-cost traversable area, the area outside the tactile paving is designated as a first-level cost traversable area, and the area containing obstructing objects is marked as an impassable area. The starting point is determined based on the user's current location, and the ending point is set based on the target direction of the tactile paving. Open and closed sets are used to manage the node states during path exploration, and the cost function comprehensively considers both the travel distance and the area traversal cost. During path expansion, the node with the lowest total cost is prioritized for exploration, and dynamic optimization is achieved by updating the parent node and path cost.

[0083] Specifically, the grid map simplifies spatial modeling complexity by discretizing the environment into a uniform grid. Zero-cost design of tactile paving areas prioritizes path extension along the tactile paving, reducing the risk of users deviating from it. Areas containing obstructing objects are marked as impassable, ensuring automatic obstacle avoidance during path planning. Collaborative management of open and closed sets avoids redundant calculations and improves search efficiency. In the cost function, movement distance is calculated using Manhattan or Euclidean distance, and the area traversal cost is cumulative based on the grid type; for example, moving one grid outside the tactile paving area increases the cost by one level. During path expansion, if an adjacent grid is impassable, that direction is skipped. If it is traversable, the sum of the actual cost from the starting point to the current node and the heuristically estimated cost is calculated, and the node with the minimum total cost is selected as the next expansion target. When the endpoint is added to the open set and the total cost is minimum, the parent node is backtracked to generate an optimized route. For example, with the grid resolution set to 10 cm × 10 cm, the actual cost in the cost function is the travel distance multiplied by the area access cost coefficient, the coefficient for the tactile paving area is 0, and the coefficient for the area outside the tactile paving is 1. The heuristic cost estimation uses Euclidean distance multiplied by a coefficient of 0.8. By dynamically updating the path cost and the relationship with the parent node, it can quickly converge to the globally optimal path, while adapting to changes in obstacle distribution in complex environments.

[0084] As a preferred embodiment, the solution of this application is specifically implemented as follows: A two-dimensional grid map is constructed, containing tactile paving markings and obstructing objects. The grid map encodes tactile paving areas as zero-cost traversable areas, defines areas outside the tactile paving as first-cost traversable areas, and encodes areas containing obstructing objects as impassable areas. For example, the grid map can be divided into 100x100 cells, each cell being 10cmx10cm. Cells containing tactile paving are assigned a value of 0, cells outside the tactile paving are assigned a value of 1, and cells containing obstructing objects are assigned a value of 100.

[0085] The start and end points are set based on the user's current location and the target direction of the tactile paving. Specifically, the user's current location can be set as the start point (0,0), and the location 10 meters away from the user in the direction of the tactile paving extension can be set as the end point (100,0).

[0086] Define an open set and a closed set and initialize the cost function values. The open set stores nodes to be explored, initially containing only the starting point. The closed set stores explored nodes and is initially empty. Further, the cost function can be defined as f(n) = g(n) + h(n), where g(n) is the actual cost from the starting point to the current node n, and h(n) is the estimated cost from the current node n to the destination.

[0087] The path is expanded by progressively selecting the node with the lowest cost from the open set, gradually moving closer to the key location coordinates. Thus, each time, the node with the smallest f(n) is selected from the open set as the current node for expansion.

[0088] Explore all possible path directions for the current node, update the path cost, and if a neighbor node is not in the open set, add the neighbor node to the open set and record its parent node. If a neighbor node is in the open set, but the new path cost is lower, update the parent node and the path cost. For example, explore the neighbor nodes in the four directions of the current node (up, down, left, and right) and calculate the f(n) value for each neighbor node.

[0089] Repeat the exploration until the current node coincides with the destination. Then, backtrack from the destination back to the starting point, and use this path as the optimized route. Specifically, when the current node coincides with the destination, start from the destination and backtrack from the parent node until you reach the starting point, thus obtaining an optimal path from the starting point to the destination.

[0090] Through the above technical solution, this application can quickly generate optimal paths to avoid obstacles in complex environments. This method fully utilizes grid map encoding of environmental information and combines it with a heuristic search algorithm to efficiently explore feasible paths, avoiding the waste of computational resources caused by blind searches. Simultaneously, by dynamically updating path costs, the global optimality of the generated path is ensured. Furthermore, this method also exhibits good real-time performance and adaptability, enabling rapid adjustments to path planning results based on environmental changes, providing timely and accurate navigation guidance for visually impaired users.

[0091] In some of the solutions mentioned above in this application, the user's direction of travel is determined based on acceleration data. However, during dynamic travel, relying solely on acceleration data may lead to unstable attitude angle estimation. Especially when there is high-frequency shaking of the user's head or external interference, the noise in the acceleration data may affect the accuracy of the direction of travel judgment, which in turn leads to errors in subsequent path comparison and feedback prompts.

[0092] This application further proposes a method for real-time acquisition of angular velocity data of the user's head around the X, Y, and Z axes using a three-axis gyroscope sensor. The acquired angular velocity data is fused with simultaneously acquired three-axis accelerometer data, and complementary filtering is used to perform stable estimation of attitude angles, outputting three-axis attitude information including pitch, roll, and yaw angles. The change in the user's orientation is estimated based on the time trend of the yaw angle, and the current orientation is corrected in conjunction with the user's travel direction. The corrected orientation vector is then used as the real-time orientation.

[0093] The three-axis gyroscope sensor collects angular velocity data at a preset sampling frequency, such as 100Hz, to ensure the accuracy of capturing dynamic attitude changes. Complementary filtering combines the high-frequency response characteristics of the gyroscope with the low-frequency stability of the accelerometer by setting weighting coefficients; for example, setting the weight of the gyroscope data to 0.98 and the accelerometer data weight to 0.02 to suppress gyroscope drift errors. The temporal trend of the yaw angle is achieved by calculating the rate of change of angle per unit time; for example, calculating the average angular velocity with a time window of 0.5 seconds, and combining this with the principal direction vector in the acceleration data for direction correction to eliminate deviations caused by instantaneous jitter.

[0094] Specifically, angular velocity data collected by the gyroscope sensor is transmitted to the processor after analog-to-digital conversion and time-aligned with synchronously acquired triaxial acceleration data. A complementary filtering algorithm performs weighted fusion of the two sets of data. The gyroscope data provides short-term, high-precision angular changes, while the acceleration data is calibrated to a long-term attitude reference using gravity direction calibration. The fused attitude angles are then converted into pitch, roll, and yaw angles using quaternion conversion. The change in yaw angle reflects the user's head rotation trend. The cumulative change in yaw angle over a continuous time window is calculated through integration. Combined with the principal direction vector extracted from the acceleration data, the current motion direction is dynamically corrected. For example, when the cumulative change in yaw angle exceeds 5 degrees, a direction vector update is triggered. The corrected motion direction vector is compared with the optimized route direction in real time to ensure the accuracy and timeliness of feedback.

[0095] As a preferred embodiment, the solution of this application is specifically implemented as follows: The system uses a three-axis gyroscope sensor to collect real-time angular velocity data of the user's head around the X, Y, and Z axes. The gyroscope sensor is mounted on the smart glasses frame, and the sampling frequency is set to 200Hz.

[0096] The acquired angular velocity data is fused with simultaneously acquired triaxial accelerometer data. The accelerometer data sampling frequency is also 200Hz. Complementary filtering is used to perform stable estimation of attitude angles, outputting triaxial attitude information including pitch, roll, and yaw angles. The time constant of the complementary filter is set to 0.5 seconds.

[0097] The change in the user's orientation is estimated based on the temporal trend of the yaw angle. The difference in yaw angle between two adjacent sampling periods is calculated, and the cumulative change in orientation is obtained. This is then combined with the user's current direction of travel to correct the current orientation. The direction of travel is obtained through acceleration data analysis.

[0098] The corrected motion direction vector is used as the real-time motion direction. The motion direction vector is represented as a unit vector, where the x and y components represent the direction in the horizontal plane, and the z component is fixed at 0.

[0099] Through the above technical solution, this application achieves accurate estimation of the user's real-time motion direction. By employing data fusion from gyroscopes and accelerometers, the accumulation of errors from a single sensor is reduced. Complementary filtering further improves the stability of attitude angle estimation. By analyzing the yaw angle change trend, the user's steering intention is accurately captured. Combined with corrections to the travel direction, the robustness of motion direction estimation is improved. This provides a reliable data foundation for subsequent path deviation judgment and feedback prompts, contributing to improved navigation accuracy and user experience.

[0100] In some of the solutions described above in this application, the real-time motion direction is obtained through the fusion processing of a gyroscope sensor and an accelerometer. However, in practical applications, factors such as sensor noise, attitude angle estimation errors, and slight head movements of the user may cause deviations between the real-time motion direction and the actual direction of travel. If the uncalibrated direction vector is directly compared with the optimized route, misjudgments may occur due to instantaneous fluctuations in the direction vector, resulting in frequent invalid feedback or missed judgments.

[0101] This application further proposes that when comparing the real-time motion direction with the optimized route, the direction vector of the current position is obtained based on all optimized routes, and then compared with the real-time motion direction. The comparison result is used to determine whether a deviation has occurred. If the angle between the direction vector and the real-time motion direction is greater than a threshold angle, a deviation is determined to have occurred. If the angle between the direction vector and the real-time motion direction is less than or equal to the threshold angle, no deviation is determined to have occurred.

[0102] The direction vector is determined by the direction of the line connecting the current position to the next path point in the optimized route, and the included angle is calculated using cosine similarity in vector space. The included angle threshold is set to a dynamically adjustable range, adaptively adjusting according to the user's walking speed; for example, it is set to 15 degrees for low-speed walking, 10 degrees for medium-speed walking, and 5 degrees for high-speed walking. Deviation from the direction is determined by the projection relationship between the real-time movement direction vector and the left or right side of the optimized route direction vector.

[0103] Specifically, during path tracking, the real-time motion direction vector and the optimized route direction vector are mapped to a two-dimensional coordinate system. The actual angle between the two vectors is calculated by inversely determining the angle. When the angle exceeds a preset threshold, deviation detection logic is triggered. For example, if the optimized route direction vector is due east and the real-time motion direction vector is 12 degrees east of north, and the current speed corresponds to a threshold of 10 degrees, then no deviation is detected. If the real-time motion direction vector is 18 degrees east of north, then deviation detection is triggered. This threshold mechanism effectively filters out instantaneous directional fluctuations caused by sensor noise, triggering feedback only when the deviation continues beyond the tolerance range. The deviation direction is determined by the sign of the vector cross product: a positive cross product indicates a leftward deviation, and a negative cross product indicates a rightward deviation, thus precisely controlling the vibration feedback of the left and right temples.

[0104] As a preferred embodiment, the solution of this application is implemented as follows: After the optimized route is generated, the yaw angle data in the user's head posture information is acquired in real time. The three-dimensional vector of the current motion direction is calculated by fusing gyroscope angular velocity and accelerometer data through complementary filtering. The travel direction vector corresponding to the user's current position in the optimized route is spatially projected with the real-time motion direction vector, and the cosine value between the two vectors is calculated using dot product operation. If the angle corresponding to the cosine value exceeds a preset threshold of 30 degrees, a deviation judgment mechanism is triggered. If the angle is within the threshold range, the normal travel state is maintained. During the deviation judgment process, the dynamic changes between the two vectors are continuously monitored. When the included angle exceeds the threshold for three consecutive sampling periods, the deviation state is confirmed.

[0105] Through the above technical solution, this application effectively solves the problems of lagging and insufficient accuracy in the direction comparison mechanism of the prior art, and realizes real-time dynamic matching between the movement direction and the planned path. By using vector space projection and angle threshold judgment, it can accurately identify the slight deviation between the user's movement direction and the planned route, avoiding misjudgments caused by environmental interference or sensor errors. At the same time, the multi-cycle continuous judgment strategy can filter out instantaneous noise interference, ensure the reliability of deviation judgment, and provide accurate triggering basis for subsequent feedback mechanisms.

[0106] In some of the solutions described above in this application, the real-time motion direction of the user is estimated by fusion processing of data from a three-axis gyroscope sensor and an accelerometer. However, in practical applications, due to slight changes in the user's head posture or environmental interference, there may be a deviation between the real-time motion direction and the direction vector of the optimized route. If the deviation cannot be detected and judged in time, it will affect the accuracy and timeliness of the feedback prompts.

[0107] This application further proposes that when comparing the real-time motion direction with the optimized route, the direction vector of the current position is obtained based on all optimized routes, and then compared with the real-time motion direction. The comparison result is used to determine whether a deviation has occurred. If the angle between the direction vector and the real-time motion direction is greater than a threshold angle, a deviation is determined to have occurred. If the angle between the direction vector and the real-time motion direction is less than or equal to the threshold angle, no deviation is determined to have occurred.

[0108] The direction vector is calculated by optimizing the coordinate difference between the current position and the next path point in the route. The vector angle is calculated using the dot product formula to convert the cosine value into an angle value. The angle threshold is dynamically adjusted according to the user's gait speed; the threshold is lowered when the gait speed increases to improve sensitivity, and increased when the gait speed decreases to reduce false alarms. The deviation judgment result triggers the response logic of the feedback prompt module, including adjustments to the vibration feedback intensity and voice prompt content.

[0109] Specifically, after the optimized route is generated by the path planning algorithm, the direction vector of the path segment corresponding to the current position is extracted. This direction vector is determined by the tangent direction of the path point sequence. The real-time motion direction is estimated by fusing gyroscope angular velocity and accelerometer data and converted into a unit vector form. The cosine value is obtained by calculating the dot product of the two vectors, and then converted into the actual angle by combining it with the inverse cosine function. When the angle exceeds a preset threshold, it is determined that the user's walking direction has deviated from the optimized route, triggering a feedback mechanism. For example, the angle threshold is initially set to 15 degrees. When the user's walking speed exceeds 1.2 meters per second, the threshold is reduced to 10 degrees. When the speed is less than 0.5 meters per second, the threshold is increased to 20 degrees. During the determination process, a sliding window mean filter is used to process the angle data of 5 consecutive frames to eliminate instantaneous jitter interference. When the data of 3 consecutive frames exceeds the threshold, the deviation state is confirmed to ensure the stability of the determination result.

[0110] As a preferred embodiment, the solution of this application is implemented as follows: When the angle between the real-time movement direction and the optimized route direction vector exceeds a preset threshold, the deviation direction is determined to be to the left. The vibration motor of the left temple is activated, and the voice prompt module is triggered to play a turning prompt sound. The vibration frequency of the vibration motor is initially set to two pulses per second, and the voice prompt is "Please adjust your direction to the right." If the deviation angle continues to increase within three consecutive detection cycles, the vibration frequency is increased to four pulses per second, and the voice prompt interval is shortened to once per second. When the user's head posture adjustment causes the angle between the real-time movement direction and the optimized route direction vector to fall below the preset threshold and remain below it for more than two detection cycles, the vibration motor stops working, and the voice prompt ends.

[0111] Through the above technical solution, this application achieves dynamic perception and multimodal feedback adjustment of the user's deviation from the travel direction. The vibration intensity and voice prompt density change in real time with the degree of deviation, effectively improving the timeliness and accuracy of feedback. When the user continues to deviate from the travel direction, the guidance intervention is strengthened through tiered enhanced tactile and auditory cues. When the user returns to the correct path, the feedback is automatically terminated to avoid interference, thereby achieving closed-loop navigation correction in complex environments and reducing the risk of path deviation due to feedback delays or misjudgments.

[0112] In some of the solutions mentioned above in this application, a method for identifying blind paths based on smart glasses was proposed. However, during the actual walking process of the user, due to the lack of a dynamic monitoring and feedback mechanism for real-time posture information, it is impossible to detect in time whether the user's movement direction deviates from the planned path, resulting in limited path guidance effect and problems of feedback lag or deviation.

[0113] In the above embodiments, an accelerometer is used to acquire the user's direction of travel in real time, combined with images captured by a camera mounted on the side of the glasses' temples. A target detection model accurately identifies various obstacles, including pedestrians, vehicles, and roadblocks. Edge detection and texture analysis are used to identify tactile paving markings, and spatial relationship modeling is used to determine whether an obstacle is on the user's path. When a path is blocked, an improved heuristic search algorithm is introduced to reconstruct the optimal obstacle avoidance route. Simultaneously, real-time attitude information from a gyroscope is used to dynamically estimate the user's actual direction of movement and compare it with the planned route, enabling immediate judgment of path deviation. Navigation information is fed back through multimodal methods such as voice and vibration, effectively improving the perception accuracy, path recognition ability, and navigation safety of visually impaired individuals in complex dynamic environments, enhancing the intelligence, human-computer interaction, and practical reliability of tactile paving identification.

[0114] In another preferred embodiment based on the above embodiments, see [reference] Figure 2 As shown, this embodiment provides a tactile paving recognition system based on smart glasses, used to apply the above-described tactile paving recognition method based on smart glasses, including: The acquisition unit is configured to acquire acceleration data, determine the user's direction of travel based on the acceleration data, acquire image data in front of the user in real time through a camera set on the side of the temple of the glasses, input the image data into the target detection model, identify obstacles in the image, and output the location information, size information and category label of the obstacles.

[0115] The judgment unit is configured to process the image data based on edge detection to identify tactile paving signs, and to perform spatial distribution modeling of the obstacles, and to determine whether the obstacles are located in the user's walking path or within the tactile paving area based on the tactile paving signs and the user's walking direction.

[0116] The processing unit is configured to generate an optimized route based on an improved heuristic search path planning method when it is determined that the obstacle is located in the user's walking path or the blind path area.

[0117] The prompting unit is configured to acquire the user's real-time posture information and real-time movement direction based on a gyroscope sensor, compare the real-time movement direction with the optimized route, and provide feedback prompts when it is determined that the user's movement direction deviates from the optimized route. The feedback prompts include voice prompts and vibration feedback.

[0118] Understandably, the system utilizes an accelerometer to acquire the user's direction of travel in real time, combined with images captured by a camera mounted on the side of the glasses' temples. A target detection model accurately identifies various obstacles, including pedestrians, vehicles, and roadblocks. Edge detection and texture analysis are then used to identify tactile paving markings, and spatial relationship modeling is used to determine if an obstacle is on the user's path. When a path is blocked, an improved heuristic search algorithm is introduced to reconstruct the optimal obstacle avoidance route. Simultaneously, real-time attitude information from a gyroscope is used to dynamically estimate the user's actual direction of movement, comparing it with the planned route to instantly detect deviations. Navigation information is fed back through multimodal methods such as voice and vibration, effectively improving the perception accuracy, path recognition ability, and navigation safety of visually impaired individuals in complex and dynamic environments. This enhances the intelligence, human-computer interaction, and practical reliability of tactile paving identification.

[0119] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program goods. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program goods embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program goods according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0121] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0122] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for recognizing a blind path based on smart glasses, characterized in that, include: Acceleration data is collected, and the user's direction of travel is determined based on the acceleration data. Image data in front of the user is collected in real time by a camera set on the side of the temple of the glasses. The image data is input into the target detection model to identify obstacles in the image and output the location information, size information and category label of the obstacles. The image data is processed based on edge detection to identify tactile paving signs, and the spatial distribution of the obstacles is modeled. Based on the tactile paving signs and the user's direction of travel, it is determined whether the obstacles are located in the user's travel path or within the tactile paving area. When it is determined that the obstacle is located in the user's walking path or the blind path area, an optimized route is generated based on an improved heuristic search path planning method; The user's real-time posture information is obtained based on the gyroscope sensor, and the real-time movement direction is obtained. The real-time movement direction is compared with the optimized route. When it is determined that the user's movement direction deviates from the optimized route, feedback prompts are given, including voice prompts and vibration feedback. 2.The smart glasses-based sidewalk recognition method according to claim 1, wherein, When collecting acceleration data and determining the user's direction of travel based on the acceleration data, the process includes: Acceleration data sequences of the user during movement are collected using a triaxial accelerometer; noise filtering is performed on the acceleration data sequences, including the use of sliding window mean filtering; The processed acceleration data sequence is segmented according to time windows, and the principal direction vectors in each time period are extracted. The system identifies continuous walking gait rhythms by detecting gait cycles and obtains the user's walking direction based on the vector angle between the main direction vector and the direction of ground gravitational acceleration. 3.The smart glasses-based sidewalk recognition method of claim 1, wherein, When inputting the image data into a target detection model to identify obstacles in the image and outputting the location information, size information, and category label of the obstacles, the process includes: The image data is preprocessed to obtain the image to be processed. The preprocessing includes image size normalization, brightness enhancement, and color space conversion. The image to be processed is input into the target detection model, which includes YOLOv5 built based on a convolutional neural network; The multi-scale spatial feature map of the image to be processed is extracted by the feature extraction layer of the model, and candidate regions of potential obstacle areas are determined. The candidate region is input into the classification branch and the regression branch. The classification branch is used to determine the target category label. The target categories include pedestrians, vehicles, roadblocks, boxes, and traffic cones. The regression branch is used to predict the bounding box position coordinates and size parameters of the corresponding target. Confidence scores and non-maximum suppression are applied to all detection results, and the final obstacle recognition result set is output. The result set includes the two-dimensional image coordinates, bounding box width and height, category label and confidence score of each obstacle. 4.The smart glasses-based sidewalk recognition method according to claim 3, wherein, When processing the image data based on edge detection to identify tactile paving markings, the process includes: The image to be processed is subjected to image enhancement processing, which includes histogram equalization and grayscale transformation. An edge detection algorithm is applied to the enhanced image to be processed to extract boundary features, including the Canny edge detection algorithm. Hough transform is used in edge images to identify candidate regions of tactile paving with continuous line structures, and noisy edges are filtered out based on line segment length and arrangement regularity. A secondary verification is performed by combining color features and tactile paving texture patterns to determine whether it is a valid tactile paving marking area; when it is determined to be a valid tactile paving marking area, the spatial location, extension direction and boundary structure information of the tactile paving marking area are extracted. 5.The smart glasses-based sidewalk recognition method according to claim 4, wherein, When determining whether an obstacle is located within the user's path or the tactile paving area based on the tactile paving markings and the user's direction of travel, the following methods are included: Extract the extension direction of the tactile paving marking area and extract the center line of the tactile paving area; The cosine similarity between the centerline direction vector and the user's walking direction is calculated to obtain the user's walking trend relative to the tactile paving. Construct a spatial relationship model based on three-dimensional reconstructed coordinates, and determine the relative spatial distance between the obstacle and the center line of the tactile paving according to the bounding box position and size of the obstacle; When the straight-line distance between the center projection of the boundary box of the obstacle and the center line of the tactile paving is less than a distance threshold, the obstacle is determined to be located in the user's walking path or within the tactile paving area. Obstacles that meet the obstacle determination criteria are marked as blocking objects.

6. The method for identifying tactile paving based on smart glasses according to claim 5, characterized in that, When generating optimized routes based on improved heuristic search path planning methods, the following are included: A two-dimensional grid map is constructed that includes the tactile paving marking area and the obstructing object. The grid map encodes the tactile paving area as a zero-cost passage area, defines the area outside the tactile paving as a first-level cost passage area, and encodes the area where the obstructing object is located as an impassable area. Set the start and end points based on the user's current location and the target direction of the tactile paving; Define an open set and a closed set and initialize the cost function value. The open set stores the nodes to be explored, and initially only contains the starting point. The closed set stores the nodes that have been explored, and is initially empty. From the open set, the path of the node with the lowest cost is gradually selected and extended towards the key location coordinates; Explore all possible path directions for the current node, update the path cost, and if the neighbor node is not in the open set, add the neighbor node to the open set and record the parent node of the neighbor node; if the neighbor node is in the open set, but the new path cost is smaller, update the parent node and the path cost. Repeat the exploration until the current node coincides with the endpoint, then backtrack from the endpoint to the parent node until the starting point, and use this path as the optimized route.

7. The method for identifying tactile paving based on smart glasses according to claim 2, characterized in that, When acquiring real-time posture information of the user based on a gyroscope sensor, and obtaining the real-time direction of motion, the following is included: The angular velocity data of the user's head around the X, Y and Z axes are collected in real time based on a three-axis gyroscope sensor. The collected angular velocity data is fused with the simultaneously collected triaxial accelerometer data, and complementary filtering is used to perform stable estimation of attitude angles, outputting triaxial attitude information including pitch angle, roll angle and yaw angle. Estimate the change in the user's orientation based on the time-varying trend of the yaw angle, and correct the current orientation based on the user's direction of travel. The corrected motion direction vector is used as the real-time motion direction.

8. The method for identifying tactile paving based on smart glasses according to claim 7, characterized in that, When comparing the real-time motion direction with the optimized route, the following steps are included: The direction vector of the current position is obtained based on all optimized routes. The direction vector is compared with the real-time movement direction, and the deviation is determined based on the comparison result. When the angle between the direction vector and the vector of the real-time motion direction is greater than the angle threshold, a deviation is determined to have occurred; When the angle between the direction vector and the vector of the real-time motion direction is less than or equal to the angle threshold, it is determined that no deviation has occurred.

9. The method for identifying tactile paving based on smart glasses according to claim 1, characterized in that, When it is determined that the user's direction of travel deviates from the optimized route, feedback is provided, including: If the user's deviation direction is to the left, control the left temple vibration motor to start the vibration mode and activate the voice prompt; If the user's deviation direction is to the right, control the vibration motor of the right temple to start the vibration mode and activate the voice prompt; The feedback frequency is adjusted according to the deviation angle. When the deviation angle continues to increase or the duration exceeds the time threshold, the vibration frequency and voice prompt interval density are increased. When the user's direction of travel is realigned with the optimized route and the deviation angle is less than the included angle threshold, the feedback prompt is terminated.

10. A tactile paving recognition system based on smart glasses, used to apply the tactile paving recognition method based on smart glasses as described in any one of claims 1-9, characterized in that, The acquisition unit is configured to acquire acceleration data, determine the user's direction of travel based on the acceleration data, acquire image data in front of the user in real time through a camera set on the side of the temple of the glasses, input the image data into the target detection model, identify obstacles in the image, and output the location information, size information and category label of the obstacles; The judgment unit is configured to process the image data based on edge detection to identify tactile paving signs, and to perform spatial distribution modeling of the obstacles, and to determine whether the obstacles are located in the user's walking path or the tactile paving area based on the tactile paving signs and the user's walking direction. The processing unit is configured to generate an optimized route based on an improved heuristic search path planning method when it is determined that the obstacle is located in the user's walking path or the blind path area. The prompting unit is configured to acquire the user's real-time posture information and real-time movement direction based on a gyroscope sensor, compare the real-time movement direction with the optimized route, and provide feedback prompts when it is determined that the user's movement direction deviates from the optimized route. The feedback prompts include voice prompts and vibration feedback.

Images