An environment perception navigation method and system based on blind lane segmentation detection and virtual reconstruction

Abstract

This application provides an environmental perception navigation method and system based on tactile paving segmentation detection and virtual reconstruction, solving the technical problems of existing guide systems such as single-dimensional environmental perception, weak adaptability to tactile paving obstruction and breakage scenarios, and lack of three-dimensional accessibility assessment. The method includes the following steps: achieving multi-dimensional environmental perception through binocular cameras, accelerometers, gyroscopes, and GPS; extracting features through a shared shallow convolutional network after image preprocessing; detecting obstacles using an improved YOLOv11 model; segmenting tactile paving and handling obstructed areas using a GRFB-UNet model; virtually reconstructing broken sections of tactile paving and assessing accessibility using three-dimensional point clouds; and finally fusing multi-source data to achieve voice navigation and path replanning in emergency situations, with iterative monitoring and updates. This invention can be widely applied to ensure safe travel for visually impaired individuals.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically relates to an environmental perception navigation method and system based on blind path segmentation detection and virtual reconstruction. Background Technology

[0002] For visually impaired people, the biggest challenges are accessing information from the outside world and ensuring their safety while walking. Existing guide systems are inadequate for route planning and handling emergencies such as broken tactile paving.

[0003] In recent years, guide systems for the visually impaired have gradually incorporated computer vision and deep learning technologies to achieve perception and analysis of the surrounding road environment. However, existing guide systems still have many limitations in algorithm design. First, existing guide systems cover a limited range of obstacle types, making it difficult to cope with complex and ever-changing road environments. They lack comprehensive recognition capabilities for various obstacles commonly found on urban roads, such as pedestrians, vehicles, bicycles, electric vehicles, roadblocks, and construction signs, resulting in significant safety hazards for visually impaired individuals during actual travel.

[0004] Secondly, it is difficult to balance detection accuracy and computational complexity. High-precision deep learning models typically involve large computational loads and numerous parameters, making them difficult to run in real time on mobile or embedded devices; while lightweight models suffer from low accuracy due to insufficient feature representation capabilities, failing to meet the needs of practical applications. This contradiction makes it difficult for existing guide systems to achieve real-time response while ensuring detection accuracy, seriously affecting the practicality and reliability of guide systems.

[0005] Furthermore, existing guidance systems for the visually impaired lack path planning capabilities deeply integrated with detection results, failing to provide users with reasonable routes. Most systems can only inform users of the presence of obstacles ahead, but cannot combine tactile paving information, obstacle locations, and the user's current location for intelligent path planning, limiting the practical value of guidance systems in real-world travel scenarios. Simultaneously, existing systems lack multi-sensor fusion capabilities, failing to comprehensively utilize multiple information sources such as vision, inertia, and positioning, resulting in insufficient robustness in complex environments.

[0006] The existing Chinese invention patent with publication number CN121121690A provides a method for identifying tactile paving based on image processing. It identifies broken areas in the tactile paving using directional gradient features, performs cross-frame target tracking, and outputs a continuous tactile paving trajectory. However, this patent relies solely on image processing technology and lacks multi-sensor fusion and deep learning models, making it difficult to achieve accurate real-time navigation and three-dimensional spatial perception.

[0007] The existing Chinese invention patent, CN121330607A, provides a method and system for assisting visually impaired travel based on improved YOLOv11 and BiseNetV2 collaborative inference. It involves inputting shared feature maps into a trained obstacle detection model and a tactile paving segmentation model for obstacle detection and tactile paving segmentation, followed by path planning. However, this patent relies solely on image processing and dual-model collaboration, failing to integrate multiple sensors for multi-dimensional environmental perception and global positioning; thus, it struggles to meet the precise navigation needs in complex scenarios. Summary of the Invention

[0008] Based on this, the present invention needs to provide an environmental perception navigation method and system based on tactile paving segmentation detection and virtual reconstruction to solve the technical problems of low detection accuracy, poor real-time performance, poor multi-task collaborative perception capability, and inability to achieve lightweight design in the existing guide systems for the blind.

[0009] To achieve the above objectives, an environmental perception navigation method and system based on tactile paving segmentation detection and virtual reconstruction includes the following steps: Step S1: Real-time acquisition of RGB images and depth maps of the environment using binocular cameras to construct a 3D point cloud; combining accelerometers and gyroscopes to obtain device attitude, and utilizing GPS to obtain global coordinates, achieving multi-dimensional environmental perception. Step S2: Perform image preprocessing including normalization and resolution adjustment, and extract edge texture and shape features using a shared shallow convolutional network. Step S3: Detect obstacles and predict bounding boxes based on the improved YOLOv11 model, segment the tactile paving area using the GRFB-UNet model, and perform masking and hole filling on the occluded areas of obstacles. Step S4: Denoise and virtually reconstruct the broken sections of the tactile paving. Combine the 3D point cloud information with curve fitting to generate a continuous binary mask for the tactile paving, and project it into 3D space to determine the accessibility. Step S5: Integrate and analyze various real-time data, and provide navigation information via voice broadcast; in case of emergencies, trigger GPS route replanning, and continuously monitor and update. Attached Figure Description

[0010] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating the steps of an environmental perception navigation method and system based on blind path segmentation detection and virtual reconstruction, provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the YOLOv11 model architecture provided in an embodiment of the present invention; Figure 3This is a schematic diagram of the structure of the GRFB-UNet blind path segmentation model provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of obstacle detection results provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the virtual reconstruction result of a blind path provided in an embodiment of the present invention; The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0011] The technical method of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention.

[0012] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor methods and / or microcontroller methods.

[0013] It should be understood that although the terms "first," "second," etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are used merely to distinguish one unit from another. For example, without departing from the scope of the exemplary embodiments, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0014] To achieve the above objectives, please refer to Figure 1 This invention provides an environmental perception navigation method based on tactile paving segmentation detection and virtual reconstruction, the method comprising the following steps: Step S1: Real-time acquisition of RGB images and depth maps of the environment using binocular cameras to construct a 3D point cloud; combined with accelerometers and gyroscopes to obtain device attitude, and GPS to obtain global coordinates to achieve multi-dimensional environmental perception.

[0015] In one embodiment, an Intel RealSense D455 binocular camera is selected as the input acquisition device. This binocular camera supports RGB color and infrared imaging, and features 4MP resolution, high frame rate output, and binocular synchronous capture, providing clear and stable image input under complex lighting conditions. The image resolution acquired by the binocular camera is 1920×1080 pixels, the frame rate is 30Hz, the baseline distance is 120mm, and the focal length is 4.5mm. A disparity map is calculated using a binocular stereo matching algorithm, and then implemented according to the disparity formula, as follows:

[0016] In the formula, Z is the actual distance corresponding to the pixel; The equivalent focal length normalized to the same image width; B is the baseline length of the left and right infrared cameras; d is the parallax; s is the depth scaling factor, which defaults to 0.001, meaning that each pixel value in the depth frame represents 0.001m. To prevent tiny correction values ​​where the denominator is zero.

[0017] In another embodiment, assuming the acquired RGB image size is 1280×720 pixels and the depth map size is also 1280×720 pixels, the depth value corresponding to each pixel ranges from 0.5m to 10m. The disparity map calculated by the binocular stereo matching algorithm has a disparity value range of 0 to 255 pixels. Assuming the disparity value of a certain pixel is 50 pixels, the focal length f is 4.5mm, and the baseline distance B is 120mm, then the depth distance corresponding to this pixel is Z=4.5×120 / (50+0.01)=10.8mm, which, after unit conversion, yields an actual distance of 10.8m. By traversing all pixels in the image, a point cloud data containing approximately 921,600 three-dimensional points can be constructed for subsequent three-dimensional spatial analysis and accessibility assessment of the tactile paving area.

[0018] In one embodiment, an inertial measurement unit (IMU) is used to acquire the device's attitude data. The IMU includes a three-axis accelerometer and a three-axis gyroscope. The three-axis accelerometer measures the linear acceleration of the device in the X, Y, and Z directions, with a measurement range of ±16g and a resolution of 16 bits. The three-axis gyroscope measures the angular velocity of the device around the X, Y, and Z axes, with a measurement range of ±2000° / s and a resolution of 16 bits. By fusing the accelerometer and gyroscope data using a Kalman filter algorithm, the device's attitude angles (pitch, roll, and yaw) can be obtained, with an attitude angle update frequency of 100Hz. Assume that at a certain moment, the accelerometer measures the X, Y, and Z axis accelerations as follows: =0.1m / s² =0.05m / s² =9.8m / s², the angular velocities of the X, Y, and Z axes measured by the gyroscope are respectively =0.02rad / s =0.01rad / s =0.03rad / s, the pitch angle of the device can be obtained through the attitude calculation algorithm. Roll angle Yaw angle The yaw angle is obtained by integration using a gyroscope. Assuming the initial yaw angle is 0°, the change in yaw angle after 1 second is... .

[0019] In another embodiment, it is assumed that the device's attitude angles change during walking, with the pitch angle varying from -15° to +15°, the roll angle from -10° to +10°, and the yaw angle from -180° to +180°. Real-time updates of the IMU data allow the device's motion state and attitude changes to be perceived, providing a reference for subsequent image stabilization processing and path planning. Assuming that during a 10-second walking period, the IMU collects 1000 sets of attitude data, with an average pitch angle of 2.3° and a standard deviation of 3.5°; an average roll angle of -1.2° and a standard deviation of 2.8°; and an average yaw angle of 15.6° and a standard deviation of 8.2°. This attitude data can be used to correct the viewing angle of the binocular camera, improving the accuracy of tactile paving detection.

[0020] In one embodiment, a GPS module is used to obtain the device's global latitude and longitude coordinates. The GPS module has a positioning accuracy of 2.5m (CEP) and an update frequency of 10Hz. The data output by the GPS module includes longitude, latitude, altitude, speed, and heading. Assuming that at a certain moment the GPS outputs a longitude of 120.153576°E, a latitude of 30.287459°N, an altitude of 15.2m, a speed of 1.2m / s, and a heading of 45°, the device's current location can be mapped onto an electronic map using GPS positioning data. Combined with pre-stored tactile paving distribution data, global path planning can be achieved. Simultaneously, GPS data can also be used to trigger path replanning when an emergency is detected, ensuring navigation continuity and safety.

[0021] In another embodiment, assuming the device is walking in an outdoor environment, the GPS signal positioning accuracy is 2.5m in open areas and 5-10m in urban canyon areas. By fusing GPS data, IMU data, and visual odometry data, higher positioning accuracy can be achieved. Assuming that over a walking distance of 100 meters, the cumulative GPS positioning error is 3.2m, the cumulative IMU positioning error is 1.8m, and the cumulative visual odometry positioning error is 0.5m, through multi-sensor fusion positioning, the final positioning error can be controlled within 1m. This multi-dimensional environmental perception capability provides comprehensive environmental information support for subsequent tactile paving detection, obstacle recognition, and path planning.

[0022] Step S2: Perform normalization and resolution adjustment preprocessing on the image, and extract edge texture and shape features through a shared shallow convolutional network.

[0023] In another embodiment, assume the original image size is 1280×720 pixels, with pixel values ​​ranging from [0, 255]. After normalization, the pixel values ​​range from [0, 1]. Assume the original RGB value of a pixel is (128, 64, 192), and the normalized value is (0.502, 0.251, 0.753). After resolution adjustment, the image size becomes 512×512 pixels. The four pixels (0,0), (0,1), (1,0), and (1,1) in the original image are mapped to pixel (0,0) in the new image after bilinear interpolation. Assuming the normalized grayscale values ​​of these four pixels are 0.5, 0.6, 0.7, and 0.8 respectively, the interpolated pixel value is (0.5 + 0.6 + 0.7 + 0.8) / 4 = 0.65. This preprocessing method unifies input images of different resolutions to the input size required by the model, ensuring consistency in subsequent feature extraction.

[0024] In one embodiment, the preprocessed image undergoes basic feature extraction via a shared shallow convolutional network. The shallow convolutional network comprises three convolutional layers, each followed by a batch normalization layer and a ReLU activation function. The first convolutional layer has a kernel size of 7×7, a stride of 2, and 64 output channels; the second convolutional layer has a kernel size of 3×3, a stride of 2, and 128 output channels; the third convolutional layer has a kernel size of 3×3, a stride of 1, and 256 output channels. Assuming the input image size is 512×512×3, after the first convolutional layer, the feature map size becomes 256×256×64; after the second convolutional layer, the feature map size becomes 128×128×128; and after the third convolutional layer, the feature map size becomes 128×128×256. The features extracted by shallow convolutional networks include low-level visual features such as edge texture and shape contour. These features are universal for obstacle detection and tactile paving segmentation tasks. By sharing shallow features, the amount of computation can be effectively reduced and the system efficiency can be improved.

[0025] In another embodiment, assume the input to the shallow convolutional network is a 512×512×3 RGB image, which outputs a 128×128×256 feature map after three convolutional layers. The feature vector at each location in the feature map has a dimension of 256, containing edge texture and shape information for that location. For example, at a location (64, 64) in the feature map, the feature vector is a 256-dimensional vector, where the first 64 dimensions represent horizontal edge features, the middle 64 dimensions represent vertical edge features, and the last 128 dimensions represent texture features. By analyzing the distribution of the feature vector, it can be determined whether the location belongs to a tactile paving area or an obstacle area. For example, if the horizontal edge feature value is high, the location may belong to a striped texture area of ​​a tactile paving; if the texture feature value is high, the location may belong to a dotted texture area of ​​a tactile paving.

[0026] Step S3: Detect obstacles and predict bounding boxes based on the improved YOLOv11 model, segment the blind path area using the GRFB-UNet model, and perform masking and hole filling on the obstacle-occluded areas.

[0027] In one embodiment, a shared feature map is input into an improved YOLOv11 model for obstacle detection. The backbone network of the improved YOLOv11 model replaces the original C3k2 module with a C3k2_PConv module. The C3k2_PConv module incorporates partial convolution techniques, performing convolution operations only on a subset of channels of the input feature map. Assuming the input feature map has 256 channels and the partial convolution ratio is 0.5, only 128 channels are convolved, while the other 128 channels remain unchanged. This design significantly reduces computational cost while maintaining feature representation capabilities. Assuming the computational cost of the original C3k2 module is M, the computational cost of the C3k2_PConv module is approximately 0.5M, representing an improvement in computational efficiency of approximately 50%.

[0028] In another embodiment, assuming the input to the improved YOLOv11 model is a shared feature map of 128×128×256, after multi-layer convolutional processing by the backbone network, the output is a multi-scale feature map. The backbone network contains five downsampling stages, and the scales of the output feature maps are 64×64, 32×32, 16×16, 8×8, and 4×4, with corresponding channel numbers of 64, 128, 256, 512, and 1024, respectively. The neck network adopts a feature pyramid structure to fuse feature maps of different scales. Assuming that on a 16×16 scale feature map, the bounding box coordinates of a detected obstacle are (8.5, 10.2, 12.8, 14.6), with a confidence score of 0.92, and the category is "pedestrian," non-maximum suppression (NMS) is used to remove overlapping detection boxes and retain the optimal detection result. Assuming the NMS threshold is 0.5, detection boxes with an overlap greater than 0.5 will be suppressed.

[0029] In one embodiment, the improved YOLOv11 model's neck network employs a Slim-Neck structure, including the GSConv module, the GS-Bottleneck module, and the VoV-GSCSP unit. The GSConv module first extracts features from a subset of channels using standard convolutional layers, assuming the input feature map has 256 channels and the standard convolutional output has 128 channels. Then, the feature map is divided into two parts: one part remains unchanged, while the other undergoes depthwise separable convolution processing with a 3×3 kernel size. Finally, the two feature maps are concatenated, and channel shuffling enhances the flow of information between features. The VoV-GSCSP unit extracts features through two paths: the first path directly extracts features through convolutional layers, and the second path uses the GS-Bottleneck module for depth feature extraction. Finally, the features from the two paths are concatenated and fused. This design significantly reduces computational cost and parameter count while maintaining detection accuracy.

[0030] In another embodiment, assuming the input to the Slim-Neck structure is a multi-scale feature map output from the backbone network, after processing by the GSConv module, the number of channels in the feature map is reduced from 512 to 256, resulting in a computational reduction of approximately 40%. After processing by the VoV-GSCSP unit, the number of channels in the feature map remains at 256, but the feature representation capability is enhanced. Assuming that at a certain feature map location, the original feature vector is a 256-dimensional vector, after Slim-Neck processing, the feature vector remains 256-dimensional, but the feature distribution is more compact, and the inter-class distance increases, which is beneficial for subsequent classification and localization tasks. Experiments verify that the improved YOLOv11 model achieves an average accuracy (mAP) of 85.3% on obstacle detection tasks, a 2.1% improvement compared to the original YOLOv11 model, while reducing computational cost by 35%.

[0031] In one embodiment, the improved YOLOv11 model employs a dynamic detection head, comprising three sub-modules: scale-aware attention, spatial awareness attention, and task-aware attention. The scale-aware attention module extracts features from different scales and dynamically fuses these features based on semantic importance. Assuming the input feature map has three scales—64×64, 32×32, and 16×16—the scale-aware attention module assigns weights to each scale, calculated using the Softmax function. Assuming the weights for the three scales are 0.3, 0.5, and 0.2, the 32×32 scale features will have the largest proportion in the fused feature map. The spatial awareness attention module extracts spatial features through variable convolution and aggregates features from the same spatial location at different levels. The task-aware attention module can dynamically enable and disable feature channels to better adapt to obstacle detection tasks in guide vision scenarios.

[0032] In another embodiment, assuming the multi-scale feature maps input to the dynamic detection head are 64×64×256, 32×32×512, and 16×16×1024 respectively, after processing by the scale-aware attention module, the feature maps of the three scales are fused into a unified feature representation. Assuming that at a certain location, the dimensions of the three scale feature vectors are 256, 512, and 1024 respectively, after scale-aware attention weighting, the dimension of the fused feature vector is 1024. The spatial awareness attention module resamples the fused features based on their spatial location, assuming the original location is... The offset of the variable convolution is ( If ), then the resampled position is ( The task-aware attention module dynamically adjusts the feature channels according to task requirements. Assuming the original number of feature channels is 1024, the task-aware attention module selects to activate 512 channels and suppress the other 512 channels, thereby improving detection efficiency.

[0033] In one embodiment, a shared feature map is input into the GRFB-UNet model for tactile paving segmentation. The GRFB-UNet network embeds GRFB modules in each stage of the encoder and decoder of the standard UNet network. The GRFB module obtains a multi-scale receptive field through a multi-branch convolutional structure, containing four parallel branches: the first branch uses a 1×1 convolution for channel adjustment and feature compression; the second branch uses a 3×3 convolution to capture local detail features; the third branch uses a 3×3 dilated convolution with a dilation rate of 2 to expand the receptive field; and the fourth branch uses a 3×3 dilated convolution with a dilation rate of 4 to further expand the receptive field. Assuming the input feature map has 256 channels, and the output channels of each branch are 64, 64, 64, and 64 respectively, the output feature maps of the four branches are fused through a concatenation operation, and then subjected to a 1×1 convolution for channel adjustment, outputting the fused multi-scale features with 256 output channels.

[0034] In another embodiment, assuming the input feature map size of the GRFB module is 64×64×256, after processing through four branches, the output feature map size of each branch is also 64×64×64. The first branch's 1×1 convolution has 256×64=16,384 parameters; the second branch's 3×3 convolution has 3×3×256×64=147,456 parameters; the third branch's dilated convolution also has 147,456 parameters, but its receptive field is expanded to 7×7; the fourth branch's dilated convolution also has 147,456 parameters, but its receptive field is expanded to 15×15. The receptive fields of the four branches are 1×1, 3×3, 7×7, and 15×15, respectively, which can simultaneously capture local details and global contextual information. This design is particularly suitable for processing narrow and long objects such as tactile paving, as it can simultaneously capture the local texture features and overall orientation features of the tactile paving, improving segmentation accuracy.

[0035] In one embodiment, the encoder of the GRFB-UNet network extracts high-level semantic features through multi-stage downsampling. The encoder contains five downsampling stages, each containing two GRFB modules and a max-pooling layer. Assuming the input image size is 512×512×3, after the first stage, the feature map size becomes 256×256×64; after the second stage, it becomes 128×128×128; after the third stage, it becomes 64×64×256; after the fourth stage, it becomes 32×32×512; and after the fifth stage, it becomes 16×16×1024. The decoder recovers the spatial resolution of the feature map through upsampling operations and fuses the detailed features from each stage of the encoder using skip connections. The decoder also contains five upsampling stages, each containing an upsampling layer and two GRFB modules. The final output is a segmentation mask of the same size as the input image, where the pixel value of the tactile paving area is 1, and the pixel value of the background area is 0.

[0036] In another embodiment, assuming the segmentation mask output by the GRFB-UNet network is 512×512 pixels, the number of pixels in the blind path region of the mask is 25,600, accounting for 9.77% of the total image area. The pixel accuracy of the segmentation mask is 94.2%, and the intersection-over-union (IoU) is 87.5%. Assuming that there are some small holes and broken areas in the segmentation mask, post-processing is performed using morphological closing operations. The morphological closing operation includes two steps: dilation and erosion. The dilation operation uses a 3×3 structuring element to expand the boundary of the blind path region outward by 1 pixel; the erosion operation also uses a 3×3 structuring element to shrink the boundary of the blind path region inward by 1 pixel. Through morphological closing operations, small holes in the blind path region can be filled, adjacent blind path regions can be connected, and the continuity and integrity of the segmentation mask can be improved. Assuming that after the morphological closing operation, the number of pixels in the blind path region increases to 26,200, and the number of holes decreases from 15 to 3.

[0037] In one embodiment, the obstructed area is masked and filled with holes. When an obstacle is detected within the tactile paving area, the detection box region corresponding to the obstacle is marked as an obstructed area in the segmentation mask. Assume the detection box coordinates of a certain obstacle are... Pixels within the detection bounding box are marked as occluded in the segmentation mask. For occluded areas, a hole-filling algorithm based on neighborhood information is used for repair. The hole-filling algorithm first identifies hole regions in the segmentation mask, and then interpolates and fills them based on the effective tactile paving pixels surrounding the hole. Assuming the hole region is 10×10 pixels in size and the average value of the effective tactile paving pixels around the hole is 0.85, then all pixel values ​​within the hole region are set to 0.85, indicating that this region may be a tactile paving area. Through the hole-filling algorithm, tactile paving areas occluded by obstacles can be effectively recovered, improving the integrity of tactile paving segmentation.

[0038] In another embodiment, assume there is a 20×15 pixel hole region in the segmentation mask. The effective tactile paving pixel values ​​around the hole are as follows: average value of 0.82 on the left, average value of 0.88 on the right, average value of 0.79 on the top, and average value of 0.91 on the bottom. The filling value of the hole region is calculated by weighted averaging, with the weights determined by distance; the closer the distance, the greater the weight. Assuming the center pixel of the hole is 10 pixels from the left boundary, 10 pixels from the right boundary, 7.5 pixels from the top boundary, and 7.5 pixels from the bottom boundary, the filling value is (0.82×0.25+0.88×0.25+0.79×0.25+0.91×0.25)=0.85. Hole filling can be completed by traversing all pixels within the hole region. Assuming that after hole filling, the number of pixels in the tactile paving region of the segmentation mask increases from 26,200 to 26,500, the continuity of the segmentation mask is significantly improved.

[0039] Step S4: Denoise and virtually reconstruct the broken sections of the tactile paving. Combine the 3D point cloud information with curve fitting to generate a continuous binary mask for the tactile paving, and project it into 3D space to determine the accessibility.

[0040] In one embodiment, break detection is performed on the segmentation mask. First, skeleton extraction is performed on the segmentation mask to obtain the center line of the tactile paving. The skeleton extraction uses a morphological thinning algorithm, shrinking the tactile paving region to a single-pixel-width skeleton line through iterative erosion operations. Assuming the width of the tactile paving region in the segmentation mask is 10-20 pixels, after skeleton extraction, a skeleton line with a width of 1 pixel is obtained. Then, break detection is performed on the skeleton line; when discontinuous endpoints exist on the skeleton line, it is determined to be a break region. Assuming two break endpoints are detected on the skeleton line, with endpoint coordinates as follows... and The distance between the two endpoints is 50 pixels, which is determined to be a broken area.

[0041] In another embodiment, assuming the segmentation mask size is 512×512 pixels, the total length of the skeleton lines after skeleton extraction is 1,250 pixels. Five breaks are detected on the skeleton lines, with break lengths of 30 pixels, 45 pixels, 25 pixels, 60 pixels, and 35 pixels, respectively. For each break, the coordinates of the endpoints and the break length are recorded. For example, the endpoint coordinates of the first break are P1(80,150) and P2(110,150), with a break length of 30 pixels; the endpoint coordinates of the second break are P3(200,280) and P4(245,280), with a break length of 45 pixels. This break information will be used for subsequent virtual reconstruction processing. The break detection threshold is set to 20 pixels; when the break length is less than 20 pixels, it is considered a false break caused by noise and no reconstruction processing is performed.

[0042] In one embodiment, the detected fracture region is virtually reconstructed. The virtual reconstruction employs linear interpolation or spline curve fitting methods. For shorter fractures (fracture length less than 50 pixels), linear interpolation is used to directly connect the endpoints of the fracture. Assume the coordinates of the fracture endpoints are... and The pixel coordinates on the connection line generated by linear interpolation are , where t [0,1]. For longer fractures (fracture length greater than or equal to 50 pixels), a cubic spline curve fitting method is used to determine the tangent direction of the curve based on the direction vectors at both ends of the fracture, generating a smooth connection curve. Assume the direction vector at fracture endpoint P1 is... fracture endpoint The direction vector at is The control points of a cubic spline curve are determined based on the endpoint coordinates and direction vector.

[0043] In another embodiment, suppose a fracture is detected with a fracture length of 60 pixels and the coordinates of the fracture endpoint are... (200, 280) and (260, 280), endpoint Angle of direction at =5°, endpoint Angle of direction at =-3°. A cubic spline curve fitting method was used for virtual reconstruction, and the generated curve equation is: , ,in 61 points were sampled on the curve. These points are connected to form the center line of a virtual tactile paving path. Assuming the width of the virtual tactile paving path is 15 pixels, it is extended 7.5 pixels to both sides of the virtual center line to generate virtual tactile paving areas. The pixel value of each virtual tactile paving area is set to 1, indicating that the area is a tactile paving area. Through virtual reconstruction, broken areas in the tactile paving path can be effectively filled, improving the continuity of tactile paving navigation.

[0044] In one embodiment, the accessibility of the tactile paving is determined by combining 3D point cloud information. A segmentation mask is projected into 3D space to obtain the 3D point cloud of the tactile paving area. Assume the coordinates of a certain pixel in the segmentation mask are... If the corresponding depth value is Z, then the three-dimensional coordinates of the pixel are: ,in , , and Let f be the coordinates of the camera's optical center and f be the focal length. Assume the camera's optical center coordinates are (320, 240), the focal length is 500 pixels, and the coordinates of a certain pixel are... If the depth is 2 meters, then the three-dimensional coordinates of this pixel are: rice, Meters, Z=2 meters. By traversing all the tactile paving pixels in the segmentation mask, the 3D point cloud data of the tactile paving area can be obtained.

[0045] In another embodiment, assume that the 3D point cloud of the tactile paving area contains 5,000 points, with the X-coordinate ranging from -1.5 meters to 1.5 meters, the Y-coordinate ranging from 0 meters to 3 meters, and the Z-coordinate ranging from 0.5 meters to 5 meters. By analyzing the height distribution of the point cloud, the smoothness and accessibility of the tactile paving can be determined. Assuming the standard deviation of the Z-coordinate in the point cloud is 0.05 meters, it indicates that the surface of the tactile paving is relatively smooth; if the standard deviation of the Z-coordinate is greater than 0.1 meters, it indicates that the surface of the tactile paving has significant undulations, which may affect passage. Simultaneously, by analyzing the density distribution of the point cloud, the width of the tactile paving can be determined. Assuming the distribution range of the point cloud in the X direction is -0.6 meters to 0.6 meters, it indicates that the width of the tactile paving is approximately 1.2 meters, meeting the passage requirements. If the width of the tactile paving is less than 0.6 meters, it is determined to be a narrow area, requiring users to be alerted.

[0046] In one embodiment, the accessibility of the virtually reconstructed tactile paving area is assessed. The accessibility assessment comprehensively considers factors such as tactile paving width, smoothness, and obstacle distance. Assuming the tactile paving width is W, the smoothness is P (the reciprocal of the standard deviation), and the closest obstacle distance is D, the accessibility score is... ,in , , The weighting coefficients are 0.4, 0.3, and 0.3 respectively. Assuming a tactile paving area is 1.2 meters wide, has a flatness score of 20 (standard deviation of 0.05 meters), and the closest distance to an obstacle is 3 meters, then the accessibility score is... The passability score ranges from 0 to 10, with a higher score indicating better passability. When the score is below 5, the system will issue a warning to remind the user to pay attention to passability safety.

[0047] In another embodiment, assuming a accessibility assessment is performed on 10 sections of tactile paving, the assessment results are: 7.38, 8.52, 6.15, 4.82, 9.21, 5.67, 7.89, 6.43, 8.05, and 5.12. Sections 4 and 10 have accessibility scores below 5, and the system will issue warnings for these two sections. Section 4 scored 4.82; analysis revealed that this section is only 0.5 meters wide and contains a step with a height difference of 0.15 meters. Section 10 scored 5.12; analysis revealed that there is an obstacle nearby, with the closest distance being only 1.5 meters. Through accessibility assessment, more accurate navigation suggestions can be provided to users, improving travel safety.

[0048] Step S5: Integrate and analyze various real-time data, and provide navigation information via voice broadcast; in case of emergencies, trigger GPS route replanning, and continuously perform monitoring and updates.

[0049] In one embodiment, the system analyzes and judges various types of real-time data using information fusion technology. Information fusion employs a multi-level fusion strategy, including data-level fusion, feature-level fusion, and decision-level fusion. Data-level fusion performs spatiotemporal alignment and registration of raw data from different sensors. Assuming the sampling frequency of the binocular camera is 30Hz, the IMU's sampling frequency is 100Hz, and the GPS's sampling frequency is 10Hz, timestamp alignment unifies data from different frequencies to the same time base. Feature-level fusion concatenates or weights features from different modalities. Assuming the visual feature vector is 1024-dimensional, the IMU feature vector is 64-dimensional, and the GPS feature vector is 32-dimensional, feature concatenation yields an 1120-dimensional fused feature vector. Decision-level fusion comprehensively judges the decision results from different modules, using weighted voting or Bayesian inference methods to obtain the final decision.

[0050] In another embodiment, assume that the system collects the following data at a certain moment: the RGB image captured by the binocular camera is 1920×1080 pixels in size, and the depth map is 1920×1080 pixels in size; the acceleration data collected by the IMU is... Angular velocity data is GPS-collected location data is Through data-level fusion, these data are unified into the device coordinate system. Assuming the origin of the device coordinate system is the device center, the X-axis points directly forward, the Y-axis points directly to the left, and the Z-axis points directly upward, the GPS location data is transformed into the device coordinate system through coordinate transformation, yielding relative position coordinates (15.2, 8.5, 0) meters. Through feature-level fusion, visual features, IMU features, and GPS features are concatenated to obtain a fused feature vector. Through decision-level fusion, the obstacle detection results, tactile paving segmentation results, and traffic availability assessment results are integrated to obtain the final navigation decision.

[0051] In one embodiment, the system provides navigation information to the user via voice broadcast. The voice broadcast includes information such as the status of the tactile paving ahead, obstacle warnings, turning instructions, and distance information. The voice broadcast uses text-to-speech (TTS) technology to convert the navigation text into natural speech output. For example, if the system detects a bicycle parked on the tactile paving 3 meters ahead, the generated navigation text would be "3 meters ahead, bicycle parked on the tactile paving, please detour to the left." The TTS engine converts this text into speech and plays it to the user through headphones or a speaker. The speech rate is set to 1.2 times the normal speaking speed, and the volume is set to a medium volume to suit outdoor usage needs.

[0052] In another embodiment, it is assumed that the system continuously monitors the environment and provides voice prompts during the user's walk. During a 10-minute walk, the system provides 35 voice prompts, including 15 for the tactile paving status, 12 for obstacle warnings, and 8 for turning guidance. The average interval for the tactile paving status announcements is 40 seconds, and the content includes "Tactile paving is clear, please go straight" and "Tactile paving narrows, please be careful," etc. The average interval for the obstacle warning announcements is 50 seconds, and the content includes "Pedestrian 2 meters ahead" and "Electric vehicle on the right," etc. The average interval for the turning guidance announcements is 75 seconds, and the content includes "Please turn left at the intersection ahead" and "Continue straight for 50 meters and then turn right," etc. Through timely voice prompts, users can accurately understand the surrounding environment and make correct walking decisions.

[0053] In one embodiment, the system can trigger GPS route replanning in the event of an emergency. Emergency situations include: complete blockage of the tactile paving, detection of dangerous obstacles, and loss of GPS signal. For example, if the system detects that the tactile paving ahead is completely blocked by construction barriers, making passage impossible, the system will immediately trigger route replanning. Route replanning first obtains the user's current location and destination location, and then searches for alternative routes on an electronic map. Assuming the user's current location is... The destination location is The system found an alternative route on the electronic map, which is 850 meters long and has an estimated walking time of 12 minutes. Detailed information about the alternative route was announced to the user via voice: "The tactile paving ahead is under construction and impassable. A new route has been planned for you. Please turn left onto the pedestrian crossing, walk straight for 200 meters, and then turn right."

[0054] In another embodiment, assuming the user encounters three unexpected situations while walking, the system performs three route replanning operations for each. The first unexpected situation is that the tactile paving is blocked by a vehicle, increasing the replanned path length by 150 meters. The second unexpected situation is the detection of a large pothole ahead, increasing the replanned path length by 80 meters. The third unexpected situation is the loss of GPS signal; the system switches to inertial navigation mode, using IMU data for positioning. When the GPS signal is restored, the system automatically corrects the position error. Assuming the processing times for the three unexpected situations are 5 seconds, 3 seconds, and 8 seconds respectively, the system can complete route replanning and notify the user within 10 seconds for each situation, ensuring the continuity and real-time performance of navigation.

[0055] In one embodiment, the system cyclically performs monitoring and updates to ensure continuous real-time monitoring of the environment and navigation updates. The monitoring cycle is 100 milliseconds, meaning 10 environmental monitoring and navigation updates are performed per second. Within each monitoring cycle, the system sequentially performs the following operations: acquiring sensor data, performing image preprocessing, performing obstacle detection and tactile paving segmentation, performing virtual reconstruction and accessibility assessment, fusing and analyzing various data, generating navigation decisions, and broadcasting voice prompts. Assuming the execution time of a certain monitoring cycle is 85 milliseconds, which is less than the 100-millisecond monitoring cycle, the system can complete all processing tasks in real time. If the execution time exceeds the monitoring cycle, the system will automatically reduce the processing frequency to prioritize the real-time performance of core functions.

[0056] In another embodiment, it is assumed that the system executes 600 monitoring cycles within a 1-minute runtime. The average execution time for each monitoring cycle is 78 milliseconds, the maximum execution time is 95 milliseconds, and the minimum execution time is 65 milliseconds. The fluctuation in execution time is mainly affected by environmental complexity and the number of obstacles. In scenarios with simple environments and fewer obstacles, the execution time is shorter; in scenarios with complex environments and more obstacles, the execution time is longer. For example, in a complex scenario where 15 obstacles are detected and 3 breaks in the segmentation mask require virtual reconstruction, the execution time for this monitoring cycle is 92 milliseconds. Through algorithm optimization and hardware acceleration, the system can maintain real-time performance in various scenarios, meeting the application requirements of blind navigation.

[0057] Preferably, the specific implementation of multi-dimensional environmental perception in step S1 is as follows: real-time acquisition of RGB images of the surrounding environment through binocular cameras, and simultaneous acquisition of depth maps and image color information; generation of depth images based on distance data, and then construction of three-dimensional point clouds; simultaneous acquisition of device attitude using accelerometers and gyroscopes; understanding of real-time road conditions by combining global latitude and longitude coordinates, comprehensively perceiving the environment from three dimensions: visual, spatial, and motion, and achieving accurate grasp of real-time conditions through GPS synchronous positioning.

[0058] In another embodiment, assuming the segmentation mask size is 512×512 pixels, after bilinear interpolation, the segmentation mask size becomes 1920×1080 pixels. The original segmentation mask has 25,600 tactile pixels, which becomes 112,500 after interpolation. The interpolated segmentation mask has the same size as the original image and can be directly used for visualization and subsequent processing. Assuming there are jagged edges in the original segmentation mask, after bilinear interpolation, the edges become smooth and continuous. By comparing the segmentation masks before and after interpolation, it can be found that the pixel value transition in the edge region of the interpolated segmentation mask is more natural, resulting in a better visual effect.

[0059] In another embodiment, assuming a 10-minute walking process, the system detects a total of 25 obstacles, including 12 high-priority obstacles and 13 low-priority obstacles. High-priority obstacles include: 5 pedestrians, 3 bicycles, 1 electric vehicle, 2 trash cans, and 1 traffic cone. The system provides voice alerts for each high-priority obstacle, including obstacle type, distance, and location information. For example, "Pedestrian on the tactile paving 2 meters ahead, please walk around to the right," or "Bicycle parked on the tactile paving 5 meters ahead, please be careful." Following the voice prompts, the user successfully avoids all high-priority obstacles and safely completes the walking task.

[0060] The second embodiment of this invention provides an environmental perception navigation system based on tactile paving segmentation detection and virtual reconstruction, comprising: a model construction module for constructing an obstacle detection model based on an improved YOLOv11 model and a tactile paving segmentation model based on a GRFB-UNet network, and training them to obtain the trained obstacle detection model and tactile paving segmentation model; a multi-sensor fusion module for acquiring the image to be tested and environmental information through a camera, an inertial measurement unit, a depth sensor, and a GPS module; and a detection and segmentation module for preprocessing the image to be tested and inputting it into a shared shallow convolutional network for feature extraction. The system obtains a shared feature map, which is then input into the trained obstacle detection model and tactile paving segmentation model to perform obstacle detection and tactile paving segmentation, respectively, resulting in detection boxes and segmentation masks. A virtual tactile paving generation module is used to interpolate and restore the segmentation mask, spatially register it with the detection boxes, and visualize it to generate a virtual tactile paving area. A path planning and navigation module is used to calculate the intersection-over-union ratio (IoU) and coverage area of ​​the visualized detection boxes and segmentation masks. If both exceed a preset threshold, the tactile paving is identified as an obstacle and an alarm is triggered; otherwise, it is determined to be a passable tactile paving path. The module then plans the tactile paving path using GPS positioning data and provides navigation prompts via voice broadcast.

[0061] In one embodiment, the model construction module is implemented as follows: First, an improved YOLOv11 model network structure is constructed, comprising a backbone network, a neck network, and a detection head. The backbone network uses the C3k2_PConv module instead of the original C3k2 module, the neck network adopts a Slim-Neck structure, and the detection head uses a dynamic detection head. Then, the GRFB-UNet network structure is constructed, embedding GRFB modules in each stage of the encoder and decoder of the standard UNet network. Next, training datasets are prepared: the obstacle dataset contains 22 types of obstacles with a total of 10,000 labeled images; the tactile paving dataset contains 1,391 labeled images. Finally, the SGD optimizer is used for end-to-end training of the model for 300 training epochs, with the learning rate dynamically adjusted using a One-Cycle strategy. After training, the model parameters are saved for subsequent inference deployment.

[0062] In another embodiment, it is assumed that the model building module is trained on a GPU server with an NVIDIA RTX 3090 GPU and 24GB of VRAM. The training time for the improved YOLOv11 model is 48 hours, with the average loss decreasing from an initial 2.5 to a final 0.15 during training; the training time for the GRFB-UNet model is 24 hours, with the average loss decreasing from an initial 0.8 to a final 0.05 during training. After training, the model performance is evaluated on the test dataset: the improved YOLOv11 model has an average accuracy (mAP) of 85.3% and an inference speed of 45 FPS; the GRFB-UNet model has an intersection-over-union (IoU) ratio of 87.5% and an inference speed of 30 FPS. Both models meet real-time requirements and can be deployed to run on mobile or embedded devices.

[0063] This invention addresses the multi-task perception needs of blind people in travel scenarios by optimizing the structure of the YOLOv11 object detection model and combining it with the GRFB-UNet segmentation network to construct a guidance system for the blind. In the obstacle detection part, YOLOv11 uses the C3k2 module as the core feature extraction unit, effectively reducing feature redundancy through a cross-stage partial connection mechanism; the SPPF module enhances global feature capture capabilities through multi-scale pooling; and the C2PSA module enhances attention to key regions through a bi-branch self-attention mechanism. The introduction of the C3k2_PConv module, the Slim-Neck structure, and a dynamic detection head further improves the model's lightweight nature and detection accuracy. The tactile paving segmentation part uses the GRFB-UNet network, which obtains multi-scale receptive fields through the multi-branch convolutional structure of the GRFB module, effectively solving the problem of poor segmentation performance of the standard UNet network when processing narrow and elongated targets. Ultimately, by combining model parallel reasoning and result-level fusion, the obstacle detection results are combined with the tactile paving segmentation results, achieving lightweight, real-time, and high-precision collaborative perception of passable areas and surrounding obstacles in guide scenarios. This is suitable for deployment on mobile devices and edge devices, and can effectively improve the safety and environmental understanding of blind people when traveling.

[0064] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0065] Therefore, the embodiments should be considered as exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of the equivalents of the application are intended to be included within the invention.

[0066] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features of the invention herein.

Claims

1. An environmental perception navigation method based on tactile paving segmentation detection and virtual reconstruction, characterized in that, Includes the following steps: Step S1: Real-time acquisition of RGB images and depth maps of the environment using binocular cameras to construct a 3D point cloud; combined with accelerometers and gyroscopes to obtain device attitude, and GPS to obtain global coordinates to achieve multi-dimensional environmental perception; Step S2: Perform normalization and resolution adjustment preprocessing on the image, and extract edge texture and shape features through a shared shallow convolutional network; Step S3: Detect obstacles and predict bounding boxes based on the improved YOLOv11 model, segment the blind path area using the GRFB-UNet model, and perform masking and hole filling on the obstacle-occluded areas. Step S4: Denoise and virtually reconstruct the broken sections of the tactile paving, combine the 3D point cloud information with curve fitting to generate a continuous binary mask for the tactile paving, and project it into 3D space to determine the accessibility. Step S5: Integrate and analyze various real-time data, and provide navigation information via voice broadcast; in case of emergencies, trigger GPS route replanning, and continuously perform monitoring and updates.

2. The environmental perception navigation method based on tactile paving segmentation detection and virtual reconstruction according to claim 1, characterized in that, The specific implementation method of the multi-dimensional environmental perception is as follows: The system uses binocular cameras to capture RGB images of the surrounding environment in real time, simultaneously acquiring depth maps and image color information; it generates depth images based on distance data, and then constructs a 3D point cloud; it also uses accelerometers and gyroscopes to obtain the device's attitude; and it combines global latitude and longitude coordinates to understand the real-time road conditions, comprehensively perceiving the environment from three dimensions: visual, spatial, and motion, and uses GPS for synchronous positioning to achieve accurate grasp of the real-time situation.

3. The environmental perception navigation method based on tactile paving segmentation detection and virtual reconstruction according to claim 1, characterized in that, The specific steps of the normalization and resolution adjustment preprocessing are as follows: First, the pixel values ​​are normalized to the range of [0,1] or [-1,1] to ensure consistency with the model training conditions. Then, the image resolution is adjusted to 512×512 or 416×416 to adapt to the network input size of YOLOv11 and GRFB-Unet. Next, common features such as edge texture and shape of the image are extracted by sharing a shallow convolutional network to accurately identify the feature information of obstacles and blind paths.

4. The environmental perception navigation method based on tactile paving segmentation detection and virtual reconstruction according to claim 1, characterized in that, The specific operations for obstacle detection and tactile paving segmentation are as follows: Real-time obstacle detection is performed, obstacle bounding boxes and confidence scores are predicted, and the best detection results are retained. For blind path areas that are occluded by obstacles, 3D point cloud information is combined to determine the integrity of the blind path. The GRFB-UNet segmentation model is used for feature extraction and convolution operations, and a segmentation mask is performed (blind path is 1, background is 0), and holes are filled in the occluded areas.

5. The environmental perception navigation method based on tactile paving segmentation detection and virtual reconstruction according to claim 1, characterized in that, The specific steps for the virtual reconstruction of the broken tactile paving are as follows: For detected breaks in the tactile paving, noise reduction is first performed to eliminate interference from obstacles; then, by combining the ground plane information of the 3D point cloud, linear interpolation or spline curve fitting is used to virtually reconstruct the broken area, generating a continuous binary mask for the tactile paving; finally, the binary mask is projected onto 3D space to determine the accessibility of the tactile paving.

6. The environmental perception navigation method based on tactile paving segmentation detection and virtual reconstruction according to claim 1, characterized in that, The specific implementation of the information fusion and navigation feedback is as follows: The system analyzes and judges various real-time situations through information fusion technology and provides feedback to users through voice broadcast; when encountering emergencies, the system triggers GPS to replan the route; the whole process is executed in a loop to ensure continuous real-time monitoring of the environment and navigation updates.

7. The environmental perception navigation method based on tactile paving segmentation detection and virtual reconstruction according to claim 2, characterized in that, The coordinate mapping formula for registering the visual, spatial, and motion data is as follows: ； In the formula, These are the pixel coordinates at the original image size. Output the width and height of the segmentation mask for the model. , These are the width and height of the original image, respectively; , These represent the width and height of the segmentation mask, respectively.

8. The environmental perception navigation method based on tactile paving segmentation detection and virtual reconstruction according to claim 4, characterized in that, The process of mapping the segmentation mask coordinates at the model size back to the original image size includes: The formula for segmentation mask interpolation restoration is: ； In the formula, The interpolated pixel values ​​for the target coordinates at the original image size. , These represent the x and y coordinates of the target pixel at the original image size. These are the original pixel values. , The model outputs the x and y coordinates of the neighboring sampling points involved in the interpolation within the segmentation mask, where w is the distance weight. The model outputs the original pixel values ​​of the neighborhood sampling points in the segmentation mask, and then smooths the edges by weighted averaging.

9. The environmental perception navigation method based on tactile paving segmentation detection and virtual reconstruction according to claim 8, characterized in that, Consistency constraints are applied before performing the registration of the three dimensions of vision, space, and motion. The consistency loss function is as follows: ； In the formula For consistency loss function, Here, b is the area calculation function, b is the detection box, and M is the segmentation mask. Joint optimization is used to prevent detection box drift.

10. An environmental perception navigation system based on tactile paving segmentation detection and virtual reconstruction, applied to the environmental perception navigation method based on tactile paving segmentation detection and virtual reconstruction as described in any one of claims 1-9, characterized in that, include: Model construction module: Build and train the improved YOLOv11 obstacle detection model and GRFB-UNet tactile paving segmentation model, and obtain the detection and segmentation models after testing; Multi-sensor fusion module: It uses a binocular camera to acquire RGB images and depth maps, combines accelerometers and gyroscopes to obtain device attitude, and obtains global coordinate data through GPS; Detection and segmentation module: Extracts image features through a shared shallow convolutional network, performs obstacle detection and tactile paving segmentation, and outputs obstacle detection boxes and tactile paving segmentation masks; Virtual tactile paving generation module: Denoises and virtually reconstructs broken areas of tactile paving, performs interpolation restoration, spatial registration and visualization processing to generate continuous virtual tactile paving; Path planning and navigation module: Determines obstacle priority and triggers alarms, or replans the path based on GPS data, and provides navigation feedback through voice broadcast.

Images