A blind guiding and obstacle avoidance system based on improved YOLOv5

Abstract

This invention discloses a navigation and obstacle avoidance system based on an improved YOLOv5s, including an image acquisition module responsible for acquiring road condition information ahead of a visually impaired person and automatically processing the video stream into a format acceptable to subsequent modules before transmitting it to a target detection and tracking module. The target detection and tracking module processes the incoming video stream frame by frame, using the improved YOLOv5s target detection algorithm to separate obstacles and guide objects from the complex pedestrian environment. The beneficial effects of this invention are: the navigation and obstacle avoidance method based on target detection technology can accurately inform visually impaired persons of the types of guide objects or obstacles ahead, enabling them to further understand the road conditions ahead; and the combination of target detection and target tracking technologies in the field of navigation and obstacle avoidance helps improve the effectiveness of navigation and obstacle avoidance and ensures the accuracy of navigation and obstacle avoidance services.

Description

Technical Field

[0001] This invention belongs to the fields of deep learning, computer vision, and obstacle avoidance for the blind, and specifically relates to a blind obstacle avoidance system based on an improved YOLOv5s. Background Technology

[0002] Visually impaired people rarely or never go out. The root cause of this situation lies in the fact that the early planning and construction of urban roads failed to keep pace with the rapid development of cities. In addition, construction workers did not pay enough attention to the construction of tactile paving during road construction. Furthermore, the rise of shared bicycles in my country in recent years has led to the frequent occurrence of tactile paving, the most common and basic barrier-free facility, being occupied by immovable infrastructure such as utility poles, bus stop signs, and movable obstacles such as shared bicycles and trash cans. This makes visually impaired people feel extremely unsafe when traveling alone, and they are prone to injury from bumps and tripping over obstacles, thus infringing on their right to travel freely and enjoy life.

[0003] In today's society, the most commonly used guide and obstacle avoidance tools for visually impaired people are white canes and guide dogs. White canes are the most common tool to assist visually impaired people in walking. They are an extension of the tactile sense of the visually impaired person's arm and have advantages such as low cost and portability. They have a long history of use, but their disadvantages are also very obvious. White canes can only detect a cone-shaped area of ​​1-2 meters in front of the user. Not only is the detection range small, but it also limits the walking speed of visually impaired people to a certain extent. In addition, white canes can only detect the presence or absence of obstacles, but cannot specifically identify what the obstacles are, their size, their material, etc., so their help to visually impaired people is very limited. Using guide dogs is an ideal means of assisting visually impaired people in traveling. However, using guide dogs to meet the travel needs of the large visually impaired population is like a drop in the ocean. In addition, the average cost of training a guide dog is about 200,000 yuan, the service life of a guide dog is 6-8 years, and the necessary expenses for guide dog owners are more than 1,000 yuan per month. Such high costs are not affordable for the general public.

[0004] Obstacle avoidance systems based on ultrasonic ranging and SLAM (Simultaneous Localization and Mapping) algorithms have been developed and applied in the daily lives of visually impaired individuals. However, both methods have certain drawbacks, preventing them from providing satisfactory guidance services to the visually impaired. Ultrasonic obstacle avoidance systems, such as the "Intelligent Terminal for Guiding the Blind (for Visually Impaired Persons)" (patent application numbers CN2023308030236) and the "An Ultrasonic Guiding System" (CN2021104305193), while simple to implement and inexpensive, have a detection radius only a few meters larger than that of a traditional white cane, and the detection accuracy decreases with distance. SLAM algorithms cannot perceive the specific type of obstacle in front of a visually impaired person, resulting in very limited information that visually impaired individuals can obtain from the obstacle avoidance system. Summary of the Invention

[0005] The purpose of this invention is to provide a guide and obstacle avoidance system based on an improved YOLOv5s, which uses a target detection algorithm as the core and is supplemented by a target tracking algorithm and a self-designed intelligent hazard analysis algorithm. It is mounted on an edge computing device and features simple operation, comprehensive functions, low cost, and good portability, providing accurate and efficient guide and obstacle avoidance services for visually impaired people.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a guidance and obstacle avoidance system for the visually impaired based on an improved YOLOv5s, comprising...

[0007] The image acquisition module is responsible for acquiring road condition information in front of the visually impaired person and automatically processing the video stream into a format that can be accepted by subsequent modules before transmitting it to the target detection and tracking module.

[0008] The target detection and tracking module is responsible for processing the incoming video stream frame by frame. It separates obstacles and guide objects from the complex pedestrian environment using the improved YOLOv5s target detection algorithm. Then, the Deep-Sort target tracking algorithm tracks the detected targets, obtains the detection box coordinates, category, and unique ID information of each target, and transmits this information to the hazard intelligent analysis module.

[0009] The hazard intelligent analysis module has a built-in hazard intelligent analysis algorithm that uses the inferred target information to perform hazard analysis and determine the condition of the road surface and obstacles ahead. When the road conditions ahead of the visually impaired person change or there is a possibility of bumping into an obstacle, the hazard intelligent analysis module will promptly generate a voice broadcast statement containing detailed information about the road surface or obstacle ahead and avoidance prompts, and transmit it to the voice broadcast module.

[0010] The voice broadcast module is responsible for broadcasting the voice messages generated by the hazard intelligent analysis module at an appropriate volume and speed to visually impaired individuals, thereby enhancing their perception of the surrounding road conditions and assisting them in avoiding obstacles in a timely manner and traveling safely.

[0011] As a preferred technical solution of the present invention, it also includes using the TensorRT deep learning inference engine to perform half-precision compression on the improved YOLOv5s algorithm model.

[0012] As a preferred technical solution of the present invention, in the image acquisition module, a Sony IMX219 camera with a field of view of 77° is selected as the image acquisition module to obtain image input of the road conditions in front of the visually impaired person when walking; in the provisional guide and obstacle avoidance system, the camera is installed at a height of 1.1 meters.

[0013] As a preferred technical solution of the present invention, it also includes target classification and dataset construction; the target classification is divided into two main categories: obstacles and guide objects for the visually impaired.

[0014] As a preferred technical solution of the present invention, the obstacle category is further subdivided into three subcategories: dynamic obstacles, static obstacles, and unpredictable obstacles.

[0015] As a preferred technical solution of the present invention, the dataset has three sources during the dataset construction process: selected from the WOTR dataset; collected from the Internet; and offline street scene photos.

[0016] As a preferred technical solution of the present invention, the improved object detection algorithm includes introducing a Ghost convolutional module, adding a CBAM attention mechanism, and replacing the PANet network with a weighted bidirectional feature pyramid network BiFPN.

[0017] As a preferred technical solution of the present invention, the improved YOLOv5s algorithm operation steps are as follows:

[0018] The image frames to be processed are fed into the backbone network of the model from the input. The backbone network consists of one CBS module, four GConv modules, four GC3 modules, one CBAM module, and one spatial pyramid pooling layer.

[0019] The CBS module is located at the very front of the backbone network and is responsible for warming up the network.

[0020] The alternating GConv and GC3 modules are responsible for progressively extracting feature maps from the input image, including shallow location feature maps and deep semantic feature maps;

[0021] The CBAM module is responsible for enhancing the extraction of spatial and semantic feature information of small targets;

[0022] The final spatial pyramid pooling layer of the backbone network is responsible for fusing feature maps of different scales and feeding them into the neck of the model;

[0023] The neck of the model consists of four GConv modules, four GC3 modules, four BiFPN networks, and two upsampling operations. The neck has an overall structure of "bottom-up and top-down". The GConv and GC3 modules are responsible for further feature extraction from the input. The BiFPN network is responsible for fusing the features from different levels. The fused feature maps from different levels are fed into the head of the model at different stages. The head of the model uses the input features to generate multiple object detection candidate boxes and confidence scores of predicted categories in the original input image and outputs this information.

[0024] Compared with the prior art, the beneficial effects of the present invention are:

[0025] The obstacle avoidance method based on target detection technology can accurately inform visually impaired people of the type of guide objects or obstacles in front of them, enabling them to better understand the road conditions ahead.

[0026] Combining target detection and target tracking technologies in the field of guide and obstacle avoidance can help improve the effectiveness of guide and obstacle avoidance and ensure the accuracy of guide and obstacle avoidance services.

[0027] By combining software-level guidance and obstacle avoidance algorithms with edge computing devices, a truly practical, easy-to-operate, portable, and highly accurate guidance and obstacle avoidance system has been achieved. Attached Figure Description

[0028] Figure 1 This is a diagram showing the overall system structure of the present invention;

[0029] Figure 2 This diagram illustrates the selection and classification of the detection targets in this invention.

[0030] Figure 3 This is a diagram showing the processing logic settings for different types of targets in this invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Please see Figures 1-3This invention provides a guide and obstacle avoidance system based on an improved YOLOv5s, which includes an image acquisition module, a target detection and tracking module, a hazard intelligent analysis module, and a voice broadcast module.

[0033] The image acquisition module is responsible for acquiring road condition information ahead of the visually impaired person and automatically processing the video stream into a format that can be accepted by subsequent modules before transmitting it to the target detection and tracking module.

[0034] The target detection and tracking module is responsible for processing the incoming video stream frame by frame. It separates obstacles and guide objects from the complex pedestrian environment using the improved YOLOv5s target detection algorithm. The Deep-Sort target tracking algorithm then tracks the detected targets, obtains information such as the detection box coordinates, category, and unique ID of each target, and transmits this information to the hazard intelligent analysis module.

[0035] The hazard intelligent analysis module has a built-in hazard intelligent analysis algorithm designed by us. It uses the target information inferred from the previous module to perform hazard analysis, mainly to judge the condition of the road surface and obstacles ahead. When the road conditions ahead of the visually impaired person change or there is a possibility of bumping into an obstacle, the hazard intelligent analysis module will generate a voice broadcast statement containing detailed information about the road surface or obstacle ahead and avoidance prompts in a timely manner, and transmit it to the voice broadcast module.

[0036] The voice broadcast module is responsible for transmitting the voice broadcast statements generated by the hazard intelligent analysis module at an appropriate volume and speed to inform visually impaired individuals, thereby enhancing their perception of the surrounding road conditions and assisting them in avoiding obstacles in a timely manner and traveling safely.

[0037] Target classification and dataset construction:

[0038] Stray cats, warning posts, bicycles, and other objects commonly found on sidewalks that may affect the walking of visually impaired people are included in the detection target sequence. In order to establish a systematic analysis and processing of these targets, the detection targets are classified into two major categories: obstacles and guide objects. The obstacle category is further subdivided into three subcategories: dynamic obstacles, static obstacles, and unpredictable obstacles.

[0039] The definitions of each class are as follows:

[0040] Dynamic obstacles: Intelligent biological entities that will actively avoid visually impaired people;

[0041] Static obstacles: Due to road planning deficiencies, various infrastructure facilities installed on and near tactile paving require visually impaired people to actively avoid them;

[0042] Uncertain obstacles: mainly various motor vehicles, sometimes stationary on the road, sometimes in motion under the driver's control, requiring visually impaired people to judge their state and then make different actions;

[0043] Guide equipment: various infrastructure that assists visually impaired people in walking.

[0044] In constructing the dataset, our dataset mainly comes from three sources: (a) selected from the WOTR dataset, (b) collected from the Internet, and (c) offline street scene photos; a total of 42,000 images were collected, and 115,000 detection box samples of obstacles or guide objects, including tactile paving, zebra crossings, pedestrians, bicycles, tree trunks, trash cans, and warning posts, were manually labeled for the object detection algorithm to train and evaluate the model.

[0045] Improvements to object detection algorithms:

[0046] To balance the real-time performance and accuracy of target detection, the YOLOv5s algorithm was selected as the target detection algorithm for this system. To balance the detection accuracy and detection speed of the model, YOLOv5s was chosen as the target detection algorithm for this system, and the model was improved with the goal of "improving the detection speed without losing detection accuracy as much as possible". The specific improvements are as follows.

[0047] Introducing the Ghost convolution module: In order to reduce the inference computation of the model as much as possible without losing detection accuracy, the Ghost convolution module replaces all standard convolution modules in the backbone network except the first one. The basic idea of ​​Ghost convolution is to break down the general convolution into three steps based on the relationship between feature maps. First, a small number of regular convolutions are performed to generate feature maps. Then, the generated feature maps are subjected to inexpensive group convolution to obtain Ghost feature maps. Finally, the feature maps obtained in the first and second steps are concatenated to generate the final feature map.

[0048] To address the issue that the native YOLOv5s algorithm cannot effectively identify small targets (such as manhole covers or fire hydrants in the distance), we added a convolutional block attention module (CBAM) to the end of the backbone network responsible for feature extraction in YOLOv5s. CBAM is a hybrid attention mechanism that combines channel attention and spatial attention. It first extracts global features channel by channel to generate channel attention feature maps, and then uses these as input to the spatial attention module to obtain a hybrid domain feature map. CBAM can consider feature information in both spatial and channel dimensions simultaneously, enhancing the model's ability to focus on target regions and enabling the model to more accurately locate and identify small targets.

[0049] Replacing PANet with a Weighted Bidirectional Feature Pyramid Network (BiFPN): To improve the feature fusion capability of the model, the original PANet path aggregation network in the YOLOv5s model is proposed to be replaced with a BiFPN network. Compared with PANet, BiFPN removes nodes with only one input end and adds skip links in the intermediate layers. These improvements make BiFPN more efficient in feature fusion and lower in computational cost than PANet.

[0050] The improved YOLOv5s algorithm operates as follows: The image frame to be processed is input into the model's backbone network. The backbone network consists of one CBS module, four GConv modules, four GC3 modules, one CBAM module, and one spatial pyramid pooling layer. The CBS module is at the very front of the backbone network and is responsible for warming up the network. The alternating GConv and GC3 modules are responsible for progressively extracting feature maps from the input image. The backbone network consists of a shallow spatial feature map and a deep semantic feature map. The CBAM module is responsible for enhancing the extraction of spatial and semantic feature information of small targets. The spatial pyramid pooling layer at the end of the backbone network is responsible for fusing feature maps of different scales and feeding them into the model's neck. The model's neck consists of four GConv modules, four GC3 modules, four BiFPN networks, and two upsampling operations. The neck has an overall structure of "bottom-up and top-down". The GConv and GC3 modules are responsible for further feature extraction from the input. The upsampling operation is to enlarge the size of the input feature map. The BiFPN network is responsible for fusing the input features at different levels. The fused feature maps at different levels are fed into the model's head at different stages. Here, the model's head uses the input features to generate multiple object detection candidate boxes and confidence scores for predicted categories in the original input image and outputs this information.

[0051] Design of intelligent algorithm for obstacle hazard analysis:

[0052] To further demonstrate that different types of obstacles require different pre-alarm times, red and yellow warning zones were manually divided into the images captured by the camera based on the distance to the visually impaired person; for obstacles, the area where the system triggers an alarm also varies depending on whether the obstacle is static, dynamic, or unpredictable.

[0053] Different processing logics are set for different types of targets. For each target input into the hazard intelligent analysis module, the system first determines its category based on its type, then determines whether the target will cause harm to the visually impaired person and the severity of the harm based on the target's location, distance from the visually impaired person, and speed, and generates corresponding voice broadcast statements. Guide objects are broadcast once at fixed intervals, while obstacles are broadcast only once per ID. The specific processing methods for obstacles and guide objects in this system are as follows:

[0054] Handling dynamic obstacles: If the target is a type of dynamic obstacle, considering that all targets in this category are intelligent beings that will actively avoid visually impaired individuals and will not significantly affect their normal walking, we choose not to process dynamic obstacles.

[0055] Handling static obstacles: Since static obstacles do not move, they pose a relatively small risk of harm to visually impaired individuals. When a static obstacle enters the red alert zone, the system generates a voice prompt reminding the visually impaired person to detour to the left or right based on the obstacle's position relative to the person. At this point, the obstacle is relatively close to the person, but the person still has ample time to detour around it. If the system prompts the person to detour when the obstacle is far away, it is not conducive to the visually impaired person controlling the detour distance, and there may be situations where the person detours and returns to the original walking route but has not yet avoided the obstacle.

[0056] Handling of indeterminate obstacles: For indeterminate obstacles, the system first uses the magnitude of the coordinate displacement of the obstacle detection box in multiple consecutive frames to determine whether the indeterminate obstacle's motion state is static or dynamic. If the target is static, it is treated as a static obstacle. If the obstacle is dynamic, such obstacles (e.g., electric bicycles coming from the opposite direction on the sidewalk or private cars parked on the sidewalk) are usually highly dangerous, and visually impaired people will suffer extremely serious injuries if they collide with these obstacles. Therefore, the system is set to generate a voice prompt to visually impaired people to stop immediately and give way when it detects a moving indeterminate obstacle entering the yellow zone.

[0057] Handling guide objects: Unlike obstacles, guide objects appear in the camera's view for extended periods. If the voice broadcast frequency for guide objects is set to once per ID, the system will broadcast voice messages too frequently, causing visually impaired individuals to be constantly distracted by the broadcast content and reducing their attention to other information, potentially leading to accidents. Furthermore, if the voice broadcast thread is occupied for an extended period by broadcasting information related to guide objects and fails to promptly broadcast voice messages for suddenly appearing obstacles, the system will be unable to alert the visually impaired individual before a collision with the obstacle. Therefore, at fixed time intervals, corresponding voice broadcast messages are generated for guide objects appearing in the camera's view based on their location.

[0058] Furthermore, since different guideways have significantly different effects on visually impaired individuals, the system needs to be configured to handle different guideways differently and generate different voice announcements. Typically, railings are used to separate roads and sidewalks. If the system detects railings on both sides, it indicates that a visually impaired person may be too close to the roadside. If the system detects a railing ahead, the road may be turning. When the system detects tactile paving, to ensure that the visually impaired person's walking direction is consistent with the paving, the position of the paving relative to the visually impaired person needs to be constantly monitored, but a corresponding voice announcement is only generated when the visually impaired person deviates from the path or the paving ends. The handling of zebra crossings is relatively simple. When a zebra crossing appears within a red warning zone, the hazard analysis module generates a voice announcement indicating that an intersection is ahead. When stairs are detected directly in front of the camera, the system generates a voice announcement saying, "There are stairs directly ahead; please walk carefully to avoid misstepping."

[0059] Physical construction of the guide and obstacle avoidance system:

[0060] The image acquisition module is responsible for recording the road conditions ahead as the visually impaired person walks and transmitting the video stream to the system for further processing. After several rounds of performance and price comparisons, this system selected a Sony IMX219 camera with a field of view of 77° as the image acquisition module to obtain image input of the road conditions ahead as the visually impaired person walks. In the tentative guidance and obstacle avoidance system, the camera is mounted at 1.1 meters. According to the sine theorem, the blind zone of this camera is approximately 1.3 meters in front of the visually impaired person, which is close to the walking experience of a sighted person.

[0061] The task of the target detection and tracking module is to receive the video stream of the road conditions ahead captured by the image acquisition module when the visually impaired person is walking, run the YOLOv5s target detection algorithm to perform target detection frame by frame on the incoming video stream, and pass the detection structure into the Deep-Sort target tracking algorithm to label the detection boxes for all detected targets and obtain data such as the unique ID, category, and detection box coordinates of each target detection box.

[0062] In the hazard intelligent analysis module, the system takes as input the data such as the category, unique ID, and detection box coordinates of all detected targets obtained by the target detection and tracking module, performs hazard analysis on the targets, and generates corresponding voice broadcast statements.

[0063] The voice broadcast module, as the part of the system that directly interacts with visually impaired individuals, is responsible for broadcasting the voice messages generated by the hazard intelligent analysis module to inform visually impaired individuals of the road conditions ahead. A relatively inexpensive and mainstream speaker was selected as the voice broadcast module for this system, and it is connected to the Jetson Nano motherboard via DuPont wires. After field testing, when the system was deployed at the waist of a visually impaired individual, the maximum volume of the YB-MVR01 voice broadcast module was sufficient for the visually impaired individual to clearly hear the voice content broadcast by the system in a slightly noisy street environment.

[0064] In addition, this system uses the TensorRT deep learning inference engine to perform half-precision compression on the improved YOLOv5s algorithm model, which improves the inference speed of the algorithm model and reduces energy consumption with almost no loss of accuracy. Finally, all the hardware and software parts of the system are integrated into the Jetson Nano B01 edge computing device as the physical terminal of the system.

[0065] Although embodiments of the invention have been shown and described in detail above, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A guidance and obstacle avoidance system for the visually impaired based on an improved YOLOv5s, characterized in that: include The image acquisition module is responsible for acquiring road condition information in front of the visually impaired person and automatically processing the video stream into a format that can be accepted by subsequent modules before transmitting it to the target detection and tracking module. The target detection and tracking module processes the incoming video stream frame by frame. Using the improved YOLOv5s target detection algorithm, it separates obstacles and tactile paving from the complex pedestrian environment. The Deep-Sort target tracking algorithm then tracks the detected targets, obtaining the bounding box coordinates, category, and unique ID information for each target. This information is then passed to the hazard analysis module. The improved YOLOv5s algorithm operates as follows: The image frames to be processed are fed into the backbone network of the model from the input. The backbone network consists of one CBS module, four GConv modules, four GC3 modules, one CBAM module, and one spatial pyramid pooling layer. The CBS module is located at the very front of the backbone network and is responsible for warming up the network. The alternating GConv and GC3 modules are responsible for progressively extracting feature maps from the input image, including shallow location feature maps and deep semantic feature maps; The CBAM module is responsible for enhancing the extraction of spatial and semantic feature information of small targets; The final spatial pyramid pooling layer of the backbone network is responsible for fusing feature maps of different scales and feeding them into the neck of the model; The neck of the model consists of four GConv modules, four GC3 modules, four BiFPN networks, and two upsampling operations. The neck has an overall structure of "bottom-up and top-down". The GConv and GC3 modules are responsible for further feature extraction from the input. The BiFPN network is responsible for fusing the features from different levels. The fused feature maps from different levels are fed into the head of the model at different stages. The head of the model uses the input features to generate multiple object detection candidate boxes and confidence scores for predicted categories in the original input image, and outputs this information. The hazard intelligent analysis module has a built-in hazard intelligent analysis algorithm that uses the inferred target information to perform hazard analysis and determine the condition of the road surface and obstacles ahead. When the road conditions ahead of the visually impaired person change or there is a possibility of bumping into an obstacle, the hazard intelligent analysis module will promptly generate a voice broadcast statement containing detailed information about the road surface or obstacle ahead and avoidance prompts, and transmit it to the voice broadcast module. The voice broadcast module is responsible for broadcasting the voice messages generated by the hazard intelligent analysis module at an appropriate volume and speed to visually impaired individuals, thereby enhancing their perception of the surrounding road conditions and assisting them in avoiding obstacles in a timely manner and traveling safely.

2. The blind guidance and obstacle avoidance system based on the improved YOLOv5s according to claim 1, characterized in that: It also includes using the TensorRT deep learning inference engine to perform half-precision compression on the improved YOLOv5s algorithm model.

3. The blind guidance and obstacle avoidance system based on the improved YOLOv5s according to claim 1, characterized in that: In the image acquisition module, a Sony IMX219 camera with a field of view of 77° is selected as the image acquisition module to obtain image input of the road conditions in front of the visually impaired person when walking; in the tentative guide and obstacle avoidance system, the camera is installed at a height of 1.1 meters.

4. The blind guidance and obstacle avoidance system based on the improved YOLOv5s according to claim 1, characterized in that: It also includes target classification and dataset construction; the target classification is divided into two main categories: obstacles and guide objects for the visually impaired.

5. A guidance and obstacle avoidance system for the visually impaired based on an improved YOLOv5s according to claim 4, characterized in that: The obstacle category is further subdivided into three subcategories: dynamic obstacles, static obstacles, and unpredictable obstacles.

6. A guidance and obstacle avoidance system for the visually impaired based on an improved YOLOv5s according to claim 4, characterized in that: During the dataset construction process, the dataset came from three sources: selected from the WOTR dataset; collected from the Internet; Actual street scene photos.

Images