Intelligent travel assistance system for the blind

Abstract

The application relates to the technical field of intelligent navigation, and discloses an intelligent blind person travel assisting system, which comprises a main system, a double-thread creation module is connected to the output end of the main system in an electrical signal mode, a feature extraction thread and a data processing thread are connected to the output end of the double-thread creation module in an electrical signal mode, an image preprocessing module is connected to the output end of the feature extraction thread in an electrical signal mode, a feature extraction module is connected to the output end of the image preprocessing module in an electrical signal mode, and a data acquisition module is connected to the output end of the data processing thread in an electrical signal mode. By adopting a target algorithm YOLO V5s based on a convolutional neural network, the object in the surrounding environment can be recognized, a low-illumination enhancement algorithm is used, the overall assisting system can perform well in the night or in a scene with weak light, obstacles in different environments can be accurately detected, and the system is adapted to processing complex environmental information, so that rich surrounding environmental information of the blind person is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent navigation, and in particular to an intelligent travel assistance system for the blind. Background Art

[0002] There are currently a large number of blind people, and in their daily travel, they mainly use tactile perception of the blind path to determine the direction of travel, and rely on a blind stick to detect whether the road ahead is clear. However, there is still a problem that the blind path is occupied, which affects the blind's safe travel. Reliance on the blind path greatly limits the blind's range of activities. Traditional guide sticks can no longer meet the current travel needs of the blind. There are some relatively simple auxiliary devices on the market, such as smart canes based on ultrasonic ranging. Its principle is to emit ultrasonic waves to the surroundings in real time, reflect the sound waves when encountering an object, and then calculate the distance to the object based on the return time of the ultrasonic wave. Since the principles of existing auxiliary devices are mostly simple physical methods, they are unable to screen obstacles and cannot determine the specific category of obstacles, and cannot provide users with richer information. Summary of the Invention

[0003] (1) Technical problems solved

[0004] In view of the deficiencies in the prior art, the present invention provides an intelligent travel assistance system for the blind.

[0005] (2) Technical solution

[0006] The present invention provides the following technical solution: an intelligent travel assistance system for the blind, including a main system, wherein the output end electrical signal of the main system is connected to a dual-thread creation module, the output end electrical signal of the dual-thread creation module is connected to a feature extraction thread and a data processing thread, the output end electrical signal of the feature extraction thread is connected to an image preprocessing module, the output end electrical signal of the image preprocessing module is connected to the feature extraction module, the output end electrical signal of the data processing thread is connected to a data acquisition module, and the output end electrical signal of the data acquisition module is connected to an image data judgment module.

[0007] Preferably, the output end electrical signal of the image preprocessing module is connected to the image acquisition module, the output end electrical signal of the image acquisition module is connected to the image brightness detection module, the output end electrical signal of the image brightness detection module is connected to the brightness judgment module, and the output end electrical signal of the brightness judgment module is connected to the image processing module.

[0008] Preferably, the output end electrical signal of the feature extraction module is connected to the feature fusion enhancement module, and the output end electrical signal of the feature fusion enhancement module is connected to the image update module.

[0009] Preferably, the output end electrical signal of the image data judgment module is connected to the feature map decoding module, and the output end electrical signal of the feature map decoding module is connected to the child thread creation module.

[0010] Preferably, the output end electrical signal of the sub-thread creation module is connected to the cluster ranging sub-thread module and the binocular ranging sub-thread module, the output end electrical signal of the cluster ranging sub-thread module is connected to the cluster coordinate calculation module, and the output end electrical signal of the cluster coordinate calculation module is connected to the distance estimation module.

[0011] Preferably, the output end electrical signal of the binocular ranging sub-thread module is connected to the view matching module, the output end electrical signal of the view matching module is connected to the object center point calculation module, and the output end electrical signal of the object center point calculation module is connected to the object distance calculation module.

[0012] Preferably, the output end electrical signals of the cluster ranging sub-thread module and the binocular ranging sub-thread module are connected to a distance judgment module, and the output end electrical signals of the distance judgment module are connected to a distance data processing module.

[0013] Preferably, the output end electrical signal of the distance data processing module is connected to a data sorting module, and the output end electrical signal of the data sorting module is connected to a voice broadcasting module.

[0014] (3) Beneficial effects

[0015] Compared with the existing technology, the present invention provides an intelligent travel assistance system for the blind, which has the following beneficial effects:

[0016] 1. This intelligent travel assistance system for the blind uses the YOLO V5s convolutional neural network-based target algorithm to identify objects in the surrounding environment. It also uses a low-light enhancement algorithm, enabling the overall assistance system to perform well at night or in low-light scenarios. It can accurately detect obstacles in different environments and adapt to processing complex environmental information, providing blind people with rich environmental information prompts.

[0017] 2. This intelligent travel assistance system for the blind performs various data adjustments for image data information by setting up a cluster ranging sub-thread module and a binocular ranging sub-thread module. At the same time, it combines a low-light enhancement algorithm with cluster ranging detection and binocular ranging detection to achieve real-time ranging of objects in the surrounding environment and improve the accuracy of voice broadcasting. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a structural schematic diagram of the present invention;

[0019] Figure 2 For the present invention Figure 1 Schematic diagram of the system structure of the image processing module;

[0020] Figure 3 For the present invention Figure 1Schematic diagram of the system structure of the feature extraction module;

[0021] Figure 4 For the present invention Figure 1 Schematic diagram of the system structure of the cluster ranging sub-thread module;

[0022] Figure 5 For the present invention Figure 1 Schematic diagram of the system structure of the binocular ranging sub-thread module;

[0023] Figure 6 This is a schematic diagram of the system process operation of the present invention;

[0024] Figure 7 For the present invention Figure 1 Schematic diagram of the process operation of image brightness detection. DETAILED DESCRIPTION

[0025] The present invention is further described in detail below with reference to the accompanying drawings, wherein the same parts are represented by the same figure numerals. It should be noted that the words "front", "rear", "left", "right", "up" and "down", "bottom" and "top" used in the following description refer to directions in the drawings, and the words "inside" and "outside" refer to directions toward or away from the geometric center of a specific component, respectively.

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0027] See also Figure 1-Figure 7 The present invention provides a technical solution: an intelligent blind travel assistance system, including a main system, the output end electrical signal of the main system is connected to a dual-thread creation module, the output end electrical signal of the dual-thread creation module is connected to a feature extraction thread and a data processing thread, the output end electrical signal of the feature extraction thread is connected to an image preprocessing module, the output end electrical signal of the image preprocessing module is connected to the feature extraction module, the output end electrical signal of the data processing thread is connected to a data acquisition module, and the output end electrical signal of the data acquisition module is connected to an image data judgment module.

[0028] The output end of the image preprocessing module is electrically connected with an image acquisition module, the output end and the input end of the image acquisition module are bidirectionally electrically connected with the output end and the input end of the image preprocessing module, the output end of the image acquisition module is electrically connected with an image brightness detection module, the output end of the image brightness detection module is electrically connected with a brightness judgment module, and the output end of the brightness judgment module is electrically connected with an image processing module.

[0029] The output end of the feature extraction module is electrically connected with a feature fusion and strengthening module, the output end of the feature fusion and strengthening module is electrically connected with an image updating module, the output end of the image data judgment module is electrically connected with a feature map decoding module, and the output end of the feature map decoding module is electrically connected with a sub-thread creation module.

[0030] The output end of the sub-thread creation module is electrically connected with a clustering distance measurement sub-thread module and a binocular distance measurement sub-thread module, the output end of the clustering distance measurement sub-thread module is electrically connected with a clustering coordinate calculation module, and the output end of the clustering coordinate calculation module is electrically connected with a distance estimation module.

[0031] The output end of the binocular distance measurement sub-thread module is electrically connected with a view matching module, the output end of the view matching module is electrically connected with an object center point calculation module, and the output end of the object center point calculation module is electrically connected with an object distance calculation module.

[0032] The output end of the clustering distance measurement sub-thread module and the binocular distance measurement sub-thread module is electrically connected with a distance judgment module, the output end of the distance judgment module is electrically connected with a distance data processing module, the output end of the distance data processing module is electrically connected with a data arrangement module, and the output end of the data arrangement module is electrically connected with a voice broadcast module.

[0033] Two feature extraction threads and data processing threads are created through a main system, the feature extraction thread first acquires an image, detects the image brightness, judges whether the brightness is too dark, adopts a green channel inverse color as a coefficient when the image brightness is too dark, multiplies an original image with the coefficient to obtain a new image layer, performs color filter mixing on the new image layer and the original image, then performs feature extraction, when the brightness is not too dark, the image can be directly subjected to feature extraction, after feature extraction, the features are fused and strengthened, and left and right views and feature maps are updated to make a detailed judgment on an obstacle.

[0034] The left and right views and feature maps are acquired by a data processing thread, it is judged whether it is the latest image data, an update flag on the shared memory is acquired, it is detected whether the update flag is 1, if it is 1, the feature map and left and right views on the shared memory are extracted, if the update flag is 0, it indicates that the data on the shared memory is old data, no data acquisition is performed, and the update flag is continuously detected, when it is detected that the update flag position is 1, the feature map and left and right views on the shared memory are acquired, and the update flag position is 0, indicating that the current data has been acquired and is old data, after the feature map is decoded, two sub-threads are created, which are a clustering ranging sub-thread module and a binocular ranging sub-thread module, the clustering ranging sub-thread module calculates the clustering point coordinates of the object, and substitutes into the corresponding clustering model to estimate the distance, the clustering model establishment method is as follows: first, images of different types of objects at different distances are collected, the distance values are 6-25 meters, and a total of 20 distance values, 100 images are collected at the same distance, the images are detected by the YOLO V5s model to obtain the object bounding box information: the length of the bounding box, the width of the bounding box and the area of the bounding box, finally, the object bounding box information is used as a clustering sample to cluster to obtain the corresponding clustering model, the clustering point coordinates of the object distance clustering are calculated according to the object bounding box information: the length of the bounding box, the width of the bounding box and the area of the bounding box, the corresponding clustering model is selected according to the specific type of the object, the Euclidean distance between the clustering point and each cluster center in the clustering model is calculated, the clustering point is divided into the nearest cluster, and the distance represented by the cluster to which the clustering point is divided is the distance finally obtained by the clustering model.

[0035] The binocular ranging sub-thread module obtains a depth map through binocular matching of the left and right views, calculates the object center point according to the bounding box information, obtains the object distance from the depth map according to the object center point, and obtains a disparity map through binocular matching of the acquired left and right views, and obtains a depth map from the disparity map, and the specific calculation formula is Where (X R -X T) is called disparity, f is the focal length of the camera, B is the baseline, both of which are fixed parameters of the camera. The center coordinates of the predicted frame are calculated according to the predicted frame information obtained after decoding the feature map, and the center coordinates of the predicted frame are used as the center coordinates of the object. The center coordinates are mapped to the depth map, and the depth value of the corresponding point on the depth map is read to obtain the distance value of the object. Wait for the two ranging threads to end. When the faster thread of the two threads ends, the feature thread is blocked. After the other thread ends, the feature thread is unblocked and continues to run. When the two ranging threads end, two different distance values ​​are obtained. When the distance obtained by binocular ranging is greater than 5 meters, the two distances are weightedly added. The formula is Z=K1×Z2+K2×Z2, where Z is the final ranging result, Z1 is the distance obtained by binocular ranging, Z2 is the distance obtained by the clustering model, K1 and K2 are the corresponding weights, and all the information obtained is sorted and voice broadcast, thereby providing accurate obstacle distance information for the blind.

[0036] The image preprocessing module acquires images through a binocular camera and preprocesses the acquired images. The feature extraction module performs target detection and feature fusion enhancement on the preprocessed images, and updates the obtained feature maps and left and right views to the shared memory. The data acquisition module is used to obtain the latest feature maps and left and right views on the shared memory and decode the acquired feature maps. The distance measurement module uses the decoded information and the acquired left and right views to measure the distance of the object and obtain the distance value of the object. The language broadcast module organizes all the information obtained and broadcasts it through the speaker.

[0037] In the image brightness detection module, when the brightness of the image is too dark, the image is enhanced under low illumination. When the brightness of the image is normal, it is not enhanced under low illumination. The image brightness detection converts the input RGB image into a grayscale image and calculates the mean da of the grayscale value of the image deviating from 128. The formula is used. Where x i is the gray value of each point in the image, N is the number of pixels in the image, and the weighted average deviation Ma of the image is calculated using the formula The Hist in the formula is the grayscale histogram of the image, and the ratio K of the absolute value of the mean da and the weighted average deviation Ma is calculated using the formula According to the calculated K value and da value, it is judged whether the brightness of the image is too dark. When K is greater than 1, it indicates that the brightness of the image is normal, and no low-light enhancement is required, and it is directly output. When K is less than 1 and da is less than 0, it indicates that the brightness of the image is too dark, and low-light enhancement is performed first, and then output. (This part describes the content of brightness detection, not the content of low-light enhancement) The low-light enhancement method is used to invert the green component value of the input RGB image as a coefficient value for subsequent processing. The formula α=255-G1 is used, where R1 is the green component value of the original RGB image. The various component values ​​of the original RGB image and the coefficient are multiplied to obtain a new layer. The formula R2=(α×R1)>>8 is used. G2=(α×G1)>>8 and B2=(α×B1)>>8, where R1, G1, and B1 are the component values ​​of the original RGB image, and R2, G2, and B2 are the component values ​​of the new layer. The new layer and the original image are subjected to a color filter mixing to complete the low-light enhancement process. The formula R3=255-((255-R1)×(255-R2)>>8), G3=255-((255-G1)×(255-G2)>>8), and B3=255-((255-B1)×(255-B2)>>8), where R3, G3, and B3 are the component values ​​of the enhanced image. After completing the low-light enhancement, the enhanced image is output for subsequent processing. (This part of the description is the content of low-light enhancement)

[0038] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent travel assistance system for the blind, including a main system, characterized by: The output end electrical signal of the main system is connected to the dual-thread creation module, the output end electrical signal of the dual-thread creation module is connected to the feature extraction thread and the data processing thread, the output end electrical signal of the feature extraction thread is connected to the image preprocessing module, the output end electrical signal of the image preprocessing module is connected to the feature extraction module, the output end electrical signal of the data processing thread is connected to the data acquisition module, and the output end electrical signal of the data acquisition module is connected to the image data judgment module; The output end electrical signal of the image preprocessing module is connected to the image acquisition module, the output end electrical signal of the image acquisition module is connected to the image brightness detection module, the output end electrical signal of the image brightness detection module is connected to the brightness judgment module, and the output end electrical signal of the brightness judgment module is connected to the image processing module; The output terminal electrical signal of the feature extraction module is connected to the feature fusion enhancement module, and the output terminal electrical signal of the feature fusion enhancement module is connected to the image update module; The output terminal electrical signal of the image data judgment module is connected to the feature map decoding module, and the output terminal electrical signal of the feature map decoding module is connected to the child thread creation module; The output end electrical signal of the sub-thread creation module is connected to the cluster ranging sub-thread module and the binocular ranging sub-thread module, the output end electrical signal of the cluster ranging sub-thread module is connected to the cluster coordinate calculation module, and the output end electrical signal of the cluster coordinate calculation module is connected to the distance estimation module; The output end electrical signal of the binocular distance measurement sub-thread module is connected to the view matching module, the output end electrical signal of the view matching module is connected to the object center point calculation module, and the output end electrical signal of the object center point calculation module is connected to the object distance calculation module; The output terminal electrical signals of the cluster ranging sub-thread module and the binocular ranging sub-thread module are connected to the distance judgment module, and the output terminal electrical signals of the distance judgment module are connected to the distance data processing module; The output terminal electrical signal of the distance data processing module is connected to the data sorting module, and the output terminal electrical signal of the data sorting module is connected to the voice broadcasting module.

2. The intelligent travel assistance system for the blind according to claim 1, characterized in that: A low-light enhancement algorithm is used in the image processing, and the cluster ranging sub-thread module uses the YOLOV5s target detection algorithm based on convolutional neural network.

Images