Intelligent blind guiding device based on machine vision

Abstract

The invention discloses an intelligent blind guiding device based on machine vision, which is characterized in that a Raspberry Pi 4 Model B (RPI 4B for short) is arranged on a blind guiding stick to serve as a microcontroller, and four cameras, a GPS (Global Positioning System) module, a voice module, keys and the like are additionally arranged. Firstly, the environment in front of and on the two sides of the stick body is monitored through the machine vision technology, obstacle information (including categories, directions and distances) is obtained to guide a user to avoid obstacles through voice, and traffic light and countdown information is obtained to guide the user to cross the road through voice. Secondly, a GPS module is arranged in the blind guiding stick, real-time positioning of a user can be converted into a visual map through a key at any time, and then the visual map is sent to a WeChat contact person. And finally, a navigation service can be awakened through voice, a destination and a travel mode (default as walking) are determined through voice recognition and feedback, route planning is performed based on a Baidu map, and a voice module guides the blind to travel. A triangular ground touching device composed of a universal wheel and a directional wheel is arranged at the bottom of the walking stick, so that in actual use, a user can not only save labor, but also effectively perceive physical tactile feedback from the ground.

Description

Technical Field

[0001] The invention belongs to the technical field of blind guiding, and in particular relates to an intelligent blind guiding device based on machine vision. Background Art

[0002] Blind paths, traditional canes and guide dogs are common ways for blind people to travel, but they cannot meet their needs. Damage and occupation of blind paths restrict blind people's travel; traditional canes cannot help blind people identify specific and changing road conditions such as obstacles and road signs on both sides; and the training cost of guide dogs is very high, and many social places prohibit guide dogs from entering. The above reasons have become the main factors hindering the travel of visually impaired patients.

[0003] To solve this problem, the foreign Wewalk company and the Stanford team have successively developed smart guide sticks and put them on the market. Although both can effectively help the blind avoid obstacles, the ultrasonic obstacle avoidance of the Wewalk cane is easily affected by the environment and has a limited recognition range. The enhanced cane of Stanford uses two-dimensional (2D) laser radar for detection, which makes the cost higher. Domestic scholars have also designed many guide sticks, but they mainly use ultrasonic obstacle avoidance and have no travel navigation function. Therefore, it is urgent to design a cane that reminds visually impaired patients to avoid surrounding obstacles, guide them to cross the road, and realize intelligent navigation. . Summary of the invention

[0004] The purpose of the present invention is to provide an intelligent blind guide device based on machine vision to ensure the travel safety of visually impaired patients, improve the intelligence and functional diversification of blind guide facilities, and ensure the travel safety of visually impaired patients in daily traffic. In order to achieve the above purpose, the present invention provides the following technical solutions:

[0005] An intelligent blind-guiding device based on machine vision, the realization of the obstacle avoidance function includes the following steps: The microcontroller controls the four cameras to simultaneously collect environmental images in front of the user and on both sides of the user, and processes the image data in parallel through the MobileNet V1 model to obtain object category information and contour boxes; Furthermore, according to the monocular camera ranging algorithm and based on the Pythagorean theorem, a correction factor obtained from the pitch angle and roll angle of the camera is introduced, and the horizontal distance between the user and the identified object is obtained through the center point of the bottom wireframe of the contour mark frame, and obstacles that need to be avoided are screened out based on the distance; Furthermore, the position of the obstacle in the image relative to the camera (i.e., the user) is obtained based on the pixel point as the criterion, according to the position of the obstacle in the image. Furthermore, the obtained obstacle information (including object category, orientation, and distance) is matched with the voice field, and three sets of data information are broadcast to the user through the voice module according to the distance from near to far.

[0006] An intelligent blind guide device based on machine vision, the realization of its traffic light and countdown recognition function includes the following steps: The image captured by the front upper camera is input into the MobileNet V1 training model every 29 frames, and the image is scaled and segmented if a traffic light is detected; Furthermore, the image is converted into HSV color space, and the color information is obtained by comparing the red and green color thresholds; Furthermore, according to the monocular camera ranging algorithm, the distance between the traffic light and the user is obtained through the center point of the bottom wireframe of the contour mark frame; Furthermore, the text information of the traffic light timer in the image is extracted through Python's easyOCR; Furthermore, the obtained color, number, and distance information of the traffic light are matched with the voice text, and a prompt voice is output through the voice module, which has a higher priority than the obstacle avoidance recognition module.

[0007] An intelligent blind guide device based on machine vision, the implementation of the function of sending a positioning map includes the following steps: The microcontroller controls the GPS module to capture the latitude and longitude information of the stick (i.e. the user) in real time; Furthermore, the Baidu map API is called to perform reverse geocoding of the longitude and latitude information to achieve the effect of location visualization and generate an HTML file; Further, the HTML file is hosted to a local address through the Node.js server function of the microcontroller; Furthermore, after the button is triggered, the URL of the location visualization is sent to the WeChat of the emergency contact through the ServerChan service; Furthermore, the contact can view the location map of the stick (ie, the user) by clicking the URL.

[0008] An intelligent blind guide device based on machine vision, the implementation of the route planning function includes the following steps: Enable this function through voice wake-up; Furthermore, the destination and travel mode are inputted by voice recognition, and the positioning information of the stick is used as the starting point; Furthermore, route planning is performed based on Baidu Map API to obtain information on all road sections, and the voice module provides voice guidance to the user, which has a priority lower than the voice reminder of the obstacle avoidance module.

[0009] The present invention also discloses an intelligent blind-guiding device based on machine vision, comprising a cane body with a three-section telescopic pull rod and a voice module, wherein the cane body is equipped with four cameras (including the front lower part, the front upper part and the left and right sides), a ground contact device, a main box, two buttons, a handle and a bracket; the main box is divided into two parts, one part is embedded with a mobile power supply, and the other part is encapsulated with a microcontroller and a GPS module; Further, among the buttons, one is a power-on button for the device, and the other is a trigger button for sending a user location map; Furthermore, the ground contact device is composed of fixed wheels and universal wheels, and a bracket is arranged on the upper surface of the device to fix the angle between the stick body and the horizontal plane; Furthermore, the camera, GPS module, and buttons are all connected to the microcontroller, and all the wires can be inserted inside the stick body; Furthermore, the voice module is connected to the microcontroller RPI 4B via Bluetooth, so that voice input and voice broadcast can be realized.

[0010] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: (1) The intelligent guide cane based on machine vision can successfully obtain the category, distance and direction of obstacles, which helps users deal with more complex traffic environments; (2) The HSV color space and easyOCR are used to extract the color and timing information of traffic lights, and the information is fed back to the user through voice to assist the user in crossing the road; (3) When a user encounters an emergency (such as getting lost), he or she can send a location map to a contact with one click. The recipient can zoom and move the map to obtain the user's location in a clearer and more intuitive way; (4) This design can use a 5V mobile power supply as the power supply device of the cane, which can be replaced by the user, making it more convenient for users to carry and use, and has a simple structure. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 It is the overall module principle diagram of the present invention; Figure 2 Flowchart for identifying obstacle avoidance module; Figure 3 This is a flow chart of the traffic light and countdown recognition module; Figure 4 A flowchart for the route planning module; Figure 5Schematic diagram of the structure of the intelligent blind guide device in the present invention (marked in the figure: 1, stick body; 2, mobile power supply; 3, microcontroller; 4, GPS module; 5, camera; 6, ground contact device; 7, main box; 8, button; 9, voice module; 10, handle; 11, bracket).

Claims

1. An intelligent blind guide device based on machine vision, It is characterized in that The implementation of the obstacle avoidance function includes the following steps: S1, the microcontroller controls the four cameras on the front upper part, front lower part and left and right sides of the stick to simultaneously collect the user's environmental image; S2, four-way image data is input into the pre-trained model for parallel processing to obtain object category information and perform contour marking; S3. According to the monocular camera ranging correction algorithm, the horizontal distance between the user and the identified object is obtained from the object frame, and obstacles that meet the distance threshold (i.e., need to be avoided) are screened out; S4, according to the position of the obstacle in the image in the image coordinate system, obtain the position of the obstacle in the camera (user); S5. Match the obstacle information (including object type, direction, and distance) obtained in S3 with the voice field, and broadcast three sets of data information from near to far according to the distance.

2. An intelligent blind guide device based on machine vision, It is characterized in that The implementation of the traffic light and countdown recognition function includes the following steps: S6, collecting images of the upper front camera every 29 frames, inputting the environment image into the pre-trained model, and if a traffic light is detected, proceeding to step S7, otherwise waiting for the next detection; S7. Obtain the distance between the traffic light and the user according to the distance measurement method of S3; S8, after scaling and segmenting the image, perform color extraction and text extraction, match the color, number, and distance information of the traffic light with the voice text, and output a prompt voice through the voice module to inform the user, which has a higher priority than the recognition and obstacle avoidance module.

3. An intelligent blind guide device based on machine vision, It is characterized in that The sending of visual positioning information includes the following steps: S9, the microcontroller controls the GPS module to collect the latitude and longitude information of the stick in real time; S10, calling Baidu Map API, performing reverse address resolution on the longitude and latitude information to achieve a location visualization effect, and generating an HTML file; S11, start the Node.js server function of the microcontroller, and load the HTML file to the local address. Under the action of the button, send it to the contact through WeChat through the ServerChan service. The contact clicks the URL to view the location map of the stick (ie, the user).

4. An intelligent blind guide device based on machine vision, It is characterized in that The implementation of the route planning function includes the following steps: S12, turn on the function by voice wake-up; S13, input the destination and mode of travel (walking is the default) using the voice recognition algorithm; S14, using the positioning information in S8 as the starting point, performing route planning based on Baidu Map API and obtaining all road section information, and the voice module guides the user to travel.

5. According to the intelligent blind guiding device based on machine vision described in claim 1 and claim 2, the pre-trained model used is the official pre-trained model of MobileNet V1.

6. According to the intelligent blind guiding device based on machine vision described in claim 3, ServerChan in step S11 is a third-party platform for communication between the microcontroller and WeChat.

7. An intelligent blind guide device based on machine vision, It is characterized in that The invention comprises a stick body with a three-section telescopic pull rod and a voice module. The stick body is equipped with four cameras (including the front lower part, the front upper part and the left and right sides), a ground contact device, a main box, two buttons, a handle and a bracket. The main box is divided into two parts, one part is embedded with a mobile power supply, and the other part is encapsulated with a microcontroller and a GPS module.

8. The intelligent blind-guiding device based on machine vision according to claim 7, It is characterized in that Among the buttons, one is a power-on button for the device, and the other is a trigger button for the user to send a location.

9. The intelligent blind-guiding device based on machine vision according to claim 7, It is characterized in that The ground contact device is composed of a fixed wheel and a universal wheel, and a bracket is arranged on the upper surface of the device to fix the angle between the stick body and the horizontal plane.

10. The intelligent blind-guiding device based on machine vision according to claim 7, It is characterized in that The voice module is connected to the RPI 4B via Bluetooth to realize voice input and voice broadcast.

Images