A blind guiding method and system based on real-time interaction and scene recognition

Abstract

The application provides a blind guiding method and system based on real-time interaction and scene recognition, divides a route into path points and path lines, takes an entry point of a path line as a navigation focus, obtains scene information through a camera arranged on glasses, detects a current scene through software, interacts with local software through a button arranged on a blind stick when needed, and is silent for detection in most time, does not interfere with self detection and judgment of a user on a road, and has the advantages of simple structure, convenient use and low cost.

Description

Technical Field

[0001] This invention relates to the field of computer applications, specifically a guide method and system for the visually impaired based on real-time interaction and scene recognition. Background Technology

[0002] Traditional white canes are the primary means of perception for visually impaired people when getting around. By using the cane to collide with the ground and surrounding objects, they can perceive obstacles ahead and thus develop an understanding of the current environment and the path ahead. However, in crowded or complex environments, it is easy to lose one's sense of direction due to open spaces and similar paths.

[0003] Currently, intelligent guide systems for the visually impaired, based on information technology, utilize technologies including GPS positioning, navigation, image recognition, and voice assistance. These systems can initially replace the functions of traditional white canes, providing visually impaired individuals with a wider range of travel options.

[0004] Most current intelligent guidance systems can perform functions such as obstacle detection, zebra crossing detection, path planning, and route indication. However, their frequent prompts may interfere with visually impaired people's judgment of the current scene. Furthermore, when faced with complex scenes and losing their sense of direction, they cannot provide visually impaired people with specific information about the current scene, thus failing to help them rebuild their sense of direction.

[0005] To address the aforementioned issues, this invention enables real-time interaction with the device through button and voice input, providing visually impaired individuals with information such as current location, future route, and surrounding conditions when needed, thereby assisting them in rebuilding their sense of direction and finding their way. Summary of the Invention

[0006] This invention provides a guide method for the visually impaired based on real-time interaction and scene recognition, comprising: (1) Use electronic maps to plan walking routes to the destination, obtain detailed routes, and classify routes into two types: waypoints and path lines; (2) After the route planning is completed, the detailed route information is delivered to the user via voice broadcast; (3) Users use satellite navigation positioning function to locate in real time and move forward according to the planned detailed route, and use a white cane with button and vibration function to interact with the software of this method; (4) When walking on a route of type path line, automatically detect the scene ahead, obstacles ahead, pedestrian crossing, tactile paving, turn ahead, current deviation, and current direction, and provide prompts according to the set prompt level and current danger level; (5) When walking on a route of type waypoint, the system automatically detects the path entry point ahead, the surrounding scene, the surrounding crowd, the surrounding obstacles, and the current direction. It indicates the current situation, the direction and distance to the path entry point ahead, and provides prompts according to the set prompt level and the current danger level. (6) Through the above detection and prompts, visually impaired people can increase their sense of direction and awareness of the surrounding situation while walking; and walk more conveniently on the planned path. (7) When the user needs to interact, he / she can ask the software through the buttons with different functions to query all the information detected by the software. The software answers the user's questions through voice, vibration and other means, thereby completing the walking navigation on the planned path.

[0007] The method uses electronic maps to plan walking routes to a destination, obtains detailed routes, and classifies routes into two types: waypoints and path lines, including: A path point is a bounded closed region that can be approximated by a polygon or a circle. A path line is a closed region with width, which can be represented by a line segment with width. All waypoints and waylines are connected to form the complete path; A path point or the connecting part of a path line that leads to the next path line is called a path entry point.

[0008] The white cane includes: The top of the white cane has physical buttons, including multiple buttons with different functions, and a vibration feedback device; the top of the white cane is also equipped with a vibration device that can vibrate to prompt the user. The white cane can also be set with a one-button alarm function.

[0009] After route planning is completed, detailed route information is delivered to the user via voice broadcast, including: Users receive voice broadcasts and interactive prompts using headphones; The interactive prompt information is the response information given by the software when the user asks the software by pressing a button; Before or during walking, users can obtain route planning information, as well as their current location, the path they have already traversed, and information about the next path, at any time by pressing a button.

[0010] When walking on a path, the system automatically detects the scene ahead, obstacles ahead, pedestrian crossings, tactile paving for the visually impaired, a turn ahead, the current veergence, and the current direction, including: Automatically detect the scene in front and store the detection results; Automatically detects the flow of people ahead and stores the results. Automatically detects obstacles ahead and stores the detection results; Automatically detect pedestrian crossings and store the detection results; Automatically detects tactile paving for the visually impaired and stores the detection results; Automatically detect situations where a turn is needed ahead in the planned route and store the results of the upcoming turn; Automatically detects whether the current position is off course and stores the current off course result; It automatically detects the current direction of travel while moving and stores the current direction result; When stopped, the current direction is automatically detected and the result is stored.

[0011] When walking on a route of type waypoint, the system automatically detects the upcoming path entry point, surrounding scenery, pedestrian flow, surrounding obstacles, and current direction. When needed, it indicates the current situation, the direction of travel to the upcoming path entry point, and the distance, including: Automatically detect and store the entry points of the path ahead; Automatically detect the current scene; store the detection results of the current scene; Automatically detect the current flow of people and store the detection results; Automatically detects current obstacles and stores the detection results; Automatically detect the current direction and store the detection results; If you rotate around in the current position, you can complete the detection of the current surrounding scene, the surrounding crowd, and the surrounding obstacles; The above detection allows users to perceive the surrounding situation; When needed by the user, it indicates the current situation, the direction of travel to the entry point of the path ahead, and the distance.

[0012] The provision of prompts based on the set prompt level and the current level of danger includes: Automatic mode: Provides voice and vibration alerts when the detected situation is dangerous; Manual mode: The stored test results will be read aloud via voice when the button is pressed; Different prompting modes can be set for different detection content.

[0013] This invention also provides a guide system for the visually impaired based on real-time interaction and scene recognition, comprising: Camera mounted on glasses: with battery and wireless network, it can transmit the current image to a smartphone at the back end; Wireless headphones: equipped with batteries and Bluetooth, they can transmit sound information emitted by a smartphone to the user's ears; A white cane with buttons and vibration: It has a battery and wireless network, and can interact with the software deployed on the smartphone by using the buttons on the cane according to the vibration command of the smartphone. Smartphone: equipped with satellite navigation and positioning functions, wireless hotspot connection to camera, wireless headphones and white cane, and software implementing the method described in this invention deployed on it; Software for this method: Software that implements the method of this invention, deployed on a smartphone.

[0014] This invention achieves positioning through a satellite positioning and navigation system, and route planning and navigation through an electronic map. By detecting the scene in real time during the journey, and interacting with the local system in real time through buttons when needed, the user can gain a sense of the scene in front of or around them, thus enhancing the function of the white cane. This invention does not require backend computing support during use, nor does it require internet support when using offline map navigation. It has the advantages of simple structure and ease of use. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a guide method for the visually impaired based on real-time interaction and scene recognition, as described in this invention.

[0016] Figure 2 This is a schematic diagram of a typical planned path that includes the entire path in this invention.

[0017] Figure 3 This is a schematic diagram of a guide system for the visually impaired based on real-time interaction and scene recognition, as described in this invention. Detailed Implementation

[0018] To more clearly demonstrate the objectives and technical solutions of this invention, the specific implementation methods and parameters of this invention will be described in more detail below, in conjunction with specific implementation processes and accompanying drawings.

[0019] Example 1: A guide method for the visually impaired based on real-time interaction and scene recognition, such as... Figure 1 As shown, it includes the following steps.

[0020] Step 1: Use an electronic map to plan a walking route to the destination, obtain a detailed planned route, and classify the route into two types: waypoints and route lines, including: A path point is a bounded closed region, which can be approximated by a polygon or a circle; A path line is a closed region with width, which can be represented by a line segment with width. All waypoints and waylines are connected to form the complete path; The part of a path point or path line that connects to the next path point or path line is called the path entry point.

[0021] A typical planned path that includes the entire route is as follows: Figure 2 As shown, the route consists of three path lines and two waypoints; the areas represented by the path lines and the areas represented by the waypoints must overlap, as shown by the four gray areas marked in the figure.

[0022] like Figure 2 As shown, gray areas 2 and 4 are the path entry points, the points from which one enters the path. When a blind person loses their sense of direction in the area where a path point is located, their primary task is to find that point on the next path.

[0023] Step 2: After the route is planned, the route information is delivered to the user via voice announcement, including: Users receive voice broadcasts and interactive prompts using headphones; The interactive prompt information is the response information given by the software when the user asks the software by pressing a button; Before or during walking, users can obtain route planning information, as well as their current location, the path they have already traversed, and information about the next path, at any time by pressing a button.

[0024] Step 3: The user uses the satellite navigation positioning function to locate themselves in real time and proceeds according to the planned detailed route.

[0025] Step 4: When walking on a route of type path line, automatically detect the scene ahead, the flow of people ahead, obstacles ahead, pedestrian crossings, tactile paving for the visually impaired, the turn ahead, the current deviation, and the current direction, and provide prompts according to the set prompt level.

[0026] Step 5: When walking on a route with a path point type, automatically detect the path entry point ahead, the surrounding scene, the surrounding crowd, the surrounding obstacles, and the current direction. When the user needs it, indicate the current situation, the direction of travel to the path entry point ahead, and the distance. The stored detection results can be used to provide prompts based on the user's set prompt level, such as indicating the current situation, the direction and distance to the entrance point of the path ahead when needed; the detection allows the user to perceive the surrounding situation.

[0027] The current level of danger refers to classifying the severity of the above test results into two levels: dangerous and normal.

[0028] The provision of prompts based on the set prompt level and the current level of danger refers to dividing the prompts into two prompt modes: Automatic mode: Provides voice and vibration alerts when the detected situation is dangerous; Manual mode: The stored test results will be read aloud via voice when the button is pressed; Different prompting modes can be set for different detection content.

[0029] Step 6: During the journey, the above-mentioned detection and prompts can enhance the visually impaired person's sense of direction and make it easier for them to walk on the planned path.

[0030] Step 7: When the user needs to interact, they can ask the software for information by setting different function buttons. The software will answer the user's questions through voice, vibration, etc., thereby completing the walking navigation on the planned route.

[0031] Example 2: A guide system for the visually impaired based on real-time interaction and scene recognition, such as... Figure 3 As shown, it includes: Device S1: A camera mounted on the glasses, with a battery and wireless network, which can transmit the current image to a smartphone at the back end; Device S2: Wireless earphones with battery and Bluetooth, which can transmit sound information emitted by the smartphone to the user's ears; Device S3: A white cane with buttons and a vibration device, equipped with a battery and wireless network. It can vibrate according to the vibration command of a smartphone, and can interact with the phone using the buttons on the cane. Device S4: A smartphone with satellite navigation and positioning, and wireless hotspot connectivity to other devices; Software S5: The software of this method deployed on a smartphone, which implements all the functions of Example 1; Devices S1 and S3 are connected to device S4 via a wireless network, and device S2 is connected to device S4 via Bluetooth.

[0032] A guide system for the visually impaired based on real-time interaction and scene recognition further includes the following steps: Step 1: Software S5 uses the satellite navigation positioning function and electronic map of a smartphone to realize route planning and obtain the planned route.

[0033] Step 2: Software S5 transmits the route information to the user via device S2 through voice broadcast.

[0034] Step 3: The user uses the satellite navigation positioning function to locate in real time and moves forward according to the route given by the software S5.

[0035] Step 4: When the user walks on a route that is a path, the software S5 uses the camera of the device S1 to obtain real-time scene information in front of them.

[0036] The S5 software automatically detects the scene ahead and stores the detection results. The scene detection results include six categories: car lane, sidewalk, pedestrian crossing, simple intersection, complex intersection, and square.

[0037] The S5 software automatically detects the flow of people ahead and stores the results. The detection results include three categories: no people, few people, many people, and crowded.

[0038] The S5 software automatically detects obstacles ahead and stores the detection results, which include three categories: no obstacles, small obstacles, and large obstacles.

[0039] The S5 software automatically detects pedestrian crossings and stores the detection results. The detection results include two categories: the pedestrian crossing ahead is for crossing the road, and the pedestrian crossing ahead is not for crossing the road.

[0040] The S5 software automatically detects tactile paving and stores the detection results, which include two categories: tactile paving ahead and no tactile paving ahead.

[0041] The S5 software automatically detects situations where a turn is needed in the planned route and stores the turn results. The detection results include five categories: straight ahead, slight left turn ahead, large left turn ahead, slight right turn ahead, and large right turn ahead.

[0042] The S5 software automatically detects whether the current position is off course and stores the current off course result. The detection result includes three categories: currently normal, currently slightly off course, and currently severely off course.

[0043] The S5 software automatically detects the current direction of travel during movement and stores the current direction result. The detection result includes eight directions: east, south, west, north, southeast, northeast, southwest, and northwest.

[0044] When software S5 stops, it automatically detects the current direction and stores the current direction result. The detection result includes eight directions: east, south, west, north, southeast, northeast, southwest, and northwest.

[0045] The stored test results can be displayed according to the prompt level set by the user.

[0046] Step 5: When walking on a route that is a waypoint, use the camera of device S1 to obtain information about the surrounding scene. Software S5 automatically detects the entry point of the path ahead and stores the entry point information. The detection results include: the angle to the left and to the left ahead, and the distance from the current position; the angle to the right and to the right ahead, and the distance from the current position. The S5 software automatically detects the current scene and stores the detection results. When it rotates around the current position, it obtains information about the surrounding scene. The S5 software automatically detects the current flow of people and stores the detection results. When it rotates around the current position, it obtains information about the surrounding flow of people. The S5 software automatically detects current obstacles and stores the detection results. When it rotates around the current position, it obtains information about surrounding obstacles. Software S5 automatically detects the current direction and stores the detection results. Software S5 thus enables the perception of the surrounding situation; The detection results stored in software S5 can be used to provide prompts based on the prompt level set by the user, such as indicating the current situation, the direction of travel to the entrance point of the path ahead, and the distance when needed by the user.

[0047] Step 6: During the journey, the above-mentioned detection and prompts can enhance the visually impaired person's sense of direction, making it easier for them to walk on the planned path until they reach the end of the route.

[0048] Step 7: When the user needs to interact, they can ask the system for information by pressing the buttons with different functions pre-set on device S3. The system will respond to the user's questions by issuing voice signals through device S2 and vibration signals through device S3, thereby completing the walking navigation on the planned route.

[0049] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit it. Although the foregoing embodiments have described the present invention in detail, those skilled in the art should understand that modifications to the technical solutions described in the foregoing embodiments, equivalent substitutions for some or all of the technical features, or modifications to the order of the steps can be made; however, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention. The values ​​of various thresholds and ranges in the present invention may also change due to different specific circumstances.

Claims

1. A guide method for the visually impaired based on real-time interaction and scene recognition, characterized in that, include: (1) Use electronic maps to plan walking routes to the destination, obtain detailed routes, and classify routes into two types: waypoints and path lines; A path point is a bounded closed region, approximated by a polygon or a circle; a path line is a closed region with width, represented by a line segment with width. All path points and path lines connected together form a complete path; the connecting part of a path point or path line that leads to the next path line is called the path entry point. (2) After the route planning is completed, the detailed route information is delivered to the user via voice broadcast; the user uses headphones to receive the voice broadcast information and interactive prompts. The interactive prompts are the responses the software provides when the user asks questions via buttons. Before or during walking, the user can obtain path planning information, as well as their current location, the path already traversed, and subsequent path information at any time via buttons. (3) Users use satellite navigation positioning function to locate in real time and move forward according to the planned detailed route; use a white cane with button and vibration function to interact with the software; physical buttons are arranged on the top of the white cane, including multiple buttons with different functions and vibration feedback device; the top of the white cane is also equipped with a vibration device that can vibrate to alert the user; the white cane is also equipped with a one-button alarm function. (4) When walking on a route of type path line, automatically detect the scene ahead, obstacles ahead, pedestrian crossings, tactile paving, turns ahead, current deviation, and current direction, store the detection results, and provide prompts according to the set prompt level and the current danger level; (5) When walking on a route of type waypoint, automatically detect the path entry point ahead, the surrounding scene, the surrounding crowd, the surrounding obstacles, and the current direction, store the detection results, indicate the current situation, the direction and distance to the path entry point ahead, and provide prompts according to the set prompt level and the current danger level. (6) The above detection and prompts can increase the sense of direction of visually impaired people while walking, as well as their perception of the surrounding situation; It makes walking on the planned route easier; (7) When the user needs to interact, he / she asks the software through the buttons with different functions set to query all the information detected by the software. The software answers the user's questions through voice, vibration and other means, thereby completing the walking navigation on the planned route.

2. The method based on claim 1, characterized in that, Provide alerts based on the set alert level and the current level of danger, including: Automatic mode: Provides voice and vibration alerts when the detected situation is dangerous; Manual mode: The stored test results will be read aloud via voice when the button is pressed; Different prompting modes can be set for different detection content.

3. A guide system for the visually impaired based on real-time interaction and scene recognition, characterized in that, include: Camera mounted on glasses: with battery and wireless network, capable of transmitting current images to a smartphone at the back end; Wireless headphones: equipped with batteries and Bluetooth, capable of transmitting sound information emitted by a smartphone to the user's ears; White cane with buttons and vibration device: It has a battery and wireless network, and can vibrate the cane according to the vibration command of the smartphone. It can also use the buttons on the cane to interact with the software deployed on the smartphone. Smartphone: Equipped with satellite navigation and positioning functions, and with wireless hotspot connectivity for camera, wireless earphones, and a white cane, and software deployed on it to implement the guide method for the blind based on real-time interaction and scene recognition as described in claim 1.

Images