A blind person auxiliary navigation device and method based on mixed reality

Abstract

The present invention relates to the field of navigation technology for the blind, and specifically to a blind auxiliary navigation device and method based on mixed reality, comprising a glasses-shaped main frame for mixed reality, the main frame having a built-in position and attitude sensor for indoor detection of its own spatial data information and a GNSS positioning chip for outdoor navigation, a microwave radio frequency identification device for indoor electronic tag identification information interaction and an image recognition camera system for obstacle image information acquisition and processing, and a binocular camera device for building a virtual three-dimensional space model on the left and right sides of the front of the main frame, the binocular camera device being connected to a cloud data computing center. The present invention efficiently matches and optimizes navigation scenes and navigation methods by adopting mixed reality technology and an intelligent head-mounted obstacle avoidance device.

Description

Technical Field

[0001] The present invention relates to the technical field of blind navigation, and in particular to a blind auxiliary navigation device and method based on mixed reality. Background Art

[0002] Existing navigation technologies for the blind use various sensors, such as laser rangefinders, cameras, and infrared cameras, to obtain three-dimensional information about the surrounding environment, and feed back some three-dimensional information about the surrounding environment and obstacles to the blind in the form of voice, giving them some auxiliary prompts to reduce the risk of injury caused by collisions between the blind and objects. However, since the blind cannot obtain their accurate position information through these devices, they cannot clearly know how to walk next to avoid colliding with obstacles. Although some technical solutions use multimodal data collected by GNSS and cameras, and process the collected data through a small processor to realize the function of providing location information to the blind, they cannot provide the surrounding obstacle information to the blind at the same time. Therefore, the acquired position information alone cannot allow the blind to clearly know how to walk next to avoid colliding with obstacles. Therefore, there is an urgent need for a blind auxiliary navigation device and method based on mixed reality. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a blind person auxiliary navigation device and method based on mixed reality in view of the deficiencies in the prior art.

[0004] The technical problem to be solved by the present invention is achieved through the following technical solutions. The present invention discloses a method for assisting navigation for the blind based on mixed reality, including a system that autonomously judges and selects a free obstacle avoidance mode and a fixed route mode according to a navigation position. (1) The free obstacle avoidance mode is used to meet the requirements of accurate assistive navigation and navigation tracking modeling of unfamiliar routes under free activity navigation, including a familiar indoor sub-mode and a familiar outdoor sub-mode. In the familiar indoor sub-mode, the system pre-models a virtual indoor space for a familiar indoor scene, and sets electronic tags at the main indoor obstacle space and space turning points and marks them correspondingly to the virtual space. When an obstacle recognition device points to the electronic tag, a navigation prompt device issues a location distance information prompt to achieve free obstacle avoidance. In the familiar outdoor sub-mode, the system pre-models a virtual road space for a familiar outdoor scene and connects to a GNSS navigation positioning system. The obstacle recognition device of the system realizes free obstacle avoidance by recognizing terrain such as the ground, sidewalks, stairs, water surface, and targets such as pedestrians and vehicles, and superimposes the recognition information into the virtual road space; (2) a fixed route mode, which is used to meet the precise navigation requirements under the fixed activity navigation, including an indoor route sub-mode and an outdoor route sub-mode. In the indoor route sub-mode, the system performs precise navigation according to the latest virtual space mapping in the real space state under the navigation instructions based on the navigation data in the familiar indoor sub-mode, and provides continuous navigation route prompts through the virtual space data; in the outdoor route sub-mode, the system performs precise navigation according to the latest virtual space mapping in the real space state under the navigation instructions based on the navigation data in the familiar outdoor sub-mode, and simultaneously turns on the infrared distance sensor device for auxiliary warning, and provides continuous navigation route prompts through the virtual space data.

[0005] Furthermore, the indoor familiarity sub-mode includes a process in which the system can autonomously control the virtual modeling update function according to an input command and update the virtual space modeling data according to the obstacle information of this free obstacle avoidance navigation process.

[0006] Furthermore, the free obstacle avoidance mode includes an unfamiliar indoor sub-mode and an unfamiliar outdoor sub-mode. In the unfamiliar indoor sub-mode, the system can identify objects and make judgments through the infrared distance sensor device and the binocular camera device and give prompts through the navigation prompt device. In the unfamiliar outdoor scene, the system can navigate and locate through GNSS and directly assist in obstacle avoidance through the obstacle recognition device.

[0007] Furthermore, both the unfamiliar indoor sub-mode and the unfamiliar outdoor sub-mode include a process in which the system automatically records the route during navigation and initiates a tracking modeling function at the autonomous choice of the blind person to generate a navigation path.

[0008] The present invention also discloses a blind person auxiliary navigation device based on mixed reality, comprising a glasses-shaped main frame for mixed reality, the main frame having a built-in position attitude sensor for indoor detection of its own spatial data information and a GNSS positioning chip for outdoor navigation, a microwave radio frequency identification device for indoor electronic tag identification information interaction and an image recognition camera system for obstacle image information acquisition and processing provided at the front center of the main frame, binocular camera devices for building a virtual three-dimensional space model provided on the left and right sides of the front of the main frame, the binocular camera devices connected to a cloud data computing center, a plurality of groups of infrared distance sensors symmetrically provided on the outside of the main frame, the microwave radio frequency identification device, the image recognition camera system, the binocular camera devices and the infrared distance sensors forming an obstacle recognition device, side clamps provided on the left and right sides of the main frame, a load-bearing belt for upper head wear connected to the upper side clamps, the side clamps on the left side are rotatably connected to a locking arm through a rotating shaft, the side clamps on the right side are connected to the locking arm through a locking pin insertion, the side clamps on the left and right sides are both connected to a rear frame plate through the locking arm, and the rear frame plate is provided with a battery and a wireless charging device.

[0009] Furthermore, the side clamp is provided with a bone conduction earphone and a voice command conversion module, the inner side of the side clamp is provided with a tactile prompt device for providing prompts through pressing and vibration, and the inner side of the front part of the main frame is provided with a thermal prompt device for providing continuous signal prompts by sensing changes in heat shielding around the eyes.

[0010] Furthermore, the bone conduction earphone, the tactile prompt device and the thermal prompt device constitute a navigation prompt device, the bone conduction earphone is configured as a main prompt device, the tactile prompt device can assist the main prompt device in the enhanced prompt process of outdoor navigation and indoor emergency avoidance, and the thermal prompt device assists the main prompt device in the continuous signal prompt process of outdoor blind road sections.

[0011] Furthermore, the wireless charging device includes a hanging vertical platform with a pull rope, and the hanging platform is used to help the blind to achieve a sight-free wireless charging positioning process after the locking pin is removed.

[0012] Compared with the prior art, the present invention has the following advantages:

[0013] (1) The present invention uses mixed reality technology and an intelligent head-mounted obstacle avoidance device to efficiently match and optimize navigation scenes and navigation methods, so that the device can perform targeted navigation autonomously or at the choice of the blind, thereby improving navigation efficiency and accuracy and avoiding a large amount of information processing operations;

[0014] (2) The present invention combines automatic navigation or autonomous walking with the obstacle avoidance function at any time, so as to cope with unexpected situations, avoid navigation errors caused by environmental changes, and update the data for the next navigation by adjusting local data;

[0015] (3) The present invention adopts a prompt device that can provide prompts through multi-sensory methods, use mixed reality technology to integrate virtual and real and build a deep learning environment for navigation data, and can simultaneously provide a rich variety of navigation and obstacle avoidance information. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the principles between various navigation modes in the navigation method of the present invention;

[0017] Figure 2 It is an overall structural diagram of the head-mounted auxiliary navigation device of the present invention. DETAILED DESCRIPTION

[0018] like Figure 1-2 As shown, the present invention discloses a blind person auxiliary navigation device based on mixed reality, including a glasses-shaped main frame 1 for mixed reality, the main frame 1 is equipped with a position attitude sensor for indoor detection of its own spatial data information and a GNSS positioning chip for outdoor navigation, a microwave radio frequency identification device a for indoor electronic tag identification information interaction and an image recognition camera system for obstacle image information acquisition and processing are arranged at the front center of the main frame 1, a binocular camera device b for building a virtual three-dimensional space model is arranged on the left and right sides of the front of the main frame 1, the binocular camera device b is connected to a cloud data computing center, and a plurality of groups of infrared distance sensors are symmetrically arranged on the outside of the main frame 1 The microwave radio frequency identification device a, the image recognition camera system, the binocular camera device b and the infrared distance sensor constitute an obstacle recognition device. The left and right sides of the main frame 1 are provided with side clamps 11, and the upper part of the side clamps 11 is connected with a load-bearing belt 2 for the upper headwear. The left side clamp 11 is rotatably connected with a locking arm 3 through a rotating shaft, and the right side clamp 11 is plug-connected with the locking arm 3 through a locking pin 5. The left and right side clamps 11 are connected with a rear frame plate 4 through the locking arm 3. The rear frame plate 4 is provided with a battery and a wireless charging device. The wireless charging device includes a hanging vertical platform with a pull rope. The hanging platform is used to help the blind to achieve the sight-free wireless charging positioning process after the locking pin 5 is removed.

[0019] The side clamp 11 is provided with a bone conduction headset and a voice command conversion module. A tactile prompt device d for prompting by pressing and vibrating is provided on the inner side of the side clamp 11. A thermal prompt device c for continuous signal prompting by sensing changes in heat shielding around the eyes is provided on the inner side of the front of the main frame 1. The bone conduction headset, the tactile prompt device d and the thermal prompt device c constitute a navigation prompt device. The tactile prompt device d can assist the main prompt device in the enhanced prompt process of outdoor navigation and indoor emergency avoidance. The thermal prompt device c assists the main prompt device in the continuous signal prompt process of outdoor blind road sections.

[0020] In order to further illustrate the principle and spirit of the present invention, the present invention also discloses a blind person assisted navigation method based on mixed reality, including the system autonomously judging and selecting a free obstacle avoidance mode and a fixed route mode according to the navigation position,

[0021] (1) Free obstacle avoidance mode, which is used to meet the requirements of accurate auxiliary navigation and navigation tracking modeling of unfamiliar routes under the requirements of free activity navigation, including familiar indoor sub-mode and familiar outdoor sub-mode. In the familiar indoor sub-mode, the system pre-models the virtual indoor space of the familiar indoor scene, and sets electronic tags at the main indoor obstacle space and space turning points and marks them correspondingly in the virtual space. When the obstacle recognition device points to the electronic tag, the navigation prompt device issues a position distance information prompt to achieve free obstacle avoidance. In addition, the familiar indoor sub-mode includes a process in which the system can autonomously control the opening of the virtual modeling update function according to the input command and update the virtual space modeling data according to the obstacle information of this free obstacle avoidance navigation process; in the familiar outdoor sub-mode, the system pre-models the virtual road space of the familiar outdoor scene and connects it to the GNSS navigation and positioning system. The obstacle recognition device realizes free obstacle avoidance by recognizing terrain such as the ground, sidewalks, stairs, water surface, and targets such as pedestrians and vehicles, and superimposes the recognition information into the virtual road space; (2) Fixed route mode, which is used to meet the precise navigation requirements under fixed activity navigation, including indoor route sub-mode and outdoor route sub-mode. In the indoor route sub-mode, the system performs precise navigation according to the mapping of the latest virtual space to the real space state under navigation instructions based on the navigation data in the familiar indoor sub-mode, and provides continuous navigation route prompts through virtual space data; in the outdoor route sub-mode, the system performs precise navigation according to the mapping of the latest virtual space to the real space state under navigation instructions based on the navigation data in the familiar outdoor sub-mode, and simultaneously turns on the infrared distance sensor device for auxiliary warning, and provides continuous navigation route prompts through virtual space data.

[0022] In order to fully utilize the navigation needs of multiple scenarios, the above-mentioned free obstacle avoidance mode also includes an unfamiliar indoor sub-mode and an unfamiliar outdoor sub-mode. In the unfamiliar indoor sub-mode, the system can identify objects and make judgments through the infrared distance sensor device and the binocular camera device b and prompt through the navigation prompt device. In the unfamiliar outdoor scene, the system can navigate and locate through GNSS, and directly assist in obstacle avoidance through the obstacle recognition device. Specifically, in order to meet the navigation needs of the blind for unfamiliar routes and gradually construct preliminary data for unfamiliar routes, the unfamiliar indoor sub-mode and the unfamiliar outdoor sub-mode both include the process of the system automatically recording the route during navigation and starting the tracking modeling function under the autonomous selection of the blind to generate a navigation path.

[0023] Therefore, according to the above scheme, the blind can trigger or independently select the navigation mode according to the system positioning, and realize efficient obstacle avoidance navigation in a variety of different scenarios such as familiar and unfamiliar indoor and outdoor. It not only reduces the computing power consumption of the equipment, but also can realize regular data updates and automatic coverage of navigation data, and provide a variety of auxiliary prompts in emergency situations, together creating a nearly visible mixed reality physical world for the blind. Its environmental perception method is friendly and the system is safe and easy to use, and has wide popular application value.

[0024] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A blind person assisted navigation method based on mixed reality, characterized in that: Including the system independently judging and selecting free obstacle avoidance mode and fixed route mode according to the navigation position, (1) Free obstacle avoidance mode, which is used to meet the requirements of accurate auxiliary navigation and navigation tracking modeling of unfamiliar routes under the requirements of free activity navigation, including familiar indoor sub-mode and familiar outdoor sub-mode. In the familiar indoor sub-mode, the system pre-models the virtual indoor space of the familiar indoor scene, and sets electronic tags at the main indoor obstacle space and space turning points and marks them in the virtual space accordingly. When the obstacle recognition device points to the electronic tag, the navigation prompt device issues a position distance information prompt to achieve free obstacle avoidance; In the familiar outdoor sub-mode, the system pre-models a virtual road space for the familiar outdoor scene and connects it to the GNSS navigation and positioning system. The obstacle recognition device can achieve free obstacle avoidance by identifying the ground, sidewalks, stairs, water surface terrain, pedestrians, and vehicle targets, and superimposes the recognition information into the virtual road space. (2) Fixed route mode, which is used to meet the precise navigation requirements under fixed activity navigation, including indoor route sub-mode and outdoor route sub-mode. In the indoor route sub-mode, the system performs precise navigation according to the mapping of the latest virtual space state to the real space state under the navigation instructions based on the navigation data in the familiar indoor sub-mode, and provides continuous navigation route prompts through the virtual space data; In the outdoor route sub-mode, the system performs precise navigation according to the mapping of the latest virtual space state in the real space based on the navigation data in the familiar outdoor sub-mode under navigation instructions, while turning on the infrared distance sensor device for auxiliary warning, and providing continuous navigation route prompts through virtual space data.

2. The method for assisting blind navigation based on mixed reality according to claim 1, characterized in that: The indoor familiarity sub-mode includes a process in which the system can autonomously control the virtual modeling update function according to an input command and update the virtual space modeling data according to the obstacle information of this free obstacle avoidance navigation process.

3. The method for assisting blind navigation based on mixed reality according to claim 1, characterized in that: The free obstacle avoidance mode includes an unfamiliar indoor sub-mode and an unfamiliar outdoor sub-mode. In the unfamiliar indoor sub-mode, the system can identify objects and make judgments through an infrared distance sensor device and a binocular camera device and provide prompts through a navigation prompt device. In an unfamiliar outdoor scene, the system can navigate and locate through GNSS and directly assist in obstacle avoidance through an obstacle recognition device.

4. The method for assisting blind navigation based on mixed reality according to claim 3, characterized in that: The unfamiliar indoor sub-mode and the unfamiliar outdoor sub-mode both include a process in which the system automatically records the route during navigation and starts a tracking modeling function at the autonomous choice of the blind person to generate a navigation path.

5. A blind person's auxiliary navigation device based on mixed reality, characterized in that: The invention comprises a glasses-shaped main frame for mixed reality, wherein the main frame is equipped with a position attitude sensor for indoor detection of its own spatial data information and a GNSS positioning chip for outdoor navigation, a microwave radio frequency identification device for indoor electronic tag identification information interaction and an image recognition camera system for obstacle image information acquisition and processing are arranged at the front center of the main frame, a binocular camera device for building a virtual three-dimensional space model is arranged on the left and right sides of the front of the main frame, the binocular camera device is connected to a cloud data computing center, a plurality of groups of infrared distance sensors are symmetrically arranged on the outside of the main frame, the microwave radio frequency identification device, the image recognition camera system, the binocular camera device and the infrared distance sensor constitute an obstacle recognition device, side clamps are arranged on the left and right sides of the main frame, a load-bearing belt for upper head wear is connected to the upper part of the side clamps, the side clamps on the left side are rotatably connected to a locking arm through a rotating shaft, the side clamps on the right side are connected to the locking arm through a locking pin insertion, the side clamps on the left and right sides are connected to a rear frame plate through a locking arm, and the rear frame plate is provided with a battery and a wireless charging device.

6. The blind person's auxiliary navigation device based on mixed reality according to claim 5, characterized in that: The side clamp is provided with a bone conduction earphone and a voice command conversion module, the inner side of the side clamp is provided with a tactile prompt device for providing prompts through pressing and vibration, and the inner side of the front part of the main frame is provided with a thermal prompt device for providing continuous signal prompts by sensing changes in heat shielding around the eyes.

7. The blind person's auxiliary navigation device based on mixed reality according to claim 6, characterized in that: The bone conduction earphone, tactile prompt device and thermal prompt device constitute a navigation prompt device. The bone conduction earphone is configured as a main prompt device. The tactile prompt device can assist the main prompt device in the enhanced prompt process of outdoor navigation and indoor emergency avoidance. The thermal prompt device assists the main prompt device in the continuous signal prompt process of outdoor blind road sections.

8. The blind person's auxiliary navigation device based on mixed reality according to claim 5, characterized in that: The wireless charging device comprises a hanging vertical platform with a pull rope, and the hanging vertical platform is used to help the blind to achieve the sight-free wireless charging positioning process after the locking pin is removed.

Images