Blind guiding stick and blind guiding method

Abstract

Provided is a blind guide stick comprising: a data communication device for acquiring a real-time map; the plurality of sensors are used for acquiring data of a current environment and positioning data of the blind guiding stick; the controller is used for generating guidance information based on the real-time map, the positioning data and the data of the current environment; and a user interface device for guiding the user based on the guidance information. Therefore, more robust navigation can be provided in various scenes.

Description

Technical Field

[0001] The present invention relates to the field of blind guiding, and in particular to the field of intelligent blind guiding. Background Art

[0002] For people with visual impairments, difficulty in traveling is an ever-present problem. In addition to guide dogs, smart guide devices have been developed to help people with visual impairments travel smoothly.

[0003] However, current intelligent guide devices are not suitable for all scenarios. For example, when taking a bus at a bus station, current intelligent guide devices are difficult to help visually impaired users identify and take the correct bus. Especially when there are multiple buses stopped at the bus station, users are likely to take the wrong bus or waste time because they do not know which bus to take.

[0004] For another example, when it is necessary to cross a crosswalk, current intelligent guide devices are difficult to help users cross the crosswalk at the right time and in the right place. Summary of the invention

[0005] It is desirable to provide an improved guide stick and guidance method that can provide more robust navigation in various scenarios, in particular, can help users identify and take the correct means of transportation, and can help users cross pedestrian crossings at the correct time and location.

[0006] According to one aspect, a guide stick for the blind is provided, comprising: a data communication device for acquiring a real-time map; a plurality of sensors for acquiring data of a current environment and positioning data of the guide stick; a controller for generating guidance information based on the real-time map, the positioning data, and the data of the current environment; and a user interface device for guiding the user based on the guidance information.

[0007] According to another aspect, a method for guiding the blind using a guide stick is provided, comprising: acquiring a real-time map via a data communication device; acquiring data of a current environment and positioning data of the guide stick; generating guidance information based on the real-time map, the positioning data, and the data of the current environment; and guiding the user based on the guidance information.

[0008] According to another aspect, a computer-readable medium is provided, which stores a computer program code. When the computer program code is executed, a controller executes the blind guiding method according to various embodiments of the present invention.

[0009] According to various embodiments of various aspects of the present invention, real-time maps can be obtained through a network connection through a data communication device and the real-time location of the user can be obtained through a positioning sensor. Combining real-time maps, real-time positioning, and data from other sensors (including environmental data) can provide users with more robust navigation.

[0010] According to one embodiment, the multiple sensors include: a radar sensor, a laser image sensor and a positioning sensor, the radar sensor is used to obtain first point cloud data representing the user's current environment, the image sensor is used to obtain an image of the user's current environment, and the positioning sensor is used to obtain positioning data of the guide stick; the controller is also used to: perform three-dimensional reconstruction of the current environment based on the first point cloud data, obtain semantic information based on the image, and generate the guidance information based on the real-time map, the positioning data, the three-dimensional reconstruction and the semantic information.

[0011] According to a further embodiment, the first point cloud data is acquired by a laser radar sensor, the multiple sensors further include a millimeter wave radar sensor, which is used to acquire second point cloud data representing the current environment, and the predetermined area includes a waiting area, wherein the controller is further used to: if it is determined that the user is in the waiting area, detect a vehicle approaching the waiting area based on the second point cloud data; if the vehicle approaching the waiting area is detected, acquire relevant information about the vehicle based on the image; determine whether the vehicle is a target vehicle based on the relevant information about the vehicle; if the vehicle is determined to be the target vehicle, determine the parking position of the target vehicle; and generate the guidance information based on the parking position of the target vehicle. The moving vehicle is detected by the millimeter wave radar sensor, and the relevant information of the vehicle is acquired based on the image to identify the target vehicle, thereby facilitating real-time monitoring, and even in the case where there are multiple buses at a bus stop, for example, the user can be guided to take the correct bus.

[0012] According to a further embodiment, the predetermined area includes a pedestrian crossing waiting area, wherein the controller is further configured to: if it is determined that the user is in the pedestrian crossing waiting area, obtain current traffic light information based on the image; if the traffic light information indicates a red light or a yellow light, cause the user interface device to output a signal indicating waiting to the user; and if the traffic light information indicates a green light, generate the guidance information based on the pedestrian crossing information obtained from the image to guide the user to cross the pedestrian crossing. Thus, the user can be guided to cross the pedestrian crossing at the correct time and location. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Embodiments are illustrated by way of example only and not by way of limitation in the figures of the accompanying drawings in which like reference numerals refer to similar elements.

[0014] Figure 1 A structural block diagram of a guide stick for the blind according to an embodiment of the present invention is shown;

[0015] Figure 2 A view showing a guide stick for the blind according to an embodiment of the present invention;

[0016] Figure 3 A flowchart of a blind guiding method according to an embodiment of the present invention is shown;

[0017] Figure 4 A flowchart of a blind guiding method according to an embodiment of the present invention is shown;

[0018] Figure 5 A flowchart of a blind guiding method according to an embodiment of the present invention is shown;

[0019] Figure 6 A flow chart of a blind guiding method according to another embodiment of the present invention is shown.

[0020] Figure 7 A flow chart of a blind-guiding method for guiding a user to cross a crosswalk according to another embodiment of the present invention is shown.

[0021] Various aspects and features of various embodiments of the present invention are described with reference to the above drawings. The above drawings are only illustrative and not restrictive. In addition, not all parts of the device of the embodiment of the present invention are marked with reference numerals in the above drawings. Only relevant parts are marked in some drawings, which does not limit the parts to those shown in the drawings of the specification. DETAILED DESCRIPTION

[0022] Figure 1 FIG. 1 is a block diagram showing a structure of a guide stick 10 according to an embodiment of the present invention. Figure 1 As shown, the guide stick 10 includes a plurality of sensors 11, a controller 12, a data communication device 13, a user interface device 14 and a power source 15 such as a battery.

[0023] The multiple sensors 11 are used to obtain data of the user's current environment and the positioning data of the guide stick. The multiple sensors 11 may include various types of sensors capable of realizing the above functions, especially radar sensors 11_1 (especially laser radar sensors), image sensors 11_2, and positioning sensors 11_3. In addition, the multiple sensors 11 may also include inertial sensors 11_4 and millimeter wave radar sensors 11_5.

[0024] The laser radar sensor 11_1 can obtain the first point cloud data of the current environment of the guide stick user when using the guide stick, and the image sensor 11_2 can obtain the image of the user's current environment. For example, the image sensor can be a fisheye camera, etc., which is convenient for imaging a wide range of the current environment. The positioning sensor 11_3 can obtain the current positioning data of the guide stick. The inertial sensor 11_4 can be used to detect the current posture of the guide stick. The millimeter wave radar sensor 11_5 can obtain the second point cloud data of the current environment. Compared with the laser radar sensor 11_1, the point cloud data obtained by the millimeter wave radar sensor 11_5 is relatively sparse, but it can provide more robust results in bad weather, such as rainy or foggy days; it is more sensitive to the detection of moving objects (the following will be combined with the scenes of taking a bus and crossing a crosswalk to specifically describe the detection of moving objects by the millimeter wave radar sensor); due to the relatively small amount of calculation, the real-time performance is also better. Therefore, in some embodiments, the laser radar sensor and the millimeter wave radar sensor can be integrated into the guide stick at the same time to provide a more robust guidance.

[0025] The data communication device 13 may be a 5G communication device, which is capable of acquiring a real-time map, and in particular, is capable of acquiring a user-related real-time map based on the user's location data. It is contemplated that the data communication device is also capable of acquiring traffic condition information indicating the current traffic condition. For example, whether each relevant road is congested or subject to traffic control. It is also contemplated that the data communication device acquires traffic light information indicating a traffic light.

[0026] The sensing results from the multiple sensors 11 (including at least the data of the current environment and the positioning data of the guide pole) and the real-time map from the data communication device 13 can be received by the controller 12 for subsequent processing, and the controller 12 includes necessary processing functions. Alternatively, the controller 12 may include a processing unit with necessary processing functions.

[0027] In one embodiment, the controller 12 can perform a three-dimensional reconstruction of the current environment based on the first point cloud data from the laser radar sensor 11_1, thereby providing a dense image for real-time navigation of the user. The controller 12 can also obtain semantic information based on the image from the image sensor 11_2. In addition, the controller 12 also generates guidance information based on the real-time map, positioning data, three-dimensional reconstruction and semantic information.

[0028] In one embodiment, semantic information can be used to identify environmental features, thereby identifying obstacles. In one embodiment, the pose data from the inertial sensor can be used for three-dimensional reconstruction of the current environment, because the positional relationship between the laser radar sensor and the inertial sensor on the guide stick is known. In one embodiment, the second point cloud data from the millimeter wave radar sensor can be further combined on the basis of the first point cloud data from the laser radar sensor. The controller 12 generates guidance information based on both the first point cloud data and the second point cloud data to provide more robust navigation for the user.

[0029] The user interface device 14 can guide the user based on the guidance information generated by the controller 12, that is, to achieve real-time navigation. Specifically, the user interface device 14 can include an auditory guidance device 14_1, a tactile guidance device 14_2, and a motion guidance device 14_3.

[0030] Auditory Guidance Devices

[0031] Based on the guidance information generated by the controller 12, the auditory guidance device 14_1 can output auditory guidance signals, such as "go straight", "turn left" and "turn right", to navigate users with visual impairments. In one embodiment, the auditory guidance device 14_1 is a bone conduction headset. When the user wears the bone conduction headset, he can communicate with the controller 12 on the guide stick body via Bluetooth (e.g., low-power Bluetooth BLE), that is, the controller 12 can generate a control signal based on the guidance information, and control the bone conduction headset to play the corresponding voice based on the control signal via Bluetooth to navigate the user. Using bone conduction headphones can give users a sense of distance and direction.

[0032] Tactile Guidance Devices

[0033] Based on the guidance information generated by the controller 12, the tactile guidance device 14_2 can be vibrated accordingly. Specifically, the controller 12 can generate a control signal based on the guidance information to control the vibration of the tactile guidance device 14_2. In one embodiment, the guide stick 10 includes two vibration devices embedded in the handle. If the user needs to turn left according to the generated guidance information, the vibration device on the left side vibrates, and conversely, if the user needs to turn right according to the generated guidance information, the vibration device on the right side vibrates. The vibration stops until it is detected that the user has indeed turned left or right according to the guidance information.

[0034] Motion Guidance Devices

[0035] In addition to auditory and tactile guidance, the user can also be directly guided to move in a certain direction when necessary. For example, when necessary, the wheels of the guide stick are directly controlled to move in a certain direction to drag the user away from danger. In one embodiment, the motion guidance device 14_3 is an omnidirectional wheel. In an emergency, for example, an object moving rapidly toward the user (the movement speed of the object is greater than a predetermined value) is detected based on the second point cloud data, then the controller 12 can generate a control signal based on the guidance information to control the omnidirectional wheel to move in a direction away from the vehicle, thereby dragging the user away from danger.

[0036] The above-mentioned auditory guidance device 14_1, tactile guidance device 14_2 and motion guidance device 14_3 can be used in any combination. On the one hand, the guide stick 10 can include any one or more of the above-mentioned devices. On the other hand, these guidance devices can be used in any way according to different scene requirements. The following only describes two combinations and usage methods as examples, and those skilled in the art can expect that other combinations and usage methods are also possible.

[0037] Auditory guidance devices and tactile guidance devices

[0038] In this embodiment, for example, the guide stick 10 includes a bone conduction headset as a user interface device 14 and two vibration devices. In a scenario of navigating the user, the controller 12 generates a control signal based on the generated guidance information to play a voice signal through the bone conduction headset, such as "go straight", "turn left" and "turn right", so as to guide the user in real time. When the guidance information indicates that the user needs to turn left, the vibration device on the left side starts to vibrate, and if it is detected that the user has indeed turned left, the vibration stops. Conversely, when the guidance information indicates that the user needs to turn right, the vibration device on the right side starts to vibrate, and if it is detected that the user has indeed turned right, the vibration stops.

[0039] In another scenario, if the guidance information indicates that there is a step or obstacle in front of the user, in addition to the voice reminder, the two vibration devices can vibrate together, and the frequency of the vibration can increase as the user approaches the step or obstacle. In one embodiment, the guidance information can be generated based on the second point cloud data from the millimeter wave radar sensor to determine whether there is a step or obstacle.

[0040] Auditory guidance devices, tactile guidance devices and motion guidance devices

[0041] In this embodiment, for example, the guide stick 10 includes a bone conduction headset as a user interface device 14, two vibration devices, and an omnidirectional wheel. In one scenario, if the guidance information indicates that an obstacle has been identified and the user needs to turn left to avoid the obstacle, then under the control of the controller 12, the bone conduction headset can play a voice signal such as "there is an obstacle 5 meters ahead, please turn left after 4 meters"; as the user approaches the left turn position or the obstacle reaches a predetermined distance, the vibration device can be vibrated while the voice reminder is given. For example, when the user is only 1 meter away from the left turn position, the left vibration device can be vibrated while the voice signal is further played to prompt the user to turn left, or when the user is only 1 meter away from the obstacle, the two vibration devices can be vibrated while the voice signal is further played to avoid the user from hitting the obstacle. The intensity and / or frequency of the vibration can vary with the distance to the left turn position or obstacle, and the closer to the left turn position or obstacle, the higher the intensity and / or frequency.

[0042] When the user is very close to the left turn position or obstacle, that is, the distance to the left turn position or obstacle is small enough (for example, less than a predetermined threshold), the controller 12 can generate a control signal to control the omnidirectional wheel to turn left or move in a direction away from the obstacle to guide the user.

[0043] Although the above description takes the example of an obstacle in front and turning left to avoid the obstacle, this is not restrictive, and there may be an obstacle on the left or right side. In this case, the vibration device on the left or right side can vibrate to prompt the user that there is an obstacle on the left or right side.

[0044] Figure 2 FIG. 2 shows a view of a guide stick 10 according to an embodiment of the present invention. Figure 2 As shown, a positioning sensor 11_3 and a tactile guide device 14_2 are provided on the handle of the guide stick. A laser radar sensor 11_1, an image sensor 11_2 and a millimeter wave radar sensor 11_5 are provided on the side of the guide stick facing forward. An inertial sensor 11_4 is provided on the body of the guide stick. In addition, the guide stick also includes an omnidirectional wheel 14_3, a battery 15 and a controller 12. Although not shown, the guide stick also includes a data communication device 13 and / or a Bluetooth module, which can be integrated with the controller 12.

[0045] Figure 3A flow chart of a blind guiding method according to an embodiment of the present invention is shown, that is, a flow chart of a method 100 for navigating a user using a blind guide stick 10. Before actually using the blind guide stick for navigation, the user can wake up the blind guide stick by, for example, waving the blind guide stick at step 105. Specifically, the posture change of the blind guide stick can be detected by an inertial sensor. If a posture change of the blind guide stick is detected in standby mode, it means that the user has woken up the blind guide stick. It is also contemplated that other methods can be used to wake up the blind guide stick, such as by voice wake-up. If any, in this step, the user can also wear a bone conduction headset to receive voice reminders later.

[0046] After the user wakes up the guide stick, the destination can be input, for example, by voice input at step 110. The user interface device 14 can also include a sound receiving device such as a microphone to receive voice input from the user. At step 115, a real-time map can be obtained through the data communication device 13, and current positioning data can be obtained through the positioning sensor 11_3. Although steps 110 and 115 are shown in a sequential manner, this is not restrictive, and these two steps can also be performed in parallel.

[0047] Next, at step 120, navigation path information to the destination can be obtained based on the real-time map, positioning data, and the destination. At step 125, first point cloud data representing the user's current environment is obtained via the lidar sensor 11_1. At step 130, an image of the user's current environment is obtained via the image sensor 11_2. At step 135, a three-dimensional reconstruction of the current environment is performed based on the first point cloud data, for example, this can be achieved by a simultaneous localization and mapping (SLAM) algorithm. At step 140, semantic information is obtained based on the image from the image sensor. At step 145, guidance information is generated based on the navigation path information, the three-dimensional reconstruction, and the semantic information. At step 150, the user is navigated based on the generated guidance information to reach the destination according to the navigation path information.

[0048] Although the description is made with reference to first obtaining navigation path information based on the real-time map, positioning data and destination, this is not restrictive. It is also conceivable to omit step 120 and generate guidance information directly based on the real-time map, positioning data, three-dimensional reconstruction and semantic information at step 145. During navigation for the user, the above steps 115-150 can be repeatedly performed until the user reaches the destination.

[0049] Although the use of a laser radar sensor and an image sensor to obtain point cloud data and images of the current environment is described with reference to steps 125 and 135, respectively, this is not restrictive, and any other method can be used to obtain the current environment data. In addition, although a specific embodiment of how to generate guidance information based on a real-time map, positioning data, and data of the current environment is described with reference to each of the above steps 115-150, this is not restrictive, and any method that can receive a real-time map through a data communication device and generate guidance information based on the real-time map, the user's positioning data, and the current environment data should be included in the present invention. Receiving a real-time map through a data communication device and generating guidance information based on the real-time map, the user's positioning data, and the current environment data can provide more robust user navigation. In one embodiment, traffic condition information and / or traffic light information are also received through a data communication device, thereby providing more robust user navigation.

[0050] In the above step 155, the method of navigating the user includes one or more of auditory navigation, tactile navigation, and motion navigation. Correspondingly, one or more of an auditory guidance device, a tactile guidance device, and a motion guidance device can be adopted.

[0051] Although the various steps are described in a sequential manner, this is not restrictive and does not prevent some of the steps from being executed in parallel. For example, the various steps of acquiring real-time maps, images, positioning data, and point cloud data can be executed in parallel.

[0052] It can be understood that steps 135 - 145 can be executed by the controller 12 , and step 150 can enable the auditory guidance device, the tactile guidance device and / or the motion guidance device to perform navigation for the user under the control of the controller 12 .

[0053] Figure 4 FIG. 2 shows a flow chart of a blind-guiding method 200 according to an embodiment of the present invention. Figure 4 The method 200 shown in FIG. 1 performs a process of navigating the user using the guide stick after waking up the guide stick at step 210, particularly as shown in FIG. Figure 3 Steps 115-150 are shown as described.

[0054] In the process of navigating the user, in step 220, it is determined whether the user is in a predetermined area, such as a waiting area for a bus or a taxi or a waiting area for a crosswalk, based on the acquired real-time map and positioning data. If not ("N"), the navigation in step 210 is continued. If yes ("Y"), in step 230, target related information is acquired based on the image acquired from the image sensor. In the case of a bus waiting area, for example, the target related information may be the number of the bus. In the case of a taxi waiting area, for example, the target related information may be the license plate number of the waiting taxi. In the case of a crosswalk waiting area, for example, the target related information may be traffic light information.

[0055] Next, guidance information is generated based on the acquired target related information in step 240. The generated guidance information can be used to further navigate the user.

[0056] Combine the following Figure 5 and Figure 6-7 A flowchart describing in detail the guidance method in the bus riding scenario and the pedestrian crossing scenario is given. Figure 5 A flow chart of a blind guidance method 300 according to an embodiment of the present invention is shown, and the method is used to guide a user in a scenario of riding a bus.

[0057] like Figure 5 As shown, step 310 is performed as follows Figure 4 The same process of navigating the user as step 210 shown in FIG. In the process of navigating the user, in step 320, it is determined whether the user is in the waiting area of ​​the bus based on the acquired real-time map and positioning data. If not ("N"), the navigation in step 310 is continued. If ("Y"), in step 330, it is determined whether the target vehicle is close to the waiting area based on the real-time position information of the target vehicle obtained from the data communication device 13, for example, whether it is within a predetermined distance from the waiting area. If it is determined that the target vehicle is not close to the waiting area ("N"), the real-time position information of the target vehicle is repeatedly obtained and the determination in step 330 is repeated. If it is determined that the target vehicle is close to the waiting area ("Y"), in step 340, the vehicle close to the waiting area is detected based on the second point cloud data from the millimeter wave radar sensor. The millimeter wave radar sensor is more sensitive to the detection of moving objects and is more suitable for detecting distant objects. The use of the millimeter wave radar sensor here facilitates the detection of vehicles that are moving and about to enter the station at a certain distance.

[0058] If the millimeter wave radar sensor does not detect a vehicle approaching the waiting area ("N"), the second point cloud data is repeatedly acquired to detect the vehicle that is about to enter the station. If the millimeter wave radar sensor detects a vehicle approaching the waiting area ("Y"), then in step 350, relevant information about the vehicle, such as the bus number, is acquired based on the image from the image sensor. If the user needs to take bus No. 300 to reach the destination, when multiple buses enter the station one after another, the bus currently entering the station may not be bus No. 300. The relevant information of the vehicle obtained can be used to determine whether the current bus is the correct bus that the user needs to take, that is, in step 360, it is determined whether it is the target vehicle based on the relevant information of the vehicle. If not ("N"), return to step 340 to continue to use the millimeter wave radar sensor to detect vehicles approaching the waiting area. If so ("Y"), the parking position of the target vehicle is determined in step 370, and guidance information is generated based on the parking position. Specifically, it is also possible to obtain first point cloud data of the current environment from a laser radar sensor, obtain semantic information based on the image, and then generate guidance information based on the first point cloud data and the semantic information and the parking position. This does not exclude the use of other sensor data.

[0059] Although the above embodiment is described with reference to taking a bus, this is not restrictive, and the method is also applicable to the case of taking a taxi. Even the method is also applicable to the case of taking a vehicle booked, for example, through a network platform taxi software.

[0060] In one embodiment, the waiting area described above may be not only a predetermined bus waiting area or a taxi waiting area, but also an area designated by a user, or an area where a user waits for a reserved vehicle.

[0061] In one embodiment, the target vehicle is determined based on information indicating the target vehicle received from the user interface device. For example, the target vehicle can be determined based on user input from the user interface device, such as the user speaks the license plate number of the target vehicle and the user interface device of the guide stick receives the user's voice input. For another example, the target vehicle can be determined based on the license plate number of a predetermined vehicle received from a network platform.

[0062] In one embodiment, when determining the path planning information based on the information indicating the destination received from the user interface device and the real-time map and traffic condition information obtained from the data communication device, the target vehicle to be taken at the current location (obtained by positioning data) has been determined, such as which bus to take.

[0063] Through the above method, the user can take the correct vehicle without accidentally taking the wrong vehicle. Figure 6The flowchart of the blind-guiding method 400 according to one embodiment of the present invention is shown. The blind-guiding method 400 can be used to guide a user to cross a pedestrian crossing.

[0064] like Figure 6 As shown, step 410 is performed as follows Figure 4 The same process of navigating the user as step 210 shown in FIG. 2 is used. In the process of navigating the user, in step 420, it is determined whether the user is in the waiting area for crossing the crosswalk based on the acquired real-time map and positioning data. If not ("N"), the navigation in step 410 is continued. If yes ("Y"), the current traffic light information is obtained in step 430. In one example, the traffic light information can be obtained based on the image from the image sensor. In another example, the traffic light information can be obtained through the data communication device 13.

[0065] Next, in step 440, if it is determined that the traffic light information does not indicate a green light ("N"), that is, a red light or a yellow light, a signal indicating waiting is output to the user through the user interface device 14, for example, a corresponding signal may be output through an auditory guidance device, a tactile guidance device, etc. Alternatively, the omnidirectional wheel may be locked and cannot be moved.

[0066] If it is determined in step 440 that the traffic light information indicates a green light ("Y"), then in step 450, pedestrian crossing information is acquired based on the image acquired from the image sensor, and guidance information is generated based on the pedestrian crossing information to guide the user to cross the pedestrian crossing. Guidance can be performed using one or more of the visual guidance means, tactile guidance means, and motion guidance means as described above.

[0067] Figure 7 FIG. 5 is a flowchart of a method 500 for guiding a blind person to cross a pedestrian crossing according to an embodiment of the present invention. The method 500 may be executed when it is determined that the traffic light information indicates a green light, that is, corresponding to the following example. Figure 6 The operation in step 450 is shown.

[0068] If moving targets, especially fast-moving targets, around users with visual impairments can be detected while crossing a crosswalk, it can help users avoid risks.

[0069] According to the blind guidance method 500, when it is determined that the traffic light information indicates a green light, at step 510, it is detected whether there is a moving object moving toward the user based on the second point cloud data from the millimeter wave radar sensor. Specifically, the movement speed and orientation of the object can be determined based on the second point cloud data, and if its orientation is toward the user and the speed is higher than a predetermined value, the moving object moving toward the user is determined. If it is determined in step 510 that there is a moving object moving toward the user ("Y"), then in step 520, guidance information is generated based on the speed and orientation of the moving object determined according to the second point cloud data to guide the user to avoid the moving object. For example, the omnidirectional wheels can be controlled to move in a direction away from the moving object to avoid a collision.

[0070] If it is determined in step 510 that there is no moving object ("N") moving toward the user, then in step 530, crosswalk information is acquired based on the image acquired from the image sensor, thereby generating guide information to guide the user to cross the crosswalk.

[0071] Next, in the process of guiding the user to cross the crosswalk, in step 540, it is determined whether the user has crossed the crosswalk. In one example, it can be determined whether the user has crossed the crosswalk based on the real-time map obtained from the data communication device and the positioning data obtained from the positioning sensor. In another example, it can be determined whether the user has crossed the crosswalk based on the first point cloud data from the lidar sensor and the image from the image sensor.

[0072] If it is determined in step 540 that the user has not crossed the crosswalk (“N”), the process returns to step 510 to detect whether there is a moving object moving toward the user based on the second point cloud data.

[0073] If it is determined in step 540 that the user has crossed the crosswalk ("Y"), then in step 550, navigation of the user continues, for example, as shown in FIG. Figure 6 The operation of step 410 is shown.

[0074] Although the above reference Figure 5 and 6-7 describes the guidance method for the scenarios of taking a bus and crossing a crosswalk. It can be understood that the guidance method for different scenarios can be used in combination. In this case, it can be determined whether the user is in the waiting area of ​​the bus or the waiting area of ​​the crosswalk based on the real-time map from the data communication device and the positioning data from the positioning sensor. If the user is in the waiting area of ​​the bus, the operation in the scenario of taking the bus can be performed. If the user is in the waiting area of ​​the crosswalk, the operation in the scenario of crossing the crosswalk can be performed. Although it is described that whether the user is in a predetermined area is determined based on the real-time map from the data communication device and the positioning data from the positioning sensor, it can be understood that this does not exclude the use of other types of data, such as image data from an image sensor.

[0075] Although the above mainly describes the guidance method using a guide stick in the scenarios of taking a bus and crossing a crosswalk, the guide stick of each embodiment of the present invention is not limited to being able to navigate only for users with visual impairments in the above scenarios.

[0076] In one embodiment, the guide stick of each embodiment of the present invention can also be used for emergency avoidance, for example, based on the second point cloud data from the millimeter wave radar sensor, a moving object moving toward the user during navigation of the user is detected, and if necessary, the user is guided to avoid collision based on the generated guidance information. As performed in the above steps 510 and 520, the difference is that the emergency avoidance described in steps 510 and 520 is performed during the process of guiding the user to cross the crosswalk, and it is also expected that the emergency avoidance described above is still performed during other navigation processes.

[0077] In another embodiment, the guide stick may also be equipped with a device for detecting landmarks. During navigation for the user, when the user passes a landmark, the corresponding information can be played to the user through the auditory guidance device. For example, a landmark can be set on a bus stop sign in a bus waiting area. When the user approaches the stop sign, the information related to the bus route can be played to the user through the auditory guidance device. The played information can be obtained via a data communication device.

[0078] References Figure 1-7 Various embodiments of the blind guide stick and the blind guide method of the present invention are described, and these embodiments can be combined with each other to obtain different effects without being limited by the subject type. Furthermore, the steps / processes mentioned above are not restrictive, and the functions of the above steps / processes can be merged / combined / changed / modified to obtain corresponding effects.

[0079] Each step of the method and device of the present invention can be implemented by software or corresponding hardware, or with the help of a processor / controller to implement the functions of the above steps. For example, the controller can read the computer programs in the memory and run these computer programs to implement the functions of the above steps.

[0080] It is understood that the methods of various embodiments of the present disclosure can be implemented under the control of computer programs / software. These software can be loaded into the working memory of the controller and used to execute the methods according to various embodiments of the present disclosure when running.

[0081] It is understood that the exemplary embodiments of the present disclosure cover both: creating / using the computer program / software of the present disclosure from the beginning, and converting an existing program / software to use the computer program / software of the present disclosure by means of an update.

[0082] According to another embodiment of the present disclosure, a computer program product is provided, such as a machine (e.g., computer) readable medium, such as a CD-ROM, including a computer program code, which, when executed, causes a computer or a processor to perform a method according to each embodiment of the present disclosure. The machine readable medium is, for example, an optical storage medium or a solid-state medium supplied with or as part of other hardware.

[0083] The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0084] The present disclosure is described above with reference to specific embodiments. It should be understood by those skilled in the art that the technical solution of the present disclosure can be implemented in various ways without departing from the spirit and basic features of the present disclosure. The specific embodiments are merely illustrative and not restrictive. In addition, these embodiments can be arbitrarily combined to achieve the purpose of the present disclosure. The scope of protection of the present disclosure is defined by the attached claims.

[0085] The word "comprising" in the specification and claims does not exclude the existence of other elements or steps. The expressions "first", "second" and the order of the steps shown in the figures do not limit their order or quantity. The functions of the various elements described in the specification or recorded in the claims can also be split or combined and implemented by corresponding multiple elements or a single element.

Claims

1. A guide stick for the blind, comprising: A data communication device for obtaining a real-time map; A plurality of sensors, used for acquiring data of the current environment and positioning data of the guide stick; A controller, configured to generate guidance information based on the real-time map, the positioning data, and the data of the current environment; and A user interface device is used to guide a user based on the guidance information.

2. The guide stick for the blind according to claim 1, wherein: The multiple sensors include: a radar sensor, an image sensor and a positioning sensor, the radar sensor is used to obtain first point cloud data representing the current environment of the user, the image sensor is used to obtain an image of the current environment of the user, and the positioning sensor is used to obtain positioning data of the guide stick; The controller is also used to: perform three-dimensional reconstruction of the current environment based on the first point cloud data, obtain semantic information based on the image, and generate the guidance information based on the real-time map, the positioning data, the three-dimensional reconstruction and the semantic information.

3. The guide stick for the blind according to claim 1, wherein: The controller is also used for: Determining whether the user is in a predetermined area based on the real-time map and the positioning data; If it is determined that the user is in the predetermined area, acquiring target related information based on the image; and The guidance information is generated based on the target related information.

4. The guide stick for the blind according to claim 3, wherein: The first point cloud data is acquired by a laser radar sensor, the multiple sensors further include a millimeter wave radar sensor, which is used to acquire second point cloud data representing the current environment, and the predetermined area includes a waiting area, wherein the controller is further used to: If it is determined that the user is in the waiting area, detecting a vehicle approaching the waiting area based on the second point cloud data; If the vehicle approaching the waiting area is detected, obtaining relevant information of the vehicle based on the image; Determining whether the vehicle is a target vehicle based on the relevant information of the vehicle; If the vehicle is determined to be the target vehicle, determining the parking position of the target vehicle; and The guidance information is generated based on the parking position of the target vehicle.

5. The guide stick for the blind according to claim 4, wherein: The data communication device is also used to obtain the real-time position of the target vehicle, and the controller is also used to: If it is determined that the user is in the waiting area, determining whether the target vehicle is close to the waiting area based on the real-time position of the target vehicle obtained from the data communication device; and If it is determined that the target vehicle is approaching the waiting area, the vehicle approaching the waiting area is detected based on the second point cloud data.

6. The guide stick for the blind according to claim 4 or 5, wherein: The waiting area includes one of a bus station waiting area and a taxi waiting area.

7. The guide stick for the blind according to claim 4 or 5, wherein: The target vehicle is determined by one of the following: determining the target vehicle based on information indicative of the target vehicle received from the user interface device; or The target vehicle is determined based on path planning information and positioning data, wherein the path planning information is determined according to information indicating a destination received from the user interface device and the real-time map and traffic condition information acquired from the data communication device.

8. The guide stick for the blind according to claim 3, wherein: The predetermined area includes a pedestrian crossing waiting area, wherein the controller is further configured to: If it is determined that the user is in the pedestrian crossing waiting area, obtaining current traffic light information based on the image; If the traffic light information indicates a red light or a yellow light, causing the user interface device to output a signal indicating waiting to the user; and If the traffic light information indicates a green light, the guide information is generated based on the crosswalk information acquired from the image to guide the user to cross the crosswalk.

9. The guide stick for the blind according to claim 8, wherein: The first point cloud data is acquired by a laser radar sensor, and the plurality of sensors further include a millimeter wave radar sensor, which is used to acquire second point cloud data representing the current environment, wherein the controller is further used to: If the traffic light information indicates a green light, detecting a moving object moving toward the user based on the second point cloud data; If the moving object is not detected, generating the guide information based on the crosswalk information to guide the user to cross the crosswalk; and If the moving object is detected, a speed and an orientation of the moving object are determined based on the second point cloud data and the guidance information is generated based on the speed and the orientation to guide the user to avoid the moving object.

10. The guide stick for the blind according to claim 9, wherein: The controller is also used for: determining whether the user has crossed the crosswalk based on the real-time map and the positioning data; and If it is determined that the user has not crossed the crosswalk, continue to detect a moving object moving toward the user based on the second point cloud data.

11. The guide stick for the blind according to claim 1 or 2, wherein: The user interface device includes one or more of the following: Bone conduction headphones; Vibration device; Omnidirectional wheels; and Components used for receiving sound.

12. The guide stick for the blind according to claim 11, wherein: The user interface device includes the omnidirectional wheel, wherein the controller is further configured to: The movement of the omnidirectional wheel is controlled based on the guidance information to guide the user.

13. The guide stick for the blind according to claim 2 or 3, wherein: The radar sensor is a laser radar sensor, and the plurality of sensors further include one or more of the following: an inertial sensor for sensing the posture of the guide stick; and a millimeter wave radar sensor for acquiring second point cloud data representing a current environment of the user; The controller is further used to perform three-dimensional reconstruction of the current environment based on the posture of the guide stick and / or generate the guidance information based on the second point cloud data.

14. A method for guiding the blind using a guide stick, comprising: Obtaining real-time maps via a data communication device; Acquire the data of the current environment and the positioning data of the guide stick; Generate guidance information based on the real-time map, the positioning data, and the data of the current environment; and The user is guided based on the guidance information.

15. The blind guiding method according to claim 14, wherein: Acquire first point cloud data representing the current environment of the user via a radar sensor, acquire an image of the current environment of the user via an image sensor, and acquire the positioning data of the guide stick via a positioning sensor; The blind guiding method further comprises: Performing three-dimensional reconstruction of the current environment based on the first point cloud data; Acquiring semantic information based on the image; and The guidance information is generated based on the real-time map, the positioning data, the three-dimensional reconstruction and the semantic information.

16. The blind guiding method according to claim 14, further comprising: Determining whether the user is in a predetermined area based on the real-time map and the positioning data; If it is determined that the user is in the predetermined area, acquiring target related information based on the image; and The guidance information is generated based on the target related information.

17. A computer-readable medium storing a computer program code, which, when executed, causes a controller to execute the blind guidance method according to any one of claims 14 to 16.

Images