CONTROL DEVICE, WALKING ASSISTANCE ... SYSTEM, CONTROL METHOD, AND PROGRAM
The control device uses image processing to set virtual guide lines and notify users via vibrations, addressing limitations of landmark-dependent systems and enhancing navigation safety for visually impaired individuals.
Patent Information
- Application Number
- JP2020072723
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-04-15
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-04-15
AI Technical Summary
Existing systems for visually impaired individuals that rely on magnetic tapes or guide lines are limited to paths with pre-installed landmarks and are prone to sensor malfunctions due to obstacles.
A control device and method that utilizes image processing to detect road discontinuities and set virtual walking guide lines, notifying users through vibrations when their cane intersects these lines, even in the absence of special landmarks.
Enables safe navigation for visually impaired individuals by providing real-time guidance on paths without pre-installed markers, enhancing safety and independence.
Smart Images

Figure 0007580938000001 
Figure 0007580938000002 
Figure 0007580938000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a control device, a walking assistance device, a walking assistance system, a control method, and a program that assist the walking of visually impaired people. [Background technology]
[0002] Generally, visually impaired people walk down the road by swinging a white cane from side to side in front of them while detecting obstacles. In order to walk along the road, visually impaired people also tap their white cane on steps and block walls that separate the sidewalk from the road at regular intervals. However, in real roads, there are many sections where steps are interrupted, such as at corners and at the entrance to parking lots, making it difficult to walk along the road.
[0003] As a technology to assist visually impaired people in walking along the road, Patent Document 1 discloses a system in which a sensor that detects the magnetism of magnetic tape installed along a path is attached to the bottom end of a white cane, and vibrations or other messages are transmitted to the user depending on the detection result.
[0004] Furthermore, Patent Document 2 discloses a system in which an illumination unit that illuminates a guide line with white light and an identification unit that identifies light reflected from the guide line are provided at the tip of a white cane, and vibrations and sounds are transmitted to the user according to the identified color. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] JP 2003-131556 A [Patent Document 2] JP 2005-342218 A Summary of the Invention [Problem to be solved by the invention]
[0006] However, the systems of Patent Documents 1 and 2 require detection of a magnetic tape or a guide line, and therefore can only operate appropriately in a passageway where a magnetic tape or a guide line has been installed in advance.
[0007] Furthermore, in the systems of Patent Documents 1 and 2, a sensor is installed at the tip of the white cane, so there is a risk that the sensor may be damaged by an obstacle.
[0008] An object of the present invention is to provide a control device, a walking assistance device, a walking assistance system, a control method, and a program that are capable of assisting walking even on a passageway that does not have any special markers. [Means for solving the problem]
[0009] A control device according to one aspect of the present invention is a device for controlling a control of an image captured by an imaging means. Regarding the discontinuity of road bumps Using the information, Walking guide lines showing walkable areas a setting means for setting the position of the specific object in the image; and an acquisition means for acquiring position information regarding the position of the specific object in the image. Walking guide line and a control means for instructing the notification means to perform a notification operation when the position information indicates that the vehicle is located in the vicinity of the vehicle.
[0010] Further, a control method according to another aspect of the present invention includes: Regarding the discontinuity of road bumps Using the information, Walking guide lines showing walkable areas and acquiring position information regarding a position of a specific object in an image. Walking guide line and when the position information indicates that the vehicle is located within the predetermined position, instructing the notification means to execute a notification operation. Effect of the Invention
[0011] According to the present invention, it is possible to provide a control device, a walking assistance device, a walking assistance system, a control method, and a program that are capable of assisting walking even on a passageway that does not have any special markers. [Brief description of the drawings]
[0012] [Figure 1] 1 is a block diagram showing an outline of the configuration of a walking assistance system according to an embodiment of the present invention. [Diagram 2] 4 is a flowchart showing a process of the imaging apparatus of the first embodiment. [Diagram 3] FIG. 13 is a diagram showing an example of an image. [Figure 4] FIG. 13 is a diagram showing a discontinuous section of a step. [Diagram 5] FIG. 2 is a diagram showing an example of walking guide lines. [Figure 6] FIG. [Figure 7] FIG. 13 is a diagram showing the position history of a white cane. [Figure 8] 4 is a flowchart showing a process of the vibration device of the first embodiment. [Figure 9] 10 is a flowchart showing a process of the imaging apparatus according to the second embodiment. [Figure 10] FIG. 13 is a diagram showing an example of walking guide lines for turning corners. [Figure 11] 13 is a flowchart showing a process of the imaging apparatus according to the third embodiment. [Figure 12] FIG. 13 is a diagram showing an example of walking guide lines for avoiding obstacles. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to refer to the same components, and duplicated descriptions will be omitted.
[0014] 1(a) is a block diagram showing an outline of the configuration of a walking assistance system according to an embodiment of the present invention. The walking assistance system includes an imaging device (walking assistance device) 100 and a vibration device 120.
[0015] The imaging device 100 has a control unit 101, a memory 102, a non-volatile memory 103, an imaging unit (imaging means) 104, an image processing unit 105, a communication I / F 106, and a GPS (Global Positioning System) 107. Each unit is connected to an internal bus 110 and configured to be able to exchange data with each other via the internal bus 110.
[0016] The memory 102 is, for example, a volatile memory that uses a semiconductor element such as a RAM (Random Access Memory).
[0017] The non-volatile memory 103 is composed of, for example, a hard disk (HD) or a ROM (Read Only Memory), and stores image data, audio data, other data, and various programs for the control unit 101 to operate.
[0018] The control unit 101 controls each unit of the imaging device 100 using the memory 102 as a work memory according to a program stored in, for example, the non-volatile memory 103. As shown in Fig. 1(b), the control unit 101 has a setting unit 101a, an acquisition unit 101b, and a control unit 101c. Note that, although the control unit 101 is mounted inside the imaging device 100 in this embodiment, it may be configured as a control device separate from the imaging device 100.
[0019] The imaging unit 104 has a lens and an optical sensor that converts light from the lens into an electrical signal, captures an image under the control of the control unit 101, and stores the image in the memory 102. As the optical sensor, a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, a CCD (Charge-Coupled Device) image sensor, or the like can be used.
[0020] The image processing unit 105 performs various image processing such as resizing, trimming, edge detection, motion vector calculation, and matrix operation on the image data stored in the memory 102 under the control of the control unit 101. The image processing unit 105 may be configured with a dedicated circuit block for performing a specific image processing. Depending on the type of image processing, the control unit 101 may perform the image processing according to a program.
[0021] The communication I / F 106 is an interface for communicating with an external device, the Internet, etc., and transmitting / receiving various data such as files, commands, etc. As the communication I / F 106, for example, Bluetooth, WiFi, etc. can be used.
[0022] The GPS 107 receives and analyzes a GPS signal transmitted from a satellite, and estimates the current position (latitude and longitude) of the image capture device 100. The current position may be estimated using a WiFi Positioning System (WPS) or the like.
[0023] The vibration device 120 has a control unit 121, a memory 122, a non-volatile memory 123, a communication I / F 124, and a vibration unit 125. Each unit is connected to an internal bus 130 and configured to be able to exchange data with each other via the internal bus 130.
[0024] The memory 122 is, for example, a RAM.
[0025] The non-volatile memory 123 is composed of, for example, a hard disk (HD) or a ROM, and stores various data and various programs for the control unit 121 to operate.
[0026] The control unit 121 controls each part of the vibration device 120 according to a program stored in, for example, a non-volatile memory 123, using the memory 122 as a work memory.
[0027] The communication I / F 124 is an interface for communicating with an external device, the Internet, etc., and transmitting / receiving various data such as files, commands, etc. As the communication I / F 124, for example, Bluetooth, WiFi, etc. can be used.
[0028] The vibration unit 125 is a vibration device such as a haptic device, and vibrates based on the control of the control unit 121. The user can know that he or she is moving in the correct direction by using the vibration transmitted from the vibration unit 125. Note that, in this embodiment, the vibration of the vibration unit 125 is used to execute a notification operation for the user, but another notification means may be used as long as it is possible to execute a notification operation for the user.
[0029] The operation of each embodiment of the present invention will now be described. EXAMPLES
[0030] In this embodiment, a configuration is described in which walking is guided by notifying the user when a white cane, which is a walking assistance stick, crosses a virtual line (walking guide line) that serves as a walking guide and is set in a section where the sidewalk is interrupted by steps or other obstacles.
[0031] First, the process of the imaging device 100 will be described with reference to Fig. 2. Fig. 2 is a flowchart showing the process of the imaging device 100 of this embodiment. Each process in the flowchart of Fig. 2 is realized by the control unit 101 expanding a program stored in the non-volatile memory 103 into the memory 102 and executing it.
[0032] In step S201, the control unit 101 acquires an image captured by the imaging unit 104. Fig. 3 is a diagram showing an example of the image acquired in step S201. The image in Fig. 3 shows a road 301, steps 302 and 303, and a white cane 304.
[0033] In step S202, the control unit 101 (setting means 101a) acquires information about steps on the road that serve as walking landmarks by analyzing the image acquired in step S201. In this embodiment, the control unit 101 acquires information about the steps 302 and 303. As a method of analyzing an image to acquire information about an object such as a step on a sidewalk, there is a method using machine learning such as Deep Learning, or a method using Simultaneous Localization and Mapping (SLAM). Note that, in this embodiment, information about steps on the road is acquired, but is not limited to this as long as it is predetermined information that can set a predetermined area for assisting walking as described later. Also, in this embodiment, information about steps is acquired by the control unit 101 analyzing the image, but may be acquired using position information based on a signal from an artificial satellite. Also, the image may be analyzed using the image processing unit 105.
[0034] In step S203, the control unit 101 (setting means 101a) uses the information about the step acquired in step S202 to determine whether or not the step has a discontinuous section. An example of a discontinuous section is the section 401 between the steps 302 and 303 shown by the dashed line in Fig. 4. If a discontinuous section is present, the process proceeds to step S204; otherwise, the process ends.
[0035] In step S204, the control unit 101 (setting means 101a) sets a predetermined area in the image to assist walking, based on the analysis result of the image. In this embodiment, the control unit 101 (setting means 101a) sets a virtual line (walking guide line) indicating a walkable area so as to connect steps, as an example of the predetermined area. FIG. 5 is a diagram showing an example of a walking guide line. Walking guide line 501 shown by a dashed line is generated parallel to steps 302, 303 in a section where steps 302, 303 are interrupted.
[0036] In step S205, the control unit 101 (acquisition means 101b) acquires information about the position of the white cane (specific object) in the image obtained by analyzing the image acquired in step S201. FIG. 6 is a schematic diagram showing a white cane. 601, 602, and 603 respectively represent the grip, handle, and ferrule of the white cane. 604 represents a painted part painted red for about 20 cm near the tip of the white cane. In this embodiment, the control unit 101 acquires information about the position of the white cane by detecting the painted part 504. Methods for detecting a part having a specific color from an image include a method using a color histogram for each divided block, and a method using machine learning such as Deep Learning, as in step S202. Note that in this embodiment, information about the position of the white cane is acquired by analyzing the image, but information about the position of the white cane may be acquired by installing a position identification transmitter in the white cane and acquiring information about the position of the white cane by a signal transmitted.
[0037] In step S206, the control unit 101 judges whether the position information regarding the position of the specific object is located within the predetermined area set in step S204. In this embodiment, the control unit 101 judges whether the white cane has crossed the walking guide line (crossed the walking guide line) on the image. FIG. 7 is a diagram showing the history of the position of the white cane. In FIG. 7, the white circle represents the position of the white cane, and the position of the white cane changes in the order of (a) to (c). In FIG. 7(a) and (b), the white cane is located on the right side of the walking guide line, and the control unit 101 judges that the white cane does not cross the walking guide line. In FIG. 7(c), the white cane moves from the right side to the left side of the walking guide line. Therefore, the control unit 101 judges that the white cane crossed the walking guide line at the time of FIG. 7(c). If the white cane crossed the walking guide line, the process proceeds to step S207, and if not, the process ends.
[0038] In step S207, the control unit 101 (control means 101c) transmits an instruction to the notification means to execute a notification operation for the user. In this embodiment, the control unit 101 transmits an instruction to the vibration unit 125 via the communication I / Fs 106 and 124 to vibrate.
[0039] Next, the processing of vibration device 120 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the processing of vibration device 120. Each process in the flowchart of Fig. 8 is realized by control unit 121 expanding a program stored in non-volatile memory 123 into memory 122 and executing it.
[0040] In step S801, the control unit 121 determines whether or not an instruction has been received from the control unit 101. If an instruction has been received, the process proceeds to step S802, and if not, the process of this step is repeated.
[0041] In step S802, the control unit 121 vibrates the vibration unit 125 to notify the user that the white cane has crossed the walking guide line.
[0042] As described above, according to the configuration of this embodiment, when the white cane crosses the walking guide line in a section where there are no steps, the vibration device attached to the white cane is vibrated. This makes it possible to assist the user in walking along the path even in a passageway where no special markers are installed. EXAMPLES
[0043] In this embodiment, a configuration for assisting the user's walking when turning a corner will be described.
[0044] Fig. 9 is a flowchart showing the processing of the imaging device 100 of this embodiment. Each process in the flowchart of Fig. 9 is realized by the control unit 101 expanding a program stored in the non-volatile memory 103 into the memory 102 and executing it. Note that the processing of steps S201, S205, and S207 in Fig. 9 are similar to the processing of steps S201, S205, and S207 in Fig. 2, respectively, and therefore description thereof will be omitted.
[0045] In step S901, the control unit 101 checks the current position of the imaging device 100 acquired from the GPS 107 against map data stored in the non-volatile memory 103 or the memory 102 to determine whether or not the user is near a corner. If the user is near a corner, the process proceeds to step S902, and if not, the process proceeds to step S905. Note that in this step, it may also be possible to determine whether or not the user is about to turn a corner by using route information set in advance.
[0046] In step S902, the control unit 101 (setting means 101a) determines whether or not there is a corner by analyzing the image acquired in step S201. As a method for determining whether or not there is a corner, there is a method using machine learning such as deep learning, as in the first embodiment. If there is a corner, the process proceeds to step S903, and if not, the process proceeds to step S905. Note that in this embodiment, information regarding the presence or absence of a corner is acquired by the control unit 101 analyzing the image, but may also be acquired using position information based on a signal from an artificial satellite.
[0047] In step S903, the control unit 101 (setting means 101a) sets a predetermined area on the image to assist walking, based on the analysis result of the image. In this embodiment, the control unit 101 (setting means 101a) sets a walking guide line for turning a corner as an example of the predetermined area. FIG. 10 is a diagram showing an example of a walking guide line for turning a corner. The image in FIG. 10 shows a road 1001 and steps 1002 and 1003. The walking guide line 1004 shown by a dashed line is a walking guide line for a user who has been walking along the step 1003 to turn the corner, and is set so as to go around the corner, starting from the part where the step 1003 ends.
[0048] In step S904, the control unit 101 judges whether the position information regarding the position of the specific object is located within the predetermined area set in step S204. In this embodiment, the control unit 101 judges whether the white cane crosses the walking guide line on the image. The process of step S904 is similar to the process of step S206, but in this step, the control unit 101 judges that the white cane crosses the walking guide line when the white cane moves from the left side to the right side of the walking guide line. This enables the user to walk inside the walking guide line. If the white cane crosses the walking guide line, the process proceeds to step S207, and if not, the process ends. Note that FIG. 10 shows a case where the user turns left, but in the case of turning right, the control unit 101 may judge that the white cane crosses the walking guide line when it moves from the right side to the left side.
[0049] In step S905, the control unit 101 performs the processes from step S202 onward in FIG.
[0050] As described above, according to the configuration of this embodiment, it is possible to assist the user in walking even when turning a corner. EXAMPLES
[0051] In this embodiment, a configuration for assisting a user in walking when there is an obstacle on the road will be described.
[0052] Fig. 11 is a flowchart showing the processing of the imaging device 100 of this embodiment. Each process in the flowchart of Fig. 11 is realized by the control unit 101 expanding a program stored in the non-volatile memory 103 into the memory 102 and executing it. Note that the processing of steps S201, S205, and S207 in Fig. 11 are similar to the processing of steps S201, S205, and S207 in Fig. 2, respectively, and therefore description thereof will be omitted.
[0053] In step S1101, the control unit 101 (setting means 101a) determines whether or not there is an obstacle on the road by analyzing the image acquired in step S201. As in the first embodiment, a method of determining whether or not there is an obstacle is to use machine learning such as deep learning. If there is an obstacle, the process proceeds to step S1102, and if not, the process proceeds to step S1104. Note that in this embodiment, information regarding the presence or absence of an obstacle is acquired by the control unit 101 analyzing the image, but may also be acquired using position information based on a signal from an artificial satellite.
[0054] In step S1102, the control unit 101 (setting means 101a) sets a predetermined area on the image to assist walking based on the analysis result of the image. In this embodiment, the control unit 101 (setting means 101a) sets walking guide lines for avoiding obstacles as an example of the predetermined area. FIG. 12 is a diagram showing an example of walking guide lines for avoiding obstacles. The image in FIG. 12 shows a road 1201 and obstacles 1202 and 1203. An inner walking guide line 1204 and an outer walking guide line 1205 shown by dashed lines are walking guide lines for walking while avoiding the obstacles 1202 and 1203. The inner walking guide line 1204 is set so as to follow the obstacle. The outer walking guide line 1205 is set so as to go around the obstacle.
[0055] In step S1103, the control unit 101 judges whether the position information on the position of the specific object is located within the predetermined area set in step S204. In this embodiment, the control unit 101 judges whether the white cane crosses the walking guide line on the image. The process of step S1103 is similar to the process of step S206, but in this step, the direction in which the control unit 101 judges that the white cane crosses the walking guide line is different for the inner walking guide line 1204 and the outer walking guide line 1205. Specifically, the control unit 101 judges that the white cane crosses the walking guide line when it moves from the right side to the left side of the inner walking guide line 1204, and judges that the white cane crosses the walking guide line when it moves from the left side to the right side of the outer walking guide line 1205. In other words, the control unit 101 judges that the white cane crosses the two walking guide lines when it moves from the inside to the outside. This enables the user to walk without hitting an obstacle with the white cane and without going too far into the roadway. If the white cane intersects with the walking guide line, the process proceeds to step S207; if not, the process ends.
[0056] In step S1104, the control unit 101 executes the processes from step S901 onward in FIG.
[0057] As described above, according to the configuration of this embodiment, it is possible to assist the user in walking even when there is an obstacle on the road. [Other Examples] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.
[0058] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]
[0059] 101 Control unit (control device) 101a Setting means 101b Acquisition method 101c Control Means 104 Imaging unit (imaging means) 125 Vibration unit (notification means)
Claims
1. a setting means for setting a walking guide line indicating a walkable area in an image captured by an imaging means, using information regarding the end of a step on a road in the image; an acquisition means for acquiring position information relating to a position of a specific object in the image; and a control means for instructing a notification means to execute a notification operation when the position information is located on the walking guide line.
2. A setting means for setting a walking guide line indicating a walkable area in an image captured by the imaging means, using information about a corner of a road in the image; an acquisition means for acquiring position information relating to a position of a specific object in the image; and a control means for instructing a notification means to execute a notification operation when the position information is located on the walking guide line.
3. a setting means for setting walking guide lines indicating walkable areas in an image captured by the imaging means, using information about obstacles on a road in the image; an acquisition means for acquiring position information relating to a position of a specific object in the image; and a control means for instructing a notification means to execute a notification operation when the position information is located on the walking guide line.
4. 2. The control device according to claim 1, wherein the setting means acquires information regarding the end of the step on the road by analyzing the image.
5. The control device described in Claim 2, characterized in that the setting means obtains information regarding corners of the road by analyzing the image.
6. The control device described in Claim 3, characterized in that the setting means obtains information regarding obstacles on the road by analyzing the image.
7. 2. The control device according to claim 1, wherein the setting means obtains information regarding the end of the step on the road by using position information based on a signal from a satellite.
8. The control device described in Claim 2, characterized in that the setting means acquires information regarding corners of the road.
9. The control device described in Claim 3, characterized in that the setting means acquires information regarding obstacles on the road.
10. The control device according to claim 1 , wherein the setting means sets the walking guide line so as to connect the discontinuities.
11. 3. The control device according to claim 2, wherein the setting means sets the walking guide line so as to go around the corner.
12. 4. The control device according to claim 3, wherein the setting means sets a first walking guide line along the obstacle, and sets a second walking guide line so as to go around the obstacle.
13. 13. The control device according to claim 1, wherein the notification means is a vibration means.
14. The control device according to any one of claims 1 to 12, wherein the specific object is a walking stick.
15. An imaging means for capturing an image; A walking assistance device comprising the control device according to any one of claims 1 to 14.
16. The walking assistance device according to claim 15 ; a notification device including notification means for performing a notification operation based on an instruction from the walking assist device.
17. setting a walking guide line indicating a walkable area in an image captured by an imaging means using information regarding the end of a step on a road in the image; obtaining position information relating to a position of a specific object in the image; and when the position information is located on the walking guide line, instructing a notification means to execute a notification operation.
18. setting a walking guide line indicating a walkable area in an image captured by an imaging means using information about corners of a road in the image; obtaining position information relating to a position of a specific object in the image; and when the position information is located on the walking guide line, instructing a notification means to execute a notification operation.
19. setting a walking guide line indicating a walkable area in an image captured by an imaging means using information about obstacles on a road in the image; obtaining position information relating to a position of a specific object in the image; and when the position information is located on the walking guide line, instructing a notification means to execute a notification operation.
20. A program for causing a computer to execute the control method according to any one of claims 17 to 19.
Citation Information
Patent Citations
Walk guide system for visually impaired person
JP2003131556A
Walking support apparatus
JP2005143892A
Guide system
JP2005342218A
Walking stick for visually impaired person
JP2019005264A
Control device, walking aid device, walking aid system, control method and program
JP2021153814A
