Control device used in mobility assistance system

Abstract

To provide a control apparatus capable of reducing power necessary for a movement support system.SOLUTION: A control apparatus is used for a movement support system for assisting walking of a user with a white cane by notifying the user of information. The white cane is provided with an imaging device, and the information is generated on the basis of an image picked up by the imaging device. The control device includes: a predetermined motion determination part for determining whether or not there has been a predetermined motion of the user using the white cane on the basis of a movement of the white cane; and an initiation processing part for executing initiation processing for the imaging device once the predetermined motion determination part determines there has been the predetermined motion.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a control device used in a mobility assistance system. [Background technology]

[0002] A known mobility support device for providing walking support to pedestrians such as visually impaired people is disclosed in Patent Document 1. Patent Document 1 discloses that the device includes a direction determination unit that determines the walking direction of a person who acts without using their vision (visually impaired person), and a guide information generation unit that generates guide information for the visually impaired person to walk in the determined direction, and determines the walking direction of the visually impaired person by matching an image from a camera carried by the visually impaired person with a pre-stored reference image, and provides guidance to the visually impaired person about the walking direction by voice or the like. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Republished Patent Publication WO2018 / 025531 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional object recognition using cameras requires the camera to be constantly on and taking pictures in order to recognize temporary obstacles such as construction structures and signs, and there is room for improvement in terms of power requirements when using it while carrying it around.

[0005] The present invention has been made in consideration of the above points, and its purpose is to provide a control device that can reduce the power required by a mobility assistance system by activating an imaging device used to assist a user in their mobility as needed by the user. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the control device used in the mobility assistance system of the present invention is a control device used in a mobility assistance system that assists a user using a white cane in walking by notifying the user of information, wherein the white cane is equipped with an imaging device and the information is generated based on images captured by the imaging device, and the control device is equipped with a predetermined action determination unit that determines whether or not a predetermined action has been taken by the user using the white cane based on the movement of the white cane, and a startup processing unit that executes startup processing for the imaging device when the predetermined action determination unit determines that the predetermined action has been taken.

[0007] This allows the imaging device to be activated in response to a predetermined action made by the user using the white cane, i.e., the imaging device can be activated in response to the user's intention to check an object, so the imaging device can be activated as needed by the user. This eliminates the need to keep the imaging device activated all the time, thereby reducing the power required by the mobility assistance system.

[0008] The movement of the white cane in the present invention is the movement of the white cane when the user is standing still, and the predetermined action is the predetermined action when the user is standing still.

[0009] Typically, a user walks while touching the ground or obstacles with their white cane. It is assumed that the user will stop when they want to check an object. Therefore, the user's intention to check an object can be determined from the movement of the white cane when the user is standing still, and as a result, the imaging device can be activated more appropriately according to the user's needs.

[0010] The predetermined action in the present invention is characterized in that the user hits the same object multiple times with the white cane.

[0011] The action of tapping the same object multiple times with a white cane is an action that a user can easily take when checking an object, and this allows the imaging device to be activated as needed by the user.

[0012] The control device of the present invention is characterized by further comprising an image acquisition unit that acquires an image captured by the imaging device after the imaging device has been started by the startup processing unit, and a stop processing unit that executes a stop process for the imaging device when the image is acquired by the image acquisition unit.

[0013] This allows the imaging device to be stopped appropriately, thereby reducing the power required by the mobility assistance system. [Effects of the Invention]

[0014] In the present invention, the imaging device used to assist the user in moving around can be activated as needed by the user, eliminating the need to keep the imaging device activated all the time, thereby reducing the power required by the mobility assistance system. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a diagram showing the system configuration of a mobility assistance system according to an embodiment and the appearance of a white cane; [Figure 2] FIG. 2 is a block diagram showing a schematic configuration of a control system of the mobility assistance system. [Figure 3] FIG. 2 is a block diagram showing the functional configuration of a control device. [Figure 4] FIG. 10 is a diagram for explaining a predetermined operation in the embodiment. [Figure 5] 1 is an example of an image captured by a camera. [Figure 6] 4 is a flowchart showing the flow of processing executed by the control device. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following, a visually impaired person may be simply referred to as a user. Furthermore, the user in the present invention is not limited to a visually impaired person.

[0017] 1 is a diagram showing the system configuration of a mobility assistance system 10 according to this embodiment and the appearance of a built-in white cane 1. The mobility assistance system 10 assists a user who uses the white cane 1 in walking by notifying the user of information. Each component of the mobility assistance system 10 is built into the grip portion 3 of the white cane 1.

[0018] The white cane 1 comprises a shaft 2, a grip 3, and a tip (ferrule) 4. The shaft 2 is a hollow rod with a roughly circular cross section, and is made of aluminum alloy, glass fiber reinforced resin, carbon fiber reinforced resin, or the like.

[0019] The grip portion 3 is provided at the base end (upper end) of the shaft portion 2, and is configured by attaching a cover 31 made of an elastic material such as rubber. The grip portion 3 has a slightly curved tip side (upper side in FIG. 1) in consideration of ease of holding when the user grips it and preventing slipping. However, the configuration of the grip portion 3 is not limited to this.

[0020] The tip portion 4 is a substantially cylindrical member with a bottom made of a hard synthetic resin or the like, and is fitted onto the tip portion of the shaft portion 2 and fixed thereto by means of adhesive, screwing, etc. The tip end surface of the tip portion 4 is hemispherical.

[0021] The white cane 1 according to this embodiment is a straight cane that cannot be folded, but it may be foldable or extendable at one or more intermediate points in the shaft portion 2.

[0022] Fig. 2 is a block diagram showing a schematic configuration of a control system of the mobility assistance system 10. As shown in Fig. 1 and Fig. 2, the mobility assistance system 10 includes a camera (imaging device) 20, a GPS module 30, a short-range wireless communication device 40, a vibration generator 50, a speaker 55, a battery 60, a charging socket 70, an inertial measurement unit (hereinafter referred to as an IMU (Inertial Measurement Unit)) 90, a control device 80, and the like.

[0023] The camera 20 is embedded in the front surface (the surface facing the user's traveling direction) of the grip portion 3 at the base of the grip portion 3, and captures an image of the area ahead in the user's traveling direction (forward in the walking direction).

[0024] As is well known, the GPS module 30 receives signals transmitted from a plurality of artificial satellites and fixed stations, and determines the position of the white cane 1 (approximate to the position of the user) with high accuracy.

[0025] The short-range wireless communication device 40 is a wireless communication device for performing short-range wireless communication between the camera 20, the GPS module 30, and the IMU 90 and the control device 80. For example, by using a well-known communication means such as Bluetooth (registered trademark), short-range wireless communication is performed between the camera 20, the GPS module 30, and the IMU 90 and the control device 80, and information on an image (camera image) captured by the camera 20, information on the user's position, and information measured by the IMU 90 (information on three-axis acceleration and three-axis angular velocity) is wirelessly transmitted to the control device 80.

[0026] The vibration generator 50 is disposed, for example, at the base of the grip portion 3. The vibration generator 50 vibrates in response to the operation of a built-in motor, and transmits the vibration to the grip portion 3, thereby enabling various notifications (instructions) to be given to the user holding the grip portion 3.

[0027] The speaker 55 is disposed, for example, at the upper end of the grip portion 3. The speaker 55 emits sounds toward the user to provide various notifications (instructions and warnings) to the user.

[0028] The battery 60 is made up of a secondary battery, and stores power for the GPS module 30, the short-range wireless communication device 40, the vibration generator 50, the speaker 55, the control device 80, and the IMU 90.

[0029] The charging socket 70 is a portion to which a charging cable is connected when storing power in the battery 60. For example, the charging cable is connected when the user is at home and wants to charge the battery 60 from a household power source.

[0030] The IMU 90 is equipped with a three-axis gyro and a three-directional accelerometer, which measure the three-dimensional acceleration (three-axis acceleration) and angular velocity (three-axis angular velocity) at the position where the IMU 90 is installed on the white cane 1. Note that the IMU 90 may also have sensors such as a pressure gauge and GPS built in to improve reliability.

[0031] The control device 80 includes, for example, a processor such as a CPU 81 (Central Processing Unit), a ROM 82 (Read-Only Memory) that stores a control program, a RAM 83 (Random-Access Memory) that temporarily stores data, and an input / output I / F 84.

[0032] The input / output I / F 84 is capable of receiving information about camera images captured by the camera 20 and information about the user's position from the GPS module 30 via the short-range wireless communication device 40. The information receiving unit 81 is also capable of receiving information about the triaxial acceleration and triaxial angular velocity measured by the IMU 90 via the short-range wireless communication device 40. At least one of the information about the camera image, the information about the user's position, and the information about the triaxial acceleration and triaxial angular velocity includes time zone information such as time. As a result, the input / output I / F 84 receives time zone information (information about the date, day of the week, time, etc.) about the current time while the user is moving. Note that a clock may be built into the control device 80, and the input / output I / F 84 may receive the current time zone information from this clock.

[0033] Fig. 3 is a block diagram showing the functional configuration of the control device 80. As shown in Fig. 3, the control device 80 includes, as its functional components, a predetermined operation determination unit 801, a start processing unit 802, an image acquisition unit 803, a stop processing unit 804, an object determination unit 805, a notification information generation unit 806, and a notification control unit 807. Each functional component is realized by the CPU 81 reading and executing a control program stored in the ROM 82.

[0034] The predetermined action determination unit 801 determines whether or not a predetermined action has been performed by the user using the white cane 1. Here, the predetermined action may be stored in advance in the control device 80, or may be stored in a server (not shown) (a server capable of mutual communication with the mobility assistance system 10).

[0035] The predetermined action will be described below with reference to FIG. 4. FIG. 4 is a diagram for explaining the predetermined action in this embodiment. FIG. 4 illustrates a situation in which a user, while walking with a white cane 1, encounters a traffic cone TC or a traffic cone bar CB on his / her path that is not included in the map data. In this embodiment, from the viewpoint of power consumption of the battery 60, the camera 20 is normally stopped, and the camera 20 is activated when the user wishes to confirm an object. In the above situation, when the white cane 1 touches a traffic cone TC on his / her path, as shown in FIG. 4, the user stops and attempts to use the white cane 1 to confirm what the object touched is. Some users may also stop and use the white cane 1 to confirm whether there are any traffic cones TC in the vicinity. The predetermined action is specifically a predetermined action taken when the user is standing still, and is defined as a user action using the white cane 1 to confirm what the object touched is, or a user action using the white cane 1 to confirm whether there are any objects in the vicinity.

[0036] For example, when the predetermined action determination unit 801 determines that (1) the user is standing still, it determines that (2) the user has performed a predetermined action in which the user hits the same object (a traffic cone TC in the scene shown in FIG. 4 ) multiple times with the white cane 1. Regarding (1), for example, it may be determined that the user is standing still if there is no change in the user's position information for a certain period of time. Regarding (2), it may be determined whether the predetermined action has occurred based on the movement of the white cane 1, i.e., based on the triaxial acceleration and triaxial angular velocity information measured by the IMU 90. For example, it may store the correlation between the triaxial acceleration and triaxial angular velocity measured when the user hits the same object with the white cane 1, and if the measured triaxial acceleration and triaxial angular velocity match the stored correlation, it may be determined that the predetermined action has occurred. The number of times the user hits the same object multiple times may be two, which is a more natural user behavior, but the number of times is not limited.

[0037] When the predetermined operation determination unit 801 determines that a predetermined operation has been performed, the startup processing unit 802 executes startup processing of the camera 20. The startup processing unit 802 outputs a camera startup signal to the camera 20 via the short-range wireless communication device 40, for example.

[0038] The image acquisition unit 803 acquires a camera image captured by the camera 20 after the camera 20 has been activated by the activation processing unit 802 .

[0039] When the image acquisition unit 803 acquires a camera image, the stop processing unit 804 executes a process of stopping the camera 20. The stop processing unit 804 outputs a camera stop signal to the camera 20 via the short-range wireless communication device 40, for example.

[0040] The object determination unit 805 determines the captured object from the information of the camera image acquired by the image acquisition unit 803. The method of determining the object captured in the camera image is, for example, R-CNN using deep learning, but various known methods can also be applied.

[0041] The notification information generation unit 806 generates notification information to be notified to the user from the user's position information, camera image information, triaxial acceleration and triaxial angular velocity information, and object information determined by the object determination unit 805. Examples of the notification information include object information captured in the camera image and guidance information for avoiding the object.

[0042] The notification information will be described below with reference to Fig. 5. Fig. 5 shows an image captured by camera 20 in the scene shown in Fig. 4.

[0043] 5 is acquired by the image acquisition unit 803, the object determination unit 805 determines that the object hit by the user is a cone TC1. In addition to the cone TC1, the camera image in Fig. 5 also shows a cone bar CB and a signboard SB, but based on the triaxial acceleration and triaxial angular velocity information measured by the IMU 90 and the camera image information, it is estimated that the object hit by the user is the cone TC1.

[0044] When object information indicating that the object hit by the user is a cone TC1 is input from the object determination unit 805, the notification information generation unit 806 generates notification information notifying the user that the object hit is a cone TC1. For example, the notification information may be generated as audio such as "What you just hit is a cone."

[0045] The notification control unit 807 outputs the notification information generated by the notification information generation unit 806 to the vibration generator 50 or the speaker 55. At this time, guidance information for avoiding the object may be simultaneously notified to the user. For example, when the camera image shown in FIG. 5 is acquired, it may be determined that the area to the far left of the camera image is impassable, and a notification may be issued to guide the user to the right. In this case, the obstacle avoidance guidance using image recognition is performed by various known methods.

[0046] Next, the flow of processing executed by the control device 80 will be described with reference to the flowchart of FIG.

[0047] First, in step S10 (hereinafter, the word "step" will be omitted) corresponding to the function of the predetermined action determination unit 801, when it is determined that the user is standing still based on information about the user's position, it is determined from information about the three-axis acceleration and three-axis angular velocity whether or not the user has performed a predetermined action using the white cane 1 (the action of the user hitting the same object multiple times with the white cane 1). If a negative determination is made in S10, the processing of S10 is repeatedly executed until a positive determination is made in S10. If a positive determination is made in S10, the processing proceeds to S11.

[0048] Next, in S11, which corresponds to the function of the activation processing unit 802, an activation process is executed in which a camera activation signal is output to the camera 20 via the short-range wireless communication device 40.

[0049] Next, in S12, which corresponds to the function of the image acquisition unit 803, an image captured by the camera 20 after the camera 20 is started by executing the start-up process in S11 is acquired.

[0050] Next, in S13, which corresponds to the function of stop processing unit 804, when a camera image is acquired in S12, a stop process is executed to output a camera stop signal to camera 20 via short-range wireless communication device 40.

[0051] Next, in S14, which corresponds to the function of the object determination unit 805, the captured object is determined from the information of the camera image acquired in S12.

[0052] Next, in S15, which corresponds to the function of the notification information generating unit 806, notification information to be notified to the user is generated from the user's position information, camera image information, triaxial acceleration and triaxial angular velocity information, and the object information determined in S12. As described above, the notification information includes object information captured in the camera image, guidance information for avoiding the object, etc.

[0053] Next, in S16, which corresponds to the function of the notification control unit 807, the notification information generated in S15 is output to the vibration generator 50 or the speaker 55 to notify the user. The sound provided to the user is not limited to the above-mentioned sound, and may be other sounds such as a beep. Furthermore, the notification information may be notified to the user by the vibration of the vibration generator 50 in a pattern corresponding to the notification information. When the notification information is notified by vibration of the vibration generator 50, the user can properly obtain the notification information even if the user is visually or hearing impaired.

[0054] When the process of S16 is completed, the control device 80 ends the series of processes in the flowchart of FIG.

[0055] Next, the effects of this embodiment will be described.

[0056] As described above, the control device 80 of this embodiment can activate the camera (imaging device) 20 used when assisting the user in moving around according to the user's intention to check an object, that is, according to the user's needs, eliminating the need to keep the camera 20 activated all the time. This reduces the power required by the movement assistance system 10.

[0057] It is also conceivable to activate the camera 20 using a switch, but having too many switches on the white cane 1 can lead to erroneous operation. By activating the camera 20 based on the movement of the white cane 1, the number of switches on the white cane 1 can be reduced.

[0058] In the above embodiment, the predetermined action is defined as the user tapping the same object multiple times with the white cane 1, but the predetermined action is not limited to the above embodiment and may be any action that indicates the user's intention to check an object, and an unusual action taken by the user using the white cane 1 may also be considered as the predetermined action. Furthermore, the predetermined action is not limited to the predetermined action taken when the user is standing still.

[0059] Furthermore, in the above embodiment, when it is determined that a predetermined action has been performed, a camera activation signal is output to camera 20 via short-range wireless communication device 40. However, when it is determined that a predetermined action has been performed, the user may be queried as to whether or not to activate the camera. That is, to prevent erroneous determination that the user intends to check an object, confirmation regarding activation of the imaging device may be performed in advance by a voice prompt such as "Do you want to activate the camera?" or vibration. In this case, the process of querying the user as to whether or not to activate the camera corresponds to the activation process. Note that the method of responding to the query is not limited to a voice response by the user or a gesture such as tapping the object with the white cane.

[0060] Furthermore, the configuration and placement position of camera 20 are not limited to those described above. Camera 20 is configured as a relatively wide-angle camera, but may incorporate a mechanism for rotating camera 20 around a vertical axis, allowing the rotation of camera 20 by this mechanism to capture images of the user's surroundings. Also, in addition to the front of grip unit 3, a camera similar to camera 20 may be embedded in the back of grip unit 3. This makes it possible to capture images not only in front of the user but also behind the user. For example, if a bicycle is approaching from behind the user, the approach of the bicycle can be recognized by the camera image. [Explanation of symbols]

[0061] 1. White cane 10. Mobility Support System 20 Camera (imaging device) 80 Control device 801 Predetermined operation determination section 802 Startup processing section 804 Stop processing unit

Claims

1. A control device used in a mobility assistance system that supports walking of a user who uses a white cane by notifying the user of information, The white cane is equipped with an imaging device, the information is generated based on an image captured by the imaging device, The control device a predetermined action determination unit that determines whether or not the user has performed a predetermined action using the white cane based on the movement of the white cane; a startup processing unit that executes startup processing of the imaging device when the predetermined operation determination unit determines that the predetermined operation has occurred; A control device used in a mobility assistance system comprising:

2. The movement of the white cane is the movement of the white cane when the user is standing still, and the predetermined action is the predetermined action when the user is standing still. A control device used in the mobility assistance system according to claim 1.

3. The predetermined action is an action in which the user hits the same object multiple times with the white cane. A control device used in the mobility assistance system according to claim 1 or 2.

4. an image acquisition unit that acquires an image captured by the imaging device after the imaging device has been started by the start-up processing unit; a stop processing unit that executes a stop process of the imaging device when the image acquisition unit acquires the image; The control device used in the mobility assistance system according to claim 1 or 2, further comprising:

Images