Surrounding environment information transmission device

Abstract

[Problem] To provide an ambient environment information transmission device which can quickly and accurately transmit ambient environment information to a visually impaired person. [Solution] An ambient environment information transmission device 1 is provided with: a body 10 worn by a user; distance image capturing units 20 that each are supported by the body 10 and capture a distance image on the frontward side of the user; a line-of-sight direction detection unit 22 that is supported by the body 10 and detects a line-of-sight direction of the user; a control unit 40 that acquires distance information of a target portion of the distance image in the line-of-sight direction; and distance information output units 30, 32, 34 that output distance information by using sound or tactile sense.

Description

[Technical Field]

[0001] The present invention relates to a surrounding environment information transmission device, and more particularly to a surrounding environment information transmission device suitable for use by visually impaired people. [Background technology]

[0002] Patent Document 1 discloses an action support device for visually impaired people that can communicate information about the surrounding environment to a user. This device uses a distance image capture unit supported by a main body worn by the user to capture a distance image in front of the user. A computing unit calculates distance information for obstacles contained in the distance image and communicates the information to the user through sound or vibration. The computing unit divides the area of ​​the distance image into a central portion and a peripheral portion, and calculates peripheral distance information, which is distance information for obstacles contained in the peripheral portion, as information that changes in stages, while calculating central distance information, which is distance information for obstacles contained in the central portion, as information that changes in more stages or continuously than the peripheral distance information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-159193 Summary of the Invention [Problem to be solved by the invention]

[0004] The device disclosed in the above-mentioned Patent Document 1 allows a user to roughly grasp the distance to obstacles around the user based on peripheral distance information, and by turning the user's head in a direction of interest, the user can grasp the detailed distance based on central distance information. However, depending on the user and the usage situation, there are cases where the user wants to grasp the surrounding environmental information more quickly, and a device that can meet such a purpose has been desired.

[0005] Therefore, an object of the present invention is to provide a surrounding environment information transmission device that can quickly and accurately transmit surrounding environmental information to visually impaired people. [Means for solving the problem]

[0006] The object of the present invention is to provide a distance image capturing device comprising: a main body to be worn by a user; a distance image capturing unit supported by the main body for capturing a distance image in front of the user; a gaze direction detecting unit supported by the main body for detecting the gaze direction of the user; a control unit for acquiring distance information of a target part in the gaze direction in the distance image; and a distance information output unit for outputting the distance information by sound or touch. When a change in the distance information due to the movement of the line of sight exceeds a reference value, the control unit outputs step information from the distance information output unit. This is achieved by a surrounding environment information transmission device.

[0007] Alternatively, the object of the present invention can be achieved by a surrounding environment information transmission device comprising: a main body worn by a user; a distance image capturing unit supported by the main body for capturing a distance image in front of the user; a gaze direction detection unit supported by the main body for detecting the user's gaze direction; a control unit for acquiring distance information of a target portion in the gaze direction in the distance image; and a distance information output unit for outputting the distance information by sound or touch, and further comprising a projection unit supported by the main body for projecting patterned light in front of the user.

[0008] It is preferable that a plurality of distance information output units are provided on the left and right sides of the user, and it is preferable that the control unit localizes the output from each distance information output unit in the line of sight direction.

[0010] It is preferable that the device further includes an ultrasonic vibrator supported on the main body for transmitting and receiving ultrasonic waves in front of the user, and the control unit can obtain distance information of the target area in the line of sight based on detection by the ultrasonic vibrator.

[0011] It is preferable that the device further includes a character information imaging unit that images character information attached to an object within the imaging area of ​​the distance image imaging unit, and a character information output unit that outputs the character information by sound or touch, and the control unit can set a reading area within the imaging area of ​​the character information imaging unit based on the distance information in the line of sight direction, and output the character information included in the reading area from the character information output unit. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide a surrounding environment information transmission device that can quickly and accurately transmit surrounding environmental information to visually impaired people. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a perspective view of a surrounding environment information transmission device according to an embodiment of the present invention; [Figure 2] 2 is a block diagram of the surrounding environment information transmission device shown in FIG. 1. FIG. [Figure 3] FIG. 10 is a diagram illustrating an example of a distance image. [Figure 4] 2 is a schematic diagram for explaining an example of the operation of the surrounding environment information transmission device shown in FIG. 1. FIG. [Figure 5] 1. FIG. 4 is a schematic diagram for explaining another example of the operation of the surrounding environment information transmission device shown in FIG. [Figure 6] 10 is a perspective view of a surrounding environment information transmission device according to another embodiment of the present invention; [Figure 7] FIG. 7 is a block diagram of the surrounding environment information transmission device shown in FIG. 6. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. Fig. 1 is a perspective view of a surrounding environment information transmission device according to one embodiment of the present invention, and Fig. 2 is its block diagram. As shown in Figs. 1 and 2, surrounding environment information transmission device 1 includes, as main components, a distance image capture unit 20, a line-of-sight direction detection unit 22, a projection unit 24, an ultrasonic vibrator 26, distance information output units 30, 32, 34, and a control unit 40, which are supported by a main body 10.

[0015] The main body 10 is formed in the shape of glasses to be worn on the face of the user. There are no particular limitations on the configuration of the main body 10 as long as it can be worn by the user, but it is preferable that the position and direction are specified to a certain extent when worn and that thereafter deviation in position and direction is unlikely to occur, and examples of the main body 10 include glasses, caps, jackets, belts, masks, etc.

[0016] The distance image capturing unit 20 is, for example, a stereo camera, and is supported on both the left and right sides of the front surface of the rim portion 10a of the main body 10 so as to capture images of the area in front of the user when the main body 10 is worn by the user. The distance image capturing unit 20 is not particularly limited as long as it is configured to capture a two-dimensional distance image having distance information for each pixel. In addition to a stereo camera, for example, a TOF (Time of Flight) camera or a monocular camera may be used, or these may be used in combination. When the distance image capturing unit 20 is a stereo camera, even if it becomes difficult to capture a distance image due to a malfunction of one camera, the other camera alone can capture a distance image by utilizing motion parallax, for example.

[0017] The viewing angle and resolution of the distance image capture unit 20 are not particularly limited, but preferably ensure a range necessary for the user's daily activities. The viewing angle is, for example, 140 to 160 degrees horizontally and 112 to 128 degrees vertically. The resolution of the distance image is preferably, for example, 1280 x 1024 px, with a bit depth of 10 bits or more. To ensure reliable acquisition of distance images under a wide range of brightness, from extremely bright environments such as sunlight to nighttime, the dynamic range is preferably, for example, 100 dB or more. The spatial resolution of the distance image is preferably approximately 10 cm at a distance of 5 m, and the lens of the distance image capture unit 20 may be appropriately designed so that the resolution in the center of the distance image is higher than that in the periphery.

[0018] Gaze direction detection unit 22 is supported on the bottom of main body 10 so that it can capture an image of one of the user's eyes using a visible light camera, an infrared camera, or the like, and detect the gaze direction from the orientation of the eyeball. The method for detecting the gaze direction is not particularly limited, and in addition to the corneal reflex method, other known methods such as the search coil method, the EOG method, and the scleral reflex method may be used. Gaze direction detection unit 22 may be positioned so that it can capture images of both eyes, and detect the gaze direction from the lines of sight of both eyes.

[0019] The projection unit 24 is supported on the right side of the rim portion 10a of the main body 10 so as to project a pattern of light, such as random dots, toward the user. Even when there is no brightness difference on the surface of an object, the projection unit 24 can project a brightness difference, allowing the distance image capture unit 20 to accurately capture a distance image. The pattern of light projected by the projection unit 24 may be visible light, but is preferably infrared light (e.g., with a wavelength of 940 nm). Infrared pattern light has a steep peak in the atmospheric absorption spectrum, is highly power efficient, and has a wavelength that can suppress ambient light, thereby reducing the adverse effects of direct visual recognition and light pollution. When pattern light projection is not required, such as when a TOF camera is used as the distance image capture unit 20, the projection unit 24 may be omitted.

[0020] The ultrasonic transducer 26 measures the distance to an object by transmitting and receiving ultrasonic waves in front of the user. A plurality of ultrasonic transducers 26 are arranged in a row in the left-right direction on the front surface of the rim portion 10a of the main body 10. Driving each ultrasonic transducer 26 using a phased array method allows for obtaining distance information over a wide range. By varying the frequency of the ultrasonic waves emitted from the ultrasonic transducer 26 over time within the operating band (e.g., 40 to 80 kHz) and filtering the signals to receive only reflected signals whose frequency matches the emitted signal, distance information over a wide range can be obtained quickly. Furthermore, if the phased array method is no longer possible due to changes in the surrounding conditions or damage to some of the ultrasonic transducers 26, or if it is determined that obtaining accurate distance information is difficult, the system can be switched to a method other than the phased array method (e.g., a method that measures distance based on the time it takes for one ultrasonic transducer to emit ultrasonic waves and receive a reflected wave, such as an ultrasonic rangefinder).

[0021] The distance information output units 30, 32, and 34 are bone conduction speakers with vibrating units positioned to contact the user's head. They transmit output information to the user through voice, acoustics, or other means. The right and left distance information output units 30 and 34 are built into a pair of temples 10b and 10c of the main body 10 and contact the upper portions of the user's right and left ears, respectively. The central distance information output unit 32 is built into the bridge portion 10d of the main body 10 and contacts the upper portion of the user's nose. The number of distance information output units 30, 32, and 34 is not particularly limited; a single unit may be provided. However, it is preferable to provide multiple units on the left and right sides of the user so that information about the surrounding environment can be localized. The distance information output units 30, 32, and 34 may be configured to output air-conducted sound instead of bone-conducted sound, and may be shaped like headphones or earphones. The distance information output units 30, 32, and 34 may output distance information through tactile sensations in addition to sound. As an example of a configuration for outputting distance information by tactile sensation, the distance information output units 30, 32, and 34 may be vibration motors or piezoelectric diaphragms that generate vibrations by rotating an eccentric rotor, etc., and may be configured to transmit distance information based on the magnitude of the vibrations. As another example of a configuration for outputting distance information by tactile sensation, the distance information output units 30, 32, and 34 may be air bags connected to a pressure pump via a valve, and may transmit distance information by compressing the user's face or other surface by controlling the opening of the valve. As yet another example of a configuration for outputting distance information by tactile sensation, the distance information output units 30, 32, and 34 may be electrodes or the like, and may transmit distance information based on the amount of electrical stimulation applied to the user's face or other surface. Multiple distance information output units 30, 32, and 34 may be arranged in a row to the left and right of the user, allowing for more precise localization of surrounding environmental information.

[0022] The control unit 40 consists of a microprocessor having a CPU and memory, and is built into one of the temples of the main body 10. Based on the detection by the gaze direction detection unit 22, the control unit 40 calculates distance information of the target area contained in the distance image acquired by the distance image capturing unit 20.

[0023] In addition to the above configuration, the ambient environment information transmission device 1 is provided with a power supply unit for operating the control unit 40 and the like, and an input unit (neither of which is shown) for turning the power supply unit on and off, switching the operating mode, etc., in the main body 10. The main body 10 may further be provided with a six-axis acceleration sensor that measures the posture of the user when wearing the device, thereby reducing the processing load of image processing.

[0024] Next, the operation of the surrounding environment information transmission device 1 having the above-described configuration will be described. When the surrounding environment information transmission device 1 is operated while worn by a visually impaired person, the distance image capture unit 20 captures a distance image in front of the user. The control unit 40 can constantly operate the projection unit 24 while the distance image capture unit 20 is operating, thereby projecting patterned light onto an object. The projection unit 24 may be controlled to operate only when it is difficult for the distance image capture unit 20 to capture a distance image, or it may be operated based on the user's operation of the input unit.

[0025] Figure 3 shows an example of a distance image captured by distance image capture unit 20. The shading in Figure 3 represents the difference in distance from distance image capture unit 20 to the object, with closer distances being displayed as white and farther distances being displayed as black. This distance image corresponds to the field of vision of a healthy person.

[0026] When the user moves their line of sight, control unit 40 identifies region of interest A in the distance image that matches the line of sight direction based on detection by line of sight direction detection unit 22, and acquires distance information of the target part of the object corresponding to region of interest A. The distance information of the target part may be, for example, distance information of the central pixel of region of interest A, or an average value of distance information of multiple pixels included in region of interest A.

[0027] The control unit 40 outputs the acquired distance information of the target area from the distance information output units 30, 32, 34. The distance information can be output using sound volume, sound pitch, spoken language, etc. For example, the farther the distance, the lower the volume, and the closer the distance, the higher the volume. If the distance information output units 30, 32, 34 are configured to transmit tactile sensations (vibration, pressure, electrical stimulation, etc.), the distance information can be output by changing the intensity of the tactile sensation, the interval between tactile sensations, the tactile sensation pattern, etc.

[0028] The control unit 40 continuously captures distance images and acquires the gaze direction at a fixed cycle, and outputs distance information of the target area in the gaze direction in real time. The user can accurately acquire distance information of the target area simply by moving their gaze, allowing them to quickly and accurately grasp information about the surrounding environment that interests them.

[0029] In parallel with acquiring distance information about the target area using the distance image, control unit 40 also acquires distance information about the target area based on detection by ultrasonic vibrator 26. If the target object is a transparent object such as a glass door, or an object made of a material that is highly light-reflective or light-absorbent, distance information may not be acquired correctly using distance image capturing unit 20 alone, so by using it in combination with ultrasonic vibrator 26, distance information about the target area can be reliably acquired.

[0030] The control unit 40 can localize the distance information output from the multiple distance information output units 30, 32, and 34 in the line of sight. For example, when the line of sight in the distance image is to the far right, output is made only from the right distance information output unit 30. As the line of sight approaches the center, the output ratio of the central distance information output unit 32 to the output of the right distance information output unit 30 gradually increases, and when the line of sight is in the center, output is made only from the central distance information output unit 32. When the line of sight is to the left, the output ratios of the central distance information output unit 32 and the left distance information output unit 34 are controlled in the same manner as above, making it easier for the user to intuitively grasp the direction of the target area for which distance information has been acquired.

[0031] 4 is a schematic diagram for explaining an example of the operation of the ambient environment information transmission device 1, and shows the states when a user U moves the line of sight S from the right to the left with respect to an object O. As shown in FIG. 4(a), when the line of sight S is at the rightmost position of the object O as seen from the user, distance information is output only from the distance information output unit 30 on the right side of the user U.

[0032] As shown in Figure 4(b), when the line of sight S moves to the left of the user U, not only output is from the distance information output unit 30 on the right side but also from the distance information output unit 32 in the center, and the proportion of output from the distance information output unit 32 in the center gradually increases. Furthermore, the output information from the distance information output units 30 and 32 reflects the distance information of the target area that is gradually getting closer.

[0033] As shown in FIG. 4(c), when the line of sight S moves to the center of object O, distance information is output only from the central distance information output unit 32. When the line of sight S moves further to the left of the user U, as shown in FIG. 4(d), the output ratio from the left distance information output unit 34 relative to the central distance information output unit 32 gradually increases, and the output distance information gradually becomes farther away. As shown in FIG. 4(e), when the line of sight S is at the leftmost point of object O as seen from the user, distance information is output only from the left distance information output unit 34. From the above-mentioned line of sight movement, the user U can grasp the approximate size and shape of object O and infer what object O is.

[0034] In addition to outputting distance information from the distance information output units 30, 32, and 34, the control unit 40 can output step information when a step on the object O is detected. Fig. 5 is a schematic diagram for explaining an example of the operation of the ambient environment information transmission device 1 when the object O has a step. As with Fig. 4, Figs. 5(a) to 5(e) show various states when the user U moves the line of sight S from the right to the left with respect to the object O. When the line of sight S is moved from the state shown in Fig. 5(a), the change in distance information due to the movement of the line of sight S suddenly becomes large when the line of sight S passes through a step, as shown in Fig. 5(b).

[0035] When the change in distance information due to movement in the line of sight S exceeds a preset reference value, the control unit 40 outputs step information to notify the user of the presence of a step using a predetermined voice, sound, vibration, or the like. This allows the user to more accurately grasp the shape and location of the object O. For example, an object placed on a desk can be easily identified by the step between the desk and the object. In the state shown in FIGS. 5(a) and 5(c) to 5(e) where no step exists in the line of sight S, distance information is output from the distance information output units 30, 32, and 34, similar to the state shown in FIGS. 4(a) and 4(c) to 4(e). Steps may be stairs, entrance porches, the edges of station platforms, etc., and when these are in the user's line of sight, step information can be accurately conveyed to the user. Note that the control unit 40 may also be configured to convey steps other than those in the line of sight S to the user, distinguishing them from steps in the line of sight S. In this case, the direction of the step and the distance to the step can be output by voice or the like, for example, "step, 2 o'clock direction, 1 meter."

[0036] It is preferable that the step information be output so that the user can distinguish between cases where the distance information suddenly decreases and cases where it increases, making it easier to grasp the shape of objects around the step. For example, if the distance information suddenly becomes closer, the output of the step information can be increased, and if the distance information suddenly becomes farther away, the output of the step information can be decreased. The reference value for the change in distance information may be determined as an absolute value of the amount of change, or may be determined as a ratio of the amount of change to the distance information.

[0037] In Figure 5, an example is explained in which the line of sight direction S is moved left and right, but it is also possible to detect steps when the line of sight direction S is moved up and down, and dangerous areas such as drops and stairs can be correctly identified.

[0038] Fig. 6 is a perspective view of a surrounding environment information transmission device according to another embodiment of the present invention, and Fig. 7 is its block diagram. The surrounding environment information transmission device 1' shown in Figs. 6 and 7 is configured by adding a text information imaging unit 28 and a text information output unit 36 ​​to the surrounding environment information transmission device 1 shown in Figs. 1 and 2, but is otherwise similar in configuration to the surrounding environment information transmission device 1 shown in Figs. 1 and 2. Therefore, in Figs. 6 and 7, components similar to those in Figs. 1 and 2 are designated by the same reference numerals, and detailed description thereof will be omitted.

[0039] The text information imaging unit 28 is supported on the left side of the rim portion 10a of the main body 10 so as to capture images in substantially the same direction as the distance image capturing unit 20, and captures text information attached to objects within the imaging area of ​​the distance image capturing unit 20. A camera with higher resolution than the distance image capturing unit 20, such as a CCD camera or CMOS camera, can be preferably used as the text information imaging unit 28. Objects to which text information can be attached are not particularly limited, but examples include product packaging, flyers, posters, signs, guide boards, and labels.

[0040] The text information output unit 36 ​​is a speaker (air conduction speaker) that outputs text information by voice, and is attached integrally with the text information imaging unit 28 to the left side of the rim portion 10a of the main body 10. The text information output unit 36 ​​may be a bone conduction speaker, similar to the distance information output units 30, 32, and 34, or may be configured to convert text information into Morse code, Braille code, or the like, and output it by tactile sensation such as vibration, pressure, or electrical stimulation.

[0041] Based on the gaze direction detected by gaze direction detection unit 22, control unit 40 acquires distance information for the target portion of the object corresponding to region of interest A, outputs the distance information from distance information output units 30, 32, and 34, and sets a reading area within the imaging area of ​​character information imaging unit 28 based on the distance information. When character information is present in region of interest A, the reading area is an area that may contain other character information that is highly relevant to the character information. For example, the reading area can be set as a single closed area by using the distance information of region of interest A as a reference and including surrounding pixels whose absolute value of the difference in distance information from this reference distance information is less than a predetermined value. Alternatively, the reading area may be set as an area including surrounding pixels radiating from region of interest A where the rate of change in distance information in each direction is less than a predetermined value.

[0042] The control unit 40 acquires all character information contained in the set reading area and outputs it as sound or touch from the character information output unit 36. The character information can be acquired, for example, by performing a known OCR (Optical Character Recognition) process on the image of the reading area. The OCR process may be performed by the control unit 40, or may be performed by an external server via a network such as the Internet.

[0043] The surrounding environment information transmission device 1' of this embodiment is configured to set a reading area within the imaging area of ​​the text information imaging unit 28 based on distance information in the line of sight, and output the text information contained in this reading area from the text information output unit 36. Therefore, even if a large number of objects with text information attached are present within the imaging area of ​​the text information imaging unit 28, the text information can be extracted as a single coherent unit according to the distance to the region of interest. Therefore, only the text information that interests the user can be reliably transmitted to the user. Note that the imaging data of the text information imaging unit 28 can also be used for purposes other than reading text, such as supplementing step detection and video recording. [Explanation of symbols]

[0044] 1. Surrounding environment information transmission device 10 Main Unit 20 Range image capturing unit 22 Gaze direction detection unit 24 Projection section 26 Ultrasonic vibrator 28 Character information imaging unit 30, 32, 34 Distance information output section 36 Text information output section 40 Control Unit

Claims

1. a main body to be worn by a user; a distance image capturing unit supported by the main body for capturing a distance image in front of the user; a gaze direction detection unit supported by the main body and detecting a gaze direction of a user; a control unit that acquires distance information of a target portion in the line of sight direction in the distance image; a distance information output unit that outputs the distance information by sound or touch, The control unit outputs step information from the distance information output unit when a change in the distance information due to a movement in the line of sight direction exceeds a reference value.

2. A main body to be worn by a user; a distance image capturing unit supported by the main body for capturing a distance image in front of the user; a gaze direction detection unit supported by the main body and detecting a gaze direction of a user; a control unit that acquires distance information of a target portion in the line of sight direction in the distance image; a distance information output unit that outputs the distance information by sound or touch, The surrounding environment information transmitting device further comprises a projection unit supported by the main body and configured to project patterned light in front of the user.

3. a plurality of distance information output units are provided on the left and right sides of the user, The surrounding environment information transmission device according to claim 1 or 2, wherein the control unit orients the outputs from the distance information output units in the line of sight direction.

4. The device further includes an ultrasonic transducer supported by the main body for transmitting and receiving ultrasonic waves to and from the front side of the user, The surrounding environment information transmission device according to claim 1 , wherein the control unit acquires distance information of a target part in the line of sight direction based on detection by the ultrasonic vibrator.

5. a character information capturing unit configured to capture character information attached to an object within an imaging area of ​​the distance image capturing unit; a character information output unit that outputs the character information by sound or touch, 5. The surrounding environment information transmission device according to claim 1, wherein the control unit sets a reading area within the imaging area of ​​the character information imaging unit based on the distance information in the line of sight direction, and outputs the character information included in the reading area from the character information output unit.

Images