Sensory substitution devices

The sensory substitution device addresses the inadequacies of existing technologies by employing shape-changing haptic interfaces that utilize computer vision and SLAM to provide precise and intuitive navigation for visually impaired individuals, enhancing navigation accuracy and intuitiveness in complex environments.

GB2613825BActive Publication Date: 2024-12-04MAKESENSE TECH LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
GB2021018227
Authority / Receiving Office
GB · GB
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-12-16
Publication Date
2024-12-04
Estimated Expiration
2041-12-16

AI Technical Summary

Technical Problem

Existing sensory substitution devices for visually impaired individuals inadequately communicate information derived from computer vision and Simultaneous Localisation and Mapping (SLAM) technologies, relying on ambiguous audio cues and vibration, which are cognitively distracting and ineffective for conveying complex navigation information like direction or distance.

Method used

A sensory substitution device that uses shape-changing haptic interfaces, combining computer vision and LIDAR with SLAM, to provide intuitive navigation guidance by altering its shape based on sensor data, allowing users to feel the direction and distance of objects or locations through proprioception, without the need for Bluetooth™ beacons or GPS.

Benefits of technology

The device enhances navigation accuracy and intuitiveness, enabling users to navigate complex environments, including 3D spaces, without modifications to infrastructure, and effectively directs users to objects or locations with high precision, surpassing the limitations of existing technologies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000001_0000
    Figure 00000001_0000
  • Figure 00000002_0000
    Figure 00000002_0000
  • Figure 00000002_0001
    Figure 00000002_0001
Patent Text Reader

Abstract

A sensory substitution device 100 comprising a spatial sensor configured to provide spatial sensor data indicative of a spatial representation of an environment in which the device is located. The device comprises a shape-change mechanism 108 operable to cause the device to change shape, based on the spatial sensor data, to provide kinaesthetic output to a user of the device. Wherein the kinaesthetic output is indicative of a target position 104 in the environment. A second such device comprises a sensor operable to output sensor data representative of an environment in which the device is being used by a user of the device and a mechanism operable to cause the device to change shape, based on the output data, with motion that is not plane-constrained to provide proprioceptive output to the user.
Need to check novelty before this filing date? Find Prior Art

Description

16 08 24 Field The present disclosure relates to sensory substitution devices. 5 Background Computer vision can identify objects. Simultaneous Localisation and Mapping (SLAM) can map and locate a person within an unknown environment. Both technologies can, in principle, benefit visually impaired people, such as blind people. However, existing devices do not adequately 10 communicate useful information derived using these technologies to visually impaired people. Summary According to first embodiments, there is provided a sensory substitution device comprising: a sensor operable, in use, to output sensor data representative of an environment in which 15 the sensory substitution device is being used by a user of the sensory substitution device; and a mechanism operable to cause the sensory substitution device to change shape, based on the output sensor data, whereby to provide proprioceptive output to the user, wherein the sensory substitution device is configured to be controlled using SLAM. 20 Brief Description of the Drawings Various embodiments will now be described, by way of example only, with reference to the accompanying drawings in which: Figure 1 shows a plan view an example of a sensory substitution device in three different configurations; 25           Figure 2 shows a perspective view of the example sensory substitution device shown in Figure 1; Figure 3 shows a perspective view of another example of a sensory substitution device; Figure 4 shows an exploded perspective view of the example sensory substitution device shown in Figure 3; 30           Figure 5 shows schematically a perspective view of another example of a sensory substitution device; and Figure 6 shows a schematic representation of example items associated with an example sensory substitution device. 16 08 24 Detailed Description As explained above, existing devices do not adequately communicate useful information derived using technologies such as computer vision and SLAM to visually impaired people. In 5 particular, such existing devices do not adequately convey such information non-visually. Some assistive technology products can identify objects using computer vision and can provide low-resolution alerts about obstacles. Such products include smart canes, smart glasses and smartphone applications which use audio cues and vibration. Audio cues allow users to find locations. However, this form of feedback is ambiguous and can obscure important environmental sounds and 10 social interactions. Vibratory sensations are cognitively distracting, can lead to short-term numbness, and have a low information bandwidth. While vibration may be effective for providing alerts, it is not as effective for conveying complex information such as direction or distance. Without significant advancements in the human-machine interface, smartphone applications, smart canes and smart glasses are not effective enough to approach the functionality of a guide dog. 15           Shape-changing, haptic interfaces for visual impairment are known. Known interfaces can change shape in the hand of a user to communicate information, such as which heading the user should walk on. Such known interfaces use Bluetooth™ beacons and Global Positioning System (GPS), rather than computer vision or SLAM, to accomplish this. Such interfaces also have limited shapechanging capabilities. For example, they may have a part that can rotate relative to another part and 20 slide forward and backward, whilst constrained to move in a single 2D plane, here called " plane-constrained". The inability of the mechanism to move out-of-plane, in other words out of that 2D plane and freely in the 3D space around the interface, restricts not only the amount of information that can be conveyed to the user but also the intuitiveness of use of such interfaces. Various other drawbacks to such existing interfaces will now be described. 25           GPS is generally only accurate to ±5 metres. This is not accurate enough to replace a guide dog. For example, a user cannot be directed to stay on a pavement. GPS also relies on receiving GPS signals from GPS satellites. This is not possible in some environments, such as environments that are deep underground. Bluetooth™ beacons need to be installed in the locations where the existing interfaces are 30 being used. This may involve changes to that environment and may mean that such interfaces can only be used in locations in which Bluetooth™ beacons have already been installed. Neither Bluetooth™ beacons nor GPS can be used to detect and navigate around temporary obstacles, such as dustbins. 16 08 24 Such existing interfaces are not optimised to direct users to point at objects in 3D space. Instead, they are optimised for directing a walking heading. Such existing interfaces use geared mechanisms and screw-driven actuators to actuate shape changes. As a result, such interfaces are relatively large and heavy, and have relatively low 5 responsiveness. The motion of such existing interfaces is not particularly organic and is not designed to be arrested by the grip of the user, in other words held tightly by the user. A user cannot use such existing interfaces to point at and identify objects. Various terms will be used in conjunction with examples described below. Such terms will now 10 be explained. The term "computer vision" is used herein to relate generally to identifying one or more objects or people from camera footage, or to make inferences about an environment such as identifying pavement versus road, or grass versus tarmac. The term "LIDAR", which stands for light detection and ranging, is used herein in relation to 15 technology used to measure distances and build a point-cloud of an environment. The term "point cloud" is used herein to mean a series of distance measurements derived from a LIDAR scan that can be used to build a 3D computer model of an environment. The term "haptic" is used herein, in relation to a device, to mean a device which communicates information through the touch sense. 20           The terms "proprioception", "kinaesthesia" and "kinesthesia" are used herein interchangeably to mean the sense of self-movement and body position. The terms "modality" and "sensory modality" are used herein interchangeably to mean a type of stimulus or what is perceived after such a stimulus. Examples of sensory modalities include, but are not limited to, smell, taste, touch, vision and hearing. 25           The term "sensory substitution" is used herein to mean passing information that is normally gathered by one sensory modality (such as vision) though the modality of another (such as touch). The terms "shape change", "change shape", "change of shape" and the like are used herein to mean a change to at least one property of a shape. A shape change may, for example, result in a change in at least one dimension of the shape, with the geometric shape remaining the same. For 30 example, a change of shape may encompass a cylindrical shape changing from having a first length to a second length. A shape change may, for example, result in a change of geometric shape. The terms "device" and "interface" are used interchangeably. 16 08 24 The term "spatial sensor" is used herein to mean any type of sensor that can be used to sense a spatial environment. The term "sensor" is used herein more generally to mean any type of sensor. The term "spatial sensor data" is used herein to mean any type of data provided or output by a spatial sensor. The term "sensor data" is used herein more generally to mean any type of data 5 provided or output by any type of sensor. The term "target position" is used herein to mean any position in 3D space that is a target for a sensory substitution device, a user of a sensory substitution device, or another device or person to locate. The term "forward-facing" is used herein to mean facing, in use, in a direction of a target 10 position. The term "SLAM", which stands for Simultaneous Localisation and Mapping, is used herein to relate to the process of using spatial sensor data to construct or update a map of an environment, whilst simultaneously keeping track of the location of a user within it. The term "plane-constrained" is used herein to mean freedom of movementwhich is confined 15 to move on parallel 2D planes. For example, plane-constrained motion may allow left and right, forwards and backwards, and rotation whilst remaining constrained to one or more parallel 2D planes, but not movement upwards and downwards as this would move out of the 2D planes and into 3D space. Motion that is not plane-constrained, may move out of a 2D plane and into 3D space. Such motion may be referred to as "out-of-plane" motion. 20          The term "shape-change mechanism" is used herein to mean a mechanism that changes a shape. The shape-change mechanism may comprise a "shape-change mechanism" which change its own shape and thereby the shape of a device in which it is incorporated. However, a shape-change mechanism may change the shape of a device in which it is incorporated without changing its own shape. 25           Examples described herein provide navigation technology in the form of a device that changes shape. Humans have a natural ability to interpret object shapes with their hands, such as whether an elongate object is curved to the right or to the left, for example. This ability is exploited to communicate navigation information non-visually. Such information is communicated via the touch sense and can be used to indicate 3D locations. The user can follow the direction of curvature thereby 30 directing the user to point in a desired direction. Such examples exploit the combination of the ability of humans (i) to sense the location of their extremities non-visually (proprioception) and (ii) to infer 3D locations by pointing with their hands. 16 08 24 Examples described herein also provide a handheld sensory substitution device which uses a shape-changing, haptic interface in combination with computer vision and / or LIDAR to direct a user towards one or more locations and / or one or more objects of interest by prompting the user to point their hand, and hence the device, in the direction of the object(s) and / or location(s) of interest. 5           Examples also provide a new human-machine interface that can direct users towards objects and / or locations of interest by passing information through the sense of touch via shape changes. In examples, the interface is in the form of a sensory substitution device which is the size of a small flashlight, and which bends in two or more dimensions whilst held in the hand, aiming or pointing in the desired direction. The user can sense which direction the device is pointed in via the shape change 10 and can intuitively follow this guidance. This non-visually conveys direction to users through the touch sense without any prior training or knowledge. Prompting a user to point in the desired direction works with their innate non-visual spatial interpretation capabilities. Examples, therefore, reduce an intractable problem of describing 3D navigation with audio cues or vibration sensations into an intuitive and straightforward experience for visually impaired people. 15           Examples described herein also provide a handheld device with forward-facing LIDAR, which scans the environment in which the device is located to build a local 3D computer model in the form of a point cloud. The point cloud is used in isolation, or is combined with computer vision using SLAM to construct a map of the local environment the user is walking through while simultaneously keeping track of the location of the user within that environment. Relevant navigation information is then 20 passed to the user though their touch sense using an advanced haptic interface. This augments the capabilities of GPS to provide enhanced and intuitive local guidance, for example keeping a person on a pavement, directing a person to a platform, or dealing with temporary obstacles such as other people or dustbins. This can be provided without modifications to infrastructure. The computer vision capability may, for example, identify bus stops, bus numbers and train doors. SLAM computations may 25 be performed on a connected device, such as a smartphone, to keep costs of the interface low and accessibility high. Referring to Figures 1 and 2, there is shown an example of a sensory substitution device (also referred to herein a "device" for convenience and brevity) in three different configurations. In Figure 1, the device is indicated with reference signs 100a, 100b, 100c in the first, second 30 and third configurations respectively. In use, the device 100a, 100b, 100c is held and gripped by a user. In Figure 1, the user is indicated with reference signs 102a, 102b, 102c when the device is in the first, second and third configurations respectively. 16 08 24 In use, the device 100a, 100b, 100c is located in an environment. The environment comprises a target position 104. The target position 104 is a position in 3D space, which is being targeted by the user 102a, 102b, 102c. The target position 104 may correspond to a target object and / or a target location. 5            In the first configuration, the device 100a is bent to the right and points at the target position 104. In the second configuration, the device 100b is straight and points at the target position 104. A straight device shape is easily sensed by the hand of the user 102b which provides unambiguous indication that the user 102b is pointing directly at the target 104. In the third configuration, the device 100c is bent to the left and points at the target position 104. 10            In this example, the device 100a, 100b, 100c is a handheld device. This provides a highly portable device, which can readily leverage proprioception. In use, the user 102a, 102b, 102c holds the device in their hand 100a, 100b, 100c like a flashlight. The user 102a, 102b, 102c can point the device 100a, 100b, 100c in any direction. In this example, the device 100a, 100b, 100c comprises a rear portion. In Figure 1, the rear 15 portion is indicated with reference signs 106a, 106b, 106c in the first, second and third configurations respectively. In this example, the device 100a, 100b, 100c comprises an exposed shape-change mechanism portion. In Figure 1, the exposed shape-change mechanism is indicated with reference signs 108a, 108b, 108c in the first, second and third configurations. The exposed shape-change mechanism 20 portion 108a, 108b, 108c is exposed in that it is at the surface of the device 100a, 100b, 100c. As will be explained in more detail below, other elements of the shape-change mechanism are within the device 100a, 100b, 100c. In this example, the device 100a, 100b, 100c comprises a gripping portion. In Figure 1, the gripping portion is indicated with reference signs 110a, 110b, 110c in the first, second and third 25 configurations. In this example, the gripping portion comprises multiple gripping regions, with a first one of the gripping regions being one side of the exposed shape-change mechanism 108a, 108b, 108c and the other one of the gripping regions being on the other side of the exposed shape-change mechanism 108a, 108b, 108c. The gripping portion 110a, 110b, 110c is designed and configured to be gripped by the user 102a, 102b, 102c. This results in the device 100a, 100b, 100c being convenient to 30 use. In addition, hands are the best-suited part of the body to receive and interpret the shapechanging output of the device 100a, 100b, 100c. In this example, the gripping portion 110a, 110b, 110c extends most of the length of the device 100a, 100b, 100c. The exposed shape-change mechanism 16 08 24 portion 108a, 108b, 108c may form part of the gripping portion 110a, 110b, 110c if it is intended to be gripped by the user 102a, 102b, 102c. In this example, the device 100a, 100b, 100c comprises a front portion. In Figure 1, the front portion is indicated with reference signs 112a, 112b, 112c in the first, second and third configurations 5 respectively. The rear portion 106a, 106b, 106c, the exposed shape-change mechanism portion 108a, 108b, 108c, the gripping portion 110a, 110b, 110c and the front portion 112a, 112b, 112c collectively define a body of the device 100a, 100b, 100c. The user 102a, 102b, 102c can direct the device 100a, 100b, 100c to find one or more target 10 objects and / or locations 104. Such object(s) 104 may be common (also referred to as "everyday") objects, or environmental features such as pavements and roads. The device 100a, 100b, 100c may comprise a microphone and / or one or more other devices to be used with the device 100a, 100b, 100c may comprise a microphone. In examples, the user 102a, 102b, 102c can control the device 100a, 100b, 100c using only their voice, for example though natural 15 language processing. In examples, the user 102a, 102b, 102c can therefore verbally describe their intentions to a controller (comprised in the device 102a, 102b, 102c or another device) using natural language processing. For example, the user 102a, 102b, 102c could ask the device 100a, 100b, 100c to find them a door, or take them to a particular train platform. This may, for example, help the user 102a, 102b, 102c navigate through public transport networks, without modifications to network 20 infrastructure. As such, in some examples, the target position 104 is identified based on processing a voice input using natural language processing. This provides an especially effective input modality for visually impaired people. The user 102a, 102b, 102c then points the device 100a, 100b, 100c in directions where the target object and / or location 104 is suspected to be positioned. 25            In this example, the device 100a, 100b, 100c comprises a spatial sensor. In Figure 1, the spatial sensor is indicated with reference signs 114a, 114b, 114c when the device is in the first, second and third configurations respectively. The spatial sensor 114a, 114b, 114c is configured to provide spatial sensor data indicative of a spatial representation of an environment in which the device 100a, 100b, 100c is located. 30            In this example, the spatial sensor 114a, 114b, 114c is in the front portion 112a, 112b, 112c of the device 100a, 100b, 100c. Such a spatial sensor 114a, 114b, 114c may be referred to as "forwardfacing". 16 08 24 In this example, the spatial sensor 114a, 114b, 114c comprises a camera. However, other types of spatial sensor may be used. Examples of other types of spatial sensor include, but are not limited to, LIDAR, ultrasonic and infrared sensors. As such, in some examples, the target position 104 is determined using computer vision and / or LIDAR. This enables the device 100a, 100b, 100c to be 5 used to identify and locate objects, to perform SLAM, and to map environments. However, ultrasound and can also be used to map environments, and infrared enables the device 100a, 100b, 100c to be used in the dark. The latter may be effective whether the device 100a, 100b, 100c is used by a visually impaired person or by a person who is not visually impaired. Through computer vision, the camera 114a, 114b, 114c 'looks' for the object(s) of interest 10   104. In this example, the device 100a, 100b, 100c and / or another device analyses the camera frame (in other words, an image captured by the camera 114a, 114b, 114c). The camera 114a, 114b, 114c can therefore capture data that can be interpreted to identify objects 104 using computer vision. In this example, the device 100a, 100b, 100c comprises object identification functionality. However, object identification functionality may be provided in another device to be used with the device 100a, 15    100b, 100c. More specifically, in this example, the device 100a, 100b, 100c is operable to identify objects when the device 100a, 100b, 100c is pointed at them. It can be more convenient and natural for the user 102a, 102b, 102c to point, compared with using a device such as a smartphone, to identify such objects. Using computer vision in this manner enables the user 102a, 102b, 102c to be located in 3D 20 space within an unknown local environment and enables the user 102a, 102b, 102c to navigate through the unknown environment using SLAM. Specifically, in this example, the device 100a, 100b, 100c is configured to be controlled using SLAM. This enables the device 100a, 100b, 100c to provide spatial guidance in any environment. Identified objects 104 may also be tracked. 25          The device 100a, 100b, 100c changes shape via a shape-change mechanism. The shape change mechanism is operable to cause the device 100a, 100b, 100c to change shape, based on the spatial sensor data provided by the camera 114a, 114b, 114c (or other type of spatial sensor), to provide kinaesthetic output to the user 102a, 102b, 102c of the device 100a, 100b, 100c. The kinaesthetic output is indicative of a target position 104 in the environment. As such, the device 100a, 30   100b, 100c indicates to the user 102a, 102b, 102c where the object 104 is in 3D space. If the object 104 is identified within the camera frame and if the device 100a, 100b, 100c is not already pointing at the object 104, the device 100a, 100b, 100c changes shape in the hand of the user 102a, 102b, 102c to point at the object 104. The user 102a, 102b, 102c can feel the change in 16 08 24 shape of the device 100a, 100b, 100c as a result of them gripping and holding the device 100a, 100b, 100c. More specifically, in this example, the user 102a, 102b, 102c can feel the change of shape of the device 100a, 100b, 100c in thegripping portion 110a, 110b, 110c of the 100a, 100b, 100c. Afterfeeling which direction the shape of the device 100a, 100b, 100c is pointed towards, the user 102a, 102b, 5    102c can move their hand in this direction. When moving their hand, and hence the device 100a, 100b, 100c, the user 102a, 102b, 102c is receiving kinaesthetic output. This kinaesthetic output is used to infer direction. The change of shape may be in at least one degree of freedom. This may enable the device 100a, 100b, 100c to indicate depth or distance. 10          The change of shape may be in at least two degrees of freedom. This may enable the device 100a, 100b, 100c to point in any direction and, therefore, indicate direction. The change of shape may be in three degrees of freedom. This may enable the device 100a, 100b, 100c to point in any direction, as well as indicate depth, distance or heading. In this example, the change of shape comprises bending of the device 100a, 100b, 100c. This 15 provides a natural-feeling shape change. In this example, the change of shape also comprises stretching and / or compressing of the device 100a, 100b, 100c. This enables the device 100a, 100b, 100c to indicate distance or heading. More generally, the shape change may be performed through bending and / or stretching in at least two degrees of freedom whilst the device 100a, 100b, 100c is held in the hand. For example, the 20 device 100a, 100b, 100c may be able to bend left, right, up and / or down. This enables the device 100a, 100b, 100c to point in any direction. The device 100a, 100b, 100c may be able to stretch forwards and compress backwards. The device 100a, 100b, 100c can indicate a distance (for example to the object 104) or heading based on the extent of stretching and / or compression. In this example, the device 100a, 100b, 100c comprises an elongate body, and the shape-25 change mechanism is operable to cause the elongate body of the device 100a, 100b, 100c to change shape by changing length. This enables intuitive pointing. In this example, the device 100a, 100b, 100c comprises a microcontroller (not visible) and the microcontroller is configured to control the shape-change mechanism based on the spatial sensor data. This can provide high-speed control when converting the spatial input into a shape-changing 30 output. The device 100a, 100b, 100c may notify the user 102a, 102b, 102c that the object 104 has been identified. Such notification may be via a vibratory alert. As such, the device 100a, 100b, 100c 16 08 24 may be operable to provide a vibratory output. This enables the device 100a, 100b, 100c to provide alerts and other signals to the user 102a, 102b, 102c. In some examples, the device 100a, 100b, 100c and / or another device comprises a loudspeaker. In examples, once the device 100a, 100b, 100c has directed the user 102a, 102b, 102c 5 to point in a direction of interest, the device 100a, 100b, 100c verbally communicates the distance to the object of interest 104. For example, whilst pointed at a coffee cup, the device 100a, 100b, 100c could communicate "the cup is 60 centimetres away". Such verbal communication may be an alternative to, or may be in addition to, the stretching and / or compression to indicate distance or heading. As such, in some examples, the device 100a, 100b, 100c is operable to communicate verbally 10 a distance from the device 100a, 100b, 100c to the target position 104. Once the user 102a, 102b, 102c is pointed at the target position 104, in 3D space, two out of three dimensions are removed for locating the target position 104. The only remaining dimension is distance. If the device 100a, 100b, 100c is already pointed directly at the object 104, the device shape is straight. In this example, the shape-change mechanism is configured such that the device 100a, 15    100b, 100c is substantially straight when the device 100a, 100b, 100c is pointed at the target position 104 in the environment. This can be best seen in the device 100b in the second configuration. The term "substantially straight" is used herein to mean generally linear from the perspective of the user 102a, 102b, 102c, for example as opposed to being curved. The device 100a, 100b, 100c may therefore be substantially straight without being perfectly linear. This enables the device 100a, 100b, 100c to 20 provide an unambiguous indication of 3D spatial direction or 3D object location. In this example, at least a portion of the device 100a, 100b, 100c is substantially cylindrical. Specifically, in this example, at least the gripping portion 110a, 110b, 110c of the device 100a, 100b, 100c is substantially cylindrical. The symmetry and the proprioceptive output enable the device 100a, 100b, 100c to be used in any orientation. The term "substantially cylindrical" is used herein to mean 25 having generally parallel sides and having a generally circular cross-section. The device 100a, 100b, 100c may therefore be substantially cylindrical without being perfectly cylindrical. For example, as can be seen in Figures 1 and 2, the exposed shape-change mechanism portion 108a, 108b, 108c of the device 100a, 100b, 100c may result in a portion the device 100a, 100b, 100c deviating from being perfectly cylindrical while still being substantially cylindrical. Additionally, in this example, the back 30 portion 106a, 106b, 106c and the front portion 112a, 112b, 112c of the device 100a, 100b, 100c are not cylindrical. In this example, the device 100a, 100b, 100c is operable to be controlled by at least one person other than the user 102a, 102b, 102c while the user 102a, 102b, 102c is using the device 100a, 100b, 16 08 24 100c. For example, the device 100a, 100b, 100c may be controlled by voice commands by the other person. The device 100a, 100b, 100c may alternatively or additionally be controlled by a device of the other person. For example, the other person may use a smartphone to identify the target position 104. This enables another person, such as a remotely located person, to help the user 102a, 102b, 5    102c navigate. Although, in this example, the device 100a, 100b, 100c analyses the camera frame, one or more other devices may perform such analysis, in addition to or as an alternative to the device 100a, 100b, 100c performing such analysis. For example, the device 100a, 100b, 100c may pair with a mobile computing device which performs the SLAM and computer vision computations. Examples of such 10 mobile computing devices include, but are not limited to, smartphones, tablet computing devices, and laptop computing devices. The device 100a, 100b, 100c may pair with such other devices wirelessly, for example using Bluetooth™, or otherwise. The use of such other devices helps keeps costs of the device 100a, 100b, 100c low and enables over-the-air updates to the device 100a, 100b, 100c without the device 100a, 100b, 100c having capability to receive such updates directly. 15            If the user 102a, 102b, 102c grips the device 100a, 100b, 100c tightly such that it cannot change shape, the user 102a, 102b, 102c will still feel the force indicating where the shape-change mechanism is trying to bend the device 100a, 100b, 100c towards, without damaging the shapechange mechanism. Referring to Figures 3 and 4, there is shown another example of a sensory substitution device 20   100. In this example, the gripping portion 110 has a single gripping region between the exposed shape-change mechanism portion 108 and the front portion 112. However, the user of the device 100 may nevertheless also use the rear portion 106 and / or the exposed shape-change mechanism portion 108 to grip the device 100. The rear portion 106 and / or the exposed shape-change mechanism portion 25    108 may therefore be considered to form part of the gripping portion 110. In this example, the shape changes are accomplished by three electrical motors 116 arranged on the front portion 112 of the device 100. In this example, the electrical motors 116 are equally spaced circumferentially around, and protrude outwardly from, the front portion 112 of the device 100. A different number of motors can be used in other examples. The position of the motor(s) can be 30 different in other examples. In this example, each motor 116 variably tensions a respective one of three antagonistic tendons 118 which run up the length of the device 100. A different number of antagonistic tendons can be used in other examples. In this example, each motor 116 drives a respective pulley 120. Only 16 08 24 two pulleys 120 are shown in Figure 4, it being understood that three pulleys 120 are present in this specific example. In this example, each antagonistic tendon 118 is drawn in around its respective pulley 120 when its respective motor 116 is driven in one direction and is allowed to be drawn out from its respective pulley 120 as its respective motor 116 is driven in the other direction. Alternatively, 5 all three pulleys 120 may draw in or pull out simultaneously to cause the device 100 to retract or extend along its length. More generally, at least two electrical motors 116 and at least two antagonistic tendons 118 may be used. However, if three independent antagonistic tendons 118 are used, the device 100 can compress and elongate, as well as bend. 10           As such, in this example the shape-changing device 100 is driven by motors 116 and antagonistic tendons 118. This may be more compact, faster-moving and cheaper than other possible mechanisms. In this example, the device 100 comprises a shape-changing membrane, the surface of which corresponds to the external shape-change mechanism portion 108. In this example, the shape-15 changing membrane comprises alternating flexible vertebrae 122 and rigid vertebrae 124. In this example, the flexible and rigid vertebrae 122,124 are ring-shaped. In this example, each antagonistic tendon 118 passes through a respective one of three holes in each of the gripping portion 110, the flexible vertebrae 122, the rigid vertebrae 124, and the end portion 106. In Figure 4, one such hole is indicated with reference sign 126, it being understood that 20 other holes are shown in in Figure 4. This arrangement results in a device body which can change shape in three degrees of freedom. In particular, the device body can bend left, right, up, down, forwards, and backwards, or anywhere in between. In this example, the flexible vertebrae 122, and hence the shape-change mechanism, 25 comprises elastomeric material. The elastomeric material enables the shape-changing membrane to compress, expand, and bend. The elastomeric material also reduces the likelihood of, or even prevents, the shape-change mechanism from pinching the hand of the user when the device changes shape. In this example, the motors 116, antagonistic tendons 118, flexible vertebrae 122 and rigid 30 vertebrae 124 all form part of the shape-change mechanism. As can best be seen in Figure 4, in this example, the end portion 106, the gripping portion 110, the flexible vertebrae 122 and the rigid vertebrae 124 are hollow. This provides a lightweight device 16 08 24 100, which uses relatively small amount of material, while still being sufficiently rigid for the intended use. The hollow centre of the device 100 can comprise other components of the device 100. In this example, the shape-change mechanism is configured to cause the device 100 to return to an equilibrium shape when the device 100 is powered off. In this example, when the device 100 is 5 powered off, the elastomeric material relaxes to its equilibrium state, and the motors 116 allow the antagonistic tendons 118 to unwind from the pulleys 120 such that the device 100 returns to being straight. The device 100 has, in effect, a self-righting design, which returns the device 100 to the straight, equilibrium configuration when the device 100 is powered off. This enables the device 100 to be more ergonomic and the device 100 better fits in a pocket of the user if the device 100 runs out 10 of power. Referring to Figure 5, there is shown another example of a sensory substitution device 100. Figure 5 demonstrates how the device 100 can change shape with motion that is not plane-constrained, whereby to provide proprioceptive output to a user. In Figure 5, a first 2D plane is indicated with reference sign 126 and using broken lines, and a second 2D plane is indicated with 15 reference sign 128 and using solid lines. In this example, the device 100 can change shape with motion that is not plane-constrained. In particular, the motion is not constrained to being in only one of the first or second planes 126,128. Instead, the motion can be in either of the first or second planes 126, 128, or outside of the first and second planes 126,128 in the 3D space surrounding the device 100. Referring to Figure 6, at item 200, an error is calculated as a difference between a desired 20 pointing direction 202 and an actual pointing direction 204. In this example, the actual pointing direction 204 is determined by a device spatial sensor 206. In this example, a controller 208 controls a shape-change mechanism 210 to change the shape of the device. At item 212, the user senses the device shape and, at 214, moves their body based on the sensed device shape. The process represented in Figure 1 can be repeated until the actual pointing direction 204 matches the desired 25 pointing position 202, at which point the output of error item 200 is zero. When this is achieved, the device is pointing at the target object(s) and / or location(s) and the shape of the device is substantially straight. In specific examples, the device shape updates twenty times per second, based on SLAM and / or computer vision input, and user movements. As such, in this example, the device is configured to change shape based on closed-loop feedback, where the closed-loop feedback is based on a 30 difference between the target position 202 in the environment and a position 204 in the environment at which the device is pointed. The shape of the device may update multiple times per second, giving closed-loop feedback. Various additional examples will now be described. 16 08 24 In some examples, the user 102a, 102b, 102c can direct the device 100,100a, 100b, 100c to navigate to locations 104 using only their voice though natural language processing. The device 100, 100a, 100b, 100c pairs with another device, such as a smartphone. Applications, such as mapping applications, are also interfaced. The user 102a, 102b, 102c walks with the device 100, 100a, 100b, 5    100c, holding the device 100, 100a, 100b, 100c forward-facing like a flashlight. In examples, the forward-facing camera 114,114a, 114b, 114c feeds video to a computer vision application to identify important navigational features 104 which are typically identified by a guide dog. Examples of such navigational features include, but are not limited to, pavements and zebra crossings. The user 102a, 102b, 102c is kept from walking into a road by navigational guidance though the touch sense. 10           In some examples, the user 102a, 102b, 102c can use the device 100, 100a, 100b, 100c to 'read' text using text-to-speech functionality. For example, the user 102a, 102b, 102c could aim the device 100,100a, 100b, 100c at a paragraph of text and use their voice to command the device 100, 100a, 100b, 100c to read the text. In some examples, the user 102a, 102b, 102c can use the forward-facing camera 114, 114a, 15    114b, 114c on the device 100, 100a, 100b, 100c to magnify items. Images of such items can be streamed to a display, such as a smartphone screen or smart glasses display, for closer inspection by the user 102a, 102b, 102c. In some examples, the device 100, 100a, 100b, 100c can be used in a mode in which the user 102a, 102b, 102c aims the device 100, 100a, 100b, 100c at an object of interest 104, initiates a 20 command (for example by pressing a button and / or issuing a voice command), and the device 100, 100a, 100b, 100c calls out the largest object 104 identified in the camera frame. In some examples, the device 100,100a, 100b, 100c does not have any forward-facing sensors 114,114a, 114b, 114c. Instead, the device 100,100a, 100b, 100c is a shape-changing, human-machine interface that can be used to convey directional information based on sensory information provided 25 by other technologies. Examples of such other technologies include, but are not limited to, a smartphone, Bluetooth™ beacons, smart glasses, GPS, or wearable technology. In some examples, the device 100, 100a, 100b, 100c is handheld in that the weight of the device is held off the ground only by the hand of the user 102a, 102b, 102c. However, the device 100, 100a, 100b, 100c may be supported by another object in other examples. Examples of other such 30 objects include, but are not limited to, guide canes and walking sticks. In some examples, the device 100,100a, 100b, 100c is used by a visually impaired user 102a, 102b, 102c. However, the device 100, 100a, 100b, 100c may be used by a person who is not visually impaired.

Claims

16 08 241. A sensory substitution device comprising:a sensor operable, in use, to output sensor data representative of an environment in which5 the sensory substitution device is being used by a user of the sensory substitution device; anda mechanism operable to cause the sensory substitution device to change shape, based on the output sensor data, whereby to provide proprioceptive output to the user,wherein the sensory substitution device is configured to be controlled using SLAM.10    2. A sensory substitution device according to claim 1, wherein the change of shape is in at leasttwo degrees of freedom.

3. A sensory substitution device according to claim 1 or 2, wherein the change of shape is in three degrees of freedom.

154. A sensory substitution device according to any of claims 1 to 3, wherein the sensory substitution device comprises an elongate body and wherein the mechanism is operable to cause the elongate body of the sensory substitution device to change shape.20   5. A sensory substitution device according to any of claims 1 to 4, wherein the sensorysubstitution device is a handheld sensory substitution device.

6. A sensory substitution device according to any of claims 1 to 5, wherein the change of shape comprises bending of the sensory substitution device.

257. A sensory substitution device according to any of claims 1 to 6, wherein the change of shapecomprises stretching and / or compressing of the sensory substitution device.

8. A sensory substitution device according to any of claims 1 to 7, wherein the proprioceptive 30 output is indicative of a target position in the environment.

9. A sensory substitution device according to claim 8, wherein the proprioceptive output is to prompt the user to point the sensory substitution device in a direction of the target position.16 08 2410. A sensory substitution device according to claim 8 or 9, wherein the sensory substitution device is configured to change shape based on closed-loop feedback, the closed-loop feedback based on a difference between the target position and a position in the environment at which the sensory 5 substitution device is pointed.

11. A sensory substitution device according to claim 10, wherein the sensory substitution device is configured to change shape multiple times per second based on the closed-loop feedback.10   12. A sensory substitution device according to any of claims 8 to 11, wherein the mechanism isconfigured such that the sensory substitution device is substantially straight when the sensory substitution device is pointed directly at the target position.

13. A sensory substitution device according to any of claims 8 to 12, wherein the mechanism is 15 configured such that the sensory substitution device is curved when the sensory substitution device is not pointed directly at the target position.

14. A sensory substitution device according to any of claims 8 to 13, wherein the device is operable to notify the user in response to the target position being identified.2015. A sensory substitution device according to claim 14, wherein the device is operable to notify the user by providing a vibratory output.

16. A sensory substitution device according to any of claims 8 to 15, wherein the target position 25 is a position determined using computer vision, LIDAR, ultrasound, and / or infrared.

17. A sensory substitution device according to any of claims 8 to 16, wherein the sensory substitution device is operable to communicate verbally a distance from the sensory substitution device to the target position.3018. A sensory substitution device according to any of claims 1 to 17, wherein the mechanism comprises elastomeric material.16 08 2419. A sensory substitution device according to any of claims 1 to 18, wherein the mechanism is configured to cause the sensory substitution device to return to an equilibrium shape when the sensory substitution device is powered off.5   20. A sensory substitution device according to claim 19, wherein the equilibrium shape issubstantially straight.

21. A sensory substitution device according to any of claims 1 to 20, wherein the sensory substitution device comprises a microcontroller and wherein the microcontroller is configured to 10 control the mechanism based on the output sensor data.

22. A sensory substitution device according to any of claims 1 to 21, wherein the sensory substitution device is operable to be controlled by at least one person other than the user while the user is using the sensory substitution device.1523. A sensory substitution device according to any of claims 1 to 22, wherein the sensor comprises a spatial sensor.

24. A sensory substitution device according to claim 23, wherein the spatial sensor comprises a 20 camera.

25. A sensory substitution device according to any of claims 1 to 24, wherein the mechanism is operable to cause the sensory substitution device to change shape with motion that is not plane-constrained.

Citation Information

Patent Citations

  • Walking assisting system for vision challenging person

    JP2010057593A

  • Flexible device for guiding a user

    WO2016116182A1