Orientation assistance system

Abstract

The present invention proposes an orientation assistance system, the orientation assistance system comprising: an acquisition device that acquires a real or virtual visual environment; a non-visual human / machine interface device; and a processing device that processes a digital representation of the visual environment to provide an electrical control signal for controlling the non-visual interface, characterized in that - the human / machine interface device comprises a bracelet having a tactile area, the surface area of the tactile area being between 60×60 mm and 150×150 mm, having a set of N×M active styli, where N is between 5 and 100 and M is between 10 and 100, - the processing of the digital representation by the processing device includes periodically extracting at least one pulsed digital activation pattern of a subset of the styli of the tactile area.

Description

Technical Field

[0001] The present invention relates to the field of orientation assistance for visually impaired people or people moving around in an environment with very low visibility (e.g., firefighters moving around in a smoky building or soldiers moving around in the dark).

[0002] Various solutions are known, ranging from guide dog assistance to marking the ground with orientation assistance tapes, implanting sound markers, or actually using a cane that allows detecting obstacles.

[0003] Recently, GPS applications have also been developed, which allow blind and visually impaired people to move more independently by making them aware of roads, points of interest at their location, intersections, bus stops, etc. The GPS application allows the user of the GPS application to receive information related to bus times in real time via verbal notifications on their smart phone, tells the user their stop, and takes the user to where they want to go once they leave the vehicle. During the user's journey, ad hoc information is provided so that the user does not get lost, avoid obstacles, encounter difficulties, and take the correct bus.

[0004] The use of tactile information transmission has also been proposed, for example, in the form of a health tracker. Tactile technology uses touch to convey items of information. WearWorks Corporation has proposed a smart bracelet called "Wayband" for guiding blind people. The user starts operating by downloading the application to an associated smart phone and stating the desired address. The bracelet connected to the GPS system guides the user to their destination. When the user deviates from their route, the bracelet vibrates. Once on the correct path, the bracelet stops vibrating. The sensitive, more intuitive, and non-intrusive tactile language alleviates hearing, which is a sense overused by visually impaired people.

[0005] Another solution sold by SUNU Company involves a bracelet that uses high-frequency waves to facilitate echolocation in space. Background Art

[0006] Prior Art

[0007] According to the prior art, a patent application US20180189567A1 for a system for assisting the blind is known. The system includes a plurality of cameras and a tactile belt worn by the user, and the tactile belt includes a plurality (two or more) of tactile transducers spaced at a certain distance. The tactile belt is worn in such a way that the user's hand is free for other tasks. The system uses its cameras, its depth processing algorithm, and its object recognition algorithm (hardware and / or software) to identify a limited number of objects important for navigation. The spatial position of each object considered important is provided to the user by correspondingly changing the output to the tactile transducers. The system is designed to identify and indicate objects as general objects, recognized objects, and possible obstacle objects. The system can also optionally provide audio information or tactile graphic display information also related to these objects.

[0008] This solution is not satisfactory because the multiple transducers act on different parts of the body, which does not allow for an intuitive and simple understanding of the information and requires significant attention to distinguish the deformation of the support areas of the transducers.

[0009] Patent US20150125831A1 describes another example of a solution that includes a set of cameras that can sense an image in front, and the image is processed thereby and used to output a three-dimensional representation of the image on a tactile pin array. This makes it possible to identify objects and obstacles in the area in front of the device, and the pin array can also be used as an adaptive Braille reader. A specific pin assembly using a stepping micro-motor is envisioned, and the global system provides various functions for visually impaired users, including navigation, face recognition, connection to a wireless network, and the ability to use various input / output methods.

[0010] Disadvantages of the Prior Art

[0011] The solutions of the prior art are not suitable for sending rich and comprehensive information related to the visual environment in a tactile form.

[0012] Tactile information is usually too basic and not very suitable for representing true orientation assistance by understanding a complex environment in a sufficiently complete way. Summary of the Invention

[0013] Solution Provided by the Present Invention

[0014] To overcome these drawbacks, the present invention relates to an orientation assistance system in its most general sense.

[0015] Advantageously, the acquisition device includes at least one image sensor that can be carried by the wearer of the bracelet, and a processing device for generating a digital depth map. The image sensor can be formed by a stereo camera or a single camera that processes successive images to determine the depth map, or in fact by a 3D scanner or a lidar for providing a point cloud.

[0016] According to a particular embodiment, the digital representation processing device calculates according to a representation model selected from a series of different representation models. Different models make it possible to provide processing suitable for a specific context (for example, sending trajectory-related information, or a haptic representation of the environment and its points of interest, or the perception of the surrounding volume) and allow the user to select one of these information modes according to their needs and preferences, or in fact to automate the selection with the aid of learning algorithms. According to a variant, the system further includes a server that can communicate with each of the individual devices to receive geolocation acquisition data of the digital environment and stored data of the geolocation digital model of the environment, and the server can send the digital model to the individual devices according to their location.

[0017] This variant makes it possible to reduce the computational processing of the acquired images and to share useful information among the various users who have traversed the area.

[0018] According to a particular embodiment, the processing device for processing the digital representation is designed to periodically extract a series of successive pulse digital activation patterns of a subset of the pins of the haptic area to provide continuous haptic information during a period of time.

[0019] According to a variant, one of the digital patterns includes a pulse activation command for aligning the pins so as to form an angle corresponding to the direction of movement relative to a reference direction of the visual environment together with the reference axis of the bracelet.

[0020] According to another variant, one of the digital patterns includes a pulse activation command for configuring the pins corresponding to the projection of the main point of interest of the digital representation of the visual environment in the horizontal plane.

[0021] According to a particular embodiment, the model of the pulse digital activation pattern of the subset of the pins of the haptic area is determined based on the attribution relationship of the visual environment to a pre-recorded environmental category.

[0022] Advantageously, the model of the pulse digital activation pattern of the subset of the pins of the haptic area is determined based on the user's experience level. Description of the Drawings

[0023] The present invention will be more clearly understood after reading the following detailed description of non - limiting embodiments of the invention with reference to the accompanying drawings, in which:

[0024] Figure 1 is a schematic diagram of the present invention.

[0025] Figure 2 shows the inner surface of a bracelet according to the present invention.

[0026] Figure 3 shows the functional architecture. Detailed Embodiments

[0027] General Description of the Hardware Architecture

[0028] The various components of the system according to the present invention include a bracelet (1), a frame (2) worn on the user's forearm (or alternatively in the thigh position), and in the described embodiment, the frame (2) is in the form of a spectacle frame, for example, an image sensor is provided in the extension of each branch to provide a stereoscopic image. The frame (2) is equipped with a georeference or geolocation sensor, and / or a module providing an indication of the orientation of the frame relative to magnetic north.

[0029] These different elements communicate with the user's paired smartphone (3) in BLE Bluetooth mode. The smartphone (3) ensures some computer processing by means of an application and communicates with a server via radio frequency communication of the 3G, 4G or 5G type or Wi - Fi.

[0030] Of course, the phone (3) can be replaced by a computer, a tablet, and more generally a calculator.

[0031] Description of the Wristband

[0032] Figure 2 shows an embodiment of the bracelet for implementing the present invention.

[0033] The bracelet includes a flexible housing (10) provided with straps (11, 12, 13, 14) for fixing around the forearm (or alternatively around the waist area).

[0034] The bracelet includes a matrix of 5×12 styli (15) which, when the electromagnetic actuator is activated by an electrical signal, can each be activated in a pulsed manner between a retracted rest position and a pulsed erected position for up to a fraction of a second.

[0035] The configuration of the tactile surface is not limited to a rectangular area with regularly distributed styli (15).

[0036] Digital Processing

[0037] The first step (100) consists in acquiring two synchronized image streams using two laterally offset sensors on the user's frame (2).

[0038] The pair of sensors forms a stereo camera oriented towards the scene visible to the user.

[0039] The camera is connected via a radio frequency link to a smartphone (3) comprising a computing unit that allows the processing (110) of the images originating from the two sensors. This processing makes it possible to calculate a depth map based on the two images and the position of the camera in space. The camera can also be connected to a computer or a smartphone via a wireless (Bluetooth, Wi-Fi, etc.) or wired connection.

[0040] A possible image processing method is a series of algorithms that make it possible to extract the depth map of the scene and then use this result together with the associated left and right side images to deduce therefrom the changes in the position and orientation of the camera between two consecutive recording instants (usually spaced one sixtieth of a second apart).

[0041] Then two processes are carried out.

[0042] First, a software module (120) controls the recording of the images or visual features constituting the point of interest and the storage in memory of the position of the frame (2) when the said point is observed. The module provides a database of points of interest that is sent by the user's phone (3) to a server that stores the said geolocation data in a database shared among all users.

[0043] The storage of a new entry in the database thus formed is preferably triggered according to collection criteria that determine the redundant information content with respect to other entries in the database. Other criteria can be used, such as manual triggering of the storage by the user, calculation of the physical distance between each position of the point of interest, or the time period elapsed between two storage instances.

[0044] The database E1 is thus constructed. The database E1 contains a set of reference positions associated with features or images. During the use of the system in the same area, the database E1 is also used to reposition the frame (2).

[0045] In parallel, a software module (140) calculates the depth map and the parameters of the stereo system are used to generate a point cloud by projecting each pixel of the image to obtain the coordinates of the points in space. Then, the points in space are subjected to a change of reference frame using information related to the position of the headset in space, said information originating from the ranging module C1, in order to place all the points sensed during the initialization phase in a common fixed reference frame.

[0046] These point sets are merged to create a dense model (cartography) of the operating area while reducing the amount of redundant information.

[0047] This process can be performed, for example, by traversing all the points and merging those identified as being close to each other based on distance or actually using a truncated signed distance function (TSDF) volume.

[0048] A subsequent step (150) includes generating a network of the point set in three dimensions. The network consists of connected triangles, thereby modeling the surface of the operating area.

[0049] The point set (voxels) is then processed to calculate the activation pattern of the needles (15) of the bracelet (1).

[0050] To achieve this, the system includes a library of different processing options.

[0051] The first processing method (200) includes determining the direction of movement or simplifying the trajectory in the form of continuous segments calculated based on the voxels corresponding to the obstacles, and recalculating its orientation with respect to the orientation of the frame (2). The result is the activation pattern of the needles, which is recalculated periodically, for example once per second, in order to send haptically the direction or trajectory to be followed.

[0052] The second processing method (210) includes calculating the projection of the voxels on a horizontal plane to determine a low-resolution digital map (resolution corresponding to the number of needles), and applying the pulse pattern corresponding to the map to the bracelet, which is oriented according to the orientation of the frame (2).

[0053] The third processing method (220) includes calculating successive transverse planes, and sending a sequence of patterns corresponding to the low-resolution transverse planes in a chronological order representing intervals along a longitudinal axis perpendicular to the transverse planes. Thus, bursts of patterns separated by periods of rest are applied to the needles, thereby allowing the user to identify the formation of their environment through successive haptic sensations.

[0054] The fourth processing method (230) includes calculating a low-resolution black-and-white image of a stereoscopic image, and calculating a pattern based on the low-resolution image to control the periodic activation of the needles (15).

[0055] The selection of one of the processing methods (200 to 230) can be performed by the user through manual or voice control. It can also be performed automatically according to the type of environment (obstacle density, known environment versus new environment, etc.) or according to the user's level of learning (some processing methods require higher sensitivity and experience than others).

[0056] According to the variant example, the wristband computer will be connected via a cable, Bluetooth or Wi-Fi. The connection of the computer to the server will be performed via Wi-Fi or 3G, 4G or 5G.

Claims

1. A directional assistance system, the directional assistance system comprising: An acquisition device, the acquisition device acquiring a real or virtual visual environment; A non-visual human / machine interface device; And a digital representation processing device, the digital representation processing device processing a digital representation of the visual environment to provide an electrical control signal for controlling the non-visual human / machine interface device, characterized in that, - The non-visual human / machine interface device includes a bracelet having a single tactile area, the surface area of the single tactile area being between 60×60 millimeters and 150×150 millimeters, having a set of N×M active styli, where N is between 5 and 100, and M is between 10 and 100, and each stylus can be activated in a pulsed manner between a retracted rest position and a pulsed erected position for up to a fraction of a second when an electromagnetic actuator is activated by an electrical signal, - The digital representation processing device processing the digital representation includes periodically extracting at least one pulsed digital activation pattern of a subset of the styli of the single tactile area, wherein the digital representation processing device is calculated according to a representation model selected from a series of different representation models, wherein the series of different representation models includes: The first processing method (200) includes determining a movement direction or a simplified trajectory in the form of continuous segments calculated based on voxels corresponding to obstacles, and recalculating the orientation of the obstacles relative to the frame (2) so as to haptically send the direction or trajectory to be followed, The second processing method (210) includes calculating the projection of the voxels on a horizontal plane to determine a low-resolution digital map that matches the number of styli, and applying a pulsed pattern corresponding to the digital map to the bracelet oriented according to the direction of the frame (2), The third processing method (220) includes calculating successive transverse planes, and sending a sequence of patterns corresponding to the low-resolution transverse planes in a time sequence representing intervals along a longitudinal axis perpendicular to the transverse planes, so that bursts of patterns separated by rest periods are applied to the styli, thereby allowing the user to identify the formation of the visual environment through successive haptic sensations, The fourth processing method (230) includes calculating a low-resolution black-and-white image of a stereoscopic image, and calculating a pattern based on the low-resolution black-and-white image to control the periodic activation of the styli.

2. The orientation assistance system according to claim 1, characterized in that, The acquisition device includes: at least one image sensor that can be carried by the wearer of the bracelet; and a processing device for generating a digital depth map.

3. The orientation assistance system according to claim 1 or claim 2, characterized in that The orientation assistance system further includes a server, the server being capable of communicating with each of the individual devices to receive geolocation acquisition data of the digital environment and stored data of the geolocation digital model of the digital environment, and the server being capable of sending the geolocation digital model to the individual devices according to its location.

4. The orientation assistance system according to claim 1 or claim 2, characterized in that, The digital representation processing device for processing the digital representation is designed to periodically extract a series of continuous pulsed digital activation patterns of a subset of the styli of the single tactile area to provide continuous haptic information during a period of time.

5. The orientation assistance system according to claim 1 or claim 2, characterized in that, One of the at least one pulsed digital activation pattern includes a pulsed activation command that aligns the stylets to form an angle with the reference axis of the bracelet corresponding to the direction of movement relative to the reference direction of the visual environment.

6. The orientation assistance system according to claim 1 or claim 2, characterized in that, One of the at least one pulsed digital activation pattern includes a pulsed activation command that configures the stylets corresponding to the projection of the main focus of the digital representation of the visual environment in the horizontal plane.

7. The orientation assistance system according to claim 1 or claim 2, characterized in that, The pulsed digital activation pattern model of the subset of stylets of the single tactile area is determined based on the attribution relationship of the visual environment with respect to a pre-recorded environmental category.

8. The orientation assistance system according to claim 1 or claim 2, characterized in that, The pulsed digital activation pattern model of the subset of stylets of the single tactile area is determined based on the user's experience level.

Images