Orientation assistance system

Abstract

To provide an orientation assistance system suited to understanding of a complex environment on the basis of transmission of visual information as haptic information.SOLUTION: An orientation assistance system comprises acquisition means for acquiring a real or virtual visual environment, non-visual human / machine interface means, and processing means for processing the digital representation of the visual environment to provide an electrical control signal for controlling a non-visual interface. The human / machine interface means consists of a bracelet having a haptic region with a surface area from 60 mm×60 mm to 150 mm×150 mm, with a set of N×M active spikes, where N is from 5 to 100 and M is from 10 to 100. The processing means for processing the digital representation is configured to periodically extract at least one pulsed digital activation pattern of a subset of the spikes of the haptic region.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to the field of orientation assistance for the visually impaired or for people moving around in very poor visibility environments, for example firefighters moving around in smoke-filled buildings or soldiers moving around in the dark. [Background technology]

[0002] Various solutions are known, ranging from the aid of guide dogs, to marking the ground with strips of orientation aids, to embedding sonic beacons or even walking sticks that allow for the detection of obstacles.

[0003] More recently, GPS applications have also been developed to allow blind and partially sighted people to be more independent in their travels by showing them routes, points of interest (POIs), intersections, bus stops, etc. Users of GPS applications can receive real-time, spoken-language notifications on their smartphones about bus departure times, bus stop directions, and information to take them where they want to go if they leave the transport. During their journey, ad hoc information is provided to help users avoid getting lost, avoid obstacles, avoid difficulties, and catch the right bus.

[0004] Further proposals have included the use of haptic information transmission, for example in the form of fitness trackers. Haptic technology uses the sense of touch to convey information. The company WearWorks proposes a smart bracelet called "Wayband" that guides blind people. The user begins by downloading the application onto a shared smartphone and then precisely stating the desired address. The bracelet, connected to a GPS system, guides the user to their destination. If the user makes a mistake about the route, the bracelet vibrates; once they are on the correct path, it stops vibrating. A sensitive, more intuitive, and unintrusive tactile language frees up hearing, a sense that is overused by people with low vision.

[0005] Another solution marketed by the company SUNU relates to a bracelet that uses radio frequency waves to facilitate echolocation in space.

[0006] The prior art is known from US Pat. No. 6,299,499, which relates to a system for assisting blind people, comprising a plurality of video cameras and a haptic band worn by a user, the haptic band including a plurality (two or more) of spaced haptic transducers. The haptic band is worn so that the user's hands are free for other tasks. The system uses video cameras, depth processing algorithms, and object recognition algorithms (hardware and / or software) to identify a limited number of objects important for navigation. The spatial location of each important object of interest is accordingly provided to the user by varying the output directed to the haptic transducers. The system is designed to identify and indicate objects as general objects, specific objects, or potential obstacles. Furthermore, the system can optionally provide audio information or tactile graphic display information further related to the aforementioned objects.

[0007] The solutions just described are not sufficient because multiple transducers act on different parts of the body, which does not allow for an intuitive and concise understanding of the information and requires considerable care to distinguish between deformations in the transducer's area of ​​action.

[0008]

[0005] US Patent No. 5,999,949 describes another example of a solution involving a set of cameras capable of sensing an image in front of the device, which is then processed and used to output a three-dimensional representation of the image onto an array of tactile pins. While what has just been described allows for the recognition of objects and obstacles in the area located in front of the device, the pin array can also be used as an adaptive Braille reader. While a specific pin assembly using step micromotors is discussed, the global system provides multiple functions to low-vision users, including navigation, facial recognition, wireless network connectivity capabilities, and various input / output means. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] US Patent Application Publication No. 2018 / 0189567 [Patent Document 2] US Patent Application Publication No. 2015 / 0125831 Summary of the Invention [Problem to be solved by the invention]

[0010] Prior art solutions are not suitable for conveying rich and comprehensive information about the visual environment in a tactile form.

[0011] In general, haptic information is too basic and inadequate for understanding complex environments in a manner that is complete enough to represent practical orientation aids. [Means for solving the problem]

[0012] In order to overcome the just mentioned disadvantages, the present invention, according to its most general meaning, relates to an orientation assistance system according to claim 1.

[0013] Advantageously, said acquisition means consist of at least one image sensor capable of being carried by the wearer of said bracelet and processing means for generating a digital depth map. Said image sensor may be formed by a stereoscopic camera, or a single camera with processing of successive images to determine the depth map, or indeed by a 3D scanner, or a lidar providing a point cloud.

[0014] According to a particular embodiment, said digital representation processing means are calculated depending on a representation model selected from a set of different representation models, which allow the user to select one of the modes of information just mentioned depending on the user's needs and preferences, or indeed to automate the selection using a learning algorithm, so as to provide processing suited to a particular context, for example, the transmission of information about the trajectory, the haptic representation of the environment and its points of interest, or the perception of the surrounding volume. According to a variant, the system further comprises a server capable of communicating with each of the individual items of equipment to receive acquired data of the geographical location of the digital environment and stored data of the digital model of the geographical location of said environment, and transmitting said digital model to each of the individual items of equipment depending on the location.

[0015] The variants just described allow for a reduction in computational processing from acquired images and allow for the sharing of useful information between different experienced users.

[0016] According to a particular aspect, the processing means for processing the digital representation is designed to periodically extract a sequence of successive pulsed digital activation patterns of a subset of spikes in the haptic region to provide progressing haptic information over a period of time.

[0017] According to a variant, one of the aforementioned digital patterns is present in a pulsed activation command for an alignment of spikes that, together with the reference axis of the bracelet, forms an angle corresponding to the direction of movement with respect to the reference direction of the visual environment.

[0018] According to another variant, one of said digital patterns is present in a pulse activation command to a configuration of spikes corresponding to a projection in the horizontal plane of a primary point of interest on said digital representation of said visual environment.

[0019] According to certain aspects, a pulsed digital activation pattern model of a subset of spikes in the aforementioned haptic region is determined based on the membership of the visual environment to pre-recorded classes of environments.

[0020] Advantageously, the pulse digital activation pattern model of the subset for spikes in said haptic region is determined based on the level of user experience.

[0021] The present invention will be more clearly understood from the following detailed description of non-limiting embodiments of the invention, taken in conjunction with the accompanying drawings, in which: [Brief explanation of the drawings]

[0022] [Figure 1]FIG. 1 is a schematic diagram of the present invention. [Figure 2] FIG. 2 shows the inside of a bracelet according to the invention. [Figure 3] Figure 3 shows the functional architecture. DETAILED DESCRIPTION OF THE INVENTION

[0023] General description of the hardware architecture The individual components of the system according to the invention include a bracelet (1) worn on the user's forearm or optionally in thigh position, and a frame (2), for example in the form of a pair of eyeglasses, provided with image sensors in the extensions of each branch to provide a stereoscopic image in the manner described. The frame (2) is equipped with georeferencing or geolocation sensors and / or modules providing an indication of the frame's orientation with respect to magnetic north.

[0024] The various elements just mentioned communicate, in BLE Bluetooth mode, with the user's smartphone (3) with which they are paired, which ensures, using applications, some computational processes and communication with a server via radio frequency communication of the 3G, 4G or 5G type, or Wi-Fi.

[0025] Of course, the telephone (3) could be replaced by a computer, a tablet, or more generally a calculator.

[0026] Bracelet Description FIG. 2 shows an embodiment of a bracelet for implementing the invention.

[0027] It consists of a flexible shell (10) provided with straps (11, 12, 13, 14) for fastening around the forearm (or optionally around the waist).

[0028] When actuated by an electromagnetic actuator using an electrical signal, it comprises a 5x12 matrix of spikes (15) that can each be actuated in a pulsed manner between a retracted rest position and a pulsed erected position in a fraction of a second.

[0029] The configuration of the haptic surface is not limited to rectangular zones with regularly distributed spikes (15).

[0030] Digital Processing The first step (100) consists in acquiring two streams of synchronized images using two laterally offset sensors in the user's frame (2).

[0031] The pair of sensors just described forms a stereoscopic camera, with the camera pointed towards the scene that the user would be able to see.

[0032] The camera just described is connected via a radio frequency link to a smartphone (3) that includes a calculation unit that allows processing (110) of the images resulting from the two sensors. The processing just described allows a depth map to be calculated from the two images, as well as the position of the camera in space. Furthermore, the camera can also be connected to a computer or smartphone via a wireless connection (Bluetooth, Wi-Fi, etc.) or a wired connection.

[0033] One possible method of image processing is a series of algorithms that extract a depth map of the scene to derive it from the changes in camera position and orientation between two successive recording occasions (typically separated by 1 / 60th of a second), and then allow the results just described to be used together with the associated left and right images.

[0034] Next, two processes take place.

[0035] First, a software module (120) controls the recording of images or visible features that constitute points of interest and the storage in memory of the positions of the frames (2) when said points are observed. Said module provides a database of points of interest that are transmitted by the users' phones (3) to a server that stores said geographical location data in a database shared among all users.

[0036] Thus, the storage of a new entry in the formed database is preferably triggered in response to a collection criterion that determines the amount of redundant information with other entries in the database. Other criteria may be used, such as manual triggering of storage by a user, calculation of the physical distance between locations of points of interest, or the period of time elapsed between two instances of storage.

[0037] Therefore, a database E1 is constructed, which contains a set of reference positions associated with features or images, and which serves to relocate the frames (2) during use of the system in the same area.

[0038] In parallel, a software module (140) calculates a depth map and the parameters of the stereoscopic system are used to generate a point cloud by projecting each pixel of the image to obtain the coordinates of a point in space, which then undergoes a change of reference frame by using information about the headset's position in space, said information coming from the odometry module C1, during an initialization phase, in order to place all sensed points in a common, fixed reference frame.

[0039] The set of points just described is merged to create a dense model (cartography) of the operating area while reducing the amount of redundant information.

[0040] For example, the process just described can be performed by going through all the points and merging those identified as being close to each other according to distance, or indeed using a TSDF (truncated signal distance function) volume.

[0041] The next step (150) consists in generating, in three dimensions, a network on the set of points, said network being made up of connected triangles and modelling the surface of the operating area.

[0042] The set of points (voxels) just mentioned is then processed to calculate the activation pattern for the spikes (15) of the bracelet (1).

[0043] To accomplish what has just been stated, the system includes a library of different processing options.

[0044] The first processing method (200) consists in determining the direction of movement or simplified trajectory in the form of a series of segments calculated relative to the voxels corresponding to the obstacle, and the orientation recalculated with respect to the orientation of frame (2). The result is a spike activation pattern that is recalculated periodically, for example once per second, to haptically convey the direction or trajectory followed.

[0045] The second processing method (210) consists in calculating the projection of the voxels in the horizontal plane to determine a low-resolution digital map (resolution according to the number of spikes) and applying to the bracelet a pulse pattern corresponding to said map, oriented according to the orientation of frame (2).

[0046] A third processing method (220) consists in calculating successive cross sections and transmitting a sequence of patterns corresponding to the lower resolution cross sections at successive times in time typified by spacing along a longitudinal axis perpendicular to the aforementioned cross sections. Thus, gaps in the pattern spaced apart by periods of quiescence are applied to the spikes, allowing the user to discern the formation of the environment by a succession of tactile sensations.

[0047] A fourth processing method (230) consists in calculating a low-resolution black and white image of the stereoscopic image and calculating a pattern based on said low-resolution image to control the periodic activation of spikes (15).

[0048] The selection of one of the processing methods (200 to 230) can be done by the user manually or by voice control, or it can be done automatically depending on the type of environment (density of obstacles, familiar vs. new environment, etc.) or depending on the user's learning level, with some processing methods requiring a greater degree of sensitivity and greater experience than others.

[0049] Depending on the variant, the bracelet computer will be connected via cable, Bluetooth or Wi-Fi. The connection to the computer's server will be via Wi-Fi or 3G, 4G or 5G.

Claims

1. an acquisition means communicatively connected to a user device held by a user for acquiring a real visual environment; a non-visual human / machine interface means worn by the user; processing means for processing the digital representation of the visual environment to provide electrical control signals for controlling a non-visual interface; Equipped with the human / machine interface means comprises a bracelet having a single haptic area with a surface area between 60 mm x 60 mm and 150 mm x 150 mm, including a set of N x M active spikes, where N is between 5 and 100, and M is between 10 and 100; the acquisition means is supported by a frame worn on the user's head, the frame adapted to hold an image sensor for providing a stereoscopic image of the visual environment; The processing means for processing the digital representation comprises periodically extracting a pulsed digital activation pattern of at least one of the subset of spikes in the haptic region by selectively applying various processing options from a library, the various processing options being calculated in response to an expression model selected from a set of identified expression models, the various processing options included in the library comprising: a process consisting of determining a direction of movement or a simplified trajectory in the form of a continuous segment calculated on the basis of a set of points (voxels) corresponding to the obstacle, the direction being recalculated relative to the orientation of said frame; a process consisting of calculating the projection of said voxels onto a horizontal plane to determine a low-resolution digital map and applying to said bracelet a pulse pattern corresponding to said digital map, oriented according to the orientation of said frame; a process consisting of calculating successive cross-sections and transmitting a sequence of patterns corresponding to the low-resolution cross-sections in a time sequence representing intervals along a longitudinal axis orthogonal to said cross-sections; a process comprising: calculating a low-resolution black-and-white image of the stereoscopic image; and calculating a pattern based on the low-resolution black-and-white image to control the periodic activation of the spikes; Including, the representation model provides context-specific processing of one of a plurality of context-related information modes based on a selection by the user or through automation based on a learning algorithm, depending on the needs of the user; An orientation aid system comprising:

2. 2. An orientation assistance system according to claim 1, characterized in that the acquisition means comprise at least one image sensor that can be carried by the wearer of the bracelet and processing means for generating a digital depth map.

3. 3. An orientation aid system according to claim 1 or 2, characterized in that the processing means for processing the digital representation are calculated in response to a representation model selected from a set of identified representation models.

4. 4. An orientation aid system according to claim 1, further comprising a server capable of communicating with each of the individual equipment items to receive geographically located acquired data of a digital environment and stored data of a geographically located digital model of said digital environment, and transmitting said digital model to said individual equipment items depending on the location of said individual equipment items.

5. 5. The orientation assistance system of claim 1, wherein the processing means for processing the digital representation is designed to periodically extract sequences of successive pulse digital activation patterns in a subset of the spikes in the haptic region to provide progressive haptic information over a period of time.

6. 6. An orientation assistance system according to claim 1, wherein one of the pulse digital activation patterns consists of a pulse activation command for the alignment of spikes that, together with a reference axis of the bracelet, form an angle corresponding to a direction of movement relative to a reference direction of the visual environment.

7. 7. An orientation aid system according to claim 1, wherein one of the pulsed digital activation patterns comprises pulsed activation commands for a configuration of spikes corresponding to a projection in a horizontal plane of a primary point of interest of the digital representation of the visual environment.

8. 8. The orientation assistance system of claim 1, wherein the pulse digital activation pattern model in the subset of spikes in the haptic region is determined depending on whether the visual environment belongs to a class of pre-recorded environments.

9. 9. The orientation aid system of claim 1, wherein the pulse digital activation pattern model in the subset of spikes in the haptic region is determined based on a level of user experience.

Images