Orientation Aid System

Abstract

The present invention provides an orientation assistance system comprising: acquisition means for acquiring a real or virtual visual environment; non-visual human / machine interface means; and processing means for processing a digital representation of said visual environment to provide electrical control signals for controlling the non-visual interface; wherein said human / machine interface means comprises a bracelet having a haptic region with a surface area of ​​between 60 mm x 60 mm and 150 mm x 150 mm, said haptic region including a set of N x M active spikes, where N is between 5 and 100 and M is between 10 and 100; and wherein said processing means for processing the digital representation periodically extracts at least one pulse digital activation pattern of a subset of the spikes in said haptic region.

Description

[Technical Field]

[0001] The present invention relates to the field of orientation assistance for the visually impaired or for people moving around in very poor visibility environments, for example firefighters moving around in smoke-filled buildings or soldiers moving around in the dark. [Background technology]

[0002] Various solutions are known, ranging from the aid of guide dogs, to marking the ground with strips of orientation aids, to embedding sonic beacons or even walking sticks that allow for the detection of obstacles.

[0003] More recently, GPS applications have also been developed to allow blind and partially sighted people to be more independent in their travels by showing them routes, points of interest (POIs), intersections, bus stops, etc. Users of GPS applications can receive real-time, spoken-language notifications on their smartphones about bus departure times, bus stop directions, and information to take them where they want to go if they leave the transport. During their journey, ad hoc information is provided to help users avoid getting lost, avoid obstacles, avoid difficulties, and catch the right bus.

[0004] Further proposals have included the use of haptic information transmission, for example in the form of fitness trackers. Haptic technology uses the sense of touch to convey information. The company WearWorks proposes a smart bracelet called "Wayband" that guides blind people. The user begins by downloading the application onto a shared smartphone and then precisely stating the desired address. The bracelet, connected to a GPS system, guides the user to their destination. If the user makes a mistake about the route, the bracelet vibrates; once they are on the correct path, it stops vibrating. A sensitive, more intuitive, and unintrusive tactile language frees up hearing, a sense that is overused by people with low vision.

[0005] Another solution marketed by the company SUNU relates to a bracelet that uses radio frequency waves to facilitate echolocation in space.

[0006] The prior art is known from US Pat. No. 6,299,499, which relates to a system for assisting blind people, comprising a plurality of video cameras and a haptic band worn by a user, the haptic band including a plurality (two or more) of spaced haptic transducers. The haptic band is worn so that the user's hands are free for other tasks. The system uses video cameras, depth processing algorithms, and object recognition algorithms (hardware and / or software) to identify a limited number of objects important for navigation. The spatial location of each important object of interest is accordingly provided to the user by varying the output directed to the haptic transducers. The system is designed to identify and indicate objects as general objects, specific objects, or potential obstacles. Furthermore, the system can optionally provide audio information or tactile graphic display information further related to the aforementioned objects.

[0007] The solutions just described are not sufficient because multiple transducers act on different parts of the body, which does not allow for an intuitive and concise understanding of the information and requires considerable care to distinguish between deformations in the transducer's area of ​​action.

[0008]

[0005] US Patent No. 5,999,949 describes another example of a solution involving a set of cameras capable of sensing an image in front of the device, which is then processed and used to output a three-dimensional representation of the image onto an array of tactile pins. While what has just been described allows for the recognition of objects and obstacles in the area located in front of the device, the pin array can also be used as an adaptive Braille reader. While a specific pin assembly using step micromotors is discussed, the global system provides multiple functions to low-vision users, including navigation, facial recognition, wireless network connectivity capabilities, and various input / output means. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] US Patent Application Publication No. 2018 / 0189567 [Patent Document 2] US Patent Application Publication No. 2015 / 0125831 Summary of the Invention [Problem to be solved by the invention]

[0010] Prior art solutions are not suitable for conveying rich and comprehensive information about the visual environment in a tactile form.

[0011] In general, haptic information is too basic and inadequate for understanding complex environments in a manner that is complete enough to represent practical orientation aids. [Means for solving the problem]

[0012] In order to overcome the just mentioned disadvantages, the present invention, according to its most general meaning, relates to an orientation assistance system according to claim 1.

[0013] Advantageously, said acquisition means consist of at least one image sensor capable of being carried by the wearer of said bracelet and processing means for generating a digital depth map. Said image sensor may be formed by a stereoscopic camera, or a single camera with processing of successive images to determine the depth map, or indeed by a 3D scanner, or a lidar providing a point cloud.

[0014] According to a particular embodiment, said digital representation processing means are calculated depending on a representation model selected from a set of different representation models, which allow the user to select one of the modes of information just mentioned depending on the user's needs and preferences, or indeed to automate the selection using a learning algorithm, so as to provide processing suited to a particular context, for example, the transmission of information about the trajectory, the haptic representation of the environment and its points of interest, or the perception of the surrounding volume. According to a variant, the system further comprises a server capable of communicating with each of the individual items of equipment to receive acquired data of the geographical location of the digital environment and stored data of the digital model of the geographical location of said environment, and transmitting said digital model to each of the individual items of equipment depending on the location.

[0015] The variants just described allow for a reduction in computational processing from acquired images and allow for the sharing of useful information between different experienced users.

[0016] According to a particular aspect, the processing means for processing the digital representation is designed to periodically extract a sequence of successive pulsed digital activation patterns of a subset of spikes in the haptic region to provide progressing haptic information over a period of time.

[0017] According to a variant, one of the aforementioned digital patterns is present in a pulsed activation command for an alignment of spikes that, together with the reference axis of the bracelet, forms an angle corresponding to the direction of movement with respect to the reference direction of the visual environment.

[0018] According to another variant, one of said digital patterns is present in a pulse activation command to a configuration of spikes corresponding to a projection in the horizontal plane of a primary point of interest on said digital representation of said visual environment.

[0019] According to certain aspects, a pulsed digital activation pattern model of a subset of spikes in the aforementioned haptic region is determined based on the membership of the visual environment to pre-recorded classes of environments.

[0020] Advantageously, the pulse digital activation pattern model of the subset for spikes in said haptic region is determined based on the level of user experience.

[0021] The present invention will be more clearly understood from the following detailed description of non-limiting embodiments of the invention, taken in conjunction with the accompanying drawings, in which: [Brief explanation of the drawings]

[0022] [Figure 1]FIG. 1 is a schematic diagram of the present invention. [Figure 2] FIG. 2 shows the inside of a bracelet according to the invention. [Figure 3] Figure 3 shows the functional architecture. DETAILED DESCRIPTION OF THE INVENTION

[0023] General description of the hardware architecture The individual components of the system according to the invention include a bracelet (1) worn on the user's forearm or optionally in thigh position, and a frame (2), for example in the form of a pair of eyeglasses, provided with image sensors in the extensions of each branch to provide a stereoscopic image in the manner described. The frame (2) is equipped with georeferencing or geolocation sensors and / or modules providing an indication of the frame's orientation with respect to magnetic north.

[0024] The various elements just mentioned communicate, in BLE Bluetooth mode, with the user's smartphone (3) with which they are paired, which ensures, using applications, some computational processes and communication with a server via radio frequency communication of the 3G, 4G or 5G type, or Wi-Fi.

[0025] Of course, the telephone (3) could be replaced by a computer, a tablet, or more generally a calculator.

[0026] Bracelet Description FIG. 2 shows an embodiment of a bracelet for implementing the invention.

[0027] It consists of a flexible shell (10) provided with straps (11, 12, 13, 14) for fastening around the forearm (or optionally around the waist).

[0028] When actuated by an electromagnetic actuator using an electrical signal, it comprises a 5x12 matrix of spikes (15) that can each be actuated in a pulsed manner between a retracted rest position and a pulsed erected position in a fraction of a second.

[0029] The configuration of the haptic surface is not limited to rectangular zones with regularly distributed spikes (15).

[0030] Digital Processing The first step (100) consists in acquiring two streams of synchronized images using two laterally offset sensors in the user's frame (2).

[0031] The pair of sensors just described forms a stereoscopic camera, with the camera pointed towards the scene that the user would be able to see.

[0032] The camera just described is connected via a radio frequency link to a smartphone (3) that includes a calculation unit that allows processing (110) of the images resulting from the two sensors. The processing just described allows a depth map to be calculated from the two images, as well as the position of the camera in space. Furthermore, the camera can also be connected to a computer or smartphone via a wireless connection (Bluetooth, Wi-Fi, etc.) or a wired connection.

[0033] One possible method of image processing is a series of algorithms that extract a depth map of the scene to derive it from the changes in camera position and orientation between two successive recording occasions (typically separated by 1 / 60th of a second), and then allow the results just described to be used together with the associated left and right images.

[0034] Next, two processes take place.

[0035] First, a software module (120) controls the recording of images or visible features that constitute points of interest and the storage in memory of the positions of the frames (2) when said points are observed. Said module provides a database of points of interest that are transmitted by the users' phones (3) to a server that stores said geographical location data in a database shared among all users.

[0036] Thus, the storage of a new entry in the formed database is preferably triggered in response to a collection criterion that determines the amount of redundant information with other entries in the database. Other criteria may be used, such as manual triggering of storage by a user, calculation of the physical distance between locations of points of interest, or the period of time elapsed between two instances of storage.

[0037] Therefore, a database E1 is constructed, which contains a set of reference positions associated with features or images, and which serves to relocate the frames (2) during use of the system in the same area.

[0038] In parallel, a software module (140) calculates a depth map and the parameters of the stereoscopic system are used to generate a point cloud by projecting each pixel of the image to obtain the coordinates of a point in space, which then undergoes a change of reference frame by using information about the headset's position in space, said information coming from the odometry module C1, during an initialization phase, in order to place all sensed points in a common, fixed reference frame.

[0039] The set of points just described is merged to create a dense model (cartography) of the operating area while reducing the amount of redundant information.

[0040] For example, the process just described can be performed by going through all the points and merging those identified as close to each other according to distance, or indeed using a TSDF (truncated signal distance function) volume.

[0041] The next step (150) consists in generating a network on the set of points in three dimensions, said network being made up of connected triangles and modeling the surface of the operating area.

[0042] The set of points (voxels) just mentioned is then processed to calculate the activation pattern for the spikes (15) of the bracelet (1).

[0043] To accomplish what has just been stated, the system includes a library of different processing options.

[0044] The first processing method (200) consists in determining the direction of movement or simplified trajectory in the form of a series of segments calculated relative to the voxels corresponding to the obstacle, and the orientation recalculated with respect to the orientation of frame (2). The result is a spike activation pattern that is recalculated periodically, for example once per second, to haptically convey the direction or trajectory followed.

[0045] The second processing method (210) consists in calculating the projection of the voxels in the horizontal plane to determine a low-resolution digital map (resolution according to the number of spikes) and applying to the bracelet a pulse pattern corresponding to said map, oriented according to the orientation of frame (2).

[0046] A third processing method (220) consists in calculating successive cross sections and transmitting a sequence of patterns corresponding to the lower resolution cross sections at successive times in time typified by spacing along a longitudinal axis perpendicular to the aforementioned cross sections. Thus, gaps in the pattern spaced apart by periods of quiescence are applied to the spikes, allowing the user to discern the formation of the environment by a succession of tactile sensations.

[0047] A fourth processing method (230) consists in calculating a low-resolution black and white image of the stereoscopic image and calculating a pattern based on said low-resolution image to control the periodic activation of spikes (15).

[0048] The selection of one of the processing methods (200 to 230) can be done by the user manually or by voice control, or it can be done automatically depending on the type of environment (density of obstacles, familiar vs. new environment, etc.) or depending on the user's learning level, with some processing methods requiring a greater degree of sensitivity and greater experience than others.

[0049] Depending on the variant, the bracelet computer will be connected via cable, Bluetooth or Wi-Fi. The connection to the computer's server will be via Wi-Fi or 3G, 4G or 5G.

Claims

1. an acquisition means for acquiring a real visual environment, the acquisition means being communicatively connected to a user device held by a user; a non-visual human / machine interface means worn by the user; processing means for processing the digital representation of the visual environment to provide electrical control signals for controlling a non-visual interface; a server communicatively coupled to a plurality of said user devices, the server receiving geographic location acquisition data of a digital environment corresponding to said visual environment and stored data of a geographic location digital model of said digital environment, the server transmitting said geographic location digital model to said plurality of user devices in response to positions of said plurality of user devices; a database communicatively connected to the server and storing point-of-interest information in geographic location acquisition data of the digital environment and stored data of the geographic location digital model, the point-of-interest information being shared among all the users via the server; and Equipped with the human / machine interface means comprises a bracelet having a single haptic area with a surface area between 60 mm x 60 mm and 150 mm x 150 mm, containing a set of N x M active spikes, where N is between 5 and 100 and M is between 10 and 100; The processing means for processing the digital representation includes periodically extracting at least one pulse digital activation pattern of a subset of the spikes in the haptic region to calculate successive cross-sections, and transmitting a sequence of patterns corresponding to the lower resolution cross-sections in a time series representing intervals along a longitudinal axis orthogonal to the cross-sections, thereby enabling the user to discern formations in the user's environment through a succession of tactile sensations. An orientation aid system characterized by:

2. 2. The orientation assistance system according to claim 1, characterized in that the acquisition means consist of at least one image sensor capable of being carried by the wearer of the bracelet and processing means for generating a digital depth map.

3. 3. An orientation aid system according to claim 1 or 2, characterized in that the processing means for processing the digital representation are calculated in response to a representation model selected from a series of identified representation models.

4. 4. The orientation assistance system of claim 1, wherein the processing means for processing the digital representation is designed to periodically extract a sequence of successive pulse digital activation patterns in a subset of the spikes in the haptic region to provide haptic information progressing over a period of time.

5. 5. An orientation assistance system according to claim 1, wherein one of the pulse digital activation patterns is characterized by a pulse activation command for an alignment of spikes that, together with a reference axis of the bracelet, forms an angle corresponding to a direction of movement relative to a reference direction of the visual environment.

6. 6. An orientation assistance system according to claim 1, wherein one of the pulsed digital activation patterns is characterized by a pulsed activation command for a configuration of spikes corresponding to a projection in a horizontal plane of a primary point of interest of the digital representation of the visual environment.

7. 7. The orientation assistance system of claim 1, wherein the pulse digital activation pattern model in the subset of spikes in the haptic region is determined depending on whether the visual environment belongs to a class of pre-recorded environments.

8. 8. The orientation aid system of claim 1, wherein the pulse digital activation pattern model in the subset of spikes in the haptic region is determined based on a level of user experience.

Images