Method for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt

Abstract

The present invention relates to a method and system for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt having an active surface of N x M actuators, where N and M are integers greater than or equal to 10. The system includes an actuator activation protocol consisting of calculating, for each acquisition of said visual environment, a sequence of P activation frames for said actuators, where P is an integer from 2 to 1000, preferably from 5 to 50. Each frame corresponds to the representation of the environment in an incremental depth plane. The system is characterized in that it further comprises a processing step consisting of applying a context-sensitive correction function that modifies the activation protocol of the haptic pixels. Abstract figure: Figure 1

Description

Title of the invention: Method for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt Field of the invention

[0001] The present invention relates to the field of orientation assistance for people suffering from visual impairments or people moving in environments with very low visibility, for example firefighters moving in a smoky building or soldiers moving in the dark.

[0002] We know the solution based on the transmission of information by a haptic mode, for example in the form of a connected bracelet. Haptic technology uses the sense of touch to convey information. The company WearWorks® offers a smart bracelet called "Wayband"®, to guide the blind. The user begins by downloading an application on an associated smartphone and states the desired address. The bracelet connected to a GPS system guides the user to the destination. When the user takes the wrong route, the bracelet vibrates. It stops vibrating once on the right path. Tactile language, sensitive, more intuitive and less intrusive, relieves hearing, a sense over-used in the visually impaired. State of the art

[0003] The applicant itself has patented by patent FR3132208Al an orientation assistance system comprising means for acquiring a real or virtual visual environment, non-visual human-machine interface means and means for processing the digital representation of said visual environment to provide an electrical signal for controlling a haptic interface, said means for processing the digital representation consisting of periodically extracting at least one digital pulse activation pattern from a subset of the pins of said haptic zone, characterized in that said haptic interface is constituted by a lumbar belt having an active surface of NxM actuators, N and M being whole numbers greater than or equal to 10,and in that said processing means provides for each acquisition of said visual environment a sequence of P activation frames of said actuators or P is an integer between 2 and 1000 and preferably between 5 and 50, each of the frames corresponding to the representation of the environment in an incremental depth plane.

[0004] Patent application WO2018058936A1 proposes a guide belt for the blind comprising a laser radar module disposed on the belt and configured to detect a surrounding environment and to form position information. of a measured object. A control device receives and processes the position information of the measured object detected by the laser radar module and forms an alert instruction; and an alert system which is controlled by the alert instruction to output alert information regarding an ambient environment condition. The wearer wears a blind guide belt, that is, the belt body of the blind guide belt is worn on the waist, and the direction in which the user advances while walking can be defined as the front of the waist body belt, and the direction in which the user uses the back is defined as On the back of the belt body, the second lidar module is arranged at the front of the belt body. In the process of personnel use, compared with other direction information, it is more important to use the direction information of the personnel in the direction of advancement.Therefore, the second laser radar module may be placed at the front of the belt body, and the front may be a range. For example, the belt body is also divided into a first part and a second part, the first part being the front part of the belt body, and the second part being the rear part of the belt body. The second laser radar module may be arranged at any position on the front of the belt body.

[0005] Optionally, the center position of the first part of the belt body is determined in advance, and the second laser radar module is disposed in the center of the first part of the belt body, so that the second laser radar module scans the route condition of the user's heading in a wider range.

[0006] Patent application WO2021 / 099993 describes another example of an orientation aid device for visually impaired people, comprising a plurality of vibrating signaling elements applicable to a support element, such as a belt or the like, to be worn at the waist by a user, at least one of said signaling elements being designed to be positioned towards the right side of the user and at least one of said signaling elements being adapted to be positioned towards the left side of the user, the device comprising a location unit provided with an antenna. Said location unit is housed in a dedicated housing separate from the signaling elements and provided with means for attachment to the body of the user and / or a guide dog in a position remote from the support element, the signaling elements being provided with at least one transceiver unit for connection with said location unit. Disadvantages of the prior art

[0007] These solutions provide only very limited information and do not allow the user to have a sufficient representation of the environment in which he is evolving.

[0008] In particular, to perceive the distances of obstacles, their extent and their height. which leads to a permanent feeling of insecurity making any movement very difficult. Solution provided by the invention

[0009] In order to overcome the drawbacks of the prior art, the present invention relates, in its most general sense, to an orientation assistance method having the technical characteristics set out in claim 1.

[0010] The invention particularly relates to a method for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt having an active surface of NxM actuators, N and M being integers greater than or equal to 10 comprising an actuator activation protocol consisting of calculating for each acquisition of said visual environment a sequence of P activation frames of said actuators or P is an integer between 2 and 1000, preferably between 5 and 50, each of the frames corresponding to the representation of the environment in an incremental depth plane characterized in that it further comprises a processing step consisting of applying a contextual type corrective function modifying the haptic pixel activation protocol.

[0011] According to variants:

[0012] - said actuator activation protocol consists of activating each Axy actuator corresponding to a voxel V(xyzmin, t) once per image I(t), in the form of N successive sequences Si(t), i being an integer varying between 0 and N, each sequence consisting of activating the actuators Axy>i corresponding to the voxels V(xyzmin, Vi, t), Vi corresponding to the distance Dxy (t) of the point of the image P relative to the camera acquiring the image I(t).

[0013] - said corrective function consists of modifying the periodicity of the images I(t) in function of the value V; of the nearest voxel V(xyzmin, V;, t).

[0014] - said corrective function consists of repeating the sequence S;(t) for the duration of a image I(t).

[0015] - said corrective function consists of weighting the value V; of each voxel V(xyzmin, Vi, t), by a coefficient Kxyijten depending on the distance of said voxel V(xyzmin, V;, t) relative to a reference point V(X0, Yo, Zo ,t).

[0016] - said corrective function consists of weighting the value V; of each voxel V(xyzmin, Vi, t), by a coefficient Kxy ijten depending on the distance Dxy (t) of the point of the image P relative to the image acquisition camera I(t).

[0017] - said corrective function consists of weighting the value V; of each voxel V(xyzmin, Vi, t), by a coefficient Kxy ijten depending on data coming from an external source distinct from said acquisition camera.

[0018] - said corrective function consists of modulating the value V; of each voxel V(xyzmin, Vi, t), for values ​​of Y less than a threshold value Yo, by assigning a value VH for the areas of the ground corresponding to a hole or a bump, and a value VB otherwise.

[0019] The invention also relates to an orientation assistance system comprising means for acquiring a real or virtual visual environment, non-visual human-machine interface means and means for processing the digital representation of said visual environment to provide an electrical signal for controlling a haptic interface constituted by a lumbar belt having an active surface of NxM actuators, N and M being whole numbers greater than or equal to 10,said means for processing the digital representation consisting of periodically extracting at least one digital pulse activation pattern from a subset of the pins of said haptic zone and providing for each acquisition of said visual environment a sequence of P activation frames of said actuators or P is an integer between 2 and 1000, preferably between 5 and 50, each of the frames corresponding to the representation of the environment in an incremental depth plane, characterized in that it comprises a microcontroller controlling the activation of said actuators according to a method mentioned above.

[0020] Detailed description of a non-limiting example of embodiment of the invention

[0021] The present invention will be described in more detail with reference to non-limiting examples of embodiment specifying the aforementioned advantages and considerations. A more particular description of the invention briefly described below.

[0022] [Fig-1] [Fig.l] represents a schematic view of the system according to the invention

[0023] [Fig.2] [Fig.2] represents a view of a visual image

[0024] [Fig.3] [Fig.3] represents a view of a haptic image

[0025] [Fig.4] [Fig.4] represents a view of a sequence of haptic frames

[0026] [Fig.5] [Fig.5] represents a schematic view of a vertical tunnel

[0027] [Fig.6] [Fig.6] represents a schematic view of the field of view as a function of the movement speed

[0028] [Fig.7] [Fig.7] represents a schematic view of a horizontal tunnel General principle of the invention

[0029] The general principle of the present invention consists in carrying out an image acquisition and processing described in the applicant's patent FR3132208Al, and in applying a corrective function modulating the pixel activation protocol, each image I(t) resulting in a sequence of N frames. A frame corresponds to the activation of a subset of pixels P(x, y, t, i) assigned the same value V;(t). This value V;(t) is calculated according to the protocol described in the applicant's patent FR3132208Al, modified by a contextual corrective function calculated according to different modalities described in a non-limiting manner below. After. Reminder of the basic protocol

[0030] The example developed below, without limitation, comprises a means for acquiring the environment, for example a frame of glasses (10) equipped with cameras (11, 12) used to acquire data on the environment in real time to provide digital images which control the actions of a haptic transducer. The haptic transducer generates actions in the form of pressures directly or indirectly on the skin by electromagnetic or electromechanical actuators, or in the form of electrical or light pulses).

[0031] It is recalled that the system which is the subject of the invention can also be used for gaming or training applications in augmented reality, with images provided by a video source.

[0032] A computer is responsible for recovering the images coming from the Sensor part, then generating a 3D depth map from them. It transmits this map to haptic equipment, which comprises for example a grid of actuators or pins (small linear actuators which can be raised or lowered), integrated into a back belt (20). This belt (20) is equipped with a set of actuators arranged on supports (21 to 24) to form a matrix, for example of 20 x 40 pixels. These actuators are arranged to form a regular matrix, preferably with a constant pitch. An electronic circuit receives the visual signals and provides the processing to control the actuators in order to produce sensations on the user's back which are easy to interpret after a learning period. The lumbar belt (20) can be worn over a light fabric garment (shirt, polo shirt, blouse) or directly on the skin.

[0033] The surface of the active matrix formed by the actuators covers an extended lumbar region to benefit from good resolution and satisfactory comfort of use.

[0034] The restitution of the image of the real environment into haptic images consists of dividing the depth map calculated from the visual image into several successive layers determining virtual or haptic images controlling the activation of the haptic equipment: thus, the closest objects are first displayed, then the slightly more distant objects, and so on until reaching the maximum viewing distance chosen (generally around 10 meters). Thus, we have a sort of scan of the environment which gradually sinks and which displays at each instant what it encounters. This scan results in a burst of virtual images lasting approximately 100 milliseconds, composed of around ten haptic images corresponding to consecutive shots, before resuming with a new burst corresponding to the new environment, resulting from the movement of the user or the change in orientation of the actual image, due to a change in the position of the head or the video image. Visual image processing

[0035] The cameras (11, 12) acquire images to reconstruct a digital image with depth information. A first processing operation consists of constructing a depth map (in gray level). For each pixel of the optical image (100), a haptic image (200) is transcribed in gray level as a function of the distance of each point from the camera.

[0036] Depth information can also be determined with a single camera with appropriate image processing.

[0037] It should be noted that this haptic image (200) could also, without departing from the invention, be calculated from the digital image provided by a lidar.

[0038] This haptic image (200) is then decomposed into a sequence of incremental haptic frames (301 to 307), each corresponding to a depth plane. The first haptic frame (301) of the sequence corresponds to the obstacle zones in the plane closest to the user, the second first haptic frame (301) of the sequence corresponds to the obstacle zones in the plane closest to the user, the following haptic frame (302) corresponding to the obstacle zones in the following plane, offset from the previous one by one step, for example 30 centimeters in distance and so on.

[0039] The gray level of the haptic frames (301 to 307) codes the type of action of the corresponding actuator, for example the frequency of the vibration or the duration of the vibration during the activation time of the corresponding haptic frame.

[0040] A haptic image (200) is thus translated into a temporal scan of haptic frames (301 to 307) which are integrated by the user to perceive a depth representation of his environment.

[0041] Other treatments are applied to improve the intelligibility of tactile perceptions: - Straightening of the visual image (100) to create a synthetic image reduced to an observation along a horizontal axis lowered to the level of the user's legs, a few tens of centimeters from the ground - Reinforcement on the visual image (100) of the pixels corresponding to small obstacles, in order to occupy at least one pixel of the haptic image (200). - Reprocessing of holes to enhance the gray level of the corresponding area on the haptic image (200) - Reinforcement of the gray level on the haptic image (200) of the area of ​​interest determined by automatic recognition, for example by ap supervised learning. Examples of processing for the production of the haptic image (200),

[0042] Examples of processing will be described below, with the following variables: - M denotes the haptic matrix applied to the user - H denotes the height of this haptic matrix M (in pixels) - W denotes the width of this haptic matrix M (in pixels) - p denotes the level of precision - DM denotes the depth map recovered by the sensors (11, 12) - dmH denotes height of the recovered depth map (in pixels) = p*H dmW denotes width of the recovered depth map (in pixels) = p*W FOVv denotes the vertical field of view of the camera used (in degrees) - FOVh denotes the horizontal field of vision of the camera used (in degrees) - hu denotes the height at which the camera is placed (user height taken at eye level) - DISTANCE_MAX denotes the maximum viewing distance set by the user - SPEED refers to the display speed set by the user - PAUSE_TIME denotes the pause time between the display of 2 images (set by the user) - MAT[x, y] corresponds to the value of the MAT matrix at coordinates [x, y]. - MAT[x] corresponds to the xth column of the matrix MAT LIST[x] corresponds to the xth value of the list LIST x = f(arguments) means that the value of x is a function of 1 or more arguments (proportionality relationship)

[0043] / / means that what is written after is a comment Step 1: Obtain a depth map

[0044] This first step consists of calculating a depth map of size dmW*dmH from the two images acquired by the cameras (11) and (12) or by a lidar, or by a source of binocular virtual images.

[0045] These are known treatments, generally comprising: a. A step of acquiring two images of the same scene at the same time by two cameras (11, 12) whose spacing is known, or else successive images coming from the same sensor. b. A calibration step consisting of determining the internal and external parameters of the geometric model of the stereoscopic sensor c. A pixel matching step to find on the two images the pairs of pixels which correspond to the projection of the same element of the scene, d. A 3D reconstruction step consisting of calculating for each pixel the position in space of the point which was projected into this pixel.

[0046] The result of this first step is a visual image (100) of size dmW*dmH where each point is constituted by a voxel defined by its coordinates in space whose origin is at the level of the user's head, and the x and y axes perpendicular to the line of sight of the cameras, and the z coordinates the distance from the user's head.

[0047] Presentation of contextual corrective functions

[0048] The following description concerns several examples of contextual corrective functions: • Display variation depending on the distance of the nearest object • Motion blur • Image distortion • Depth distortion • Centralized 3D capture • Display beyond the cameras' capture speed • Display of two information channels • Display of hole in the ground or absence of information • Part that may be trade secrets that we want to file • Displaying a higher resolution than the number of pixels • Multitasking • Simplification of images. • Display bypassing the actuator activation speed limitation. • Adaptive display As stated above, the invention relates to the display of images by scanning. The invention relates to the continuity of the display of depth by scanning.

[0049] Each actuator activates once per image. First, what is near is displayed, then what is far.

[0050] Each pixel(x,y) has an associated value which is the distance.

[0051] At t=0 we display the pixels having the value 0

[0052] At t=l we display the pixels having the value 1

[0053]

[0054] And we return to 0 to display a new image.

[0055] The pseudo codes in the description of the detailed examples which follow are implementation examples to explain the principle of the corresponding corrective functions.

[0056] Variation of the display according to the distance of the nearest object

[0057] According to a first exemplary embodiment, the contextual corrective function controls the duration of an image based on the distance to the nearest object by digital processing on the digital data.

[0058] For this purpose, the processing of the data provided by the camera at a time t determines the voxel V(x,y,z,t). Depending on the distance of the nearest voxel V(xyzmin,t), the duration separating two consecutive images varies between a minimum value, for example 1 millisecond when the voxel V(xyzmin,t) is at a minimum distance, for example 10 cm and a maximum value, for example 2 seconds when the voxel V(xyzmin,t) is at a maximum distance, for example 100m.

[0059] The N frames scroll during the duration of each image, with a frequency that accelerates when the user approaches an obstacle.

[0060] The pseudo code is for example the following: • T is the duration of an image (between 1ms and 3 seconds) • D is the distance covered by the image on T (between 10cm and 100m) • V is the speed of the display, during X ms, the display covers f(X) m. (lem / s to Ikm / s) • T=VxD • dMin=distance to the nearest voxel.

[0061] The display speed varies according to the voxel (x, y, z, t) closest to the user. The minimum and maximum display speed is defined, (the display scanning speed in km / h varies according to the distance of the nearest object)

[0062] V=f(dMin) with a minimum V and a maximum V

[0063] T is unchanged and therefore D is changed

[0064] The display time of a scan depends on the distance of the nearest voxel. The display time has a minimum and a maximum, (the duration of an image composed of several layers depends on the distance of the nearest object)

[0065] T=f(dMin) with a minimum T and a maximum T

[0066] V is unchanged and therefore D is impacted

[0067] According to a variant, the maximum display distance depends on the distance of the nearest voxel. The further away the nearest object is, the greater the distance one perceives. For example, max distance = X times the distance of the nearest object.

[0068] The pseudo code is for example the following:

[0069] D=f(dMin) with a minimum D and a maximum T

[0070] V is unchanged and T is impacted

[0071] Another variant is to control the omission of a part of the display. The display can start at the nearest voxel and / or end at the furthest voxel. • Start of display = ffdMin) with a maximum start of display • dMax is the furthest voxel • End of display = ffdMax) with a minimum End of display. • D=End of display-Start of display • V is unchanged and therefore T is impacted. These variants can be combined.

[0072] With gray it is if these modifications are not combined if everything is combined: • V=ffdMin) with a minimum V and a maximum V • T=f(dMin) with a minimum T and a maximum T • ( D=f(dMin) with a minimum D and a maximum T ) or ( Start of display = ffdMin) with a maximum display start • Display End = ffdMax) with a minimum Display End. dMax is the furthest voxel • D=End of display-Start of display) Motion blur correction function

[0073] This variation is intended to create motion blur to highlight moving objects. The faster an object moves, the more it will be stretched in the direction of its movement.

[0074] Motion blur is caused by displaying images slower than the capture speed. Thus the image to be displayed is updated several times while displaying a single image. Each time the image to be displayed is updated, each voxel is updated by the value of the closest voxel between the image to be displayed and the new image. Each voxel is displayed only once.

[0075] The pseudo code is for example the following:

[0076] Matrix to display[x,y]=Distance from voxel (updated more frequently than the displayed image)

[0077] Displayed matrix[x,y]=Distance from voxel

[0078] Actuators already activated [x,y ]=0 / / 0 for actuator that has not been activated, 1 for actuator that has already been activated

[0079] Actuators activated[x,y]=0 / / 0 for actuator not activated, 1 for actuator to activate

[0080] Current distance = 0

[0081] Increment distance - x / / distance in cm between 0.1 and 1000

[0082] Maximum distance - x / / between 2 and 10,000 times the increment distance

[0083] while(Current Distance <Distance max){

[0084] for(i=0;i <x;i++)

[0085] for(j=0;j <y;j++)f

[0086] iffMatrix displayed [i,j] <Matrice à ajficher[i,j])

[0087] Displayed matrix [i,j] = Matrix to be displayed [i,j]

[0088] iffDisplayed matrix[i,j] <Distance actuelle and Actuators déjà activés[i,j] = = 0 )

[0089] Actuators activés[i,j]=1

[0090] Actuators already activated[i,j]=1

[0091] }

[0092] activation of actuators (Actuators activated)

[0093] wait(x ms) / / x between 0.1 and 1000

[0094] Current Distance + = Increment Distance

[0095] Actuators activated[x,y]=0 / / reset all values ​​to 0

[0096] One variation involves artificial motion blur, based on the speed of movement of objects in the environment, creating motion blur to make moving objects more prominent. Objects can be rendered up to 10 times their sizes without motion blur. Corrective function “Image distortion”

[0097] The pixels in the center of the display are magnified, and the closer you go to the edges of the images, the smaller they are. The distortion also varies along the vertical axis. A pixel can be magnified up to 10 times and reduced by a factor of 10.

[0098] This function controls a variable deformation depending on the distance of the voxels. If a deformation can be achieved by a single lens, it must be imagined that for each different voxel distance a different lens is applied. The further the pixels are, the greater the deformations. The impact of voxel deformations ranges from x 100 to divided by 100.

[0099] These deformations will complement the elements trimmed according to the claims of patent FR3132208A1.

[0100] Corrective function “Depth distortion”

[0101] This corrective function controls the display of distances in a non-linear way. Depth is currently displayed linearly and must be converted to compress distant distances. For example, switching to a perception accuracy in % of the pixel distance and not an accuracy in m. So a layer that displays voxels 10ms later means that it is x% further than the previous one and not x meters further.

[0102] The pseudo code is for example the following:

[0103] Value displayed for distance = / (Distance of the Voxel) Corrective function “Centralized 3D capture”

[0104] 3D image capture is currently done locally and in real time or almost. To ensure that the images to be displayed are of much better quality, this corrective function controls the display made of collaborative depth images. Each person who looks with the device collects data so we know exactly where they are and what they are looking at, they send their depth image to the server which can refine its model and they receive the depth image calculated by all the people who are already been there. And for latency, it is easy to predict what the person will have in their field of vision in the next minute and therefore send them in advance the objects they will see.

[0105] Corrective function "Display beyond the cameras' capture speed"

[0106] This corrective function controls the display of images faster than the perception capabilities of the sensors, therefore image redundancy. Display of the last captured image in a loop.

[0107] The pseudo code is for example the following:

[0108] Matrix to display[x,y] / / contains the values ​​to display

[0109] Displayed matrix[x,y]

[0110] while(true) [YES] Displayed matrix[x,y]=Matrix to display[x,y]

[0112] Display(Displayed matrix[x,y])

[0113] Alternatively, this corrective function controls the prediction of the following images in relation to the images already calculated (therefore prediction of errors).

[0114] The pseudo code is for example the following:

[0115] Matrix to display[t,x,y] reads corresponds to the image number. Contains the x images.

[0116] Displayed matrix[x,y]

[0117] while(true)

[0118] Displayed matrix[x,y]=prediction of the next image(Matrix to display[t,x,y])

[0119] Display(Displayed matrix[x,y])

[0120] Alternatively, this corrective function controls the display of the displayed images slightly delayed to display transitions between each image (no prediction so it is certain or almost certain).

[0121] The pseudo code is for example the following:

[0122] Matrix to display[t,x,y] / / t corresponds to the image number. Contains the last x images (between 2 and 10).

[0123] Displayed matrix[x,y]

[0124] t intermediate^ / / percentage of the display progress between the front last image and the last

[0125] while(true)

[0126] Displayed matrix[x,y]=image prediction(Matrix to display[t,x,y],t intermediate)

[0127] Display(Displayed matrix[x,y])

[0128] t intermediate + =x / / x calculated so that 100% corresponds to the penultimate display before having the new image to display.

[0129] if(t intermediate^)

[0130] intermediate t = 0

[0131] Corrective function “Display of two information channels”,

[0132] In the prior art, the distance is displayed as a black and white screen which only displays the brightness. The method according to the invention has the ability to display two channels at a time, thus as a color blind.

[0133] Channels can display: Depth, brightness, red, green, blue, cyan, magenta, yellow, hue, speed, object size, materials.

[0134] The pseudo code is for example the following:

[0135] Image to display[x,y]={Canall,Canal2}

[0136] Channel 1 Scan Display

[0137] T=x / / duration of an image

[0138] t=0 / / current time

[0139] while(t <T){

[0140] for(i=0;i <x;i++)

[0141] for(j=0;j <y;j++){

[0142] image to display[i,j]. Channel 1 ==t)

[0143] activateSolenoide(i,j) / / activation of an actuator pulse

[0144] }

[0145] }

[0146] wait(y) / / duration to wait

[0147] t+=y

[0148] }

[0149] Channel 2

[0150] A number of pulses of the actuator at the time of its activation relative to the scanning display rules. This number of pulses corresponds to the value to be transmitted.

[0151] The pseudo code is for example the following:

[0152] activateSolenoide(i,j) / / is replaced by activateSolenoideXImplusion(i,j)

[0153] activateSolenoideXImplusion(i,j){ / / activates each actuator several times

[0154] for(k=0;k <lmage à afficher[i,j].canal2;k++){

[0155] activateSolenoide(i,j)

[0156] wait(x)

[0157] }

[0158] }

[0159] A frequency with a number of pulses defined according to the value to be transmitted at the time of the layer. The frequency gives the transmitted value.

[0160] The pseudo code is for example the following:

[0161] activateSolenoide(i,j) / / is replaced by activateSolenoideXFrequency(ij)

[0162] activateSolenoideXFrequency(i,j){ / / activates each actuator several times

[0163] for(k=0;k <x;k++ff / / x compris entre 1 et 20

[0164] activateSolenoide(ij)

[0165] wait(1 / Image to displayfi,j].channel2)

[0166] }

[0167] }

[0168] A number of pulses and a frequency depending on the value to be transmitted at the time of the layer. The frequency and the number of pulses give the transmitted value. The pseudo code is for example the following:

[0169] activateSolenoide(i,j) / / is replaced by activateSolenoideXImpulse&Frequency(ij)

[0170] activateSolenoideXImpulse&Frequency(i,j){ / / activates each ac- several times tioner

[0171] for(k=0;k <lmage à afficher[i,j].canal2*x;k++){ / / x facteur pour faire une corres consistent correspondence between frequency and pulse number

[0172] activateSolenoide(i,j)

[0173] wait(l / Image to display[i,j].channel2)

[0174] }

[0175] }

[0176] According to a variant, the variable to be transmitted is transmitted by the ratio of the number of times that the actuator is displayed on several images. For example, on 10 successive images displayed, the actuator is activated 5 times on the intended layer, therefore one image out of two. The information transmitted is therefore 50%.

[0177] The pseudo code is for example the following:

[0178] T=x / / duration of an image

[0179] for(k=0;k <kMax;k++){ / / kMaxprenant une valeur entre 1 et 100

[0180] t=0 / / current time

[0181] while(t <T){

[0182] for(i=0;i <x;i++)

[0183] for(j=0;j <y;j++){

[0184] if(Image to display[i,j].Channel 1 ==t &&

[0185] k multiple of the rounding (kMax / Image to display[i,j]. Channel2))

[0186] activateSolenoide(i,j)}

[0187] }

[0188] wait(y) / / duration to wait

[0189] t+=y

[0190] }

[0191] }

[0192] Other variants consist of informing about the information transmitted by the channels using colors projected onto the glasses or the retina. Regularly (between 1 and 120 seconds), the color is displayed to inform that the information from channel 1 and / or channel 2 correspond to x information.

[0193] A watermark on the displayed images may inform about the data contained in each channel. The watermark may inform about the data type of one or both channels.

[0194] Corrective function “Display of hole in the ground or absence of information”

[0195] According to this variant, a percentage of the bottom of the display is used to display negative heights, for example a curb drop, a hole in the ground. Thus instead of permanently displaying the ground, it is only displayed when there is no longer any ground.

[0196] The pseudo code is for example the following:

[0197] Matrix to display[x,y]=z / / source matrix

[0198] Hole[x,z]=y / / the matrix contains the estimated depth of each of the points of the plane corresponds to the one on which the user walks, the depth goes from 0 to Variable 1% of y. For a Variable 1% corresponding to the ordinate devoted to the information of the holes

[0199] HoleToXY[x,y]=MatrixConversion(Hole[x,z]) / / Converts the matrix to x,y = z format

[0200] Displayed matrix[x,y *VanaZ?Ze7%]=Concatenate(HoleInXy / %, y / ,ZVtorzce to display[x,y])

[0201] According to a variant, a percentage of the bottom of the display is used to display the relief of the ground. Whether there is a bump or a hole in the ground.

[0202] The pseudo code is for example the following:

[0203] Matrix to display[x,y]=z / / source matrix

[0204] Hole[x,z]=y / / the matrix contains the estimated depth of each of the points of the plane corresponds to the one on which the user walks, the depth goes from 0 to Variable 1% of y. For a (Variablel + Variable2)% corresponding to the ordinate devoted to the information of the relief of the ground

[0205] HoleInXY[x,y]=ConversionOfMatrix(Hole[x,z]) / / Converts the matrix for that it passes to the format x,y = z

[0206] Bosse[x,z]=y / / the matrix contains the estimated positive height of each of the points of the plane corresponding to the one on which the user walks, the depth goes from 0 to Variable2% of y. For (Variablel + Variable2)% corresponding to the ordinate dedicated to the information of the relief of the ground

[0207] BosseEnXY[x,y]=ConversionDeEaMatrice(Bosse[x,z]) / / Converts the matrix so that it passes to the format x,y = z

[0208] Displayed matrix[x,y*(Variablel + Variable2)%] = Concatenate(HoleInXY[x,y], BosseEnXY[x,y], Matrix to display[x,y])

[0209] Displaying better resolution than pixel count

[0210] Using multiple images to display better resolution when the display speed is very fast. Each displayed pixel corresponds to a set of pixel to display. For each image displayed, a displayed pixel corresponds to one of the pixels it must represent selected according to a pattern. The set of images displayed covers a large part, if not all, of the pixels it must represent.

[0211] On average the pixel values ​​correspond to the average value of the pixels which here represents the pixel worth once 0 and once 1 or 0.5 and it represents the pixels 0 and 1 or also 0.5. The brain gathers the objects by their largest set. So the brain will perceive more pixels than display. Multitasking

[0212] Multitasking will allow you to have several windows visible at the same time, just like on a computer.

[0213] Our brain is only able to analyze one shape / image at a time. If the screen is separated into several zones, our brain will be able, with a lot of learning, to analyze the zone we want, but the technology will not be sufficient to offer a decent resolution.

[0214] The screen will not be separated into several zones but into display periods. A bit like a processor with a single core that sequences the processing of several tasks to process them "at the same time", or as we do with our brain. Our eyes abruptly change target and we immediately analyze this information without mixing it with the previous ones.

[0215] Operation:

[0216] Over a long period (example 1 second) for two videos / images to be displayed:

[0217] The images alternate at the screen display speed, so for example if the screen displays at 30 frames per second, odd frames are frame 1 and even frames are frame 2. Each frame will be displayed 15 times.

[0218] For short periods, display 1 and 2 alternate every x ms.

[0219] The number of displays at the same time is between 1 and 5.

[0220] The importance of the displays can be variable, some displays can be displayed more than others. The importance varies between 1 and 100. 100 means that the display appears 100 times more than a display with an importance of 1. Simplifying images

[0221] One variation for simplifying images is to remove pixels in the middle of similar pixels to reduce power consumption and improve perception and to keep the outlines unchanged and remove pixels following a pattern within these shapes.

[0222] For example for a layer

[0223] 0011111

[0224] 0011111

[0225] 0001111

[0226] 0001111

[0227] 0000000

[0228] becomes

[0229] 0011111

[0230] 0010101

[0231] 0001011

[0232] 0001111

[0233] 0000000

[0234] Display bypassing the actuator activation speed limitation

[0235] The images activate only one actuator out of x (between 2 and 4). For example, to display one actuator out of two. The even images display the actuators that meet the condition:

[0236] If (x is even and y is even) or (x is odd and y is odd)

[0237] And the other actuators for odd images.

[0238] Following a pattern seeking to activate the maximum number of actuators present on the contour of the shapes while never having an actuator activated more than all the x images. For an x ​​between 2 and 4.

[0239] Thus the display can be faster and / or consume less energy Adaptive display

[0240] Another variant consists of adapting the display to the environment so that in a calm environment without much change, the display is “lighter”, consumes less and uses the components less. A calm environment is an environment which does not change much (object movement, appearance and disappearance of objects) and whose sensors move little or slowly.

[0241] Adapting the display speed can reduce its operating speed by up to 1%.

[0242] With the reduction in the number of active pixels, each frame can have up to 90% of the actuators inactive and the inactive actuators are constantly changing.

[0243] APPENDIX

[0244] Example algorithm for processing the horizontal field of view Where is the stata ssssstafta at 41? <h2 style=";text-align:left;direction:ltr">.tas\<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> O&r istatata r » â&*t ™ “ 1 | V ïjf<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> _ V .. . . . / <h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> {wmte««M8»|Fteit |^v~ ||<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> taSM ta S: s 1$<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> Ow wî?wW &<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> & EO<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> , & - 7 “77-----~—~:<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> 8^ * •? <$W W8S^« «}$<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> / / hw$s ®s tta-csta* à® ¥ w®w:®<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> / / ^tatatasta:, ®t<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> < V .. ÎSt'<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> & Sa fe®£tiass Ssg est ona&ssg&is sîs^Ssmssst e& Sangla»: idf&--■♦■ $<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> * fess SS<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> Æf w » & «08$ W«? " sfe / <h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> FW ste tesscS® i<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> ÿsssïS ® s Sis sswSS'fé® ^gw -^»af #©$!»■ ' sçswiïSés' Jss «sfejfs ssf^ffsw®s<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> & a W<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> fe®- ïsssife t®sti$sW ;<h2 style=";text-align:left;direction:ltr"> W»f that is & «® ££W Sea sstasfta father : state » Order.? &that &■ ; $mjwta£^w Ff stays ta g to W s^s»wtat^s®£g - » ümwnsg to father s^tata®' Fta is ta^tas / ®»* taïta® *

[0245] Exemplary algorithm for haptic imaging application j'îTVSÇ I am? 4? s H-' jf secret i$? ;J to & '« •■ {'ass^yes' » WJ-», / 1 > 5 <i s5\.><h2 style=";text-align:left;direction:ltr">« .S: ;<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> «sss&œss^S^^ £ :<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> : * & « j<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> ■^^ïssis^i ;<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> ■- i unJ»*o8ftssr ^^s'ô;»:<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> :£<»'s? Sssss ® »»;» sssss^ $ a^M^mÿjoæ-aeM! ;<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> ■ æ ««S^SS^to<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> ss^jKMzvii' JA^r £cs<£ : / / r sssssg» «mue» «ss «ss»sssf<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> &&$?>«* «S»*$$SS$<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> ; §ÿ sæ<5 issssiwn^s "'■wrsisîw asf s. tsss<.«s. s È J jfe£S®.ï C'ss- <r< sv<* j» f-ïs'w iS? fr» > -f<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> i asassâàsÆs^^^ M:StÇ-f\ïWÆ<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> . < / «"«Afï-twf z^- i* swÿs^.'w* j V'i « S s-wïs;,» ; ÜtVtSsS? u;5> s / we bs>s' «f»»»*-? xïrfs' i- ^s.-î^KïtïM.f ÿssjxS».wt^i#Sf_ ÎW£'<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> <Ce^s? wm&¥ ® wib&f istes M :<h2 style=";text-align:left;direction:ltr"> ; oiær w^W wæflâæs* s< : £réé<«®^ »l»w« •(»» ; Crwer jïwsOsSssk.® ~ X£*«&»£st' ! ? &ir ® i ' î&WOTH^e~ ; O««e mssÈriŒ $Oêse îss^ Wf «arM $< scfee.it = AtQRS €fèe* ÿstr^^e tea^j^ejtttsnr^ ffee&eæ setue^e -: we-mÉ^aü à esÏV sphere S

[0246] Exemplary algorithm for accelerated haptic imaging application :: ; .* SJ'-SSS^ «$S • f^iW^W i - fj^S^S®S§h^®B8» 13^ ■ ■ $ with sw : : SJ IPjWWWÿOi sgasss iasï®M»^a^®tK^®s© ■ - $ •' ^SSHW® FRIENDSHIP Sÿ SixSixSix âf-^K^' ; «$■ for: ,+. ï-ïs:^}<^î' 1^5 *»"■ '«^s^SV5 * À*.' ^«^S; ^JÎ'Jïw^ *4^'5?^ '^V Si 4&WS5 î» J* ,}'S\ Mg *ww 'â$&Am« ^'SS^si'^ ; HKsqEæg.^' ;

Claims

Claims

1. - Method for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt having an active surface of NxM actuators, N and M being integers greater than or equal to 10 comprising an actuator activation protocol consisting of calculating, for each acquisition by a camera of said visual environment, a sequence of P activation frames of said actuators or P is an integer between 2 and 1000, preferably between 5 and 50, each of the frames corresponding to the representation of the environment in an incremental depth plane characterized in that it further comprises a processing step consisting of applying a contextual type corrective function modifying the activation protocol of said actuators.

2. - Method for processing the digital representation of a visual environment to control a haptic interface constituted by a lumbar belt according to claim 1 characterized in that said activation protocol of said actuators consists of activating each actuator Axy corresponding to a voxel V(xyzmin, t) once per image I(t), in the form of N successive sequences S;(t), i being an integer varying between 0 and N, each sequence consisting of activating the actuators Axy>i corresponding to the voxels V(xyzmin, V;, t), V; corresponding to the distance Dxy (t) of the point of the image P relative to the camera acquiring the image I(t).

3. - Method for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt according to claim 1 characterized in that said corrective function consists of modifying the periodicity of the images I(t) as a function of the value V; of the nearest voxel V(xyzmin, Vi, t).

4. - Method for processing the digital representation of a visual environment to control a haptic interface constituted by a lumbar belt according to claim 2 characterized in that said corrective function consists of repeating the sequence S;(t) for the duration of an image I(t).

5. - Method for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt according to claim 1 characterized in that said corrective function consists of weighting the value V; of each voxel V(xyzmin, V;, t), by a coefficient Kxy>i>ten depending on the distance of said voxel V(xyzmin, V;, t) relative to a reference point V(X0, Yo, Zo ,t).

6. - Method for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt according to claim 1 characterized in that said corrective function consists of weighting the value V; of each voxel V(xyzmin, V;, t), by a coefficient Kxy>i>ten depending on the distance Dxy (t) of the point of the image P relative to the image acquisition camera I(t).

7. - Method for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt according to claim 1 characterized in that said corrective function consists of weighting the value Empty each voxel V(xyzmin, Vi, t), by a coefficient Kxy ijten as a function of data coming from an external source distinct from said acquisition camera.

8. - Method for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt according to claim 1 characterized in that said corrective function consists of modulating the value V; of each voxel V(xyzmin, V;, t), for values of Y lower than a threshold value Yo, by assigning a value VH for the areas of the ground corresponding to a hole or a bump, and a value VB otherwise.

9. - Orientation assistance system comprising means for acquiring a real or virtual visual environment, non-visual human-machine interface means and means for processing the digital representation of said visual environment to provide an electrical signal for controlling a haptic interface constituted by a lumbar belt having an active surface of NxM actuators, N and M being integers greater than or equal to 10, said means for processing the digital representation consisting of periodically extracting at least one digital pulse activation pattern from a subset of actuators of said haptic zone and providing for each acquisition of said visual environment a sequence of P activation frames of said actuators or P is an integer between 2 and 1000, preferably between 5 and 50, each of the frames corresponding to the representation of the environment in an incremental depth plane characterized in that it comprises a microcontroller controlling the activation of said actuators according to the method according to the preceding claims.

Images