Method for processing the digital representation of a visual environment for controlling a haptic interface consisting of a lumbar belt

Abstract

The present invention relates to a method and a system for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt having an active surface of N×M actuators, N and M being integers greater than or equal to 10, including an actuator activation protocol consisting in calculating, for each acquisition of the visual environment, a sequence of P activation frames for the actuators, where P is an integer between 2 and 1000, preferably between 5 and 50, each of the frames corresponding to the representation of the environment in an incremental depth plane, characterised in that it further includes a processing step that consists in applying a contextual corrective function modifying the haptic pixel activation protocol.

Description

Method for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt Field of invention

[0001] The present invention relates to the field of orientation assistance for people suffering from visual impairments or people moving in environments with very low visibility, for example firefighters moving in a smoky building or soldiers moving in the dark.

[0002] We know the solution based on transmitting information through haptic means, for example in the form of a connected bracelet. Haptic technology uses the sense of touch to convey information. The company WearWorks offers a smart bracelet called "Wayband" to guide the blind. The user begins by downloading an application on a paired smartphone and stating the desired address. The bracelet, connected to a GPS system, guides the user to their destination. When the user takes the wrong route, the bracelet vibrates. It stops vibrating once they are on the right path. Tactile language, sensitive, more intuitive, and less intrusive, relieves hearing, a sense that is overused in the visually impaired. State of the art

[0003] The applicant itself patented by patent FR3132208A1 an orientation assistance system comprising means for acquiring a real or virtual visual environment, non-visual human-machine interface means and means for processing the digital representation of said visual environment to provide an electrical signal for controlling a haptic interface, said means for processing the digital representation consisting of periodically extracting at least one digital pulse activation pattern from a subset of the pins of said haptic zone, characterized in that said haptic interface is constituted by a lumbar belt having an active surface of NxM actuators, N and M being whole numbers greater than or equal to 10,and in that said processing means provides for each acquisition of said visual environment a sequence of P activation frames of said actuators or P is an integer between 2 and 1000 and preferably between 5 and 50, each of the frames corresponding to the representation of the environment in an incremental depth plane.,

[0004] Patent application US2019065854 relates to a device for providing tactile feedback of an environment using a haptic device, to enable physical detection of objects in an environment. The device uses a plurality of vibration elements arranged in a grid pattern on a part of the user's body, and activates different subsets of the vibration elements in response to the shape of objects in the user's vicinity. The intensity of the vibrations represents the distance to the object or various parts of an object.

[0005] Patent application US2020 / 034980 relates to a system for providing information about an environment to a user within the environment. An electronic processor is configured to receive an input comprising a user selection of an object of interest from among potential objects of interest. The electronic processor is further configured to provide an output for guiding the user to move the sensing apparatus so as to position the object of interest proximate to a reference point in a field of view of the sensing apparatus, obtain multiple images of the object of interest during the user's movement of the sensing apparatus, and crop each of the images to maintain the object of interest proximate to a reference point in each of the images. Disadvantages of the prior art

[0006] These solutions provide only very limited information and do not allow the user to have a sufficient representation of the environment in which he is evolving.

[0007] In particular, to perceive the distances of obstacles, their extent and their height which leads to a permanent feeling of insecurity making any movement very difficult. Solution provided by the invention

[0008] In order to overcome the drawbacks of the prior art, the present invention relates, in its most general sense, to an orientation assistance method having the technical characteristics set out in claim 1.

[0009] The invention particularly relates to a method for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt having an active surface of NxM actuators, N and M being integers greater than or equal to 10, comprising an actuator activation protocol consisting of calculating for each acquisition of said visual environment a sequence of P activation frames of said actuators or P is an integer between 2 and 1000, preferably between 5 and 50, each of the frames corresponding to the representation of the environment in an incremental depth plane, characterized in that it further comprises a processing step consisting of applying a contextual type corrective function modifying the haptic pixel activation protocol.

[0010] According to variants:

[0011] - said actuator activation protocol consists of activating each actuator A xy corresponding to a voxel V(xyz min , t) once per image I(t), in the form of N sequences S i (t) successive, i being an integer varying between 0 and N, each sequence consisting of activating the actuators A xy , i corresponding to the voxels V(xyz min , V i , t), V i corresponding to the distance D xy (t) of the image point P relative to the image acquisition camera I(t).

[0012] - said corrective function consists of modifying the periodicity of the images I(t) as a function of the value V i of the voxel V(xyz min , V i , t) the nearest.

[0013] - said corrective function consists of repeating the sequence S i (t) for the duration of an image I(t).

[0014] - the said corrective function consists of weighting the value Vi of each voxel V(xyz min , V i , t), by a coefficient K xy , i,t depending on the distance from said voxel V(xyz min, Vi, t) par rapport à un point de référence V(X0, Y0, Z0,t).

[0015] - the said corrective function consists of weighting the value V i of each voxel V(xyz min , V i , t), by a coefficient K xy , i,t depending on the distance D xy(t) du point de l’image P par rapport à la caméra d’acquisition de l’image I(t).

[0016] - the said corrective function consists of weighting the value V i of each voxel V(xyz min , V i , t), by a coefficient K xy , i,t based on data from an external source separate from said acquisition camera.

[0017] - the said corrective function consists of modulating the value V i of each voxel V(xyz min , V i , t), for values ​​of Y less than a threshold value Y0, by assigning a value V Hfor areas of the ground corresponding to a hole or a bump, and a value V B Otherwise.

[0018] The invention also relates to an orientation assistance system comprising means for acquiring a real or virtual visual environment, non-visual human-machine interface means and means for processing the digital representation of said visual environment to provide an electrical signal for controlling a haptic interface constituted by a lumbar belt having an active surface of NxM actuators, N and M being integers greater than or equal to 10, said means for processing the digital representation consisting of periodically extracting at least one digital pulse activation pattern from a subset of the pins of said haptic zone and providing for each acquisition of said visual environment a sequence of P activation frames of said actuators where P is an integer between 2 and 1000, preferably between 5 and 50,each of the frames corresponding to the representation of the environment in an incremental depth plane characterized in that it comprises a microcontroller controlling the activation of said actuators according to a method mentioned above.,

[0019] Detailed description of a non-limiting example of embodiment of the invention

[0020] The present invention will be described in more detail with reference to non-limiting exemplary embodiments specifying the above-mentioned advantages and considerations. A more particular description of the invention is briefly described below.

[0021] represents a schematic view of the system according to the invention

[0022] it represents a view of a visual image

[0023] represents a view of a haptic image

[0024] represents a view of a sequence of haptic frames

[0025] represents a schematic view of a vertical tunnel

[0026] represents a schematic view of the field of view as a function of the speed of movement

[0027] represents a schematic view of a horizontal tunnel

[0028] represents an example of an algorithm for processing the horizontal field of view

[0029] represents an example of an algorithm for the application of haptic image

[0030] represents an example of an algorithm for the application of accelerated haptic image General principle of the invention

[0031] The general principle of the present invention consists of carrying out an image acquisition and processing described in the applicant's patent FR3132208A1, and applying a corrective function modulating the pixel activation protocol, each image I(t) resulting in a sequence of N frames. A frame corresponds to the activation of a subset of pixels P(x, y, t, i) assigned the same value V i (t). This value V i (t) is calculated according to the protocol described in the applicant's patent FR3132208A1, modified by a contextual corrective function calculated according to different methods described in a non-limiting manner below. Reminder of the basic protocol

[0032] The example developed below, without limitation, comprises a means of acquiring the environment, for example a frame of glasses (10) equipped with cameras (11, 12) used to acquire data on the environment in real time to provide digital images which control the actions of a haptic transducer. The haptic transducer generates actions in the form of pressures directly or indirectly on the skin by electromagnetic or electromechanical actuators, or in the form of electrical or light pulses).

[0033] It is recalled that the system which is the subject of the invention can also be used for gaming or training applications in augmented reality, with images provided by a video source.

[0034] A computer is responsible for recovering the images from the Sensor part, then generating a 3D depth map from them. It transmits this map to haptic equipment, which includes for example a grid of actuators or pins (small linear actuators that can be raised or lowered), integrated into a back belt (20). This belt (20) is equipped with a set of actuators arranged on supports (21 to 24) to form a matrix, for example of 20 x 40 pixels. These actuators are arranged to form a regular matrix, preferably with a constant pitch. An electronic circuit receives the visual signals and provides the processing to control the actuators in order to produce sensations on the user's back that are easy to interpret after a learning period. The lumbar belt (20) can be worn over a light fabric garment (shirt, polo shirt, blouse) or directly on the skin.

[0035] The surface of the active matrix formed by the actuators covers an extended lumbar region to benefit from good resolution and satisfactory user comfort.

[0036] The restitution of the image of the real environment into haptic images consists of dividing the depth map calculated from the visual image into several successive layers determining virtual or haptic images controlling the activation of the haptic equipment: thus, the closest objects are first displayed, then the slightly more distant objects, and so on until reaching the maximum chosen viewing distance (generally around 10 meters). Thus, we have a sort of scan of the environment which gradually sinks and which displays at each instant what it encounters.This scan results in a burst of virtual images lasting approximately 100 milliseconds, composed of around ten haptic images corresponding to consecutive shots, before resuming with a new burst corresponding to the new environment, resulting from the user's movement or the change in orientation of the real image, due to a modification of the position of the head or the video image. Visual image processing

[0037] The cameras (11, 12) acquire images to reconstruct a digital image with depth information. A first processing operation consists of constructing a depth map (in grayscale). For each pixel of the optical image (100), a haptic image (200) is transcribed in grayscale as a function of the distance of each point from the camera.

[0038] Depth information can also be determined with a single camera with appropriate image processing.

[0039] It should be noted that this haptic image (200) could also, without departing from the invention, be calculated from the digital image provided by a lidar.

[0040] This haptic image (200) is then decomposed into a sequence of incremental haptic frames (301 to 307), each corresponding to a depth plane. The first haptic frame (301) of the sequence corresponds to the obstacle zones in the plane closest to the user, the second first haptic frame (301) of the sequence corresponds to the obstacle zones in the plane closest to the user, the next haptic frame (302) corresponding to the obstacle zones in the next plane, offset from the previous one by one step, for example 30 centimeters in distance and so on.

[0041] The gray level of the haptic frames (301 to 307) encodes the type of action of the corresponding actuator, for example the frequency of the vibration or the duration of the vibration during the activation time of the corresponding haptic frame.

[0042] A haptic image (200) is thus translated into a temporal scan of haptic frames (301 to 307) which are integrated by the user to perceive a depth representation of his environment.

[0043] Other treatments are applied to improve the intelligibility of tactile perceptions: Straightening of the visual image (100) to create a synthetic image reduced to an observation along a horizontal axis descended to the level of the user's legs, a few tens of centimeters from the ground Reinforcement on the visual image (100) of the pixels corresponding to small obstacles, in order to occupy at least one pixel of the haptic image (200). Reprocessing of the holes to reinforce the gray level of the corresponding zone on the haptic image (200) Reinforcement of the gray level on the haptic image (200) of the zone of interest determined by automatic recognition, for example by supervised learning. Examples of treatments for the production of the haptic image (200).

[0044] Examples of processing will be described below, with the following variables:M denotes the haptic matrix applied to the userH denotes the height of this haptic matrix M (in pixels)W denotes the width of this haptic matrix M (in pixels)p denotes the precision levelDM denotes the depth map retrieved by the sensors (11, 12)dmH denotes the height of the retrieved depth map (in pixels) = p*H dmW denotes the width of the retrieved depth map (in pixels) = p*W FOVv denotes the vertical field of view of the camera used (in degrees)FOVh denotes the horizontal field of view of the camera used (in degrees)hu denotes the height at which the camera is placed (user height taken at eye level)DISTANCE_MAX denotes the maximum viewing distance set by the userSPEED denotes the display speed set by the userPAUSE_TIME denotes the pause time between the display of 2 images (set by the user)MAT[x,y] corresponds to the value of the MAT matrix at coordinates [x, y].MAT[x] corresponds to the xth column of the MAT matrix LIST[x] corresponds to the xth value of the list LISTx = f(arguments) means that the value of x is a function of 1 or more arguments (proportionality relation),

[0045] / means that what is written after is a comment Step 1: Obtain a depth map

[0046] This first step consists of calculating a depth map of size dmW*dmH from the two images acquired by the cameras (11) and (12) or by a lidar, or by a source of binocular virtual images.

[0047] These are known treatments, generally comprising: A step of acquiring two images of the same scene at the same time by two cameras (11, 12) whose spacing is known, or else successive images coming from the same sensor. A calibration step consisting of determining the internal and external parameters of the geometric model of the stereoscopic sensor A pixel matching step to find on the two images the pairs of pixels which correspond to the projection of the same element of the scene, A 3D reconstruction step consisting of calculating for each pixel the position in space of the point which was projected into this pixel.

[0048] The result of this first step is a visual image (100) of size dmW*dmH where each point is constituted by a voxel defined by its coordinates in space whose origin is at the level of the user's head, and the x and y axes perpendicular to the line of sight of the cameras, and the z coordinates the distance from the user's head.

[0049] Introducing contextual correction functions

[0050] The following description concerns several examples of contextual corrective functions:Variation of the display according to the distance of the nearest objectVariation of the maximum depth distance according to the position of the pixelMotion blurImage distortionDepth distortionCentralized 3D captureDisplay of depths relative to a rectangleDisplay of depths relative to a planeHigh display frequencyDisplay beyond the capture speed of the camerasDisplay of two information channelsDisplay of a better resolution than the number of pixelsMultitaskingSimplification of images.Display bypassing the limitation of activation speed of the actuators.Adaptive displayAs explained above, the invention relates to the display of images by scanning. The invention relates to the continuity of the display of depth by scanning.

[0051] Each actuator activates once per frame. Nearby images are displayed first, then faraway images.

[0052] Each pixel(x,y) has an associated value which is the distance.

[0053] At t=0 we display the pixels with the value 0

[0054] At t=1 we display the pixels having the value 1

[0055] …

[0056] And we return to 0 to display a new image.

[0057] The pseudo codes in the description of the detailed examples that follow are implementation examples to explain the principle of the corresponding correction functions.

[0058] Display variation depending on the distance to the nearest object

[0059] According to a first exemplary embodiment, the contextual corrective function controls the duration of an image according to the distance of the nearest object by digital processing on the digital data.

[0060] For this purpose, the processing of the data provided by the camera at a time t determines the voxel V (x,y,z,t). Depending on the distance from the voxel V(xyz min , t) the closest, the duration separating two consecutive images varies between a minimum value, for example 1 millisecond when the voxel V(xyz min , t) is at a minimum distance, for example 10 cm and a maximum value, for example 2 seconds when the voxel V(xyz min , t) is at a maximum distance, for example 100m.

[0061] The N frames scroll during the duration of each image, with a frequency that accelerates as the user approaches an obstacle.

[0062] The pseudo code is for example the following:T is the duration of an image (between 1ms and 3 seconds)D is the distance that the image covers on T (between 10cm and 100m)V is the speed of the display, during X ms, the display covers f(X) m. (1cm / s to 1km / s)T=VxDdMin=distance of the nearest voxel.

[0063] The display speed varies depending on the voxel (x, y, z, t) closest to the user. The minimum and maximum display speed is defined. (The display scan speed in km / h varies depending on the distance to the nearest object)

[0064] V=f(dMin) with a minimum V and a maximum V

[0065] T is unchanged and therefore D is changed

[0066] The display time of a scan depends on the distance to the nearest voxel. The display time has a minimum and a maximum. (The duration of an image composed of several layers depends on the distance to the nearest object)

[0067] T=f(dMin) with a minimum T and a maximum T

[0068] V is unchanged and therefore D is impacted

[0069] Alternatively, the maximum display distance depends on the distance of the nearest voxel. The further away the nearest object, the greater the distance we perceive. For example, max distance = X times the distance of the nearest object.

[0070] The pseudo code is for example the following:

[0071] D=f(dMin) with a minimum D and a maximum T

[0072] V is unchanged and T is impacted

[0073] Another variation is to command the omission of a part of the display. The display can start at the nearest voxel and / or end at the furthest voxel.Start of display = f(dMin) with a maximum start of displaydMaxis the furthest voxelEnd of display = f(dMax) with a minimum End of display.D=End of display-Start of displayVis unchanged and therefore T is impactedThese variations can be combined.

[0074] With gray it is if these modifications are not combined if everything is combined:V=f(dMin) with a minimum V and a maximum VT=f(dMin) with a minimum T and a maximum T(D=f(dMin) with a minimum D and a maximum T) or (Start of display = f(dMin) with a maximum start of displayEnd of display = f(dMax) with a minimum End of display.dMax is the furthest voxelD=End of display-Start of display)Variation of the maximum depth distance according to the position of the pixel

[0075] The maximum depth distance chosen varies depending on the position of the pixel in the image being represented. The closer the pixel is to the center, the higher the maximum distance will be, and conversely, the closer the pixel is to the edges, the lower the maximum distance will be.

[0076] This variation in maximum depth distance roughly follows the shape of a half-ellipsoid. Motion blur correction function

[0077] This variation is intended to create motion blur to highlight moving objects. The faster an object moves, the more it will be stretched in the direction of its movement.

[0078] Motion blur is caused by displaying images slower than the capture rate. Thus, the image to be displayed is updated several times while displaying a single image. Each time the image to be displayed is updated, each voxel is updated by the value of the closest voxel between the image to be displayed and the new image. Each voxel is displayed only once.

[0079] The pseudo code is for example the following:

[0080] Matrix to display[x,y]=Distance from voxel (updated more frequently than the displayed image)

[0081] Displayed matrix[x,y]=Distance from voxel

[0082] Actuators already activated[x,y]=0 / 0 for actuator that has not been activated, 1 for actuator that has already been activated

[0083] Actuatorsactivated[x,y]=0 / 0 for actuator not activated, 1 for actuator that must be activated

[0084] Current distance = 0

[0085] Increment distance = x / distance in cm between 0.1 and 1000

[0086] Maximum distance = x / between 2 and 10,000 times the increment distance

[0087] while(Current Distance <Distance max){

[0088] for(i=0;i <x;i++)

[0089] for(j=0;j <y;j++){

[0090] if(Displayed matrix[i,j] <Matrice à afficher[i,j])

[0091] Displayed matrix[i,j]=Matrix to display[i,j]

[0092] if(Displayed matrix[i,j] <Distance actuelle andActuatorsdéjà activés[i,j] == 0 )

[0093] Actuatorsactivated[i,j]=1

[0094] Actuators already activated[i,j]=1

[0095] }

[0096] activationdesactuators(Actuatorsactivated)

[0097] wait(x ms) / x between 0.1 and 1000

[0098] Current Distance + = Increment Distance

[0099] Actuatorsactivated[x,y]=0 / reset all values ​​to 0

[0100] A variation involves artificial motion blur, which uses motion blur to emphasize moving objects based on the speed of movement of objects in the environment. Objects can be rendered up to 10 times their size without motion blur. Corrective function “Image distortion”

[0101] Pixels in the center of the display are magnified, and the closer you move to the edges of the images, the smaller they are. The distortion also varies along the vertical axis. A pixel can be magnified up to 10 times and reduced by a factor of 10.

[0102] This function controls a variable deformation depending on the voxel distance. If a deformation can be achieved by a single lens, it must be imagined that for each different voxel distance a different lens is applied. The further the pixels are, the greater the deformations. The impact of voxel deformations ranges from x100 to divided by 100.

[0103] These deformations will complement the elements trimmed according to the claims of patent FR3132208A1.

[0104] Corrective function “Depth distortion”

[0105] This corrective function controls the display of distances in a non-linear way. Depth is currently displayed linearly and must be converted to compress long distances. For example, switching to a perception accuracy in % of the pixel distance and not in m. So a layer that displays voxels 10ms later means that it is x% further than the previous one and not x meters further.

[0106] The pseudo code is for example the following:

[0107] Value displayed for distance =f(Voxel Distance) “Centralized 3D Capture” corrective function

[0108] 3D image capture is currently done locally and in real time or almost. To ensure that the images to be displayed are of much better quality, this corrective function controls the display made of collaborative depth images. Each person who looks with the device collects data so we know exactly where they are and what they are looking at, they send their depth image to the server which can refine its model and receives the depth image calculated by all the people who have already been there. And for latency, it is easy to predict what the person will have in their field of vision in the next minute and therefore send them in advance the objects they will see.

[0109] Displaying depths relative to a rectangle

[0110] This correction function allows the depth of objects to be represented by taking the user as the reference point and not the camera. For the sake of simplification, the user is represented as a rectangle in space.

[0111] In the prior art, the user was represented by a segment running from his feet to his head. Displaying depths relative to a plane

[0112] This corrective function allows the depth of objects to be represented by taking the vertical plane on which the user is located as a reference and not the camera. High display frequency

[0113] This corrective function controls the display of a large number of images per second.

[0114] When the display frequency is low, the user may have the impression that the images are jerky. From a certain display speed, the perceived information is much more fluid, the user has the impression that the flow of images is continuous.

[0115] Fixed "Display beyond camera capture speed"

[0116] This corrective function controls the display of images faster than the sensors' perception capabilities, thus providing image redundancy. Display of the last captured image in a loop.

[0117] The pseudo code is for example the following:

[0118] Matrix to display[x,y] / contains the values ​​to display

[0119] Displayed matrix[x,y]

[0120] while(true)

[0121] Displayed matrix[x,y]=Matrix to display[x,y]

[0122] Display(Displayed matrix[x,y])

[0123] Alternatively, this corrective function controls the prediction of the following images in relation to the images already calculated (therefore prediction of errors).

[0124] The pseudo code is for example the following:

[0125] Matrix to display[t,x,y] / t corresponds to the image number. Contains the x images.

[0126] Displayed matrix[x,y]

[0127] while(true)

[0128] Displayed matrix[x,y]=next frame prediction(Matrix to display[t,x,y])

[0129] Display(Displayed matrix[x,y])

[0130] Alternatively, this corrective function commands the display to delay the displayed images a little to display transitions between each image (no prediction so it is certain or almost).

[0131] The pseudo code is for example the following:

[0132] Matrix to display[t,x,y] / t corresponds to the image number. Contains the last x images (between 2 and 10).

[0133] Displayed matrix[x,y]

[0134] t intermediate=0 / percentage of display progression between the penultimate image and the last

[0135] while(true)

[0136] Displayed matrix[x,y]=image prediction(Matrix to display[t,x,y],t intermediate)

[0137] Display(Displayed matrix[x,y])

[0138] t intermediate+=x / x calculated so that 100% corresponds to the penultimate display before having the new image to display.

[0139] if(intermediate t>1)

[0140] intermediate t = 0

[0141] Corrective function “Display of two information channels”.

[0142] In the prior art, distance is displayed as a black and white screen that only displays brightness. The method according to the invention has the ability to display two channels at once, thus as a color blind.

[0143] Channels can display: Depth, brightness, red, green, blue, cyan, magenta, yellow, hue, speed, object size, materials.

[0144] The pseudo code is for example the following:

[0145] Image to display[x,y]={Channel1,Channel2}

[0146] Channel 1 Scan Display

[0147] T=x / duration of an image

[0148] t=0 / current time

[0149] while(t <T){

[0150] for(i=0;i <x;i++)

[0151] for(j=0;j <y;j++){

[0152] if(Image to display[i,j].Channel1==t)

[0153] activateSolenoide(i,j) / activation of a pulse from the actuator

[0154] }

[0155] }

[0156] wait(y) / duration to wait

[0157] t+=y

[0158] }

[0159] Channel 2

[0160] A number of pulses of the actuator at the time of its activation compared to the scanning display rules. This number of pulses corresponds to the value to be transmitted.

[0161] The pseudo code is for example the following:

[0162] activateSolenoide(i,j) / is replaced by activateSolenoideXImplusion(i,j)

[0163] activateSolenoideXImplusion(i,j){ / activates each actuator several times

[0164] for(k=0;k <Image à afficher[i,j].canal2;k++){

[0165] activateSolenoide(i,j)

[0166] wait(s)

[0167] }

[0168] }

[0169] A frequency with a number of pulses defined according to the value to be transmitted at the time of the layer. The frequency gives the transmitted value.

[0170] The pseudo code is for example the following:

[0171] activateSolenoide(i,j) / is replaced by activateSolenoideXFrequency(i,j)

[0172] activateSolenoideXFrequency(i,j){ / activates each actuator several times

[0173] for(k=0;k <x;k++){ / x compris entre 1 et 20

[0174] activateSolenoide(i,j)

[0175] wait(1 / Image to display[i,j].channel2)

[0176] }

[0177] }

[0178] A number of pulses and a frequency depending on the value to be transmitted at the time of the layer. The frequency and the number of pulses give the transmitted value. The pseudo code is for example the following:

[0179] activateSolenoide(i,j) / is replaced by activateSolenoideXImpulse&Frequency(i,j)

[0180] activateSolenoideXImpulse&Frequency(i,j){ / activates each actuator several times

[0181] for(k=0;k <Image à afficher[i,j].canal2*x;k++){ / x facteur pour faire une correspondance cohérente entre les fréquence et le nombre d’impulsion

[0182] activateSolenoide(i,j)

[0183] wait(1 / Image to display[i,j].channel2)

[0184] }

[0185] }

[0186] Alternatively, the variable to be transmitted is transmitted by the ratio of the number of times the actuator is displayed on several images. For example, out of 10 successive images displayed, the actuator is activated 5 times on the intended layer, therefore one image out of two. The information transmitted is therefore 50%.

[0187] The pseudo code is for example the following:

[0188] T=x / duration of an image

[0189] for(k=0;k <kMax;k++){ / kMaxprenant une valeur entre 1 et 100

[0190] t=0 / current time

[0191] while(t <T){

[0192] for(i=0;i <x;i++)

[0193] for(j=0;j <y;j++){

[0194] if(Image to display[i,j].Channel1==t &&

[0195] kmultiple of the rounding (kMax / Image to display[i,j].Channel2))

[0196] activateSolenoide(i,j)}

[0197] }

[0198] wait(y) / duration to wait

[0199] t+=y

[0200] }

[0201] }

[0202] Other variants involve informing about the information transmitted by the channels using colors projected onto the glasses or the retina. Regularly (between 1 and 120 seconds), the color is displayed to inform that the information from channel 1 and / or channel 2 corresponds to x information.

[0203] A watermark on the displayed images can inform about the data contained in each channel. The watermark can inform about the data type of one or both channels.

[0204] Corrective function “Display of hole in the ground or absence of information”

[0205] In this variant, a percentage of the bottom of the display is used to display negative heights, for example a curb drop, a hole in the ground. So instead of permanently displaying the ground, it is only displayed when there is no longer any ground.

[0206] The pseudo code is for example the following:

[0207] Matrix to display[x,y]=z / source matrix

[0208] Hole[x,z]=y / the matrix contains the estimated depth of each of the points of the plane corresponding to the one on which the user walks, the depth goes from 0 to Variable1% of y. For a Variable1% corresponding the ordinate devoted to the information of the holes

[0209] HoleToXY[x,y]=MatrixConversion(Hole[x,z]) / Converts the matrix to x,y=z format

[0210] Displayed matrix[x,y*Variable1%]=Concatenate(HoleInXY[x,y],Matrix to display[x,y])

[0211] Alternatively, a percentage of the bottom of the display is used to display the ground relief. Whether there is a bump or a hole in the ground.

[0212] The pseudo code is for example the following:

[0213] Matrix to display[x,y]=z / source matrix

[0214] Hole[x,z]=y / the matrix contains the estimated depth of each of the points of the plane corresponding to the one on which the user walks, the depth goes from 0 to Variable1% of y. For a (Variable1+Variable2)% corresponding to the ordinate dedicated to the information of the relief of the ground

[0215] HoleToXY[x,y]=MatrixConversion(Hole[x,z]) / Converts the matrix to x,y=z format

[0216] Bosse[x,z]=y / the matrix contains the estimated positive height of each of the points of the plane corresponding to the one on which the user walks, the depth goes from 0 to Variable2% of y. For (Variable1+Variable2)% corresponding to the ordinate dedicated to the information of the relief of the ground

[0217] BosseEnXY[x,y]=ConvertDeLaMatrice(Bosse[x,z]) / Converts the matrix so that it passes to the format x,y= z

[0218] Displayed matrix[x,y*(Variable1+Variable2)%] = Concatenate(HoleInXY[x,y],BumpInXY[x,y], Matrix to display[x,y])

[0219] Displaying a higher resolution than the number of pixels

[0220] Using multiple images to display better resolution when the display speed is very fast. Each displayed pixel corresponds to a set of pixels to be displayed. For each displayed image, a displayed pixel corresponds to one of the pixels it must represent selected according to a pattern. The set of displayed images covers a large part, if not all, of the pixels it must represent.

[0221] On average, the pixel values ​​correspond to the average value of the pixels, which here represents the pixel worth once 0 and once 1, i.e. 0.5, and it represents pixels 0 and 1, i.e. also 0.5. The brain groups objects by their largest set. So the brain will perceive more pixels than display. Multitasking

[0222] Multitasking will allow you to have several windows visible at the same time, just like on a computer.

[0223] Our brain is only able to analyze one shape / image at a time. If the screen is separated into several zones, our brain will be able to analyze the desired area with a lot of learning, but the technology will not be sufficient to offer a decent resolution.

[0224] The screen will not be divided into several zones but into display periods. A bit like a processor with a single core that sequences the processing of several tasks to process them "at the same time," or as we do with our brain. Our eyes abruptly change target and we immediately analyze this information without mixing it with the previous ones.

[0225] Functioning :

[0226] Over a long period (example 1 second) for two videos / images to display:

[0227] The images alternate at the screen display speed, so for example if the screen is displaying at 30 images per second, the odd images are image 1 and the even images are image 2. Each of the images will be displayed 15 times.

[0228] For short periods, display 1 and 2 alternate every x ms.

[0229] The number of displays at the same time is between 1 and 5.

[0230] The importance of displays can vary; some displays may be displayed more than others. Importance ranges from 1 to 100. 100 means that the display appears 100 times more than a display with an importance of 1. Simplifying images

[0231] A variation for image simplification is to delete pixels in the middle of similar pixels to reduce power consumption and improve perception, and to keep the outlines unchanged and delete pixels following a pattern within these shapes.

[0232] For example for a layer

[0233] 0011111

[0234] 0011111

[0235] 0001111

[0236] 0001111

[0237] 0000000

[0238] becomes

[0239] 0011111

[0240] 0010101

[0241] 0001011

[0242] 0001111

[0243] 0000000

[0244] Display bypassing actuator activation speed limitation

[0245] The images activate only one actuator out of x (between 2 and 4). For example, to display one actuator out of two. The even images display the actuators that meet the condition:

[0246] If (x is even and y is even) or (x is odd and y is odd)

[0247] And the other actuators for odd frames.

[0248] Following a pattern seeking to activate the maximum number of actuators present on the outline of the shapes while never having an actuator activated more than every x images. For an x ​​between 2 and 4.

[0249] This way the display can be faster and / or consume less energy Adaptive display

[0250] Another variation is to adapt the display to the environment so that in a quiet environment without much change, the display is “lighter”, consumes less and uses less components. A quiet environment is one that changes little (object movement, object appearance and disappearance) and whose sensors move little or slowly.

[0251] Adapting the display speed can reduce its operating speed by up to 1%.

[0252] With the reduction in the number of active pixels, each frame can have up to 90% of the actuators inactive, and the inactive actuators are constantly changing.

Claims

– Method for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt having an active surface of NxM actuators, N and M being integers greater than or equal to 10 comprising an actuator activation protocol consisting of calculating for each acquisition of said visual environment a sequence of P activation frames of said actuators or P is an integer between 2 and 1000, preferably between 5 and 50, each of the frames corresponding to the representation of the environment in an incremental depth plane characterized in that it further comprises a processing step consisting of applying a contextual type corrective function modifying the haptic pixel activation protocol. - Method for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt according to claim 1 characterized in that said actuator activation protocol consists of activating each actuator A xy corresponding to a voxel V(xyz min , t) once per image I(t), in the form of N sequences S i (t) successive, i being an integer varying between 0 and N, each sequence consisting of activating the actuators A xy , i corresponding to the voxels V(xyz min , V i , t), V i corresponding to the distance D xy (t) of the image point P relative to the image acquisition camera I(t). - Method for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt according to claim 1 characterized in that said corrective function consists of modifying the periodicity of the images I(t) as a function of the value V i of the voxel V(xyz min , V i , t) the nearest. - Method for processing the digital representation of a visual environment to control a haptic interface constituted by a lumbar belt according to claim 1 characterized in that said corrective function consists of repeating the sequence S i (t) for the duration of an image I(t). - Method for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt according to claim 1 characterized in that said corrective function consists of weighting the value V iof each voxel V(xyz min , V i , t), by a coefficient K xy , i,t depending on the distance from said voxel V(xyz min, Vi, t) par rapport à un point de référence V(X0, Y0, Z0,t). - Method for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt according to claim 1 characterized in that said corrective function consists of weighting the value V i of each voxel V(xyz min , V i , t), by a coefficient K xy , i,t depending on the distance D xy(t) du point de l’image P par rapport à la caméra d’acquisition de l’image I(t). - Method for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt according to claim 1 characterized in that said corrective function consists of weighting the value V i of each voxel V(xyz min , V i , t), by a coefficient K xy, i,t based on data from an external source separate from said acquisition camera. - Method for processing the digital representation of a visual environment to control a haptic interface consisting of a lumbar belt according to claim 1 characterized in that said corrective function consists of modulating the value V i of each voxel V(xyz min , V i , t), for values of Y less than a threshold value Y0, by assigning a value V H for areas of the ground corresponding to a hole or a bump, and a value V B Otherwise. - Orientation assistance system comprising means for acquiring a real or virtual visual environment, non-visual human-machine interface means and means for processing the digital representation of said visual environment to provide an electrical signal for controlling a haptic interface constituted by a lumbar belt having an active surface of NxM actuators, N and M being integers greater than or equal to 10, said means for processing the digital representation consisting of periodically extracting at least one digital pulse activation pattern from a subset of the pins of said haptic zone and providing for each acquisition of said visual environment a sequence of P activation frames of said actuators where P is an integer between 2 and 1000, preferably between 5 and 50,each of the frames corresponding to the representation of the environment in an incremental depth plane characterized in that it comprises a microcontroller controlling the activation of said actuators according to the method according to the preceding claims.,

Images