Method and system for assisting the movement of a visually impaired person

Abstract

The invention relates to a method for assisting the movement of a visually impaired person, said method comprising a spatial data acquisition step (101), a pixelated image generation step (102). For each pixel, the method comprises a pixel coordinate determination step (103), a pixel depth determination step (104), and a pixel danger level determination step (105). The method further comprises a step (107) for determining a direction (Dir) of movement and an audio guidance step (108) for the visually impaired person along the determined direction (Dir). Figure 1 is shown in the abstract.

Description

Title of the invention: Method and system for assisting the movement of a visually impaired person technical field

[0001] The present invention relates to the field of mobility assistance for visually impaired persons. In particular, the invention concerns a method and a portable system for mobility assistance for such persons. The term "visually impaired person" here encompasses both moderately visually impaired persons and persons with severe visual impairments (such as blindness). Prior Art

[0002] According to a recent study, more than 335 million people worldwide have severe visual impairment, of whom 42 million are blind. These individuals require assistance from a third party to get around or the help of an animal. Guide dogs play a crucial role in this area. Unfortunately, such dogs are a scarce resource, and few people can benefit from them. Furthermore, dog allergies also hinder their use.

[0003] Numerous technological solutions exist to address the mobility challenges faced by visually impaired individuals. One such solution is based on ultrasonic obstacle detection integrated into a white cane. Sweeping the cane detects obstacles and returns the reflected ultrasonic signal. A sound is then emitted by a speaker integrated into the cane or by a Bluetooth™-type earpiece to inform the user of the obstacle's presence.

[0004] A solution based on a helmet positioned on the head of the person being assisted, which allows for assistance without a cane, is also known. However, the use of such a helmet is not very intuitive for the user, and its weight is also a deterrent to its use.

[0005] There is therefore a need to offer a mobility assistance solution for visually impaired people that is simple to implement and practical to use. Summary of the invention

[0006] A first object of the invention relates to a method for assisting visually impaired persons, said method comprising:

[0007] - a step of acquiring, by a camera, spatial data of an environment in which the visually impaired person is able to move around;

[0008] - a step of generating a pixelated image from said spatial data said pixelated image comprising a plurality of pixels;

[0009] - for each pixel of the plurality of pixels:

[0010] - a step of determining the coordinates of the pixel;

[0011] - a step of determining a pixel depth;

[0012] - a step of determining a level of danger of the pixel;

[0013] - a step of generating a 3D point cloud, each point of the cloud of points being associated with a pixel of the pixelated image, each point of the point cloud being positioned in the point cloud by point coordinates, said point coordinates being determined from the coordinates of the pixel associated with said point and from the depth of said pixel, said point of the point cloud being affected by the danger level of said pixel;

[0014] - a step of determining a direction of movement for the person to visual impairment based on the coordinates of all or part of the points in the 3D point cloud and the danger levels assigned to said points;

[0015] - an audio guidance step for the visually impaired person according to the determined direction of movement.

[0016] The invention thus makes it possible to model and describe the user's environment. This environment is constituted as a three-dimensional geometric point cloud. The invention proposes using a single, very low-cost camera and constructing the 3D environment as a point cloud, each point of which has three metric coordinates along the x, Y, and Z axes relative to the camera, from a single image, by assigning, on the one hand, an object class semantics, and on the other hand, a danger semantics to each point of the 3D representation.

[0017] The invention thus makes it possible to guide a person with a visual impairment using spatial sound in order to avoid obstacles by indicating a safe direction of movement to follow.

[0018] In an alternative embodiment, the coordinates of the point are determined according to the following equation: HN • =[«■' * À UV .1. . 1. with 2 a scaling factor related to camera type and [K] an intrinsic camera matrix, such that: [K] = fx 0 o fy . oo , with fx a focal length of the camera in number of pixels following a first direction x, fy a focal length of the camera in number of pixels along a second direction y, Cx a first pixel coordinate of the intersection of an optical axis of the camera with an image plane, Cy a second pixel coordinate of the intersection of the optical axis of the camera with said image plane.

[0019] In one embodiment, the step of determining the danger level of the pixel includes a step of determining for said pixel a pixel semantics, said pixel danger level being obtained from said pixel semantics.

[0020] In one embodiment, each point of the 3D point cloud having a height relative to a ground, the step of determining a direction of movement includes a step of eliminating points from the 3D point cloud whose height is greater than a certain value, to form a corrected 3D point cloud.

[0021] In one embodiment, said elimination step takes into account an inclination and / or an irregularity of the ground.

[0022] In one embodiment, the step of determining a direction of movement includes a step of projecting onto the ground the points of the corrected 3D point cloud in order to determine the direction of movement.

[0023] In one embodiment, the audio guidance step for the visually impaired person according to the determined direction of movement includes a step for generating a spatial 3D sound.

[0024] In one embodiment, the audio guidance step includes a step for correcting the generated spatial 3D sound, said correction being carried out from a 3x3 rotation matrix R and a 1x3 translation matrix T of the camera so as to guide the person independently of the rotation and translation movements of said camera.

[0025] Another object of the invention relates to a system for assisting the movement of a visually impaired person, said system comprising a portable device intended to be positioned on the head of said person and a processing unit, said portable device comprising a camera adapted to acquire spatial data of an environment in which the visually impaired person is able to move, said portable device comprising an audio module for providing audio guidance to the visually impaired person along a direction of movement determined from the spatial data, said system comprising at least one processor configured to implement:

[0026] - a step of acquiring spatial data by the camera;

[0027] - a step of generating a pixelated image from the spatial data, said pixelated image containing a plurality of pixels;

[0028] for each pixel of the plurality of pixels:

[0029] - a step of determining the coordinates of the pixel in the pixelated image, said pixel coordinates;

[0030] - a step of determining a pixel depth;

[0031] - a step of determining a level of danger of the pixel;

[0032] - a step of generating a 3D point cloud, each point of the cloud of points being associated with a pixel of the pixelated image, each point of the point cloud being positioned in the point cloud by point coordinates, said point coordinates being determined from the coordinates of the pixel associated with said point and from the depth of said pixel, said point of the point cloud being affected by the danger level of said pixel;

[0033] - a step of determining a direction of movement for the person to visual impairment based on the coordinates of all or part of the points in the 3D point cloud and the danger levels assigned to said points;

[0034] - an audio guidance step, using the audio module, for the person with a disability visual according to the determined direction of movement.

[0035] Another object of the invention relates to a computer program comprising software instructions which, when executed by computer, implement a method for assisting the movement of a person with a visual impairment.

[0036] Another object of the invention relates to a method for acquiring spatial data to assist the movement of a visually impaired person, said method comprising:

[0037] - a step of acquiring spatial data by a camera in which the A person with a visual impairment is able to move around;

[0038] - a step of generating a pixelated image from said spatial data, said pixelated image comprising a plurality of pixels;

[0039] said spatial data being intended to be able to implement, for each pixel of the plurality of pixels;

[0040] - a step of determining the coordinates of the pixel in the acquired image;

[0041] - a step of determining a pixel depth;

[0042] - a step of determining a level of danger for said person;

[0043] - a step of generating a 3D point cloud, each point of the cloud of points being associated with a pixel of the pixelated image, each point of the point cloud being positioned in the point cloud by point coordinates, said point coordinates being determined from the coordinates of the pixel associated with said point and from the depth of said pixel, said point of the point cloud being affected by the danger level of said pixel;

[0044] - a step of determining a direction of movement for the person to visual impairment based on the coordinates of all or part of the points in the 3D point cloud and the danger levels assigned to said points;

[0045] - said acquisition method comprising, furthermore, an audio guidance step of the person with visual impairment according to the determined direction of movement?

[0046] Another object of the invention relates to a portable device intended to be positioned on the head of a visually impaired person, said portable device comprising:

[0047] - a camera adapted for acquiring spatial data of an environment in which the visually impaired person is able to move around and to generate a pixelated image from said spatial data, said pixelated image comprising a plurality of pixels;

[0048] - an audio module for audio guidance of a visually impaired person according to a determined direction of movement based on spatial data;

[0049] - a processor configured to implement an acquisition step, by the camera, spatial data, said spatial data being intended to be implemented by said processor and / or by another processor, for each pixel of the plurality of pixels:

[0050] - a step of determining the coordinates of the pixel;

[0051] - a step of determining a pixel depth;

[0052] - a step of determining a level of danger of the pixel;

[0053] - a step of generating a 3D point cloud, each point of the cloud of points being associated with a pixel of the pixelated image, each point of the point cloud being positioned in the point cloud by point coordinates, said point coordinates being determined from the coordinates of the pixel associated with said point and from the depth of said pixel, said point of the point cloud being affected by the danger level of said pixel;

[0054] - a step of determining a direction of movement for the person to visual impairment based on the coordinates of all or part of the points in the 3D point cloud and the danger levels assigned to said points;

[0055] - said processor and / or said other processor being configured to implement an audio guidance (108) of the visually impaired person according to the determined direction (Dir) of movement.

[0056] Another object of the invention relates to a computer program comprising software instructions which, when executed by computer, implement a method for acquiring an image to assist the movement of a person with a visual impairment.

[0057] Another object of the invention relates to a treatment method for assisting the movement of a visually impaired person, said method comprising:

[0058] - a step of generating a 3D point cloud, each point of the cloud points being associated with a pixel of a pixelated image obtained from data spatial via a camera, each point of the point cloud being positioned in the point cloud by point coordinates, said point coordinates being determined from the coordinates of the pixel associated with said point and from the depth of said pixel, said point of the point cloud being assigned a danger level of said pixel, the of the pixel, the depth of the pixel, the danger level of said pixel the coordinates being obtained from said spatial data;

[0059] - a step of determining a direction of movement for the person to visual impairment based on the coordinates of all or part of the points in the 3D point cloud and the danger levels assigned to said points, said direction determined allowing audio guidance of the person.

[0060] Another object of the invention relates to a data processing device for assisting the movement of a visually impaired person, said processing device comprising a processor configured to implement:

[0061] - a step of generating a 3D point cloud, each point of the cloud points being associated with a pixel of a pixelated image obtained from spatial data via a camera, each point of the point cloud being positioned in the point cloud by point coordinates, said point coordinates being determined from the coordinates of the pixel associated with said point and from the depth of said pixel, said point of the point cloud being assigned a danger level of said pixel, the pixel coordinates, the pixel depth, the danger level of said pixel being obtained from said spatial data;

[0062] - a step of determining a direction of movement for the person to visual impairment based on the coordinates of all or part of the points in the 3D point cloud and the danger levels assigned to said points, said direction determined allowing audio guidance of the person.

[0063] Another object of the invention relates to a computer program comprising software instructions which, when executed by computer, implement a data processing method for assisting the movement of a visually impaired person.

[0064] Another object of the invention relates to a portable device intended to be positioned on the head of a visually impaired person, said portable device comprising:

[0065] - a main body;

[0066] - a first branch;

[0067] - a second branch, said first branch and said second branch extending said main body;

[0068] - said portable device comprising:

[0069] - a camera on the first branch, said camera being a pinhole camera;

[0070] - a first speaker on the first branch and a second speaker on the second branch, said first speaker and said second speaker being capable of generating spatial 3D sound for audio guidance of the visually impaired person along a predetermined direction of movement;

[0071] - a human-machine interface dial for controlling the portable device by the person with a visual impairment or a voice impairment.

[0072] In one embodiment, the first branch comprises a head, a body and an intermediate part between said head and said body and the second branch comprises a head, a body and an intermediate part between said head and said body.

[0073] In one embodiment, the head of the first branch includes the camera and the human-machine interface wheel.

[0074] In one embodiment, the first loudspeaker and the second loudspeaker are respectively arranged on the first branch and on the second branch, opposite each other.

[0075] In one embodiment, the device is capable of implementing part of a method for assisting the movement of a person with a visual impairment.

[0076] In one embodiment, the device is part of a system for assisting the movement of a visually impaired person. Description of figures

[0077] Other features and advantages of the invention will become apparent upon reading the detailed description that follows, for an understanding of which reference should be made to the accompanying drawings in which:

[0078] Figure 1 illustrates in general terms a method for assisting movement of a person with a visual impairment according to the invention;

[0079] Figure 2 illustrates a geometric visualization of spatial data environment in which the visually impaired person is able to move around;

[0080] Figure 3 illustrates a pixelated Zpex image obtained from the spatial data of [Fig.2];

[0081] Figure 4 illustrates in more detail a step in determining a level danger of the process of the [Fig.l];

[0082] Figure 5 illustrates in more detail a step in determining a direction of movement of the process of the [Fig.l];

[0083] Figure 6 illustrates in more detail an audio guidance step of the person with visual impairment of the process of the [Fig.l];

[0084] [Fig.7] illustrates a system for assisting the movement of a person with a visual impairment for the implementation of the method of [Fig.1];

[0085] Fig. 8 illustrates a portable device of the system of Fig. 7 according to a first view;

[0086] Fig.9 illustrates a portable device of the system of Fig.7 according to a second view;

[0087] Fig. 10 illustrates an internal operation of the portable device of figures 8 and 9;

[0088] Fig. 11 illustrates the internal workings of a processing device in the system of Fig. 7.

[0089] Description of embodiments

[0090] Fig. 1 illustrates the overall operation of a method for assisting the movement of a person with a visual impairment.

[0091] This method includes an acquisition step 101, by a camera, of spatial data D of an environment in which the visually impaired person is able to move around and a generation step 102 of a pixelated image from the spatial data D.

[0092] The camera here is a pinhole camera. It is a simple device that offers many advantages in terms of manufacturing cost.

[0093] The pinhole camera is thus capable of capturing a plurality of M3V points constituting the spatial data D. By the process of the invention, it is possible to obtain a point j. belonging to the pixelated image Ipex from one of the M3D points of the spatial data D via a linear transformation (called "mapping" in English).

[0094] Such a linear transformation is illustrated in particular from figures 2 and 3.

[0095] Figure 2 shows an orthogonal coordinate system with axes X, F, Z centered at O, O being the optical center of the camera.

[0096] In such a Euclidean space, each point Mx, y can be defined by a triplet of coordinates such that: M3D = XI, where X is the coordinate of M3D along the x-axis, Y is the coordinate of M3D along the x-axis and Z is the coordinate of M3D along the Z-axis

[0097] By fixing a value for Z, for example the value 1, it is possible to determine a plane P passing through the point y. This plane P corresponds to the image that the camera can acquire.

[0098]

[0099] In this plane P visible in figure 2, the point M3D has the following coordinates: m3D= m. Y 1. It is also possible to determine homogeneous coordinates for this M3D point by adding an additional parameter to the coordinate triplet, for example a constant, such as 1. The homogeneous coordinates of such a point are then written: M3D

[0100]

[0101]

[0102]

[0103]

[0104] Z 1. As illustrated in Figure 3, the image generated in step 102 is a pixelated Ipex image obtained from the spatial data D, including those present on the plane P. The pixelated image Ippx is divided into a plurality of pixels Pix, i ranging from 1 to N. Each pixel Pix;a here has the shape of a square with a center M r In step 103, the coordinates of the pixel Pix are determined; in the pixelated image Ipex, these are called the coordinates of the pixel (M, v, 1); The coordinates of point m'u v are thus determined in a coordinate system with axes U, V and center O'. The center O' is represented here at the center of the pixelated image Ipex- The linear transformation between point M3D and point Mu r is performed using the following equations:

[0105] (Equation 1)

[0106] With [Al = ïfx 0 Cx 01 fy Cy 0 01

[0107] Where:

[0108] fx denotes a focal length of the camera in number of pixels along the x direction;

[0109] fy denotes the focal length of the camera in number of pixels along the y direction;

[0110] Cx and Cy denote coordinates of an intersection C of an optical axis AO with The plane P passes through point M3D. The optical axis AO extends the z-axis on [Fig.2]. The intersection C between the optical axis AO and the plane P is positioned at the center of this plane.

[0111] is a scaling factor that links the relative consistency of a distribution along the Z-axis for a given image.

[0112] The matrix [K] is called the intrinsic matrix of the pinhole camera. This matrix is ​​calculated from the characteristics of the camera.

[0113] Thus, this matrix [K] allows the linear transformation (mapping) of a point in the 3D space of the real world defined by (X, Y, Z, 1)T on the image plane from an image sensor of the camera (M, V, 1)T.

[0114] From equation 1 we can define the following 3 relations (Equations 2):

[0115] At = X*fx + Z*Cx;

[0116] Tov = X*fy + Z*Cy;

[0117] 2 = Z.

[0118] After simplification we have the following system of equations (Equations 3):

[0119]

[0120] uX^+Cx', v=Y^+Cy

[0121] Equations 3 thus allow the passage from a 3D point (real world) to 2D (image).

[0122] The process in Figure 1 also includes a step 104 for determining a depth Z for each pixel Pix;. It is necessary to reconstruct this value. Indeed, the pixelated image Ipix is ​​a 2D image. However, the objects associated with the different pixels Pix1 represent different depths in the environment. For example, in Figure 3, the first pixel Pix1 represents the first corner of a bank 300. The second pixel Pix2 represents a second corner of the bank 300 diametrically opposite the first pixel Pix2. The second corner is in the depth of the image. It is therefore necessary to retrieve this information. Each depth Z of pixel Pix; is obtained using a first artificial intelligence module capable of analyzing the pixelated image Ipix-. This first artificial intelligence module is previously trained using a proprietary data sample.The artificial intelligence module is thus adapted to provide a depth map.

[0123] The process of [Fig.1] also includes a step of determining 105 a level of danger for said person, called danger level L; of the pixel Pix;.

[0124] As illustrated in [Fig.4], this determination step 105 comprises:

[0125] - a determination step 1051 of a Si semantics of the pixel Pix; ;

[0126] - a determination step 1052 of the hazard level L; associated with said pixel Pix;

[0127] The danger level L; of the pixel Pix; is obtained from a semantics Si of the pixel Pixi.

[0128] By semantics S;> we mean the type of object covered by the pixel Pix;, such as a bench, a road, a bus shelter or others.

[0129]

[0130]

[0131]

[0132]

[0133]

[0134]

[0135]

[0136]

[0137]

[0138]

[0139]

[0140] It is thus possible to assign a danger level L; to the object of the pixel Pix;. A bench A bench or bus shelter can have the same danger level L;. A road's danger level will be higher than the danger level of the bench or bus shelter due to an increased risk of collision with a moving vehicle. The danger levels for each pixel Pix; are assigned via a second artificial intelligence model of Semantic segmentation. This second model is capable of analyzing the pixelated image Ipix to assign each pixel Pixi a danger level L;. This second artificial intelligence module is pre-trained using a proprietary data sample. Alternatively, the danger level L; is also defined according to the context detected in parallel. The process in Figure 1 also includes a step 106 of generating a 3D point cloud. Each point Pt; of the 3D point cloud is associated with a Pixi pixel of the pixelated image Ipix-. In addition, each point Pf of the point cloud is positioned in said cloud by coordinates of the point Xf, Z, 1. These coordinates of the point X', Yr, Z, 1 are determined from the coordinates u, v, 1 of the Pixi pixel associated with said point Pt; and from the depth Z of the Pix pixel; according to equation 4 below: (Equation 4) The depth Z is measured and the coordinates X' and Y' are calculated. The matrix [ / C1] is the inverse matrix of [AT]. This matrix [JC1] is constant. dimension 4x3. It is calculated using linear algebra. When 2 = Z, equation 4 becomes: w (Equation 5) Z v .1. Equation 5 allows us to find the metric coordinates of the point Pt; of the point cloud if the depth Z (in metric system) of a point is known. Knowing the matrix ] and the depth Z, it is possible to calculate the coordinates X', F' of the point Pf from “, r. By applying equation 5 to each point Mu v, it is possible to obtain the different points Pt; of the 3D point cloud. It is possible to assign to each point Pt a danger level L; of the associated pixel Pix;

[0141] The method of [Fig.1] also includes a step of determining 107 a direction Dir of movement for the visually impaired person from the coordinates of all or part of the points Pf of the 3D point cloud and the levels L of danger assigned to said points.

[0142] As can be seen in [Fig. 5], determination step 107 comprises:

[0143] - a step of eliminating 1071 points from the 3D point cloud whose height is greater than a certain value. This forms a corrected 3D point cloud;

[0144] - a 1072 projection step onto the ground of the points of the corrected 3D point cloud;

[0145] - a step of obtaining 1073 of the direction Dir of movement from the cloud of 3D points corrected.

[0146] In one embodiment, the elimination step 1071 takes into account an inclination and / or an irregularity of the ground.

[0147] The process of [Fig.1] finally includes an audio guidance step 108 of the visually impaired person according to the direction Dir of movement.

[0148] As can be seen in [Fig.6], this audio guidance step 108 includes a spatial 3D sound generation step 1081;

[0149] This spatial 3D sound is a broadband sound whose source is virtually positioned spatially. Thus, the principle is to follow a sound based on the spatial perception of this sound source in 3D space and to walk in the direction of the sound.

[0150] Audio guidance step 108 also includes:

[0151] - a step 1082 of correction of the generated 3D spatial sound, said correction being made from a 3x3 rotation matrix R and a 1x3 translation matrix T;

[0152] - a step 1083 of guiding the person independently of the movements of rotation and translation of said camera.

[0153] The rotation matrix and the translation matrix can be generated using two successive images and their associated depth maps. First, the characteristic point pairs PI and P2 are determined using a known algorithm such as SIFT (Scale-Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF) from the two successive images. Then, the metric positions of points PI and P2 are extracted from the depth maps. By calculating the difference between the two positions of the same characteristic point PI (Xi, Yi, Zi) and P2 (X2, Y2, Z2) on the three axes, the two rotation and translation vectors are determined. To minimize errors, this operation should be performed on several point pairs.

[0154] Considering two series of 3D unit vectors as follows:

[0155] ©!= ..., P)„}

[0156] 02= [P21,P22,P23,P2„]

[0157] With Pry corresponding to a unit vector in Euclidean space calculated from a metric point characterized by the coordinates (XY, Z) whose origin is the optical axis AO. To calculate a unit vector, the 3 coordinates of a metric point are divided by the square root of the sum of the squares of its 3 coordinates.

[0158] P a and P^ are point pairs corresponding to a characteristic point of the two successive images.

[0159] To obtain the angles &x, and Sz between two unit vectors, we calculate the dot product of the projections of the two vectors onto the orthogonal plane of the desired axis.

[0160] _ 1 ^COS4 (P*P2Î\Y + Pïi'Z *P2i'Z)

[0161] 1 *Pn',x + Pwz )

[0162] Qz = l E",C0S4 (*P^ + Pwy '^Y)

[0163] To calculate the translation vector, we take the differences in coordinates for each pair of points and average them over the number of points in the series. [°1641 t= i EX-a)

[0165] Alternatively, the rotation and translation matrices can be generated using a 9-axis inertial system (3 axes for the magnetometer, 3 for the gyroscope, and 3 for the accelerometer). The rotation vector is calculated using the magnetometers and the gyroscope, and the translation vector is calculated by double integration of the acceleration delivered by the accelerometer. Since the acceleration signal is very noisy, algorithms such as Extended Kalman filtering should be used to minimize the influence of noise in the calculation of the translation matrix.

[0166] In one embodiment, the person is guided using useful information, such as a scene description or an indication of useful objects. Thus, instead of generating a sound source, a word (e.g., "bus") or a phrase (e.g., "bus stop") is generated along with its distance ("bus at 3.5 meters"), always spatially positioned in the direction of the object. During the person's movement, the information is generated in real time using 3D Text-to-Speech (TTS) technology or 3D sound. This technology involves placing the source of the speech within the space of a word or short text, such that the user perceives the voice as coming from a point in space. It is this sound source that the person must follow.

[0167] In one embodiment, the environment around the user is described at the user's request. This environment includes obstacles and useful information. 3D TTS technology is used with metric distances.

[0168] In one embodiment, the guidance is tactile guidance.

[0169] In one embodiment, the guidance is haptic guidance.

[0170] Figure 7 illustrates a system 2 for assisting the movement of the person. This System 2 includes:

[0171] - a portable device 20 intended to be positioned on a person's head;

[0172] - a data processing device 21.

[0173] Figures 8 and 9 present the portable device 20 in more detail.

[0174] This portable device 20 includes:

[0175] - a main body 30A;

[0176] - a first branch 30B;

[0177] - a second branch 30C;

[0178] - a 201 camera;

[0179] - a first loudspeaker (not visible in Figures 8 and 9) and a second loud- speaker 202;

[0180] - a human-machine interface wheel 203.

[0181] The main body 30A has an arc-shaped form. It is designed to allow optimal adjustment of the portable device 20 behind the user's head. This main body 20A is extended on one side by the first arm 30B and on the other side by the second arm 30C.

[0182] The first branch 30B comprises:

[0183] - a 301 head;

[0184] - a 302 body;

[0185] - an intermediate part 303.

[0186] The head 301 is adapted to be positioned near the user's temple. It comprises a rear face 3011 arranged opposite the body 302, a front face 3012 arranged on the head 301 opposite the rear face 3011, and a lower face 3013 arranged between the front face 3012 and the rear face 3011. The lower face 3013 includes a cavity extending from the front face 3012.

[0187] The camera 201 is located on the front face 3012. As already mentioned, this camera 201 is of the pinhole type.

[0188] The human-machine interface wheel 203 is positioned in the cavity of the lower face 3013. This interface wheel 203 allows the portable device 20 to be controlled by the user or by voice.

[0189] The second arm 30C also includes a head, a body, and an intermediate portion. The intermediate portion of the arm 30C includes an inner face 301 designed to make contact with the user's temple. This inner face 301 includes the second loudspeaker 202. This second loudspeaker 202 is adapted to reproduce 3D spatial sound for the user's guidance.

[0190] It should be noted that the first branch 30B also includes the first loudspeaker (not visible in Figures 7 and 8). The first loudspeaker and the second loudspeaker 202 are thus capable of generating spatial or binaural stereophonic sound.

[0191] In one embodiment, the wearable device 20 is in the form of glasses comprising a body and two temples extending from said body. In this configuration, each temple includes a speaker and the body includes the camera.

[0192] As illustrated in [Fig. 7], the treatment device 21 is in the form of a rectangular case. Such a case can, for example, be placed in a user's pocket or in a bag carried by the user.

[0193] Fig. 10 illustrates the internal workings of the portable device 20.

[0194] Fig. 11 illustrates an internal operation of the processing device 21.

[0195] The proposed invention thus offers the following advantages:

[0196] - use of a simple image sensor for calculating the mapping of depth and semantic segmentation compared to a solution using a stereoscopic system or a ToF (Time of Flight) system. A camera using a monocular sensor offers several advantages, such as:

[0197] - minimal bulk;

[0198] - a depth of mapping with constant accuracy over distance;

[0199] - an ease of industrialization because there is no need to align the sensors;

[0200] - there is no interpolation to adjust the size of the depth map to the RGB image or vice versa;

[0201] - reduced energy consumption.

[0202] The invention also provides the following advantages:

[0203] - a construction of the environment in metric 3D point cloud from from a single image with hierarchical hazard attributes (for a pedestrian) for each point in the cloud. The primary benefit lies in converting the point cloud into a ground projection of the points, which are presented as a 2D matrix. Once this operation is performed, it is easy to carry out obstacle avoidance calculations and direction angle calculations;

[0204] - a spatial description of the environment with 3D sound composed of words or short phrases. The advantage of this solution is to generate, in a very short time, a sensory map of the objects surrounding the user while minimizing their mental load.

[0205] - a correction of the movement and orientation of the head in order to generate a sound Stationary spatial motion is achieved using two matrices (rotation and translation). When using 3D sound, it is important to position the sound source at a fixed point. when the camera is in motion characterized by two rotation vectors R and translation vectors T.

Claims

Demands

1. A method for assisting the movement of a visually impaired person, said method comprising: - a step of acquiring, by a camera (201), spatial data ( / )) of an environment in which the visually impaired person is able to move; - a step of generating (102) a pixelated image (Ipix) from said spatial data ( / )), said pixelated image (Ipix) comprising a plurality of pixels (Pix;); - for each pixel (Pix;) of the plurality of pixels: - a step of determining the coordinates (M, V, L) of the pixel (¾); - a step of determining the depth (Z) of the pixel (¾); - a step of determining the danger level (Li) of the pixel (Pixi); - a generation step (106) of a 3D point cloud, each point (Ph) of the point cloud being associated with a pixel (Pix;) of the pixelated image (Ipjx), each point of the point cloud being positioned in the point cloud by coordinates (V', Y', Z, 1) of the point (Pf), said coordinates of the point (X', y', Z, 1) being determined from the coordinates (a, V, 1) of the pixel (Pix;) associated with said point (Pt;) and from the depth (Z) of said pixel, said point (Pf) of the point cloud being assigned the danger level (Li) of said pixel (Pix;); - a step of determining (107) a direction (Dir) of movement for the visually impaired person from the coordinates of all or part of the points (Pt;) of the 3D point cloud and the danger levels (Li) assigned to said points (Pt;); - a step of audio guidance (108) of the visually impaired person according to the determined direction (Dir) of movement.;

2. A method according to claim 1, wherein the coordinates (X\ F', Z, 1) of point (Ph) are determined according to the following equation: with 2 a scaling factor related to the type of camera and [Æ] an intrinsic camera matrix, such that: [K] = ïfx 0 Cx 0, with fx a focal length of the camera in

3.

4.

5.

6.

7. 0 fy Cy 0 0 0 1 01 number of pixels along a first direction x, fy a focal length of the camera in number of pixels along a second direction Cx a first pixel coordinate of the intersection of an optical axis of the camera with an image plane, Cy a second pixel coordinate of the intersection of the optical axis of the camera with said image plane. A method according to any one of claims 1 or 2, wherein the step of determining the hazard level (L) of the pixel includes a determination step (1051) for said pixel (Pixi) of a semantics (Si)of the pixel, said danger level (L;)of the pixel being obtained (1052) from said semantics (Si)of the pixel (¾). A method according to any one of claims 1 to 3, wherein, each point of the 3D point cloud having a height relative to a ground, the step of determining a direction (Dir) of displacement includes a step of eliminating (1071) points of the 3D point cloud whose height is greater than a certain value, to form a corrected 3D point cloud. Method according to claim 4, wherein said elimination step (1071) takes into account an inclination and / or an irregularity of the ground. A method according to any one of claims 4 or 5, wherein the step of determining a direction of movement includes a step of projecting (1072) onto the ground the points of the corrected 3D point cloud in order to determine the direction (Dir) of movement (1073). A method according to any one of claims 1 to 6, wherein the audio guidance step (108) of the visually impaired person according to the determined direction of movement includes a generation step (1081) of a spatial 3D sound.

8. A method according to claim 7, wherein the audio guidance step (108) includes a step (1082) for correcting the generated spatial 3D sound, said correction being carried out from a 3x3 rotation matrix R and a 1x3 translation matrix T of the camera so as to guide (1083) the person independently of the rotation and translation movements of said camera.

9. System for assisting the movement of a visually impaired person, said system comprising a portable device (20) intended to be positioned on the head of said person and a processing unit (21), said portable device (20) comprising a camera (201) adapted to acquire spatial data (®) of an environment in which the visually impaired person is able to move, said portable device (20) comprising an audio module (202) for audio guidance of the visually impaired person along a direction (Dir) of movement determined from the spatial data (^), said system (2) comprising at least one processor (10, 11) configured to implement: - a stage of acquisition (101) by the camera of the spatial data ( ^);- a generation step (102) of a pixelated image (Ipix) from the spatial data (^), said pixelated image (Ipix) comprising a plurality of pixels (Pix;); - for each pixel of the plurality of pixels: - a determination step (103) of the pixel's coordinates in the pixelated image, called the coordinates (H, R 1) of the pixel (Pix;); - a determination step (104) of a depth (Z) of the pixel (¾); - a determination step (105) of a danger level (L;) of the pixel (¾); - a generation step (106) of a 3D point cloud, each point (Ph) of the point cloud being associated with a pixel (Pix;) of the pixelated image (Ipix), each point of the point cloud being positioned in the point cloud by coordinates of the point (X\ Z, 1), said coordinates of the point (X\ Y'. Z, 1) being determined from the coordinates (M, V, 1) of the pixel associated with said point and from the depth (Z) of said pixel, said point (Pt,) of the; point cloud being affected by the danger level (L;)of said pixel (Pix; ); - a step of determining (107) a direction (Dir) of movement for the visually impaired person from the coordinates of all or part of the points (Pf) of the 3D point cloud and the danger levels (L;)assigned to said points; - a step of audio guidance (108), by the audio module (202), of the visually impaired person according to the determined direction (Dir) of movement.

10. A computer program comprising software instructions which, when executed by computer, implement a method for assisting the movement of a visually impaired person according to any one of claims 1 to 8.

11. A method for acquiring spatial data (D) to assist the movement of a visually impaired person, said method comprising: - a step of acquiring (101), by a camera (201), spatial data (D) in which the visually impaired person is able to move; - a step of generating (102) a pixelated image (Ipix) from said spatial data (Ipix), said pixelated image (Ipix) comprising a plurality of pixels (Pix); - said spatial data (D) being intended to enable the implementation, for each pixel (Pix) of the plurality of pixels: - a step of determining the coordinates (H, V, L) of the pixel (Pix) in the acquired image; - a step of determining the depth (Z) of the pixel; - a step of determining the level of danger (L) for said person;- a generation step (106) of a 3D point cloud, each point (Ph) of the point cloud being associated with a pixel (Pix;) of the pixelated image ( / pixY each point of the point cloud being positioned in the point cloud by coordinates of the point ( X\ Y', Z, 1), said coordinates of the point (X\ Y', Z, 1) being determined from the coordinates (M, V, 1) of the pixel associated with said point and from the depth of said pixel (Z), said point (Pt,) of the; point cloud being assigned the danger level (L;)of said pixel (Pix; ); - a step of determining (107) a direction (Dir) of movement for the visually impaired person from the coordinates of all or part of the points (Pf) of the 3D point cloud and the danger levels (L;)assigned to said points (Pf); - said acquisition method comprising, furthermore, an audio guidance step (108) of the visually impaired person according to the determined direction (Dir) of movement.

12. A wearable device (20) intended to be positioned on the head of a visually impaired person, said wearable device (20) comprising: - a camera (201) adapted to acquire spatial data (D) of an environment in which the visually impaired person is able to move and to generate a pixelated image (Zpix) from said spatial data (Zpix), said pixelated image (Zpix) comprising a plurality of pixels (Pixix); - an audio module (202) for audio guidance of the visually impaired person along a determined direction (Dir) of movement from the spatial data (D); - a processor (10) configured to implement an acquisition step (101), by the camera (201), of the spatial data (D), said spatial data (D) being intended to be able to be implemented by said processor (10) and / or by another processor (11), for each pixel (Pixix);) of the plurality of pixels: - a step of determining (103) coordinates (w, V, 1) of the pixel (¾); - a step of determining (104) a depth (Z) of the pixel (¾); - a step of determining (105) a danger level (L;) of the pixel; - a step of generating (106) a 3D point cloud, each point (Ph) of the point cloud being associated with a pixel (Pix;) of the pixelated image (Zpixh) each point of the point cloud being positioned in the point cloud by coordinates (X', Y', Z, 1) of the point (Ph), said coordinates (X', Y', Z, 1) of the point (Ph) being determined from the coordinates (M, V, 1) of the pixel; (Pixi) associated with said point (Pt;) and from the depth (Z) of said pixel, said point (Pti) of the point cloud being affected by the danger level (L;) of said pixel (Pix;); - a step of determining (107) a direction (Dir) of movement for the visually impaired person from the coordinates of all or part of the points (Pf) of the 3D point cloud and the danger levels (L;) assigned to said points (Pf); - said processor (10) and / or said other processor (11) being configured to implement audio guidance (108) of the visually impaired person in the determined direction (Dir) of movement.

13. A computer program comprising software instructions which, when executed by computer, implement a method for acquiring an image for assisting the movement of a visually impaired person according to claim 11.

14. A processing method for assisting the movement of a visually impaired person, said method comprising: - a step of generating a 3D point cloud (106), each point (Ph) of the point cloud being associated with a pixel (Pix) of a pixelated image (Jpix) obtained from spatial data (D) via a camera (201), each point of the point cloud being positioned in the point cloud by point coordinates (X, Y', Z, 1), said point coordinates (X', Y', Z, 1) being determined from the coordinates (M, V, 1) of the pixel associated with said point and from the depth of said pixel (Z), said point (Pt) of the point cloud being assigned a danger level (L) of said pixel (Pix), the coordinates (M, V, 1) of the pixel, the depth (Z) of the pixel, the danger level (L) of said pixel (Pix;) being obtained from said spatial data (D); - a step of determining (107) a direction (Dir) of movement for the visually impaired person from the coordinates of all or part of the points (Pt;) of the 3D point cloud and the danger levels (L;) assigned to said points (Pf), said direction (Dir) determined allowing audio guidance (108) of the person.

15. A data processing device for assisting the movement of a visually impaired person, said processing device comprising a processor (11) configured to implement: - a generation step (106) of a 3D point cloud, each point (Ph) of the point cloud being associated with a pixel (Pix;) of a pixelated image (Jpjx) obtained from spatial data (D) via a camera (201), each point of the point cloud being positioned in the point cloud by point coordinates (Y', Z, 1), said point coordinates (Y', Z, 1) being determined from the coordinates (M, V, 1) of the pixel associated with said point and from the depth of said pixel (Z), said point (Pt,) of the point cloud being assigned a danger level (L;) of said pixel (Pix; ), the coordinates (M, V, 1) of the pixel, the depth (Z) of the pixel, the danger level (L;) of said pixel (Pix;) being obtained from said spatial data (D);- a step of determining (107) a direction (Dir) of movement for the visually impaired person from the coordinates of all or part of the points (Pt;) of the 3D point cloud and the danger levels (L;) assigned to said points (Pf), said direction (Dir) determined allowing audio guidance (108) of the person.;

16. A computer program comprising software instructions which, when executed by a computer, implement a data processing method for assisting the movement of a visually impaired person according to claim 1 A

17. 1H-. A portable device intended to be positioned on the head of a visually impaired person, said portable device being capable of implementing part of a method for assisting the movement of a visually impaired person according to any one of claims 1 to 8, said portable device comprising: - a main body (30A); - a first arm (30B); - a second branch (30C), said first branch (30B) and said second branch (30C) extending said main body (30A); - said portable device (20) comprising: - a camera (201) on the first branch (30B), said camera (201) being a pinhole camera; - a first loudspeaker on the first branch (30B) and a second loudspeaker (202) on the second branch (30C), said first loudspeaker and said second loudspeaker being capable of generating spatial 3D sound for audio guidance of the visually impaired person along a predetermined direction (Dir) of movement; - a human-machine interface wheel (203) for controlling the portable device by the visually impaired person or by voice.

18. Portable device according to claim 17, wherein the first arm (30B) comprises a head (301), a body (302) and an intermediate part (303) between said head (301) and said body (302) and the second arm (30C) comprises a head (401), a body (402) and an intermediate part (403) between said head (401) and said body (402).

19. A portable device according to claim 18, wherein the head (301) of the first arm (30B) comprises the camera (201) and the human-machine interface wheel (203).

20. Portable device according to any one of claims 18 or 19, wherein the first loudspeaker and the second loudspeaker (202) are respectively arranged on the first arm (30B) and on the second arm (30C), opposite each other.

21. A portable device according to any one of claims 17 to 20, wherein said device is part of a system for assisting the movement of a visually impaired person according to claim 9.

Images