Apparatus and method for determining the position of an object in front of a display screen

Abstract

The invention relates to an apparatus comprising: A screen (1) having a substrate (2) which supports: photoemitters (3); a photodetector (4); a laser beam emission source (5); and a beam scanning system (6) coupled to said source and capable of being controlled to scan the laser beam in a scene located in front of the substrate; an electronic system configured to: control the beam scanning system by means of a scanning angle command; acquire from the photodetector (4) a photogenerated signal following the detection of the laser beam backscattered by an object in the scene illuminated by the laser beam in accordance with the scanning angle command; determine a position of the object in the scene, from the photogenerated signal and the scanning angle command. Figure for the abstract: Figure 1A

Description

Title of the invention: Apparatus and method for determining the position of an object in front of a display screen. Technical field

[0001] The field of the invention is that of image display screens. The invention relates more particularly to an interactive screen combining a light emission function and a 3D optical capture function enabling depth mapping of the scene in front of the screen. Previous technique

[0002] A 3D touchless technology called "Infra-Red Intelligent Surface" (IRIS) uses a sensor composed of interlaced infrared (IR) emitters and receivers distributed over a surface. This IRIS technology is described, for example, in Santoul, E., Hemery, E. and Tuckey, J. (2023), Infra-Red Intelligent Surface for Near-Field Touchless Displays. Information Display, 39: 18-21. https: / / doi.org / 10.1002 / msid.1408

[0003] IRIS technology is based on measuring intensity and knowing the geometry of the sensor to deduce a 3D point cloud of the scene above the sensor. The sensor operates in the near field, from the surface up to a few tens of centimeters above in a continuous detection field.

[0004] The sensor operates more precisely by first emitting a controlled pattern of IR light from the emitter array. The light is then reflected by objects in front of the sensor, and the reflected light is detected by the receiver array. The detected light is processed by a signal processing circuit to eliminate noise and ambient light sources. The resulting 2D image contains information about the type and the x, y, z positions of the objects in front of the sensor. This is used to create a 3D point cloud of the scene in front of the sensor. This data allows the system to infer a user's intent and interact accordingly.

[0005] The maximum detection range of the sensor is on the order of magnitude of the size of the surface, which means that a sensor the size of a multifunction mobile phone could detect objects up to a few tens of centimeters away, while a sensor the size of a 55-inch screen could detect objects up to a few meters away.

[0006] The industrialized IRIS sensor will consist of a stack containing several layers, including a substrate, photoelements (IR emitters and receivers or RGB pixels), and a collimation layer. The collimation layer allows the infrared light to be manipulated so that it converges towards the receivers. However, this collimation layer introduces additional technological complexity. Description of the invention

[0007] The invention aims to eliminate the collimation layer while improving 3D capture performance in the far field, typically greater than 30 cm.

[0008] To this end, the invention proposes a device comprising: - a screen having a substrate that supports: • a set of photoemitters; • a photodetector; • a laser beam emission source; and • a beam scanning system coupled to said source and capable of being controlled to scan the laser beam emitted by said source in a scene located in front of the substrate; - an electronic system configured to: • control the beam scanning system using a scanning angle command; • acquire from the photodetector a photo-generated signal following the detection of the laser beam backscattered by an object in the scene illuminated by the laser beam in accordance with the scanning angle instruction; • determine a position of the object in the scene, from the photo-generated signal and the scanning angle command.

[0009] Some preferred but not limiting aspects of this device are as follows: - the electronic system is further configured to modify a control command for the photoemitters in response to the determination of the object's position; - the laser beam emission source and the beam scanning system are arranged on the substrate; - the screen further includes a cover hood covering the entire photoemitter assembly and the photodetector and the laser source for emitting the laser beam and the beam scanning system are arranged on the cover hood; - the cover hood includes a waveguide configured to route the laser beam from the emission source of a laser beam to the beam scanning system; - the beam scanning system includes an optical phase-controlled array and a deflector; - the laser beam emission source is a tunable monochromatic source and the deflector is a diffraction grating; - the beam scanning system is an opto-mechanical system; - the beam scanning system includes a programmable metasurface; - the photodetector is incorporated into an elementary chip which includes at least one of the photoemitters and an electronic circuit; - to determine the position of the object in the scene, the electronic system is configured to determine a time of flight, directly or indirectly, of the laser beam from its emission until its detection by the photodetector; - to determine the position of the object in the scene, the electronic system is configured to perform a differential measurement of arrival time. Brief description of the drawings

[0010] Other aspects, objects, advantages and features of the invention will become more apparent upon reading the following detailed description of preferred embodiments thereof, given by way of non-limiting example, and made with reference to the accompanying drawings in which:

[0011] - [Fig. 1A] and [Fig. 1B] are diagrams, respectively in side view and in view from above, of a screen of a device according to a first possible embodiment of the invention;

[0012] - [Fig. 2] is a side view diagram of a screen of a device according to a second possible embodiment of the invention;

[0013] - [Fig. 3] is a top view diagram of a screen of a device according to a third possible embodiment of the invention;

[0014] - [Fig. 4] is a diagram illustrating the principle of a distance measurement that can be implemented by a device conforming to the invention;

[0015] - [Fig. 5] is a diagram illustrating the advantage of providing several emission sources laser in a device according to the invention;

[0016] - [Fig. 6] is a schematic of a scanned emission source that can be used within the scope of the invention.

[0017] DETAILED DESCRIPTION OF PARTICULAR EMBODIMENTS

[0018] The invention relates to a device comprising a display screen 1, 10, 100, for example, for a computer, multifunction mobile device, television, or tablet. With reference to Figures IA, IB, 2, and 3, the screen comprises a substrate 2, for example, made of glass, which supports a set of photoemitters 3 and one or more photodetectors 4. The screen also comprises a cover 7, for example, made of glass, which covers the photoemitters 3 and the photodetector(s) 4.

[0019] The substrate 2 preferably supports a plurality of photodetectors 4 in order to make the measurement described below more robust (for example by averaging the signals of photodetection delivered by the photodetectors 4). It is therefore possible to distribute several photodetectors on the surface of the screen, without there necessarily being the same number of photodetectors 4 as photoemitters 3.

[0020] The photoemitters 3 of said assembly form the pixels of the screen. They are typically arranged in a matrix array. In one possible embodiment, the photoemitters 3 are microLEDs, for example GaN-based. The microLEDs can be smart pixels, as described in Templier, F. (2023), MicroLED Technology: A Unique Opportunity Toward “More Than Displays”. Information Display, 39: 13-17. https: / / doi.org / 10.1002 / msid.1407. Each pixel of the screen can thus be formed by an elementary light-emitting chip comprising at least one LED and an electronic circuit including a control element for at least one LED. Preferably, each elementary chip comprises a plurality of sub-pixels, each comprising one LED; typically three sub-pixels associated with LEDs emitting respectively in red, green, and blue.The screen may also include a CMOS (Complementary Metal Oxide Semiconductor) driver circuit on the substrate (glass for example) for supplying control signals to the smart-pixels, this CMOS circuit taking for example the form of a set of conductive lines and columns.

[0021] MicroLEDs are not necessarily of the "smart-pixel" type and can be driven by a TFT ("Thin-Film Transistor") circuit rather than a CMOS circuit. The photoemitters can be organic diodes known as "OLEDs" or liquid crystal pixels known as "LCDs".

[0022] The photodetector(s) 4 are capable of detecting incident radiation and delivering a photodetection signal. For example, they are capable of capturing radiation in the near-infrared, within the silicon detection range (less than 1 pm). Alternatively, they may be capable of capturing radiation in the short-wave infrared, between 1 and 2 pm, typically 1.55 pm, which allows them to be less sensitive to the emission from the photoemitters 3.

[0023] In one possible embodiment, a photodetector 4 can be carried by an elementary light emission chip as described above and whose electronic circuit includes a reading and possible processing element for the photodetection signal delivered by the photodetector.

[0024] The substrate 2 further supports (at least) a laser beam emission source 5 and (at least) a beam scanning system 6 coupled to said source 5 and capable of being driven to scan the laser beam emitted by said source in a scene located in front of the substrate.

[0025] The laser beam can be a beam in the near-infrared or short-wavelength infrared. It has a range of several meters, or even tens of meters, whereas IRIS technology is limited to about thirty centimeters.

[0026] The beam scanning system 6 can, in particular, be controlled by means of a scanning angle command so that the laser beam emitted towards the scene has a given emission angle relative to the substrate. For a given scanning angle command, the emission source 5 and the scanning system 6 thus provide point illumination of the scene. Together, the emission source 5 and the scanning system 6 form a scanning emission source.

[0027] As shown in Figures IA, IB and 3, the laser beam emission source 5 and the beam scanning system 6 can be arranged on the substrate 2, preferably at its periphery. Alternatively, as shown in [Fig. 2], the laser beam emission source 5 and the beam scanning system 6 can be arranged on the cover cap 7, preferably at its periphery.

[0028] In a possible embodiment illustrated in [Fig.3], the cover hood may include a waveguide 8 configured to direct the laser beam from the emission source 5 to one or more beam scanning systems 6.

[0029] As illustrated in [Fig. 6], the beam scanning system 6 may include an optical phase-controlled grating 61 and a deflector 62. The optical phase-controlled grating 61 may be driven, in particular by means of a heater, to perform beam scanning in the plane of the substrate 2 (horizontal scanning in x, y). The deflector is an active (e.g., opto-mechanical) or passive deflector that allows beam scanning in a plane perpendicular to the substrate 2 (vertical scanning at an angle θ with respect to a normal to the substrate). In one possible embodiment, the laser beam emission source 5 is a tunable monochromatic source, and the deflector is a passive deflector in the form of a diffraction grating 62, enabling vertical scanning at an angle θ fixed by the wavelength of the tunable monochromatic source.The tunable monochromatic source can be constituted by the association of a superluminescent diode 51 and a Bragg reflector 52 allowing the wavelength to be tuned by thermo-optical effect, as for example described in Kim, SM., Lee, ES., Chun, KW. et al. Compact solid-state optical phased array beam scanners based on polymeric photonie integrated circuits. S ci Rep 11, 10576 (2021).

[0030] The beam scanning system 6 may include several deflectors (typically exhibiting different angular ranges of deflection) associated with the same emission source. The beam scanning system 6 may also include a deflector associated with one or more emission sources. It is It is also possible to plan for several different emission sources associated with different deflectors.

[0031] In another possible embodiment, the beam scanning system may be an opto-mechanical system using, for example, one or more MEMS (Micro Electro-Mechanical Systems) mirrors. The opto-mechanical system may include a Laser Beam Scanner (LBS). In yet another possible embodiment, the beam scanning system may include a programmable metasurface based on phase-change materials.

[0032] The device according to the invention further comprises an electronic system enabling the electro-optical control of the laser source(s) and the photodetector(s), and the processing of data. This electronic system may take the form of a monolithic or distributed integrated circuit. For example, part of the processing carried out by the electronic system may be performed by the electronic circuit of a "smart pixel" housing the photodetector. This may involve the basic functions of signal control and low-noise amplification. Another part of the processing may be performed by an integrated circuit located at the edge of the screen or remotely outside the screen, while remaining electrically connected to the electro-optical components of the screen.This integrated circuit is, for example, responsible for aggregating and processing data in order to ensure time synchronization and extract useful information (for example, a time-of-flight measurement).

[0033] The electronic system is configured to control the beam scanning system so that the beam scans the scene in front of the substrate, illuminating, for example, each point of the scene successively. At a given instant, the electronic system provides a scanning angle command to the beam scanning system so that it directs the beam towards a point on the scene at a specific emission angle.

[0034] The electronic system is also configured to: - acquire from photodetector 4 a photo-generated signal following the detection of the laser beam backscattered by an object in the scene illuminated by the laser beam in accordance with the scanning angle setting; and - determine a position of the object in the scene, from the photo-generated signal and the scanning angle command.

[0035] For example, the electronic system can be configured to determine the arrival time of the laser beam backscattered by the object, from a photo-generated signal and determine the position of the object from the arrival time and the scan angle setpoint.

[0036] The electronic system can also be configured to modify a control command for the photoemitters 3 in response to the determination of the object's position. The screen is thus interactive in that its display is modified according to the scene in front of the substrate, for example, according to a non-contact interaction of a user with the screen.

[0037] In a first embodiment, to determine the position of the object in the scene, the electronic system is configured to determine a time of flight, direct or indirect, of the laser beam from its emission until its detection by the photodetector.

[0038] By determining a direct time of flight, we mean a measurement of the time elapsed between the emission, by the scanning emission source 5, 6, of a light pulse towards an object in the scene and the time of arrival of this pulse at the photodetector 4. In this case, the photodetector 4 can be an APD avalanche photodiode (acronym for "Avalanche PhotoDiode") or a SPAD single photon avalanche photodiode (acronym for "Single Photon Avalanche Diode").

[0039] By determining an indirect time of flight, we mean an emission, by the scanning emission source 5, 6, of modulated light (for example in amplitude) and a measurement (by demodulation) of the phase shift of the light received by the photodetector. In this case, the photodetector 4 can be a QE modulation detector (QEM), a lateral electron field (LEF) detector or a current assisted photon demodulator (CAPD).

[0040] As shown in [Fig. 4], the laser beam is emitted from a point B at an emission angle θ with respect to the surface of the substrate or hood. The laser beam illuminates a point C of an object O in the scene and is backscattered in all directions, including in the direction of the photodetector 4 arranged at point A.

[0041] Knowing the angle 0 defined at each instant by the angle instruction of scanning and the position of the photodetector 4, the position in space of point C is determined by the distance d separating points A and C.

[0042] We have: d2 = dl2 + d22 + 2dld2£osd, where dï corresponds to the distance separating points B and C and where d2 corresponds to the distance separating points B and A.

[0043] The flight time T between points B, C and A is written y - , where c is the speed of the light in a vacuum. The distance d is then written as:

[0044] j _ “ ~ 2(T.c+ < / 2cos6)

[0045] It is therefore possible to determine the distance d separating the illuminated point C from the photodetector 4 arranged at point A using a time-of-flight measurement, knowing Furthermore, the scanning angle setting 0 and the geometry d2 of the laser beam emission-detection system are known. Given this distance d and the angle 0, it is possible to determine the 3D position of point C. By controlling the scanning system to successively illuminate different points of the scene, it then becomes possible to map the entire scene in front of the screen in 3D.

[0046] The solution described above relies on a distance estimation (d and dl) based on a photon arrival time measurement. To ensure an unambiguous position estimation in 3D space, it may be necessary to use at least three photodetectors and perform triangulation measurements. This proves particularly useful in cases of stray reflections.

[0047] Furthermore, to improve the accuracy of the time of arrival estimation, it may be useful to provide a highly accurate frequency and phase-synchronization clock device between the scanning emission source and the photodetector(s) to accurately estimate the time of arrival. One difficulty lies in the screen size, which can be relatively large compared to the size of the photodetector and the scanning emission source. Basic solutions for common time / clock between the scanning emission source and the photodetector, or for phase-locked loop (PLL) clock recovery within the photodetector, can be implemented to satisfy this synchronization requirement.

[0048] In a second embodiment, to determine the position of the object in the scene, the electronic system is configured to perform a differential time-of-arrival measurement. By providing several photodetectors distributed on the substrate, it is indeed possible to perform a differential time-of-arrival measurement between the different sensors thus distributed. These sensors are synchronized with each other but not necessarily with the scanning emission source, and their positions are predefined and known.

[0049] Figure 5 illustrates the advantage of having several scanning emission sources (here, for the emission of a beam Fl from point B1 and the emission of a beam F2 from point B2) and distributing them around the periphery of the screen to measure the distance to highly convex / concave shapes or shapes obscured by the foreground. In this Figure 5, object 01 obscures object 02 for beam Fl. Conversely, object 02 can be reached by beam F2, and the arrival time of this beam F2 at the photodetector located at point A can be measured.

[0050] The invention is not limited to the apparatus described above but also extends to a method for controlling such an apparatus, this method comprising the implementation of the following steps by the electronic system of the apparatus: - control the beam scanning system using a scanning angle command; - acquire from the photodetector a photo-generated signal following the detection of the laser beam backscattered by an object in the scene illuminated by the laser beam in accordance with the scanning angle instruction; - determine a position of the object in the scene, from the photo-generated signal and the scanning angle command.

[0051] The invention also extends to a computer program product comprising instructions which, when executed by a computer, lead the computer to implement this control method.

Claims

Demands

1. Device comprising: - a screen (1, 10, 100) having a substrate (2) which supports: • an array of photoemitters (3); • a photodetector (4); • a laser beam emission source (5, 51, 52) (F, Fl, F2); and • a beam scanning system (6, 61, 62) coupled to said source and capable of being driven to scan the laser beam emitted by said source in a scene located in front of the substrate; - an electronic system configured to: • drive the beam scanning system by means of a scanning angle command; • acquire from the photodetector (4) a photo-generated signal following the detection of the laser beam backscattered by an object (0, 01, 02) in the scene illuminated by the laser beam in accordance with the scanning angle command; • determine a position of the object in the scene, from the photo-generated signal and the scanning angle command.

2. Device according to claim 1, wherein the electronic system is further configured to modify a control setpoint for the photoemitters in response to the determination of the object's position.

3. Apparatus according to any one of claims 1 and 2, wherein the laser beam emission source (5) and the beam scanning system (6) are arranged on the substrate.

4. Apparatus according to any one of claims 1 to 3, wherein the screen (1, 10, 100) further comprises a cover hood (7) covering the photoemitter assembly and the photodetector and wherein the laser beam emission source (5) and the beam scanning system (6) are arranged on the cover hood.

5. Device according to claim 4, wherein the cover hood comprises a waveguide (8) configured for to route the laser beam from the source of emission of a laser beam to the beam scanning system.

6. Apparatus according to any one of claims 1 to 5, wherein the beam scanning system comprises an optical phase-controlled array (61) and a deflector (62).

7. Apparatus according to claim 6, wherein the laser beam emission source is a tunable monochromatic source (51, 52) and wherein the deflector (62) is a diffraction grating.

8. Apparatus according to any one of claims 1 to 5, wherein the beam scanning system is an opto-mechanical system.

9. Device according to any one of claims 1 to 5, wherein the beam scanning system comprises a programmable metasurface.

10. Apparatus according to any one of claims 1 to 9, wherein the photodetector is incorporated into an elementary chip which includes at least one of the photoemitters and an electronic circuit.

11. Apparatus according to any one of claims 1 to 10, wherein in order to determine the position of the object in the scene the electronic system is configured to determine a time of flight, directly or indirectly, of the laser beam from its emission until its detection by the photodetector.

12. Apparatus according to any one of claims 1 to 10, wherein to determine the position of the object in the scene the electronic system is configured to perform a differential measurement of arrival time.

13. A method for controlling an apparatus according to any one of claims 1 to 12, comprising implementing the following steps by the electronic system: - controlling the beam scanning system by means of a scanning angle setpoint; - acquiring from the photodetector (4) a photo-generated signal following the detection of the backscattered laser beam by an object (0, 01, 02) in the scene illuminated by the laser beam in accordance with the scanning angle setpoint; - determining a position of the object in the scene, from the photo-generated signal and the scanning angle setpoint.

14. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to claim 13.