Device and method for determining position of object in front of display screen, and computer program

Abstract

The invention provides a device and method for determining the position of an object in front of a display screen and a computer program. The apparatus comprises: a screen (1) comprising a substrate (2) supporting a set of photoemitters (3), a photodetector (4), a light source (5) for emitting a laser beam, and a beam scanning system (6) coupled with the light source and drivable to scan the laser beam in a scene in front of the substrate; an electronic system configured to: drive the beam scanning system by a scanning angle set value; acquiring, from the photodetector (4), a light generation signal generated by detecting a laser beam backscattered by an object in the scene when the laser beam irradiates the scene according to the scanning angle set value; and determining a position of an object in the scene based on the light generation signal and the scan angle setpoint.

Description

Technical Field

[0001] The present invention pertains to image display screens and provides an apparatus, method, and computer program for determining the position of an object in front of the display screen. More specifically, the present invention relates to an interactive screen that combines light emission and 3D optical capture functions, allowing depth mapping of a scene located in front of the screen. Background Technology

[0002] A non-contact 3D technology called Infra-Red Intelligent Surface (IRIS) uses a sensor consisting of staggered infrared (IR) emitters and receivers distributed on a surface. This IRIS technology is described, for example, in "Infra-Red Intelligent Surface for Near-Field Touchless Displays" by Santoull, E., Hemery, E., and Tucky, J., published in Information Display in 2023 (Information Display, 39:18-21. https: / / doi.org / 10.1002 / msid.1408).

[0003] IRIS technology uses intensity measurements and knowledge of sensor geometry to derive a 3D point cloud of the scene above the sensor. The sensor operates in the near field, forming a continuous detection field from the surface up to tens of centimeters above.

[0004] More specifically, the sensor operates by first emitting a controlled pattern of IR light from an emitter array. The light is then reflected by an object in front of the sensor, and the reflected light is detected by a receiver array. The detected light is processed by signal processing circuitry to remove noise and ambient light. The resulting 2D image contains information about the type of object in front of the sensor and its x, y, z positions. This is used to create a 3D point cloud of the scene in front of the sensor. This data allows the system to infer the user's intent and interact accordingly.

[0005] The maximum detection range of a sensor is on the order of surface size, which means that a sensor the size of a multi-functional mobile phone can detect objects up to tens of centimeters in size, while a sensor the size of a 55-inch screen can detect objects several meters away.

[0006] Industrial-grade IRIS sensors will consist of a stack of multiple layers, including a substrate, optoelectronic elements (IR emitters and receivers or RGB pixels), and a collimation layer. The collimation layer allows for the manipulation of infrared light, causing it to converge toward the receiver. However, this collimation layer introduces additional technical complexity. Summary of the Invention

[0007] The present invention aims to eliminate the need for aligned layers while improving the performance of 3D capture in far fields typically greater than 30 cm.

[0008] Therefore, the present invention provides an apparatus comprising:

[0009] The screen includes a substrate, which supports:

[0010] A set of photoelectric transmitters;

[0011] Photodetector;

[0012] A light source for emitting laser beams; and

[0013] A beam scanning system coupled to a light source and capable of being driven to scan a laser beam emitted by the light source in a scene in front of a substrate;

[0014] The electronic system is configured as follows:

[0015] The beam scanning system is driven by the scanning angle setting value;

[0016] The light generation signal is obtained from the photodetector. This light generation signal is generated by detecting the laser beam that is backscattered by objects in the scene when the laser beam illuminates the scene according to the scanning angle set value.

[0017] The position of objects in the scene is determined based on the light generation signal and the scanning angle setting.

[0018] Some preferred but non-limiting aspects of the device are as follows.

[0019] The electronic system is also configured to modify the drive settings of the photoelectric emitter in response to the determination of the object's position.

[0020] The laser source and beam scanning system used to emit the laser beam are both arranged on the substrate.

[0021] The screen also includes a cap that covers a set of photoelectric emitters and photoelectric detectors, and a laser source and beam scanning system for emitting laser beams are arranged on the cap.

[0022] The cap includes a waveguide configured to guide a laser beam from a laser beam emission source to a beam scanning system.

[0023] The beam scanning system includes an optical phased array and a deflector.

[0024] The laser beam emission source is a tunable monochromatic light source, and the deflector is a diffraction grating.

[0025] A beam scanning system is an optomechanical system.

[0026] The beam scanning system includes a programmable metasurface.

[0027] The photodetector is integrated into a base chip, which includes at least one of a photoelectric emitter and electronic circuitry.

[0028] In order to determine the position of objects in a scene, the electronic system is configured to directly or indirectly determine the flight time of the laser beam from emission to detection by a photodetector.

[0029] In order to determine the position of objects in the scene, the electronic system is configured to perform differential measurements of arrival time.

[0030] The present invention also provides a method for monitoring the above-mentioned device, comprising the following steps performed by an electronic system: driving a beam scanning system by a scanning angle setting value; acquiring a light generation signal from a photodetector, the light generation signal being generated by detecting a laser beam backscattered by an object in the scene when the laser beam illuminates the scene according to the scanning angle setting value; and determining the position of the object in the scene based on the light generation signal and the scanning angle setting value.

[0031] The present invention also provides a computer program, including instructions, which, when executed by a computer, cause the computer to perform the above-described method. Attached Figure Description

[0032] Other aspects, objects, advantages, and features of the invention will become more apparent upon reading the following detailed description of preferred embodiments, given by way of non-limiting example and described with reference to the accompanying drawings.

[0033] Figure 1A and Figure 1B These are, respectively, a side view and a top view of the screen of the device according to a first possible embodiment of the present invention.

[0034] Figure 2 This is a side view of the screen of a device according to a second possible embodiment of the present invention.

[0035] Figure 3 This is a top view of the screen of the device according to a third possible embodiment of the present invention.

[0036] Figure 4 This is a diagram illustrating the principle of distance measurement that can be achieved by the device according to the invention.

[0037] Figure 5 The figure illustrates the advantages of providing several laser emission sources in the device according to the invention.

[0038] Figure 6 This is a diagram of a scanning emission source that can be used within the framework of this invention. Detailed Implementation

[0039] This invention relates to a device including displays 1, 10, and 100, for example, for use in a computer, multi-functional mobile phone, television, or tablet computer. (Reference) Figure 1A , Figure 1B , Figure 2 and Figure 3 The screen includes, for example, a substrate 2 made of glass, which supports a set of photoelectric emitters 3 and one or more photodetectors 4. The screen also includes, for example, a cover 7 made of glass, which covers the photoelectric emitters 3 and one or more photodetectors 4.

[0040] The substrate 2 preferably supports a plurality of photodetectors 4 to make the measurements described below more robust (e.g., by averaging the photodetector signals transmitted by the photodetectors 4). Thus, several photodetectors can be distributed on the surface of the screen, and the number of photodetectors 4 need not be the same as the number of photoemitters 3.

[0041] The set of photoelectric emitters 3 forms the pixels of the screen. They are typically arranged in a matrix array. In one possible embodiment, the photoelectric emitters 3 are microLEDs, such as GaN-based microLEDs. MicroLEDs can be smart pixels, as described in "MicroLED Technology: A Unique Opportunity Toward 'More Than Displays'" by Templier, F., published in Information Display, 39:13-17 (https: / / doi.org / 10.1002 / msid.1407). Therefore, each pixel of the screen can be formed by a basic light-emitting chip, which includes at least one LED and electronic circuitry including components for controlling the at least one LED. Preferably, each basic chip includes multiple sub-pixels, each sub-pixel including an LED. Typically, three sub-pixels are associated with LEDs emitting red, green, and blue light, respectively. The screen may also include complementary metal-oxide-semiconductor (CMOS) driving circuitry on a substrate (e.g., made of glass) for transmitting control signals to smart pixels. For example, the CMOS circuitry may take the form of a set of conductive rows and columns.

[0042] Miniature LEDs are not necessarily of the smart pixel type and can be driven by thin-film transistor (TFT) circuits instead of CMOS circuits. The photoelectroluminescent device can be an organic diode called an OLED or a liquid crystal pixel called an LCD.

[0043] One or more photodetectors 4 are capable of detecting incident radiation and transmitting photodetector signals. For example, photodetectors can capture near-infrared radiation within the silicon detection range (less than 1 μm). Alternatively, photodetectors can capture short-wave infrared radiation (between 1 μm and 2 μm, typically 1.55 μm), making them less sensitive to the emission of photoemitter 3.

[0044] In one possible embodiment, the photodetector 4 may be carried by a basic light-emitting chip as described above, and the electronic circuitry of the basic light-emitting chip includes components for reading and possibly processing the photodetector signal transmitted by the photodetector.

[0045] The substrate 2 also supports at least one laser beam emitting source 5 and at least one beam scanning system 6, which is coupled to the source 5 and can be driven to scan the laser beam emitted by the source in a scene in front of the substrate.

[0046] The laser beam can be a near-infrared beam or a short-wave infrared beam. It has a range of several meters or even tens of meters, while IRIS technology is limited to about thirty centimeters.

[0047] Specifically, the beam scanning system 6 can be driven by a scanning angle setting, so that the laser beam emitted toward the scene has a given emission angle relative to the substrate. For a given scanning angle setting, the emission source 5 and the scanning system 6 provide point illumination for the scene. The emission source 5 and the scanning system 6 together form a scanning emission source.

[0048] like Figure 1A , Figure 1B and Figure 3 As shown, the laser beam emission source 5 and the beam scanning system 6 can be arranged on the substrate 2, preferably on the periphery of the substrate 2. Alternatively, as... Figure 2 As shown, the laser beam emission source 5 and the beam scanning system 6 can be arranged on the cap 7, preferably on the periphery of the cap.

[0049] exist Figure 3 In one possible embodiment shown, the cap may include a waveguide 8 configured to guide a laser beam from the emission source 5 to one or more beam scanning systems 6.

[0050] like Figure 6As shown, the beam scanning system 6 may include an optical phased array 61 and a deflector 62. Specifically, the optical phased array 61 can be driven by means of a heater to scan the beam in the plane of the substrate 2 (horizontally scanning in x and y). The deflector is an active (e.g., optomechanical) or passive deflector that allows scanning the beam in a plane perpendicular to the substrate 2 (vertically scanning at an angle θ relative to the normal of the substrate). In one possible embodiment, the laser beam emission source 5 is a tunable monochromatic light source, and the deflector is a passive deflector in the form of a diffraction grating 62, which allows vertical scanning at an angle θ set by the wavelength of the tunable monochromatic light source. A tunable monochromatic light source can be composed of a superluminescent diode 51 and a Bragg reflector 52, used to tune the wavelength via a thermo-optical effect, as described, for example, in “Compact solid-state optical phased array beam scanners based on polymeric photonic integrated circuits” (Sci Rep 11, 10576) by Kim, SM., Lee, ES., Chun, KW et al., published in 2012.

[0051] The beam scanning system 6 may include several deflectors (typically with different angular deflection ranges) associated with the same emission source. The beam scanning system 6 may also include deflectors associated with one or more emission sources. Several different emission sources associated with different deflectors may also be provided.

[0052] In another possible embodiment, the beam scanning system may be an optomechanical system using, for example, one or more microelectromechanical system (MEMS) type mirrors. The optomechanical system may include a laser beam scanner (LBS). In yet another possible embodiment, the beam scanning system may include a programmable metasurface based on a phase change material.

[0053] The apparatus according to the invention also includes an electronic system for electro-optically driving a laser source and a photodetector, as well as an electronic system for data processing. This electronic system can be in the form of a monolithic or distributed integrated circuit. For example, a portion of the processing implemented by the electronic system can be performed by electronic circuitry housing smart pixels of the photodetector. This portion can be the basic functions of signal control and low-noise amplification. Another portion of the processing can be performed by an integrated circuit located at or away from the screen edge, while being electrically connected to the electro-optical elements of the screen. This integrated circuit, for example, is responsible for aggregating and processing data to ensure time synchronization on the one hand, and extracting useful information (e.g., time-of-flight measurement) on the other.

[0054] The electronic system is configured to drive the beam scanning system such that the beam scans the scene in front of the substrate by, for example, sequentially illuminating each point in the scene. At a given moment, the electronic system provides the beam scanning system with a scanning angle setting, causing the beam scanning system to guide the beam to a point in the scene according to the emission angle.

[0055] The electronic system is also configured to: acquire a light generation signal from a photodetector 4, which is generated by detecting a laser beam backscattered by an object in the scene when the laser beam illuminates the scene according to a scanning angle set value; and determine the position of the object in the scene based on the light generation signal and the scanning angle set value.

[0056] For example, the electronic system can be configured to determine the arrival time of a laser beam backscattered by an object based on a light generation signal, and to determine the position of the object based on the arrival time and a scanning angle setpoint.

[0057] The electronic system can also be configured to modify the drive settings of the photoelectric emitter 3 in response to the determination of the object's position. Therefore, the screen is interactive because its display is modified based on the scene in front of the substrate, for example, based on touchless interaction between the user and the screen.

[0058] In a first embodiment, in order to determine the position of an object in a scene, the electronic system is configured to determine the direct or indirect flight time of a laser beam from its emission to its detection by a photodetector.

[0059] Determining the direct time of flight refers to measuring the time elapsed between the emission of a light pulse from scanning sources 5 and 6 towards an object in the scene and the arrival of that pulse at photodetector 4. In this case, photodetector 4 can be an avalanche photodiode (APD) or a single-photon avalanche diode (SPAD).

[0060] Determining the indirect time of flight involves scanning the emitted modulated light (e.g., amplitude) from sources 5 and 6 and measuring (by demodulating) the phase shift of the light received by the photodetector. In this case, the photodetector 4 can be a QE modulation detector (QEM), a Lateral Electron Field (LEF) detector, or a Current-assisted Photonic Demodulator (CAPD).

[0061] like Figure 4As shown, the laser beam is emitted from point B at an emission angle θ relative to the surface of the substrate or cover. The laser beam illuminates point C of object O in the scene and backscatters in all directions, especially in the direction of the photodetector 4 arranged at point A.

[0062] The angle θ at each moment is known to be defined by the scanning angle setting and the position of the photodetector 4. The position of point C in space is determined by the distance d between point A and point C.

[0063] We have: d 2 =d1 2 +d2 2 +2d1.d2.cosθ,, where d1 corresponds to the distance between point B and point C, and d2 corresponds to the distance between point B and point A.

[0064] The flight time T between points B, C, and A is written as Where C is the speed of light in a vacuum. Then the distance d is written as:

[0065]

[0066] Therefore, given the known scanning angle setting θ and the geometry d2 of the laser beam emission-detection system, the distance d between the illumination point C and the photodetector 4 positioned at point A can be determined using time-of-flight measurement. Knowing this distance d and the angle θ, the 3D position of point C can be determined. By driving the scanning system to sequentially illuminate different points in the scene, the entire scene in front of the screen can be mapped in 3D.

[0067] The above solution is based on distance estimates (d and d1) relative to measurements of photon arrival time. To ensure precise position estimation in 3D space, it may be necessary to use at least three photodetectors and perform triangulation. This is particularly useful in cases of interferometric reflection.

[0068] Furthermore, to improve the accuracy of arrival time estimation, providing a high-precision frequency and phase synchronization clock between the scanning transmitter and the photodetector can be useful for accurate arrival time estimation. One challenge lies in the screen size, which can be relatively large compared to the size of the photodetector and the scanning transmitter. This synchronization condition can be met by implementing a common time base / clock solution between the scanning transmitter and the photodetector, or a phase-locked loop (PLL) clock recovery solution within the photodetector.

[0069] In the second embodiment, to determine the position of an object in the scene, the electronic system is configured to perform differential time-of-arrival measurements. Differential time-of-arrival measurements can be performed among various sensors distributed on a substrate. These sensors are synchronized with each other, but not necessarily with the scanning emission source, and their positions are predefined and known.

[0070] Figure 5 This demonstrates the advantages of having several scanning emission sources (here, beam F1 is emitted from point B1, beam F2 is emitted from point B2) and distributing them around the perimeter of the screen to measure the distance to highly convex / concave shapes or shapes hidden by the foreground. Figure 5 In the diagram, for beam F1, object O1 hides object O2. However, object O2 can be reached by beam F2, and the time it takes for beam F2 to reach the photodetector located at point A can be measured.

[0071] The present invention is not limited to the previously described apparatus, but extends to a method for monitoring such an apparatus, the method comprising the steps performed by the electronic system of the apparatus: driving a beam scanning system by a scanning angle setpoint; acquiring a light generation signal from a photodetector, the light generation signal being generated by detecting a laser beam backscattered by an object in the scene when the laser beam illuminates the scene according to the scanning angle setpoint; and determining the position of the object in the scene based on the light generation signal and the scanning angle setpoint.

[0072] The invention also extends to computer program products that include instructions that, when executed by a computer, cause the computer to implement the monitoring method.

Claims

1. An apparatus comprising: a screen (1, 10, 100) comprising a substrate (2) supporting: a set of optoelectronic emitters (3); an optoelectronic detector (4); a light source (5, 51, 52) for emitting a laser beam (F, F1, F2); and a beam scanning system (6, 61, 62) coupled with the light source and drivable to scan the laser beam emitted by the light source in a scene in front of the substrate; an electronic system configured to: drive the beam scanning system by a scan angle setting value; acquire, from the optoelectronic detector (4), a light generation signal generated by detecting the laser beam backscattered by an object (O, O1, O2) in the scene when the scene is illuminated by the laser beam according to the scan angle setting value; determine a position of the object in the scene based on the light generation signal and the scan angle setting value. The electronic system is further configured to modify a drive setting value of the optoelectronic emitters in response to the determination of the position of the object.

2. The apparatus of claim 1, wherein, The laser source (5) for emitting the laser beam and the beam scanning system (6) are both arranged on the substrate.

3. The apparatus of claim 1 or 2, wherein, The screen (1, 10, 100) further comprises a cover cap (7) covering the set of optoelectronic emitters and the optoelectronic detector, and the laser source (5) for emitting the laser beam and the beam scanning system (6) are arranged on the cover cap.

4. The apparatus of any one of claims 1 to 3, wherein, The cover cap comprises a waveguide (8) configured to conduct the laser beam from the laser beam emission source to the beam scanning system.

5. The apparatus of claim 4, wherein, The beam scanning system comprises an optical phased array (61) and a deflector (62).

6. The apparatus of any one of claims 1 to 5, wherein, The laser beam emission source is a tunable monochromatic light source (51, 52) and the deflector (62) is a diffraction grating.

7. The apparatus of claim 6, wherein, The beam scanning system is an optomechanical system.

8. The apparatus of any one of claims 1 to 5, wherein, The beam scanning system comprises a programmable metasurface.

9. The apparatus of any one of claims 1 to 5, wherein, The optoelectronic detector is integrated into a base chip comprising at least one of the optoelectronic emitters and the electronic circuitry.

10. The apparatus of any one of claims 1 to 9, wherein, To determine the position of the object in the scene, the electronic system is configured to determine, directly or indirectly, a time of flight experienced by the laser beam from emission to detection by the optoelectronic detector.

11. The apparatus of any one of claims 1 to 10, wherein, To determine the position of the object in the scene, the electronic system is configured to perform a differential measurement of time of arrival.

12. The apparatus of any one of claims 1 to 10, wherein, 13. A method of monitoring an apparatus according to any one of claims 1 to 12, comprising implementing by the electronic system the steps of: driving the beam scanning system by a scan angle setting value; acquiring, from the optoelectronic detector (4), a light generation signal generated by detecting the laser beam backscattered by an object (O, O1, O2) in the scene when the scene is illuminated by the laser beam according to the scan angle setting value; and determining a position of the object in the scene based on the light generation signal and the scan angle setting value.

14. A computer program comprising instructions which, when executed by a computer, cause the computer to perform the method of claim 13. ​

Images