Three-dimensional user interface device

Abstract

Three-Dimensional User Interface Device A three-dimensional user interface device comprises: * a stereoscopic or time-of-flight camera (4) arranged to return an image associated with a time marker, the data from the camera (4) further allowing the association of depth data with each pixel of said image, * a detector (6) arranged to analyze the image associated with a given time marker and to return an interaction region in said image, and * a computer (8) arranged to return interaction data derived from the comparison between, on the one hand, the coordinates of the pixels in the interaction region and their associated depth data, and on the other hand, the coordinates of the pixels associated with a predefined interaction region and their associated depth data. Fig. 1

Description

Title of the invention: Three-dimensional user interface device

[0001] The invention relates to the field of user interfaces, and in particular three-dimensional user interfaces.

[0002] User interfaces represent an area in which a significant amount of work has been done. In the case of computers, interfaces are various types of mice and keyboards that allow interaction with a display on a screen. More recently, with the rise of tablets and smartphones, interfaces have become touch-sensitive and / or wireless.

[0003] All these interfaces are based on the interaction of a user with a screen. With the advent of virtual reality and augmented reality, the interaction medium for user interfaces has changed, since screens have been replaced by the user's environment. When devices track the movement of the user or their hand, this data is not used to define an interaction. Added to this paradigm shift is the challenge of detecting user movements, since the user's environment is three-dimensional.

[0004] To detect a movement or interaction of the user in the context of virtual reality or augmented reality applications, additional devices have traditionally been used to detect the movements of the user, such as controllers or joysticks, generally equipped with inertial sensors. Advantageously, these additional devices are generally provided with buttons, which allow manual interactions for the user.

[0005] The use of additional devices is cumbersome, impractical, requires new learning and poses battery problems. For this reason, new three-dimensional user interfaces are desired that allow a more practical application of virtual reality or augmented reality.

[0006] The invention improves the situation. To this end, it proposes a three-dimensional user interface device comprising a stereoscopic or time-of-flight camera, arranged to return an image associated with a time marker, the data from the camera also making it possible to associate depth data with each pixel of said image, a detector arranged to analyze the image associated with a given time marker and to return an interaction region in said image, and a calculator arranged to return interaction data derived from the comparison between, on the one hand, the coordinates of the pixels of the interaction region and the depth data associated with them and on the other hand coordinates of the pixels associated with a predefined interaction zone and the depth data associated with them.

[0007] This device is particularly advantageous because it makes it possible to detect an interaction of a user with his environment, whether in a virtual reality environment or in an augmented reality environment, in an extremely efficient manner and with little cost in computing power. In addition, this device avoids the use of an additional device.

[0008] According to various embodiments, the invention may have one or more of the following characteristics: - the detector or the calculator is arranged to associate a bounding box with each interaction region, and in which the calculator is arranged to compare the coordinates of the pixels of this bounding box and the depth data associated with them with the coordinates of the pixels associated with a predefined interaction zone and the depth data associated with them, - the calculator is arranged to return interaction data indicating a touch as soon as the comparison between, on the one hand, the coordinates of the pixels of the interaction region and the depth data associated with them and, on the other hand, the coordinates of the pixels associated with a predefined interaction zone and the depth data associated with them indicates that the pixel coordinates of the interaction region have a distance to the coordinates of the pixels of the interaction zone that is less than a first given threshold, and that the depth data associated with the pixels of the interaction region have a distance to the depth data associated with the pixels of the interaction zone that is less than a second given threshold, - the calculator is arranged to return interaction data indicating a gesture when the calculator detects interaction data indicating a touch in images whose time markers are consecutive, and when the interaction determined in the images whose time markers are consecutive corresponds to a movement on the interaction zone defining a chosen shape, - the detector comprises a YOLOV-type neural network, or an R-CNN, or an SSD trained to detect an interaction region, - the detector is arranged, after detecting an interaction region, to process at least some of the images having a later time marker with a displacement tracking algorithm of the type to determine the interaction region, - the detector is arranged to execute a movement tracking algorithm of the Deep Sort, FairMOT, TransMOT, or ByteTrack type, - the detector is arranged to process with the displacement tracking algorithm the images whose time markers are between the time markers of two images which are processed by said neural network and which have time markers whose difference corresponds to the processing time of an image by said neural network, and - the detector is arranged to determine several interaction regions in a given image.

[0009] The invention also relates to a virtual reality or augmented reality device which comprises a device according to the invention, and in that it is arranged to use the interaction data.

[0010] The invention also relates to a method for detecting interaction in three dimensions comprising the following operations: a) acquiring images with a stereoscopic or time-of-flight camera, arranged to return an image associated with a time marker, the data from the camera also making it possible to associate depth data with each pixel of said image, b) analyze the image associated with a given time marker, return an interaction region in said image, (c) returning interaction data derived from the comparison between, on the one hand, the coordinates of the pixels of the interaction region and the depth data associated with them and, on the other hand, the coordinates of the pixels associated with a predefined interaction zone and the depth data associated with them.

[0011] According to various embodiments, the method according to the invention may have one or more of the following characteristics: - operation b) or operation c) comprises associating a bounding box with each interaction region, and wherein operation c) comprises comparing the coordinates of the pixels of this bounding box and the depth data associated with them to the coordinates of the pixels associated with a predefined interaction zone and the depth data associated with them, - operation c) comprises returning interaction data indicating a touch as soon as the comparison between, on the one hand, the coordinates of the pixels of the interaction region and the depth data associated with them and, on the other hand, the coordinates of the pixels associated with a predefined interaction zone and the depth data associated with them indicates that the pixel coordinates of the interaction region have a distance to the coordinates of the pixels of the interaction zone less than a first given threshold, and that the depth data associated with the pixels of the interaction region have a distance to the depth data associated with the pixels of the interaction zone less than a second given threshold, and - operation c) includes returning interaction data indicating a gesture when operation c) returns interaction data indicating a touch in frames whose time markers are consecutive, and when the interaction determined in the frames whose time markers are consecutive corresponds to a movement on the interaction zone defining a chosen shape.

[0012] The invention also relates to a method for determining interaction in augmented reality or virtual reality comprising the following operations: 1) presenting an augmented reality or virtual reality image to a user, 2) performing a three-dimensional interaction detection method according to the invention to determine the user's interaction with the environment defined by the augmented reality or virtual reality image to a user.

[0013] The invention also relates to a computer program comprising instructions for executing the method according to the invention, a data storage medium on which such a computer program is recorded and a computer system comprising a processor coupled to a memory, the memory having recorded such a computer program.

[0014] Other characteristics and advantages of the invention will appear more clearly on reading the following description, taken from examples given for illustrative and non-limiting purposes, taken from the drawings in which: - [Fig.l] represents a schematic diagram of a device according to the invention, - [Fig.2] represents an example of an operating loop of the device of [Fig.l], - [Fig.3] represents an example of use of the device of [Fig.l], and - [Fig.4] represents another example of use of the device of [Fig.l].

[0015] The drawings and the description below contain, for the most part, elements of a certain character. They may therefore not only serve to better understand the present invention, but also contribute to its definition, if necessary.

[0016] [Fig.l] schematically represents a three-dimensional user interface device 2 according to the invention. The device 2 comprises a camera 4, a detector 6, a calculator 8 and a memory 10.

[0017] In the example described here, the camera 4 is a stereoscopic camera, for example of the OAK-D W PoE model. Other stereoscopic cameras could be used. Alternatively, the camera 4 could be a time-of-flight camera. In both cases, the camera 4 is arranged to capture a user environment that includes at least the part of the user that is intended to be used to perform the interface interaction.

[0018] For example, this part of the user may be a finger, and the interaction may be of the "click" type or derived, and this interaction may correspond to the user pressing with his finger on a given surface, or on a given location. In an example of an augmented reality application, a button can be added on a surface of the environment, and the interface interaction be detected when the user presses with his index finger on this virtual button.

[0019] Since the environment is that of the user, it is necessarily in 3 dimensions, which makes detection difficult. The use of a stereoscopic or time-of-flight camera 4 makes it possible to produce two-dimensional images of the user's environment in which depth information can be added.

[0020] In the example described here, the camera 4 is of the digital type, that is to say that the data that it captures is in file form and is not analog. These files have a video frequency that depends on the acquisition frequency of the camera 4.

[0021] Thus, the data from the camera can be seen as sequences of images each associated with a time marker which follow one another at the video frequency. Each image acquired by the camera 4 associates color data with each pixel that composes it. Due to the use of a stereoscopic or time-of-flight camera, it is possible to associate depth data relative to the camera with each pixel of the image acquired by the camera 4. The depth information being available for each pixel, this makes it possible, as will be seen below, to determine an interaction between the user and his environment.

[0022] The depth dimension indicates for each pixel a depth relative to the camera.

[0023] When considering the coordinates of a pixel and the depth dimension associated with it, we cannot speak of 3-dimensional coordinates, because the depth data are not considered in the same frame of reference as the pixel coordinates. Rather, the images associated with the depth data provide information in 2+1 dimensions: 2 dimensions indicating the coordinates of each pixel in the image, and a depth dimension associated with each pixel in the image captured by the camera, representing the distance of that pixel from the reference point.

[0024] In the case of the OAK-D W PoE type camera, each image directly associates with each pixel a triplet of coordinates in 2+1 dimensions as described in the previous paragraph. Thus, each image allows direct access to the depth data. This is particularly advantageous because the depth data are directly accessible without any calculation being necessary. As will be seen below, having this data directly available makes it possible to increase the frequency at which the device 2 can perform interaction detection.

[0025] Alternatively, the camera 4 may return, in addition to the image data (i.e. the color data of each pixel), data that makes it possible to determine the depth data for each point of the image. In this case, the camera 4 therefore produces image data associated with a time marker, as well as data that makes it possible to determine the depth data for each pixel of the image.

[0026] All of this data is stored in the memory 10. The memory 10 can be any type of data storage suitable for receiving digital data: hard disk, hard disk with flash memory, flash memory in any form, RAM, magnetic disk, locally distributed storage or in the cloud, etc.

[0027] In the example described here, the memory 10 further receives all the data relating to the device 2, i.e. the programs and software instantiating the detector 6 and the calculator 8, the parameters and hyperparameters of the neural networks which may compose them, the weights of such neural networks, the outputs and intermediate data of the neural networks, the data received as input (if applicable), the data stored in buffer memory, as well as the interaction data as output. The data calculated by the device may be stored on any type of memory similar to the memory 10, or on it. This data may be erased after the device has performed its tasks or retained.

[0028] Interaction detection is based on the following principle: an interaction of the user with his environment can be defined as any correspondence between on the one hand a portion of the image which corresponds to a part of the user supposed to generate the interaction (for example a finger) and a portion of the image which corresponds to an interaction element (for example a button added in augmented reality, or a surface to define a click).

[0029] There are therefore two steps: a first step to determine in an image which portion corresponds to the part of the user supposed to generate the interaction, and then a second step to determine whether this portion of the image corresponds to an interaction taking into account the portion of the image which corresponds to an interaction element.

[0030] The detector 6 is used to carry out the first step, while the calculator 8 is used to carry out the second step. In the example described here, the detector 6 and the calculator 8 directly or indirectly access the memory 10. They can be implemented in the form of appropriate computer code executed on one or more processors. By processors, it is meant any processor suitable for the calculations described below. Such a processor can be implemented in any known manner, in the form of a microprocessor for a personal computer, laptop, tablet or smartphone, a dedicated chip of the FPGA or SoC type, a computing resource on a grid or in the cloud, a cluster of graphics processing units (GPUs), a mi microcontroller, or any other form capable of providing the computing power necessary for the implementation described below. One or more of these elements can also be implemented in the form of specialized electronic circuits such as an ASIC. A combination of processor and electronic circuits can also be considered. Processors dedicated to machine learning may also be considered.

[0031] In the example described here, the detector 6 comprises a neural network that is trained specifically to detect the part of the user in the images that is used to carry out the interaction. This part is subsequently called the “interaction region”. For example, a YOLOv7 type engine, which is recognized in the field of object detection, can be trained to distinguish human fingers - the fingers then constitute the interaction region sought in the images acquired by the camera 4).

[0032] The article by Wang, Chien-Yao & Bochkovskiy, Alexey & Liao, Hong-yuan. (2022), “YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors”, 10.48550 / arXiv.2207.02696 describes the architecture of this engine. For this, a specific training dataset of human hands can be used to specialize the engine of the detector 6 in the detection of a finger on an image acquired by the camera 4, regardless of its size, shape, color, coverage by a glove, etc. Alternatively, the YOLOv7 engine could be replaced by an R-CNN (Region-based Convolutional Neural Network) or even an SSD (for “Single Shot Detector”).

[0033] Once the detector 6 has detected a finger in an image from the camera 4, the tracking of its movement in the following images can be carried out using a Deep Sort type algorithm. This algorithm is described at https: / / github.com / levan92 / deep_sort_realtime. Alternatively, it would be possible to use the FairMOT algorithm, the TransMOT algorithm, or the ByteTrack algorithm.

[0034] The implementation by the detector 6 of the Deep Sort algorithm has the advantage of increasing the processing frequency because this algorithm is less expensive in terms of calculation time than the neural network. Indeed, the Deep Sort type algorithm allows image-by-image processing (i.e. it is lower than the acquisition frequency of the camera 4), whereas the camera 4 acquires several images while the neural network processes one.

[0035] Advantageously, to compensate for the fact that the Deep Sort type algorithm is less precise than the detection by the neural network 6, an alternation between the neural network and the Deep Sort type algorithm can be carried out using a buffer: - the neural network is launched on an image further in the corresponding time at the most advanced end in time of the buffer, - during the time of processing of this image by the neural network 6, the previous images in the buffer are processed by the Deep Sort type algorithm to track the finger in each image, and - when the image processing by the neural network is finished, the finger is detected again with great precision, which allows what is usually called a reconciliation operation to be carried out.

[0036] Still as a variant, the tracking algorithm could not be used and the detector 6 could apply the YOLOv7 type engine continuously.

[0037] The motor of the detector 6 can also be driven with any other part of the user's body that can be used as an interaction region, whether it be his head, a leg, a foot, or an instrument carried by the user. The most important thing is that at the output of the detector 6, thanks to the depth coordinates associated with each pixel of the image thanks to the acquisition of the camera 4, the depth of the pixels in the interaction region are available in each image.

[0038] Once the depth data and the pixels of the interaction region are known, the computer 8 can act to determine the presence or absence of an interaction. This is very simply achieved by determining whether, in a given image, the coordinates of the interaction region correspond to the coordinates of the portion of the user environment which is dedicated to the interaction.

[0039] Depending on the type of detected interaction implemented, the calculator 8 can be implemented in various ways.

[0040] Indeed, according to a first approach, the detection of an interaction can be triggered by the detection of contact with a surface, whatever it may be. In this case, the computer 8 can determine a bounding box around the determined interaction region in a given image, and search whether this bounding box corresponds to a surface in the image. By corresponding to a surface, it must be understood that the bounding box has a distance to a surface of the user environment less than a given threshold, for example 5 mm, which amounts to saying that one or more pixels in the image which are neighbors of the bounding box have depth data whose difference to the depth data of the bounding box is less than the given threshold.

[0041] Alternatively, the computer 8 can operate without a bounding box, for example by determining a contour of the interaction region or with a different and / or adaptive threshold depending on the distance between the user and the camera 4. The bounding box or its equivalent can also be determined by the detector 6 and not by the computer 8. This type of detection is particularly facilitated thanks to the coordinates in 2+1 dimensions to which the computer 8 has access.

[0042] In a version where the interactions are more complex, the interaction may for example be characterized by touching a particular region of the environment, for example a virtual button displayed superimposed on a sleeve of a garment worn by the user, or another type of region of interest in the environment.

[0043] In this case, the detection is a little more sophisticated, because it is necessary to detect that the interaction region is in contact with a part of the user's environment, but not with just any part.

[0044] Thus, the detection of a part added in augmented reality will be simpler than the detection of a part chosen arbitrarily in the environment, for example a switch in a room. In the first case, it is a matter of resetting in each acquired image the location in this image of the part added in augmented reality. In the example of the second case, it will be necessary to detect the switch in addition to the interaction region and compare its depth data to those of the interaction region and verify that the pixels of the image corresponding to them are sufficiently close.

[0045] In the two previous examples, the detected interactions are of the “touch” type. It goes without saying that the interactions can be more complex. For example, the detector 6 can be adapted to detect several interaction regions, for example several fingers, which makes it possible to carry out multi-contact interactions (“multitouch” in English), or even detect typing on a virtual keyboard. Similarly, the computer 8 can be configured to detect gesture-type interactions (“gestures” in English), in the sense of the gestural interactions of web browsers, that is to say conditioned by the detection of a click-type interaction maintained on several consecutive images. Here again, this is a fairly direct extension once the touch detection is acquired.

[0046] [Fig.2] represents an example of an operating loop of the device of the [Fig.l].

[0047] In an operation 200, the operating loop begins with the acquisition of an image by the camera 4. This acquisition is carried out by the execution of a function Acq(). This function is based on the recovery of the image Img captured by the camera 4, as well as the determination of the coordinate data Coord[] for each of the points of this image. As described above, the coordinate data Coord[] can be directly available at the output of the camera 4, or be the subject of post-processing.

[0048] Then, in an operation 210, a function Detect() calls the detector 6 with the image Img to try to detect a region of interest there, that is to say to detect if the image Img contains a portion which corresponds to a finger or another shape determined as being used to determine the interaction. The result (potentially empty) is stored in a vector RI[] which can receive either the coordinates of the shape of the interaction region detected in the image Img, either the three-dimensional coordinates corresponding to the interaction region in the image Img, or a shape derived from the shape of the interaction region, for example a bounding box, or the three-dimensional coordinates associated with this bounding box.

[0049] An operation 220 then determines whether the vector RI[] is empty, which means that no interaction region was detected in the operation 210, or whether the calculator 8 must be called in order to test whether an interaction is detected.

[0050] If the vector RI[] is empty, then the loop ends and the next loop resumes with operation 200. If the vector RI[] is not empty, then the loop continues with the call to the calculator 8 to determine whether an interaction has just been detected. As explained above, the interaction detection function Calc() depends greatly on the type of interaction to be detected. The Calc() function will therefore implement various calculations depending on whether a simple interaction with any surface is to be detected, an interaction with a specific portion of the user environment, a multiple touch interaction, or a gesture type interaction, etc.

[0051] Although [Fig. 2] represents a loop that appears linear, certain steps may be performed asynchronously. Thus, the camera 4 acquires images continuously, at its acquisition frequency, so that it is mainly the frequency of production of the three-dimensional coordinates of the operation 200 that is relevant. The operation 210 may be performed on all the images, or be deliberately omitted for certain images depending on the processing speed by the neural network of the detector 6. In addition, as described above, the tracking of the region of interest in the detector 6 may be performed by an algorithm other than the neural network once a first detection of the region of interest has taken place. In the same way, the operation 220 may be performed at a distinct frequency depending on the computation times associated with it.Ultimately, the rendering on a screen or other interaction rendering element can be set to the lowest frequency of all the frequencies mentioned above.

[0052] Although the detector 6 and the computer 8 have been presented here in the form of two separate units, this distinction is mainly functional. Thus, their operations could be carried out within a single functional module, or on the contrary be divided more finely into sub-modules.

[0053] What emerges from the above is that the invention is based on two strong principles: - the possibility, with modern cameras, of obtaining directly or almost directly at the sensor output coordinates in 2+1 dimensions for each of the pixels of the video images acquired by the camera, - image processing to detect one or more regions of interest and even actually one or more interaction zones thanks to an artificial intelligence engine,

[0054] Once the relevant portions of the image have been determined, use the 2+1 dimensional coordinates that correspond to them to determine whether there is interaction. Once this idea is stated, it seems almost obvious. However, before cameras that allow sufficient depth precision existed, it was out of the question to consider the question.

[0055] And even once these have become available, the idea of ​​processing the image using a neural network to segregate the relevant 2+1 dimensional coordinates to determine the existence of an interaction is itself extremely innovative.

[0056] Indeed, the invention uses both pixel coordinates in the image and depth data which are linked to a real reference point, which is not conventional, and makes it possible to combine both the advantages linked to acquisition by cameras and the speed of image processing combined with the fact that it is easier to determine whether neighboring points of a two-dimensional image have similar depth data than to seek to reconstruct a three-dimensional image of the environment and determine objects therein to see whether they are in contact.

[0057] Furthermore, the invention mixes several philosophies and solutions, where current developments tend to be mono-philosophical, that is to say either based on artificial intelligence exclusively, or on sensors, or on gesture tracking, or based on point cloud processing, etc.

[0058] [Fig.3] and [Fig.4] represent two series of three images resulting from processing by the device of [Fig.l].

[0059] In [Fig.3] as in [Fig.4], image (a) represents the environment before detecting a touch-type interaction on a surface, image (b) represents the environment when the user's finger touches the surface, and image (c) represents the environment when the finger has left the surface.

[0060] More precisely, these images are frames taken from the video output of the device in [Fig.l] which makes it possible to indicate by visible overlay on the image (b) of figures 3 and 4 when a touch-type interaction is detected.

[0061] The difference between [Fig.3] and [Fig.4] lies in the acquisition conditions: as can be seen in these figures, [Fig.3] was obtained with a bare hand, while [Fig.4] was obtained with a gloved hand. This shows the versatility that can be achieved with the detector 6. In addition, figure (b) of these figures also shows the bounding box that is used to detect the interaction.

[0062] These figures make it possible to understand how the device 2 can be used in the context of an augmented reality ("AR" or "Augmented Reality" type application in English): it is sufficient to display augmented reality data superimposed on an image taken from camera 4. The application to virtual reality appears in a similar way, the stream from camera 4 being then used only to detect depth data for the interaction region, this data can then be compared to the data from the virtual environment to detect interactions.

Claims

Claims

1. Three-dimensional user interface device comprising: * a stereoscopic or time-of-flight type camera (4), arranged to return an image associated with a time marker, the data from the camera (4) further enabling depth data to be associated with each pixel of said image, * a detector (6) arranged to analyze the image associated with a given time marker and to return an interaction region in said image, and * a calculator (8) arranged to return interaction data derived from the comparison between, on the one hand, the coordinates of the pixels of the interaction region and the depth data associated with them and, on the other hand, the coordinates of the pixels associated with a predefined interaction zone and the depth data associated with them.

2. Device according to claim 1, in which the detector (6) or the calculator (8) is arranged to associate a bounding box with each interaction region, and in which the calculator (8) is arranged to compare the coordinates of the pixels of this bounding box and the depth data associated with them with the coordinates of the pixels associated with a predefined interaction zone and the depth data associated with them.

3. Device according to claim 1 or 2, in which the calculator (8) is arranged to return interaction data indicating a touch as soon as the comparison between on the one hand the coordinates of the pixels of the interaction region and the depth data associated with them and on the other hand the coordinates of the pixels associated with a predefined interaction zone and the depth data associated with them indicates that the coordinates of pixels of the interaction region have a distance to the coordinates of the pixels of the interaction zone less than a first given threshold, and that the depth data associated with the pixels of the interaction region have a distance to the depth data associated with the pixels of the interaction zone less than a second given threshold.

4. Device according to one of the preceding claims, in which the computer (8) is arranged to return interaction data indicating a gesture when the computer (8) detects data interaction indicating a touch in images whose time markers are consecutive, and when the interaction determined in the images whose time markers are consecutive corresponds to a movement on the interaction zone defining a chosen shape.

5. Device according to one of the preceding claims, wherein the detector (6) comprises a YOLOV type neural network, or an R-CNN, or an SSD trained to detect an interaction region.

6. A device according to claim 5, wherein the detector (6) is arranged, after detecting an interaction region, to process at least some of the images having a later time marker with a displacement tracking algorithm of the type to determine the interaction region.

7. Device according to claim 6, wherein the detector (6) is arranged to execute a movement tracking algorithm of the Deep Sort, FairMOT, TransMOT, or ByteTrack type.

8. Device according to claim 6 or 7, wherein the detector (6) is arranged to process with the movement tracking algorithm the images whose time markers are between the time markers of two images which are processed by said neural network and which have time markers whose difference corresponds to the processing time of an image by said neural network.

9. Device according to one of the preceding claims, in which the detector (6) is arranged to determine several interaction regions in a given image.

10. Virtual reality or augmented reality device characterized in that it comprises a device according to one of the preceding claims, and in that it is arranged to use the interaction data.

11. A method for detecting interaction in three dimensions comprising the following operations: a) acquiring images with a stereoscopic or time-of-flight camera (4), arranged to return an image associated with a time marker, the data from the camera (4) also making it possible to associate depth data with each pixel of said image, b) analyzing the image associated with a given time marker, returning an interaction region in said image, c) returning interaction data derived from the comparison between, on the one hand, the coordinates of the pixels of the interaction region and the depth data associated with them and, on the other hand, co- ordinates of the pixels associated with a predefined interaction zone and the depth data associated with them.

12. The method of claim 11, wherein operation b) or operation c) comprises associating a bounding box with each interaction region, and wherein operation c) comprises comparing the coordinates of the pixels of that bounding box and the depth data associated with them to the coordinates of the pixels associated with a predefined interaction area and the depth data associated with them.

13. The method of claim 12, wherein operation c) comprises returning interaction data indicating a touch as soon as the comparison between, on the one hand, the coordinates of the pixels of the interaction region and the depth data associated with them and, on the other hand, the coordinates of the pixels associated with a predefined interaction zone and the depth data associated with them indicates that the pixel coordinates of the interaction region have a distance to the coordinates of the pixels of the interaction zone less than a first given threshold, and that the depth data associated with the pixels of the interaction region have a distance to the depth data associated with the pixels of the interaction zone less than a second given threshold.

14. The method of claim 12, wherein operation c) comprises returning interaction data indicating a gesture when operation c) returns interaction data indicating a touch in frames whose time markers are consecutive, and when the interaction determined in the frames whose time markers are consecutive corresponds to a movement on the interaction zone defining a chosen shape.

15. Method for determining interaction in augmented reality or virtual reality comprising the following operations: 1) presenting an augmented reality or virtual reality image to a user, 2) executing a method according to one of claims 11 to 14 to determine the interaction of the user with the environment defined by the augmented reality or virtual reality image to a user.

16. A computer-implemented computer program comprising instructions for performing the method according to one of claims 11 to 15.

17. Data storage medium on which the computer program according to claim 16 is recorded.

Images