Method for calibrating the display of images in a mixed reality headset

Abstract

This method (19) for calibrating a display in a mixed reality headset comprises the following steps: - Using a digital screen to display a reference image (step 23); - Using a camera to acquire (step 25) at least one image distorted by an optical system positioned between the digital screen and the camera; - Determining the change of coordinates of the pixels of the reference image with the counterpart pixels of the distorted image (step 27); and - Creating a distortion matrix (step 29) configured to apply a change of coordinates to each pixel of each image intended to be displayed in a mixed reality headset, such that the distortion induced by the optical system is partially compensated for.

Description

[0001] DESCRIPTION

[0002] TITLE: Method for calibrating the display of images in a mixed reality headset

[0003] Technical field

[0004] The present invention relates to the calibration of the display in a virtual or mixed reality headset.

[0005] In particular, the present invention relates to compensating for image distortion induced by an optical system disposed between a digital screen and an eye of a user viewing the digital screen in a mixed reality headset.

[0006] Generally speaking, the invention applies to any calibration of the display of an image intended to be observed through an optical system.

[0007] Previous techniques

[0008] A mixed reality headset is a display device configured to be placed on a user's head so that said user can observe an image displayed on a digital screen of said mixed reality headset.

[0009] A mixed reality headset includes, for example, two cameras configured to film the environment outside the headset, two digital screens in the headset transmitting the images or part of the images filmed by the cameras, as well as two locations configured to accommodate the two eyes of a user. Each eye observes a digital screen through an optical system of the mixed reality headset. The digital screen can thus display the environment outside the headset and / or other images adding to or replacing this external environment. A mixed reality headset can thus serve as a virtual reality headset.

[0010] The design of a mixed reality headset requires minimizing the mass and volume of the headset while promoting the best possible display quality for user viewing comfort. The optical system positioned between a digital screen and the user's eye must meet these specifications. Its primary objective is to allow the eye to see the image displayed on the digital screen as clearly and with the least distortion possible.

[0011] A commonly used optical system is a layer-stacked optical system known as a "pancake" comprising a succession of optical elements such as a lens, a reflective polarizer, a quarter-wave plate, a semi-reflective lens, and a circular polarizer. The principle is to control the polarization of light in order to achieve a large focal length in a minimal footprint. This so-called "pancake" optical system provides good viewing comfort for the user but is very expensive to manufacture.

[0012] Another optical system that can be used in a mixed reality headset is a freeform lens. This can be a freeform lens with 4 optical paths, or more generally a freeform lens comprising an even number of paths, also called lobes. A freeform lens is simpler to manufacture, for example by molding, and less expensive than an optical system by stacking layers. However, it induces distortions, such as optical aberrations, in the image observed by the user.

[0013] Statement of the invention

[0014] The present invention therefore aims to overcome the aforementioned drawbacks and to provide a mixed reality headset whose display system is simple and inexpensive to manufacture, and presents very little distortion in the image seen by the user.

[0015] The present invention relates to a method for calibrating the display of images in a virtual or mixed reality headset comprising a digital screen configured to be observed by a user through an optical system, the method comprising the following steps: - Positioning on the same optical axis a digital screen, an optical system and a camera, the camera being configured to image an image displayed on the digital screen through the optical system;

[0016] - Display of a reference image on the digital screen;

[0017] - Acquisition by the camera of at least one image distorted by the optical system of the reference image displayed on the digital screen;

[0018] - Determination by a computer of the change in coordinates of the pixels of the reference image with the homologous pixels of the distorted image by comparing the distorted image with the reference image; and

[0019] - Creation of a distortion matrix from the step of determining the change in coordinates of the pixels of the reference image with the homologous pixels of the distorted image, the distortion matrix being configured to be applied to each image intended to be displayed on a digital screen in a virtual or mixed reality headset, the distortion matrix being configured to apply a change in coordinates to each pixel of each displayed image so that the distortion induced by the optical system is partly compensated by the change in coordinates of the pixels of the image displayed by the digital screen of the virtual or mixed reality headset.

[0020] Thus, the creation of a distortion matrix applicable to images intended to be displayed in a mixed reality headset makes it possible to calibrate together the display of a digital screen and an optical system of a mixed reality headset for the most pleasant viewing comfort possible for the user.

[0021] Advantageously, the step of acquisition by the camera comprises the acquisition of several distorted images, preferably at least 5 distorted images, the camera being moved in a volume of between 20 mm3 and 70 mm3 between the acquisition of each distorted image, the step of determining the change in coordinates of the pixels of the reference image with the homologous pixels of the distorted image, as well as the step of creating the distortion matrix being carried out on the basis of each distorted image acquired.

[0022] Advantageously, the step of determining the change in coordinates of the pixels of the reference image with the homologous pixels of the distorted image comprises, for defective pixels of the reference image or of the distorted image for which the coordinates are aberrant and / or unknown, the determination of the change in coordinates of said defective pixels by interpolation between neighboring pixels.

[0023] In one embodiment, the reference image is a plain background on which a grid of points and / or lines is represented.

[0024] Advantageously, the method further comprises for a single optical system, and before the integration of said single optical system into a virtual or mixed reality headset, the following steps:

[0025] - Positioning on the same optical axis of a digital screen, the single optical system and a camera, the camera being configured to image an image displayed on the digital screen through the single optical system;

[0026] - Display on the digital screen of a reference image to which the distortion matrix is ​​applied;

[0027] - Acquisition by the camera of an image distorted by the unique optical system of the reference image displayed on the digital screen and to which the distortion matrix is ​​applied;

[0028] - Determination by a computer of the change in coordinates of the pixels of the reference image to which the distortion matrix is ​​applied with the homologous pixels of the distorted image by comparing the distorted image with the reference image to which the distortion matrix is ​​applied; and

[0029] - Modifying the distortion matrix into a single distortion matrix from the step of determining the change in coordinates of the pixels of the reference image to which the distortion matrix is ​​applied with the homologous pixels of the distorted image, the single distortion matrix being configured to be applied to each image intended to be displayed on a digital screen in a virtual or mixed reality headset, the single distortion matrix being configured to apply a change in coordinates to each pixel of each image intended to be displayed so that the distortion induced by the single optical system is partly compensated by the change in coordinates of the pixels of the image displayed by the digital screen of the virtual or mixed reality headset.

[0030] Advantageously, the method is implemented for each unique optical system produced from an optical system production line, a unique distortion matrix being associated with a unique optical system, the method further comprising a step of integrating the unique distortion matrix into an internal memory of a virtual or mixed reality headset comprising the unique optical system associated with said unique distortion matrix.

[0031] Advantageously, the method comprises a step of integrating two unique distortion matrices into the internal memory of the virtual or mixed reality headset, said virtual or mixed reality headset comprising two unique optical systems associated respectively with said unique distortion matrices.

[0032] The present invention also relates to the use in a virtual or mixed reality headset of a distortion matrix created by the implementation of the method as defined previously by applying said distortion matrix to each image intended to be displayed in a digital screen of the virtual or mixed reality headset, or of a single distortion matrix created by the implementation of the method as defined previously by applying said single distortion matrix to each image intended to be displayed in a digital screen configured to be observed by a user through a single optical system associated with said single distortion matrix.

[0033] The present invention also relates to a calibration bench configured to implement the method as defined previously, the calibration bench comprising a digital screen, an optical system and a camera configured to image an image displayed on the digital screen through the optical system.

[0034] Advantageously, the calibration bench further comprises a motorized platform configured to slide on several axes of movement in a volume of at least 70 mm3, the camera being mounted on the platform.

[0035] Brief description of the drawings

[0036] Other aims, characteristics and advantages of the invention will appear on reading the following description, given solely by way of non-limiting example, and made with reference to the appended drawings in which:

[0037] [Fig 1] is a schematic view of a calibration bench according to the invention; and [Fig 2] is a schematic representation of the different steps of the method for calibrating the display of images in a mixed reality headset according to the invention.

[0038] Detailed description of at least one embodiment

[0039] Figure 1 schematically shows a calibration bench 1 comprising on the same optical axis a digital screen 3, an optical system 5, for example a free-form lens with four optical channels, and a camera 7 configured to image an image displayed on the digital screen 3 through the optical system 5.

[0040] The digital screen 3 is for example a square-shaped screen, one side of which comprises between 1000 and 10000 pixels. Alternatively, the screen is round or rectangular in shape.

[0041] The camera 7 includes a sensor whose definition is equal to or greater than the definition of the digital screen 3.

[0042] The calibration bench 1 is configured to enable the calibration of the display of images in a mixed reality headset. Consequently, the digital screen 3 and the optical system 5 are similar respectively to a digital screen 3 and to an optical system 5 of said mixed reality headset. On the same principle, the distance between the optical system 5 and the digital screen 3 is similar to the distance between a similar optical system 5 and a digital screen 3 of a mixed reality headset, for example between 1 and 20 millimeters. Also, the distance between the optical system 5 and the camera 7 is substantially the same as the distance provided between a similar optical system 5 and an eye of a user when using a mixed reality headset, for example between 1 and 50 millimeters.

[0043] The calibration bench 1 further comprises a motherboard 9 configured to transmit to the digital screen 3 the images to be displayed by the latter.

[0044] The calibration bench also includes a computer 11 connected to the camera 7 as well as to the motherboard 9.

[0045] Optionally, the calibration bench 1 comprises a motorized platform 13, for example by three motors 15 directed in three orthogonal directions x, y and z, the platform being configured to slide on several axes of movement, for example the directions x, y and z. The motors 15 are for example powered by a power supply 17.

[0046] The camera 7 is mounted on the platform 13. The platform 13 is controlled so as to enable the camera 7 to capture images in a volume of at least 70 mm 3. The movement possibilities of the platform 13 in this volume allow the camera 7 to simulate the movements of an eye and thus to be able to acquire images as an eye would see them. An eye, as well as the camera 7, can move over a range of 3 to 4 mm in each direction x, y and z.

[0047] Figure 2 shows the different steps of a method 19 for calibrating the display of images in a mixed reality headset. Said method 19 can be implemented using the calibration bench 1 shown in Figure 1.

[0048] Firstly, a step 21 is carried out for positioning on the same optical axis a digital screen 3, an optical system 5 and a camera 7, the camera 7 being configured to image an image displayed on the digital screen 3 through the optical system 5.

[0049] Then, a step 23 of displaying a reference image on the digital screen 3 is carried out.

[0050] The reference image is, for example, a plain background on which a grid of points and / or lines is represented. The grid structure makes it easy to detect distortions of the grid, while a contrast between the grid and the plain background allows the identification of said grid. For a 3-dimensional digital screen with a side of between 1000 and 2000 pixels, the reference image comprises, for example, points represented by a square of 3 pixels on a side, spaced 25 to 50 pixels apart.

[0051] Then, a step 25 is carried out of acquisition by the camera 7 of at least one image distorted by the optical system 5 of the reference image displayed on the digital screen 3.

[0052] The optical system 5 can induce distortions, the distorted image represents for example a grid distorted by the optical system 5.

[0053] Optionally, step 25 of acquisition by the camera 7 comprises the acquisition of several distorted images, preferably at least 5 distorted images, more preferably at least 25 distorted images, the camera 7 being moved in a volume of between 20 mm 3 and 70 mm 3 between the acquisition of each distorted image.

[0054] For example, the camera 7 is moved between each distorted image acquisition by the platform on which it is arranged, said platform being configured to move between 0 and 4 mm in one or more of the x, y and z directions.

[0055] In order to minimize the calculation time of the following steps 27 and 29, the acquisition step 25 is optionally carried out by acquiring four series of images, for example a series of images corresponding to a quadrant of a distorted image, the recombination of the images four by four forming a distorted image.

[0056] Then, a step 27 is carried out for determining by the computer 11 the change in coordinates of the pixels of the reference image with the homologous pixels of each distorted image by comparing each distorted image with the reference image. A homologous pixel is understood to mean a pixel of a distorted image which contains the same information as a pixel of the reference image, the distorted image having a number of pixels identical to the number of pixels of the reference image, either by acquiring said same number of pixels, or by resizing the number of pixels of the distorted image. Thus, for a reference image with a black background comprising a pink pixel, the homologous pixel of the distorted image is the pink pixel of the distorted image, regardless of whether the coordinates of said pink pixel have been modified.

[0057] This step 27 makes it possible to identify the deformations induced by the optical system 5 for each pixel of the reference image, and this, for each possible viewing angle when several distorted images are acquired.

[0058] In a particular embodiment, some pixels of the reference image or of the distorted image are defective. It is then not possible to link the homologous pixels of the reference image to the distorted image. Defective is understood to mean a pixel that is not seen, or whose coordinates are not known or not consistent with the neighboring pixels. In this embodiment, step 27 of determining the change in coordinates comprises determining the change in coordinates of said defective pixels by interpolation between neighboring pixels.

[0059] More generally, an interpolation or an averaging filter can be used during this step 27 of determining the change of coordinates in order to avoid errors of association between a pixel of the reference image with a pixel of the distorted image.

[0060] Then, a step 29 is performed for creating a distortion matrix from the step 27 for determining the change in coordinates of the pixels of the reference image with the homologous pixels of each distorted image, the distortion matrix being configured to be applied to each image intended to be displayed on a digital screen 3 in a mixed reality headset. In particular, the distortion matrix is ​​configured to apply a change in coordinates to each pixel of each image intended to be displayed on a digital screen 3 so that the distortion induced by the optical system 5 is partly compensated by the change in coordinates of the pixels of the image displayed by the digital screen 3 of the mixed reality headset.

[0061] The distortion matrix thus generated, in particular when the method comprises the acquisition of several distorted images from different points of view, allows very precise compensation for the deformation induced by the optical system 5 used for implementing the method.

[0062] However, during the mass production of optical systems, each manufactured optical system 5 comprises its own defects which can in turn induce image distortions slightly different from the distortions induced by the optical system 5 used for implementing the method described above. Each optical system 5 is thus referred to as a single optical system 5B and the previously generated distortion matrix only partially compensates for the distortions induced by each single optical system 5B.

[0063] Optionally, in order to more precisely compensate for distortions in the image display of mixed reality headsets, the previous steps can be performed a second time, this time implemented with a different single optical system 5B. In other words, the following steps 21B, 23B, 25B, 27B, 29B are performed for each single optical system 5B coming off a production line. The single optical systems are, for example, free-form lenses.

[0064] A step 21B is therefore carried out for positioning on the same optical axis a digital screen 3, the single optical system 5B and a camera 7, the camera 7 being configured to image an image displayed on the digital screen 3 through the single optical system 5B. Then, a step 23B is carried out for displaying on the digital screen 3 a reference image to which the previously created distortion matrix is ​​applied.

[0065] Then, a step 25B is carried out for the camera 7 to acquire an image distorted by the single optical system 5B of the reference image displayed on the digital screen 3 and to which the distortion matrix is ​​applied. A single distorted image is acquired for the sake of speed at the output of the production line, a single image making it possible to adjust the distortion matrix already generated beforehand.

[0066] Then, a step 27B is carried out for determining by a computer the change in coordinates of the pixels of the reference image to which the distortion matrix is ​​applied with the homologous pixels of the deformed image by comparing the deformed image with the reference image to which the distortion matrix is ​​applied.

[0067] A step 29B of modifying the distortion matrix into a single distortion matrix is ​​then performed from the step 27B of determining the change in coordinates of the pixels of the reference image to which the distortion matrix is ​​applied with the homologous pixels of the distorted image. The single distortion matrix is ​​specific to a single optical system 5B and is configured to be applied to each image intended to be displayed on a digital screen 3 in a mixed reality headset. The single distortion matrix is ​​configured to apply a change in coordinates to each pixel of each image intended to be displayed so that the distortion induced by the single optical system 5B is partly compensated by the change in coordinates of the pixels of the image displayed by the digital screen 3 of the mixed reality headset.

[0068] Optionally, the method further comprises a step 31 of integrating the single distortion matrix into an internal memory of a mixed reality headset comprising the single optical system 5B associated with said single distortion matrix. In the case where said mixed reality headset comprises two locations configured to accommodate the eyes of a user, two single optical systems 5B, and two digital screens, the method comprises the integration of two single distortion matrices into the internal memory of the mixed reality headset, the single optical systems 5B being respectively associated with said single distortion matrices. In a variant, the two digital screens are combined into a single digital screen 3 comprising a first portion intended to be observed via a single optical system 5B, and a second portion intended to be observed via another single optical system 5B.

Claims

CLAIMS 1. Method (19) for calibrating the display of images in a virtual or mixed reality headset comprising a digital screen (3) configured to be observed by a user through an optical system (5), characterized in that it comprises the following steps: Positioning on the same optical axis of a digital screen (3), an optical system (5) and a camera (7) (step 21), the camera (7) being configured to image an image displayed on the digital screen (3) through the optical system (5); Display on the digital screen (3) of a reference image (step 23); Acquisition (step 25) by the camera (7) of at least one image distorted by the optical system (5) of the reference image displayed on the digital screen (3); Determining by a computer (11) the change in coordinates of the pixels of the reference image with the homologous pixels of the distorted image by comparing the distorted image with the reference image (step 27); and Creating a distortion matrix (step 29) from the step (27) of determining the change in coordinates of the pixels of the reference image with the homologous pixels of the distorted image, the distortion matrix being configured to be applied to each image intended to be displayed on a digital screen (3) in a virtual or mixed reality headset, the distortion matrix being configured to apply a change in coordinates to each pixel of each displayed image so that the distortion induced by the optical system (5) is partly compensated by the change in coordinates of the pixels of the image displayed by the digital screen (3) of the virtual or mixed reality headset.

2. Method according to claim 1, in which the step (25) of acquisition by the camera (7) comprises the acquisition of several distorted images, preferably at least 5 distorted images, the camera (7) being moved in a volume of between 20 mm 3 and 70 mm 3 between the acquisition of each distorted image, the step (27) of determining the change in coordinates of the pixels of the reference image with the homologous pixels of the distorted image, as well as the step (29) of creating the distortion matrix being carried out on the basis of each acquired distorted image.

3. Method according to one of claims 1 and 2, in which the step (27) of determining the change in coordinates of the pixels of the reference image with the homologous pixels of the distorted image comprises, for defective pixels of the reference image or of the distorted image for which the coordinates are aberrant and / or unknown, the determination of the change in coordinates of said defective pixels by interpolation between neighboring pixels.

4. Method according to any one of claims 1 to 3, in which the reference image is a plain background on which a grid of points and / or lines is represented.

5. Method according to any one of claims 1 to 4, further comprising for a single optical system (5B) and before the integration of said single optical system (5B) into a virtual or mixed reality headset, the following steps: Positioning (step 21B) on the same optical axis of a digital screen (3), of the single optical system (5B) and of a camera (7), the camera (7) being configured to image an image displayed on the digital screen (3) through the single optical system (5B); Display (step 23B) on the digital screen (3) of a reference image to which the distortion matrix is applied; Acquisition (step 25B) by the camera (7) of an image distorted by the single optical system (5B) of the reference image displayed on the digital screen (3) and to which the distortion matrix is applied; Determining by a computer (11) the change in coordinates of the pixels of the reference image to which the distortion matrix is applied with the homologous pixels of the distorted image by comparing the distorted image with the reference image to which the distortion matrix is applied (step 27B);and Modifying the distortion matrix into a single distortion matrix (step 29B) from the step (27B) of determining the change in coordinates of the pixels of the reference image to which the distortion matrix is applied with the homologous pixels of the distorted image, the single distortion matrix being configured to be applied to each image intended to be displayed on a digital screen (3) in a virtual or mixed reality headset, the single distortion matrix being configured to apply a change in coordinates to each pixel of each image intended to be displayed so that the distortion induced by the single optical system (5B) is partly compensated by the change in coordinates of the pixels of the image displayed by the digital screen (3) of the virtual or mixed reality headset.; 6. Method according to claim 5, implemented for each unique optical system (5B) produced from an optical system production line (5; 5B), a unique distortion matrix being associated with a unique optical system (5B), the method further comprising a step (31) of integrating the unique distortion matrix into an internal memory of a virtual or mixed reality headset comprising the unique optical system (5B) associated with said unique distortion matrix.

7. Method according to claim 6, comprising a step (31) of integrating two unique distortion matrices into the internal memory of the virtual or mixed reality headset, said virtual or mixed reality headset comprising two unique optical systems (5B) associated respectively with said unique distortion matrices.

8. Use in a virtual or mixed reality headset of a distortion matrix created by implementing the method according to any one of claims 1 to 4 by applying said distortion matrix to each image intended to be displayed in a digital screen (3) of the virtual or mixed reality headset, or of a single distortion matrix created by implementing the method according to any one of claims 5 to 7 by applying said single distortion matrix to each image intended to be displayed in a digital screen (3) configured to be observed by a user through a single optical system (5B) associated with said single distortion matrix.

9. Calibration bench (1) configured to implement the method according to any one of claims 1 to 7, comprising a digital screen (3), an optical system (5; 5B) and a camera (7) configured to image an image displayed on the digital screen (3) through the optical system (5; 5B).

10. Calibration bench according to claim 9, further comprising a motorized platform (13) configured to slide on several axes of movement (x, y, z) in a volume of at least 70 mm 3 , the camera (7) being mounted on the platform (13).

Images