A method for dynamically viewing stereoscopic images of 3D video, a head-mounted display, and a 3D display device implementing this method.
By modifying stereoscopic 2D image pairs on head-mounted displays or 3D display devices and utilizing viewpoint changes and depth information, the images are dynamically adjusted to provide 3D rendering, solving the problems of insufficient immersive experience and high resource costs in existing technologies, and achieving efficient dynamic 3D video display.
Patent Information
- Application Number
- CN202480091164.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-25
- Publication Date
- 2026-11-17
AI Technical Summary
Existing 3D video display devices cannot provide an immersive experience for dynamically observing scenes, and the time and resource costs of generating synthetic 3D images are high.
By modifying stereoscopic 2D image pairs on a head-mounted display or 3D display device, and utilizing viewpoint changes and depth information, the images are dynamically adjusted to provide 3D rendering. This includes moving, expanding, or shrinking pixels or groups of pixels in the image, and combining depth maps and interpupillary distance information to achieve viewpoint changes.
It enhances the user's immersive experience, reduces the time and resource costs of generating 3D images, and enables 3D video display of dynamically observed scenes.
Smart Images

Figure CN123125848A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to stereoscopic 3D video intended to be displayed in a head-mounted display or on a 3D display device to provide 3D rendering during display.
[0002] This invention relates to the dynamic observation display of stereoscopic 3D video in a head-mounted display or a 3D display device. Background Technology
[0003] 3D video is known in the prior art and can be used to introduce perspective effects from 2D images into displayed 3D video, as would be perceived by an observer directly viewing the imaged scene. The 3D video according to the invention is not a computer-generated image or three-dimensional computer graphics, but a three-dimensional image that has been generated or has had perspective or depth effects introduced using computer-aided design (CAD). 3D images can be generated directly from models of real-world scenes. The production of 3D images is costly in terms of the time and IT resources required to process and store these large images. 3D images can be understood, for example, as relief images obtained through modeling a 3D surface in space or from two images with different parallaxes (typically two images obtained from two discrete viewpoints). The workload increases tenfold when 3D image streaming is involved, particularly when it involves 3D image streaming generated from modeling a 3D surface in space. Furthermore, the processing time is incompatible with one or more 2D images acquired in real-time for dynamic observation. The time required to generate a composite 3D image is incompatible with one or more 2D images acquired in real-time for dynamic observation. The production of composite 3D images is costly in terms of the time and IT resources required to process and store these images.
[0004] This invention relates to 3D video, also known as stereoscopic video, which provides 3D rendering by reconstructing depth and providing the observer with a sense of depth when displayed on a suitable display medium by utilizing the observer's binocular vision.
[0005] Stereoscopic vision is known in existing technology. Human binocular vision (or more broadly, animal binocular vision) based on the comparison of two images by the brain is the most widespread example. The biological processing of the acquired images is extremely powerful because it provides a real-time concept of the depth of the observed scene, making it possible, for example, to know the relative distance of the observed object while moving around.
[0006] Observing stereoscopic 3D videos based on or mimicking the principles of stereoscopic vision is known in the prior art. In practice, stereoscopic images comprise two discrete images of the same scene acquired at two discrete locations in space. Human stereoscopic vision enables the reconstruction of a scene, including depth information, from a 3D video comprising a set of stereoscopic images. The principle of stereoscopic observation lies in enhancing the immersive nature of image observation by increasing the depth effect through projecting a single synthetic image or monocular 3D video (which comprises a mixture of two 2D images of the same scene acquired from different viewpoints) onto a single 3D display screen, or by projecting 2D images of the same scene acquired from different viewpoints onto two discrete display screens (each screen viewed by one of the observer's eyes).
[0007] Head-mounted displays, 3D glasses, and 3D displays are also known. These display devices enable the viewing of videos in 3D. However, a drawback of these display devices is that they provide a fixed view of the scene, which remains the view seen by the camera when the video is captured and moves with the display device. During dynamic observation of a scene, the viewpoint from which the scene is observed remains fixed. The observed scene gives the observer the impression of moving with the display device, which fails to provide an immersive experience when watching video.
[0008] One object of the present invention is to provide a method for processing 3D video, a head-mounted display, and a 3D display device:
[0009] - This enables the overcoming of the problems of existing technological methods, and / or
[0010] - The viewpoint changes of the video it renders to the user of a head-mounted display or 3D display device, and / or
[0011] - This enables users of head-mounted displays or 3D display devices to have an immersive experience by enhancing and / or improving stereoscopic 3D video display scenes, and / or
[0012] - It constitutes an alternative to head-mounted displays and 3D display devices that display synthesized 3D images. Summary of the Invention
[0013] To address this, a method for generating stereoscopic images from 3D video for dynamically viewed scenes is proposed. The 3D video comprises a set of stereoscopic 2D image pairs, referred to as one or more source image pairs, which can provide 3D rendering of the video when displayed in a head-mounted display or a 3D display device.
[0014] According to the present invention, the term source image can be understood to mean any stored or available image, such as an image stack, particularly an image stack from 3D video. As stated elsewhere, source images may originate from or be transmitted by an external device, preferably in real time, so that they can preferably be processed in real time by the method according to the present invention.
[0015] Preferably, the images of the source image pair have been acquired by one or more imaging systems.
[0016] At least one source image pair is a 2D image that may include depth information.
[0017] The method includes the following steps: obtaining the viewpoint change relative to the imaged scene.
[0018] The term "imaged scene" can be understood as referring to the image of the source image pair, i.e., the unmodified image or the image before modification.
[0019] The method further includes the following steps: modifying at least one source image pair, preferably each of the images in each source image pair, preferably each image, more preferably each of two images, to render at least one stereoscopic 2D image pair of the scene, referred to as at least one rendered image pair.
[0020] Preferably, the two images of at least one rendered image pair correspond to the two images of at least one source image pair modified by implementing the method.
[0021] Preferably, the number of rendered image pairs corresponds to the number of modified source image pairs.
[0022] At least one rendered image is intended or is capable of providing 3D rendering when displayed in a head-mounted display or a 3D display device.
[0023] At least one rendered image is provided or intended to provide a viewpoint change in 3D video relative to the scene, preferably when displayed in a head-mounted display or a 3D display device.
[0024] The image of at least one source image pair is modified, or the steps of modifying each image of at least one source image pair are performed, according to the following: by moving and / or expanding and / or shrinking pixels or one or more groups of pixels of at least one source image pair, and / or by adding pixels or groups of pixels to the image of at least one source image pair and / or removing pixels or groups of pixels from the image of at least one source image pair.
[0025] - Depth map of the imaged scene
[0026] - The resulting viewpoint changes.
[0027] Preferably, according to the invention, only some source image pairs (i.e., at least a portion of the source image pairs) are modified among all source image pairs. In practice, in some cases, there may be no viewpoint change, or the viewpoint change may only occur on some images of all source image pairs in the 3D video. This can happen, for example, if the observer of the 3D video remains stationary for a period of time while watching the 3D video, and therefore, no viewpoint change occurs during that time period.
[0028] Preferably, the step of modifying at least one source image pair of the scene is performed by a processing unit.
[0029] Preferably, the viewpoint change corresponds to a shift from one viewpoint (referred to as the previous viewpoint) to a different viewpoint (referred to as the new viewpoint). Preferably, the images of the image pair obtained by implementing the method, or the images modified according to the method, correspond to images of the scene as they will be observed from the new viewpoint.
[0030] The term viewpoint can be understood as referring to a location or coordinate in space.
[0031] Preferably, each data processing step of the method according to the invention, including, for example, any operation performed on or on the data (e.g., calculation, determination, or comparison), is implemented by a processing unit or any device or system capable of processing data.
[0032] Preferably, the two images of the source image pair correspond to two discrete viewpoints of the scene.
[0033] Preferably, a pixel corresponds to or is associated with a point in the imaged scene, and a group of pixels corresponds to or is associated with a portion of an object in the imaged scene.
[0034] Preferably, the movement of some pixels or one or more groups of pixels in each image of the source image pair enables, is intended to reproduce, or includes the reproduction of the parallax effect caused by the viewpoint change. Preferably, moving some pixels or one or more groups of pixels is understood to mean: modifying the position of some pixels or one or more groups of pixels in each image of the source image pair, and / or translating and / or expanding and / or shrinking some pixels or one or more groups of pixels in each image of the source image pair, preferably in a plane or reference frame specific to each of the two images of at least one source image pair, or in a reference frame shared by the two images of at least one source image pair.
[0035] Preferably, contraction can be understood as meaning settling, reduction, or compression.
[0036] A plane or reference frame can be understood to mean either a plane or reference frame common to the images under consideration, or a plane or reference frame separate from the images under consideration, preferably a plane or reference frame specific to each image in the images under consideration.
[0037] Preferably, each pixel or group of pixels is moved and / or translated and / or expanded and / or shrunk relative to a reference depth. Preferably, the reference depth is associated with one or more objects or a group of objects in the scene.
[0038] Depth information can be understood as data in an image of a scene that represents, corresponds to, or provides information about the depth of a pixel or group of pixels associated with a point in the imaged scene, or a group of pixels associated with an object in the imaged scene.
[0039] Depth information can be included in a graph or image. Depth information can be included in a depth map or an inverse depth map. Furthermore, according to the present invention, a depth map can replace, or be equivalent to, an inverse depth map.
[0040] Depth can be understood as the distance between a point or object in the imaged scene and the optical sensor of the imaging device that acquires the image of the scene.
[0041] Depth information can be understood as referring to a depth field. This depth field can include the depth information of all points or objects in the scene, or some points or objects, preferably the depth field of the imaged scene. The depth field can be contained within the channels of all pixels or some pixels of one or more images in a source image pair.
[0042] At least some pixels of one or more images from at least one source image pair of the scene may include, but do not necessarily include, depth information and / or sharpness information.
[0043] Preferably, the modification steps involve:
[0044] - Modify each of the two images in the considered source image pair in the same way so as to render at least one image pair including the two modified images, or
[0045] - Modify each of the two images in the source image pair under consideration differently in order to render at least one image pair that includes the two modified images.
[0046] Preferably, the step of modifying the image of at least one source image pair is also performed relative to or based on a region, which is preferably a single region of each of the two images of at least one source image pair, or a region of multiple portions of each of the two images of at least one source image pair, which comes from at least one source image pair and corresponds to a region called the region of interest.
[0047] The region of interest can be understood as one image in a pair of source images or one or more pixels in each image.
[0048] Preferably, the region of interest is shared by the two images of at least one source image pair, or within each of the two images of the at least one source image pair, it corresponds to or is associated with the same region of the same object or scene in the scene.
[0049] Preferably, the step of modifying the image of at least one source image pair is also performed relative to or based on the depth associated with the region of interest.
[0050] Preferably, the area of interest is:
[0051] - Remains unchanged when at least one source image pair is modified, or
[0052] - Modified when at least one source image pair is modified.
[0053] The region of interest can be modified based on modifications to the images of at least one source image pair, and can be modified in the same or different ways as the images of at least one source image pair have been modified.
[0054] Preferably, modifications to at least one image of the scene are performed, implemented, or carried out on all images in each of at least one source image pair, except for the region of interest. In other words, preferably, the movement of some pixels or one or more groups of pixels in one or each image of the source image pair, and / or the addition of pixels or groups of pixels to one or each image of the source image pair, and / or the removal of pixels or groups of pixels from one or each image of the source image pair, are preferably performed only outside the region of interest.
[0055] Preferably, one or more pixels in the region of interest remain unchanged and / or remain stationary during the modification step.
[0056] Preferably, when modifying a pair of source images, the region of interest is kept constant by anchoring it relative to one or more planes or reference frames of at least one source image pair or each image.
[0057] Preferably, the images of at least one source image pair are modified proportionally to the viewpoint change.
[0058] According to the present invention, modifying at least one image can be understood as meaning: moving and / or expanding and / or shrinking pixels or one or more groups of pixels within the image of at least one source image pair, and / or adding an amount or number of pixels or groups of pixels to the image of at least one source image pair, and / or removing an amount or number of pixels or groups of pixels from the image of at least one source image pair.
[0059] Preferably:
[0060] - Modifications to a portion of pixels or pixel groups (referred to as the first portion) of at least one source image pair are inversely proportional to the depth associated with the pixels or pixel groups of the first portion, and / or
[0061] - Modifications to a portion of the image of at least one source image pair that differs from the first portion (referred to as the second portion) are proportional to the depth associated with the pixels or pixel group of the second portion, and
[0062] - More preferably, a portion of the image of at least one source image pair that differs from the first and second portions (referred to as the third portion, corresponding to the region of interest):
[0063] • Remain unchanged, or
[0064] • Modified in whole or in part according to Part I, or in whole or in part according to Part II, or modified in a manner different from Part I and Part II.
[0065] According to the first alternative, the first portion may consist of all pixels or pixel groups of the images of at least one source image pair, preferably excluding one or more pixels or pixel groups of the region of interest. According to the first alternative, the modification of all pixels of at least one source image pair (preferably excluding one or more pixels or pixel groups of the region of interest) is inversely proportional to the depth associated with the pixels or pixel groups of the first portion.
[0066] According to the second alternative, the second portion may consist of all pixels or pixel groups of the images of at least one source image pair, preferably excluding one or more pixels or pixel groups of the region of interest. According to the second alternative, the modification of all pixels of at least one source image pair (preferably excluding one or more pixels or pixel groups of the region of interest) is inversely proportional to the depth associated with the pixels or pixel groups of the second portion.
[0067] Preferably:
[0068] - For objects in the scene whose depth is greater than the depth of the region of interest, referred to as distant objects, the modification steps include shifting the distant objects (or the pixels or groups of pixels corresponding to each object) in the same direction and direction as the viewpoint change, and / or
[0069] - For objects in the scene whose depth is less than the depth of the region of interest, called near objects, the modification steps include shifting the near objects (or the pixels or groups of pixels corresponding to each near object) in the same direction as the viewpoint change and in the opposite direction to the viewpoint change.
[0070] Preferably, the method includes: for each pixel or one or more groups of pixels in at least one image, calculating:
[0071] - The depth deviation or difference between the region of interest and each pixel or one or more groups of pixels located outside the region of interest, and / or
[0072] - The deviation or difference between the inverse of the depth of the region of interest (or the depth associated with one or more pixels in the region of interest) and the inverse of the depth of each pixel or group of pixels outside the region of interest (or the depth associated with each pixel or group of pixels outside the region of interest).
[0073] Preferably, the step of modifying at least one image is performed based on the calculated deviation.
[0074] Preferably, the calculation of the depth deviation between the region of interest and each pixel or one or more pixel groups located outside the region of interest can be defined as follows: calculating the deviation between the inverse of the depth associated with the region of interest and the inverse of the depth associated with each pixel or one or more pixel groups located outside the region of interest.
[0075] Preferably, the method includes calculating at least one angle associated with or corresponding to a change in viewpoint.
[0076] Preferably, the step of modifying the image of at least one source image pair is performed based on at least one calculated angle.
[0077] At least one angle associated with or corresponding to a change in viewpoint can be defined as the angle formed between the observer's optical axis before the change in viewpoint and the observer's optical axis after the change in viewpoint.
[0078] Preferably, the method includes calculating a distance change relative to the imaged scene that is associated with or corresponds to a change in viewpoint.
[0079] Preferably, the step of modifying the image of at least one source image pair is performed based on the calculated distance change.
[0080] The change in angle and / or distance relative to the imaged scene associated with a change in viewpoint can be understood as the change in the observer's viewpoint relative to the scene they will see.
[0081] For example, a change in viewpoint may include or correspond to the movement of the observer relative to the 3D video of the imaged scene, preferably a change in angle and / or a change in distance.
[0082] Preferably, the modification step is applied to a stream or stack of source image pairs that are stored, transmitted (e.g., via a data transmission device and / or an imaging device), or acquired (e.g., via an imaging device).
[0083] Preferably, the stream or stack of the stored source image pairs forms a 3D video.
[0084] Preferably, the modification step is applied to the stream of source image pairs. Preferably, the obtaining step and the modification step are applied to the stream of source image pairs.
[0085] Preferably, the modification of at least one source image pair depends on the interpupillary distance.
[0086] The step of modifying the image of at least one source image pair is performed based on interpupillary distance, preferably when the distance between the viewpoint of one image of the two images of the considered source image pair and the other viewpoint of the other image of the two images of the considered source image pair is too large and / or too small relative to the interpupillary distance.
[0087] Preferably, the image of at least one source image pair is also modified according to the following:
[0088] - Obtain the distance between the two discrete imaging devices for each source image pair, and / or
[0089] - The position or arrangement or spatial coordinates of two discrete imaging devices relative to each other and / or relative to the scene, and / or
[0090] - Distance deviation between the two discrete imaging devices and the scene.
[0091] Preferably, the two discrete imaging devices are arranged at two discrete spatial locations, and one image of each source image pair originates from one of the two imaging devices, or has been acquired by one of the two imaging devices, or is being acquired by one of the two imaging devices, and the other image of each source image pair originates from the other of the two imaging devices, or has been acquired by the other of the two imaging devices, or is being acquired by the other of the two imaging devices.
[0092] Preferably, in addition to modifications implemented to enable dynamic observation based on viewpoint changes, this step is performed when it is observed, determined, or calculated that the two images of at least one source image pair must also be modified to improve the stereoscopic 3D rendering that the rendered image pair is intended to provide.
[0093] Depth can be understood as the distance between a point or object in the imaged scene and the optical sensor of the imaging device that acquires the image of the scene (e.g., an image of a source image pair).
[0094] During the step of modifying the images of at least one source image pair, the positions of the two discrete imaging devices relative to each other and / or relative to the scene are considered, preferably such that each rendered image pair can consist of an image called the right image and an image called the left image, the right image as it will be perceived by the right eye of the scene observer, and the left image as it will be perceived by the left eye of the scene observer.
[0095] Preferably, by moving and / or expanding and / or shrinking pixels or one or more groups of pixels of at least one source image pair of the scene, and / or by adding pixels or groups of pixels to the image of at least one source image pair and / or removing pixels or groups of pixels from the image of at least one source image pair, the viewpoint difference between the two images of at least one rendered image pair corresponds to the interpupillary distance, thereby also modifying the images of at least one source image pair relative to each other.
[0096] Preferably, the method includes the step of displaying an image of at least one rendered image pair or at least one source image pair in a virtual reality / augmented reality (VR / AR) head-mounted device (referred to as a head-mounted display) or on a 3D display device, referred to as a display step.
[0097] The display step may also include displaying one or more source image pairs, for example, when no viewpoint change is obtained.
[0098] Preferably, the head-mounted display or 3D display device includes two discrete imaging devices.
[0099] Preferably, the viewpoint change corresponds to, is, or is based on the change in position of the head-mounted display or 3D display device in space and / or the movement relative to the scene that the observer will see.
[0100] Preferably, in each image of at least one image pair, or for each image of at least one image pair, the region of interest corresponds to the same object (one or more) in the imaged / displayed scene. Preferably, in each image of at least one image pair, or for each image of at least one image pair, the region of interest corresponds to one or more pixels or groups of pixels corresponding to the same one or more objects in the imaged / displayed scene.
[0101] Preferably:
[0102] - The resulting viewpoint change depends on the position and / or orientation and / or tilt of the head-mounted display or 3D display device in space, or on the position and / or orientation and / or tilt of the observer relative to the 3D display device in space, and / or
[0103] - The area of focus corresponds to the area in the displayed scene where the observer's gaze is focused.
[0104] The viewpoint can correspond to the center of a straight line segment connecting the same part of each of the observer's eyes, such as the center of a straight line connecting the fovea, cornea, or iris of each of the observer's eyes.
[0105] Preferably, the viewpoint corresponds to the corresponding position of each of the observer's eyes, such as the corresponding position of the fovea, cornea, or iris of each of the observer's eyes. In this case, the viewpoint change on which the modification of at least one source image pair is based will be different for each of the two images in the at least one source image pair.
[0106] A change in an observer's viewpoint can be defined as a shift in the observer's perspective from a previous viewpoint to a new viewpoint.
[0107] The observer's viewpoint can be defined as:
[0108] - The position and / or orientation of the observer, preferably the position and / or orientation of the observer's head and / or eyes, or
[0109] - The position and / or orientation of each of the observer's eyes, preferably the corresponding position and / or orientation of the fovea, cornea, or iris of each of the observer's eyes.
[0110] Preferably, the region of interest corresponds to a point or object, pixel, or group of one or more pixels, or region of the image of at least one source image pair that is focused on or intended to be focused on by the observer's gaze. Preferably, the region of the image of at least one source image pair that is focused on by the observer's gaze corresponds to the same point, the same object, or group of one or more objects in the imaged scene.
[0111] Preferably, the region of interest can be determined or calculated based on information relating to the region of at least one source image pair or the displayed image that is focused on or intended to be focused on by the observer's gaze.
[0112] Preferably, the region of interest corresponds to a point, such as a pixel or group of pixels, located or intended to be located within the corresponding field of view cone of each of the images in at least one source image pair. For example, the region of interest may correspond to a point or region with the lowest depth within the observer's field of view cone in the images of at least one source image pair of the scene. Preferably, the observer's field of view cone extends about the rotation axis of the optical axis of one of the observer's eyes, and is formed by a solid angle of about 60°, including a solid angle between 3° and 5° relative to the optical axis of one of the observer's eyes, in which visual acuity is maximized.
[0113] Preferably:
[0114] - Viewpoint changes are obtained from position data (e.g., position and / or orientation and / or tilt data) or data related to changes in the position of the head-mounted display and / or data related to movement of the head-mounted display (e.g., changes or variations in position data), said position data or data related to changes in position and / or movement being referred to as head-mounted display data, and / or
[0115] - Viewpoint changes are obtained from the observer's image and / or the image of the observer's environment and / or the image of the head-mounted display or 3D display device, and / or
[0116] - The area of interest is obtained from the image from the observer's eye.
[0117] Preferably, at least one image of the observer's eye is obtained from an imaging device, preferably from a head-mounted display or 3D display device, or one or more imaging devices capable of imaging the observer and / or the observer's eye.
[0118] The observer's viewpoint and / or area of focus can be determined through eye tracking.
[0119] Data from a head-mounted display or a 3D display device can be measured or detected by the head-mounted display or the 3D display device. Changes in viewpoint can be determined or calculated based on data from the head-mounted display or the 3D display device.
[0120] Images of the observer and / or the head-mounted display or 3D display device may be acquired by an external imaging device, which is preferably not part of or constituting a component of the head-mounted display or 3D display device. Images of the observer's environment may be acquired by the external imaging device and / or by one or more imaging devices of the head-mounted display or 3D display device.
[0121] External imaging devices can be configured to image observers and / or head-mounted displays or 3D display devices.
[0122] Preferably, in this application, location data is understood to mean position (preferably relative position) and / or orientation (preferably relative orientation) and / or tilt (preferably relative tilt).
[0123] Preferably, the method includes: linearly or nonlinearly filtering the data from the head-mounted display or 3D display device to limit the magnitude of the obtained viewpoint change over time to a value below a limit.
[0124] Preferably, the method includes: non-linearly limiting the data from the head-mounted display or 3D display device to limit the magnitude of the obtained viewpoint change to a value below a limit.
[0125] Preferably, the method includes: nonlinearly and dynamically compressing the gain of the data applied to the head-mounted display or 3D display device to attenuate or amplify the amplitude variation or magnitude of the obtained viewpoint change over time, such that the amplitude does not exceed an amplitude variation threshold or amplitude threshold.
[0126] Preferably, the method includes: nonlinearly and dynamically amplifying the gain of the data applied to the head-mounted display or 3D display device to vary the magnitude of the change in the obtained viewpoint over time.
[0127] The steps of gain filtering and / or limiting and / or dynamic compression have the effect of limiting and / or reducing the amplitude and / or amplitude variation of viewpoint changes over time.
[0128] Changes in the position and / or orientation and / or tilt of a head-mounted display or 3D display device and / or an observer in space can be defined or characterized as actual viewpoint changes or effective viewpoint changes.
[0129] Effective viewpoint change can be defined as a change measured or detected by a head-mounted display or 3D display device, or a viewpoint change determined or calculated based on data from a head-mounted display or 3D display device.
[0130] The obtained viewpoint change can be equal to, correspond to, or be proportional to the effective viewpoint change. This is especially true when the magnitude of the viewpoint change and / or the magnitude variation is less than one or more limit values or thresholds.
[0131] The resulting viewpoint change may differ from the effective viewpoint change. This is especially true when the magnitude of the viewpoint change and / or the magnitude variation exceeds one or more limits or thresholds.
[0132] Preferably, the method includes an acquisition step, which is preferably performed by a head-mounted display or a 3D display device, the acquisition step acquiring:
[0133] - Images of the scene, the acquired images of the scene include all source image pairs, and / or
[0134] - An image of the observer, which may include an image of the observer's eyes, and / or
[0135] - An image of at least a portion of the environment of a head-mounted display or an image of at least a portion of the environment of a 3D display device.
[0136] According to the present invention, a data processing apparatus is also proposed, which includes means arranged and / or programmed and / or configured to implement the method according to the present invention.
[0137] According to the present invention, a computer program comprising instructions is also provided, which, when executed by a computer, cause the computer to implement the method according to the present invention.
[0138] According to the present invention, a computer-readable medium is also provided, comprising instructions that, when executed by a computer, cause the computer to perform the method according to the present invention.
[0139] According to the present invention, a virtual reality / augmented reality (VR / AR) head-mounted display, hereinafter referred to as a head-mounted display, is also provided, comprising one or more 3D video display devices, and:
[0140] - A means arranged and / or programmed and / or configured to implement the method according to the invention, and / or
[0141] - A communication device arranged and / or programmed and / or configured to and / or capable of communicating with an external device, said external device being arranged and / or programmed and / or configured to implement the method according to the invention.
[0142] Preferably, one or more display devices of the head-mounted display are capable of or arranged to provide an observer with 3D rendering of a displayed 3D video, which comprises or consists of a set of stereoscopic 2D image pairs.
[0143] Head-mounted displays are capable of being or can be configured to display 3D video using monocular and / or binocular vision. The term head-mounted display can be understood as referring to 3D glasses.
[0144] Preferably, the head-mounted display includes at least one imaging device arranged to image the eyes of an observer wearing the head-mounted display.
[0145] Preferably, the head-mounted display includes:
[0146] - At least one imaging device arranged to image at least a portion of the environment of a VR / AR head-mounted display and / or at least a portion of the observer's environment, preferably different from a device for imaging the observer's eyes, preferably comprising two discrete imaging devices, and / or
[0147] - A device arranged to detect the movement and / or position and / or orientation and / or tilt of a VR / AR head-mounted display.
[0148] According to the present invention, a 3D display device is also provided, which is arranged to display 3D video or includes one or more means for displaying 3D video, the 3D display device comprising:
[0149] - A means arranged and / or programmed and / or configured to implement the method according to the invention, and / or
[0150] - A communication device arranged and / or programmed and / or configured to and / or capable of communicating with an external device, wherein the external device is arranged and / or programmed and / or configured to implement the method according to the invention.
[0151] Preferably, the 3D display device includes at least one imaging device arranged to image the observer's eye.
[0152] Preferably, the 3D display device includes:
[0153] - At least one imaging device arranged to image at least a portion of the environment of the 3D display device and / or an observer and / or at least a portion of the observer's environment, preferably different from a device imaging the observer's eye, preferably comprising two discrete imaging devices, and / or
[0154] - A device arranged to detect the movement and / or position and / or orientation and / or tilt of an observer relative to a 3D display device.
[0155] As a non-limiting example, a 3D display device can be a screen (e.g., an LCD or plasma screen), a video projector, a tablet computer, or a smartphone.
[0156] Preferably, at least a portion of the environment of the head-mounted display or 3D display device includes at least a portion of the scene in front of the head-mounted display (i.e., the scene facing the observer or the scene opposite the observer, preferably the entire scene). Preferably, the scene in front of the head-mounted display is the imaged scene; in other words, the scene in front of the head-mounted display is the scene contained in each image of the set of source image pairs, or the scene represented in each image of the set of source image pairs.
[0157] Preferably, at least a portion of the observer's environment includes all or part of the environment located to the side and / or behind the head-mounted display or 3D display device, i.e., all or part of the environment located to the side and / or behind the observer.
[0158] At least one source image pair or each source image pair may (but not necessarily must) be acquired by, or originate from, the imaging device of the head-mounted display or 3D display device, or be acquired by the imaging device of the head-mounted display or 3D display device.
[0159] Preferably, the head-mounted display or 3D display device includes two discrete imaging devices.
[0160] Preferably, the head-mounted display or 3D display device is arranged to measure and / or detect changes in viewpoint, or to determine and / or calculate changes in viewpoint based on data from the head-mounted display or 3D display device.
[0161] Preferably, the head-mounted display and 3D display device according to the invention are arranged to provide 3D rendering to a pair of rendered stereoscopic 2D images, or to enable the pair of rendered stereoscopic 2D images to be 3D rendered, or to supply 3D rendering to the pair of rendered stereoscopic 2D images. Where appropriate, the head-mounted display may include, or the 3D display device may be combined with, or used in conjunction with:
[0162] - In the case of 3D display devices, the synchronization shutter mechanism of active (or synchronous) glasses,
[0163] - Polarizing devices or polarized glasses, such as synchronization glasses, polarized glasses, or glasses equipped with red / blue filters.
[0164] Preferably, according to the present invention, 3D rendering of a stereoscopic 2D image is understood to mean any display method or process that enables 3D rendering from a pair of displayed 2D images.
[0165] Those skilled in the art will be able to envision various display modes that may generate 3D renderings from a pair of 2D images to be displayed.
[0166] Preferably, the two images of each rendered image pair (where the stream of the rendered image pair forms or constitutes 3D video) can be displayed simultaneously in a binocular head-mounted display, with one displayed image viewed by the right eye and the other displayed image viewed by the left eye.
[0167] Preferably, the two images of each rendered image pair (where the streams of the rendered image pair form or constitute 3D video) can be spatially multiplexed and simultaneously displayed in a monocular head-mounted display or 3D display device, for example, in the form of a single composite image or raster image, the image comprising a portion of each of the two images of each rendered image pair. In this case, only one portion of each of the two images of each rendered image pair is displayed in a single image.
[0168] Preferably, the two images of a rendered image pair can be simultaneously displayed in a monocular head-mounted display or 3D display device by superimposing the two images of each rendered image pair, and each of the two images in each rendered image pair has discrete polarization.
[0169] Preferably, the two images of the scene in each rendered image pair can be displayed continuously, alternately, or iteratively, for example, through time multiplexing, in a synchronous monocular head-mounted display or on a 3D display device, preferably in conjunction with a user wearing synchronous glasses.
[0170] Preferably, the head-mounted display is a binocular head-mounted display, wherein each image of the scene for each rendered image pair is displayed as if seen by only one eye, enabling stereoscopic viewing of the scene.
[0171] Preferably, the head-mounted display is worn by the user during the implementation of the method.
[0172] Preferably, the method according to the invention is suitable, more preferably particularly suitable, and even more preferably designed and particularly advantageously designed specifically for implementation by a VR / AR head-mounted display or 3D display device according to the invention. Therefore, any feature of the method according to the invention can be directly applied to a VR / AR head-mounted display or 3D display device according to the invention, and conversely, any feature of a VR / AR head-mounted display or 3D display device according to the invention can also be directly applied to the method according to the invention. Attached Figure Description
[0173] Other advantages and features of the present invention will become apparent upon review of the detailed description of the fully non-limiting embodiments and examples, and from the accompanying drawings:
[0174] Figure 1 This is an example of a spatially immersive 3D video viewed through existing virtual reality glasses.
[0175] Figure 2 A flowchart illustrating an embodiment of the method according to the present invention is shown.
[0176] Figure 3A and Figure 3B These are schematic diagrams illustrating two examples of changes in the observer's viewpoint within the scene.
[0177] Figure 4A and Figure 4B This is a schematic diagram showing a pair of stereoscopic 2D source images of a scene.
[0178] Figure 4C and Figure 4D This is a schematic diagram showing a pair of stereoscopic 2D images of a rendered scene.
[0179] Figures 5 to 7 It shows Figure 4A and 4B A schematic diagram showing the perspective view of the scene.
[0180] Figure 8 yes Figure 4Aand Figure 4B A schematic diagram of the depth map of the scene shown. Detailed Implementation
[0181] The embodiments described below are not intended to be limiting. In particular, variations of the invention may be considered that include only a selection of a subset of the disclosed features, isolated from other disclosed features (even if the selection is isolated within a sentence that includes other features), provided that the feature selection is sufficient to provide a technical advantage or to distinguish the invention from the prior art. Such selection may include at least one preferred functional feature lacking structural detail or having only a subset of structural detail, provided that the subset is only sufficient to provide a technical advantage or to distinguish the invention from the prior art.
[0182] refer to Figures 1 to 7 An embodiment of a method for dynamically observing 3D video according to the present invention is shown, the method being referred to as "the method" in the remainder of this disclosure.
[0183] Those skilled in the art will be able to grasp the concepts and implementations of 3D images, stereo vision, monocular vision, and binocular vision.
[0184] Figure 1 An example of monocular 3D video viewed through existing virtual reality glasses is shown, which are designed to provide dynamic viewing for enhanced immersion. Figure 1 Four frames from a static scene in a 3D video are shown, in which the observer's feet are visible, and the observer is facing a sunlit beach in the background. These images illustrate the rendering of the scene as seen by the virtual reality glasses during a lateral scan movement from left to right and back again. The image on the left corresponds to the image rendered when the observer wearing the virtual reality glasses turns their head to the left, and the image on the right corresponds to the image rendered when the observer turns their head to the right.
[0185] from Figure 1 It can be seen that existing devices and methods for dynamic observation pivot the image of a scene to improve the observer's experience. Furthermore, vanishing perspective lines (e.g., the foreground and the sun) maintain their orientation relative to the observer as the observer turns their head. From the observer's perspective, everything happens as if the scene rotates with the observer. In reality, the observer continuously views the scene from the same viewpoint as when viewing from a fixed camera. In contrast, pivoting movement visible at the outer edge of the image aims to provide an effect of movement relative to the scene. However, the resulting effect is disorienting and actually detracts from the immersive experience.
[0186] The method for dynamically viewing 3D video according to the present invention aims to address these shortcomings.
[0187] Figure 2 A flowchart of the method according to the present invention is shown.
[0188] Scene 1's 3D video comprises a set of n stereoscopic 2D image pairs, referred to as one or more source image pairs, denoted as (IS1, IS2)n. The considered source image pair is denoted as (IS1, IS2)i, where, for the considered source image pair i, IS1... i It is one of the two images considered for pair i, IS2 i It is the other of the two images in the considered pair i. The 3D video comprises a set, stream, or stack of n source image pairs (IS1, IS2). IS1 is an image acquired by a first imaging device, and IS2 is an image acquired by a second imaging device. Each image IS1 of each source image pair i is acquired by the first imaging device, and each image IS2 of each source image pair i is acquired by the second imaging device. The imaging device according to the invention is a camera or, among other things, a lens associated with an image sensor.
[0189] Each source image pair consists of two discrete images, I1 and I2, representing the same scene. These discrete images are preferably acquired simultaneously by two discrete cameras at two discrete locations in space. The two images I1 and I2 of each source image pair correspond to observations of the scene from different viewpoints, the distance between which corresponds to the spacing between the two cameras I1 and I2. Therefore, each stereoscopic 2D image pair (I1, I2) of a 3D video can or is intended to provide 3D rendering of the video by displaying a stream of source image pairs (I1, I2) in a head-mounted display or on a 3D display device.
[0190] Each stereoscopic 2D source image pair, consisting of two discrete images of the same scene acquired at two discrete locations in space, can generate and provide 3D rendering by displaying the source image pair in a head-mounted display or on a 3D display device. Furthermore, according to the invention, the statement "capable of generating 3D images" should not be understood to refer only to the case where two images are acquired at a horizontal distance equal to the interpupillary distance. This case corresponds to a specific situation.
[0191] The method according to the invention is intended to be implemented by a virtual reality / augmented reality (VR / AR) head-mounted display (hereinafter referred to as a head-mounted display), or by any 3D display device capable of providing 3D rendering. Furthermore, a head-mounted display (VR / AR) and a 3D display device according to the invention are also proposed. The head-mounted display (VR / AR) and the 3D display device each include one or more means for displaying 3D video (consisting of a set of n stereoscopic 2D image pairs) in monocular and / or binocular modes. The head-mounted display and the display device also each include:
[0192] - A means arranged and / or programmed and / or configured to implement the method according to the invention, and / or
[0193] - A communication device arranged and / or programmed and / or configured to and / or capable of communicating with an external device, said external device being arranged and / or programmed and / or configured to implement the method according to the invention, and / or
[0194] - At least one imaging device.
[0195] The method according to the invention includes the step of obtaining changes in the observer's viewpoint. According to a non-limiting embodiment, the viewpoint change depends on changes in the position and / or orientation and / or tilt of the observer, particularly the observer's head, in space.
[0196] According to one improvement, the viewpoint change depends on the observer's relative position and / or orientation and / or tilt. The step of obtaining the viewpoint change has the effect of obtaining a new viewpoint or a new position from which, if the observer directly observes Scene 1, they will see Scene 1.
[0197] For this purpose, the head-mounted display includes means arranged to detect movement and / or position and / or orientation and / or tilt of the head-mounted display. As a non-limiting example, the means arranged to detect movement and / or position and / or tilt and / or orientation may be a gyroscope or an accelerometer. The means arranged to detect movement and / or position and / or orientation and / or tilt may be an inertial measurement unit.
[0198] For this purpose, 3D display devices and optionally head-mounted displays in addition to inertial measurement units include means arranged to detect the movement and / or position and / or orientation and / or tilt of an observer relative to the 3D display device or head-mounted display. In this case, the viewpoint change is obtained from an image of the observer and / or an image of the observer's environment and / or an image of the 3D display device or head-mounted display. The viewpoint change may alternatively or in combination be obtained from:
[0199] - Data derived from an observer's image, such as images from at least one imaging system of a 3D display device or from at least one imaging system of a head-mounted display, said imaging system being arranged to image the observer's eye, and / or
[0200] - Data derived from the observer's environment, such as images derived from a 3D display device or from at least one imaging device arranged to image at least a portion of the environment, or images derived from a head-mounted display or from at least one imaging device arranged to image at least a portion of the environment.
[0201] Advantageously, the obtained viewpoint change is calculated and / or determined based on data from a device arranged to detect the movement and / or position and / or tilt and / or orientation of the head-mounted display and / or the observer relative to the 3D display device and / or the head-mounted display.
[0202] Preferably:
[0203] - Viewpoint changes include movement (du, dv), which is a lateral (du) (along the U-axis) and / or vertical (dv) (along the V-axis) movement of the observer's, preferably the observer's, head, i.e., within the observer's forehead plane; in some cases, this movement (du, dv) may be within or relative to the reference frame of the images in the source image pair, or within or relative to the reference frame of one image in the source image pair, and / or
[0204] - Viewpoint changes include lateral and / or vertical movement (du, dv) of the head-mounted display, or lateral and / or vertical movement (du, dv) of the observer relative to the 3D display device, i.e., within the observer's forehead plane, and / or
[0205] - Viewpoint changes include longitudinal movement (dz) of the observer's, preferably the observer's, head (along the Z-axis), i.e., in the observer's sagittal plane; in some cases, this movement (dz) may be perpendicular to the reference frame of the source image pair or perpendicular to the reference frame of one image in the source image pair, and / or
[0206] - Viewpoint changes include movement (dz) of a head-mounted display perpendicular to the plane formed by the axis (du, dv), or movement (dz) of a 3D display device or the display device of a 3D device perpendicular to the plane formed by the axis (du, dv) (i.e., along the axis connecting the observer to the 3D display device).
[0207] refer to Figure 3A and 3B This presents two illustrative examples of changes in the observer's viewpoint. Figure 3A and Figure 3B The left image in the image shows the observer's viewpoint before the viewpoint change, and Figure 3A and 3B The image on the right shows the change in viewpoint that an observer would see if they directly observed Scene 1. The observer's field of view 11 is also shown. Figure 3A In this context, the change in viewpoint corresponds to a movement dv parallel to the plane of Scene 1 and / or the 3D display device. If the observer actually observes Scene 1 as it is being imaged, this movement will correspond to a horizontal movement of the observer. In this case, the distance between the observer and Scene 1 and / or between the observer and the 3D display device remains unchanged. Figure 3B In this context, the viewpoint change corresponds to a movement dv parallel to the plane of Scene 1 and / or the 3D display device, and a movement dz perpendicular to the plane of Scene 1 and / or the 3D display device. The movement dz will correspond to a horizontal movement that moves the observer away from or closer to the actual observed scene 1 as it is being imaged.
[0208] The optical axis 6 of the observer before the viewpoint changes and the optical axis 7 of the observer after the viewpoint changes are depicted. Figure 3A The diagram illustrates the change in viewpoint during which the observer keeps their gaze focused on the same area, object, or group of objects in Scene 1 (referred to as the area of interest 12).
[0209] According to the invention, the method includes filtering data from a head-mounted display and a 3D display device to limit the magnitude of the obtained viewpoint change (particularly over time) to a value below a limit. For example, the limit could be the absolute value or norm of a movement vector determined based on data from the head-mounted display or the 3D display device. Preferably, the filtering is linear.
[0210] For this purpose, an effective viewpoint change can be defined, which corresponds to a viewpoint change measured or detected by a head-mounted display or 3D display device, or to a viewpoint change determined or calculated based on data from the head-mounted display or 3D display device. In other words, the effective viewpoint change corresponds to the effective viewpoint change made by the head-mounted display and the user or by the observer relative to the 3D display device or head-mounted display. In some cases, the obtained viewpoint change may correspond to or be equal to the effective viewpoint change. However, in most cases, the obtained viewpoint change will depend on or be calculated or determined based on the effective viewpoint change, but will not be equal to the effective viewpoint change.
[0211] The primary objective of the filtering stage is to truncate viewpoint changes with high amplitude. In other words, the filtering step aims to prevent sudden shifts or gradients in viewpoint change, such as when an observer wearing a head-mounted display suddenly turns or pivots 90°. In practice, since modifications to the source image pair (IS1, IS2) are based on viewpoint changes, exceeding a certain amplitude of the obtained viewpoint change will cause objects in the rendered image pair (IR1, IR2) to be excessively distorted and / or stretched, making the modified images (IR1, IR2) blurry or unobservable. While it is specified to add some pixels, pixel groups, or objects to and / or remove some pixels, pixel groups, or objects from the source image pair, this invention is not intended to generate data or information not present in the source image pair. In other words, the filtering step aims to truncate or ignore excessively large movements over a short period, so that the resulting viewpoint change does not deviate too far from the initial position and orientation of the observer's head while wearing a head-mounted display or observing a 3D display device.
[0212] The filtering step may also include the following steps: filtering or smoothing small variations (e.g., minor physiological movements of the head). This avoids the need to render frequent, slight variations in the source image pair (IS1, IS2) to the observer, variations that could cause motion sickness in observers wearing head-mounted displays or viewing the screen of 3D display devices. In other words, the filtering step can be defined as including applying a bandpass filter, particularly a high-pass filter; or subtracting an estimate of the average position of the head-mounted display or the average position of the observer relative to the 3D display device from the actually detected viewpoint changes. This smooths the actually detected viewpoint changes.
[0213] Variations in viewpoint (which can be described as high-amplitude viewpoint changes) are amplitude changes or variations that occur at low frequencies. High amplitude can be defined as an amplitude greater than 1 cm. Low-frequency amplitude changes or variations can be defined as having a frequency below 1 Hz. High-frequency variations in viewpoint can typically be observed as low-amplitude movements of the head-mounted display, usually less than 0.1 cm. High-frequency viewpoint changes can be defined as having a frequency greater than 10 Hz. The concepts of "low" and "high" amplitude changes or variations in frequency and amplitude are relative.
[0214] According to an embodiment, the method includes non-linearly limiting data from a head-mounted display or 3D display device to restrict the magnitude of the acquired viewpoint change to below a limit value. Limiting can be defined as applying a cut to the effective viewpoint change or data associated with the effective viewpoint change to limit or keep the magnitude of the acquired viewpoint change used to modify the source image pair (IS1, IS2) below a maximum magnitude. In practice, since the modifications made to the images of the source image pair (IS1, IS2) are based on viewpoint changes, exceeding a certain magnitude of the acquired viewpoint change will cause objects contained in the rendered image pair (IR1, IR2) to be excessively deformed and / or stretched, making the thus modified images (IR1, IR2) blurry or unobservable. The purpose of this step is to limit high values of the effective viewpoint change, keeping them below a predefined maximum value.
[0215] This method may include nonlinear dynamic compression of the gain applied to data from a head-mounted display or 3D display device to decay the amplitude variation or magnitude of the obtained viewpoint change over time, such that the amplitude does not exceed an amplitude variation threshold or magnitude threshold. This nonlinear dynamic gain compression of the effective viewpoint change complements the limiting of the effective viewpoint change. Nonlinear dynamic gain compression can be defined as modulating the gain applied to or related to the effective viewpoint change in order to limit the amplitude of the obtained viewpoint change used to modify the source image pair (IS1, IS2). In practice, in addition to truncating sudden large fluctuations in the effective viewpoint change, it may be meaningful to limit the amplitude of the viewpoint change to be rendered to restrict the modification of the source image pair (IS1, IS2) to below a predefined value, beyond which the rendered image pair (IR1, IR2) is no longer available.
[0216] Finally, the method includes applying nonlinear dynamic gain amplification to the data from the head-mounted display or 3D display device to attenuate or reduce the magnitude of the change in the amplified viewpoint over time. Preferably, this dynamic amplification step can be advantageously performed after a nonlinear dynamic gain compression step of the data applied to the head-mounted display or 3D display device.
[0217] When the direction of movement of the head-mounted display or the user relative to the 3D display device changes, particularly in the opposite direction, nonlinear dynamic amplification of the data applied to the head-mounted display or the 3D display device can be advantageously implemented. Nonlinear dynamic amplification can be defined as modulation of the gain applied to the data related to the effective viewpoint change. In other words, when the previous movement (or the magnitude of the previous movement) of the head-mounted display or the user relative to the 3D display device has made it necessary to nonlinearly dynamic compress the gain applied to the data from the head-mounted display or the 3D display device, nonlinear dynamic amplification of the gain applied to the data can be advantageously implemented. Therefore, in this case, when the movement of the head-mounted display or the user relative to the 3D display device is in the opposite direction, the gain applied to the effective viewpoint change can be increased to restore the user's perception of rapid movement.
[0218] The method further includes a modification step, which involves modifying each of the two images in at least a portion of a set of n source image pairs (IS1, IS2) n to render at least one stereoscopic 2D image pair (IR1, IR2) of the scene, referred to as at least one rendered image pair (IR1, IR2). The rendered image pair (IR1, IR2) is designed to provide 3D rendering when displayed in a head-mounted display or a 3D display device and to provide viewpoint changes relative to a 3D video of the scene. Furthermore, each rendered image pair (IR1, IR2) provides viewpoint changes relative to a corresponding source image pair (IS1, IS2) (the two images (IS1, IS2) of which have been modified according to the method to render the two images (IR1, IR2)).
[0219] The method further includes the steps of: displaying at least one rendered image pair (IR1, IR2) on a head-mounted display or a 3D display device when a viewpoint change is obtained; or displaying at least one source image pair (IS1, IS2) on a head-mounted display or a 3D display device when no viewpoint change is obtained. In practice, at certain moments during a 3D video, there may be no viewpoint change for some images of all n image pairs in the 3D video (e.g., if the observer remains stationary at a reference position for a period of time while watching the video).
[0220] Preferably, the source image pair (IS1, IS2) and the rendered image pair (IR1, IR2) can:
[0221] - Display simultaneously with monocular or binocular vision, or
[0222] - Displayed continuously, alternately, or iteratively using monocular vision.
[0223] The expression "continuous display of image pairs or a stream of image pairs" can be understood as meaning: displaying image pairs of a scene continuously, alternately, sequentially, or cyclically. It can also be understood as: time-division multiplexing of two images within an image pair.
[0224] The phrase "simultaneously displaying an image pair" can be understood as meaning: superimposing two images from an image pair, where each image in the displayed pair has discrete polarization. Alternatively, it can be understood as spatial multiplexing of the two images in an image pair, i.e., displaying a single composite image or raster image that includes a portion of each of the two images in the displayed image pair.
[0225] The two images in an image pair, or the stream of images that form or constitute a 3D video, can be displayed simultaneously with binocular vision, with one image displayed for viewing by the right eye and the other for viewing by the left eye.
[0226] For the same reason, the modification step of the method can be implemented only on a portion of the set of n source image pairs (IS1, IS2)n. In fact, if no viewpoint change is obtained, the modification step is not implemented, and therefore the source image pair (IS1, IS2) is displayed without any viewpoint change.
[0227] Advantageously, once the viewpoint change has been obtained, the source image pair (IS1, IS2) is modified according to the method to render the viewpoint change of the scene by displaying a rendered image pair (IR1, IR2) in a head-mounted display or on a 3D display device that provides a view of the scene from the observer's new viewpoint.
[0228] The modification steps include moving and / or expanding and / or shrinking pixels or one or more groups of pixels within the images of at least one source image pair (IS1, IS2) and / or adding pixels or groups of pixels to the images of at least one source image pair (IS1, IS2) and / or removing pixels or groups of pixels from the images of at least one source image pair (IS1, IS2).
[0229] The modification steps are as follows:
[0230] • Depth map of the imaged scene,
[0231] • The resulting viewpoint changes.
[0232] For a simple implementation of the present invention, the modification of at least one source image pair (IS1, IS2) can be proportional to the change of viewpoint.
[0233] According to the present invention, the modification is performed, among other things, through the parallax effect.
[0234] For example, the modification is implemented via a matrix transformation operator (called a transformation operator). Advantageously, the transformation operator is arranged and / or configured to apply a parallax effect to the image modified during the modification step.
[0235] The two images modified in this way constitute a pair of rendered images (IR1, IR2) that provide a change in viewpoint.
[0236] Specifically, the modification steps include moving and / or shrinking by means of any function, operator, matrix, or grid (called a transformation operator), or may include or contain the coordinates or positions of all or some pixels or groups of pixels within a pixel grid or matrix of the image to be modified, moved (e.g., by translation).
[0237] In some cases, the modification steps may include expanding and / or shrinking and / or weighting the movement of a portion of the coordinates or positions of pixels or groups of pixels within a pixel grid or matrix of the image to be modified, using a transformation operator. Specifically, the expansion and / or shrinking and / or weighting of the movement may be applied to pixels or groups of pixels where the values or local variances of parameters or channels include, but are not limited to, sharpness, depth, or any other parameters known to those skilled in the art, such as, as a non-limiting example, R, G, B parameters (R (red), G (green), and B (blue)), dilation parameters, and / or contour parameters. Furthermore, in these cases, the modification steps may include removing and / or generating pixels, particularly pixels near the edges of the image to be modified, using a transformation operator. Pixels to be generated may be generated, for example, by transposing or expanding and / or shrinking adjacent or neighboring pixels or groups of pixels.
[0238] The transformation operators for the modification steps described above may include all the features described below, may be supplemented by all the features described below, or may implement all the features described below, and the modification steps are implemented based on all the features described below.
[0239] Furthermore, the modification steps may involve pixel interpolation to, for example, add missing pixels (especially during significant viewpoint changes). Those skilled in the art are familiar with pixel interpolation techniques, such as linear interpolation (based on the respective distances to adjacent pixels used as interpolation reference points) or interpolation based on spline functions.
[0240] Advantageously, refer to Figures 5 to 7 The method includes calculating at least one angle corresponding to a change in viewpoint. The at least one angle corresponds to the angle formed between the observer's optical axis 6 before the viewpoint change and the optical axis 7 after the viewpoint change. The source image pairs 21 and 22 are modified based on the at least one calculated angle.
[0241] This method may include calculating at least one angle, called angle alpha (α), which has the U-axis as its axis of rotation. In other words, angle α defines rotation about the U-axis. According to this embodiment, the U-axis is a vertical axis. According to this embodiment, the U-axis is parallel to an axis as seen by the observer in scene 1 or contained within scene 1.
[0242] This method may include calculating at least one angle, called angle beta(β), which has the V-axis as its axis of rotation. In other words, angle β defines rotation about the V-axis. The V-axis is perpendicular to the U-axis. According to this embodiment, the V-axis is a horizontal axis. According to this embodiment, the V-axis is parallel to an axis as seen by the observer in scene 1 or contained within scene 1.
[0243] This method may include calculating at least one angle, called angle gamma (γ), which has the Z-axis as its axis of rotation. In other words, angle γ defines rotation about the Z-axis. According to this embodiment, the Z-axis is a horizontal axis. According to this embodiment, the Z-axis is perpendicular to the axis as seen by the observer in scene 1 or contained within scene 1.
[0244] Therefore, the method may include calculating angle α, angle β and / or angle γ.
[0245] Angles α, β, and γ can be defined relative to a reference frame as seen by the observer as scene 1; or relative to a reference frame associated with or corresponding to scene 1 as seen by the observer; and / or relative to a reference frame of the user and / or the head-mounted display and / or the 3D display device.
[0246] In most cases, each of the two images in the considered modified source image pair (IS1, IS2) is modified identically during the modification step. In some specific cases, the two images in the considered modified source image pair (IS1, IS2) are modified differently during the modification step.
[0247] In practice, in most cases, the resulting viewpoint change will reflect the same movement and / or translation and / or rotation in each of the observer's two eyes. In this case, the modification applied to each of the two images in the considered source image pair (IS1, IS2) is the same. However, in some specific cases, the viewpoint change of one of the observer's eyes differs from that of the other eye, for example, in the case of rotation around the optical axis of one of the observer's eyes.
[0248] In one particular case, according to this embodiment, the method may further include calculating an angle called gamma (γ), which corresponds to a change in viewpoint or a component of the change in viewpoint achieved by rotation (by precession) around the optical axis of the observer (the region of interest 12 preferably intersects the optical axis of the observer).
[0249] In a specific case where the viewpoint change, or a component of the viewpoint change, includes the rotation angle γ, the viewpoint change of one eye of the observer differs from that of the other eye. In this case, the angle γ is applied to the considered source image pair i(IS1, IS2). i The modifications to the two images can be different, or the modifications applied according to the angle γ can be applied only to the considered source image pair i (IS1, IS2). i One of the images.
[0250] Furthermore, the viewpoint change may optionally include translation, referred to as longitudinal translation, denoted as dz, in combination with translation du or dv or du+dv, or in combination with angles α, β, and γ. This viewpoint change may be equivalent to, or related to, the movement of a person directly observing scene 1 along or according to their optical axis. Preferably, the person (observer) keeps their gaze fixed on the region of interest 12 in the direction connecting the observer and scene 1 (the observer moves toward scene 1) or in the opposite direction (the observer moves away from scene 1). In this case, the modification steps may, for example, include, a homothetic transformation, or consist of, for example, a homothetic transformation.
[0251] According to an advantageous embodiment, when modifying the image of at least one source image pair (IS1, IS2), the step of modifying the image of at least one source image pair (IS1, IS2) is further performed relative to, or based on, a region 12 (referred to as the region of interest 12) of the image of said at least one source image pair (IS1, IS2). The region of interest 12 corresponds to a region of the image ((IR1, IR2) or (IS1, IS2), as appropriate) or an object in scene 1 where the observer's gaze is focused. Furthermore, the region of interest 12 is a single region of the image of the scene, particularly the image ((IR1, IR2) or (IS1, IS2)) or an object in scene 1, i.e., a single region of two images ((IR1, IR2) or (IS1, IS2)) of a 3D video at a given time. The region of interest 12 specifically corresponds to one or more objects in the scene, particularly one or more objects of the same depth in the displayed image of the scene.
[0252] Modifying the images of at least one source image pair (IS1, IS2) according to the region of interest 12 improves the visual experience for viewers of 3D videos. Furthermore, modifying the images of at least one source image pair (IS1, IS2) according to the region of interest 12 increases the immersion for viewers of 3D videos. Additionally, modifying the images of at least one source image pair (IS1, IS2) according to the region of interest 12 limits the resources used to process the stream of the source image pair (IS1, IS2).
[0253] Preferably, the region of interest 12 is obtained from an image of the observer's eye.
[0254] According to a non-limiting embodiment, the region of interest 12 remains unchanged during the step of modifying the images of at least one source image pair (IS1, IS2).
[0255] Furthermore, pixel movement and / or expansion and / or contraction are performed through a parallax effect relative to the region of interest 12, which remains unchanged during the modification step. In other words, the transformation operator is arranged to move the coordinates or positions of pixels relative to the pixels in the region of interest 12 via the parallax effect.
[0256] Where appropriate, when considering the region of interest 12 during the modification step, the modification step may include removing and / or generating pixels, particularly pixels near the region of interest, using transform operators. Pixels to be generated may be generated, for example, by transposing or expanding and / or shrinking adjacent or neighboring pixels or groups of pixels. However, pixels located at the contour of the region of interest 12 are only slightly moved by the parallax effect implemented relative to the region of interest 12.
[0257] Considering the region of interest 12 during the modification step preferably involves modifying the source image pair (IS1, IS2) once a change in the region of interest 12 is detected or obtained, but this has several advantages. In practice, observers watching the video will focus their attention on different areas of scene 1 or different objects in scene 1 over time. Furthermore, modifying the image overall may reduce immersion because the objects in scene 1 that the observer's gaze is focused on while watching the 3D video are modified.
[0258] Advantageously, during the modification step, the object is moved proportionally to a depth deviation ΔP, preferably translated, which corresponds to the difference between the depth of the object (located outside the region of interest 12) in the considered scene 1 and the depth of the region of interest 12.
[0259] The region of interest 12 can be derived from data related to the observer's gaze. Data related to the user's gaze can be obtained through eye tracking. Therefore, the method may include the steps of acquiring the region of interest 12 and / or determining the region of interest 12 based on that data. In this case, the data related to the user's gaze can be acquired by one or more imaging systems of a head-mounted display or 3D display device arranged to image the eyes of an observer wearing the head-mounted display. Those skilled in the art are familiar with eye-tracking technology and with data that can be derived from it, such as foveal paths. The method may include the step of acquiring data related to the observer's gaze, for example, using one or more optical systems that may include one or more cameras.
[0260] refer to Figure 4A , Figure 4B , Figure 4C and Figure 4D An advantageous embodiment of the method is shown, wherein the modification step is implemented based on the depth map of the imaged scene, the obtained viewpoint changes, and the region of interest 12. Figure 4AThe pair of source images (IS1, IS2) under consideration (denoted as i) are shown. i The left image IS1i; Figure 4B The pair of source images (IS1, IS2) under consideration are shown. i The right image IS2i. Figure 4C The source images (IS1, IS2) of the pair i under consideration are shown. i A pair of related rendered images (IR1, IR2) (denoted as i). i The left image IR1i, and Figure 4D The source images (IS1, IS2) for the considered pair i are shown. i The associated rendered images (IR1, IR2) for pair i. i The left image IR1i.
[0261] According to this non-limiting embodiment, the observer maintains focus on the same region of interest 12 during viewpoint changes. This embodiment makes it possible to account for cases where the region of interest 12 that the considered source image pair (IS1, IS2)i remains focused on during viewpoint changes.
[0262] Those skilled in the art will be able to adapt and transfer the description of this non-limiting embodiment to situations where an observer changes their region of interest 12 over time, for example, between modifying the source image (IS1, IS2)i of the considered pair i (at time t) and modifying the source image (IS1, IS2)i of the pair i+1 (at time t+1).
[0263] In this embodiment, modifications to the source images (IS1, IS2)i of the considered pair are also based on region 12. According to this embodiment, region 12 can remain unchanged during modifications to the source images (IS1, IS2)i of the considered pair by anchoring region 12 relative to the observer or relative to a reference frame other than the observer or located outside the observer.
[0264] For ease of description, Figures 5 to 7 Shown in top perspective view Figure 3A The imaged scene 1 (IS1, IS2)i depicted in 3D is a scene 1 that includes the concept of depth. The observer's optical axis 6 before the viewpoint change and the observer's optical axis 7 after the viewpoint change are shown. In the presented scene, a given viewpoint change of the observer corresponds to a movement of the plane parallel to the image of the displayed source image (IS1, IS2)i of the considered pair i. If the observer actually observes the imaged scene 1, this movement will correspond to a horizontal movement of the observer.
[0265] During changes in viewpoint, the observer's gaze remains focused on area 12, for Figure 6 In other words, it is the tree 8 located behind individual 9 but in front of the landscape 10 in the background. During changes in viewpoint, the observer's gaze remains focused on the area 12. Figure 5 The center is background 10, that is, the landscape within background 10. During changes in viewpoint, the observer's gaze remains focused on area 12. Figure 7 The middle is prospect 9, which is individual 9. Figure 4A and 4B The modifications to the source image pair (IS1, IS2) shown in the figure correspond to... Figure 6 The embodiment shown illustrates this when the observer's gaze (i.e., the area of focus 12) remains focused on tree 8 during a change in viewpoint.
[0266] Advantageously, the modification of the source image pair (IS1, IS2) includes: for each pixel or one or more groups of pixels in each image of the source image pair (IS1, IS2), calculating a deviation ΔP between the inverse of the depth associated with the region of interest 12 and the inverse of the depth associated with each pixel or one or more groups of pixels in each image of the source image pair (IS1, IS2) located outside the region of interest 12. This deviation ΔP is calculated based on data from the depth map. The source image pair (IS1, IS2) is modified according to the calculated depth deviation ΔP.
[0267] Advantageously, according to the described non-limiting embodiment, the translation of a given object in the source image pair (IS1, IS2) other than the object in the region of interest 12 will be proportional to the angle α.
[0268] Figures 5 to 7 This shows the modification of Scene 1 as observed by the observer after a change in viewpoint.
[0269] The translation will be performed along the axis connecting the optical axis 6 of the observer before the viewpoint change and the optical axis 7 after the viewpoint change. Since the object is translated in the plane of the image to be modified, a trigonometric relationship can be established between the calculated depth deviation ΔP, angle α, angle β and / or angle γ and the distance (denoted in m) that the object must move to render the viewpoint change.
[0270] For example, refer to Figure 6 and Figures 4A to 4D For individual 9 in the foreground of the imaged scene 1, the depth deviation ΔP is equal to P1 - P0, or equal to the absolute value of P1 - P0, where P0 corresponds to the depth of 9 in scene 1, and P1 corresponds to the depth of regions of interest 8 and 12 in scene 1. In the case of the background 10 of the image, the depth deviation ΔP is equal to P1 - P2, or equal to the absolute value of P1 - P2, where P2 corresponds to the depth of the background 10 in scene 1.
[0271] Therefore, as a non-restrictive example, a relationship can be established. .
[0272] Furthermore, the method may include calculating the distance change Δd of the observer relative to the imaged scene 1. Modifications to the images of at least one source image pair (IS1, IS2) will advantageously also be performed based on the calculated distance change Δd. The distance change may be associated with a viewpoint change. In practice, in most cases, this distance change will be negligible compared to the depth of the objects in the imaged scene 1. However, where the change is not negligible relative to the depth of the objects in the imaged scene 1, as a non-limiting example, the relationship can be... Modifications to be applied to at least one source image pair (IS1, IS2).
[0273] It is important to emphasize that the value “m” and the relationship described above correspond to the physical modifications in Scene 1 (i.e., the actual modifications caused by the change of viewpoint or the modification of the scale of Scene 1), as would be perceived by the observer of Scene 1.
[0274] It should be noted that the relationships described in this application are provided for informational purposes only. Those skilled in the art will be able to derive any matrix operator based on simple mathematical tools that allow the calculation of the movement (expansion and / or contraction and / or shift) of each pixel or group of pixels in the image of the source image pair (IS1, IS2) (e.g., trigonometry and basic geometry (Thales' theorem, etc.)).
[0275] The remainder of the specification describes specific, non-limiting examples of the steps to modify images IS1 and IS2. This description relates to the modification of images IS1 and IS2 of scene 1 (i.e., modifications to pixels within the image (or within the image grid itself) caused by a change in viewpoint), as if they were to be rendered to an observer via a display. In this embodiment, the movement of one or more pixels or groups of pixels described is inversely proportional to depth. The modification (movement) of a portion of a pixel located at a given depth Z, denoted as dpi, can be expressed as:
[0276] dpi = F*((1 / ZR) – (1 / Z))*TT, Equation 1
[0277] Where F is the so-called focal length of the sensor, or more precisely, the distance between the optical center of the lens used to acquire the source image (IS1, IS2) of the considered pair i and the sensor; TT is the so-called lateral component of the viewpoint translation (TT can be equal to du, dv, or du+dv, or correspond to du, dv, or du+dv); and ZR is the depth of the region of interest 12 or the depth associated with the region of interest 12.
[0278] TT can be defined as the direction perpendicular to the central axis of the imaging camera module, or as the translation in a plane perpendicular to the observer's optical axis (or more precisely, parallel to the plane of the rendered image), and dpi is the movement of pixels in the same direction as the translation TT.
[0279] As a non-limiting example, in the case of a viewpoint change due to a longitudinal translation dz, the modification of the coordinates (U1, V1) of the considered pixel (or group of pixels) in each of the source image pair (IS1, IS2) can be defined as follows: U2 = (Z / (Z-TL))*U1 and V2 = (Z / (Z-TL))*V1, where Z is the depth of the considered point or object in the scene (or the depth associated with the pixel or group of pixels corresponding to the considered object in the scene), TL is the longitudinal translation (dz), and (U2, V2) are the coordinates of the considered pixel after the movement occurs following the implementation of this method. Note that when Z is large (the point or object in scene 1 has a large depth), the shift is small. Furthermore, objects located at infinity (in practice, objects located at depths greater than a threshold, such as objects located 50 meters or more away from the camera acquiring the source image, can be considered to be at infinity) will not move, and the similarity ratio will be equal to 1.
[0280] In other words, the modification made to the source images IS1 and IS2 by shrinking corresponds to a homothetic transformation with a ratio less than 1. Conversely, expanding corresponds to a homothetic transformation with a ratio greater than 1.
[0281] As a non-limiting example, in the case of viewpoint changes caused by lateral translation du or dv or du+dv, the modification of the coordinates (U1, V1) of a given pixel (or group of pixels) in each of the source image pair (IS1, IS2) can be defined as: U2 = U1 + ((1 / (ZR)) – (1 / Z))*F*TTU and V2 = V1 + ((1 / (ZR)) – (1 / Z))*F*TTV, where TTU is the lateral (or vertical) translation component along the U-axis and TTV is the lateral (or vertical) translation component along the V-axis.
[0282] Preferably, in this case, the units of U, V, TTU, TTV, and F are preferably consistent; they can be physical distances (for U, V, and focal length F, physical distances or distances expressed in units associated with the camera sensor; for TTU and TTV, and depths Z and ZR, physical distances of movement). Alternatively, TTU, TTV, ZR, and Z are consistent (distances expressed in physical distances or units associated with the camera sensor), and U, V, and F use the same unit (e.g., pixels, where F is in pixels / radians, where radians are dimensionless), which is different from the units of TTU, TTV, ZR, and Z.
[0283] In practice, viewpoint changes can include a vertical translation component TT and a longitudinal translation component TL. Furthermore, for viewpoint changes that include multiple translation components, the modification of the coordinates (U1, V1) of a given pixel (or pixel group) of each of the source image pair (IS1, IS2) can be defined as: U2 = (U1 + F*((1 / ZR) – (1 / Z))*TTU)*(1 / (1 - TL / Z)) and V2 = (V1 + F*((1 / ZR) – (1 / Z))*TTV)*(1 / (1 - TL / Z)).
[0284] When the viewpoint change includes very large rotation angles (α, β, and / or γ) that must be considered when modifying the source images IS1 and IS2, the modification of the coordinates (U1, V1) of a given pixel (or group of pixels) applied to each of the source image pair (IS1, IS2) can be defined as follows:
[0285] U2 = [(U1 + F*((1 / ZR) – (1 / Z))*TTU)*(1 / (1-TL / Z))]*cos(γ) – [(V1 + F*((1 / ZR) – (1 / Z))*TTV)*(1 / (1-TL / Z)]*sin(γ), and
[0286] V2 = [(U1 + F*((1 / ZR) – (1 / Z))*TTU)*(1 / (1-TL / Z))]*sin(γ) + [(V1 + F*((1 / ZR) – (1 / Z))*TTV)*(1 / (1-TL / Z)]*cos(γ).
[0287] It should be noted that in specific cases where the viewpoint change includes a rotation component with angle γ, the source images (IS1, IS2) of the considered pair i can be used. i The image is applied to one of the source images (IS1, IS2) and applied to the considered pair of i.i Another image with different modifications.
[0288] In most cases, the viewpoint change of an observer viewing a head-mounted display or a 3D display is small enough to keep the entire area of interest 12 constant. Furthermore, processing of the data from the head-mounted display (filtering, clipping, and / or compression) enables viewpoint changes to be kept within tolerable limits, or at least helps to keep viewpoint changes within tolerable limits, thereby keeping the area of interest 12 constant.
[0289] In certain specific cases involving significant viewpoint changes, particularly those involving translation dz and / or significant rotation (particularly according to angles α, β, and γ), the source image pair (IS1, IS2), including the region of interest 12, can also be modified by expanding or dilating or stretching and / or contracting or compressing. In cases of significant viewpoint changes caused by translation dz, the modification steps may include modifying the region of interest 12, for example, by a homothetic transformation. In cases of viewpoint changes involving significant rotation (α, β, γ), the modification steps may include, for example, stretching and / or compressing the region of interest along one or both axes. However, even in rare cases where modification of the region of interest 12 is necessary, the modification applied to the region of interest 12 may differ from, or be limited to, the modification applied to the remainder of the source images (IS1, IS2).
[0290] refer to Figures 4A to 4D and Figure 6 In scenes 9 and 10, all objects except those focused on by the observer's gaze (trees 8 and 12) will be translated in the plane of the image in response to the obtained viewpoint change. Furthermore, some objects or portions of objects corresponding to a few pixels at the contours of regions of interest 8 and 12 (i.e., pixels immediately adjacent to regions of interest 8 and 12) may be occluded by the object focused on by the observer's gaze, and some objects or portions of objects corresponding to a few pixels at the contours of regions of interest 8 and 12 that were occluded before the viewpoint change can be generated in the rendered image pair (IR1, IR2) according to this method.
[0291] according to Figures 4A to 4D and Figure 6 The specific embodiment shown corresponds to the case where the region of interest 12 is located between the background and the foreground of the imaged scene 1 (in other words, the region of interest 12 does not correspond to an object with the maximum depth or is not an object with the maximum depth, and does not correspond to an object with the minimum depth or is not an object with the minimum depth). The modification steps are implemented according to a non-limiting embodiment:
[0292] - For objects in Scene 1 whose depth differs from the depths of interest regions 8 and 12 (individual 9, tree, and landscape 10 and 12), the depth of these objects is inversely proportional to the depth of the objects under consideration in Scene 1, and is inversely shifted relative to the depth field of interest regions 8 and 12.
[0293] - For objects whose depth differs only slightly from the depths of interest regions 8 and 12, as a first-order approximation, these objects can be shifted proportionally to the depth difference between the object under consideration and the depths of interest regions 8 and 12.
[0294] - For objects farther away than the depth of the region of interest (8, 12), these objects are shifted along the same direction and direction as the viewpoint change, and...
[0295] - For nearby objects with a depth less than the depth of the region of interest 8 or 12, these objects are shifted in the same direction as the viewpoint change and in the opposite direction to the viewpoint change.
[0296] according to Figure 5 The specific embodiment shown corresponds to the case where the region of interest 12 is located in the background of the imaged scene 1 (in other words, the region of interest 12 corresponds to an object in the background and / or an object with the maximum depth, or an object in the background and / or an object with the maximum depth), and the object is shifted inversely to the depth of the object under consideration in scene 1.
[0297] according to Figure 7 The specific embodiment shown corresponds to the case where the region of interest 12 is located in the foreground of the imaged scene 1 (in other words, the region of interest 12 corresponds to an object in the foreground and / or an object with the minimum depth, or an object in the foreground and / or an object with the minimum depth), and the object is shifted inversely to the depth of the object under consideration in scene 1. In this disclosure, proportional (or inversely proportional) is understood to mean a proportional relationship of the phase difference constant (in particular the depth value of the region of interest 12), i.e., an affine relationship.
[0298] In almost all 3D videos, each source image pair (IS1, IS2) includes a left image IS1 and a right image IS2. However, in some cases, the two images of the source image pair (IS1, IS2) are not allowed to be viewed binocularly (right eye sees right image, left eye sees left image); in other words, they do not constitute a right image and a left image. In fact, as mentioned above, the method according to the invention is applicable to any type of 3D video, as long as it includes two 2D image streams of the same scene capable of providing 3D rendering. Therefore, in some cases, such as when two videos are simultaneously acquired by two camera sensors of the same connected device (e.g., a smartphone or tablet) capturing the same scene, the source image pair (IS1, IS2) does not include a left image and a right image, but instead includes, for example, an upper image and a lower image. Although this type of video can be displayed with monocular 3D vision, in order to be able to display the 3D video with binocular vision, it is necessary to consider the distance between the two discrete cameras that acquired the source image pair (IS1, IS2), as well as the positions of the two discrete cameras relative to each other and / or relative to scene 1, during the step of modifying the images of at least one source image pair (IS1, IS2).
[0299] In this way, each rendered image pair (IR1, IR2) consists of image IR2 (referred to as the right image IR2, as it will be perceived by the right eye of the observer of scene 1) and image IR1 (referred to as the left image IR1, as it will be perceived by the left eye of the observer of scene 1).
[0300] The entire description relating to the modification steps can be applied to modifications of the distance between the two discrete cameras acquiring the source image pair (IS1, IS2) relative to each other, and / or the positions of the two discrete cameras relative to each other and / or relative to scene 1. Therefore, according to a non-limiting embodiment, the modification steps can also be implemented based on the distance between the two discrete cameras acquiring the source image pair (IS1, IS2), the positions of the two discrete cameras relative to each other, and / or the positions of the two discrete cameras relative to scene 1.
[0301] In some cases, the source image pair (IS1, IS2) may not provide optimal 3D rendering. This may occur, for example, when the two discrete imaging devices acquiring the source image pair (IS1, IS2) are too far apart from each other. In this case, according to a non-limiting embodiment, a modification step can also be implemented based on the interpupillary distance.
[0302] Those skilled in the art are familiar with interpupillary distance (IPD). An average IPD can be used, such as one corresponding to the average IPD of a user or, for example, to the IPD of a head-mounted display and / or synchronized, polarized, or red / blue filter glasses. As a non-limiting example, IPD can be considered to be between 45 mm and 75 mm.
[0303] More preferably, in order to provide the observer with optimal 3D rendering, it may be preferable to combine consideration of interpupillary distance with data regarding the relative positions of the two discrete cameras acquiring the source image pair (IS1, IS2). Thus, advantageously, in this case, the step of modifying the source image pair (IS1, IS2) also depends on the interpupillary distance and the distance between the two discrete cameras acquiring the source image pair (IS1, IS2), the positions of the two discrete cameras relative to each other, and / or their positions relative to scene 1.
[0304] Specifically, this can be implemented by further modifying the images IS1, IS2 of at least one source image pair (IS1, IS2), preferably relative to each other. The step of modifying the images IS1, IS2 of at least one source image pair (IS1, IS2) relative to each other is performed by moving and / or expanding and / or shrinking pixels or one or more pixel groups of at least one source image pair (IS1, IS2) of scene 1, and / or by adding pixels or pixel groups to the images of at least one source image pair and / or removing pixels or pixel groups from the images of at least one source image pair, such that the viewpoint difference between the two images IR1, IR2 of at least one rendered image pair (IR1, IR2) corresponds to the interpupillary distance.
[0305] In this case, the modifications applied to the source image pair (IS1, IS2) can be different for each image in the pair. For example, when the two images of the source image pair (IS1, IS2) originate from two cameras with an interval distance different from the pupil distance and / or when the two cameras are not aligned along the basic horizontal axis, the interval between the observation points can be readjusted.
[0306] When the two images of the source image pair (IS1, IS2) originate from two cameras with an interval (e.g. along the V-axis, such as the observer's eye) that is different from the pupillary distance, a translation of TTV = (DPU - IC) can be applied to the right image (IS2), where IC is the distance between the two cameras and DPU is the pupillary distance.
[0307] This can then be combined with other viewpoint changes (which, if needed, can be algebraically represented as a common transformation so that only one transformation is applied to each image).
[0308] When the two images of a source image pair (IS1, IS2) originate from two cameras that are not aligned along the basic horizontal axis (along the V-axis), the image of the source image pair (IS1, IS2) (e.g., the left image (IS1)) can be modified to be corrected by pure rotation γ.
[0309] When the two images of the source image pair (IS1, IS2) originate from two cameras with an interpupillary distance different from their interpupillary distance, and when the two cameras are not aligned along the basic horizontal axis, a modification by translation (TTU and / or TTV) should be applied to the right image of the source image pair (IS1, IS2), followed by a modification by rotation angle γ. If the interpupillary distance DPU between the observer's eyes is initially oriented along the V-axis and rotated perpendicular to the observation axis by angle γ, then the translation values are TTV = DPU*(1 - cos(γ)) and TTU = DPU * sin(γ).
[0310] In this application, it is believed that those skilled in the art are familiar with the concept of depth maps and methods for obtaining depth maps.
[0311] As a non-limiting example, the method may include the step of generating a depth map of the imaged scene based on one or more images of the source image pair (IS1, IS2) and / or based on the sharpness information of one or more images of the source image pair (IS1, IS2) and / or based on the local variance.
[0312] Those skilled in the art will know how to select the most appropriate sharpness information for a given situation. Sharpness information contained in a pixel or pixel block may include the entropy, energy, and / or variance of image parameters between adjacent pixels.
[0313] Preferably, any energy or entropy estimator conventionally used for focus detection can be used to determine the variance. Various techniques for calculating local variance are known to those skilled in the art. As a non-limiting example, the local variance can be the variance of pixel intensities (e.g., the average intensity of the R, G, and B channels), calculated over a given set of pixel matrices and adjusted according to the image resolution and / or the processing power / power of the processing unit. Then, for each pixel, group of pixels, or object in the image from which the depth map is to be generated, the scene image with the highest variance is selected.
[0314] A depth map can correspond to an image that includes a single depth channel. Depth is understood to refer to the physical distance between a point or physical object in scene 1 (corresponding to a pixel or group of pixels in the image under consideration) and the sensor that acquired the image. For illustration, refer to Figures 4 to... Figure 7 The simplified schematic depth map associated with the image of scene 1 illustrated in the example is shown in... Figure 8 The image is depicted in the image. In this depth map, the white area 4 corresponds to an object located, for example, five meters away from the camera's optical sensor, and the gray area 5 in the image background corresponds to an object located, for example, more than fifty meters away from the optical sensor. The distance of five meters corresponds, for example, to the minimum focusing distance of the camera lens, and the distance of fifty meters corresponds, for example, to the maximum focusing distance of the camera lens.
[0315] A disparity map of the imaged scene 1 can also be generated. A depth map can be calculated and / or determined based on the disparity map. In this way, any feature related to the depth map can be transferred to the disparity map, and vice versa. A disparity map can be defined as a graph or image containing depth information. In other words, a disparity map can be an image or graph including at least one depth channel of the imaged scene. Preferably, the depth information constitutes the depth field of the imaged scene.
[0316] Furthermore, a pixel group or pixel block corresponding to an object in the imaged scene 1 will have different depth information than one or more other pixel groups or pixel blocks corresponding to one or more other objects in the imaged scene 1. Therefore, each image acquired by the same moving imaging system can be enhanced with depth field, meaning that each pixel or pixel group or pixel block corresponding to an object in the scene (representing the pixel matrix or pixel table of the scene image) will have different depth information than that corresponding to the distance between the object and the optical sensor; in most cases, the depth of various objects in the imaged scene is different for at least some objects in the scene.
[0317] According to the present invention, a VR / AR head-mounted display and a 3D display device are also provided. The head-mounted display includes a monocular image display device or a binocular image display device arranged to provide 3D rendering from two stereoscopic 2D images. The 3D display device includes an image display device arranged to provide 3D rendering from two stereoscopic 2D images.
[0318] During the optional display step, the rendered image pair (IR1, IR2) can be displayed in a binocular head-mounted display for binocular vision. One of the two images in the rendered image pair (IR1, IR2), image IR1, is displayed in monocular vision to the left eye of the observer wearing the head-mounted display by the display device of the head-mounted display, and the other image in the rendered image pair (IR1, IR2), image IR2, is displayed in monocular vision to the right eye of the observer wearing the head-mounted display by the display device of the head-mounted display.
[0319] During the optional display step, the rendered image pair (IR1, IR2) can be displayed monocularly on a head-mounted display or a 3D display device. The two images of the rendered image pair (IR1, IR2) are displayed simultaneously or alternately on the display device of the 3D display or head-mounted display, such that one image IR1 or a portion of one image IR1 of the two images of the rendered image pair (IR1, IR2) is viewed only by one eye of the observer, and the other image IR2 or a portion of the other image IR2 of the two images of the rendered image pair (IR1, IR2) is viewed only by the other eye of the observer (through an additional viewing device, e.g., glasses, where appropriate). As described in detail above, those skilled in the art will be able to grasp the concepts of spatial and temporal multiplexing displays and polarized image displays, among others.
[0320] Those skilled in the art will be able to grasp the concepts and implementations of stereoscopic vision, as well as monocular and binocular vision. In particular, those skilled in the art will observe that the two images of a rendered image pair correspond to two different viewpoints of scene 1, enabling an observer wearing a head-mounted display or viewing a 3D display device to observe a perspective or three-dimensional view of the scene.
[0321] Advantageously, the head-mounted display and 3D display device include means arranged and / or programmed and / or configured to implement the method according to the invention. In practice, the means arranged and / or programmed and / or configured to implement the method according to the invention is a processing unit. In this case, the stream of the source image pair (IS1, IS2)n and / or the stream of the rendered image pair (IR1, IR2)n of scene 1 can be loaded onto the storage device of the head-mounted display or 3D display device. Therefore, the method will be autonomously implemented by the head-mounted display and 3D display device.
[0322] However, head-mounted displays and 3D display devices may include communication devices arranged and / or programmed and / or configured to communicate with external or remote devices. In this case, the stream of the source image pair (IS1, IS2)n and / or the stream of the rendered image pair (IR1, IR2)n of Scene 1 can preferably be transmitted in real time from the external device to the head-mounted display and from the 3D display device.
[0323] Alternatively, external devices may be arranged and / or programmed and / or configured to implement the method according to the invention. In this case, the head-mounted display and 3D display device may not implement the method according to the invention, or may not implement all the steps of the method according to the invention. In extreme cases, the head-mounted display and 3D display device may not implement any steps of the method according to the invention and may only display the stream of the source image pair (IS1, IS2)n and / or the stream of the rendered image pair (IR1, IR2)n of scene 1.
[0324] According to an advantageous embodiment, the head-mounted display and 3D display device include at least one imaging system arranged to image an observer and / or the observer's eyes.
[0325] The processing unit may be arranged and / or programmed and / or configured to determine and / or calculate viewpoint changes based on data from at least one imaging system arranged to image the observer and / or the observer's eye.
[0326] Head-mounted displays and 3D display devices each include means arranged to detect movement and / or relative position of the head-mounted display and the 3D display device. As a non-limiting example, the means arranged to detect movement and / or position may be a gyroscope or accelerometer for a head-mounted display, and may be one or more imaging systems for a 3D display device.
[0327] Furthermore, advantageously, the obtained viewpoint changes depend on, or are calculated and / or determined based on, data such as that derived from a device arranged to detect the movement and / or relative position of the head-mounted display (referred to as head-mounted display data), and from one or more imaging systems of a 3D display device.
[0328] Of course, the present invention is not limited to the examples just described, and many modifications can be made to these examples without departing from the scope of the present invention.
[0329] Therefore, in the interchangeable variations of the previously described embodiments:
[0330] - Viewpoint changes include movement (du, dv, and / or dz) in the reference frame of one or both images of the source image pair (IS1, IS2) and / or in the user's reference frame of the head-mounted display or 3D imaging device, and / or
[0331] - The method may include acquiring all or part of a source image pair (IS1, IS2) of scene 1 originating from two discrete imaging systems at two discrete locations or two discrete viewpoints, and / or
[0332] - This method may include acquiring an image of the observer, and / or
[0333] - This method may include acquiring an image of the observer's eye, and / or
[0334] - The method may include acquiring an image of at least a portion of the environment of the head-mounted display or at least a portion of the environment of the 3D display device, and / or
[0335] - In this disclosure, the two discrete imaging systems can be one or more imaging systems of a VR / AR head-mounted display or a 3D display device, and / or
[0336] - In this disclosure, the image of the observer and / or the image of the observer's eyes and / or the image of at least a portion of the environment of the head-mounted display or the environment of the 3D display device can be acquired by one or more cameras of the head-mounted display or the 3D display device, and / or
[0337] - Two discrete imaging systems acquiring images of the source image pair (IS1, IS2) are stationary and / or at a constant distance and constant position relative to each other during image acquisition, and / or
[0338] - This invention proposes a computer program comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of the disclosed embodiments, and / or
[0339] - The present invention proposes a readable medium including instructions, particularly a medium readable by a computer or any device including a processing unit, which, when executed by the computer or the device, causes the computer or the device to perform the method according to any one of the described embodiments.
[0340] Furthermore, the various features, forms, variations, and embodiments of the present invention can be combined with each other in various combinations, as long as they are incompatible or not mutually exclusive.
Claims
1. A method for dynamic observation of stereoscopic images of a 3D video of a scene, the 3D video comprising a set of stereoscopic 2D image pairs, referred to as one or more source image pairs, which can provide a 3D video rendering by display in a head-mounted display or on a 3D display device, wherein, The method includes the following steps: - Obtain the viewpoint change relative to the imaged scene. - Modify each image in at least one source image pair to render at least one stereoscopic 2D image pair of the scene, referred to as at least one rendered image pair, wherein the at least one rendered image pair: • Designed to provide 3D rendering when displayed in head-mounted displays or 3D display devices. • Provides viewpoint changes relative to the 3D video of the scene; The image of at least one source image pair is modified by moving and / or expanding and / or shrinking pixels or one or more groups of pixels of at least one source image pair of the scene, and / or by adding pixels or groups of pixels to the image of the at least one source image pair and / or removing pixels or groups of pixels from the image of the at least one source image pair, according to the following: • Depth map of the imaged scene, • The resulting viewpoint changes.
2. The method according to the preceding claim, wherein, The modification steps include: - Modify each of the two images in the considered source image pair in the same way to render at least one image pair including the two modified images, or - Modify each of the two images in the source image pair under consideration differently to render at least one image pair including the two modified images.
3. The method of claim 1 or 2, wherein, The step of modifying the image of the at least one source image pair is also performed relative to a region of the image of the at least one source image pair, the region being referred to as the region of interest.
4. The method according to the preceding claim, wherein, The area of interest: - Remains unchanged when the image of at least one source image pair is modified, or - Modified when the image of the at least one source image pair is modified.
5. The method according to any one of the preceding claims, wherein, The modification of the image of the at least one source image pair is proportional to the change in viewpoint.
6. The method according to any one of the preceding claims, wherein: - A portion of the pixels or pixel group of the image of the at least one source image pair, referred to as the first portion, wherein the modification of the first portion is inversely proportional to the depth associated with the pixels or pixel group of the first portion, and / or - A portion of the image of the at least one source image pair that differs from the first portion is referred to as the second portion, and the modification of the second portion is proportional to the depth associated with the pixels or pixel group of the second portion.
7. The method according to the preceding claim and in combination with claim 3 or 4, comprising: For each pixel or one or more groups of pixels in the image of the at least one source image pair, calculate: - The depth deviation between the region of interest and each pixel or one or more pixel groups located outside the region of interest, and / or - The deviation between the inverse of the depth of the region of interest and the inverse of the depth of each pixel or one or more pixel groups located outside the region of interest; The modification of the images of the at least one source image pair is performed based on the calculated deviation.
8. The method according to any one of the preceding claims, the method comprising calculating at least one angle associated with or corresponding to the viewpoint change; the image of the at least one source image pair being modified according to at least one calculated angle.
9. The method according to any one of the preceding claims, the method comprising calculating a distance change relative to the imaged scene that is associated with or corresponds to the viewpoint change; and modifying the image of the at least one source image pair according to the calculated distance change.
10. The method of any of the preceding claims, wherein, The modification steps are applied to the stream or stack of the stored source image pairs.
11. The method of any of the preceding claims, wherein, The image of at least one source image pair is modified according to the interpupillary distance.
12. The method of any of the preceding claims, wherein, Modify the image of the at least one source image pair according to the following: - Obtain the distance between the two discrete imaging devices for each source image pair, and / or - The positions of the two discrete imaging devices relative to each other and / or relative to the scene, and / or - The distance deviation between the two discrete imaging devices and the scene; The two discrete imaging devices are arranged at two discrete spatial locations, and one image of each source image pair originates from one of the two imaging devices, and the other image of each source image pair originates from the other of the two imaging devices.
13. The method according to the preceding claim and in combination with claim 11, wherein, By moving and / or expanding and / or shrinking pixels or one or more groups of pixels of at least one source image pair of the scene, such that the viewpoint difference between the two images of the at least one rendered image pair corresponds to the interpupillary distance, the images of the at least one source image pair are also modified relative to each other.
14. The method according to any one of the preceding claims, the method comprising the step of displaying an image of the at least one rendered image pair or the at least one source image pair in a virtual reality / augmented reality (VR / AR) head-mounted device called a head-mounted display or on a 3D display device.
15. The method according to the preceding claim, wherein: - The resulting viewpoint change depends on the position and / or orientation and / or tilt of the head-mounted display or the 3D display device in space, or the position and / or orientation and / or tilt of the observer relative to the 3D display device in space, and / or - The area of focus corresponds to the area in the displayed scene where the observer's gaze is focused.
16. The method according to the preceding claim, wherein: - The viewpoint change is obtained from the position data and / or movement data of the head-mounted display, the position data and / or movement data being referred to as head-mounted display data, and / or - The viewpoint change is obtained from the observer's image and / or the image of the observer's environment and / or the image of the head-mounted display or the image of the 3D display device, and / or - The area of interest is obtained from the image of the observer's eye.
17. The method of claim 16, comprising: The data from the head-mounted display or the 3D display device is filtered to limit the magnitude of the change in viewpoint over time to a value below a limit.
18. The method of claim 16 or 17, comprising: The head-mounted display data or 3D display device data is non-linearly limited to restrict the magnitude of the obtained viewpoint change to a value below a limit.
19. The method of any one of claims 16-18, comprising: The gain applied to the head-mounted display data or 3D display device data is non-linearly dynamically compressed to decay the amplitude variation or magnitude of the viewpoint change obtained over time, such that the amplitude variation does not exceed an amplitude variation threshold or magnitude threshold.
20. The method of any one of claims 16-18, comprising: The gain applied to the head-mounted display data or 3D display device data is non-linearly and dynamically amplified to vary the magnitude of the change in viewpoint over time.
21. The method according to any one of the preceding claims, the method comprising the following steps: - Acquire images of the scene, including all source image pairs, and / or - Obtain the observer's image, and / or - Acquire an image of at least a portion of the environment of the head-mounted display or an image of at least a portion of the environment of the 3D display device.
22. A data processing apparatus comprising means arranged and / or programmed and / or configured to implement the method according to any one of claims 1 to 21.
23. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 21.
24. A computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 21.
25. A virtual reality / augmented reality (VR / AR) head-mounted display, said head-mounted display including one or more means for displaying 3D video, and: - A means arranged and / or programmed and / or configured to implement the method according to any one of claims 1 to 21, and / or - A communication device arranged and / or programmed and / or configured to and / or capable of communicating with an external device, wherein the external device is arranged and / or programmed and / or configured to implement the method according to any one of claims 1 to 21.
26. The VR / AR head-mounted display according to the preceding claim, the head-mounted display comprising at least one imaging device arranged to image the eyes of an observer wearing the VR / AR head-mounted display.
27. The VR / AR head-mounted display according to claim 25 or 26, wherein the head-mounted display comprises: - At least one imaging device arranged to image at least a portion of the environment of the VR / AR head-mounted display and / or at least a portion of the observer's environment, and / or - A device arranged to detect the movement and / or position and / or orientation and / or tilt of the VR / AR head-mounted display.
28. A 3D display device, the 3D display device being arranged to display 3D video, the 3D display device comprising: - A means arranged and / or programmed and / or configured to implement the method according to any one of claims 1 to 21, and / or - A communication device arranged and / or programmed and / or configured to and / or capable of communicating with an external device, wherein the external device is arranged and / or programmed and / or configured to implement the method according to any one of claims 1 to 21.
29. The 3D display device according to the preceding claim, the 3D display device comprising at least one imaging device arranged to image an observer's eye.
30. The 3D display device according to claim 28 or 29, wherein the 3D display device comprises: - At least one imaging device arranged to image at least a portion of the environment of the 3D display device and / or at least a portion of the observer's environment and / or the observer's environment. - A device arranged to detect the movement and / or position and / or orientation and / or tilt of an observer relative to the 3D display device.
