Viewer adjustment of a stereoscopic image display of a camera

Abstract

The present disclosure relates to an image modification process for generating a selected type of stereoscopic video of a common field of view shared by two cameras without requiring any presetting and arrangement normally required for stereoscopic display. The process may use left and right video streams of images from various camera types that share part of their respective fields of view that are not limited to specific relative orientations (convergent, parallel, or divergent), and may be used to adjust stereoscopic video to be displayed for viewing to a viewer in real-time, such as to display stereoscopic images to the viewer. Wherein the dynamic input of the viewer is continuously updated. The proposed method can be used to optimize, correct and / or allow the use of various devices (web cameras, cameras, scanners, etc.), which may be from a production line and have defective capture cameras.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 348,743 filed on June 3, 2022, the contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to a stereoscopic image display. Background Art

[0004] Conventional stereoscopic imaging involves acquiring left-eye and right-eye images with camera positions and orientations corresponding to nominal viewing geometry (e.g., a person sitting in the middle of a movie theater) to provide a comfortable 3D viewing experience. When stereoscopic videos acquired for movie theaters are to be viewed at home, the 3D experience is often unsatisfactory. The depth of objects may appear unnatural and cause strain or fatigue.

[0005] Alternative methods of performing image processing to reformat and / or optimize stereo images have been previously proposed. Applicants have previously proposed a technique for adjusting stereo images acquired from parallel cameras that has been shown to reduce eye fatigue and improve the 3D viewing experience, as described in U.S. Patent 10,917,623. However, this technique imposes a restrictive constraint on the capture cameras, namely that the capture cameras are parallel. Summary of the invention

[0006] In this patent, the term "capture camera" refers to a physical camera used to capture images or video of a selected scene in a real environment (ie, not a virtual environment).

[0007] The present disclosure relates to an image modification process for generating selected types of stereoscopic images and / or videos of a common field of view shared by two capture cameras without requiring any setup or pre-arrangement normally required for stereoscopic display. The process can use left and right video and / or image streams of images from various camera types that share a portion of their respective fields of view that are not limited to a particular relative orientation (converging, parallel, or diverging), and the process can be used to adjust the stereoscopic video to be displayed to a viewer in real time, with the viewer's dynamic input continuously updating.

[0008] Applicants have discovered that differences in viewing angles of images sharing a common portion of their field of view caused by misalignment and / or non-optimal orientation of the capture cameras can be overcome.

[0009] A method for processing images using a graphics processing unit (GPU) for providing a stereoscopic display to a viewer, the images being captured by a left capture camera and a right capture camera having a common portion of a captured field of view, the method comprising: generating a left polygon mesh and a right polygon mesh using specification parameters of the left capture camera and the right capture camera having a central axis, and defining a position of a virtual camera relative to the mesh to correspond to the field of view of the capture cameras; texturing the left polygon mesh and the right polygon mesh with at least one image from the left capture camera and the right capture camera, respectively, to create a left virtual 3D object and a right virtual 3D object, respectively; and

[0010] A left eye image and a right eye image are rendered using a left virtual camera and a right virtual camera, wherein the left virtual camera and the right virtual camera are oriented to capture at least a portion of a left virtual 3D object and a right virtual 3D object, respectively; wherein orientation parameters of the left virtual camera and the right virtual camera are determined in the following manner: (a) determining a reference line extending between a position of the left capture camera and a position of the right capture camera in a capture space; (b) determining the orientations of the left virtual camera and the right virtual camera in the capture space so that the orientations of the left virtual camera and the right virtual camera are parallel to each other and located in a selected viewing horizontal plane; (c) determining a left orientation offset vector and a right orientation offset vector, which are defined as the amount by which the orientations of the left virtual camera and the right virtual camera change relative to the orientations of the optical axes of the left capture camera and the right capture camera corresponding to the central axis; (d) using the left orientation offset vector and the right orientation offset vector, orienting the left polygon mesh and the right polygon mesh around the central axis in the virtual space relative to the left virtual camera and the right virtual camera, respectively; and (e) determining the viewing cone parameters of the left virtual camera and the right virtual camera using display parameters, wherein the display parameters include display size, distance from the display, interocular distance, and viewing direction.

[0011] In some embodiments, the display parameters include a magnification, wherein the viewing cone parameters are determined using the magnification.

[0012] In some embodiments, step (d) includes determining lens offset parameters of the left virtual camera and the right virtual camera.

[0013] In some embodiments, the optical axis orientations of the left capture camera and the right capture camera are coplanar.

[0014] In some embodiments, the optical axes of the left capture camera and the right capture camera are parallel, wherein the left capture camera and the right capture camera have different fields of view, and wherein at least one of the optical axes of the left capture camera or the right capture camera is non-perpendicular to the reference line.

[0015] In some embodiments, the original image may be a still image.

[0016] In some embodiments, the original image may be a video image.

[0017] In some embodiments, the left-eye image and the right-eye image are displayed to the viewer on a single screen of a stereoscopic display.

[0018] In some embodiments, the left-eye and right-eye images are generated as anaglyph format images, column-interleaved format images displayed on an autostereoscopic display, a sequence of flipping images viewed with shutter glasses, or a sequence of row-interleaved images for polarized display.

[0019] In some embodiments, the stereoscopic display includes a left image display and a right image display, wherein a left-eye image is displayed to a viewer's left eye with the left image display, and a right-eye image is displayed to a viewer's right eye with the right image display.

[0020] In some embodiments, the viewer includes a plurality of viewers, and the inter-eye distance is selected to be a minimum inter-eye distance among the plurality of viewers.

[0021] In some embodiments, the relative orientation between the left capture camera and the right capture camera is divergent.

[0022] In some embodiments, the relative orientation between the left capture camera and the right capture camera is parallel.

[0023] In some embodiments, the relative orientation between the left capture camera and the right capture camera is converging.

[0024] In some embodiments, at least one of the left-eye image and the right-eye image has an aspect ratio different from an aspect ratio determined by the size of the stereoscopic display, and the method further includes providing border padding for at least one of the left-eye image and the right-eye image displayed to the viewer.

[0025] In some embodiments, the captured field of view includes a panoramic image, and the method further includes determining a selected optical axis within a portion of the panoramic image contained in a common portion of the captured field of view, wherein the portion of the panoramic image and the selected optical axis are processed by the steps of the method.

[0026] Another broad aspect is a device for processing a stereoscopic image for display to a viewer, the device comprising a graphics processing unit, a central processing unit and a memory readable by the central processing unit, the memory storing instructions for executing the method as described above.

[0027] Another broad aspect is a computer program product comprising a non-transitory memory storing instructions for a processor connected to a GPU to execute the method as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The present invention will be better understood by describing the following detailed description of embodiments of the present invention with reference to the accompanying drawings, in which:

[0029] Figure 1A is a top view of a diagram showing a scene featuring a sphere, a cube, and a sun as viewed by a viewer.

[0030] Figure 1B is an illustration of an image of a scene captured from the axis origin.

[0031] Figure 1C is an illustration of the image of the scene as seen by the left eye of an observer in front of the axis origin.

[0032] Figure 1D is an illustration of the image of the scene as seen by the right eye of an observer in front of the axis origin.

[0033] Figure 1E is an illustration of superimposed images of the scene perceived by the left and right eyes, such that the sun at infinity appears in the same position to better visualize the parallax between the left and right recaptured images.

[0034] Figure 2A is a schematic diagram of a top view of two different capture cameras recording a common scene from different known orientations and positions.

[0035] Figure 2B is a schematic isometric view of the field of view of the two capture cameras.

[0036] Figure 2C yes Figure 2B Schematic plan view of the field of view of the two capture cameras is shown in FIG.

[0037] Figure 2D is to move the virtual camera along Figure 2C Schematic diagram of an embodiment in which reference axes are placed in a capture space reference system at an offset relative to a capture camera.

[0038] Figure 2E The left virtual camera is Figure 2D The offset determined in the schematic plan view is used to determine the orientation and position of the placed in the virtual space.

[0039] Figure 2F The right virtual camera is Figure 2D The offset determined in the schematic plan view is used to determine the orientation and position of the placed in the virtual space.

[0040] Figure 2G It is shown Figures 2B to 2F A flowchart of the steps involved in an embodiment of the present invention.

[0041] Figure 2His a schematic diagram of a top view of a virtual environment with generated virtual objects and two similar virtual cameras positioned parallel to each other, wherein the two similar virtual cameras are in relative positions equivalent to the positions of their corresponding capture cameras and are at a specific known distance from each other to recapture stereo images while preserving the geometric correspondence of the capture cameras.

[0042] Fig.2I is a schematic diagram of a top view of an alternative embodiment, wherein two virtual environments each have a generated virtual object and a similar virtual camera, wherein the similar virtual camera is set to be virtually parallel to the virtual camera in the other virtual environment.

[0043] Figure 2J A three-dimensional schematic diagram of a virtual environment is shown with an exemplary arrangement of some of the various elements of the virtual environment (base axes, virtual camera, optical axes, etc.) to ensure that the geometric correspondence with the real environment is preserved.

[0044] Figure 3A is a top view diagram showing a scene featuring a sphere, a cube, and a sun as viewed by a diverging wide angle lens capture camera with a wide field of view.

[0045] Figure 3B is a three-dimensional side view of a diagram showing a polygonal mesh of a wide angle lens capture camera.

[0046] Figure 3C is a two-dimensional illustration of the front view of a virtual object captured by a camera corresponding to a wide-angle lens. Figure 3B It is obtained by texturing the polygonal mesh.

[0047] Figure 4A is a top view of the diagram of the camera transformation step, in which the angular size of the virtual camera's viewing frustum can be adjusted.

[0048] Figure 4B A top view of a diagram showing a camera transformation step in which the field of view of the left virtual camera may be offset using image sensor translation.

[0049] Figure 4C is an illustration of a rendered left final image captured with a virtual camera of a left virtual environment to be displayed to a viewer's left eye.

[0050] Figure 4D is an illustration of a rendered right final image captured with a virtual camera of a right virtual environment to be displayed to the viewer's right eye.

[0051] Figure 4EIt is a complementary color stereographic representation of the superposition of the left and right final images, so that the sun at infinite distance appears in the same position, realizing the visualization of parallax between other objects.

[0052] Figure 4F It is an anaglyphic representation of a superposition of a left final image and a right final image, which superposition has a lateral offset of the interocular distance (Io) of the viewer when it can be displayed on a display screen.

[0053] Figure 4G is an illustration of an exemplary rendered final image having a common portion that is smaller than or has a different aspect ratio than a display screen, where areas (bounding boxes) of the image displayed outside the common area may be rendered as neutral pixels.

[0054] Figure 5A is a flow chart of an exemplary method for processing dynamic parameters that may be considered to adjust a stereoscopic image to be viewed on a display.

[0055] Figure 5B is a block diagram presenting the inputs to the methods of some embodiments and some of their corresponding objectives.

[0056] Figure 6 is a graph used to select recaptured images from the virtual object to generate the viewing cone for the final image projected to the viewer.

[0057] Figure 7 is a three-dimensional diagram illustrating the effect of adjusting or changing the viewing cone on the selected virtual object area and therefore on the final image to be displayed for viewing by the viewer.

[0058] Figure 8 is a block diagram of the hardware components required to process and execute the proposed method for producing a stereoscopic display for viewing by a viewer.

[0059] Fig. 9A is an illustration of a top view of a head mounted stereoscopic device for displaying stereoscopic images on left and right screens of the head mounted stereoscopic device for an initial position of a viewer and perceived by the left and right eyes of the viewer, respectively.

[0060] Fig. 9B is an illustration of a top view of an image that has not been adjusted for new dynamic parameter inputs associated with a new orientation of the viewer's viewing angle and a new region of interest for the viewer.

[0061] Fig. 9C is an illustration of a top view of a stereoscopic image processed to comply with a new dynamic parameter input that provides a view of a new region of interest and optimal depth perception for the viewer.

[0062] Fig. 10A is an illustration of an embodiment in which the smartphone's camera is slightly misaligned within its housing.

[0063] Fig. 10B is a diagram showing the camera slightly misaligned within its housing Fig. 10A Cross-sectional view at the cutting plane line AA.

[0064] Fig.11 is a block diagram of some main steps of one embodiment of a method for generating a stereoscopic image that is dynamically adjusted according to parameters of a viewer.

[0065] Fig.12 A block diagram is presented of some of the steps that may be required to ensure that each virtual camera is set up in the correct manner to ensure that the rendering of stereo images is practical, optimal and realistic and has a geometric correspondence with the real world. DETAILED DESCRIPTION

[0066] The image processing for stereoscopic display will be better understood through the following detailed description of embodiments with reference to the accompanying drawings.

[0067] The methods described herein may be used to provide a 3D experience using a pair of rendered stereo images that may be generated from images captured by capture cameras that share a common portion of their field of view, which may not be intended for stereoscopic vision, and displaying these final rendered stereo images using a selected stereo technique (autostereoscopic, anaglyph, row interleaved, column interleaved, etc.) and a display device.

[0068] The method may include generating, for each capture camera, a polygonal mesh in the virtual environment based on distortion specification parameters of the capture camera lens type (standard, wide angle, fisheye, etc.), magnification, position and / or orientation of the capture camera.

[0069] In a virtual environment, the captured images may be used to texture corresponding polygonal meshes to generate virtual objects, which may be 3D (3D objects). The virtual environment may allow positioning of virtual objects at a selected position and orientation relative to a corresponding virtual camera, or vice versa, based on positional specifications of the capturing camera.

[0070] The virtual cameras may be used to capture selected portions of their respective virtual objects (projective transformation). The final stereoscopic image may be displayed to a viewer for viewing after operations that may include cropping and rendering selected portions of the virtual objects.

[0071] These virtual cameras with viewing cones may have the following dimensions: the aspect ratio of the stereoscopic display's aspect ratio, the viewer's interocular distance, any vergence accommodation, his distance from the stereoscopic display, and any magnification input selected by the viewer and any zoom selected by the viewer.

[0072] The lens offset of the virtual camera may be determined by the viewer's interocular distance, his distance from the stereo display, the size of the stereo display, any convergence / accommodation compensation, any off-axis viewing angles of the viewer (the viewer's area of ​​interest), or a combination thereof.

[0073] In some embodiments, the rendered image may be further processed using methods such as disclosed in US Pat. No. 10,917,623 B2 to generate a pair of optimized final stereo images.

[0074] Figure 1A A schematic diagram of a top view of a scene featuring a sphere located in the middle of the ground, the sun located in the middle of the sky at infinity (D∞), and a cube placed to the right of the sphere and slightly further away. Also shown are the left and right eyes of an observer centered on the axis origin and the scene without any convergence orientation (parallel), as if the observer is looking at an object (or in this case, the sun) at infinity.

[0075] Figure 1B An illustration of the same scene as if captured with a camera placed at the origin is presented, which helps to compare the following stereoscopic illustration with what can be considered a passive visualization of the scene.

[0076] Figure 1C and 1D The common areas of the scene observed by the left and right eyes of the observer are shown respectively.

[0077] Figure 1E A previous image of the scene seen by the observer's left eye is presented (from Figure 1C , and shown here as faded out) and the previous image of the scene seen by the observer's right eye (from Figure 1D , and shown here with dashed lines). This superposition can be done so that an object that the viewer is likely to focus on (the sun) appears as a single object without any parallax, thus accounting for the various parallaxes of other objects in the scene.

[0078] As those skilled in the art will appreciate, functional stereoscopic images are typically achieved using capture cameras of limited types and configurations using common stereo viewing techniques described in the prior art. For example, to generate a stereo image similar to Figure 1EStereoscopic images of complementary color stereoscopic representations typically require two identical capture cameras with lens arrangements specific to stereoscopic capture / filming (which further requires very specific settings (e.g., Figure 1A In practice, if these guidelines are not strictly followed, the viewer may misperceive depth and / or experience negative physiological reactions.

[0079] However, the Applicant has developed an image processing technique that allows the generation of functional stereoscopic images with a geometric correspondence between reality (the convergence point of the viewer's eyes) and a stereoscopic display for using images captured by various capture camera types (standard, wide angle, fisheye, etc.), the capture cameras not requiring restricted pre-selected positioning as long as they share a common portion of their respective fields of view.

[0080] In practice, as will be noted by those skilled in the art, the types of capture cameras and their relative positioning may often be specific to those considered to be optimal stereo imaging conditions or functional correspondences. Some embodiments of the developed process may be used to generate stereoscopic video that adapts in real time to the viewer's perspective and dynamic parameters, or may accelerate the computational process of some existing stereoscopic image reformatting techniques.

[0081] Note that "functional" stereo images may be defined here as stereo images that provide the correct perception of depth, amplitude, scale, etc., without causing noticeable negative physiological reactions in the viewer, such as dizziness, headaches, any uncomfortable feelings, etc.

[0082] Figure 2A The accompanying drawing of shows a schematic diagram of a top view of a wide angle left capture camera 20L and a coplanar standard right capture camera 20R, the two capture cameras recording a common scene from different positions and from possibly different orientations. Figure 2A In the example of , the right capture camera can be located on the x-axis and have an optical axis 22R that has an orientation parallel to the z-axis.

[0083] The orientation and distance between the center points of the two capture cameras relative to the reference axis 26 can be extracted from the static parameters of the capture cameras by using the relative three-dimensional positions of the capture cameras. The relative orientation 30L or 30R of the optical axis 22L or 22R of the capture camera relative to the relative reference axis 26 can be measured or calculated using the parameters of the capture camera. Some real environment characteristics (including relative orientation, relative reference axis, etc.) can then be used to locate some elements included in the virtual environment.

[0084] Figure 2BAn isometric view of the field of view of two capture cameras is shown. As shown, the proposed method can use a pair of various capture cameras that do not need to have similar lens types or fields of view. In this example, the left capture camera 20L has a field of view that can be narrower than the right capture camera 20R.

[0085] like Figure 2C As shown, the optical axes 22L and 22R do not have to be parallel. The cameras 20 may even have a roll angle between them, which means that the captured fields of view may be at different angles in their images. The parameter data from the capture cameras 20L and 20R may be recorded based on measurements performed by the camera operator, or may be done by analyzing images acquired from the capture cameras. The parameter data of the capture cameras (capture camera parameters) may include specifications for the field of view angle and shape (e.g., rectangular, circular, elliptical) and the relative orientation and relative position between the capture cameras. It should be noted that the optical axes 22L and 22R do not need to be coplanar.

[0086] Note that reference axis 26 is defined herein as a line passing through the positions of capture cameras 20L and 20R. In virtual space, reference axis 26 can be used as a base axis to position various virtual components (virtual cameras, meshes, 3D objects, etc.) relative to it. In a preferred embodiment, the virtual cameras can be virtually positioned so that reference axis 26 passes through both virtual cameras.

[0087] As described below, a GPU may be used to process the captured camera image to provide an image suitable for display to a viewer. In the case of parallel cameras with a common baseline, the applicant's PCT patent application publication WO2019 / 041035 published on March 7, 2019 describes the use of a graphics processing unit (GPU) for image processing to provide an image suitable for display to a viewer.

[0088] In a GPU, one way to process an image might be to create a virtual surface or object, texture the object with the captured image and then render the image using a virtual camera that actually performs the required scaling, offset and optional dewarping using the virtual surface or object's specification parameters and the virtual camera parameters. Following this approach, real-time video processing can be easily performed using a GPU.

[0089] In the present application, the technique can be extended to cameras whose capture properties are not restricted in terms of capture orientation. This restriction can be addressed by determining a reference axis 26 in the capture space and determining the position of the virtual camera to be located on the reference axis 26 at the position of the central axis of the capture camera and making the virtual camera axes parallel to each other, also as Figure 2C The direction of the virtual camera, i.e., the left-right angle and the up-down angle, is the viewing direction. The spacing between the virtual cameras 21 may correspond to the eye spacing of the viewer. Figure 2D As shown in and referenced below Figure 2J Angular orientation offset is described in more detail.

[0090] Figure 2E and 2F Schematically illustrates how virtual cameras 21L and 21R are positioned relative to their corresponding 3D surfaces or objects 25. Although 3D objects 25L and 25R are shown as planar 3D surfaces, it should be understood that camera optics may result in non-uniform images and the corresponding 3D objects 25 may not be planar, as described below with reference to Figure 3B and 4A described.

[0091] like Figure 2E and 2F As shown, in a virtual space, a 3D object 25 textured with a source capture camera image can be positioned at a predetermined position. The GPU virtual camera can be used as a tool to extract images for display, which captures a view of a portion of the 3D object 25. In addition, many GPUs support a "lens offset" parameter for determining the properties of the virtual camera 21. The lens offset may be a parameter that makes the camera cone tilted (i.e., asymmetric). For example, the lens offset can offset the lens horizontally or vertically from the center. The value of the lens offset can be a multiple of the sensor size, for example, an offset of 0.5 along the X axis will offset the sensor by half its horizontal size.

[0092] Figure 2G Some processes of the present embodiments are shown, which may include step 201, where capture camera parameters can be retrieved; step 202, where a 3D object 25 can be determined based on the capture camera parameters and can be established in the GPU; step 203, where a source image can be loaded into the GPU and the GPU textures the object 25 using the corresponding source image; step 204, where a reference axis 26 can be retrieved and used in the virtual space, which reference axis can be provided as part of the parameters of the capture camera, or can be determined in advance in the capture space based on the capture camera parameters (e.g., the position of the capture camera in 3D space).

[0093] Parameters of the viewer may include distance from the display (ie, viewer-screen distance), display size, interocular distance, and / or viewing orientation of the viewer (which may be provided or measured).

[0094] In some embodiments, step 204 may utilize the viewer's parameters and / or the capture camera parameters to set / position the virtual camera 21 on the reference axis 26 (e.g., at each end) and / or may be used to determine / set the orientation of the parallel virtual camera axis 23, which may extend in the same direction from the reference axis 26. This may enable orienting the optical axis 23 of the virtual camera in the virtual space.

[0095] In some embodiments, step 205 may be used to calculate the relative orientation between the virtual camera axes 23 and their corresponding 3D objects (ie, 3D meshes textured with the captured image / texture).

[0096] In some embodiments, step 206 may be used to determine or calculate the view cone and / or lens offset for each virtual camera field of view, which may be done using viewer parameters and / or capture camera parameters.

[0097] In some embodiments, step 207 can be used in the following situation: the parameters of the virtual camera 21 can be passed to the GPU, which can then process the rendering of the left eye image and the right eye image for the viewer, and then various methods and devices can be used to display the image to the viewer through a display screen.

[0098] like Figure 2H As shown in the exemplary embodiment of , the left virtual camera 21L, the right virtual camera 21R and the polygon mesh 25 (geometry) can be positioned using some real environment features. It should be understood that the calculation of the orientation mentioned above can be similar to a simple change of the coordinate system.

[0099] For example, in Figure 2H In some embodiments, each virtual camera 21L / 21R can be fixed at each corresponding end of the relative reference axis 26 to position and orient the virtual cameras. In some embodiments, the distance between the two virtual cameras 21L / 21R can be positioned to have the same relative distance between the real distances between the capture cameras 20L / 20R.

[0100] Most importantly, the virtual cameras 21L / 21R may be provided with optical axes 23L / 23R having an orientation perpendicular to the relative reference axis 26, and wherein the optical axes may be parallel to each other. Their respective polygonal meshes (25L for the left camera and 25R for the right camera) may be positioned along a virtual object axis 22', which is an axis equivalent to the capture camera optical axis 22, and which may be oriented with a relative orientation 30 (30L for the left virtual camera 21L and 30R for the right virtual camera 21R) equivalent to the relative reference axis 26.

[0101] It should be understood that all these requirements can be integrated into or translated by a virtual orientation vector 28 (28L for the left virtual camera 21L and 28R for the right virtual camera 21R) between the optical axes 23 of the virtual cameras 21L / 21R in the direction of the virtual object 22' (equivalent to the optical axis of the capture camera 22L / 22R), which may be an orientation equivalent to the difference between the orientation of the optical axis 23 of the virtual camera relative to the relative reference axis and the orientation relative orientation 30.

[0102] Thus, in some embodiments, the virtual camera may be oriented at the virtual orientation vector 28 from the orientation (direction) of the virtual object, or similarly, the orientation (direction) of the virtual object may be oriented at the virtual orientation 28 from the virtual camera.

[0103] In some embodiments, the distance between the virtual camera 21 and its corresponding grid 25 can be adjusted along the virtual object axis using some parameters of the corresponding capture camera to account for the distance between the viewer and the stereoscopic display (e.g., display screen) or its equivalent.

[0104] Those skilled in the art will understand that Figure 2H In the case of the illustrative example of , the original images (herein referred to as textures 24) captured by the capture cameras and used to texture their corresponding polygonal meshes will only be visible from their corresponding virtual cameras. In other words, to prevent the texture and mesh of one camera from blocking the field of view of the other camera, the left virtual camera 21L can only perceive the 3D object 39L of the left capture camera 20L (i.e., the left 3D mesh 25L textured with image / texture 24L), while the right virtual camera 21R can only perceive the 3D object 39R of the right capture camera 20R (i.e., the right 3D mesh 25R textured with the right image / texture 24R), just as they are in Fig.2I as shown in the alternative virtual environment.

[0105] In some embodiments, the virtual camera 21L / 21R can be positioned relative to the corresponding polygonal mesh 25L / 25R using the specification parameters of the capture camera 20L / 20R, or vice versa, to ensure a geometric correspondence between the real environment and the virtual environment (i.e., the position of the virtual camera is equivalent to the position of the corresponding capture camera in the virtual environment).

[0106] In some embodiments, to ensure this geometric correspondence, the virtual camera 21L / 21R may be positioned on an axis equivalent to the capture camera's optical axis (eg, an axis perpendicular to the center of the polygon mesh), which may be extracted or calculated using the capture camera's specification parameters.

[0107] In some embodiments, to ensure this geometric correspondence, the virtual camera 21L / 21R can be positioned at a distance from the polygonal mesh 25L / 25R such that when facing the polygonal mesh from this distance, a virtual camera having a field of view equivalent to the field of view of the corresponding capture camera 20L / 20R (i.e., having the same angular size, which can be extracted or calculated using the specification parameters of the capture camera) can accurately see (i.e., no more and no less) the entirety of the polygonal mesh (i.e., so that the perimeter of the field of view exactly coincides with the edge or perimeter of the polygonal mesh).

[0108] Figure 2J A three-dimensional schematic diagram of various elements used to calculate a virtual orientation vector 28 for each of the left and right capture cameras is presented, which, as previously described, can be used to ensure geometric correspondence in the virtual environment with the real environment. The illustrated embodiment presents a representation of a relative reference axis 26 (extending between the two capture cameras 20L / 20R), a base axis 29 perpendicular to the relative reference axis 26, and an optical axis 22 of the capture camera 20. As previously explained, characteristics of some of these elements (e.g., position and orientation) can be extracted or calculated using static parameters of the capture camera (a portion of the capture camera parameters) using their relative three-dimensional positions and the relative orientation of the capture camera optical axes, or derived from the static parameters.

[0109] In some embodiments, the base axis 29 may be defined to have a selected pitch associated with the pitch component 200p of the viewing horizon 200 and to have a yaw perpendicular to the relative reference axis 26. The relative reference axis 26 may have an orientation of a line between the center points of the two capture cameras, which orientation is calculated using the static parameters (relative three-dimensional positions) of the capture cameras. In some embodiments, the static parameters (specification parameters) of the capture cameras may be used to calculate the virtual orientation 28 (the orientation between the base axis 29 and the optical axis 23 of the virtual camera).

[0110] In some embodiments, it is possible to calculate Figure 2J Components of virtual orientation 28 (roll 28r, pitch 28p or yaw 28y) are shown in FIG. In some embodiments, pitch component 28p (pitch orientation) of virtual orientation 28 may correspond to the difference in pitch between optical axes (arbitrary pitch of capture camera 20) relative to base axis 29.

[0111] It should be understood that in some embodiments, when the base axis 29 may have a pitch 29p from a reference coordinate system of static parameters (e.g., for determining the pitch 30p of a virtual camera), the pitch 28p of the virtual orientation may be calculated alternatively using the following formula 28p=|29p-30p|.

[0112] In some embodiments, the yaw component of the virtual orientation (yaw orientation) 28y may correspond to the yaw difference between the optical axis 22 and the base axis 29 , which in turn may be calculated using the yaw 30y between the optical axis 22 and the relative reference axis 26 using the following formula 28y=|90°−30y|.

[0113] In some embodiments, the roll component (roll orientation) 28 r of the virtual orientation that may correspond to the roll between the virtual object and the virtual camera may be equivalent to the roll difference 28 r between the capture camera and the viewing field of view 200 .

[0114] In some embodiments, base axis 29 may be coplanar with optical axis 22 so that pitch component 28p of the virtual axis is zero. In some embodiments, the viewing orientation of a viewer may be used to directly determine the pitch 200p of viewing horizon 200 .

[0115] In some embodiments, the viewer's viewing orientation may be used to directly determine the pitch of base axis 29. In some embodiments, the pitch component of the viewer's viewing orientation (looking up or down) may be used to determine the pitch component 200p of the viewing horizon or the pitch component of the base axis.

[0116] It should be understood that those skilled in the art may calculate and characterize the virtual orientation 28 using alternative calculation methods, equations, coordinate systems, etc.

[0117] Figure 3A Shows Figure 1A , but in this embodiment, the scene may be captured using an exemplary diverging wide angle lens capture camera 20.

[0118] Figure 3B A schematic 3D representation of the geometric representation of the field of view of the exemplary diverging wide angle lens capture camera 20 is shown in side view. The geometric representation may be similar to that for correction purposes. Figure 3A The wide angle lens used in the illustrated embodiment captures the image captured by the camera and generates the geometric figure 25, also known in the art as a polygonal mesh.

[0119] Figure 3C A 2D representation of a polygonal mesh 25 with a deformed mesh pattern (mesh) 32 of a capture camera lens is shown from a front view. In this embodiment, a 3D object 39 (virtual image) can be obtained by texturing the polygonal mesh (geometry) 25 with the texture 24 (original capture image) of a wide angle lens capture camera. Those skilled in the art will understand that the process of this embodiment is not limited to any capture camera type (standard, wide angle, fisheye, etc.), and a virtual 3D object can be generated for any of them.

[0120] In a virtual environment, the field of view or cone of view of a virtual camera is often referred to as the view frustum, and it can be determined by two parameters: aspect ratio and angular size (field of view with an aspect ratio corresponding to or related to the aspect ratio of the display screen). While the aspect ratio of the view frustum can be adjusted based on the aspect ratio of the viewport (corresponding to the height and width of the display screen), the angular size of the view frustum (field of view) can be adjusted based on the viewer's interocular distance, any visual convergence / accommodation compensation, his distance from the stereoscopic display, any magnification (zoom) input selected by the viewer, or a combination thereof.

[0121] In some embodiments, the interocular distance may be selected or fixed to a given value (e.g., the average interocular distance of an adult or child). In some embodiments, multiple viewers may be considered, and the interocular distance may be selected to be the smallest interocular distance among the multiple viewers.

[0122] Figure 4A and 4B is a schematic top view illustrating an example of the view transformation steps of Applicant's method in a virtual environment, wherein a cross section of a virtual object 39 (polygonal mesh 25 textured with image 24 of capture camera 20) may be selected (i.e., captured) using virtual camera 21 (within its view cone 44). According to some embodiments, one of the view transformations may be used to adjust and modify the field of view (view cone 44) of the virtual camera to follow relevant dynamic parameters, such as any zoom selected by the viewer, the aspect ratio and / or resolution of the display screen, the interocular distance of the user and his distance from the stereo display.

[0123] Figure 4A An exemplary embodiment is presented showing such a resizing of the viewing cone 44, wherein the angular dimension (horizontal field of view) of the width 45 of the viewing cone can be adjusted to a larger width 45' and thus have a larger viewing cone 44', which can be obtained by or is equivalent to a horizontal resizing (i.e., enlarging) the image sensor 42 to a resized image sensor 42".

[0124] According to some embodiments, one of the view transformations may be used to adjust and modify the symmetry of the geometry of the view cone 44 to make it tilted (asymmetric). Such a view transformation may be referred to in the art as a lens shift of the virtual camera 21 which may be similar to a lateral translation. Figure 4B As shown in the exemplary embodiment of FIG. 4 , the lens shift 40 may include translating 40′ of the image sensor 42 of the virtual camera 21 to an alternative position (see 42′), which results in a shift (translation) of the viewing cone 44, thereby forming an asymmetric viewing cone 44. It should be understood that the lens shift may not cause the following. Figure 4B Any change of the optical axis 22 of the virtual camera 21 is shown to be the same for both viewing cones 44 and 44 ″.

[0125] The lens offset operation, which is well known in the art, is often used with a virtual camera. For example, it can be used in the 3D development platform Unity to "offset the lens along any axis to tilt the camera's viewing frustum."

[0126] The selected lens offset may be determined by the interocular distance of the viewer, his distance from the stereo display, the size of the stereo display, any convergence / accommodation compensation, any off-axis viewing angles of the viewer (the viewer's area of ​​interest), or a combination thereof. In some embodiments, the lens offset may be used to adjust various aspects of the stereo image, including but not limited to centering the 3D object at infinity, corresponding to infinity of the captured image, and / or ensuring a correspondence between the real environment and the virtual environment.

[0127] When the required view transform is complete, the virtual camera 21 can capture a virtual image within the viewport (the projection of the view cone 44 on the virtual object) by rendering the pixels within the region of interest (also called cropping). This can render functional stereo images for the viewer's left and right eyes, such as Figure 4C and 4D In some embodiments, these recaptured images can be further processed by simple manipulation and image adjustment techniques before being displayed to the viewer by the selected device. In some embodiments, it can be similar to the method described in US Pat. No. 10,917,623 B2.

[0128] To better visualize the disparity between these two images, Figure 4E The previous left final stereo image (with Figure 4C and appears here as a fade-out) and the right final stereo image (with Figure 4D 62)) is a diagram of a superposition of the left and right recaptured images 62 (the final stereo image is the rendered recaptured image 62) which is positioned so that the sun appears as a single object without any parallax. 左 , for the right virtual camera 21R is 62 右 ) are displayed to the corresponding eyes of the viewer (the spacing between these images is equivalent to the distance between the viewers' eyes, relative to the distance between the viewer and the display), which can achieve an optimized three-dimensional (3D) viewing angle of the scene.

[0129] Figure 4F An anaglyphic representation of a superposition of left and right recaptured images 62 is shown with such a lateral offset of the viewer's interocular distance (Io) as may be displayed on a display screen.

[0130] Figure 4G An exemplary embodiment is presented in which a pair of final stereo images 62 have a common area that is smaller than the display screen 48 or has a different aspect ratio than the display screen, wherein the corresponding Figure 4GIn the shaded area of ​​the display screen, areas of the image displayed outside the common area may not be subjected to the rendering process or may be displayed as neutral pixels (white, black, grayscale, an alternative selected uniform color, a selected color gradient, etc.), while the size of the recaptured image 62 may remain unchanged.

[0131] The method of the presented embodiment innovates by creating a virtual environment for recapturing the captured images using parallel virtual cameras, which virtual environment can be used to ensure geometric correspondence between the captured pattern and the generated functional stereo image of a portion of the common field of view of the capture cameras.

[0132] In some embodiments, the geometry generated and used to create virtual 3D objects in the virtual environment can be used to correct and eliminate distortions that may be associated with the lens of the capture camera. This can allow the use of parallel virtual cameras to efficiently and quickly recapture images, which can be dynamically updated to optimize their view cones and optimize the geometric correspondence between the capture environment and the image for stereoscopic display, and correct lens and perspective distortions in the process when necessary. Those skilled in the art will understand that the process of this embodiment is not limited to any capture camera type, and the visual reproduction mode (rendering process) can be adjusted to produce a variety of stereoscopic display outputs.

[0133] This integrated recapture method using parallel virtual cameras with fixed positions is one of the key aspects of Applicant's image processing method, which allows it to use a variety of capture camera types and arrangements without the need to pre-set them specifically for stereo purposes and correct for potential misalignments between their relative positions. In some embodiments, the processing load can be reduced by pre-generating virtual images 39 (geometrically textured with the original captured images in the virtual environment) and saving them for later re-upload and use to quickly generate stereo images and update them in real time with new or updated viewer dynamic parameters.

[0134] Figure 5AAn exemplary flow chart of a method for processing variables and inputs to generate and display a stereoscopic image XII to a viewer based on dynamic parameters V of a viewer and capture camera parameters II (e.g., static parameters of a capture camera) is presented. These dynamic parameters V can be considered as variable inputs considered in various steps. Step III: Positioning and generating a polygonal mesh in a virtual environment can be accomplished using the following capture camera parameters II: lens type, aspect ratio, projection type (equirectangular, spherical, planar, etc.), media name, horizontal field of view, vertical field of view, inclination, relative position of left and right cameras, etc. It should be understood that capture cameras can have their own specific parameters, meaning that they do not have to be similar cameras, as long as they share a common portion of their field of view. In some embodiments, all parameters (display parameters and viewer parameters) can be obtained simultaneously. In some embodiments, display parameters IV (aspect ratio, stereoscopic vision reproduction mode, etc.) can be used to adjust some virtual camera parameters (e.g., cone of view) in step VI, while some viewer dynamic parameters V (interocular distance, distance between eyes and display screen, etc.) can be used to adjust some dynamic parameters. Some viewer dynamic parameters V (e.g., magnification, offset) may also be used to implement a scaling factor (increase / decrease the view cone), offset, or orientation change of the virtual camera view cone after steps VII, VIII, and IX. In some embodiments, additional processing may be added, such as methods and parameters similar to those disclosed in U.S. Pat. No. 10,917,623, B2 implemented in step VIII.

[0135] In some embodiments, the left and right images may be processed simultaneously (eg, each image has its own GPU partition and texture), and the values ​​of the dynamic parameters may be updated every frame, while in other embodiments the dynamic parameter update frequency may be adjusted (reduced or increased).

[0136] It should be understood that this method can be used to generate various stereoscopic display modes as needed, such as autostereoscopic, anaglyph, row interleaved, column interleaved, etc. The visual reproduction method can be open to any color encoding and can vary from device to device. For example, for an autostereoscopic device, the final shader will depend on the display device and may be provided by the device manufacturer, or information about the stereoscopic display specifications may be required to correctly map the rendered image to provide the best results.

[0137] In some embodiments, for example, during cropping step X, the selected area of ​​the left virtual camera may be compared to the selected area of ​​the right virtual camera so that pixels corresponding to portions of the selected area outside the initially captured common area are omitted (blanked / darkened) during rendering. In some embodiments, where this may result in stereoscopic images that are smaller than the intended display screen, border padding may be provided to these images prior to displaying them to ensure that they fill the stereoscopic display.

[0138] Figure 5B A block diagram of inputs and parameters is presented, which may include Figure 5A The parameters mentioned in the for setting virtual environment elements (meshes, textures, 3D objects, view cones, position and orientation of virtual cameras, etc.) contain and summarize the relationship between the inputs described above and the method steps of some embodiments.

[0139] In some embodiments, the proposed method may be used using the following steps: step 501, capturing images with capture cameras having a common portion of their FOVs; step 502, determining and generating one or more polygon meshes, which may require using capture camera parameters 552; step 503, generating a virtual 3D object (e.g., texturing the polygon mesh with the image from the capture camera), which may require using the image captured by the capture camera 553; step 504, positioning the virtual 3D object relative to the virtual camera, which may require using the position and orientation of the optical axis of the capture camera with respect to a line extending between the two capture cameras 554; step 505, determining the orientation of the virtual camera (e.g., calculating the angle between the orientation of the virtual object and the optical axis), which may require using various Step 506, determine the dimensions of the aspect ratio of the viewing cone, which may require the use of the aspect ratio of the display 556; step 507, determine the dimensions of the viewing cone (e.g., the width and height of the viewport), which may require the use of various parameters 557 (e.g., interocular distance; vergence; viewer-screen distance; and / or any zoom selected by the viewer); step 508, determine and apply a lens offset for the virtual camera, which may require the use of various parameters 558 (e.g., interocular distance; vergence; viewer-screen distance; screen size; and / or any off-axis viewing angle of the viewer); and step 509, format (e.g., crop) and generate (e.g., render) a stereoscopic image for stereoscopic display to a viewer.

[0140] Figure 6 Also shown is an example of a recaptured image 62 corresponding to a selected area of ​​a virtual object 39 within a view cone projection 60 (also referred to as a viewport of a virtual camera). In some embodiments, the view cone projection 60 may be defined as a shape or area corresponding to the projection of a view cone (the field of view or view cone of a virtual camera) onto a given virtual object 39. Figure 6In the exemplary embodiment shown, the frustum projection 60 may be a rectangle having an aspect ratio that may be the same as the aspect ratio of the display screen having a width (Ws) and a height (Hs), wherein the width (Wp) and the height (Hp) of the frustum projection may have values ​​corresponding to the product of the corresponding size of the display screen (Ws or Hs) and the scale factor (SF) between the real world and the virtual world. The recaptured image 62 may be used to perform cropping of the virtual medium (selectively enabling or disabling rendering operations of pixels within a determined region of interest) before rendering, so that only pixels of the recaptured image 62 corresponding to the selected region (within the frustum 44) of the left virtual object and the right virtual object 39 may be rendered to optimize GPU resources. In some embodiments, the resolution of the recaptured image within the viewport (frustum projection) to be rendered may be selected to be the selected screen resolution so that the GPU can render the correct number of pixels (subdivisions of the rendered image).

[0141] In some embodiments, the viewer can select the screen resolution to be a maximum resolution, a minimum resolution, or both. The resolution can vary and can include, but is not limited to, values ​​such as: standard definition (480p) with 640x480 pixels, high definition (720p) with 1280x7200 pixels, full high definition (1080p) with 1920x1080 pixels, etc.

[0142] Figure 7 3D schematic diagram showing some of the changes induced on the visual cone, which may be similar to or analogous to Figure 4A The view transforms of the previously rendered ones are associated with each other. Figure 7 In an exemplary embodiment of the present invention, the view transformation may include increasing the angular height 46 of the view cone (also referred to as the vertical line of the field of view) to another modified angular height 46' while maintaining its aspect ratio, so that the initial view cone projection 60 corresponding to the initial view cone 44 can be enlarged to the projection 60' of the modified view cone corresponding to the modified view cone 44' in this embodiment. Such a view transformation may thus modify the output image recaptured with the virtual camera, shown here as the adjusted final image 62'.

[0143] Those skilled in the art will understand that the parameters affecting the viewing cone are not limited to the specification parameters described in this embodiment, and various parameters can be added to the process as needed.

[0144] In these embodiments, it is possible to set several pairs of corrected left and right images using selected techniques to produce a better stereoscopic display for the viewer. L,RMust always be parallel to each other, so the selected technology can be similar to but not limited to the viewer-adjusted stereoscopic image display method described in US10,917,623B2 patent, and can utilize one or a combination of single operations such as cropping, scaling, translation, rotation, etc. In some embodiments, the image can be, for example, no different from that in the present patent reference, and in the case of stereoscopic content, the impression of depth can be given by using parallax differences that usually conflict with other visual cues. One of the main problems may come from the difference between two important information; the visual convergence of the eyes (yaw orientation between the left eye and the right eye) and the distance at which the eyes focus (for example, the distance between the display screen and the viewer's eyes) is called visual accommodation. It should be understood that those skilled in the art can calculate the visual convergence using the viewer's interocular distance (Io) and his distance to the display screen (Ds).

[0145] The brain regularly updates these two pieces of information, taking them into account and comparing them to each other to achieve clear vision and better interpretation of visual information. While in the real world, the brain is accustomed to a certain learned correspondence between convergence and visual accommodation (for example, if one changes, the other information will change in some way), in some embodiments, these two pieces of information may conflict (be inconsistent) with this correspondence, for example, the viewer's eyes can remain focused on the display screen, while the convergence may change. Those skilled in the art know that in the case of convergence and visual accommodation conflict, many adverse effects may occur, such as loss of depth perception or discomfort, pain, and diplopia. In order to adapt to and manage this problem, the maximum or farthest distance (perceived as "inside" the screen) and the minimum or closest distance ("outside" the screen) associated with the angular constraint can be determined as described in the above-mentioned patent reference. In fact, in order to allow the viewer's perception of depth to adapt to this difference between convergence information, these maximum (Df) and minimum (Dc) distances are determined in the US10,917,623B2 patent and can be expressed as a function of the optical base (Bo).

[0146] In some embodiments, the relative reference offset can be used to further scale and / or position the stereoscopic images to make the farthest objects appear closer to the screen and / or to make the closest objects appear closer to the single or multiple screens. This further scaling and / or positioning, while reducing eye fatigue that may result from differences in visual accommodation for focusing on the single or multiple screens and visual accommodation for focusing on near and / or far objects, can be used to keep objects that appear at a given depth appearing at approximately the same depth. In some embodiments, a cropping process can be used to control maximum and / or minimum distances and allow for vergence accommodation.

[0147] In some embodiments, stereo zoom, which means changing the optical base (Bo) defined as the distance between the centers of the two images on the screen, may be calculated using the viewer's interocular distance (Io), the vergence factor (V) which may be an average value, and the screen / viewer distance (Ds) by the following equation:

[0148]

[0149] Wherein, the vergence factor is a selected angular offset of visual convergence (e.g., about 1 or 2 degrees), which can be used to control the maximum angular variation required for focusing between objects at the maximum and / or minimum distances, which implements visual convergence / accommodation compensation, can reduce conflicts with certain visual cues, and thus provide a better stereoscopic experience for the viewer. In some embodiments, the vergence factor can be selected or fixed to a given value. In some embodiments, the vergence factor can be modified or actively adjusted according to the screen / viewer distance (Ds). It should be understood that in embodiments where the vergence factor can be ignored and can be approximately zero (e.g., when Ds is less than about 3 meters), the optical base can also have a negligible value (v=0→Bo=0). As previously described, in some embodiments, such an optical base can be used to determine the maximum (Df) or minimum (Dc) distance of a stereoscopic image by the following exemplary relationships: Df=Ds / (1-Bo / Io) or Dn=Ds / (Bo / Io+1). In some embodiments, by modifying the viewing cone, more specifically the lens offset, the value of the optical base can be used to control and adjust the width and / or depth (distance) perceived by the viewer. In some embodiments, the size of the view frustum (FoV) can be further adjusted to follow the optical base while maintaining the same aspect ratio. In some embodiments, the view frustum can be resized to adjust the size of the projection of the view frustum on the virtual object, referred to herein as the width (Wp) and height (Hp) of the view frustum projection, which can be calculated using the width (Ws) or height (Hs) of the display screen and the scale factor (SF) between the real world and the virtual world using the following equation.

[0150]

[0151] It should be understood that Wp or Hp can also be calculated using the aspect ratio of the screen (Hs / Ws), such as Hp=Wp·Hs / Ws, thus also ensuring that the aspect ratio can be respected. In some embodiments, the viewing cone can be resized by determining and adjusting the angular width (Wa) (also referred to as the horizontal line of the field of view) and the angular height (Ha) (also referred to as the vertical line of the field of view) of the viewing cone, which in some embodiments can be calculated using the following equation.

[0152]

[0153] In some embodiments, the lens offset can be adjusted to allow for a vertical offset of the cone projection from the optical base (shift_PF) and the width of the display screen (Ws) to better maintain the ratio of the real world (capture environment) and the virtual environment, for example, shift_PF = Bo·Wp / Ws = Bo·SF.

[0154] Once all final corrections have been implemented, the resulting pair of left and right optimized images can be projected onto a selected display device using selected techniques and processes for the benefit of the viewer. In some embodiments, a stereoscopic image can be displayed using separate outputs (one for each eye), such as a screen for a projector or virtual reality headset.

[0155] Figure 8 A block diagram of the following hardware electronic components for processing and executing the proposed method for producing a stereoscopic display for viewing by a viewer is presented. It should be understood that all presented components can be interconnected with a circuit, such as a motherboard, allowing communication between many components of the system. Input data 80 can give information about the raw images of the capture camera, their respective parameters, the input parameters of the viewer, and / or other parameters from various capturers or sensors (controller / mouse / touchpad, accelerometer, gyroscope, etc.). A data memory 82 can be used to store a plurality of information, such as instructions for executing the proposed method; the raw images 24 of the capture camera 20 and their associated parameters; the pre-generated polygon mesh 25 and the parameters of its virtual environment; the stereoscopic video produced by the embodiment of the method previously reproduced and recorded; the panoramic image of the scene from two different locations that can be cropped to extract the selected area; or more. It should be understood that the computing device 84 can include a central processing unit (CPU) and a random access memory (RAM), which can allow the required commands to be calculated. Following the CPU instructions, a graphics processing unit (GPU) 86 can be used to calculate and render the stereoscopic image to be displayed for viewing by the viewer. The display 88 for presenting stereoscopic images to the viewer can be a variety of screens or projectors or head-mounted displays or more. It should be understood that the program product containing all instructions for executing the proposed method can be stored in a data storage unit, which can be an external data storage unit, for later processing in an alternative device containing the above components.

[0156] Some embodiments use various images captured with various types of capture cameras to process and generate viewer-adjusted stereo images, where the viewer's viewpoint may be dynamic, such as in a virtual reality (VR) environment. Fig. 9AA schematic diagram of a dynamic head-mounted 3D VR stereoscopic display having two screens (left screen 48L and right screen 48R) or a single screen divided into two pieces, such as a smartphone equipped with compatible VR glasses, each piece presenting a corresponding image in a pair of processed stereoscopic images (left image 94L and right image 94R) to each eye of the viewer, with the left eye 90L having a corresponding left eye field of view 96L and the right eye 90R having a corresponding right eye field of view 96R. Fig. 9B shows a transition state between frames, where a certain change in the orientation of the VR device (here represented as a clockwise rotation) corresponds to a change in the viewer's region of interest, thus changing the viewer's dynamic parameters ( Figure 5A Step V) Fig. 9B In the example, the image 94L / 94R of the previous frame can be represented by the previous dynamic parameters (with Fig. 9A ) are shown to help visualize the orientation changes of the VR device and the parameter changes to be achieved for the following images. Fig. 9C The following adjusted stereo images 94'L / 94'R are presented, which are corrected to be consistent with the perspective of the new dynamic parameter input, processed into each frame and displayed on the display screen 48' L；R Displayed on the screen to facilitate the viewer's new perspective 90' L；R Note that the dynamic parameters of the viewer may be extracted from, but not limited to, a computer's mouse position, accelerometer and gyroscope, or any type of controller (e.g., a video game controller).

[0157] It should be understood that the proposed method can be used in most smartphones equipped with two rear cameras that share a common part of their field of view, regardless of the camera type or its magnification, to capture raw images to be processed within the smartphone to generate stereo images that can be visualized with its display.

[0158] Fig. 10A An illustration of a camera 20 coupled to the rear of a smartphone 100 (eg, the smartphone's camera or an added camera lens that can be clipped onto a smartphone) is shown, wherein cutting plane line AA can be seen. Fig. 10B Presented Fig. 10A The cross-sectional view AA on the right side allows to see the position of the two types of cameras 20L and 20R coupled within their housing 101. In this embodiment, the cameras can be slightly offset from each other by a gap 27, which can be taken into account and corrected by the applicant's method, such as Figure 2A , 2H Similar to 2I described previously.

[0159] Those skilled in the art will appreciate that such a gap need not be as noticeable as the gap 27 of the present example, as it may be a minor defect, perhaps as small as a few microns, that does not affect the viewer's stereoscopic experience. If desired, the proposed method can compensate and correct such defects (e.g., gaps and orientation offsets) between cameras.

[0160] In some embodiments, the orientations (optical axes) 22 of the capture cameras may be parallel, may be nearly parallel (i.e., intended to be parallel by design, but in practice deviate slightly from parallel due to, for example, production tolerances and / or imperfections), or may be non-parallel. The proposed method may compensate for and correct for such offsets between camera orientations, if desired.

[0161] This method can also compensate and correct for magnification differences between cameras if necessary.

[0162] In some embodiments, when the smartphone is placed horizontally, the stereoscopic images can be displayed on the left and right halves of the screen and can be viewed through or with the help of a set of lenses for visualizing stereoscopic images, which can be mounted on the smartphone, such as Google VR Cardboard.

[0163] It should be understood that Fig. 10B It may also represent any alternative camera pair assembly / arrangement sharing a common portion of a captured field of view integrated into a device for capturing images / video, such as a 3D webcam, a 3D scanner, a 3D camera, or any other such device known in the art.

[0164] The proposed method may require better results or may produce better results by completing a characterization of the capture camera parameters (e.g., position, orientation, field of view, lens distortion / warp) of any capture camera. It should be understood that such characterization can be used to adjust or fine-tune the values ​​of the capture camera parameters to be used in the proposed method by characterizing, measuring and / or identifying various minor to significant variations and / or defects (e.g., position and / or angular offsets) that may exist between capture camera parameters of similar devices that are theoretically expected to be identical to each other (e.g., between the same camera model and / or camera components / arrangements).

[0165] It should be understood that a "defective" 3D product / device (e.g., a 3D webcam that may have its capture camera offset from the desired arrangement during the production process at an undesirable angle and / or position) may not provide the viewer with an optimal (or in some cases functional) stereoscopic experience.

[0166] The proposed method can be used to use "defective" products, for example, devices coming out of a production line with camera parameters slightly different from what was designed and expected. It should therefore be understood that the proposed method can be used to optimize, correct and / or allow the use of various devices (webcams, cameras, scanners, etc.) that may come from a production line and have defective capture cameras.

[0167] Depending on the user's requirements, this characterization of capture camera parameters can be done once before the first use of the proposed method (e.g., once for a new device), until before each new image capture is completed (e.g., before taking any new photos), or not at all.

[0168] Fig.11 A block diagram of the main steps of a method embodiment is presented, wherein image source data 110 may be processed with viewer parameters 115 to render a stereoscopic image for stereoscopic display 119. In some embodiments, capture parameters 111 may be used in a step 112 of generating a polygonal mesh, which may be done according to parameters of a capture camera to which the image source data may be attached; texturing the polygonal mesh may be done 113 with the image source (texture) to obtain a virtual image; and positioning the virtual camera may be done relative to the virtual object 114. In step 116, the parameters of the virtual camera may be adjusted relative to the display parameters and viewer parameters 115. After formatting (e.g., cropping) 117 a selected area of ​​the virtual object, the stereoscopic image may be rendered 118 with the selected stereoscopic format prior to stereoscopic display 119.

[0169] Fig.12 A block diagram of some steps is presented that may be required to ensure that each virtual camera can be set up in the correct manner to ensure that the rendering of the stereoscopic image is practical, optimal and realistic and has a geometric correspondence with the real world. The steps may include calculating the position, orientation, lens offset, size of the view frustum (viewport), etc. as described above based on various parameters and values, which may include various parameters of the capture camera, various parameters of the display (resolution, aspect ratio, stereoscopic vision type and / or number of screens), and various parameters of the viewer (static parameters and / or dynamic parameters).

Claims

1. A method of processing images using a graphics processing unit (GPU) for providing a stereoscopic display to a viewer, the images being captured by a left capture camera and a right capture camera having a common portion of a captured field of view, the method comprising: generating a left polygon mesh and a right polygon mesh using specification parameters of the left capture camera and the right capture camera having a central axis, and defining a position of a virtual camera relative to the meshes to correspond to a field of view of the capture camera; texturing the left polygon mesh and the right polygon mesh using at least one image from the left capture camera and the right capture camera, respectively, to create a left virtual 3D object and a right virtual 3D object, respectively; as well as Rendering a left-eye image and a right-eye image using a left virtual camera and a right virtual camera, the left virtual camera and the right virtual camera being oriented to capture at least a portion of the left virtual 3D object and the right virtual 3D object, respectively; The orientation parameters of the left virtual camera and the right virtual camera are determined in the following manner: a) determining in capture space a reference line extending between the position of the left capture camera and the position of the right capture camera; b) determining the orientations of the left virtual camera and the right virtual camera in the capture space so that the orientations of the left virtual camera and the right virtual camera are parallel to each other and located in a selected viewing horizontal plane; c) determining a left orientation offset vector and a right orientation offset vector, which are defined as the amount by which the orientations of the left virtual camera and the right virtual camera change relative to the orientations of the optical axes of the left capture camera and the right capture camera corresponding to the central axis; d) orienting the left polygon mesh and the right polygon mesh around the central axis in virtual space relative to the left virtual camera and the right virtual camera, respectively, using the left orientation offset vector and the right orientation offset vector; and e) determining viewing cone parameters of the left virtual camera and the right virtual camera using display parameters, wherein the display parameters include display size, distance from the display, interocular distance, and viewing direction.

2. The method of claim 1, wherein the display parameters include a magnification, wherein the viewing cone parameters are determined using the magnification.

3. The method according to claim 1 or 2, wherein the step of orienting comprises determining lens offset parameters of the left virtual camera and the right virtual camera. 4 . The method of claim 1 , wherein an optical axis orientation of the left capture camera is coplanar with an optical axis orientation of the right capture camera. 5 .

5. The method according to any one of claims 1 to 4, wherein the optical axis of the left capture camera is parallel to the optical axis of the right capture camera, wherein the left capture camera and the right capture camera have different fields of view, and wherein at least one of the optical axis of the left capture camera and the optical axis of the right capture camera is non-perpendicular to the reference line. 6 . The method according to claim 1 , wherein the left-eye image and the right-eye image are still images.

7. The method according to any one of claims 1 to 6, wherein the left-eye image and the right-eye image are video images.

8. The method according to any one of claims 1 to 7, wherein the left-eye image and the right-eye image are displayed to the viewer on a single screen of a stereoscopic display. 9 . The method according to claim 1 , wherein the left-eye image and the right-eye image are generated as anaglyph format images.

10. The method according to any one of claims 1 to 8, wherein the left-eye image and the right-eye image are generated as column-interleaved format images displayed on an autostereoscopic display.

11. The method according to any one of claims 1 to 8, wherein the left-eye image and the right-eye image are generated as a sequence of page-turning images viewed with shutter glasses.

12. The method according to any one of claims 1 to 8, wherein the left-eye image and the right-eye image are generated as a sequence of line-interleaved images for polarization display.

13. The method according to any one of claims 1 to 7, wherein the stereoscopic display comprises a left image display and a right image display, wherein the left-eye image is displayed to the left eye of the viewer using the left image display, and the right-eye image is displayed to the right eye of the viewer using the right image display.

14. The method according to any one of claims 1 to 12, wherein the viewer comprises a plurality of viewers, and the inter-eye distance is selected to be a minimum inter-eye distance among the plurality of viewers.

15. The method of any one of claims 1 to 14, wherein the relative orientation between the left capture camera and the right capture camera is divergent.

16. The method of any one of claims 1 to 14, wherein the relative orientation between the left capture camera and the right capture camera is parallel.

17. The method of any one of claims 1 to 14, wherein the relative orientation between the left capture camera and the right capture camera is converging.

18. A method according to any one of claims 1 to 17, wherein at least one of the left-eye image and the right-eye image has an aspect ratio different from an aspect ratio determined by the display size, and the method further includes providing border padding for at least one of the left-eye image and the right-eye image displayed to the viewer.

19. A method according to any one of claims 1 to 18, wherein the captured field of view includes a panoramic image, the method further comprising determining a selected optical axis within a portion of the panoramic image contained in the common portion of the captured field of view, wherein the portion of the panoramic image and the selected optical axis are processed using the steps of the method.

20. A device for processing a stereoscopic image for display to a viewer, the device comprising a graphics processing unit, a central processing unit and a memory readable by the central processing unit, the memory storing instructions for executing the method according to any one of claims 1 to 19.

21. A computer program product comprising a non-volatile memory storing instructions for causing a processor connected to a GPU to execute the method according to any one of claims 1 to 20.

Images