Stereoscopic display tailored to the viewer

Abstract

To provide a method, device, and program for processing stereoscopic photographed images to generate an optimal stereoscopic image pair matching viewing conditions.SOLUTION: A stereoscopic video playback device provides stereoscopic video of viewing conditions with new display / screen parameters on a single screen for captured images (right eye image and left eye image) taken using parallel axis cameras, such that an image processor 25 processes the original stereoscopic image pair with the crop and scale parameters generated by a crop and scale parameter calculator 20.SELECTED DRAWING: Figure 16C

Description

[Technical Field]

[0001] This application claims priority to U.S. Provisional Patent Application No. 62 / 551,942, filed August 30, 2017, and U.S. Provisional Patent Application No. 62 / 627,825, filed February 8, 2018, the contents of which are incorporated herein by reference.

[0002] This application relates to stereoscopic image displays. [Background technology]

[0003] Stereoscopic video or cinematography is an art. Positioning the cameras to capture left and right video streams for playback with the best 3D effect is not an easy task. Camera placement requires knowledge of the cinema and audience placement, as well as an understanding of how most people perceive 3D under those viewing conditions. In the field of stereoscopic cinematography, it is generally accepted that placing two cameras so that their optical axes converge results in the best 3D effect. The camera spacing and convergence angle are selected based on knowledge of the viewer's average distance from the display screen, the average eye separation, and the average viewing position relative to the center of the screen. Failure to observe these viewing conditions compromises the quality of the 3D experience.

[0004] If viewing conditions are to be changed from those originally intended by the stereoscopic cinematographer, it is known in the art to reformat stereoscopic video for the new viewing conditions. Typically, reformatting involves analyzing stereoscopic image pairs to determine the depth of individual pixels, and then generating stereoscopic image pairs using the original image and depth information so that a stereoscopic image stream suitable for the new viewing conditions can be recreated. Such reformatting is computationally intensive and is performed in the new viewing conditions. If viewing conditions change, the computationally intensive process is repeated. Summary of the Invention

[0005] Applicant has discovered that the potential loss in quality of the 3D experience caused by the use of parallel non-converging cameras is overcome by an improvement in the quality of the 3D experience when such stereoscopic video is reformatted to fit the viewing conditions of a viewer on a single screen.

[0006] Therefore, a playback device is provided that processes the original stereoscopic image pairs that were presented for viewing under the original viewing conditions by scaling and cropping to present stereoscopic video for new viewing conditions on a single screen.

[0007] To avoid reformatting the stereoscopic images as described above, it is possible to display the stereoscopic images originally intended for display in a first field of view on a new single display having a second field of view.

[0008] Applicant has further discovered that capturing and storing 3D images using a parallel-axis camera with a wider field of view than would normally be expected to be used for viewing has the advantage of allowing the recorded 3D images to be processed on the viewing device (or within the viewing system) when viewed in a wider range of viewing conditions.

[0009] A broad aspect is a method of processing stereoscopic images for display to a viewer on a single screen, the stereoscopic images being captured using parallel-axis cameras having a first field of view. The method includes defining a second field of view provided by the single screen, a viewer's interocular distance Io, and a viewing angle Ib. and a distance between the viewer and the single screen relative to the first field of view, thereby displaying an image on the single screen at a distance from the viewer relative to the first field of view, cropping the image relative to the screen if the scaled stereoscopic image is larger than the screen, and providing a border for the image relative to the screen if the scaled stereoscopic image is smaller than the screen.

[0010] In some embodiments, the method may further include selecting a zoom window within the stereoscopic image to modify the first field of view, and the stereoscopic image may be scaled relative to the modified first field of view.

[0011] In some embodiments, the zoom window may be offset from the center of the stereoscopic image to allow viewing of an area of ​​interest within the stereoscopic image.

[0012] In some embodiments, viewer input may be used to move the offset while viewing the stereoscopic image.

[0013] In some embodiments, the stereoscopic image may be a still image.

[0014] In some embodiments, the stereoscopic image may be a video image.

[0015] In some embodiments, the stereoscopic images may be converted into combined anaglyph format images.

[0016] In some embodiments, the stereoscopic images may be converted into column-interleaved images for display on an autostereoscopic display.

[0017] In some embodiments, the stereoscopic images may be converted into a series of page-flipping images for viewing with shutter glasses.

[0018] In some embodiments, the stereoscopic image may be converted into a series of line interleaves for polarized display.

[0019] In some embodiments, the method may include obtaining user input to obtain a definition of a second field of view provided by the single screen.

[0020] In some embodiments, the method may include acquiring sensor data to obtain a definition of a second field of view provided by the single screen.

[0021] In some embodiments, the stereoscopic images may be arranged on a single screen to correspond to an object spacing of Io between the right-eye and left-eye images of a distant object.

[0022] In some embodiments, the viewer includes multiple viewers, and the interocular distance Io may be selected to be the smallest interocular distance among the multiple viewers.

[0023] In some embodiments, the stereoscopic images are further scaled and / or positioned using a relative base offset to make the most distant objects appear closer to the screen and / or make the closest objects appear closer to the screen, with the goal being to accommodate differences between eye accommodation to focus on a single screen and eye accommodation to focus on near and / or far objects. This further scaling and positioning of the relative base offsets allows objects that appear at a single screen depth to remain at the same depth.

[0024] Another broad aspect is an apparatus for processing stereoscopic images for display to a viewer on a single screen, the apparatus comprising a processor and a memory readable by the processor, the memory storing instructions for performing the methods defined herein.

[0025] Another broad aspect is a computer program product including instructions stored in a non-transitory memory for a processor or reconfigurable hardware that performs the methods defined herein. [Brief explanation of the drawings]

[0026] The present invention will be better understood from the following detailed description of embodiments thereof, taken in conjunction with the accompanying drawings, in which:

[0027] [Figure 1A] FIG. 1 is a diagram of an exemplary parallel camera system.

[0028] [Figure 1B] The top image is taken with the left camera and the bottom image is taken with the right camera.

[0029] [Figure 1C] This figure shows how each image is qualitatively positioned within the frame of a single screen, with a magnification factor of 1 and an appropriate horizontal offset to accommodate a viewer's interocular distance where the display field of view is larger than the capture field of view.

[0030] [Figure 1D] 1 shows a schematic diagram of the change in field of view depending on the viewing distance to the screen.

[0031] [Figure 1E] 1 shows a schematic representation of the change in field of view for a fixed viewing distance as screen size changes.

[0032] [Figure 1F] This figure shows how each image is qualitatively positioned within the frame of a single screen, with a magnification factor of 1 and an appropriate horizontal offset to accommodate a viewer's interocular distance where the display field of view is smaller than the capture field of view.

[0033] [Figure 1G] This figure shows how each image is qualitatively positioned within the frame of a single screen, with the appropriate horizontal offset to correspond to the interocular distance of a viewer with a magnification of 1.5 and a display field of view approximately the same as the capture field of view.

[0034] [Figure 1H]This figure shows how each image is qualitatively positioned within the frame of a single screen, with an appropriate horizontal offset to correspond to the interocular distance of a viewer with a magnification of 0.5 and a display field of view approximately the same as the capture field of view.

[0035] [Figure 2] FIG. 1 illustrates ratios tied to parallax calculations for an exemplary parallel camera system.

[0036] [Figure 3A] FIG. 1 is a diagram of dual parallel screens S1 and S2 placed in front of a user.

[0037] [Figure 3B] FIG. 1 illustrates ratios for calculating the perceived distance of an object Op on dual parallel screens S1 and S2 placed in front of a user.

[0038] [Figure 4A] This is a diagram corresponding to width perception in the real world.

[0039] [Figure 4B] 1A and 1B are diagrams corresponding to monoscopic width perception in the real world and the perceived world.

[0040] [Figure 5] A diagram of the left-eye screen, one of the dual screens in a stereoscopic system, showing the line occupying exactly the right half of the image displayed on the screen.

[0041] [Figure 6A] FIG. 10 is a diagram showing the proportions of objects perceived in the real world at distance Drn.

[0042] [Figure 6B] FIG. 1 illustrates how objects are perceived on the left screen of a dual screen system in the perceived world.

[0043] [Figure 7] FIG. 1 is a diagram of dual screens S1 and S2 of an exemplary stereoscopic system, where S1 and S2 are perpendicular to an imaginary line Io between the right and left eyes, and S1 and S2 are centered at the pupil of the left eye and the pupil of the right eye, respectively.

[0044] [Figure 8A] FIG. 1 is a diagram of dual parallel screens S1 and S2 placed in front of a user.

[0045] [Figure 8B] FIG. 1 is a diagram of different ratios related to where an object Op is perceived when a user is facing a dual screen system at a distance Ds from the viewer's eyes.

[0046] [Figure 9A] FIG. 1 is a diagram of two theoretically overlapping screens S1′ and S2′ located further away from the user than the dual screens S1 and S2.

[0047] [Figure 9B] 1 shows the ratios associated with how the right eye perceives an object Op on at least part of the dual screens S1′ and S2′.

[0048] [Figure 9C] A schematic image from the left eye camera includes the distant sun near the optical axis and a central tree along the optical axis.

[0049] [Figure 9D] A schematic image from the right-eye camera with its axis parallel to the optical axis of the left-eye camera, thus showing the distant sun aligned perpendicular to the optical axis, and the central tree offset to the left.

[0050] [Figure 10A]FIG. 1 is a diagram of a single screen with two overlapping sections S1' and S2' located further from the user than the dual screens S1 and S2.

[0051] [Figure 10B] FIG. 1 shows different measures linked to how a user perceives an object Op on a single screen with two overlapping sections S1′ and S2′.

[0052] [Figure 10C] 9C is a schematic left-eye image for display on a typical screen, the image corresponds to the camera image of FIG. 9C, with an interocular offset of Io / 2 to the left included in the displayed image, and the image shows the sun in the distance perpendicular to the optical axis and a tree in the center perpendicular to the left eye's optical axis.

[0053] [Figure 10D] 10C and 10D , where the image corresponds to the camera image of FIG. 9D , and an interocular offset of Io / 2 to the right is included in the displayed image, and the image shows a distant sun aligned perpendicular to the optical axis and a central tree offset to the left of the right eye's optical axis, and the interocular distance Io is shown between the distant sun object in the images of FIG. 10C and FIG. 10D .

[0054] [Figure 11A] FIG. 1 shows a simple screen system in which parts of the screens S1′ and S2′ are shared by both eyes.

[0055] [Figure 11B] FIG. 1 shows measurements of a simple screen system corresponding to the part of the screen S1′ and S2′ shared by both eyes.

[0056] [Figure 12A] FIG. 10 shows a partial image of an image intended for the left eye by using a simple screen with the same ratio Ds / Ls1′ as the dual screen system.

[0057] [Figure 12B] A diagram showing a partial image of the image for the right eye by using a simple screen having the same ratio Ds / Ls1’ as the dual screen system.

[0058] [Figure 13] A diagram showing sections of the original image viewed as the left-eye final image and the right-eye final image.

[0059] [Figure 14A] Since Lse = Ls1’, a diagram of an exemplary single screen system where the width of screen S1’ is compared with the effective width of the image and there is no need to adjust the image perceived by the left eye.

[0060] [Figure 14B] Since Lse < Ls1’, a diagram of an exemplary single screen system where the width of screen S1’ is compared with the effective width of the image and it is necessary to add black strips on both sides to the image perceived by the left eye.

[0061] [Figure 14C] Since Lse > Ls1’, a diagram of an exemplary single screen system where the width of screen S1’ is compared with the effective width of the image and it is necessary to cut the image perceived by the left eye.

[0062] [Figure 15A] A diagram of an exemplary single screen system perceived by the user when Lse = Ls1’.

[0063] [Figure 15B] A diagram of an exemplary single screen system perceived by the user when Lse < Ls1’.

[0064] [Figure 15C]FIG. 10 is a diagram of an exemplary single screen system as perceived by a user when Lse>Ls1′.

[0065] [Figure 16A] FIG. 1 is a block diagram of an exemplary stereoscopic system for cropping and scaling images for viewing on a display. [Figure 16B] FIG. 1 is a block diagram of an exemplary stereoscopic system for cropping and scaling images for viewing on a display. [Figure 16C] FIG. 1 is a block diagram of an exemplary stereoscopic system for cropping and scaling images for viewing on a display.

[0066] [Figure 17A] FIG. 1 is a diagram of image acquisition or rendering using a virtual camera in an exemplary volume reader method.

[0067] [Figure 17B] FIG. 10 is a diagram of a single screen format in an exemplary volume reader method.

[0068] [Figure 18A] This is a graph of object depth when the scale depth is 1, i.e., when objects are perceived at the same distance as in the real world.

[0069] [Figure 18B] This is a graph of object depth when the depth scale is less than 1, i.e., when objects are perceived closer than in the real world.

[0070] [Figure 18C] This is a graph of object depth when the depth scale is greater than 1, i.e., when objects are perceived as farther away than they would be in the real world.

[0071] [Figure 19A]The geometric arrangement of Io and Bo for the image displayed on the screen is shown.

[0072] [Figure 19B] Shows the resulting change in object width as a result of changing Bo.

[0073] [Figure 20A] This shows the geometry of distant objects when Bo is equal to Io. [Figure 20B] 1 shows the effect of using an optical base Bo lower than Io on the appearance of distant objects.

[0074] [Figure 21A] 1 is a graph showing that perceived space is not linear.

[0075] [Figure 21B] 1 is a graph showing the ratio between real world distance and perceived world distance.

[0076] [Figure 22] FIG. 1 is a diagram of a nearby object that is out of the field of view, making it difficult to perceive the depth of the nearby object due to conflict with the associated screen edge or frame.

[0077] [Figure 23A] The same vergence and focal distances are shown schematically for real-world viewing. [Figure 23B] 10A and 10B illustrate schematic diagrams of the convergence distance and closer focal distance of screens for stereoscopic 3D viewing. [Table 1]

[0078] [Figure 24A] FIG. 1 shows the viewing geometry for convergence behind the screen.

[0079] [Figure 24B]FIG. 1 illustrates the geometric arrangement of screen vergence angle and object vergence angle.

[0080] [Figure 25A] Illustrates the viewing geometry when converging in front of the screen.

[0081] [Figure 25B] 1 shows the geometry of the screen vergence angle and the object vergence angle. DETAILED DESCRIPTION OF THE INVENTION

[0082] Before describing the geometry behind the image processing techniques involved in the embodiments described herein, a qualitative overview of image processing is provided.

[0083] FIG. 1A shows a schematic of how parallel cameras, i.e., a left camera and a right camera, can be arranged to capture the same scene. The two cameras can have the same properties, such as resolution, focus, and field of view, and they have parallel optical axes. The two cameras can be separated by a distance that can correspond to the distance between the viewer's eyes. An object at infinity appears at the same position in each camera image. Closer objects have different parallax depending on the object's position in the field of view, including the distance between the cameras and the object's distance from the cameras.

[0084] In Figure 1B, the left-eye image is shown over the right-eye image, with the sun appearing in the same position in each image. The tree in the center is in a different position in the two images due to parallax.

[0085] As shown in Figure 1C, the camera image in Figure 1B must be modified for display on a single screen. Single-screen viewing can be achieved using known techniques. For example, anaglyph color filter glasses can be worn by the viewer, and the screen image consists of both left and right color-encoded image data. For page-flip operations, the viewer can wear shutter glasses that allow the right and left eyes to see in alternating time slots while the screen image alternates between the right and left images. In autostereoscopic displays, the viewer does not need glasses, but the screen contains lenses or a screen mask that allow the right eye to see the right-eye pixels and the left eye to see the left-eye pixels.

[0086] The field of view (FOV) of the display screen in FIG. 1C is larger than the original FOV of the camera images. As shown in FIG. 1C, each image is qualitatively placed within a window or frame of the single screen, offset laterally appropriately to correspond to the interocular distance of the viewer. This distance may vary from person to person. As explained below, if the screen is to be viewed by two viewers, it may be best to use the minimum interocular distance of the viewers to avoid discomfort for the viewers. The resulting images are then displayed on the single screen according to stereoscopic display techniques.

[0087] It will be appreciated that as a viewer changes their distance from the screen, the FOV of the display or screen changes, as shown in Figure 1D. The closer the viewer is to the screen, the larger the FOV. Similarly, for a viewer at a constant distance from the screen, the FOV will be larger for a larger screen than for a smaller screen. Figures 1D and 1E are important for qualitatively understanding the dependencies between FOV, viewing distance, and screen size.

[0088] In the embodiment of FIG. 1C, the display screen provides a larger FOV than the original FOV, so some of the borders may be padded or blacked out. In the embodiment of FIG. 1F, the display FOV is smaller than the capture FOV. This means that the display screen is essentially too small for the viewing distance. This is qualitatively illustrated in FIG. 1F. As shown, the original capture image is cropped so that the two images can be combined to fit on the display screen. Some of the edges of the original capture image are lost, but the image remains three-dimensionally faithful to the original capture.

[0089] In the embodiment of FIG. 1G, the stereoscopic output should be enlarged by 1.5 times. Qualitatively, it can be seen that the image of FIG. 1B (repeated on the drawing sheet for ease of understanding) is first enlarged, and from the enlarged image, a portion that can fit on the display screen is extracted and placed within a single display screen with a suitable interocular offset (Io) according to the stereoscopic display technology. The display screen FOV can be the same as the capture FOV, but as a result of the enlargement, important boundary portions of the capture image will be lost. However, the stereoscopic effect of the enlarged image is easy to view.

[0090] Scaling an image in the manner shown in Figure 1G causes objects to appear closer because the magnification factor affects object size and perceived disparity, creating the impression that zooming involves getting closer to objects in the image. The perception of depth variations between objects in the scene is reduced or flattened, but the image remains aligned with the two eyes, so the 3D effect still works well even when zoomed in.

[0091] The ability of the stereoscopic effect to withstand adjustments to the originally captured image according to changing viewing conditions is facilitated by the originally captured image from a parallel axis camera, as it will be appreciated that a camera with axes that are close to parallel will provide an adequate image.

[0092] It will be appreciated that the magnification of the captured image illustrated in Figure 1G need not be taken about the center of the captured image, but rather that a window of interest is effectively selected when performing such a magnification. It will further be appreciated that this functionality allows the viewer to move the window of interest in a manner that simulates panning of the original captured scene.

[0093] In the embodiment of FIG. 1H, the stereoscopic output should be scaled up (i.e., shrunk) by 0.5 times. Qualitatively, it can be seen that the image in FIG. 1B (repeated on the drawing sheet for ease of understanding) is first scaled down, and the smaller image is placed within the single display screen at a suitable interocular offset (Io) in accordance with stereoscopic display techniques. The display screen FOV can be the same as the capture FOV, but as a result of the shrinking, no part of the captured image is lost. The magnification factor can be selected so that the image fits exactly within the available FOV of the single display screen. As before, in the embodiment of FIG. 1H, the perception of depth variation between objects in the scene increases, but the image remains aligned with the two eyes, so the 3D effect still works well even when scaled up.

[0094] While certain embodiments have been described qualitatively, other embodiments are described below using precise geometric calculations.

[0095] Captured with parallel cameras

[0096] A stereoscopic capture system consisting of two identical cameras arranged in a parallel configuration, as shown in Figure 1A, is a system in which the disparity of an object captured by such a stereoscopic system is the measured difference between the position of this object on the image captured by the left camera and the position of this same object on the image captured by the right camera.

[0097] As shown in Figure 2, these two cameras are defined as having sensors with a width Lc and a focal length of length F. The centers of their respective lenses are placed a distance B from each other. This distance is called the base. If an object is exactly on the central axis of the left camera, then this object will be represented exactly at the center of the left camera's sensor. The disparity of this object is equal to the distance between the point formed by this object on the right camera's sensor and the center of the right camera's sensor, which is shown by segment pc in the graph above. The disparity of an object located at distance Drn can be determined by comparing sides Drn and F with the equivalent right triangle with corresponding sides Pc and B. We get Pc=B*F / Drn.

[0098] Depth Scale-Parallel Screen

[0099] To view stereoscopic images, a parallel-screen system can be used, in which each eye is presented with a separate image (left image and right image) on a separate screen. These two identically sized screens (designated S1 and S2) are aligned directly with the center of each eye's pupil (see Figure 3A).

[0100] If we take an object whose representation on the left screen is at Og and which lies on an axis perpendicular to the screen and passing through its center, then its representation on the right screen is at a distance Ps (screen parallax) from Od or the center of the screen. Thus, the perceived distance of object Op given by the disparity information is Dp. Given two equivalent right triangles and matching sides, we obtain the following ratio:

number

[0101] The following simplifications can be made:

number

[0102] The screen parallax (Ps) can be calculated by multiplying the sensor parallax (Pc) by the on-screen magnification. This magnification (M) corresponds to the ratio between the effective width of the image displayed on the screen (Lse) and the width of the captured image, which for all practical purposes is equal to the sensor width (Lc). In this case, we ensure that the image displayed on the screen is the image that was originally captured in its entirety. We get:

number

[0103] Combining the two previous equations, we get:

number

[0104] For a given audience member sitting a fixed distance from a given stereoscopic screen and viewing content shot at a fixed base, Io, B, Ds, Lse, Lc, and F can be said to be constant. Then the equation becomes:

number

[0105] In other words, the depth perception represented by a stereoscopic system is linearly proportional to that in the real world, with the depth scale equal to C. For a unitary change in distance in the real world (capture), there is a C-variation in the distance perceived by the observer (visualization). When Epr=1, the depth perception is identical to the real world. If Epr<1, the observer perceives the world as shallower than it actually is. If Epr>1, the observer perceives a deeper world than it actually is.

[0106] Spatial Scale - Parallel Screen

[0107] Knowledge of relative distance is essential to ascertaining the actual width of an object observed in the real world. Indeed, in monoscopic vision, objects of different sizes placed at different distances can give the impression of being the same size. This is illustrated in Figures 4A and 4B, where lines Lr1, Lr2, and Lr3 are all three different lengths, but appear to be the same length to an observer blind to their relative distances, and therefore, in monoscopic vision, Lp1 = Lp2 = Lp3.

[0108] Therefore, the perceived width of an object is directly related to the distance information of the observer of this object.

[0109] A stereoscopic image is obtained that displays a line that occupies exactly the right half of the image displayed on the left eye's screen in a stereoscopic system with parallel screens, as shown in Figure 5. The width of this line on the screen is therefore equal to Lse / 2.

[0110] In stereopsis, this line may be at an apparent distance different from the distance separating the viewer from the screen. As shown in Figures 6A and 6B, this line can be assumed to be perceived at a distance Dp from the viewer. At this perceived distance, the line has a perceived width Lp. In other words, , this line is perceived as being much wider because it is located further away.

[0111] Since there are two equivalent right triangles, it can be established as follows:

number

[0112] We have shown above how to calculate the perceived depth (Dp) of an object in such a stereo system, and substituting this calculation into the Dp term in the above equation gives:

number

[0113] We can now determine that a line of real-world width (Lrn) forms a line in the left eye image of perceived width Lp. This line occupies the entire right portion of the left eye image, and therefore occupies fully half of the sensor of the camera that captured this image, as shown in the graph on the previous page. By applying Thales' theorem, we establish that:

number

[0114] It is established that the scale ratio between the perceived width of this line and any object in the perceived world (Lp) and their real-world equivalent (Ln) is as follows:

number

[0115] In other words, the width perception represented by the stereo system is linearly proportional to that in the real world, with a spatial scale equal to Io / B. For each variation in width in the real world (capture), there is an Io / B variation in width perceived by the observer (visualization). When Esp=1, the perception of width is the same as in the real world. If Esp<1, the observer perceives the world as smaller than it actually is (i.e., squeezed). · If Esp>1, the observer perceives the world as larger than it actually is (i.e., magnification).

[0116] Proportion of three-dimensional expression

[0117] Knowing the depth and spatial scale of a three-dimensional representation, one can ascertain the proportional scale of this representation. This proportional scale is intended to determine whether the representation is flattened, extended, or proportional to reality. The ratio is established as follows: Z=Epr / Esp ·When Z=1, the observer perceives a proportional world (desirable). · When Z<1, the observer perceives a flattened world (more pleasant, less effective). · If Z>1, the observer perceives a stretched world (wider, more spectacular). If Z is equal to 1, then Epr=Esp, and therefore,

number

[0118] In other words, the captured field specified by a focal point and sensor pair is equal to the field of view of a stereoscopic system specified by a pair of image width (screen) and image distance. A system with a screen parallel to the ratio Ds / Lse will provide an equivalent experience regardless of screen size, and that ratio is given by Io / B. For example, a stereo image captured for a 10m wide screen with an observer 30m from the screen (Ds / Lse=3) will provide the same stereoscopic experience on a 10cm screen with an observer 30cm from the screen (Ds / Lse=3).

[0119] However, there is a problem associated with the fact that above a certain size (wider than Io), parallel screens touch each other. This makes the use of parallel displays impractical unless magnifying lenses such as a stereoscope or virtual reality headset are used, severely limiting their use. In the next section, we will explain how to circumvent this limitation and use the parallel camera method for presentations on very large single screens such as 3DTV or movie screens.

[0120] Single Screen Conversion

[0121] The equations developed above only work for parallel screens, i.e., screens perpendicular to an imaginary line separating the two eyes and with centers located exactly at the centers of the pupils of each eye.

[0122] It has been demonstrated above that stereoscopic representations on systems with the same Ds / Lse ratio (the ratio of the distance to the screen to the width of the image displayed on the screen) will provide an identical experience in all respects, i.e. the perceived size and distance of objects will be exactly the same.

[0123] Take an object Op with a perceived distance of Dp, as shown in Figures 8A and 8B. This point is represented by point Og on the left eye screen (S1) and point Od on the right eye screen (S2). Point Og is located exactly on the central axis of screen S1, while point Od is located a distance Ps from the center of screen S2. Thus, the two eyes converge on point Op, which is where the observer perceives this point to be localized, as shown in Figure 8B.

[0124] As shown in Figures 9A and 9B, we take two theoretical screens S1' and S2', which have the same ratio Ds / Lse as screens S1 and S2, but are located further away from the screens. These screens are theoretical because they overlap, and not in the real world. This is not possible. Therefore, we can see that Ls1 / Ds1 is equal to Ls2 / D2, which is also equal to Ls1' / Ds1' and Ls2' / Ds2'. Since screens S1 and S1' are at the center of the left eye's pupil, we can argue that points Og and Og' both lie on the central axis of screens S1 and S1', respectively. As shown in Figures 9A and 9B, points Od and Od' are located at distances Ps and Ps' from the centers of screens S2 and S2', respectively.

[0125] For point Op to be perceived in the same location in the two representations, points Od and Od' must form the same angle, or the ratio Ps / Ds2 must be equal to Ps' / Ds2'. Since S2' is a linear magnification of S2, we know that Ps' will have the same magnification compared to Ps. In other words, Ls2' / Ls2 = Ps' / Ps. Since the system is designed around this constraint, we also know that Ls2' / Ds2' = Ls2 / Ds2. Therefore, we can deduce the following:

number

[0126] Therefore, when using either of these two systems, the Op object is perceived in the same location, thus demonstrating that the two systems provide an identical and comparable stereoscopic experience in all respects.

[0127] As shown in Figures 9C and 9D, images captured using a parallel-axis camera show distant objects, such as the sun, in the same position, while closer objects are in different positions. Such images can be viewed using a head-mounted display.

[0128] 9B, the image seen by each eye may be resized or scaled to fit onto a screen placed at a first depth corresponding to where objects Od and Og are found, or to fit onto a screen placed at a second depth corresponding to where objects Od' and Og' are found. Thus, a small screen used at a first depth can be replaced with a larger screen at a second depth that provides the same field of view. Scaling the image for the larger screen may change Od to ​​Od', but the stereo position of object Op remains the same.

[0129] As will be appreciated from the discussion below with reference to Figures 10 to 13, while the images of Figures 9C and 9D can be scaled as a function of screen position as described above, scaling images 10C and 10D will adversely affect interocular distance, and therefore any scaling requires a positional offset (or maintaining the position of the left-eye and right-eye axes during the scaling process) to maintain interocular distance.

[0130] If a different screen size is required for either of the two depths, the field of view changes due to image scaling. In monocular viewing, the field of view is generally appreciated more if a normal field of view is provided and the resolution is of high quality. Nevertheless, viewers can sit closer or farther from the screen, or change from a 30-inch screen to a 50-inch screen at the same viewing distance, and the ability to display a monocular image is not adversely affected by changing the field of view of the original image. In stereoscopic viewing, changing the field of view reduces the perception of the stereoscopic depth of an object.

[0131] For example, referring to Figure 9B, if the image displayed to the right eye were displayed on a larger screen at a second depth, Od' would appear further to the left as a result of scaling to fit the larger screen at the same second depth. Object Og' would thus appear closer in depth because object Og' would remain in the same central position in the left-eye image. This would cause distortion in the stereoscopic display.

[0132] In fact, if a larger screen is used at a second depth, a larger screen can be used to display an image with the same field of view without adversely affecting stereoscopic vision, which may involve placing a border around the image on the screen, without changing the effective field of view.

[0133] If you use a smaller screen at the second depth, you can use the smaller screen. A portion of the image can be displayed. This is like looking at the world through a smaller window, in the sense that objects displayed on the smaller display are the same size as objects displayed on the larger display, and only a portion of the original field of view is displayed on the smaller screen. If the smaller screen has the same resolution as the larger screen, the image is enlarged and cropped to maintain the same object size and show only a portion of the image. The edges of the original image are lost, but the stereoscopic effect is not distorted due to the use of the smaller screen at the second depth.

[0134] Now that this equivalence has been established, we can translate it into a stereoscopic system based on a single screen, as shown in Figures 11A and 11B. To do this, a single screen is considered, centered on each pupil, as two partial sections of two separate screens, as shown in Figures 11A and 11B. In fact, a screen S is taken, the center of which is on an axis perpendicular to and centered on the observer's two eyes. This screen is a partial representation of screen S1' (the right portion) and screen S2' (the left portion), and the field of view of each eye is said to be asymmetric (wider on one side of the center of both eyes than on the other).

[0135] 10C and 10D, when images are viewed dichroically on the same display, for example, using anaglyph glasses to view an anaglyph image (e.g., cyan for the right eye and red for the left eye), LC shutter glasses to view alternating left-eye and right-eye images, or an autostereoscopic display, the images contain distant objects with differences in Io. Images captured using parallel-axis cameras have distant objects, such as the sun, in the same position, and closer objects in different positions relative to the distant objects, once the offset Io is taken into account.

[0136] It will be appreciated that scaling a stereoscopic image captured by a camera for a first field of view for display on a screen for an observer with a second field of view is not limited to displaying the entire stereoscopic image. As shown in Figures 9C, 9D, 10C, and 10D, a region of interest zoom window can be selected within the source stereoscopic image. This window provides a first field of view that is smaller than the entire source image, but the window can be acquired as a source image and displayed as described herein. Selecting the window may result in less cropping of the image to fit on the new screen.

[0137] This window selection does not have to be in the center of the image, and is shown slightly to the left of the image in the figure. This window selection can be controlled by user input, allowing navigation of the viewing window within the source image.

[0138] Thus, the source image can be a wide-angle or panoramic scene, and window selection allows the viewer to explore the scene by changing the viewing direction within it.

[0139] As shown in Figures 12A and 12B, to obtain an experience equivalent to that of a parallel screen system using a single screen with the same ratio Ds / Ls1', a partial image of the image intended for the left eye (right portion) and a partial image of the image intended for the right eye (left portion) are displayed, and the partial images are calculated in the following way:

[0140] For the left eye, as shown in Figures 12A and 12B, the right half of the width of the screen S1' is equal to the screen width divided by 2 (Ls / 2) plus the interocular distance divided by 2 (Io / 2). Therefore, the complete screen width S1' is equal to Ls / 2 + Io / 2 multiplied by 2, or Ls1' = Ls + Io. Because the screen can only display the right portion of the left-eye image (Ls1' - Io), the portion of the left image equivalent to Io is cropped. The image is cropped according to the ratio Io / Ls1' or Io / (Ls + Io).

[0141] For example, consider an image with a resolution of 1920x1080 to be displayed on a 140cm wide screen (observer interocular distance is 6.5cm). The left part of the image for the left eye needs to be cropped by 85 pixels.

number

[0142] To maintain the aspect ratio of the original image, all images are cropped vertically by the same amount.

number

[0143] Thus, the final image for the left eye will be a section of the original 1835x1032 image, as shown in Figure 13. It should be noted that the vertical portion of the image can be any part of the image (top, bottom, center, etc.), as long as the number of pixels is taken into account and the same selection is made for both eyes (stereoscopic arrangement). To obtain the right eye image, simply make the equivalent left section of the original image for the right eye one of the 1835x1032 sections that is vertically aligned with the left eye section.

[0144] These images can be scaled back to the resolution on the screen they are displayed on without affecting the final stereoscopic result. Thus, the formulas for obtaining the final horizontal and vertical resolution of the image are:

number

[0145] This method therefore allows a capture system with parallel cameras to be used for display on a simple screen such as a 3D television or 3D cinema screen, providing an equivalent user experience in all respects.

[0146] Adaptation of non-identical Ds / L and F / Lc ratios

[0147] When the ratio Ds / Lse is the same as the ratio F / Lc, it has been established above that it is possible to obtain a truly proportional three-dimensional experience (Z = 1). However, there may be constraints in the three-dimensional display system that make it impossible to satisfy this ratio. Nevertheless, it is possible to modify the image to restore this ratio and the desired three-dimensional proportionality. For simplicity, a parallel screen system is used. A virtual reality headset with a very wide viewing angle is provided to the user thanks to the magnifying lens. The final screen width is given by the formula Ls1 = Ls1 * G, where G represents the magnification provided by the lens used.

[0148]

[0149] The effective width of the three-dimensional image is determined taking into account the distance from the observer to the screen. For this purpose, the following formula is used.

Equation

[0150] Step 1: Determination of the effective width

[0151] Next, the width of the screen S1’ is compared with the effective width of the image. As shown in FIGS. 14A, 14B, and 14C, · When Lse = Ls1’, the image can be displayed on the screen as it is. · When Lse < Ls1’, the size of the image is reduced (black bar, central window). · When Lse > Ls1’, it is necessary to cut the image to fit the actual size of the screen.

[0152] Step 3(A): Adjustment of the image when Lse < Ls1’

[0153] Method 1:

[0154] In this case, you can add black bars across the image to keep it centered on the eye and preserve the original aspect ratio of the image. To do this, use the following formula:

number

[0155] The resulting image is reset to the screen resolution and displayed in full screen mode. For example, an image with a resolution of 1920x1080 must have an effective width (Lse) of 45 cm and be displayed on a screen with a final width (Ls1') of 60 cm. The image is:

number

[0156] Therefore, the final image has a resolution of 2560x1440 pixels, and the aspect ratio is kept at 1.78:1. This new image is reset to the screen resolution and displayed in full screen mode. For example, if the screen has a resolution of 800 pixels, the active portion (displaying the image data) will be 1920 / 2560*800=600 pixels.

[0157] Method 2:

[0158] Alternatively, the image can be created to be displayed in a window that is centered horizontally and preferably also vertically on the screen. The image resolution is:

number

[0159]

number

[0160] The image (downscaled) is reduced from 1920 pixels to 600 pixels and placed in the center of the screen, giving the exact same result as above (the active part of the image).

[0161] Step 3(B): Image adjustment when Lse>Ls1'

[0162] If the image's effective width is greater than the effective width of the screen, the image will be scaled down, cutting off both sides of the image to maintain horizontal centering. You can use the following methods:

number

[0163] For example, an image with a horizontal resolution of 1920 pixels and an effective width at a distance from the screen that should be 50 cm (Lse), but the actual screen width is only 30 cm, the image can be cropped as follows:

number

[0164] The final image will therefore have a resolution of 1152 x 648 pixels, with the same aspect ratio of 1.78:1. All that remains is to adjust the resolution of the resulting image to match the screen resolution and display it in full-screen mode.

[0165] Parallel screen method adaptation

[0166] Currently, single screen systems are being considered.

[0167] As shown in Figures 15A, 15B, and 15C, the user is viewing an image on a television that provides a field of view limited by the distance from where the user is sitting to view the screen. As seen earlier, the final screen width is given by the equation Ls1' = Ls + Io. can be done.

[0168] To adjust the image on the screen, you can take two steps:

[0169] Step 1: Adjust the image resolutions (Rimg_h and Rimg_v) of the two images (left and right eye) so that the images respect the initial ratio Ds / Lse.

[0170] Step 2: Cut the right side of the image of the new left and right eyes obtained using the technique in Section 5.

[0171] Consider the example of an observer with an interocular distance of 6.5 cm looking at a television with a width (Ls) of 140 cm (an image with a resolution of 1920x1080 pixels and a width (Lse) of 200 cm). Step 1 is performed first.

[0172] Step 1:

[0173] Ls1' is determined first, which is equal to Ls+Io, which is 146.5 cm. Since Lse is greater than Ls1', the left and right eye images are scaled down in the following manner:

number

[0174] Therefore, the intermediate image has a resolution of 1406 x 791 pixels and the initial aspect ratio remains the same at 1.78:1. Step 2 is performed.

[0175] Step 2:

[0176] Using the intermediate image as the basis for the calculation, the right part for the left eye and the left part for the right eye are cut out as follows:

number

[0177] To maintain the aspect ratio of the original image, the image is cropped along the vertical axis by the same proportion.

number

[0178] Therefore, the final image for the left eye has a resolution of 1344 x 756 pixels and a resolution of 1.78 The resulting image is a section of the original image (right part) with an aspect ratio of 1:1. The right eye image consists of the equivalent left section of the original image for the right eye, i.e. a 1344x756 section aligned vertically in the same direction as the left eye section. All that remains is to adjust the resolution of the left and right eye images to match the screen resolution and obtain the final image for display in full screen mode.

[0179] Stereoscopic zoom: Change image size (Lse')

[0180] In monoscopy, zooming corresponds to the enlargement of the image in the x- and y-axes at a given magnification: if you zoom in by 2, the image will appear twice as large as the original. On the other hand, as we saw earlier, this enlargement in stereoscopy affects not only the size of the image on the screen, but also the perceived depth of the object.

[0181] In the example of a stereo image displayed to scale for a given screen (Z=1, Io / B=1), the pair of stereo images (left and right images) are modified identically by a factor of X, so that Lse' / Lse=X. The effect of this modification for a given user at the same distance from the screen is observed.

[0182] Effects on perceived distance

[0183] According to the equations established above, the following is established:

number

[0184] Thus, if the image magnification factor is X, the perceived distance of the object decreases proportionally to 1 / X.

[0185] Effect on perceived width

[0186] According to the equation established above, the following is also established:

number

[0187] So if the image magnification factor is X, the perceived width of the object remains the same.

[0188] Effect on Proportionality

[0189] Finally, according to the equations established above, the following is established:

number

[0190] Thus, if the image magnification factor is X, the proportional scale changes by an inverse proportional factor of 1 / X.

[0191] In summary (see graphs in Figures 18A, 18B, and 18C): [Table 2]

[0192] To maintain the proportionality of the stereoscopic representation, a change in the perceived distance of the image is accompanied by an equally proportional change in the perceived size of the image. In other words, the change in spatial scale (Esp = Io / B) is equal to the change in depth scale (Epr), so that the proportional scale remains equal to 1.

[0193] However, the base of the stereoscopic camera system is already fixed for shooting, and the distance between the user's two eyes obviously cannot be changed, so the spatial scale components (Io and B) cannot be changed. In other words, if the scale or magnification of the image on the screen changes, there is no way to keep the experience proportional.

[0194] So, for zooming by image magnification:

[0195] When you zoom in, you "enter" the 3D world. Your field of view narrows and the 3D world becomes flat.

[0196] Zooming out takes the user out of the 3D world, widening the field of view and stretching the 3D world.

[0197] Stereoscopic zoom: Change of optical base (Bo)

[0198] The following example diagram is provided: When a zoom occurs, the scale (x, y, z) changes globally by a factor X as follows:

number

[0199] Although the user's interocular distance (Io) cannot be changed, the position of the image relative to the center of the optical axis of the two eyes can be changed. The optical base (Bo) is defined as the distance between the centers of the two images on the screen. We can show how basic optical changes affect the perceived width and depth of an object.

[0200] Effect on perceived width

[0201] FIG. 19A shows the geometry of Io and Bo for an image displayed on a screen, and FIG. 19B shows the resulting change in object width as a result of changing Bo.

[0202] The optical base is positioned so that its center is perfectly centered between the observer's eyes.

[0203] The following is established:

number

[0204] By replacing "La" in the first equation with the result of the second equation, the following is established:

number

[0205] A ratio Lp' to Lp is established.

number

[0206] Therefore, for a given image width, the perceived change in an object's width is equal to the perceived change in that object's distance. The optically based change allows the scale proportionality condition to be satisfied. We next discuss how optically based changes affect the perceived distance of objects in a stereoscopic representation.

[0207] Effects on perceived distance

[0208] An object is photographed located at points Ag and Ad in the left and right eye images, respectively. Figures 20A and 20B show the effect of using an optical base Bo lower than Io.

[0209] Based on the properties of a right triangle, the following is established:

number

[0210] The ratio Dp' / Dp is established as follows:

number

[0211] This relationship demonstrates that the orthostereoscopic effect is lost when using a different optical base. In fact, while the variations of Dp are linear, the variations of the ratio Dp' / Dp are not, because they vary according to Ps, that is, as a function of the distance of the captured object in the real world. For a unit variation of Drn, the variation of Dp' changes according to the value of Drn. This can be seen as several zones with approximately linearly proportional variations. This relationship allows us to calculate the value of X (3D magnification).

number

[0212] This result demonstrates that the magnification factor X is only valid for a specific real distance Drn. For example, if we obtain a 3D magnification factor equivalent to one-third of the original representation, we can specify the original distance (Drn) to achieve this relationship. Arbitrarily, the distance Drn is chosen to be the real reference distance displayed in the zero plane, i.e., the distance from the screen (Dp = Ds), in orthostereoscopic mode. The result is as follows:

number

[0213] Conversely, if Bo is determined to give a desired 3D magnification, Bo can be separated as follows:

number

[0214] The graphs in Figures 21A and 21B show the effect of optically based modifications on the ratio between real-world and perceived-world distances.

[0215] As shown in the graph above, the perceived space is not linear (the ratio Dp' / Drn is not constant) and is also not orthostereoscopic. When changing the optical base and zooming in the image, a plateau is quickly reached in the perceived distance. The maximum perceived distance (when Drn = infinity) is calculated as follows:

number

[0216] First, it is not possible to zoom out because objects span distances from Drn to infinity. An object at infinity will normally appear in the center of both eyes if Bo = Io. If Bo is greater than Io, the point will be seen to the left of the left eye and to the right of the right eye, respectively. Since it is not possible to separate the eyes, this method makes fusion impossible and causes pain to the user.

[0217] Also, this method does not significantly increase the perceived image portion: the image is only moved by a few centimeters, so the desired effect cannot be achieved by zooming (i.e., significantly changing the field of view).

[0218] Finally, optically based modifications cause significant spatial distortions, disrupting the linearity and orientation of the space. This causes a loss of the stereoscopic effect. For all these reasons, changing the optical base is not the recommended method for 3D zooming of stereoscopic images.

[0219] Comfortable 3D display that takes into account vergence and accommodation management

[0220] To establish the depth of an object, the brain uses many visual cues and combines them to achieve a higher level of certainty. For example, it can use interposition, motion parallax, blur, perspective, and of course, stereopsis (parallax). In traditional / monoscopic video games and movies, many of these techniques are used to give the content a greater sense of depth, and sometimes even provide the impression of depth to the spectator / player.

[0221] For stereoscopic content, differences in parallax are used to give the impression of depth. However, parallax information often competes with other visual cues. For example, one case is when an object should be in front of the screen but its image is "cropped by the edge of the screen", as shown in Figure 22.

[0222] In the image in Figure 22, the baseball should emerge from the screen according to the stereoscopic information, but it "touches" the frame, meaning that the screen frame appears to block the image of the ball. However, in everyday life, the brain learns that there is an object in front of it that visually blocks another object (the phenomenon of interposition). Therefore, because there is a conflict between visual cues and interposition is commonly used by the visual cortex, the brain rejects the stereoscopic information and decides to place the ball on the screen (e.g., refusing to perceive it as being in front of the screen). Stereophotographers are familiar with this phenomenon and carefully frame objects that should appear in front of the screen.

[0223] Another major problem arises from the difference between vergence information (where the eyes converge) and accommodation (how far the eyes focus). The brain regularly manages these two pieces of information simultaneously to enable clear vision. These two pieces of information should be consistent with each other, and the brain uses both pieces of information together to make better decisions (accommodations). While vergence is achieved at a given distance (Dp), in stereoscopic vision, these two pieces of information do not necessarily coincide because the eyes focus at a screen distance (Ds). Figure 23A shows a schematic representation of the same vergence distance and focal distance for real-world viewing, while Figure 23B shows a schematic representation of the screen vergence distance and closer focal distance for stereoscopic 3D viewing.

[0224] Literature shows that if there is too much conflict between vergence and accommodation in stereopsis, many adverse effects can occur, such as discomfort, pain (sometimes persistent), and diplopia (double vision, without fusion). This conflict between vergence and accommodation affects not only the level of comfort, but also the perception of the depth of objects in a stereoscopic representation.

[0225] Experiments were conducted using parallel cameras and computer-generated objects positioned at various distances. Upon observation, it was observed that, despite significant differences in parallax (measured and verified on the screen), the perception of object distance changed only slightly for objects positioned very far from the screen. When there is a conflict between vergence and accommodation information, the human brain can prioritize accommodation information, and the perceived distance of an object is related to its distance from the screen. This effect can be accentuated when there are many objects in the field of view near the screen that corroborate accommodation information.

[0226] To manage this issue, a maximum or furthest distance (perceived "inside" the screen) and a minimum distance ("outside" the screen) for the angle constraint are determined.

[0227] Maximum distance (Df)

[0228] The article "Visual Discomfort and Visual Fatigue of Stereoscopic Displays: A Review" by Marc Lambooij et al., published in the May-June 2009 Journal of Imaging Science and Technology (53(3):030201-030201-14, 2009), suggests considering a 1° limit between the angle formed by the eyes as they converge on the screen (the "accommodation" angle) and the maximum or minimum convergence angle to maintain a comfortable experience. This principle is used as the basis for determining the maximum and minimum distances for stereoscopic perception. It is important to note that there are important differences between visualization with lenses (e.g., stereoscopes, virtual reality headsets), where accommodation occurs at infinity, and traditional screens, where accommodation occurs at the screen's distance. We will first discuss the case of traditional screens.

[0229] Figure 24A shows the case of convergence inside the screen. From this figure, finding the value of the object distance (Do) is established as follows:

number

[0230] With reference to Figure 24B, when the eyes converge on the screen, the vergence angle for the left eye is equal to Θ. As the eyes converge "inside" the screen, the angle formed decreases to θ' for each eye. A value for P is determined that satisfies the vergence angle constraint (V, expressed in radians, is the angle multiplied by π and divided by 180°) while maintaining stereoscopic perception and legibility.

number

[0231] Now we have the value of P that satisfies the congestion condition, and by integrating P into the previous equation, we get the maximum distance as follows:

number

[0232] Starting from the distance to the screen, all objects up to infinity can be comfortably merged because they are within the vergence constraint. To establish this distance, the value of Ds is established as Df tends to infinity as follows:

number

[0233] Taking the example of a person with an interocular distance of 6.5 cm and a convergence limit of 2° in radians or 2°π / 180° as an example, the distance to the screen that allows comfortable fusion to infinity is:

number

[0234] This shows that for stereoscopic presentations on screens relatively close to the average user, the stereoscopic effect can have a natural and very significant depth (up to infinity). This corresponds well to indoor projections as well as 3D viewing on television. On the other hand, viewing on stereoscopic screens closer to the user (e.g., mobile phones, computer screens, tablets, etc.) has serious depth limitations. For example, for the same user as in the previous example, if the screen (e.g., laptop) is placed 60 cm from the user, the maximum allowable depth under the 2° constraint is only 88.6 cm, or a very limited 28.6 cm inside the screen.

[0235] Minimum Distance (Dn)

[0236] Referring to Figures 25A and 25B, the minimum distance perceived by the user for an object to exit the screen, i.e., located between the user and the screen, is calculated.

[0237] Figure 25A shows the case of convergence in front of the screen. From this figure, finding the value of the object distance (Do) is established as follows:

number

[0238] Referring to Figure 25B, when the eyes converge on the screen, the vergence angle for the left eye is equal to Θ. As the eyes converge in front of the screen, the angle formed increases to θ' for each eye. A value for P is determined that takes into account the angle constraint of vergence (V, expressed in radians, is the angle multiplied by π and divided by 180°) while maintaining stereoscopic perception and legibility.

number

[0239] Now that the P value corresponding to the congestion constraint has been determined, we can integrate the P value into the previous equation to obtain the minimum distance (Dn) as follows:

number

[0240] Parameterization to account for vergence and accommodation conflicts

[0241] The maximum and minimum distances of the stereoscopic representation for vergence and accommodation conflicts are determined as follows.

number

[0242] It has been shown that changing the field of view (Lse') does not reduce the overall depth of the perceived world in a stereoscopic representation. In fact, a point at infinity captured by parallel cameras is perceived as infinitely far away in a stereoscopic representation, regardless of how the stereoscopic field of view is modified (always at the center of each eye). On the other hand, it has been demonstrated that the depth of the perceived world can be reduced in a representation by changing the optical base of the system.

[0243] An optical base for considering Df constraints was determined. The farthest point captured by a parallel camera system (Drn = infinity) is perceived in the stereoscopic representation at the maximum distance that enables a comfortable experience (Dp’ = Df). [Number]

[0244] Note that this adjustment is made for any distance to a screen shorter than the minimum distance that enables infinity fusion so that Ds < Io / V (V expressed in radians). If any screen distance is greater than Io / V, the optical base can be set to Io.

[0245] When the optical base is established with a value less than Io, the linearity of the space also changes. One result of this spatial change is that the object of orthostereoscopy usually becomes the screen distance and "deviates" from the screen. Therefore, the part of the image that should be inside the screen comes out of the screen, causing discomfort and a framing problem.

[0246] To solve this problem, the change in image size (Lse) can be used to make the actual distance (distance from the screen) of the object displayed on the zero plane equivalent to the orthostereoscopic representation scale. For example, in an image captured with a scale of 1 (proportional to nature) and having a stereoscopic representation on a screen located 60 cm from the user, the object perceived at a distance of 60 cm from the screen is preferably located 60 cm from the camera when the image where the object is located was captured.

[0247] To do this, the actual distance of the object displayed on the screen is established for the case of an orthostereoscopic representation with Bo = Io. This distance can be calculated as follows. Drn = Ds * B / Io The perceived distance (Dp’) is equal to the distance to the screen (Ds) The image widths Lse' and Drn are established so that the image width Lse' is

number

[0248] When Bo is equal to Io (the user is a large enough distance away from the screen), Lse' becomes equal to Lse, resulting in a return to orthostereoscopic mode.

[0249] 16A and 16B are schematic block diagrams of an apparatus for processing parallel-camera stereoscopic video to accommodate different viewing conditions. Capture parameters can determine the original field of view. These parameters can be encoded into the image or video stream, set by the user, or detected by video analysis. Block 12 therefore represents a memory store of captured field of view parameters and optionally includes an interface for receiving field of view parameters from image data stores or video streams 22a and 22b.

[0250] The display / screen parameters may be screen distance, screen resolution, screen size, and interocular distance of the viewer. These parameters may be stored in memory 14. The interocular distance may be a variable set in memory 14, or may be fixed to a nominal value in calculator 20, which determines the crop and scale parameters, as described in detail above. If the screen is shared by multiple viewers, the interocular distance may be selected to be the interocular distance of the person with the smallest interocular distance to avoid the problem of that person's distraction.

[0251] The computer 20 may also modify the base offset to bring distant objects closer to the screen and reduce the image in view, taking into account the vergence constraints described above with reference to Figures 17 to 25. The scaling determines crop and scale parameters that reduce the difference in vergence angle between the screen and the displayed object.

[0252] The distance between the viewer and the screen can be entered using the user interface or other suitable method.

[0253] When there is a change in interocular distance, the scale parameters include image shift parameters even if other viewing conditions account for the original recording. However, when the 3D scene is viewed on a display that is smaller / larger than the original field of view, the scale parameters include image shifts to maintain the basic distance between the centers of each image on displays of different sizes.

[0254] Thus, the 3D images, i.e., the right-eye and left-eye images stored in stores 22a and 22b, are shifted, scaled, and cropped / border-padded as necessary in image processor 25, for example, as shown schematically in Figure 16B. The image processor may be a GPU, CPU, FPGA, or any other suitable processing device. The source of images 22a and 22b may be a stereoscopic image stream, as known in the art.

[0255] As mentioned above, stereoscopic viewing can be achieved using known techniques. In the block diagram of FIG. 16B, stereoscopic formatting is performed in block 28. Such image processing can be performed by a CPU, but can also be performed using, for example, a GPU or FPGA. In an anaglyph display, color filter glasses can be worn by the viewer, and the screen image consists of color-encoded left and right image data. In a page-flip operation, the viewer can wear shutter glasses that allow the right and left eyes to see in alternating time slots while the screen image alternates between the right and left images. In an autostereoscopic display, the viewer does not need glasses, but the screen includes a lens or screen mask that allows the right eye to see the right-eye pixels and the left eye to see the left-eye pixels. In a polarized line-interleaved display, odd and even lines have different polarizations of light (the pattern of pixels for each polarization need not be limited to alternating horizontal lines), and polarized glasses are worn so that one eye sees the odd lines and the other sees the even lines. As shown generally in Figure 16B, stereo formatting for the desired display technology is performed by a formatter module 28 before sending the display signal to the display device. If desired, the stereo formatter operation or function can be performed within the image processor. The formatted image or images are then displayed using a corresponding display device 30.

[0256] In the embodiment of FIG. 16C, the image source is a large field-of-view source, such as a wide-angle (e.g., 180-360 degree panoramic source), a fisheye lens, or a computer-generated image source, and cropping or dewarping and cropping can provide a desired image for a given viewing direction. Viewing direction module 18 can be part of a user interface that allows a user to select the viewing direction. The process of cropping or dewarping and cropping is known in the art and is performed in module 19. As shown, the original fisheye camera image cannot be displayed as a 2D image until it is dewarped. Alternatively, dewarping module 19 can be integrated into image processor 25, such that the necessary cropping and scaling involves selecting the portion of the source image to be dewarped.

[0257] It will be appreciated that image processing, i.e. cropping and scaling, can be performed using a volume reader, which is a system that places the original image in 3D space and creates a virtual representation of the original image at the correct position for the viewpoint or camera, taking into account the original capture parameters. This means capturing a virtual "camera" view, which can be done within most conventional GPUs, for example.

[0258] The details are as follows: nomenclature D=distance L=width H = height F=focal point RES = resolution [Table 3]

[0259] 1) Place the image in space (see Figure 17A) - Place the left image at any width Lo Place the right image at the same width Lo.

[0260] 2) Camera placement Position the camera at the center of the image, with x, y, and z coordinates set to 0.0.0 (origin) Placed away from the image to respect the proportions

number

[0261] 3) Render the image Create left and right eye images using: i) Ratio Lv / Hv=(Ls+Io) / Hs ii) Ratio Fcv / Lccdv=Ds / (Ls+Io) iii) Resolution=RESs*(Ls+Io) / Ls 4) Single-Screen Format (See Figure 17B) - Overlay two images Shift the left image to the left by a distance equal to (Io / 2) / (Ls+Io)*Lv Shift the right image to the right by the same distance -Keep the intersection of two images Anaglyph format

[0262] Alternatively, in step 4, the left eye image can be cut (from the left) by a number of pixels equal to RESs (resolution), and the right eye image can be cut from the right by the same number of pixels.

[0263] In step 2, you can zoom in or out by moving the camera closer or further away.

[0264] In the context of streaming or online services, images can be processed on a server and sent to a remote client display. In this context, the display or user of the display device can relay display / screen parameters to the server, where the image is processed and encoded for transmission to the client display. Cropping and scaling can be performed before transmission to achieve a reduction in the data transmitted.

Claims

1. 1. A system for stereoscopic image streaming, comprising: a server comprising a processor and a non-transitory memory for storing instructions; The instructions, when executed by the processor, operate to perform a method for processing stereoscopic images for display to a viewer on a single screen of a remote client display; the stereoscopic images are captured using a parallel-axis camera having a first field of view (Lc / F) provided by a sensor with a focal length of width Lc and length F; The method comprises: receiving and using a definition of a second field of view ((Ls+Io) / Ds) provided by the viewer's interocular distance Io, the width Ls of the single screen, and the distance Ds between the viewer and the remote client display, to position and scale the stereoscopic image, thereby displaying the stereoscopic image on the single screen of the remote client display at the distance Ds from the viewer relative to the first field of view (Lc / F); processing the stereoscopic images to crop the stereoscopic images for the single screen if the stereoscopic images scaled for the single screen are larger than the single screen, and to border the stereoscopic images and present them for the single screen if the stereoscopic images scaled for the single screen are smaller than the single screen; encoding the processed stereoscopic images for streaming transmission to the remote client display.

2. 2. The system of claim 1, wherein the instructions further comprise selecting a zoom window within the stereoscopic image to modify the first field of view, and the stereoscopic image is scaled relative to the modified first field of view.

3. The system of claim 2 , wherein the zoom window is offset from a center of the stereoscopic image to enable viewing of a region of interest within the stereoscopic image.

4. The system of claim 3 , wherein viewer input is used to move the offset while viewing the stereoscopic image.

5. The system of claim 1 , wherein the stereoscopic images are still images.

6. The system of claim 1 , wherein the stereoscopic images are video images.

7. The system of claim 1 , wherein the stereoscopic images are converted into combined anaglyph format images.

8. 7. A system according to claim 1, wherein the stereoscopic images are converted into column-interleaved images for display on an autostereoscopic display.

9. 7. The system of claim 1, wherein the stereoscopic image is converted into a series of page-flipping images for viewing with shutter glasses.

10. 7. The system of claim 1, wherein the stereoscopic image is converted into a series of line interleaves for polarized display.

11. 11. The system of claim 1, wherein the instructions further comprise obtaining user input to obtain a definition of the second field of view provided by the single screen of the remote client display.

12. 12. The system of claim 1, wherein the instructions further comprise acquiring sensor data to obtain a definition of the second field of view provided by the single screen of the remote client display.

13. 13. The system of claim 1, wherein the stereoscopic images are arranged on the single screen of the remote client display to correspond to an object spacing of Io between right-eye and left-eye images of a distant object.

14. 14. The system of claim 13, wherein the viewer includes a plurality of viewers, and the interocular distance Io is selected to be the smallest interocular distance among the plurality of viewers.

15. The stereoscopic images are further scaled and positioned using relative base offsets so that objects appearing at greatest depth appear closer and objects appearing in front of the single screen appear closer to the single screen, thereby: a binocular angle between a focal point at the single screen depth and a focal point on an object appearing at the modified maximum depth; and the angle of the eye between the focal point at the depth of the single screen and the focal point on the nearest appearing object in front of the single screen 15. The system of claim 1, wherein the system limits at least one of the following to reduce eye strain:

16. 16. The system of claim 15, wherein the stereoscopic images are further scaled and positioned to limit both the canthus between a focal point at the single screen depth and a focal point on an object appearing at the modified maximum depth, and the canthus between a focal point at the single screen depth and a focal point on an object appearing closest in front of the single screen.

17. 17. The system of claim 16, wherein the stereoscopic images are further scaled to maintain objects that appear at the single screen depth appearing at the same depth.

18. 18. The system of claim 15, 16, or 17, wherein the viewing angle between objects appearing at depth on the single screen and objects appearing behind and / or in front of the single screen is less than 1 degree.

19. 19. The system of claim 1, wherein the stereoscopic image comprises a panoramic image, and the instructions further comprise: defining a viewing direction within the panoramic image; and extracting a portion of the panoramic image using the viewing direction.

20. 20. The system of claim 19, wherein the panoramic image is a wide-angle camera lens image, and the instructions further comprise dewarping at least a portion of the panoramic image.

21. The system of claim 20 , wherein the panoramic image is a fisheye lens image.

22. 10. A computer program for stereoscopic image streaming recorded in a non-transitory memory storing instructions for a processor or reconfigurable hardware, the instructions operative when executed by the processor or reconfigurable hardware to perform the method of processing stereoscopic images of claim 1 to display the stereoscopic images captured using a parallel-axis camera having a first field of view to a viewer on a single screen of a remote client display.

Images