Stereo image pair according to monoscopic image saliency

Abstract

An autostereoscopic display system can generate a stereoscopic image pair from a monoscopic image. Processing circuitry can determine saliency information indicative of a region of interest of the monoscopic image from the monoscopic image. The processing circuitry can determine a convergence plane for a stereoscopic representation of the monoscopic image. The convergence plane can be a function of disparity characteristics of the region of interest of the monoscopic image. The processing circuitry can generate a disparity map based on the monoscopic image and the convergence plane, such as by generating a provisional disparity map based on the monoscopic image; generating a saliency map based on the monoscopic image; and generating the disparity map based on the provisional disparity map and the saliency map. The processing circuitry can generate a stereoscopic image pair using the monoscopic image and the disparity map.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 623,742, filed January 22, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This document generally relates to display systems, and more specifically to multi-view displays, 3D displays, or autostereoscopic displays. Background Technology

[0004] Multi-view displays provide viewers with different views of a multi-view image. Stereoscopic displays provide two different views of a three-dimensional image to each of the viewer's eyes. Autostereoscopic displays provide two different views to each of the viewer's eyes without requiring special glasses or goggles. Efforts are underway to improve autostereoscopic displays. Summary of the Invention

[0005] In the example, the autostereoscopic display system may include processing circuitry capable of performing operations. These operations may include: receiving data from a specified single-field-of-view image; determining saliency information from the single-field-of-view image indicating a region of interest (ROI); determining a convergence plane for a stereoscopic representation of the single-field-of-view image, the convergence plane being a function of the disparity characteristics of the ROI in the single-field-of-view image; generating a disparity map based on the single-field-of-view image and the convergence plane; and generating a stereoscopic image pair using the single-field-of-view image and the disparity map.

[0006] In one example, a method may include generating a stereo image pair from a single-view image. The stereo image pair may optionally be displayed using an automatic stereo system. The method may include: receiving data specifying the single-view image using processing circuitry; determining saliency information indicating a region of interest (ROI) of the single-view image using the processing circuitry; determining a convergence plane for a stereo representation of the single-view image using the processing circuitry, the convergence plane being a function of the disparity characteristics of the ROI of the single-view image; generating a disparity map based on the single-view image and the convergence plane using the processing circuitry; and generating the stereo image pair using the single-view image and the disparity map using the processing circuitry.

[0007] The content of this document is intended to provide an overview of the subject matter. It is not intended to provide an exclusive or exhaustive explanation of the invention. Detailed descriptions are included to provide further information about the subject matter. Attached Figure Description

[0008] Figure 1 An exploded perspective diagram of an example of a multi-view display system is shown.

[0009] Figure 2 A front view of an example display panel including an array of light-emitting diodes is shown.

[0010] Figure 3 A front view of an example display panel including backlight and light valve array is shown.

[0011] Figure 4 A front view of an example of a parallax-generating optics device including a lenticular lens is shown.

[0012] Figure 5 It shows Figure 4 A cross-sectional view of a cylindrical lens.

[0013] Figure 6 A front view of an example of a parallax-generating optics device including a parallax barrier is shown.

[0014] Figure 7 A transmission slit is shown. Figure 6 A cross-sectional view of the parallax barrier.

[0015] Figure 8 An example of generating stereo image pairs from a single field-of-view image is shown.

[0016] Figure 9 A flowchart illustrating an example of a method for generating stereo image pairs from a single field-of-view image is shown.

[0017] Figure 10 An example of a single-view image including a plus sign ("+") on the background and the corresponding saliency map is shown. Detailed Implementation

[0018] Typically, parallax can be expressed as the horizontal distance between the left and right images of a stereoscopic image pair for the same point in a scene. In other words, parallax corresponds to the distance between a pixel in the left (right) image and its horizontal counterpart in the right (left) image. This assumes the viewer is orthogonal, with their left and right eyes at the same height, and the axis connecting the left and right eyes is horizontal.

[0019] A disparity map quantifies the disparity of a stereo image pair. For example, a disparity map can be an array, such as one with the same size as the images of the stereo image pair. The value of the disparity map at a particular pixel can correspond to the disparity value of the images of the stereo image pair at that same pixel.

[0020] The convergence plane is the plane in the scene where the parallax is zero. Objects on one side of the convergence plane have positive parallax. Objects on the other side of the convergence plane have negative parallax.

[0021] When an autostereoscopic display shows stereoscopic image pairs, objects in the scene located at the convergence plane appear to be located at the plane of the autostereoscopic display. Objects in the scene that are closer to the viewer than the convergence plane appear to be above the autostereoscopic display. Objects in the scene that are farther from the viewer than the convergence plane appear to be below the autostereoscopic display.

[0022] Typically, a disparity map can include the same amount of information as a depth map, allowing one disparity map to be created from another. Furthermore, for a data structure comprising a left image, a right image, and a disparity map, any two of the three data structures can be used to create the third.

[0023] Furthermore, image processing software exists that can be executed by processing circuitry to generate artificial or simulated disparity maps from a (single) single-view image using artificial intelligence. For example, the processing circuitry can use a fully convolutional neural network model. The neural network model can include an encoder based on a residual neural network (ResNet). The neural network model can include a decoder configuration with skip connections, similar to those in a U-Net. The architecture of a U-Net can include encoder paths (also called shrinking paths) and decoder paths (also called expanding paths). The encoder can extract features from the input image. The decoder can project these features onto the pixel space to produce dense classifications. The encoder and decoder can be symmetric and connected by paths, which can give the model a "U" shape. The convolutional neural network model can include millions (or more) of trainable parameters, which can be trained on multiple (e.g., thousands, millions) images with known disparity characteristics. The processing circuitry can use this model to process a (single) single-view image and provide the corresponding disparity map.

[0024] Typically, saliency refers to data about visually important or prominent regions within an image. A saliency map quantifies the saliency of one or both images in a (single) field-of-view image or a stereo image pair. For example, a saliency map can be an array, such as having the same size as the left and right images of a field-of-view image or a stereo image pair. Image processing software, executable by processing circuitry, can analyze one or both images in a field-of-view image or a stereo image pair, such as by examining one or more of the color variance, brightness variance, or texture variance, to determine saliency information about the images and generate a corresponding saliency map.

[0025] For example, to generate a saliency map of an image, image processing software can calculate the local variance of the colors surrounding each pixel in the image. The software can identify regions with high color variations, which typically correspond to the edges or boundaries of objects in the image. The software can consider regions that have high visual perceptual importance to human viewers as salient objects or salient regions. The software can perform saliency analysis at a coarse scale. For example, the software can generate fairly large clusters of information-rich objects or objects that stand out from their surroundings. When performing coarse analysis, the software can highlight the most salient regions with relatively high granularity, rather than focusing on finer details in the image.

[0026] Autostereoscopic display systems can partially utilize saliency properties to generate stereoscopic image pairs from a single-field-of-view image. For example, an autostereoscopic display system can use saliency information from a single-field-of-view image to identify a region of interest (ROI) within that image. The autostereoscopic display system can then determine the location of the convergence plane based on parallax information within the ROI. When the autostereoscopic display system displays a stereoscopic image pair on an autostereoscopic display, the viewer can perceive the ROI as being approximately located at or near the plane of the autostereoscopic display, rather than too close to the viewer (e.g., in front of the autostereoscopic display) or too far away (e.g., behind the autostereoscopic display).

[0027] As a specific example Figure 10 An example of a single-field-of-view image 1002 including a plus sign 1004 (“+”) on a background and a corresponding saliency map 1008 is shown. An autostereoscopic display system can determine that the plus sign 1004 forms a region of interest 1006, such that the perimeter of the plus sign 1004 defines the perimeter of the region of interest 1006. The autostereoscopic display system can generate the saliency map 1008, which quantifies the saliency information of the single-field-of-view image 1002. The saliency map 1008 can include relatively large (e.g., positive) saliency values ​​(S) within the region of interest 1006 and relatively small (e.g., zero) saliency values ​​(S) outside the region of interest 1006. The autostereoscopic display system can generate stereoscopic image pairs based on disparity information in the region of the plus sign 1004 (e.g., without using information from the background). When the autostereoscopic display system displays a pair of stereoscopic images on the autostereoscopic display, the plus sign 1004 can be perceived as being located at or near the plane of the autostereoscopic display, rather than being much higher than the autostereoscopic display or far behind the plane of the autostereoscopic display.

[0028] The preceding paragraphs only outline some aspects of the image generation and display techniques described in detail below, and should not be construed as limiting in any way.

[0029] Figure 1An exploded perspective view of an example of a multi-view display system 100 including a multi-view display 110 is shown. Figure 1 The configuration is just one example of the multi-view display system 100; other configurations can be used.

[0030] Figure 1 The notation shown and used below assumes that the multi-view display 110 is in The z-axis extends in the plane, and extends away from the multiview display 110 and approximately toward the viewer 42 in a direction orthogonal to the plane of the multiview display 110. Other notation conventions may be used.

[0031] The multiview display system 100 may include a multiview display 110. The multiview display 110 may provide different views of a multiview image to a viewer 42. For example, as the viewer 42 moves in space, the multiview display 110 may direct different views of the multiview image to the viewer 42's left and right eyes, allowing the viewer 42 to observe different views of the multiview image from different positions or orientations. In some configurations, the multiview display 110 may provide multiple views in various fixed location areas in space, allowing the multiview display 110 to operate without eye tracking. In other configurations, such as the autostereoscopic configuration described in detail below, the multiview display system 100 may use eye tracking to dynamically and continuously (or at relatively frequent discrete times) determine the viewer 42's position, and in response, may dynamically and continuously control how the multiview display 110 displays the multiview image, such that multiple views follow the viewer 42 or the tracked eye(s) of the viewer 42 as the viewer 42 moves in space relative to the position of the multiview display 110.

[0032] For a multi-view display 110 to be configured to provide only two different views of a multi-view image, the multi-view display 110 can be an autostereoscopic display or a three-dimensional (3D) display. An autostereoscopic display can provide the left image to the left eye of viewer 42 and the right image to the right eye of viewer 42. The left and right images can correspond to different views of an object or scene, and can allow viewer 42 to perceive the object or scene in 3D using only their naked eye, without using additional glasses or a head-mounted device.

[0033] The multi-view display system 100 may include a viewer tracker 120 that can dynamically determine the position of viewer 42. The multi-view display system 100 can use the determined position of viewer 42 to guide a left image to viewer 42's left eye and a right image to viewer 42's right eye. Because the viewer's position can change as viewer 42 moves in space, eye tracking allows the multi-view display system 100 to follow viewer 42, enabling the autostereoscopic display to automatically guide the left image to viewer 42's left eye at its (dynamically changing) position and automatically guide the right image to viewer 42's right eye at its (dynamically changing) position. The viewer tracker 120 can provide the tracked position of viewer 42, such as the viewer 42's head, one or both eyes of viewer 42, or another anatomical feature of viewer 42. Viewer tracker 120 may be coupled to processing circuitry 130 (described below) or a controller, such as by providing viewer 42 positioning data representing the measured position or location of viewer 42 (in Figure 1 The coordinates are shown in the figure. xv , yv and zv Viewer tracker 120 may provide viewer 42 positioning data to processing circuitry 130 at regular or irregular intervals. In a particular example of viewer tracker 120, a camera may capture an image of viewer 42. Viewer tracker 120 may also include an image processor (or a general-purpose computer programmed as an image processor) that can determine the position of viewer 42 within the captured image to provide the tracked position. In some examples, processing circuitry 130 may include the image processor of viewer tracker 120. In other examples, processing circuitry 130 may be separate from the image processor of viewer 42 tracker. Other suitable viewer trackers may be used, including those based on lidar (e.g., using the time-of-flight of reflected light across a scene to determine distances to one or more objects in the scene, such as the viewer's head or the viewer's eyes) or other technologies. Processing circuitry 130 may use the output of viewer tracker 120, along with other data, to perform one or more downstream calculations relating to providing a left view or left image to viewer 42's left eye and a right view or right image to viewer 42's right eye.

[0034] An autostereoscopic display can be a lenticular autostereoscopic display. In a lenticular autostereoscopic display, display panel 112 can display multi-view images, and parallax generating optics 118 can guide light from display panel 112 to viewer 42, such that the left image is visible from viewer 42's left eye and the right image is visible from viewer 42's right eye. During use of the lenticular autostereoscopic display, processing circuitry 130 can track the position of viewer 42 and can use the tracked position to dynamically determine how to distribute the content of the multi-view images on the surface area of ​​display panel 112 (e.g., using pixels distributed on display panel 112), such that the left image remains visible from viewer 42's left eye and the right image remains visible from viewer 42's right eye, even if viewer 42 changes position. In this way, position tracking and image content distribution can be performed in software, so that following the viewer 42's position does not involve physically moving any components of the lenticular autostereoscopic display relative to each other. Examples of suitable display panels and examples of suitable parallax generating optics are described below.

[0035] In the example, display panel 112 can display a multi-view image. Display panel 112 may have an array of subpixels 114 that can display the image according to stereoscopic coordinates associated with viewer 42. Subpixels 114 may be located at subpixel positions in a grid with grid axes (e.g., x-axis and y-axis). Each subpixel 114 can generate light of a specified color. For example, subpixels 114 may include red subpixels, green subpixels, and blue subpixels that generate red, green, and blue light, respectively. Other color / wavelength schemes may be used. Subpixels 114 may be grouped into pixels, where each pixel includes at least two subpixels 114 that generate light of different colors. Display panel 112 may receive display panel drive electrical signals 138 from processing circuitry 130 (described below), which may specify how the content of the multi-view image is distributed across the pixels and / or subpixels 114 of display panel 112. This is described below and... Figure 2 and Figure 3 The diagram shows two possible configurations for the display panel 112. Other configurations can also be used.

[0036] Figure 2A front view of an example display panel 112A including an array 202 of light-emitting diodes 204 (such as an array 202 of organic light-emitting diodes) is shown. Each light-emitting diode 204 may correspond to a sub-pixel. The array 202 of light-emitting diodes 204 may include a red light-emitting diode 204R, a green light-emitting diode 204G, and a blue light-emitting diode 204B, which correspond to the red, green, and blue sub-pixels, respectively. Each light-emitting diode 204 may controllably generate light in response to an electrical signal provided by processing circuitry 130 (such as a display panel drive signal 138) or by a suitable light-emitting diode drive circuit communicating with processing circuitry 130. Processing circuitry 130 may directly power a designated light-emitting diode 204, with the power varying according to the intensity at the corresponding location in the image. The power delivered to the light-emitting diodes 204 may optionally be pulse-width modulated at a modulation frequency higher than that perceptible to the human eye. Using pulse-width modulation simplifies the design of the light-emitting diode array controller because it can generate arbitrary average power levels from relatively small amounts of instantaneous power levels by changing the duty cycle of the power. In some examples, the array 202 of light-emitting diodes 204 may be arranged in a rectangular or square repeating pattern on the surface region 206 of the array 202. For example, the array 202 may have grid axes 208 that are orthogonal to each other. In some examples, the grid axes 208 may be parallel to the edges 210 of the array 202 of light-emitting diodes 204.

[0037] Figure 3 A front view of an example display panel 112B including a backlight 302 and a light valve array 304 is shown. Although Figure 3While the backlight 302 and the light valve array 304 are shown as separate, in reality, the backlight 302 and the light valve array 304 may be in contact or located as close as possible together. The backlight 302 may provide illumination with a uniform or substantially uniform intensity over a surface area of ​​the backlight 302. The backlight 302 may provide illumination with a relatively broad spectrum, such as including most or all of the visible portion of the electromagnetic spectrum. The backlight 302 may provide illumination toward the light valve array 304 over a continuous range of propagation angles. The backlight 302 may provide unmodulated illumination to the light valve array 304. The light valve array 304 may include light valves 306 (described below) that can be individually controlled by the processing circuitry 130 or controlled in one or more groups. Each light valve 306 may controllably attenuate the illumination from the backlight 302, such as in response to an electrical signal provided by the processing circuitry 130, such as a display panel drive electrical signal 138, or an electrical signal provided by a suitable light valve drive circuit in communication with the processing circuitry 130. Each light valve 306 may have a corresponding color filter that allows only a portion of the electromagnetic spectrum to pass through the light valve. For example, light valve 306 may include a red light valve 306R with a red color filter that allows only red light to pass through it; a green light valve 306G with a green color filter that allows only green light to pass through it; and a blue light valve 306B with a blue color filter that allows only blue light to pass through it. Other color schemes and numbers of colors may be used. Suitable light valves may include liquid crystal light valves, electrophoretic light valves, electrowetting-based light valves, etc. In some examples, the light valves 306 of the light valve array 304 may be arranged in a rectangular or square repeating pattern on the surface region 308 of the light valve array 304. For example, the light valve array 304 may have grid axes 208 that are orthogonal to each other. In some examples, the grid axes 208 may be parallel to the edges 312 of the light valve array 304.

[0038] Refer again Figure 1 The autostereoscopic display may include a parallax generating optics 118 that directs light from the display panel 112 to the viewer 42 such that a left view or left image is visible from the viewer 42's left eye, and a right view or right image is visible from the viewer 42's right eye. Two possible configurations of the parallax generating optics 118 are described below, and in Figure 4 and Figure 5 as well as Figure 6 and Figure 7 As shown in the diagram. Other configurations can be used. Figure 4 and Figure 5 as well as Figure 6 and Figure 7 Each configuration in can be compared with Figure 2 and Figure 3 Any configuration of the display panel 112 shown (e.g., Figure 2 An array of light-emitting diodes 204 or Figure 3 The backlight 302 and the light valve array 304 are used in combination.

[0039] Figure 4 A front view of an example of a parallax generating optics 118A including a cylindrical lens 402 is shown. Figure 5 It shows Figure 4 A cross-sectional view of the cylindrical lens 402. The cylindrical lens 402 may include a plurality of cylindrical lenses 504 spaced apart at equal intervals. The cylindrical lens 402 may have a focal plane that coincides with the display panel 112. The cylindrical lens 402 may be positioned to receive light from the display panel 112 and at least partially focus the received light to direct the light to a designated area near the viewer's eye.

[0040] Figure 6 A front view of an example of a parallax generating optics 118B including a parallax barrier 602 is shown. The parallax barrier may include a plurality of equally spaced transmission slits 704. Figure 7 A parallax barrier 602 with a transmission slit 704 is shown. Figure 6 A cross-sectional view of the image is shown. The parallax barrier 602 may include an array of opaque strips 706 and thin transmission slits 704, arranged to obscure portions of the left and right viewing areas of the displayed image. The transmission slits 704 may be spatially arranged to ensure that the left / right image portions are visible only in their intended corresponding left / right viewing areas. The parallax barrier 602 may be provided by a static physical layer in which the slits are precisely positioned, or electronically generated on an adaptive intermediate liquid crystal display layer.

[0041] The parallax generating optics 118 can be kept constant along the optical axis (OA) at an angle relative to the grid axes (e.g., the x-axis and y-axis), such as at a tilt angle of 45 degrees or approximately 45 degrees relative to the grid axis 208. For example, the parallax generating optics 118 may have transmission characteristics, such as cylindrical lenses or transmission slits, which are constant along the optical axis (OA) and periodic along an axis orthogonal to the optical axis (OA).

[0042] Refer again Figure 1An autostereoscopic display may include a material 116 disposed between a display panel 112 and a parallax generating optics 118. In some examples, the material 116 may extend entirely between the display panel 112 and the parallax generating optics 118, such that light originating from the display panel 112 passes only through the material 116 (and does not pass through any air or unfilled volume) before reaching the parallax generating optics 118. In other examples, the material 116 may occupy only a portion of the volume between the display panel 112 and the parallax generating optics 118, such that light originating from the display panel 112 passes through at least a portion of the material 116 and through a volume of air before reaching the parallax generating optics 118. The material 116 may have a quantity of Refractive index. The value can be between about 1.3 and about 2, although other suitable values ​​may be used. Suitable material 116 may include glass, plastic, transparent optical adhesive, etc. In some examples, material 116 may be dispensed in liquid form and then cured in situ, such as by exposure to ultraviolet light or heat. In other examples, material 116 may be manufactured as a solid unit and placed in its position in the autostereoscopic display. For example, material 116 may be used as a cover glass for display panel 112. In some examples, material 116 may be used as a relatively precise spacing element. For example, material 116 may be manufactured to have a specified thickness within a specified thickness tolerance, and the spacing between display panel 112 and parallax generating optics 118 may be set to a value equal to the specified thickness when the autostereoscopic display is assembled.

[0043] like Figure 1 As shown, the multi-view display system 100 may include processing circuitry 130. Processing circuitry 130 may include a non-transitory computer-readable storage medium 132, such as a hard disk, solid-state drive, memory, optical media, magnetic media, semiconductor media, punched card, etc. The non-transitory computer-readable storage medium 132 may be locally incorporated with the processing circuitry 130, or may be located remotely from the processing circuitry 130 and accessible via a wired or wireless connection. The non-transitory computer-readable storage medium 132 may store instructions 134 for performing some or all of the steps of a specific task or a series of tasks or a method of execution.

[0044] Instruction 134, when executed by processing circuitry 130, can cause processing circuitry 130 to perform operation 136. For example, instruction 134 can be used to generate a stereo image pair from a single-view image. Suitable operation 136 for generating a stereo image pair from a single-view image may include, among other operations: receiving data of a specified single-view image using processing circuitry 130; determining saliency information indicating a region of interest (ROI) of the single-view image using processing circuitry 130; determining a convergence plane for a stereo representation of the single-view image using processing circuitry 130, the convergence plane being a function of the disparity characteristics of the ROI of the single-view image; generating a disparity map using processing circuitry 130 based on the single-view image and the convergence plane; and generating a stereo image pair using processing circuitry 130 using the single-view image and the disparity map. By performing these operations 136, processing circuitry 130 can cause multi-view display system 100 to perform the generation of a stereo image pair from a single-view image. Operation 136 may also include displaying the stereo image pair using multi-view display system 100. These operations 136 are described in detail below.

[0045] Figure 8 An example of generating a stereo image pair from a single field-of-view image is shown. Quantity 800 can be a data structure, such as an array, matrix, or variable. Quantity 800 can be stored in memory. When processing circuitry 130 performs operation 136, processing circuitry 130 can read one or more of quantities 800 from memory and write one or more of quantities 800 into memory. Quantity 800 is merely an example of quantities that can be used to generate a stereo image pair from a single field-of-view image. Other suitable quantities can be used.

[0046] Processing circuitry 130 can receive data specifying a single field-of-view image 802. This data can be received from a memory, a local storage device, or a remote storage device. This data can be included as one or more frames of a video signal, specifying the single field-of-view image 802 as one of a series of sequential single field-of-view images.

[0047] The processing circuit 130 can determine saliency information indicating the region of interest 806 of the single field of view image 802 from the single field of view image 802.

[0048] To quantify this saliency information, the processing circuit 130 can generate a saliency map 804 based on the single-view image 802. The saliency map 804 can be an array or matrix, such as having the same size as the single-view image 802 (e.g., in pixels). The saliency map 804 can include values ​​corresponding to and determining the region of interest 806 of the single-view image 802 in relation to its importance to the human visual system. The saliency map 804 can have saliency values ​​(S) in an array corresponding to the single-view image 802. For a pixel in the saliency map 804, a relatively large S value can indicate a relatively high saliency for the corresponding pixel in the single-view image 802. For a pixel in the saliency map 804, a relatively small S value (such as zero) can indicate a relatively low saliency for the corresponding pixel in the single-view image 802. In some examples, the value of S can take a finite or infinite number of values. In some examples, the saliency map 804 may be binary, such as having a (single) positive or non-zero S-value in the region of interest and a zero S-value outside the region of interest. For example, the values ​​of the saliency map 804 may include non-zero values ​​in the region of interest 806 and zero values ​​outside the region of interest 806. In other examples, the value may be any suitable value, such as a magnitude corresponding to the relative importance of the human visual system. The region of interest 806 may include less than the entire single-field image 802. The region of interest 806 may include one or more regions considered important. In other words, the region of interest 806 may include a continuous region in the single-field image 802 or multiple non-contiguous regions in the single-field image 802.

[0049] The processing circuit 130 can determine a convergence plane for a stereoscopic representation of the single-field-of-view image 802. The convergence plane can be a function of the disparity characteristics of the region of interest 806 of the single-field-of-view image 802. To determine the convergence plane, the processing circuit 130 can generate a temporary disparity map 808 based on the single-field-of-view image 802. As described above, the processing circuit 130 can generate the temporary disparity map 808 using artificial intelligence; the temporary disparity map can be artificial or simulated. The temporary disparity map 808 can include values ​​corresponding to depth information within the single-field-of-view image 802 and including the disparity characteristics of the single-field-of-view image 802. The temporary disparity map 808 can extend over the entire range of the single-field-of-view image 802, such as including regions within and outside the region of interest 806.

[0050] To limit the disparity characteristics to only the region of interest 806, the processing circuit 130 can generate a weighted disparity map 810. This weighted disparity map 810 can use the values ​​of a temporary disparity map 808 weighted by the corresponding values ​​of the saliency map 804. For a binary saliency map 804, or a saliency map 804 having zero values ​​outside the region of interest 806, such weighting ensures that the weighted disparity map 810 uses the values ​​of the temporary disparity map 808 within the region of interest 806 and does not use the values ​​of the temporary disparity map 808 outside the region of interest 806.

[0051] The convergence plane can be located using disparity information from the region of interest 806. Typically, a disparity map defines the convergence plane as the plane where the disparity map value is zero. To ensure the convergence plane is located at the plane of the autostereoscopic display, the processing circuit 130 can subtract the average disparity in the region of interest 806, making the average disparity in the region of interest 806 zero. Subtracting the average in this way can move the convergence plane from an (unspecified) initial position to a final position at or near the plane of the autostereoscopic display. The temporary disparity map 808 can represent the disparity in the region of interest 806 before average correction; the final disparity map 812 can represent the disparity in the region of interest 806 after average correction.

[0052] The processing circuit 130 can generate a final disparity map 812 based on the single-view image 802 and the convergence plane. More specifically, the processing circuit 130 can generate the final disparity map 812 based on the temporary disparity map 808 and the saliency map 804. As a specific example, the processing circuit 130 can perform the following three operations to generate the final disparity map 812. First, the processing circuit 130 can multiply the value of the temporary disparity map 808 by the corresponding value of the saliency map 804 to form the weighted disparity map 810. Second, the processing circuit 130 can average the values ​​of the weighted disparity map 810 to form an average value. Third, the processing circuit 130 can subtract the average value from the values ​​of the weighted disparity map 810 to form the final disparity map 812. The three operations detailed above are merely examples of generating the final disparity map 812 based on the temporary disparity map 808 and the saliency map 804. Other operations can be used.

[0053] The processing circuit 130 can use the single-field-of-view image 802 and the final parallax map 812 to generate a stereoscopic image pair 814. The stereoscopic image pair 814 may include a left image 816 and a right image 818. When the autostereoscopic display shows the stereoscopic image pair 814 to a viewer, the autostereoscopic display directs the left image 816 to the viewer's left eye and the right image 818 to the viewer's right eye.

[0054] For method 900 discussed below, the single-view image 802 is the input, and the stereoscopic image pair 814 is the output. The stereoscopic image pair 814 can be displayed on an autostereoscopic display, such as... Figure 1 A multi-view display system 100.

[0055] Figure 9 A flowchart illustrating an example of a method 900 for generating stereoscopic image pairs from a single field-of-view image is shown. This method 900 can be used in automated stereoscopic display systems (such as...) Figure 1 The method 900 is executed on a multi-view display system 100, applied to, or may be used by or in conjunction with an autostereoscopic display system. The method 900 can be stored as instruction 134 on a non-transitory computer-readable storage medium 132. When executed by the processing circuitry 130, instruction 134 causes the processing circuitry 130 to perform operation 136. Operation 136 may include operations detailed below with respect to the method 900. The method 900 is merely one method for generating stereoscopic image pairs from a single-view image; other suitable methods may be used.

[0056] At operation 902, the processing circuit 130 can receive data from a specified single-view image 802.

[0057] At operation 904, the processing circuitry 130 can determine saliency information from the single-view image 802 that indicates a region of interest 806 of the single-view image 802. The region of interest 806 may include less than the entire single-view image 802. The region of interest 806 may include one or more regions considered important, such as regions important to the human visual system.

[0058] At operation 906, the processing circuit 130 can determine the convergence plane for stereo representation of the single field-of-view image 802 based on the parallax characteristics of the region of interest 806 of the single field-of-view image 802.

[0059] At operation 908, the processing circuit 130 can generate a final disparity map 812 based on the single field-of-view image 802 and the convergence plane.

[0060] At operation 910, the processing circuit 130 can use the single field-of-view image 802 and the final disparity map 812 to generate a stereoscopic image pair 814.

[0061] The autostereoscopic display (such as multi-view display 110) can display the stereoscopic image pair 814 such that when the stereoscopic image pair 814 is displayed on the autostereoscopic display: the convergence plane coincides with the plane of the autostereoscopic display (and the region of interest), a first portion of the stereoscopic image pair 814 appears to be above the plane of the autostereoscopic display, and a second portion of the stereoscopic image pair 814 appears to be below the plane of the autostereoscopic display. A viewer tracker (such as viewer tracker 120) can determine the position of a viewer (such as viewer 42).

[0062] The autostereoscopic display may include a display panel, such as display panel 112. The display panel may have a subpixel array, such as an array of subpixels 114, which can display the stereoscopic image pair 814. The autostereoscopic display may include parallax generating optics, such as parallax generating optics 118, which can direct light from the display panel to the viewer. The parallax generating optics may include one of a lenticular lens (such as lenticular lens 402) or a parallax barrier with a transmission slit (such as parallax barrier 602 with a transmission slit 704). The parallax generating optics may be constant along an optical axis (such as the optical axis (OA)) having a tilt angle relative to the display panel. This tilt angle may be within a specified angular tolerance of forty-five degrees, such as one, two, five, ten degrees, or another suitable angular tolerance. The parallax generating optics may be periodic along an axis orthogonal to the optical axis. The processing circuit 130 can arrange the left image 816 and the right image 818 of the stereoscopic image pair 814 on the sub-pixel array, such that the parallax generating optics guides the left image 816 and the right image 818 to the viewer's corresponding eye at the viewer's position.

[0063] To further illustrate the systems and methods disclosed herein, a list of non-limiting examples is provided below. Each of these non-limiting examples may exist independently or may be combined with any one or more other examples in any permutation or combination.

[0064] In Example 1, an automatic stereoscopic display system may include processing circuitry configured to perform operations. These operations may include: receiving data of a specified single-field-of-view image; determining saliency information from the single-field-of-view image indicating a region of interest (ROI) of the single-field-of-view image; determining a convergence plane for a stereoscopic representation of the single-field-of-view image, the convergence plane being a function of the disparity characteristics of the ROI of the single-field-of-view image; generating a disparity map based on the single-field-of-view image and the convergence plane; and generating a stereoscopic image pair using the single-field-of-view image and the disparity map.

[0065] In Example 2, the autostereoscopic display system of Example 1 may optionally be configured such that the operation further includes: generating a temporary disparity map based on the single field-of-view image; generating a saliency map based on the single field-of-view image; and generating the disparity map based on the temporary disparity map and the saliency map.

[0066] In Example 3, the autostereoscopic display system of any of Examples 1-2 may optionally be configured such that the operation further includes: the temporary disparity map including values ​​corresponding to depth information within the single field-of-view image and including values ​​of the disparity characteristics of the single field-of-view image; the saliency map including values ​​corresponding to importance to the human visual system and determining the region of interest of the single field-of-view image; and the disparity map defining the convergence plane as a plane where the value of the disparity map is equal to zero.

[0067] In Example 4, the autostereoscopic display system of any of Examples 1-3 may optionally be configured such that the operation further includes generating the disparity map using the values ​​of the temporary disparity map weighted by the corresponding values ​​of the saliency map.

[0068] In Example 5, the autostereoscopic display system of any of Examples 1-4 may optionally be configured such that the operation further includes generating the disparity map using the value of the temporary disparity map in the region of interest and not using the value of the temporary disparity map outside the region of interest.

[0069] In Example 6, the autostereoscopic display system of any of Examples 1-5 may optionally be configured such that: the value of the saliency map is binary; the value of the saliency map includes non-zero values ​​in the region of interest and zero values ​​outside the region of interest; and the operation further includes: multiplying the value of the temporary disparity map by the corresponding value of the saliency map to form a weighted disparity map; averaging the values ​​of the weighted disparity map to form an average value; and subtracting the average value from the values ​​of the weighted disparity map to form the disparity map.

[0070] In Example 7, the autostereoscopic display system of any of Examples 1-6 can optionally be configured such that the region of interest includes less than the entire single field of view image.

[0071] In Example 8, the autostereoscopic display system of any of Examples 1-7 can optionally be configured such that the region of interest includes one or more regions considered important.

[0072] In Example 9, the autostereoscopic display system of any one of Examples 1-8 may optionally further include: an autostereoscopic display configured to display the stereoscopic image pair such that when the stereoscopic image pair is displayed on the autostereoscopic display: the convergence plane coincides with the plane of the autostereoscopic display; a first portion of the stereoscopic image pair appears to be above the plane of the autostereoscopic display; and a second portion of the stereoscopic image pair appears to be below the plane of the autostereoscopic display; and a viewer tracker configured to determine the position of a viewer.

[0073] In Example 10, the autostereoscopic display system of any one of Examples 1-9 may optionally be configured such that the autostereoscopic display includes: a display panel having a subpixel array configured to display the stereoscopic image pair; and a parallax generating optics configured to direct light from the display panel to the viewer, the parallax generating optics including one of a lenticular lens or a parallax barrier having a transmission slit, the parallax generating optics being constant along an optical axis having an angle of inclination relative to the display panel within a specified angular tolerance of forty-five degrees, the parallax generating optics being periodic along an axis orthogonal to the optical axis; and wherein the operation further includes: arranging the left and right images of the stereoscopic image pair on the subpixel array such that the parallax generating optics directs the left and right images to the viewer's respective eyes at the viewer's position.

[0074] In Example 11, a method for generating a stereo image pair from a single-view image may include: receiving data specifying the single-view image using a processing circuit; determining saliency information indicating a region of interest (ROI) of the single-view image using the processing circuit; determining a convergence plane for a stereo representation of the single-view image using the processing circuit, the convergence plane being a function of the disparity characteristics of the ROI of the single-view image; generating a disparity map based on the single-view image and the convergence plane using the processing circuit; and generating the stereo image pair using the single-view image and the disparity map using the processing circuit.

[0075] In Example 12, the method of Example 11 may optionally further include: generating a temporary disparity map based on the single field-of-view image using the processing circuit; generating a saliency map based on the single field-of-view image using the processing circuit; and generating the disparity map based on the temporary disparity map and the saliency map using the processing circuit.

[0076] In Example 13, the method of any one of Examples 11-12 may optionally be configured such that: the temporary disparity map includes values ​​corresponding to depth information within the single field-of-view image and including the disparity characteristics of the single field-of-view image; the saliency map includes values ​​corresponding to importance to the human visual system and determining the region of interest of the single field-of-view image; and the disparity map defines the convergence plane as the plane where the value of the disparity map is equal to zero.

[0077] In Example 14, the method of any one of Examples 11-13 may optionally further include generating the disparity map using the values ​​of the temporary disparity map weighted by the corresponding values ​​of the saliency map.

[0078] In Example 15, the method of any one of Examples 11-14 may optionally further include: using the processing circuitry to generate the disparity map using the value of the temporary disparity map in the region of interest and not using the value of the temporary disparity map outside the region of interest.

[0079] In Example 16, a non-transitory computer-readable storage medium may store instructions for generating a stereo image pair from a single-view image. When executed by processing circuitry, the instructions may cause the processing circuitry to perform operations. These operations may include: receiving data specifying the single-view image using the processing circuitry; determining saliency information from the single-view image indicating a region of interest in the single-view image using the processing circuitry; determining a convergence plane for a stereo representation of the single-view image, the convergence plane being a function of the disparity characteristics of the region of interest in the single-view image using the processing circuitry; generating a disparity map based on the single-view image and the convergence plane using the processing circuitry; and generating the stereo image pair using the single-view image and the disparity map using the processing circuitry.

[0080] In Example 17, the non-transitory computer-readable storage medium of Example 16 may optionally be configured such that the operation further includes: generating a temporary disparity map based on the single-view image using the processing circuitry; generating a saliency map based on the single-view image using the processing circuitry; and generating the disparity map based on the temporary disparity map and the saliency map using the processing circuitry.

[0081] In Example 18, the non-transitory computer-readable storage medium of any of Examples 16-17 may optionally be configured such that: the temporary disparity map includes values ​​corresponding to depth information within the single field-of-view image and including the disparity characteristics of the single field-of-view image; the saliency map includes values ​​corresponding to importance to the human visual system and determining the region of interest of the single field-of-view image; and the disparity map defines the convergence plane as a plane where the value of the disparity map is equal to zero.

[0082] In Example 19, the non-transitory computer-readable storage medium of any of Examples 16-18 may optionally be configured such that the operation further includes generating the disparity map using the values ​​of the temporary disparity map weighted by the corresponding values ​​of the saliency map.

[0083] In Example 20, the non-transitory computer-readable storage medium of any of Examples 16-19 may optionally be configured such that the operation further includes: using the processing circuitry to generate the disparity map using the value of the temporary disparity map in the region of interest and not using the value of the temporary disparity map outside the region of interest.

[0084] The above detailed description includes references to the accompanying drawings, which form part of the detailed description. The drawings illustrate, by way of illustration, specific embodiments in which the invention can be practiced. These embodiments are also referred to herein as “examples.” Such examples may include elements other than those shown or described. However, examples providing only those elements shown or described are also contemplated. Furthermore, other examples may use any combination or arrangement of those elements (or one or more aspects thereof) shown or described, relative to a particular example (or one or more aspects thereof) or relative to other examples shown or described herein (or one or more aspects thereof).

[0085] In this document, as is common in patent documents, the terms “a” or “an” are used to include one or more, independent of any other instances or uses of “at least one” or “one or more.” In this document, the term “or” is used to refer to a non-exclusive “or,” such that “A or B” can include “A but not B,” “B but not A,” and “A and B,” unless otherwise stated. In the appended claims, the terms “including” and “in which” are used as concise English equivalents to the corresponding terms “comprising” and “wherein.” Furthermore, in the following claims, the terms “comprising” and “including” are open-ended, meaning that a system, apparatus, article, or process that includes elements other than those listed after such terms in the claim is still considered to be within the scope of that claim. Additionally, in the following claims, the terms “first,” “second,” and “third,” etc., are used merely as designations and are not intended to impose numerical requirements on their objects.

[0086] The above description is intended to be illustrative and not restrictive. For example, the examples (or one or more aspects thereof) described above can be used in combination with each other. Other embodiments can be used by those skilled in the art after reading the above description. It should be understood at the time of filing that it is not intended to interpret or limit the scope or meaning of the claims. Moreover, in the above detailed description, various features can be combined together to simplify this disclosure. This should not be construed as meaning that any unclaimed disclosed feature is essential to any claim. Rather, the inventive subject matter may lie in all features of fewer than the embodiments of a particular disclosure. Therefore, the following claims are incorporated herein by reference, wherein each claim is an independent, separate embodiment, and these embodiments are contemplated to be combined with each other in various combinations or arrangements. The scope of the invention should be determined with reference to the appended claims and the full scope of their equivalents.

Claims

1. An automatic stereoscopic display system, comprising: The processing circuit is configured to perform operations, the operations including: Receives data from a specified single field-of-view image; From the single-view image, determine saliency information indicating the region of interest of the single-view image; Determine a convergence plane for a stereo representation of the single-view image, the convergence plane being a function of the disparity characteristics of the region of interest in the single-view image; A disparity map is generated based on the single-view image and the convergence plane; and A stereo image pair is generated using the single-view image and the disparity map.

2. The automatic stereoscopic display system according to claim 1, wherein the operation further includes: A temporary disparity map is generated based on the single-view image; A saliency map is generated based on the single-view image; as well as The disparity map is generated based on the temporary disparity map and the saliency map.

3. The automatic stereoscopic display system according to claim 2, wherein: The temporary disparity map includes values ​​corresponding to the depth information within the single field-of-view image and also includes values ​​of the disparity characteristics of the single field-of-view image; The saliency map includes values ​​corresponding to the importance to the human visual system and determining the region of interest in the single field of view image; as well as The disparity map defines the convergence plane as the plane whose value in the disparity map is equal to zero.

4. The automatic stereoscopic display system according to claim 3, wherein the operation further includes: The disparity map is generated using the values ​​of the temporary disparity map, which are weighted by the corresponding values ​​of the saliency map.

5. The automatic stereoscopic display system according to claim 3, wherein the operation further includes: The disparity map is generated using the values ​​of the temporary disparity map within the region of interest but not using the values ​​of the temporary disparity map outside the region of interest.

6. The automatic stereoscopic display system according to claim 3, wherein: The saliency plot has binary values; The values ​​of the saliency map include non-zero values ​​in the region of interest and zero values ​​outside the region of interest; as well as The operation also includes: The value of the temporary disparity map is multiplied by the corresponding value of the saliency map to form a weighted disparity map; The values ​​of the weighted disparity map are averaged to form an average value; and The average value is subtracted from the values ​​of the weighted disparity map to form the disparity map.

7. The automatic stereoscopic display system according to claim 3, wherein the region of interest includes less than all of the single field-of-view images.

8. The automatic stereoscopic display system according to claim 3, wherein the region of interest includes one or more regions considered important.

9. The automatic stereoscopic display system according to claim 1, further comprising: An autostereoscopic display is configured to display the stereoscopic image pair such that when the stereoscopic image pair is displayed on the autostereoscopic display: The convergence plane coincides with the plane of the automated stereoscopic display. The first portion of the stereoscopic image pair appears to be positioned above the plane of the autostereoscopic display; as well as The second part of the stereoscopic image pair appears to be located below the plane of the autostereoscopic display; as well as A viewer tracker is configured to determine the viewer's location.

10. The automatic stereoscopic display system according to claim 9, wherein the automatic stereoscopic display comprises: The display panel has a sub-pixel array configured to display the stereoscopic image pairs; as well as Parallax generating optics are configured to direct light from the display panel toward the viewer. The parallax generating optics include either a cylindrical lens or a parallax barrier with a transmission slit. The parallax generating optics remain constant along the optical axis, which has an angle of inclination relative to the display panel. The tilt angle is within a specified 45-degree tolerance. The parallax generating optics are periodic along an axis orthogonal to the optical axis; as well as The operation further includes: The left and right images of the stereoscopic image pair are arranged on the sub-pixel array such that the parallax generating optics guides the left and right images to the viewer's corresponding eye at the viewer's position.

11. A method for generating a stereo image pair from a single field-of-view image, the method comprising: The processing circuit receives data for the specified single-view image. The processing circuitry is used to determine saliency information indicating the region of interest in the single field of view image. The processing circuit is used to determine a convergence plane for a stereo representation of the single-view image, the convergence plane being a function of the disparity characteristics of the region of interest in the single-view image; The processing circuit generates a disparity map based on the single-view image and the convergence plane. as well as Using the processing circuit, a stereo image pair is generated using the single-view image and the disparity map.

12. The method of claim 11, further comprising: The processing circuit generates a temporary disparity map based on the single-view image. The processing circuit generates a saliency map based on the single-view image. as well as The processing circuit generates the disparity map based on the temporary disparity map and the saliency map.

13. The method according to claim 12, wherein: The temporary disparity map includes values ​​corresponding to the depth information within the single field-of-view image and also includes values ​​of the disparity characteristics of the single field-of-view image; The saliency map includes values ​​corresponding to the importance to the human visual system and determining the region of interest in the single field of view image; as well as The disparity map defines the convergence plane as the plane whose value in the disparity map is equal to zero.

14. The method of claim 13, further comprising: The disparity map is generated using the values ​​of the temporary disparity map, which are weighted by the corresponding values ​​of the saliency map.

15. The method of claim 13, further comprising: The processing circuit is used to generate the disparity map by using the values ​​of the temporary disparity map within the region of interest but not by using the values ​​of the temporary disparity map outside the region of interest.

16. A non-transitory computer-readable storage medium storing instructions for generating stereoscopic image pairs from a single field-of-view image, the instructions causing the processing circuitry to perform operations when executed by the processing circuitry, the operations including: The processing circuit receives data specifying the single field-of-view image. The processing circuitry is used to determine saliency information indicating the region of interest in the single field of view image. The processing circuit is used to determine a convergence plane for a stereo representation of the single-view image, the convergence plane being a function of the disparity characteristics of the region of interest in the single-view image; The processing circuit generates a disparity map based on the single-view image and the convergence plane. as well as Using the processing circuit, a stereo image pair is generated using the single-view image and the disparity map.

17. The non-transitory computer-readable storage medium according to claim 16, wherein, The operation also includes: The processing circuit generates a temporary disparity map based on the single-view image. The processing circuit generates a saliency map based on the single-view image; and The processing circuit generates the disparity map based on the temporary disparity map and the saliency map.

18. The non-transitory computer-readable storage medium according to claim 17, wherein: The temporary disparity map includes values ​​corresponding to the depth information within the single field-of-view image and also includes values ​​of the disparity characteristics of the single field-of-view image; The saliency map includes values ​​corresponding to the importance to the human visual system and determining the region of interest in the single field of view image; as well as The disparity map defines the convergence plane as the plane whose value in the disparity map is equal to zero.

19. The non-transitory computer-readable storage medium according to claim 18, wherein, The operation also includes: The disparity map is generated using the values ​​of the temporary disparity map, which are weighted by the corresponding values ​​of the saliency map.

20. The non-transitory computer-readable storage medium according to claim 18, wherein, The operation also includes: The processing circuit is used to generate the disparity map by using the values ​​of the temporary disparity map within the region of interest but not by using the values ​​of the temporary disparity map outside the region of interest.

Images