Stereoscopic image pair re-convergence based on saliency

Abstract

An autostereoscopic display system can re-converge a stereoscopic image pair. Processing circuitry can determine saliency information indicative of a region of interest of an initial stereoscopic image pair, determine a convergence plane based on disparity characteristics of the region of interest of the initial stereoscopic image pair, and generate a final stereoscopic image pair using the initial stereoscopic image pair and the convergence plane. The processing circuitry can introduce a horizontal offset between a left image and a right image of the initial stereoscopic image pair to form a left image and a right image of the final stereoscopic image pair. The horizontal offset can have a selected offset value such that a difference between the left image and the right image of the final stereoscopic image pair in the region of interest is less than a difference between the left image and the right image of the initial stereoscopic image pair in the region of interest.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 623,748, filed January 22, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This document generally relates to display systems, and more specifically to multi-view displays, 3D displays, or autostereoscopic displays. Background Technology

[0004] Multi-view displays can provide viewers with different views of a multi-view image. Stereo displays can provide two different views of a three-dimensional image to both of the viewer's eyes. Autostereo displays can provide two different views to both of the viewer's eyes without requiring the viewer to wear special glasses or goggles. Efforts are underway to improve autostereo displays. Summary of the Invention

[0005] In one example, the autostereoscopic display system may include processing circuitry capable of performing operations. These operations may include: receiving data of a specified initial stereoscopic image pair; determining saliency information from the initial stereoscopic image pair that indicates the region of interest (ROI) of the initial stereoscopic image pair; determining a convergence plane based on the disparity characteristic of the ROI of the initial stereoscopic image pair; and generating a final stereoscopic image pair using the initial stereoscopic image pair and the convergence plane.

[0006] In the example, the method may include reconverging the stereo image pair, and optionally using an automatic stereo system to display the image pair. The method may include: receiving data for a specified initial stereo image pair using processing circuitry; determining saliency information from the initial stereo image pair indicating the region of interest (ROI) of the initial stereo image pair using processing circuitry; determining a convergence plane based on the parallax characteristics of the ROI of the initial stereo image pair using processing circuitry; and generating a final stereo image pair using the initial stereo image pair and the convergence plane using processing circuitry.

[0007] The content of this document is intended to provide an overview of the subject matter. It is not intended to provide an exclusive or exhaustive explanation of the invention. Detailed descriptions are included to provide further information about the subject matter. Attached Figure Description

[0008] Figure 1 An exploded perspective diagram of an example of a multi-view display system is shown.

[0009] Figure 2A front view of an example display panel including an array of light-emitting diodes is shown.

[0010] Figure 3 A front view of an example display panel including backlight and light valve array is shown.

[0011] Figure 4 A front view of an example of a parallax-generating optics device including a cylindrical lens is shown.

[0012] Figure 5 It shows Figure 4 A cross-sectional view of a cylindrical lens.

[0013] Figure 6 A front view of an example of a parallax-generating optics including a parallax barrier is shown.

[0014] Figure 7 A transmission slit is shown. Figure 6 A cross-sectional view of the parallax barrier.

[0015] Figure 8 A flowchart illustrating an example of a method for reconverging stereo image pairs is shown.

[0016] Figure 9 A front view of an example of the left and right images of the initial stereoscopic image pair is shown.

[0017] Figure 10 A front view showing an example of saliency information indicating the region of interest of an initial stereo image pair.

[0018] Figure 11 A schematic diagram illustrating an example of determining a convergence plane based on the parallax characteristics of the region of interest from an initial stereo image pair.

[0019] Figure 12 A front view of an example of the left and right images of the final stereoscopic image pair is shown. Detailed Implementation

[0020] An autostereoscopic display system can receive pairs of stereoscopic images for display on an autostereoscopic display. The received stereoscopic image pairs may require additional processing before display to improve the autostereoscopic viewing experience. For example, to enhance viewer comfort, the autostereoscopic display system can select different convergence planes for the stereoscopic image pairs in a process known as "reconvergence." More specifically, the autostereoscopic display system can receive an initial pair of stereoscopic images, reconverge the initial pair to form a final pair of stereoscopic images with convergence planes selected in a specific manner, and display the final pair of stereoscopic images on an autostereoscopic display.

[0021] In the reconvergence technique described in detail below, the autostereoscopic display system can use saliency information from the initial stereoscopic image pair to identify the region of interest (ROI) in the stereoscopic image. The autostereoscopic display system can then reconverge the initial stereoscopic image pair based on parallax information within the ROI. When the autostereoscopic display system displays the final stereoscopic image pair on the autostereoscopic display, the viewer can perceive the ROI as being approximately located at or near the plane of the autostereoscopic display, rather than too close to the viewer (e.g., in front of the autostereoscopic display) or too far away (e.g., behind the autostereoscopic display). As used herein, the term parallax is intended to represent the positional difference of a given location in a scene between the left and right images of a stereoscopic image pair representing the scene.

[0022] As a specific example, the initial stereoscopic image pair may include a flower in the background. The autostereoscopic display system can determine that the flower forms a region of interest, such that the perimeter of the flower defines the perimeter of the region of interest. The autostereoscopic display system can reconverge the initial stereoscopic image pair based on parallax information in the flower region (such as information from the background not being used) to form a final stereoscopic image pair. When the autostereoscopic display system displays the final stereoscopic image pair on an autostereoscopic display, the flower can be perceived as being at or near the plane of the autostereoscopic display, rather than far above or far behind the plane of the autostereoscopic display.

[0023] Figure 1 An exploded perspective schematic diagram of an example of a multi-view display system 100 including a multi-view display 110 is shown. Figure 1 The configuration is just one example of the multi-view display system 100; other configurations can be used.

[0024] Figure 1 The notation shown and used below assumes that the multi-view display 110 is in ( x , y The z-axis extends in the plane, and extends away from the multiview display 110 along a direction orthogonal to the plane of the multiview display 110 and generally toward the viewer 42. Other notation conventions may be used.

[0025] The multiview display system 100 may include a multiview display 110. The multiview display 110 may provide different views of a multiview image to a viewer 42. For example, as the viewer 42 moves in space, the multiview display 110 may direct different views of the multiview image to the viewer 42's left and right eyes, allowing the viewer 42 to observe different views of the multiview image from different positions or orientations. In some configurations, the multiview display 110 may provide multiple views at various fixed location areas in space, allowing the multiview display 110 to operate without eye tracking. In other configurations, such as the autostereoscopic configuration described in detail below, the multiview display system 100 may use eye tracking to dynamically and continuously (or at relatively frequent discrete times) determine the viewer 42's position, and in response, may dynamically and continuously control how the multiview display 110 displays the multiview image, such that as the viewer 42 moves in space relative to the position of the multiview display 110, the multiple views follow the viewer 42 or follow the tracked eye position of the viewer 42.

[0026] For a multi-view display 110 configured to provide only two different views of a multi-view image, the multi-view display 110 can be an autostereoscopic display or a three-dimensional (3D) display. An autostereoscopic display can provide a left image to the left eye of viewer 42 and a right image to the right eye of viewer 42. The left and right images can correspond to different views of an object or scene, and can allow viewer 42 to perceive the object or scene in 3D using only the viewer's naked eye without the use of additional glasses or a head-mounted device.

[0027] The multi-view display system 100 may include a viewer tracker 120 capable of dynamically determining the position of a viewer 42. The multi-view display system 100 can use the determined position of the viewer 42 to guide a left image to the viewer 42's left eye and a right image to the viewer 42's right eye. Because the viewer's position can change as the viewer 42 moves in space, eye tracking allows the multi-view display system 100 to follow the viewer 42, enabling the autostereoscopic display to automatically guide the left image to the left eye and the right image to the right eye at the viewer 42's (dynamically changing) position. The viewer tracker 120 can provide the tracking position of the viewer 42, such as the tracking position of the viewer 42's head, the tracking position of one or both eyes of the viewer 42, or the tracking position of another anatomical feature of the viewer 42. Viewer tracker 120 may be coupled to processing circuitry 130 (described below) or a controller, such as by providing viewer 42 positioning data representing the measured position or location of viewer 42 (in Figure 1The coordinates are shown in the figure. xv , yv and zv Viewer tracker 120 may provide viewer 42 position data to processing circuitry 130 at regular or irregular intervals. In a particular example of viewer tracker 120, a camera may capture an image of viewer 42. Viewer tracker 120 may also include an image processor (or a general-purpose computer programmed as an image processor) that can determine the position of viewer 42 within the captured image to provide the tracked position. In some examples, processing circuitry 130 may include the image processor of viewer tracker 120. In other examples, processing circuitry 130 may be separate from the image processor of viewer 42 tracker. Other suitable viewer trackers may be used, including those based on lidar (e.g., using the time-of-flight of reflected light on a scene to determine distances to one or more objects in the scene, such as the viewer's head or the viewer's eyes) or other technologies. Processing circuitry 130 may use the output of viewer tracker 120, along with other data, to perform one or more downstream calculations involving providing a left view or left image to viewer 42's left eye and a right view or right image to viewer 42's right eye.

[0028] An autostereoscopic display can be a lenticular autostereoscopic display. In a lenticular autostereoscopic display, display panel 112 can display multi-view images, and parallax generating optics 118 can guide light from display panel 112 to viewer 42, such that the left image can be seen from viewer 42's left eye and the right image from viewer 42's right eye. During the use of the lenticular autostereoscopic display, processing circuitry 130 can track the position of viewer 42 and can use the tracked position to dynamically determine how to distribute the content of the multi-view images on the surface area of ​​display panel 112 (e.g., using pixels distributed on display panel 112), so that even if viewer 42 changes position, the left image remains visible from viewer 42's left eye and the right image remains visible from viewer 42's right eye. In this way, position tracking and image content distribution can be performed in software, so that following viewer 42's position does not involve physically moving any components of the lenticular autostereoscopic display relative to each other. Examples of suitable display panels and suitable parallax generating optics are described below.

[0029] In the example, display panel 112 can display a multi-view image. Display panel 112 may have an array of subpixels 114 that can display the image according to stereoscopic coordinates associated with viewer 42. Subpixels 114 may be located in a grid with axes (e.g., x shaft andy The sub-pixel 114 is located at a sub-pixel position in the grid of the axis. Each sub-pixel 114 can generate light with a specified color. For example, sub-pixels 114 may include red sub-pixels, green sub-pixels, and blue sub-pixels that generate red, green, and blue light, respectively. Other color / wavelength schemes may be used. Sub-pixels 114 can be divided into pixels, wherein each pixel includes at least two sub-pixels 114 that generate light of different colors. The display panel 112 can receive display panel drive electrical signals 138 from the processing circuit 130 (described below), which can specify how the content of the multi-view image is distributed on the pixels and / or sub-pixels 114 of the display panel 112. The following description and Figure 2 and Figure 3 The diagram shows two possible configurations for the display panel 112. Other configurations can also be used.

[0030] Figure 2 A front view of an example display panel 112A including an array 202 of light-emitting diodes 204 (such as an array 202 of organic light-emitting diodes) is shown. Each light-emitting diode 204 may correspond to a sub-pixel. The array 202 of light-emitting diodes 204 may include a red light-emitting diode 204R, a green light-emitting diode 204G, and a blue light-emitting diode 204B, which correspond to the red, green, and blue sub-pixels, respectively. Each light-emitting diode 204 may generate light in response to an electrical signal (such as a display panel drive signal 138) provided by processing circuitry 130 or by a suitable light-emitting diode driving circuitry communicating with processing circuitry 130. Processing circuitry 130 may directly power a designated light-emitting diode 204, the power varying according to the intensity at the corresponding location in the image. The power delivered to the light-emitting diodes 204 may optionally be pulse-width modulated at a modulation frequency greater than that perceptible to the human eye. Using pulse-width modulation simplifies the design of the light-emitting diode array controller because it can generate arbitrary average power levels from relatively small amounts of instantaneous power levels by changing the power duty cycle. In some examples, the array 202 of light-emitting diodes 204 may be arranged in a rectangular or square repeating pattern on the surface region 206 of the array 202. For example, the array 202 may have grid axes 208 that are orthogonal to each other. In some examples, the grid axes 208 may be parallel to the edge 210 of the array 202 of light-emitting diodes 204.

[0031] Figure 3 A front view of an example display panel 112B including a backlight 302 and a light valve array 304 is shown. Although Figure 3The backlight 302 and the light valve array 304 are shown as separate, but in practice, the backlight 302 and the light valve array 304 can be in contact or can be brought as close as possible together. The backlight 302 can provide illumination with a uniform or substantially uniform intensity over a surface area of ​​the backlight 302. The backlight 302 can provide illumination with a relatively broad spectrum, such as including most or all of the visible portion of the electromagnetic spectrum. The backlight 302 can provide illumination to a continuous propagation angle toward the light valve array 304. The backlight 302 can provide unmodulated illumination to the light valve array 304. The light valve array 304 can include light valves 306, which can be individually controlled by the processing circuitry 130 or can be controlled by the processing circuitry 130 in one or more groups (described below). Each light valve 306 can controllably attenuate the illumination from the backlight 302, for example, in response to an electrical signal provided by the processing circuitry 130, such as a display panel drive electrical signal 138, or an electrical signal provided by a suitable light valve drive circuit in communication with the processing circuitry 130. Each light valve 306 may have a corresponding color filter that allows only a portion of the electromagnetic spectrum to pass through the light valve. For example, light valve 306 may include a red light valve 306R, a green light valve 306G, and a blue light valve 306B. The red light valve 306R has a red filter that allows only red light to pass through it, the green light valve 306G has a green filter that allows only green light to pass through it, and the blue light valve 306B has a blue filter that allows only blue light to pass through it. Other color schemes and numbers of colors may be used. Suitable light valves may include liquid crystal light valves, electrophoretic light valves, electrowetting-based light valves, etc. In some examples, the light valves 306 of the light valve array 304 may be arranged in a rectangular or square repeating pattern on the surface region 308 of the light valve array 304. For example, the light valve array 304 may have grid axes 208 that are orthogonal to each other. In some examples, the grid axes 208 may be parallel to the edge 312 of the light valve array 304.

[0032] Refer again Figure 1 The autostereoscopic display may include a parallax generating optics 118 that directs light from the display panel 112 to the viewer 42 such that a left view or left image is visible from the viewer 42's left eye, and a right view or right image is visible from the viewer 42's right eye. Two possible configurations of the parallax generating optics 118 are described below and in Figure 4 and Figure 5 China and in Figure 6 and Figure 7 As shown in the diagram. Other configurations can be used. Figure 4 and Figure 5 as well as Figure 6 and Figure 7 Each configuration in can be compared with Figure 2 and Figure 3 Any configuration of the display panel 112 shown (e.g., Figure 2 LED array 204 or Figure 3 The backlight 302 and the light valve array 304 are used in combination.

[0033] Figure 4 A front view of an example of a parallax generating optics 118A including a cylindrical lens 402 is shown. Figure 5 It shows Figure 4 A cross-sectional view of the cylindrical lens 402. The cylindrical lens 402 may include a plurality of cylindrical lenses 504 spaced equidistantly. The cylindrical lens 402 may have a focal plane that coincides with the display panel 112. The cylindrical lens 402 may be positioned to receive light from the display panel 112 and at least partially focus the received light to direct the light to a designated area near the viewer's eye.

[0034] Figure 6 A front view of an example of a parallax generating optics 118B including a parallax barrier 602 is shown. The parallax barrier may include a plurality of equally spaced transmission slits 704. Figure 7 A parallax barrier 602 with a transmission slit 704 is shown. Figure 6 A cross-sectional view of the image is shown. The parallax barrier 602 may include an array of opaque strips 706 and thin transmission slits 704, arranged to obscure portions of the displayed image in the left and right viewing areas. The transmission slits 704 may be spatially arranged to ensure that the left / right image portions are visible only in their intended corresponding left / right viewing areas. The parallax barrier 602 may be provided by a static physical layer, where the slits are precisely positioned, or electronically generated on an adaptive intermediate liquid crystal display layer.

[0035] Parallax generating optics 118 can be positioned along a grid axis (e.g., x shaft and y The optical axis is at an angle to the axis. OA ) remains unchanged, such as the tilt angle of 45 degrees or approximately 45 degrees relative to the grid axis 208. α For example, the parallax generating optics 118 can have transmission characteristics, such as a cylindrical lens or a transmission slit, which are along the optical axis ( OA ) remains unchanged and along the optical axis ( OA Orthogonal axes are periodic.

[0036] Refer again Figure 1An autostereoscopic display may include a material 116 disposed between a display panel 112 and a parallax generating optics 118. In some examples, the material 116 may extend completely between the display panel 112 and the parallax generating optics 118, such that light originating from the display panel 112 passes only through the material 116 (and does not pass through any air or unfilled volume) before reaching the parallax generating optics 118. In other examples, the material 116 may occupy only a portion of the volume between the display panel 112 and the parallax generating optics 118, such that light originating from the display panel 112 passes through at least some of the material 116 and through a volume of air before reaching the parallax generating optics 118. The material 116 may have a quantity of n Refractive index. n The value can be between about 1.3 and about 2, but other suitable values ​​can be used. Suitable material 116 may include glass, plastic, transparent optical adhesive, etc. In some examples, material 116 may be dispensed in liquid form and then cured in place, such as by exposure to ultraviolet light or heat. In other examples, material 116 may be manufactured as a solid unit and placed in its position in the autostereoscopic display. For example, material 116 may be used as a cover glass for display panel 112. In some examples, material 116 may be used as a relatively precise spacer element. For example, material 116 may be manufactured to have a specified thickness within a specified thickness tolerance and the spacing between display panel 112 and parallax generating optics 118 may be set to a value equal to the specified thickness when assembling the autostereoscopic display.

[0037] like Figure 1 As shown, the multi-view display system 100 may include processing circuitry 130. Processing circuitry 130 may include non-transitory computer-readable storage medium 132, such as a hard disk drive, solid-state drive, memory, optical media, magnetic media, semiconductor media, punched card, etc. Non-transitory computer-readable storage medium 132 may be locally incorporated with processing circuitry 130, or may be located remotely from processing circuitry 130 and accessible via a wired or wireless connection. Non-transitory computer-readable storage medium 132 may store instructions 134 for performing some or all of the steps of a specific task or a series of tasks or a method of execution.

[0038] When executed by processing circuit 130, instruction 134 can cause processing circuit 130 to perform operation 136. For example, instruction 134 can be used to reconverge a stereoscopic image pair. Suitable operation 136 for reconverging the stereoscopic image pair may include the following and other operations: receiving data for a specified initial stereoscopic image pair using processing circuit 130; determining saliency information indicating the region of interest (ROI) of the initial stereoscopic image pair using processing circuit 130; determining a convergence plane using processing circuit 130 based on the parallax characteristics of the ROI of the initial stereoscopic image pair; and generating a final stereoscopic image pair using processing circuit 130 with the initial stereoscopic image pair and the convergence plane. By performing these operations 136, processing circuit 130 can cause multi-view display system 100 to perform the reconvergence of the stereoscopic image pair. Operation 136 may also include displaying the stereoscopic image pair using multi-view display system 100. These operations 136 are described in detail below.

[0039] Figure 8 A flowchart illustrating an example of a method 800 for reconverging stereoscopic image pairs is shown. Method 800 can be used in automated stereoscopic display systems (such as...) Figure 1 The method 800 is executed on a multi-view display system 100, applied to an autostereoscopic display system, or may be used by or with an autostereoscopic display system. The method 800 may be stored as instructions 134 on a non-transitory computer-readable storage medium 132. When executed by processing circuitry 130, instructions 134 may cause processing circuitry 130 to perform operation 136. Operation 136 may include operations detailed below with respect to method 800. Method 800 is only one method for reconverging stereoscopic image pairs; other suitable methods may be used. Figures 9 to 11 Additional details regarding operations 802 to 808 of method 800 are provided.

[0040] At operation 802, processing circuitry 130 can receive data specifying an initial stereoscopic image pair. Data can be received from memory, local storage, or remote storage. The data can include one or more frames as a video signal, which can specify the initial stereoscopic image pair as one of a series of sequential stereoscopic image pairs. Figure 9 The example front view of the left image 912L and right image 912R of the initial stereoscopic image pair 910 is shown. As an example, the initial stereoscopic image pair 910 shows two views of a scene including the "+" symbol. Other scenes can be used.

[0041] return Figure 8At operation 804, processing circuitry 130 can determine saliency information indicating the region of interest (ROI) of the initial stereo image pair 910 from the initial stereo image pair. The ROI may include the entire left image of the initial stereo image pair 910 and the entire right image of the initial stereo image pair 910. The ROI may include one or more regions considered important to the human visual system.

[0042] Processing circuit 130 can use one or more techniques to determine saliency information. For example, processing circuit 130 can use a fully convolutional neural network model. The neural network model can include an encoder based on a residual neural network (ResNet). The neural network model can include a decoder configuration with skip connections similar to those in U-Net. The architecture of U-Net can include encoder paths (also called shrinking paths) and decoder paths (also called expanding paths). The encoder can extract features from the input image. The decoder can project these features onto the pixel space to generate dense classifications. The encoder and decoder can be symmetric and connected by paths, which can give the model a "U" shape. The convolutional neural network model can include millions (or more) of trainable parameters, such as those that can be trained on multiple (e.g., thousands, millions) images with known saliency characteristics.

[0043] Figure 10 A front view is shown as an example indicating saliency information for the region of interest of the initial stereo image 910. The processing circuit 130 can analyze the specified image and generate saliency information from the image.

[0044] exist Figure 10 In the example, processing circuitry 130 has determined that the region of interest (ROI) in the initial stereo image pair 910 is a "+" sign. One way to represent saliency information is through a saliency map, which can have saliency values ​​in an array corresponding to the left and / or right images of the initial stereo image pair 910. For a pixel in the saliency map, a relatively large S value can indicate a relatively high saliency for the corresponding pixel in the image. For a pixel in the saliency map, a relatively small value of S (such as zero) can indicate a relatively low saliency for the corresponding pixel in the image. In some examples, the value of S can take a finite or infinite number of values. The saliency map can be binary, for example, having positive values ​​in the ROI and zero values ​​outside the ROI. Figure 10 In a specific example, the value of S can have one or more positive values ​​in the region of interest and a zero value outside the region of interest.

[0045] Processing circuit 130 can determine a left saliency map 1022L for the left image 912L of the initial stereo image pair 910, indicating the region of interest (ROI) of the left image 912L. The left saliency map 1022L can have a positive value S within the ROI 1024L of the left image 912L. Processing circuit 130 can also determine a right saliency map 1022R for the right image 912R of the initial stereo image pair 910, indicating the ROI of the right image 912R. The right saliency map 1022R can have a positive value S within the ROI 1024R of the right image 912R.

[0046] return Figure 8 At operation 806, processing circuit 130 can determine the convergence plane based on the parallax characteristics of the region of interest of the initial stereo image 910. Figure 11 A schematic diagram illustrating an example of determining the convergence plane based on the parallax characteristics of the region of interest of 910 from an initial stereo image is shown.

[0047] Operation 806 may include introducing a horizontal offset between the left image 912L and the right image 912R of the initial stereoscopic image pair 910 to form the left and right images of the final stereoscopic image pair. (In this document, the term "horizontal" is intended to correspond to the axis between the viewer's left and right eyes.) Processing circuitry 130 may select the amount of horizontal offset to reduce or eliminate the difference between the left and right images of the final stereoscopic image pair in the region of interest. Specifically, processing circuitry 130 may select a horizontal offset with an offset value such that the difference between the left and right images of the final stereoscopic image pair in the region of interest is less than the difference between the left image 912L and the right image 912R of the initial stereoscopic image pair 910 in the region of interest. Processing circuitry 130 may effectively ignore portions of the left and right images outside the region of interest. For example, when determining how much horizontal offset to introduce, processing circuitry 130 may use only image information within the region of interest.

[0048] exist Figure 11 In the example, the right image 912R is offset by a variable offset from the left image 912L. Alternatively, the left image 912L can be offset by a variable offset from the right image 912R. As a further alternative, the left image 912L and the right image 912R can be offset by a variable offset from each other.

[0049] To determine how much horizontal offset to introduce, the processing circuitry 130 can specify multiple horizontal offset values. These multiple horizontal offset values ​​can include consecutive values ​​separated by intervals corresponding to the pixel spacing of the initial stereo image pair. For example, the horizontal offset values ​​can include consecutive values ​​separated by 1 pixel, 2 pixels, or integer multiples of the pixel size. Other horizontal offsets can be used.

[0050] Then, the processing circuit 130 can generate multiple temporary stereo image pairs based on the initial stereo image pair and the corresponding multiple horizontal offset values. Each temporary stereo image pair can be formed by the left image 912L and the right image 912R of the initial stereo image pair 910, with the left image 912L and the right image 912R horizontally offset relative to each other, and the offset amount corresponding to their respective horizontal offset values.

[0051] Then, the processing circuit 130 can determine multiple differences. Each difference can represent the difference between the left and right images of a corresponding temporary stereo image pair in multiple temporary stereo image pairs in the region of interest.

[0052] Specific examples of the difference can include L1 distance values. To calculate the L1 distance value between the left and right images of a temporary stereo image pair, processing circuit 130 can perform the following calculations for each pixel of the left and right images. Processing circuit 130 can calculate a first quantity as the product of the left image intensity value (such as a value that increases with increasing brightness) and the left image saliency value (S, such as a non-zero value in the region of interest and zero outside the region of interest). Processing circuit 130 can calculate a second quantity as the product of the right image intensity value (such as a value that increases with increasing brightness) and the right image saliency value (S, such as a non-zero value in the region of interest and zero outside the region of interest). Processing circuit 130 can calculate a third quantity as the absolute value of the difference between the first and second quantities. To generate an image width difference, processing circuit 130 can sum the corresponding third quantity for all pixels in the left and right images of the temporary stereo image pair. The image range difference represents the difference between the left and right images, considering only the regions of interest of the left and right images (or weighting the individual regions in the image according to the values ​​in the corresponding saliency maps).

[0053] Processing circuit 130 can repeat the above calculation for different horizontal offset values ​​to generate a result as the horizontal offset (along...). Figure 11 The graph 1120 shows the difference of a function along the horizontal axis of the graph (along the vertical axis of the graph 1120). The processing circuit 130 can select a convergence plane from the data in the graph 1120. Below are two examples of techniques used for selecting a convergence plane.

[0054] In the first example, the processing circuit 130 can select the smallest difference 1132 among a plurality of differences as the selected difference. Then, the processing circuit 130 can select the temporary stereo image pair corresponding to the selected difference from a plurality of temporary stereo image pairs as the final stereo image pair.

[0055] In the second example, processing circuit 130 can fit a difference curve by interpolating multiple differences. Processing circuit 130 can determine a minimum value 1130 of the difference curve. Processing circuit 130 can set the convergence plane at the position corresponding to the minimum value of the difference curve. In many cases, the minimum value of the difference curve can lie between two adjacent differences among multiple differences, and the final stereo image pair can lie between two adjacent temporary stereo image pairs among multiple temporary stereo image pairs. These two examples are merely examples for selecting a convergence plane. Other techniques for selecting a convergence plane can be used.

[0056] return Figure 8 At operation 808, the processing circuit 130 can use the initial stereo image pair and the convergence plane to generate the final stereo image pair. Figure 12 The image shows an example front view of the left image 1242L and right image 1242R of the final stereoscopic image pair 1240. As an example, the final stereoscopic image pair 1240 shows two views of a scene including a "+" symbol as the region of interest. Other scenes can be used.

[0057] An autostereoscopic display, such as multi-view display 110, can display a final stereoscopic image pair 1240 such that when the final stereoscopic image pair 1240 is displayed on the autostereoscopic display: the convergence plane coincides with the plane of the autostereoscopic display (and the region of interest), the first portion of the final stereoscopic image pair appears to be above the plane of the autostereoscopic display, and the second portion of the final stereoscopic image pair appears to be below the plane of the autostereoscopic display. A viewer tracker, such as viewer tracker 120, can determine the position of a viewer, such as viewer 42.

[0058] An autostereoscopic display may include a display panel, such as display panel 112. The display panel may have a subpixel array, such as subpixel array 114, capable of displaying a final stereoscopic image pair 1240. The autostereoscopic display may include parallax generating optics, such as parallax generating optics 118, which can direct light from the display panel to the viewer. The parallax generating optics can include one of a lenticular lens (such as lenticular lens 402) or a parallax barrier with a transmission slit (such as parallax barrier 602 with a transmission slit 704). The parallax generating optics can be positioned along an optical axis (such as an optical axis with an angle of inclination relative to the display panel)... OAThe tilt angle remains unchanged. It can be within a specified angular tolerance of forty-five degrees, for example, within one, two, five, ten degrees, or another suitable angular tolerance. The parallax generating optics can be periodic along an axis orthogonal to the optical axis. The processing circuit 130 can arrange the left image 1242L and right image 1242R of the final stereoscopic image pair 1240 on a subpixel array, such that the parallax generating optics guides the left image 1242L and right image 1242R to the viewer's corresponding eye at the viewer's position.

[0059] To further illustrate the systems and methods disclosed herein, a non-limiting list of examples is provided below. Each of the following non-limiting examples may exist independently or may be combined with any one or more of the other examples in any permutation or combination.

[0060] In Example 1, an automatic stereoscopic display system may include: processing circuitry configured to perform operations including: receiving data of a specified initial stereoscopic image pair; determining saliency information from the initial stereoscopic image pair indicating a region of interest (ROI) of the initial stereoscopic image pair; determining a convergence plane based on the parallax characteristics of the ROI of the initial stereoscopic image pair; and generating a final stereoscopic image pair using the initial stereoscopic image pair and the convergence plane.

[0061] In Example 2, the autostereoscopic display system of Example 1 may optionally be configured such that the operation further includes: introducing a horizontal offset between the left and right images of the initial stereoscopic image pair to form the left and right images of the final stereoscopic image pair, the horizontal offset having a selected offset value such that the difference between the left and right images of the final stereoscopic image pair in the region of interest is less than the difference between the left and right images of the initial stereoscopic image pair in the region of interest.

[0062] In Example 3, the autostereoscopic display system of any of Examples 1-2 may optionally be configured such that the operation further includes: specifying a plurality of horizontal offset values; generating a plurality of temporary stereoscopic image pairs based on an initial stereoscopic image pair and the corresponding plurality of horizontal offset values, each temporary stereoscopic image pair being formed by a left image and a right image of the initial stereoscopic image pair, the left image and the right image being horizontally offset relative to each other by an amount corresponding to the corresponding horizontal offset values; and determining a plurality of differences, each difference representing the difference between the left image and the right image of a corresponding temporary stereoscopic image pair in the plurality of temporary stereoscopic image pairs in the region of interest.

[0063] In Example 4, the automatic stereoscopic display system of any of Examples 1-3 may optionally be configured such that the operation further includes: selecting the smallest difference among a plurality of differences as the selected difference; and selecting the temporary stereoscopic image pair among a plurality of temporary stereoscopic image pairs that corresponds to the selected difference as the final stereoscopic image pair.

[0064] In Example 5, the autostereoscopic display system of any of Examples 1-4 may optionally be configured such that the operation further includes: fitting a difference curve by interpolation of a plurality of differences; determining the minimum value of the difference curve; and setting the convergence plane to be arranged at a position corresponding to the minimum value of the difference curve, wherein: the minimum value of the difference curve is between two adjacent differences among the plurality of differences; and the final stereoscopic image pair is located between two adjacent temporary stereoscopic image pairs among the plurality of temporary stereoscopic image pairs.

[0065] In Example 6, the autostereoscopic display system of any of Examples 1-5 can be optionally configured such that the plurality of horizontal offset values ​​include consecutive values ​​separated by spacing values ​​corresponding to the pixel spacing of the initial stereoscopic image pair.

[0066] In Example 7, the autostereoscopic display system of any of Examples 1-6 may optionally be configured such that the operation further includes: determining a left saliency map from the left image of the initial stereoscopic image pair that indicates the region of interest of the left image of the initial stereoscopic image pair; and determining a right saliency map from the right image of the initial stereoscopic image pair that indicates the region of interest of the right image of the initial stereoscopic image pair.

[0067] In Example 8, the autostereoscopic display system of any of Examples 1-7 may optionally be configured such that: the region of interest includes the entire left image smaller than the initial stereoscopic image pair and the entire right image smaller than the initial stereoscopic image pair; and the region of interest includes one or more regions considered important.

[0068] In Example 9, the autostereoscopic display system of any one of Examples 1-8 may optionally further include: an autostereoscopic display configured to display a final stereoscopic image pair such that when the final stereoscopic image pair is displayed on the autostereoscopic display: the convergence plane coincides with the plane of the autostereoscopic display; a first portion of the final stereoscopic image pair appears to be above the plane of the autostereoscopic display; and a second portion of the final stereoscopic image pair appears to be below the plane of the autostereoscopic display; and a viewer tracker configured to determine the position of a viewer.

[0069] In Example 10, the autostereoscopic display system of any one of Examples 1-9 may optionally be configured such that the autostereoscopic display includes: a display panel having a subpixel array configured to display a final stereoscopic image pair; and a parallax generating optics configured to guide light from the display panel to a viewer, the parallax generating optics including either a lenticular lens or a parallax barrier having a transmission slit, the parallax generating optics being constant along an optical axis having an angle of inclination relative to the display panel within a specified angular tolerance of forty-five degrees, and the parallax generating optics being periodic along an axis orthogonal to the optical axis; and wherein the operation further includes: arranging the left and right images of the final stereoscopic image pair on the subpixel array such that the parallax generating optics guides the left and right images to the viewer's corresponding eye at the viewer's position.

[0070] In Example 11, a method for reconverging a pair of stereo images may include: receiving data of a specified initial stereo image pair using a processing circuit; determining saliency information indicating the region of interest of the initial stereo image pair from the initial stereo image pair using the processing circuit; determining a convergence plane based on the disparity characteristics of the region of interest of the initial stereo image pair using the processing circuit; and generating a final stereo image pair using the initial stereo image pair and the convergence plane using the processing circuit.

[0071] In Example 12, the method of Example 11 may optionally further include: using processing circuitry to introduce a horizontal offset between the left and right images of the initial stereo image pair to form the left and right images of the final stereo image pair, the horizontal offset having a selected offset value such that the difference between the left and right images of the final stereo image pair in the region of interest is less than the difference between the left and right images of the initial stereo image pair in the region of interest.

[0072] In Example 13, the method of any one of Examples 11-12 may optionally further include: specifying a plurality of horizontal offset values ​​using processing circuitry; generating a plurality of temporary stereo image pairs using processing circuitry based on an initial stereo image pair and the corresponding plurality of horizontal offset values, each temporary stereo image pair being formed by a left image and a right image of the initial stereo image pair, the left and right images being horizontally offset relative to each other by an amount corresponding to the corresponding horizontal offset values; and determining a plurality of differences using processing circuitry, each difference representing a difference between the left and right images of a corresponding temporary stereo image pair in the plurality of temporary stereo image pairs in the region of interest.

[0073] In Example 14, the method of any one of Examples 11-13 may optionally further include: using processing circuitry to select the minimum difference among a plurality of differences as a selected difference; and using processing circuitry to select a temporary stereo image pair among a plurality of temporary stereo image pairs that corresponds to the selected difference as a final stereo image pair.

[0074] In Example 15, the method of any one of Examples 11-14 may optionally further include: fitting a difference curve using a processing circuit via interpolation of a plurality of differences; determining a minimum value of the difference curve using a processing circuit; and setting a convergence plane to be positioned at a location corresponding to the minimum value of the difference curve using a processing circuit, wherein: the minimum value of the difference curve is between two adjacent differences among the plurality of differences; and the final stereo image pair is located between two adjacent temporary stereo image pairs among the plurality of temporary stereo image pairs.

[0075] In Example 16, a non-transitory computer-readable storage medium may store instructions for reconverging stereo image pairs. When executed by processing circuitry, the instructions may cause the processing circuitry to perform operations. These operations may include: receiving data specifying an initial stereo image pair using the processing circuitry; determining saliency information indicating the region of interest (ROI) of the initial stereo image pair from the initial stereo image pair using the processing circuitry; determining a convergence plane based on the disparity characteristics of the ROI of the initial stereo image pair using the processing circuitry; and generating a final stereo image pair using the initial stereo image pair and the convergence plane using the processing circuitry.

[0076] In Example 17, the non-transitory computer-readable storage medium of Example 16 may optionally be configured such that the operation further includes: using processing circuitry to introduce a horizontal offset between the left and right images of the initial stereo image pair to form the left and right images of the final stereo image pair, the horizontal offset having a selected offset value such that the difference between the left and right images of the final stereo image pair in the region of interest is less than the difference between the left and right images of the initial stereo image pair in the region of interest.

[0077] In Example 18, the non-transitory computer-readable storage medium of any of Examples 16-17 may optionally be configured such that the operation further includes: specifying a plurality of horizontal offset values ​​using processing circuitry; generating a plurality of temporary stereo image pairs using processing circuitry based on an initial stereo image pair and the corresponding plurality of horizontal offset values, each temporary stereo image pair being formed by a left image and a right image of the initial stereo image pair, the left and right images being horizontally offset relative to each other by an amount corresponding to the corresponding horizontal offset values; and determining a plurality of differences using processing circuitry, each difference representing a difference between the left and right images of a corresponding temporary stereo image pair in the plurality of temporary stereo image pairs in the region of interest.

[0078] In Example 19, the non-transitory computer-readable storage medium of any of Examples 16-18 may optionally be configured such that the operation further includes: using processing circuitry to select the smallest difference among a plurality of differences as a selected difference; and using processing circuitry to select a temporary stereo image pair among a plurality of temporary stereo image pairs that corresponds to the selected difference as a final stereo image pair.

[0079] In Example 20, the non-transitory computer-readable storage medium of any of Examples 16-19 may optionally be configured such that the operation further includes: fitting a difference curve using processing circuitry via interpolation of a plurality of differences; determining a minimum value of the difference curve using processing circuitry; and setting a convergence plane using processing circuitry to be arranged at a position corresponding to the minimum value of the difference curve, wherein: the minimum value of the difference curve is between two adjacent differences among the plurality of differences; and the final stereo image pair is located between two adjacent temporary stereo image pairs among the plurality of temporary stereo image pairs.

[0080] The above detailed description includes reference to the accompanying drawings, which form part of the detailed description. The drawings illustrate specific embodiments in which the invention can be practiced. These embodiments are also referred to herein as “examples.” Such examples may include elements other than those shown or described. However, it is contemplated that examples are provided only with those elements shown or described. Furthermore, other examples may be any combination or substitution of those elements (or one or more aspects thereof) shown or described with respect to a particular example (or one or more aspects thereof) or with respect to other examples (or one or more aspects thereof) shown or described herein.

[0081] In this document, as is common in patent documents, the terms “a” or “an” are used to include one or more, independent of any other instances or uses of “at least one” or “one or more.” In this document, the term “or” is used to refer to a non-exclusive “or,” such that “A or B” can include “A but not B,” “B but not A,” and “A and B,” unless otherwise stated. In the appended claims, the terms “including” and “in which” are used as concise English equivalents to the corresponding terms “comprising” and “wherein.” Furthermore, in the following claims, the terms “comprising” and “including” are open-ended, meaning that a system, apparatus, article, or process that includes elements other than those listed after such terms in the claim is still considered to be within the scope of that claim. Additionally, in the following claims, the terms “first,” “second,” and “third,” etc., are used merely as designations and are not intended to impose numerical requirements on their objects.

[0082] The above description is intended to be illustrative and not restrictive. For example, the examples (or one or more aspects thereof) described above can be used in combination with each other. Other embodiments can be used by those skilled in the art after reading the above description. It should be understood at the time of filing that it is not intended to interpret or limit the scope or meaning of the claims. Moreover, in the above detailed description, various features can be combined together to simplify this disclosure. This should not be construed as meaning that any unclaimed disclosed feature is essential to any claim. Rather, the inventive subject matter may lie in all features of fewer than the embodiments of a particular disclosure. Therefore, the following claims are incorporated herein by reference, wherein each claim is an independent, separate embodiment, and these embodiments are contemplated to be combined with each other in various combinations or arrangements. The scope of the invention should be determined with reference to the appended claims and the full scope of their equivalents.

Claims

1. An automatic stereoscopic display system, comprising: The processing circuit is configured to perform operations, the operations including: Receive data for a specified initial stereo image pair; From the initial stereo image pair, determine saliency information indicating the region of interest of the initial stereo image pair; The convergence plane is determined based on the disparity characteristics of the region of interest in the initial stereo image pair; and The initial stereo image pair and the convergence plane are used to generate the final stereo image pair.

2. The automatic stereoscopic display system according to claim 1, wherein the operation further includes: A horizontal offset is introduced between the left and right images of the initial stereo image pair to form the left and right images of the final stereo image pair. The horizontal offset has a selected offset value such that the difference between the left and right images of the final stereo image pair in the region of interest is smaller than the difference between the left and right images of the initial stereo image pair in the region of interest.

3. The automatic stereoscopic display system according to claim 1, wherein the operation further includes: Specify multiple horizontal offset values; Multiple temporary stereo image pairs are generated based on an initial stereo image pair and corresponding multiple horizontal offset values. Each temporary stereo image pair is formed by the left and right images of the initial stereo image pair, and the left and right images are horizontally offset relative to each other by an amount corresponding to the corresponding horizontal offset values. and Multiple differences are determined, each difference representing the difference between the left and right images of a corresponding temporary stereo image pair in the plurality of temporary stereo image pairs in the region of interest.

4. The automatic stereoscopic display system according to claim 3, wherein the operation further includes: Select the smallest difference among the plurality of differences as the selected difference; and The temporary stereo image pair that corresponds to the selected difference among the plurality of temporary stereo image pairs is selected as the final stereo image pair.

5. The automatic stereoscopic display system according to claim 3, wherein the operation further includes: The difference curve is fitted by interpolating the plurality of differences; Determine the minimum value of the difference curve; and The convergence plane is positioned at the location corresponding to the minimum value of the difference curve, wherein: The minimum value of the difference curve is between two adjacent differences among the plurality of differences; and The final stereo image pair is located between two adjacent temporary stereo image pairs among the plurality of temporary stereo image pairs.

6. The automatic stereoscopic display system of claim 3, wherein the plurality of horizontal offset values ​​comprises consecutive values ​​separated by an interval value corresponding to the pixel spacing of the initial stereoscopic image pair.

7. The automatic stereoscopic display system according to claim 1, wherein the operation further includes: Determine a left saliency map from the left image of the initial stereo image pair, indicating the region of interest of the left image of the initial stereo image pair; and Determine a right saliency map from the right image of the initial stereo image pair, indicating the region of interest of the right image of the initial stereo image pair.

8. The automatic stereoscopic display system according to claim 1, wherein: The region of interest includes the entire left image smaller than the initial stereo image pair and the entire right image smaller than the initial stereo image pair; and The region of interest includes one or more regions that are considered important.

9. The automatic stereoscopic display system according to claim 1, further comprising: An autostereoscopic display is configured to display the final stereoscopic image pair such that when the final stereoscopic image pair is displayed on the autostereoscopic display: The convergence plane coincides with the plane of the automated stereoscopic display. The first portion of the final stereoscopic image pair appears to be positioned above the plane of the autostereoscopic display; and The second portion of the final stereoscopic image pair appears to be located below the plane of the autostereoscopic display; and A viewer tracker is configured to determine the viewer's location.

10. The automatic stereoscopic display system according to claim 9, wherein the automatic stereoscopic display comprises: The display panel has a sub-pixel array configured to display the final stereoscopic image pair; and Parallax generating optics are configured to direct light from the display panel toward the viewer. The parallax generating optics include either a cylindrical lens or a parallax barrier with a transmission slit. The parallax generating optics remain constant along the optical axis, which has an angle of inclination relative to the display panel. The tilt angle is within a specified angular tolerance of 45 degrees. The parallax generating optics are periodic along an axis orthogonal to the optical axis; and The operation further includes: The left and right images of the final stereoscopic image pair are arranged on a subpixel array such that the parallax generating optics guide the left and right images to the viewer's corresponding eyes at the viewer's position.

11. A method for reconverging a stereo image pair, the method comprising: The processing circuit receives data from a specified initial stereo image pair. The processing circuitry is used to determine saliency information indicating the region of interest of the initial stereo image pair from the initial stereo image pair. The processing circuit determines the convergence plane based on the disparity characteristics of the region of interest in the initial stereo image pair; and Using the processing circuit, a final stereo image pair is generated using the initial stereo image pair and the convergence plane.

12. The method of claim 11, further comprising: The processing circuit introduces a horizontal offset between the left and right images of the initial stereo image pair to form the left and right images of the final stereo image pair. The horizontal offset has a selected offset value such that the difference between the left and right images of the final stereo image pair in the region of interest is smaller than the difference between the left and right images of the initial stereo image pair in the region of interest.

13. The method of claim 11, further comprising: The processing circuitry is used to specify multiple horizontal offset values; The processing circuit generates multiple temporary stereo image pairs based on an initial stereo image pair and corresponding multiple horizontal offset values. Each temporary stereo image pair is formed by the left and right images of the initial stereo image pair, and the left and right images are horizontally offset relative to each other by an amount corresponding to the corresponding horizontal offset value. and The processing circuit is used to determine multiple differences, each difference representing the difference between the left and right images of a corresponding temporary stereo image pair in the plurality of temporary stereo image pairs in the region of interest.

14. The method of claim 13, further comprising: The processing circuit selects the smallest difference among the plurality of differences as the selected difference; and The processing circuit selects the temporary stereo image pair that corresponds to the selected difference from the plurality of temporary stereo image pairs as the final stereo image pair.

15. The method of claim 13, further comprising: The processing circuit is used to fit a difference curve by interpolating the plurality of differences; The minimum value of the difference curve is determined using the processing circuit. and The processing circuit is used to position the convergence plane at a location corresponding to the minimum value of the difference curve, wherein: The minimum value of the difference curve is between two adjacent differences among the plurality of differences; and The final stereo image pair is located between two adjacent temporary stereo image pairs among the plurality of temporary stereo image pairs.

16. A non-transitory computer-readable storage medium storing instructions for reconverging stereoscopic image pairs, the instructions causing the processing circuitry to perform operations when executed by the processing circuitry, the operations including: The processing circuit receives data for a specified initial stereoscopic image pair. The processing circuitry is used to determine saliency information indicating the region of interest of the initial stereo image pair from the initial stereo image pair. The processing circuit determines the convergence plane based on the disparity characteristics of the region of interest in the initial stereo image pair; and Using the processing circuit, a final stereo image pair is generated using the initial stereo image pair and the convergence plane.

17. The non-transitory computer-readable storage medium according to claim 16, wherein, The operation also includes: The processing circuit introduces a horizontal offset between the left and right images of the initial stereo image pair to form the left and right images of the final stereo image pair. The horizontal offset has a selected offset value such that the difference between the left and right images of the final stereo image pair in the region of interest is less than the difference between the left and right images of the initial stereo image pair in the region of interest.

18. The non-transitory computer-readable storage medium according to claim 16, wherein, The operation also includes: The processing circuitry is used to specify multiple horizontal offset values; The processing circuit generates multiple temporary stereo image pairs based on an initial stereo image pair and corresponding horizontal offset values. Each temporary stereo image pair is formed by the left and right images of the initial stereo image pair, wherein the left and right images are horizontally offset relative to each other by an amount corresponding to the corresponding horizontal offset values. The processing circuit is used to determine multiple differences, each difference representing the difference between the left and right images of a corresponding temporary stereo image pair in the plurality of temporary stereo image pairs in the region of interest.

19. The non-transitory computer-readable storage medium according to claim 18, wherein, The operation also includes: The processing circuit selects the minimum difference among the plurality of differences as the selected difference; and The processing circuit selects the temporary stereo image pair that corresponds to the selected difference from the plurality of temporary stereo image pairs as the final stereo image pair.

20. The non-transitory computer-readable storage medium according to claim 18, wherein, The operation also includes: The processing circuit is used to fit a difference curve by interpolating the plurality of differences; The minimum value of the difference curve is determined using the processing circuit; and The processing circuit is used to position the convergence plane at a location corresponding to the minimum value of the difference curve, wherein: The minimum value of the difference curve is between two adjacent differences among the plurality of differences; and The final stereo image pair is located between two adjacent temporary stereo image pairs among the plurality of temporary stereo image pairs.

Images