Multi-view display system and method employing a tilted convergence plane for multi-view images

Abstract

The system and method involve loading views of a multi-view image into memory. The views can be formatted as bitmaps defined by a pixel coordinate system. The distance between the view and the central viewpoint can be identified. Subsequently, the views can be rendered in the graphics pipeline as a sheared view based on a shear function applied along the axes of the pixel coordinate system. The shear strength of the shear function is related to the distance between the view and the central viewpoint.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 115,531, filed November 18, 2020, the entire contents of which are incorporated herein by reference.

[0003] Statement regarding federally sponsored research or development

[0004] none Background Technology

[0005] Depending on the viewing angle, scenes in three-dimensional (3D) space can be viewed from multiple perspectives. Furthermore, when viewed by a user with stereoscopic vision, multiple views representing different perspectives of the scene can be perceived simultaneously, effectively creating a sense of depth that can be perceived by the user. Multiview displays present images with multiple views to represent how a scene is perceived in a 3D world. Multiview displays render different views simultaneously to provide a realistic experience to the user. Multiview images can be dynamically generated and processed by software. They can then be rendered in real-time by the graphics pipeline. When rendering multiview images for display, the graphics pipeline can apply various operations to the multiview images. Attached Figure Description

[0006] The various features of the examples and embodiments based on the principles described herein can be more readily understood by referring to the accompanying drawings and the following detailed description, wherein the same reference numerals denote the same structural elements, and wherein:

[0007] Figure 1 A multi-view image is shown as an example of an embodiment consistent with the principles described herein.

[0008] Figure 2 An example of generating a multi-view image is shown, based on an embodiment consistent with the principles described herein.

[0009] Figure 3 An example of a convergence plane of a tilted multi-view image is shown, according to an embodiment consistent with the principles described herein.

[0010] Figure 4 An example of a computational system for a convergence plane of a tilted multi-view image, based on an embodiment consistent with the principles described herein, is shown.

[0011] Figure 5A and Figure 5B Examples of applying the shear function according to embodiments consistent with the principles described herein are shown.

[0012] Figure 6 An example of interfacing with a graphics pipeline is shown, based on an embodiment consistent with the principles described herein.

[0013] Figure 7A and Figure 7B An example of a user interface for configuring the rendering of a multi-view image is shown, based on an embodiment consistent with the principles described herein.

[0014] Figure 8 An example of a computing system is shown that applies a shear function as pixels are sampled, according to an embodiment consistent with the principles described herein.

[0015] Figure 9 A flowchart illustrating a system and method for a convergence plane of tilted multi-view images according to embodiments consistent with the principles described herein.

[0016] Figure 10 A schematic block diagram illustrating an example of a multi-view display system providing multi-view display according to an embodiment consistent with the principles described herein.

[0017] Some examples and embodiments have other features that complement or replace the features shown in the above figures. These and other features are described in detail below with reference to the above figures. Detailed Implementation

[0018] Examples and embodiments based on the principles described herein provide techniques for improving the user's experience of perceiving multi-view images by tilting a convergence plane. By default, a convergence plane is typically parallel to the camera lens at a certain distance from the camera. Objects intersecting the convergence plane appear in focus, resulting in no parallax between different views of these objects. However, when the viewpoint changes to extreme angles (such as a bird's-eye view as an example), objects of interest in a multi-view image may experience parallax in a way that negatively impacts the viewing experience. As described herein, embodiments involve tilting a convergence plane to improve the way objects are perceived based on the angle of the camera view. In the case of a bird's-eye view, the convergence plane can be tilted so that it is substantially parallel to the ground. As a result, objects closer to the ground can be perceived more clearly from the viewer's perspective.

[0019] The embodiments involve applying graphics-level operations to tilt a convergence plane in a real-time graphics pipeline. For example, as a multi-view image is rendered in real time, the convergence plane can be tilted as part of a post-processing operation. Shaders in the graphics pipeline can be configured to apply shearing functionality to different views to effectively tilt the convergence plane. For instance, the shader can sample pixels from views of a multi-view image in a shearing manner. The shearing amount can correspond to how far a particular view is from the central viewpoint. To this end, the shearing functionality shears the different views of the multi-view image in real time to effectively tilt the convergence plane, thereby creating a better viewing experience for the viewer.

[0020] Figure 1A multi-view image 103 is shown as an example of an embodiment consistent with the principles described herein. The multi-view image 103 has a plurality of views 106. Each of the views 106 corresponds to a different viewing orientation 109. The views 106 are rendered for display on a multi-view display 112. Figure 1 The multi-view image 103 shown depicts views of various buildings on the ground from a bird's-eye view. Each view 106 represents a different viewing angle of the multi-view image 103. Therefore, the different views 106 have a certain level of parallax relative to each other. In some embodiments, a viewer can perceive one view 106 with her right eye while perceiving different views 106 with her left eye. This allows the viewer to perceive different views simultaneously, thus creating stereoscopic vision. In other words, the different views 106 create a three-dimensional (3D) effect.

[0021] In some embodiments, as a viewer physically changes her viewing angle relative to the multiview display 112, the viewer's eye can capture different views 106 of the multiview image 103. As a result, the viewer can interact with the multiview display 112 to see the different views 106 of the multiview image 103. For example, as the viewer moves to the left, the viewer can see more of the left side of the building in the multiview image 103. The multiview image 103 may have multiple views 106 along a horizontal plane and / or multiple views 106 along a vertical plane. Therefore, as the user changes their viewing angle to see different views 106, the viewer can obtain additional visual details from the multiview image 103. When the multiview image 103 is processed for display, it is stored as data in a format that records the different views 106.

[0022] As discussed above, each view 106 is presented by the multi-view display 112 at a different corresponding view orientation 109. When the multi-view image 103 is presented for display, the view 106 may actually appear on or near the multi-view display 112. A 2D display may be substantially similar to the multi-view display 112, except that the 2D display is typically configured to provide a single view (e.g., only one view) opposite to the different views 106 of the multi-view image 103.

[0023] In this document, a "two-dimensional display" or "2D display" is defined as a display configured to provide substantially the same image view regardless of the viewing direction (i.e., within a predefined viewing angle or range of the 2D display). Conventional liquid crystal displays (LCDs) found in many smartphones and computer monitors are examples of 2D displays. Conversely, a "multi-view display" is defined as an electronic display or display system configured to simultaneously provide different views of a multi-view image from different viewing directions or from different viewing directions, from the user's perspective. Specifically, the different views 106 may represent different perspective views of the multi-view image 103.

[0024] The multi-view display 112 can be implemented using various techniques to adapt the presentation of different image views so that they can be perceived simultaneously. One example of a multi-view display is one that employs diffraction gratings to control the subject orientation of different views 106. According to some embodiments, the multi-view display 112 can be a light field display, which is a light field display that presents multiple beams of light corresponding to different colors and directions for different views. In some examples, the light field display is a so-called "glasses-free" 3D display that can use diffraction gratings to provide an automatic stereoscopic representation of multi-view images without requiring special eyewear to perceive depth. In some embodiments, the multi-view display 112 may require glasses or other eyewear to control which views 106 are perceived by each of the user's eyes.

[0025] In some embodiments, the multi-view display 112 is part of a multi-view display system that renders multi-view images and 2D images. In this regard, the multi-view display system may include multiple backlights operating in different modes. For example, the multi-view display system may be configured to provide wide-angle emitted light using a wide-angle backlight during 2D mode. Additionally, the multi-view display system may be configured to provide directional emitted light using a multi-view backlight with a multi-beam element array during multi-view mode, the directional emitted light comprising multiple directional beams provided by each multi-beam element of the multi-beam element array. The multi-view display system may be configured to time-multiplex the 2D mode and the multi-view mode using a mode controller to sequentially activate the wide-angle backlight during a first sequential time interval corresponding to the 2D mode and the multi-view backlight during a second sequential time interval corresponding to the multi-view mode. The direction of the directional beams in the directional beams may correspond to different view orientations of the multi-view image.

[0026] For example, in 2D mode, wide-angle backlighting can generate an image, enabling the multi-view display system to operate like a 2D display. By definition, "wide-angle" emitted light is light with a cone angle greater than the cone angle of the multi-view image or the view of the multi-view display. Specifically, in some embodiments, wide-angle emitted light may have a cone angle greater than about 20 degrees (e.g., >±20°). In other embodiments, the cone angle of wide-angle emitted light may be greater than about 30 degrees (e.g., >±30°), or greater than about 40 degrees (e.g., >±40°), or greater than about 50 degrees (e.g., >±50°). For example, the cone angle of wide-angle emitted light may be greater than about 60 degrees (e.g., >±60°).

[0027] Multi-view mode can use multi-view backlighting instead of wide-angle backlighting. The multi-view backlighting can have an array of multi-beam elements that scatter light into multiple directional beams with different main directions from each other. For example, if the multi-view display 112 operates in multi-view mode to display a multi-view image with four views, the multi-view backlighting can scatter light into four directional beams, each corresponding to a different view. The mode controller can sequentially switch between 2D mode and multi-view mode, such that the multi-view image is displayed using the multi-view backlighting at a first sequential time interval, and the 2D image is displayed using the wide-angle backlighting at a second sequential time interval.

[0028] In some embodiments, the multi-view display system is configured to guide light as guided light within a light guide. Herein, "light guide" is defined as a structure that guides light within a structure using total internal reflection, or "TIR". Specifically, a light guide may include a core that is substantially transparent at the operating wavelength of the light guide. In various examples, the term "light guide" generally refers to a dielectric optical waveguide that employs total internal reflection to guide light at the interface between the dielectric material of the light guide and the material or medium surrounding the light guide. By definition, total internal reflection is conditional upon the light guide having a refractive index greater than the refractive index of the surrounding medium adjacent to the surface of the light guide material. In some embodiments, in addition to or instead of the aforementioned refractive index difference, the light guide may include a coating to further facilitate total internal reflection. For example, the coating may be a reflective coating. A light guide may be any of several light guides, including, but not limited to, one or both of plate or sheet light guides and strip light guides. Light guides may be shaped like plates or sheets. Light guides may be illuminated by the edge of a light source (e.g., a light-emitting device).

[0029] In some embodiments, the multi-view display system is configured to use multi-beam elements in a multi-beam element array to scatter a portion of guided light as directional emission light, each multi-beam element in the array including one or more of a diffraction grating, a micro-refractive element, and a micro-reflective element. In some embodiments, the diffraction grating of the multi-beam element may include multiple individual sub-gratings. In some embodiments, the micro-reflective element is configured to reflectively couple or scatter a portion of the guided light as multiple directional beams. The micro-reflective element may have a reflective coating to control how the guided light is scattered. In some embodiments, the multi-beam element includes a micro-refractive element configured to couple or scatter a portion of the guided light as multiple directional beams by or using refraction (i.e., refractively scattering the guided light portion).

[0030] like Figure 1 As shown, the multi-view display 112 includes a screen for displaying multi-view images 103. For example, the screen may be the display of a telephone (e.g., a mobile phone, smartphone, etc.), a tablet computer, a laptop computer, a computer monitor for a desktop computer, a camera monitor, or essentially any other device's electronic display.

[0031] Figure 2 An example of generating a multi-view image 115 according to an embodiment consistent with the principles described herein is shown. Figure 2 The multi-view image 115 contains various objects, such as a tree 118 on the ground 120. The tree 118 and the ground 120 can be referred to as objects, and together they form at least a part of the scene. This can be described in a manner similar to... Figure 1 The discussion focuses on the methods of displaying and viewing multi-view images 115. To generate multi-view images 115, a camera 121 can be used to capture a scene. In some embodiments, camera 121 may include one or more physical cameras. For example, a physical camera includes a lens for capturing light and recording it as an image. Multiple physical cameras can be used to capture different views of the scene to create multi-view images 115. For example, each physical camera may be spaced apart at a defined distance to allow capturing different perspectives of objects in the scene. The distance between different physical cameras allows for the ability to capture scene depth, similar to how the distance between a viewer's eyes allows for 3D vision.

[0032] Camera 121 can also represent a virtual (e.g., simulated or hypothetical) camera relative to a physical camera. Scenes can be generated using computer graphics techniques that manipulate computer-generated information. In this example, camera 121 is implemented as a virtual camera with a viewpoint used to generate the scene using software tools for editing images. The virtual camera can be defined based on the viewing angle and the coordinates of the 3D model. The 3D model can define various objects captured by the virtual camera.

[0033] In some embodiments, one or more views of the multi-view image 115 may be generated by automated algorithms (e.g., computer vision, artificial intelligence, image batch processing, etc.). For example, after views of a scene are generated or captured using a physical or virtual camera, one or more additional views may be artificially generated by predicting, interpolating, or extrapolating from the original views. For example, various computer vision techniques may generate additional views based on one or more input views. This may involve employing a trained computer vision model that predicts, interpolates, and / or extrapolates different views from one or more input views.

[0034] When using camera 121 to generate or capture views of a scene, multi-view images may have a convergence plane 127. A "converging plane" or "convergence plane" is defined as a set of locations where different views are aligned such that there is almost no parallax between them. Convergence plane 127 may also be referred to as the zero parallax plane (ZDP). Convergence plane 127 appears in front of camera 121. Objects between camera 121 and convergence plane 127 appear closer to the viewer, while objects behind convergence plane 127 appear farther away. In this respect, the farther an object is from convergence plane 127, the greater the degree of parallax between the different views. Objects along convergence plane 127 appear in focus relative to the viewer. Therefore, when generating multi-view images 115, a creator who wishes to characterize certain objects as primary subjects may want convergence plane 127 to fall on the primary subject. Pixels rendered on ZDP can appear as if they are on the display, pixels rendered in front of ZDP can appear as if they are in front of the display, and pixels rendered behind ZDP can appear as if they are behind the display.

[0035] Camera 121 captures the scene falling within a frustrum 130 of camera 121. The frustrum 130 is shown as having upper and lower boundaries defining the viewing angle range of the scene. Typically, the default convergence plane 127 is parallel to the plane formed by the camera lens of camera 121, such that it forms a trapezoid relative to the frustrum 130. Figure 2 In the image, the convergence plane 127 intersects the bottom and back of tree 118 (relative to camera 121). As a result, the bottom of tree 118 appears to be in focus and, as it appears to be positioned on the display, will appear to be a feature point of interest to the viewer.

[0036] Figure 2A disparity map 133 for one of the views of the multi-view image 115 is also shown. A disparity map 133 can be generated for at least one of the views. In some cases, no disparity map is generated. In any case, the disparity map 133 is discussed to illustrate the concepts relevant to the embodiments discussed herein. The disparity map 133 associates each pixel (or potentially a cluster of pixels) with a corresponding disparity value. The disparity value quantifies disparity based on the distance relative to the corresponding pixel in different views. For example, a pixel having a large disparity value in the first view means that, relative to the corresponding pixel in the second view, there is a large difference in the position of the pixel and the corresponding pixel as viewed by the viewer from a particular viewing angle.

[0037] In this document, a "disparity map" is defined as information indicating the apparent pixel differences between at least two views of a multi-view image 115. In this respect, a disparity map can indicate the positional differences between two pixels in two views of a multi-view image. When the disparity is zero (e.g., equal to or close to zero), the pixels representing an object appear to be in the same position to the viewer. In other words, the object focused by the user has zero disparity across multiple views (e.g., left-eye and right-eye views). Areas with almost no disparity are considered to correspond to the convergence plane 127 (or ZDP). Objects appearing in front of or behind the focused object will have different degrees of disparity and thus extend beyond the convergence plane. For example, pixels representing objects between camera 121 and the convergence plane 127 may have positive disparity values, while pixels representing objects behind the convergence plane 127 may have negative disparity values. The larger the absolute value of the disparity, the further away it is from the convergence plane 127. Disparity is inversely proportional to depth.

[0038] Figure 2A disparity map 133 with three regions 135a-c is shown. Each region 135a-c contains pixels representing various disparity values. For example, the lower region 135a corresponds to pixels representing the ground 120 in front of the tree 118, the middle region 135b corresponds to pixels representing the bottom of the tree 118, and the upper region 135c corresponds to pixels representing the top of the tree 118. The disparity values ​​in the lower region 135a can be relatively large and positive because they represent pixels in front of the convergence plane 127. The disparity values ​​in the middle region 135b can be close to zero because they represent pixels on the convergence plane 127. The disparity values ​​in the upper region 135c can be relatively large and negative because they represent pixels behind the convergence plane 127. When rendered on a multi-view display, the multi-view image 115 is perceived by the user, resulting in a wide range of disparity relative to the ground 120. In other words, only a small portion of the ground 120 will appear to be in focus and on the display. The remaining portion of the ground plane 120 will appear in front of or behind the display. This result may be undesirable in some applications. For example, multi-view content characterized by a subject moving along the ground plane 120 or multiple subjects at different locations on the ground may not be presented to the viewer in an optimal way from a bird's-eye view. In such cases, it may be desirable to tilt the convergence plane.

[0039] Figure 3 An example is shown of tilting the convergence plane 127 of a multi-view image according to an embodiment consistent with the principles described herein. For example, the convergence plane 127, which is initially parallel to the plane formed by the lens (virtual or physical) of the camera 121, can be tilted to form a tilted convergence plane 138 that is not parallel to the plane formed by the lens (virtual or physical) of the camera 121. Figure 3 The tilt amount 141 is shown, which can be quantified as the angle between the convergence plane 127 (which may be referred to as the initial convergence plane) and the tilted convergence plane 138. The tilted convergence plane 138 can be generated as a result of the convergence plane 127 rotating about a point of rotation (shown at the intersection of the initial convergence plane 127 and the tilted convergence plane 138).

[0040] By applying a tilted convergence plane 138, multi-view images can produce a more aesthetically pleasing viewing experience. For example, the tilted convergence plane 138 can correspond to a plane formed by the ground 120. As a result, objects along the ground will have no parallax, thus drawing the viewer's attention to the ground as it crosses the multi-view image. For example, objects located on or near the ground will appear as if they are on the display, objects above the ground will appear in front of the display, and objects below the ground will appear behind the display.

[0041] In terms of mathematical relations, the convergence plane 127 can be tilted along the vertical (y) axis by modifying the disparity map according to the following equation (1):

[0042] D ′ (X,Y)=D(X,Y)+T*Y+C (1)

[0043] Where "D" refers to the disparity value, "D'" refers to the updated disparity value, and the disparity value is a function of the pixel's X and Y coordinates. "T" quantizes the amount of tilt 141, and "C" corresponds to the rotational position of the convergence plane 127 defined by the rotation axis 150. By applying the above equation to the disparity map 133, the disparity is modified along the vertical axis such that the change in disparity is greater the further away from the rotation axis 150.

[0044] In some embodiments, modifying the disparity map 133 to create a tilted convergence plane 138 may not be an option because the disparity map 133 may not be readily available. For example, in a real-time rendering environment, multi-view images may be rendered and post-processed in-run when there is no bandwidth or the ability to generate a disparity map. Therefore, manipulating the disparity map may not allow for real-time rendering in the graphics pipeline. The following figures illustrate the use of the graphics pipeline to tilt the convergence plane 127 in a real-time rendering environment.

[0045] When generating or rendering multi-view images, various visual properties or effects control how the images are displayed. These visual properties include, for example, parallax, depth of field (DoF), baseline, convergence plane, convergence offset, and transparency. These visual properties of multi-view images can be applied as post-processing operations during rendering.

[0046] As used in this article, "parallax" is defined as the difference between at least two views of a multiview image at corresponding locations. For example, in the context of stereoscopic vision, the left and right eyes see the same object, but at slightly different locations, due to the difference in viewing angles between the eyes. This difference can be quantified as parallax. Changes in parallax across multiview images convey a sense of depth.

[0047] As used in this article, “depth of field” is defined as the depth difference between two objects that are considered to be in focus. For example, a large depth of field in a multi-view image results in a small amount of parallax between relatively large depth ranges.

[0048] As used herein, the “baseline” or “camera baseline” is defined as the distance between two cameras capturing corresponding views of a multi-view image. For example, in the context of stereoscopic vision, the baseline is the distance between the left and right eyes. A larger baseline results in increased parallax and can enhance the 3D effect of multi-view images.

[0049] As used in this article, "convergence offset" refers to the distance between the camera and a point along the convergence plane. Modifying the convergence offset will change the position of the convergence plane so that the multi-view image can be refocused on new objects at different depths.

[0050] As used in this article, "transparency" refers to an object property that defines how much other objects behind an object can be seen. Objects can be rendered as layers that form the final view of a multi-view image. Increasing the transparency of a foreground layer will allow the background layer to be seen. Minimal transparency (e.g., no transparency) will prevent the background layer from being seen, while maximum transparency will make a particular layer invisible so that the background layer is fully revealed.

[0051] Furthermore, as used herein, the article “a” is intended to have its common meaning in the patent field, namely “one or more”. For example, “a processor” means one or more processors, and therefore, “the memory” herein means “one or more memory components”.

[0052] Figure 4 An example of a calculation system for the convergence plane of a tilted multi-view image, according to an embodiment consistent with the principles described herein, is shown. The operation of tilting the convergence plane can be performed in the graphics pipeline 200 without using a disparity map. Figure 4 An embodiment is shown that mates with graphics pipeline 200 to tilt a convergence plane. As used herein, a “graphics pipeline” is defined as a computer-implemented environment that renders models for display. A graphics pipeline may include one or more graphics processing units (GPUs), GPU cores, or other dedicated processing circuitry optimized for rendering image content to a screen. For example, a GPU may include a vector processor that executes a set of instructions to operate on an array of data in parallel. Graphics pipeline 200 may include a graphics card, a graphics driver, or other hardware and software for rendering graphics. Graphics pipeline 200 may be configured to render images on multi-view display 112. Graphics pipeline 200 may map pixels to corresponding locations on the display and control the display to emit light to render the image.

[0053] Figure 4The computing system shown may also include one or more central processing units (CPUs) 202. The CPU 202 may be a general-purpose processor that executes instructions, supports an operating system, and provides user-level applications 205. In some embodiments, the graphics pipeline 200 is a subsystem separate from the CPU 202. For example, the graphics pipeline 200 may include a dedicated processor (e.g., a GPU) separate from the CPU 202. In some embodiments, the graphics pipeline 200 is implemented purely as software by the CPU 202. For example, the CPU 202 may execute software modules that operate as the graphics pipeline 200 without dedicated graphics hardware. In some embodiments, portions of the graphics pipeline 200 are implemented in dedicated hardware, while other portions are implemented as software modules by the CPU 202.

[0054] Application 205 can be a user-level application that generates a user interface rendered by the graphics pipeline 200 for display on the multi-view display 112. For example, application 205 can be a navigation application that loads various maps depicting streets, buildings, and other geographic landmarks. The navigation application can provide a user interface for generating a 3D model of a geographic region. The navigation application can dynamically update the viewing angle of a virtual camera in the 3D model to generate a visual output of a portion of the 3D model based on the orientation of the virtual camera.

[0055] The computing system may also include memory 208. Memory 208 may include main memory (e.g., system memory), cache, or other fast memory for fast data processing. As discussed in further detail below, memory 208 may be volatile memory, but may also include non-volatile memory. Memory 208 may include memory for CPU 202 and memory for graphics pipeline 200, such that CPU 202 and graphics pipeline 200 share the same memory resources. In some embodiments, memory 208 includes a first memory dedicated to the CPU (e.g., CPU memory) and a second memory dedicated to graphics pipeline 200 (e.g., GPU memory, texture memory, etc.). In this embodiment, graphics pipeline 200 may load, copy, or otherwise move content from CPU memory to GPU memory.

[0056] As described above, application 205 can use computer graphics techniques for 3D modeling to generate a 3D model. A 3D model is a mathematical representation of various surfaces and textures of different objects and can include spatial relationships between objects. Application 205 can generate and update the 3D model based on user input. User input can involve navigating the 3D model by clicking or dragging the cursor, pressing direction buttons, converting the user's physical position to a virtual position within the 3D model, etc. The 3D model can be loaded into memory 208 and subsequently updated.

[0057] A 3D model can be converted into a multi-view image 211, thereby exposing a window into the 3D model. The window can be defined by a virtual camera having a set of coordinates, viewing angle, focal length, baseline, etc., within the 3D model. A sequence of multi-view images 211 can form a video displayed at a specific frame rate (e.g., 30 frames per second). Each multi-view image 211 can consist of multiple views 214a-d. Figure 4 The example shows a multi-view image 211 formatted as a four-view image with four views; however, any number of views can be used. Views 214a to d can be configured to provide horizontal parallax, vertical parallax, or both. For example, when horizontal parallax is present, views 214a-d appear to change as the viewer moves relative to the multi-view display 112 from left to right.

[0058] Application 205 can load views 214a-d of the multi-view image 211 into memory 208. For example, application 205 can be configured to convert a 3D model into a rendering scene for displaying the multi-view image 211 derived from the 3D model. One or more views 214a-d are generated by application 205 and placed in a specific block of memory 208. Views 214a-d can be formatted as bitmaps 217 defined by a pixel coordinate system. For example, views 214a-d can be represented as a two-dimensional bitmap array along the horizontal (X) axis and the vertical (Y) axis. Each pixel in bitmap 217 has a corresponding position on the display. For example, the top-left pixel of bitmap 217 controls the output of the top-left pixel on the display. In addition, each view 214a-d can have a corresponding view index number 220. View index number 220 can be an ordered view number of the views within the multi-view image 211. For example, in a four-view multi-view format, each of the four views can be numbered one, two, three, and four. View index number 220 indicates the position of the view relative to other views. For example, view... Figure 1 It can be the leftmost view, view Figure 2 It can be the left center view, view Figure 3 It can be a right-center view, and the view... Figure 4 This could be the rightmost view. In this case, the maximum parallax will be at the viewpoint. Figure 1 With vision Figure 4 between.

[0059] Once views 214a-d are generated and loaded into memory 208, application 205 can invoke rendering command 221 on graphics pipeline 200. Rendering command 221 instructs graphics pipeline 200 to begin rendering multiview image 211. Rendering command 221 can be a function call to the graphics driver to cause graphics pipeline 200 to render multiview image 211. Rendering command 221 can identify a specific multiview image 211 to be rendered. For example, rendering command 221 can identify the address block storing multiview image 211.

[0060] The graphics pipeline 200 may include one or more shaders 226 to render the multi-view image 211. Shader 226 may be a hardware device (e.g., a shader core), a software module, or a combination thereof. Shader 226 may be executed by the GPU of the graphics pipeline 200. Initial rendering of the multi-view image 211 may be performed by a module that performs various techniques (such as rasterization, as an example) to render a simple or coarse version of the multi-view image 211. The initial rendering operation may be a fast, efficient operation to convert scene geometry into pixels for display. Initial rendering may not include more advanced optical effects. In some embodiments, shader 226 may be used in the initial rendering.

[0061] After the initial rendering is performed, one or more advanced optical effects can be applied to the initially rendered multi-view image. One or more shaders 226 can be used to apply these optical effects. Shaders 226 are considered to perform post-processing effects by manipulating the initially rendered multi-view image 211. As used herein, "post-processing" is defined as an operation performed on an image that was initially rendered as part of the rendering process in the graphics pipeline 200. Different shaders 226 can be configured to perform post-processing. Some examples of post-processing include, but are not limited to, modifying color saturation, modifying hue, adjusting brightness, adjusting contrast, applying blur, performing volumetric lighting, applying depth effects, performing cel shadows, generating bokeh effects, and applying one or more filters. As used herein, a "shader" is defined as a graphics component in the graphics pipeline that applies specific graphics operations, including, for example, initial rendering or post-processing.

[0062] Application 205 can interface with graphics pipeline 200 using one or more application programming interfaces (APIs). An example of an API is OpenGL, which provides an interface that allows application 205 to call functions performed in graphics pipeline 200. For example, application 205 can use the API to call a specific shader 226 to perform post-processing on the initially rendered multi-view image 211.

[0063] The embodiment relates to implementing the functionality of tilting a convergence plane during real-time rendering in the graphics pipeline 200. Examples of functions and operations that can occur in a computing system are provided below. As described above, application 205 can generate views 214a-d of a multi-view image 211 and load them into memory 208. Application 205, executing on an operating system, can instruct the CPU to load views 214a-d into memory block 208.

[0064] Views 214a-d can be formatted as bitmaps 217 defined by a pixel coordinate system. Views 214a-d can be generated from a 3D model by identifying a specific viewpoint and viewing angle of the 3D model and converting it into bitmaps 217. This can be performed for each view of the multi-view image 211. Application 205 can then invoke rendering command 221 to initially render views 214a-d of the multi-view image 211. For example, application 205 can use an API to request the graphics pipeline 200 to perform initial rendering. In response, the graphics pipeline 200 can generate the initially rendered views 214a-d by performing, for example, rasterization. In a real-time graphics rendering environment, the graphics pipeline 200 can be optimized to render views 214a-d quickly in runtime. This provides a seamless experience for the viewer as new multi-view images 211 are dynamically generated (and not pre-rendered).

[0065] Next, application 205 is configured to tilt the convergence plane in real time. For example, application 205 can identify the distances between views 214a-d and the central viewpoint. Assuming that different views 214a-d have different degrees of horizontal parallax relative to the central viewpoint, the distance along the horizontal axis between each view and the central viewpoint can be determined. This distance depends on the baseline (e.g., the distance between views). For example, the larger the baseline, the greater the distance from the central viewpoint. In some embodiments, the distance between views 214a-d and the central viewpoint is identified by determining the ordered view number (e.g., view index number 220) of views 214a-d within the multi-view image 211. For example, if the views 214a-d in the multi-view image 211 are ordered from one to four, where the views... Figure 1 Located on the far left and viewed Figure 4Located on the far right. View index 220 corresponds to the distance between views 214a-d and the center viewpoint. For example, view index 220 of 1 could correspond to a distance of 50 pixels to the left of the center, view index 220 of 2 could correspond to a distance of 25 pixels to the left of the center, view index 220 of 3 could correspond to a distance of 25 pixels to the right of the center, and view index 220 of 4 could correspond to a distance of 50 pixels to the right of the center. The distance from the center can be a signed number (e.g., positive or negative) to indicate whether the view is to the left of the center. For example, a negative distance could indicate that the view is to the left of the center, while a positive distance could indicate that the view is to the right of the center.

[0066] Determining the distance of views 214a-d from the central viewpoint is part of determining how to tilt the convergence plane in the real-time graphics pipeline 200. Application 205 can generate rendering instructions to tilt the convergence plane using a shear function. Application 205 can send instructions to the graphics pipeline 200 to render the views as sheared views. In this regard, application 205 can invoke instructions to post-process the initially rendered multi-view image to be sheared according to the shear function applied along the axes of the pixel coordinate system. Specifically, the graphics pipeline 200 can render views 214a-d in the graphics pipeline 200 as sheared views according to the shear function. The shear intensity of the shear function is related to the distance between views 214a-d and the central viewpoint. As used herein, a "shear function" is defined as a graphics operation that shifts the pixels of an image along a direction according to the shear intensity. The shear intensity quantifies the amount of shearing effect applied to the image by the shear function. The shear intensity of the shear function can be related to the position of the view relative to other views in the multi-view image. Figure 5, discussed below, provides a visual explanation of the shear function.

[0067] As the multiview image 211 is rendered in the graphics pipeline, a shear function is performed to tilt the convergence plane in real time. Shader 226 can be customized to implement the shear function. In this regard, application 205 can invoke shader 226 to perform the shear function on the initially rendered multiview image 211. After applying the shear function to views 214a-d of the multiview image, the graphics pipeline 200 can load the result into memory 208 as the sheared multiview image 232. The graphics pipeline 200 can either overlay the multiview image 211 with the sheared multiview image 232, or load the sheared multiview image 232 into a separate portion of memory 208. Additional post-processing can be applied to the sheared multiview image 232 before it is finally rendered on the multiview display 112.

[0068] Figure 5A and Figure 5BExamples of applying the shear function according to embodiments consistent with the principles described herein are shown. Figure 5A It depicts different views of a multi-view image. Each view can be formatted for storage or loaded into memory (such as, for example...). Figure 4 The bitmap image is stored in memory 208. Although four views are shown, a multi-view image can have any number of views. Figure 5A The view shown (e.g., view) Figure 1 ,See Figure 2 ,See Figure 3 Heshi Figure 4 It has horizontal parallax. Figure 5A A multi-view image can have a central viewpoint 235. Viewers can see around objects in the multi-view image by moving along the horizontal axis (e.g., from left to right or from right to left).

[0069] Each view can have a corresponding distance to the central viewpoint 235. Although Figure 5A This shows the distance measured from the center of each view, but the distance can be measured from any point in the view (such as the left or right edge in this example). Figure 1 The distance from the central viewpoint 235 is defined as "D1". Figure 2 The distance from the central viewpoint 235 is "D2". Figure 3 The distance from the central viewpoint 235 is "D3". Figure 4 The distance away from the central viewpoint 235 is "D4". Distances D1 and D2 can be negative values ​​indicating that they are to the left of the central viewpoint 235, while distances D3 and D4 can be positive values ​​indicating that they are to the right of the central viewpoint 235.

[0070] Figure 5A This shows how a multiview image appears when it is loaded into memory and how it is initially rendered by the graphics pipeline before post-processing. Figure 5B This is shown as a real-time graphics pipeline (e.g., Figure 4 The post-processing part of the graphics pipeline 200) transforms the multi-view image via a shear function. Specifically, Figure 5B Showing from view Figure 1 The generated sheared view Figure 1 From the perspective Figure 2 The generated sheared view Figure 2 From the perspective Figure 3 The generated sheared view Figure 3 And from the perspective Figure 4 The generated sheared view Figure 4 The view after shearing Figure 1-4Each of these values ​​is generated by a shear function that applies shear intensity. The shear intensity is determined based on the distance (e.g., D1-D4) between the view and the central viewpoint 235. For example, the greater the distance from the central viewpoint 235, the greater the shear intensity. Furthermore, the sign of the shear intensity (e.g., positive or negative) is defined by the sign of the distance. The sign of the shear intensity controls the direction of the shear applied by the shear function.

[0071] The shear function can also be defined by shear line 238. Shear line 238 can extend along a specific axis that controls how each view is sheared. The shear function operates according to shear line 238. Figure 5B The example shows a shear line 238 along the horizontal axis. As a result, the shear function is configured to skew the view only along the horizontal axis of the pixel coordinate system. In this respect, pixels in the pixel coordinate system are only horizontally shifted. The direction and extent of the pixel displacement depend on the distance of the view from the central viewpoint 235 (e.g., positive or negative distance) and whether the shifted pixel is above or below the shear line 238. For example, in the view after shearing... Figure 1 and the view after shear Figure 2 In the image, pixels above shear line 238 are skewed to the right, while pixels below shear line 238 are skewed to the left. This is reflected in the viewfinder after the shearing. Figure 3 and the view after shear Figure 4 In the middle, the pixels above shear line 238 are skewed to the left, while the pixels below shear line 238 are skewed to the right.

[0072] Figure 5B It also shows the view after the corresponding shear. Figure 1-4 The shear effect 241a-d. A stronger shear effect will cause the view to be sheared more. The shear effect is determined by the shear intensity, such that a larger shear intensity will result in a larger shear effect 241a-d. For example, the shear intensity can be based on the tilt of the convergence plane. Furthermore, the shear intensity increases as the view moves away from the central viewpoint 235. For example, the sheared view... Figure 1 The shear effect 241a is compared to the visual effect after shearing. Figure 2 The shear effect of 241b is stronger because the sheared view Figure 1 Further away from the central viewpoint 235. Additionally, the viewpoint after shearing... Figure 1 The shear effect 241a is similar to the sheared view. Figure 4 The shear effect is 241d, because the sheared view Figure 1 and the view after shear Figure 4 Equidistant from the central viewpoint 235. However, the sheared viewpoint... Figure 1 and the view after shear Figure 4 The shear effects 241a and 241d are opposite in direction because they are on opposite sides of the central viewpoint 235.

[0073] Shear line 238 can be formed by default as a horizontal line positioned in the middle along the vertical axis. In other embodiments, shear line 238 can be located at different vertical positions and can be user-specified. Although Figure 5B A horizontal shear line 238 is shown, but the shear line 238 can be vertical. In this embodiment, the shear function is configured to skew the first and second views along the vertical axis of the pixel coordinate system. In some embodiments, the shear line 238 is sloped or curved, such that it has points that change along both the horizontal and vertical axes. In this example, pixels can shift along both the X and Y directions.

[0074] One embodiment envisions using a navigation application to dynamically generate multi-view images of a map scene as the user navigates physical or virtual space. If the camera angle is similar to or close to a bird's-eye view, the convergence plane can be tilted around the horizontal axis. As a result, the shear function is configured to skew the view only along the horizontal axis of the pixel coordinate system.

[0075] Figure 6 Examples of interfacing with the graphics pipeline according to embodiments consistent with the principles described herein are shown. As discussed above, application 205 may interface with graphics pipeline 200. For example, application 205 may be a user-level application running on the operating system of a client device. Application 205 may also be implemented as a cloud-based application running on a server and provided to users via client devices. Application 205 may interface with graphics pipeline 200 using one or more APIs. Application 205 is responsible for calculating the view. As the user provides input, the view can be dynamically calculated from the 3D model. For example, the user may provide instructions or input to change the viewpoint, zoom, orientation, or position of the camera capturing the scene defined by the 3D model. In response, application 205 may calculate a view of multi-view images in real time. In this case, the multi-view images may be frames of video to be rendered in the real-time graphics pipeline.

[0076] The views of a multi-view image can be dynamically calculated in response to user interaction. Application 205 can generate commands to the graphics pipeline 200 to perform real-time rendering of any or all views as any or all views are calculated by application 205. For example, application 205 can send API function calls to the graphics pipeline 200 to render views.

[0077] Real-time rendering may include an initial rendering portion and a post-processing portion. The initial rendering portion involves the graphics pipeline 200 to render an initial view. As described above, the initial rendering of the view is to quickly render the pixels of the multi-view image on the display without advanced optical effects. Shaders may be used to perform the initial rendering. Subsequently, application 205 may invoke one or more post-processing operations to transform the initial rendering into a final rendering. Post-processing may apply image editing operations to improve the quality or realism of the initially rendered image. According to an embodiment, application 205 instructs the graphics pipeline 200 to tilt the convergence plane. For example, the graphics pipeline 200 applies a shear function to each view. The shear intensity of the shear function is related to the position of the view relative to other views in the multi-view image. Shaders may be used to implement the shear function. Application 205 may provide the shear intensity, shear line, or both as input to the graphics pipeline. The sheared views of the multi-view image are then rendered on the multi-view display 112. This process occurs continuously as application 205 generates new multi-view images to be rendered in real time.

[0078] Figure 7A and Figure 7B An example of a user interface 244 for configuring the rendering of multi-view images is shown, based on an embodiment consistent with the principles described herein. There are generally two software development and usage modes: configuration mode and runtime mode. Configuration mode refers to the mode in which developers create and configure software applications. For example, an application might allow developers to create a navigation application during configuration mode. Developers can specify desired camera angles, post-processing operations, and other aspects of how multi-view images will be rendered. Runtime mode refers to the mode in which end users execute software that has already been configured by the developers.

[0079] User interface 244 can be rendered on a client device and used by a developer during configuration mode, who is developing an application that will ultimately render multi-view images during runtime mode. The user interface may include windows containing information (e.g., text and graphics) presented to the user. User interface 244 can be generated by an application used to design an end-user application. For example, a developer can use user interface 244 to design a navigation application, a game application, or another application. User interface 244 can be used by a developer designing graphics and how those graphics are presented to other users. User interface 244 can also be rendered by an end-user application. User interface 244 can allow a user to configure shaders during configuration mode by making user selections related to different post-processing operations. Once the shaders are configured based on user input, they can post-process the multi-view images at runtime.

[0080] User interface 244 may have a first portion 247 for displaying a multi-view image or a representation thereof. The first portion 247 may include rendering of the multi-view image. For example, the rendering of the multi-view image may simulate how user settings will be applied to the multi-view image during runtime. User interface 244 may have a second portion 250 containing a menu. The menu may include various input elements, such as sliders, text boxes, checkboxes, radio buttons, drop-down menus, etc., as examples. The menu allows the user to change various visual parameters of the multi-view image while it is being rendered in the first portion. These visual parameters include, for example, camera baseline, convergence offset, ZDP rotation, auto ZDP option, depth of field (DoF) threshold, DoF intensity, transparency threshold, transparency intensity, and potentially other visual parameters. The user can provide input by manipulating one or more input elements. As a result, user input is received from user interface 244.

[0081] Figure 7A An example is depicted where user input is received from the user interface and the shear intensity is determined based on that input. For instance, a user can slide a ZDP rotation slider to select the range of ZDP rotation. When set to the minimum value (e.g., zero rotation), the convergent plane does not rotate when the slider is at one end. When set to the maximum value by moving the slider to the other end, the convergent plane tilts accordingly. In other words, the amount of ZDP rotation specified by the user is used to determine the amount of tilt. This quantifies the shear intensity when the shear function is applied during runtime.

[0082] The shear function can also calculate the shear intensity based on a baseline, which can be user-specified. The baseline controls the distance between each view and the central viewpoint by increasing the distance between at least two views. Therefore, increasing the baseline can move a view further away from the central viewpoint, thus causing that view to experience a stronger shear effect. For this purpose, outer views will be sheared to a greater extent to achieve the effect of tilting the converging plane.

[0083] Figure 7BAn example user interface is depicted, allowing the user to select an option to automatically determine the tilt amount of the convergence plane during configuration mode. For example, an application can automatically determine the shear intensity by calculating the disparity value at a common point between one view and another in a multi-view image. For example, an application can identify features at predetermined locations. Features can be pixels, or sets of pixels with a common color. The predetermined location can be the midpoint along the horizontal or vertical axis of the view. The application can identify the position of the corresponding feature in another view to determine the amount by which it has shifted due to the different perspective of the other view. The application can invoke a ray casting operation to identify features at predetermined locations. Ray casting refers to projecting hypothetical rays from a specific angle toward a specific location on a 3D model. The output identifies the features of the 3D model. Ray casting can be used to determine the disparity between different views of a 3D model. The amount by which a feature shifts between two views is equal to the disparity. Once the disparity between two views is identified at a specific location, an optimal shear intensity can be determined such that the disparity at that specific location is removed due to tilting the convergence plane. In this respect, tilting the convergence plane using the calculated shear intensity will cause the views to align at the predetermined location, such that there is no disparity.

[0084] Figure 7A and Figure 7B A user interface for selectively applying post-processing operations is also shown. In some embodiments, post-processing operations can be applied to selected regions of a multi-view image. For example, the application can be configured to receive user input from the user interface, determine a range of disparity values ​​based on the user input, and configure shaders to operate on pixels of the view in response to pixels having disparity values ​​within the range of disparity values. The user interface 244 may include a menu for selecting a threshold, such as, for example, a DoF threshold, a transparency threshold, or a threshold for other post-processing operations. The threshold can be a range of values ​​corresponding to the disparity range. A low threshold can cover view regions corresponding to low amounts of disparity (e.g., zero or near-zero disparity). A higher threshold can cover view regions corresponding to large amounts of disparity, such that the entire view is selected. Threshold selection expands the selection of the view in both directions (in and out) of the zero disparity plane. Therefore, based on the threshold selection, the application can determine a range of disparity values ​​and select those regions in the view that fall within the range of disparity values.

[0085] After selecting a region of the view, the application applies shader operations (e.g., post-processing operations) only to the selected region. The shader is configured to perform opacity or depth-of-field operations, or other possible post-processing operations. Opacity operations change the degree to which other objects behind an object are visible. This degree can be specified by the user through the user interface. For example, as... Figure 7A and 7BAs shown, users can specify the transparency intensity to control the transparency of pixels within a transparency threshold. The shader that performs the transparency operation is configured based on the transparency intensity and operates on selected pixels defined by the transparency threshold.

[0086] Depth-of-field operations modify the depth difference between two objects considered to be in focus. For example, depth-of-field operations can change the disparity value of pixels within a selected set of pixels. For instance, if a depth-of-field threshold selects pixels with disparity values ​​between -30 and +30, a large depth-of-field intensity can specify the degree of blur applied to the selected pixels. Depth-of-field operations blur the selected pixels in a manner corresponding to the depth-of-field intensity.

[0087] User interface 244 allows users to make specific selections regarding thresholds and post-processing operation parameters. These settings are used to configure the shader. During runtime, the shader operates based on the settings applied via user interface 244.

[0088] Figure 8 An example of a computing system is shown that applies a shear function as pixels are sampled, according to an embodiment consistent with the principles described herein. Figure 8 A computing system comprising at least a processor and a memory 303 is depicted, wherein the memory 303 stores a plurality of instructions, which, when executed, cause the processor to perform various operations. The memory 303 may be similar to... Figure 4 The memory is 208. Regarding... Figure 10 An example of this computing architecture, showing the processor and memory, is described in more detail.

[0089] The computing system can load views of a multi-view image into memory 303. For example, as mentioned above... Figure 4 The application discussed (e.g., application 205) can generate one or more multi-view images 309 and load different views 312 into memory 303 in real time. Views 312 can be formatted as bitmaps defined by a pixel coordinate system. For example... Figure 8 As shown, a bitmap can have a horizontal (X) axis and a vertical (Y) axis. For illustrative purposes, each pixel can be labeled with a column letter (A to G) and a row number (1 to 7). The top-left pixel is referred to as pixel A1 of view 312. It should be understood that the number of pixels in each view 312 can be significantly greater than [the number of pixels in the view]. Figure 8 The number of pixels displayed.

[0090] Next, the computing system can send an instruction (rendering instruction 317) to the graphics pipeline 315 to render the view as a sheared view 320 according to a shear function applied along axis 323 of the pixel coordinate system. The graphics pipeline 315 can be similar to... Figure 4The graphics pipeline 200. Rendering instruction 317 can be an API function call to render a view as a sliced ​​view 320 by invoking a shader configured to apply the slice function. The sliced ​​view 320 is part of a sliced ​​multiview image 326, such that each view 312 has a corresponding sliced ​​view 320. Rendering instruction 317 can identify the view to be sliced. Rendering instruction 317 can be an instruction sent in real-time to the graphics pipeline 315 to render the multiview image 211 as it is dynamically generated by the application. The slice function can be implemented by a shader, such as, for example... Figure 4 Shader 226. Shaders can be used by the user during configuration mode via a user interface (such as, for example...) Figure 7A and 7B Configure the user interface (244).

[0091] The shear strength of the shear function is related to the position of view 312 relative to other views in the multi-view image 309. For example, a view index number can identify the position of view 312 relative to other views. As explained above, in some embodiments, the shear strength can be determined by a user during configuration mode, with the user providing user input via a user interface. The shear strength is determined by user input and applied during runtime.

[0092] Graphics pipeline 315 is configured to implement a shearing function as pixels of a bitmap are sampled by graphics pipeline 315. For example, the shearing function may involve forming a sheared view by sampling pixels from view 312. Instead of sampling uniformly in a one-to-one correspondence, the shearing function samples pixels along a shear line using a shear intensity to cause the shearing effect. For example, the shearing function operates according to axis 323 that forms the shear line. Pixels of the sheared view 320 are sampled from positions close to corresponding positions in view 312. As pixels of the sheared view 320 move further away from axis 323 (in the vertical direction), the horizontal displacement increases relative to the position where the pixel was sampled.

[0093] To illustrate, pixel D3 of the sheared view 320 is close to the axis, which is close to rows 3 and 4. This pixel of the sheared view 320 is sampled from pixel D3 of view 312. Because pixel sampling is performed at the same corresponding position, this is equivalent to no shearing effect. However, as the pixel is positioned further upward in the vertical direction, the shearing effect becomes more pronounced. Pixel D1 of the sheared view is sampled from pixel C1 of view 312. In this respect, pixels north of axis 323 are skewed to the right. This is the sampling offset that causes the shearing effect to be applied to the sheared view. Similarly, pixel D7 of the sheared view 320 is sampled from pixel E7 of view 312. Pixels south of axis 323 are skewed to the left. This skew function can cause pixels to be sampled at invalid locations when operating near specific edges of view 312. For example, pixel G7 of the sheared view 320 is sampled from a position outside the view (shown as X). In this scenario, a default pixel can be used to generate the G7 pixel of the sheared view 320. The default pixel can be a pixel with a zero color value (e.g., a black pixel) or can have any other default pixel value. In some embodiments, the default pixel value can be determined by matching the nearest pixel in the boundary.

[0094] Figure 8 This illustrates a shear function configured to skew view 312 only along the horizontal axis of the pixel coordinate system (e.g., axis 323). However, any axis orientation can be applied. Additionally, as mentioned above regarding... Figure 7A and Figure 7B The post-processing operations (including tilting the convergence plane) can be configured during configuration mode using a user interface (e.g., user interface 244). Subsequently, shear functionality can be applied during runtime in the graphics pipeline 315.

[0095] Figure 9 This is a flowchart illustrating a system and method for a convergence plane of tilted multi-view images according to embodiments consistent with the principles described herein. Figure 9 The flowchart provides an example of different types of functionality implemented by a computing device (e.g., a multi-view display system) that executes an instruction set. Alternatively, Figure 9 A flowchart can be viewed as an example of elements depicting a method implemented in a computing device according to one or more embodiments.

[0096] At item 404, the computing device generates multiple views of a multi-view image. For example, applications (e.g., Figure 4 Application 205) can dynamically generate views of multi-view images in response to user input. The application can load the views into memory (e.g., ...). Figure 4 memory 208, Figure 8 In the memory 303).

[0097] At item 407, the computing device identifies the distance between each view and the central viewpoint. For example, an application could identify this distance based on a view index number that indicates the position of each view relative to another view. The view index number could indicate whether the view is to the right or left of the center, and how close it is to the center as the index number is ordered. Distance can also be calculated based on a baseline. If the baseline is predetermined, the view index number may be sufficient to infer the distance between the view and the central viewpoint.

[0098] At point 410, the computing device applies the shear function to each view to generate a sheared view. For example, the graphics pipeline (e.g., Figure 4 Graphics pipeline 200, Figure 8 The memory (303) can be instructed by the application to apply a post-processing shear function. The graphics pipeline can render a first view in the graphics pipeline as a first-sheared view based on a first shear intensity applied along an axis of the pixel coordinate system. The graphics pipeline can render a second view in the graphics pipeline as a second-sheared view based on a second shear intensity applied along an axis of the pixel coordinate system. The first and second shear intensities are different and based on the distance between the view and the central viewpoint. For example, the first shear intensity can be a negative shear intensity, and the second shear intensity can be a positive shear intensity. The sign of the shear intensity controls the direction of the shear applied to the view, which depends on the view's relative position to the central viewpoint.

[0099] At item 413, the computing device displays a rendered, sheared view. The sheared view effectively has a tilted convergence plane controlled by the shear variable applied to each view. The view can be rendered as a multi-view image on a multi-view display. For example, the graphics pipeline can communicate with the multi-view display using, for example, a graphics driver and / or firmware, to render a multi-view image for display.

[0100] The above discussion Figure 9 A flowchart can illustrate a system or method for real-time tilting convergence planes, with the functionality and operation of an instruction set implementation. If embodied in software, each block can represent a module, segment, or portion of code comprising instructions for implementing the specified logical functions(s). Instructions can be embodied in source code comprising human-readable statements written in a programming language, object code compiled from source code, or machine code comprising numerical instructions recognizable by a suitable execution system (such as a processor, computing device). Machine code can be converted from source code, etc. If embodied in hardware, each block can represent a circuit or several interconnected circuits for implementing the specified logical functions(s).

[0101] although Figure 9The flowchart illustrates a specific execution order; however, it should be understood that the execution order may differ from the depicted order. For example, the execution order of two or more boxes may be disrupted relative to the illustrated order. Furthermore, the two or more boxes shown may be executed simultaneously or partially simultaneously. Additionally, in some embodiments, one or more boxes may be skipped or omitted.

[0102] Figure 10 This is a schematic block diagram illustrating an example illustration of a multi-view display system 1000 providing a multi-view display according to embodiments consistent with the principles described herein. The multi-view display system 1000 may include a system of components that perform various computational operations for a user of the multi-view display system 1000. The multi-view display system 1000 may be a laptop computer, tablet computer, smartphone, touchscreen system, smart display system, or other client device. The multi-view display system 1000 may include various components such as, for example, processor(s) 1003, memory 1006, input / output (I / O) components(s) 1009, display 1012, and potentially other components. These components may be coupled to a bus 1015, which serves as a local interface to allow the components of the multi-view display system 1000 to communicate with each other. Although the components of the multi-view display system 1000 are shown as being contained within the multi-view display system 1000, it should be understood that at least some components may be coupled to the multi-view display system 1000 via external connections. For example, components can be externally inserted into or otherwise connected to the multiview display system 1000 via external ports, sockets, plugs, or connectors.

[0103] Processor 1003 may be a central processing unit (CPU), a graphics processing unit (GPU), any other integrated circuit that performs computational processing operations, or any combination thereof. The processor(s) 1003 may include one or more processing cores. The processor(s) 1003 includes circuitry for executing instructions. Instructions include, for example, computer code, programs, logic, or other machine-readable instructions received and executed by the processor(s) 1003 to perform computational functions embodied in the instructions. The processor(s) 1003 may execute instructions to manipulate data. For example, the processor(s) 1003 may receive input data (e.g., an image), process the input data according to an instruction set, and generate output data (e.g., the processed image). As another example, the processor(s) 1003 may receive instructions and generate new instructions for subsequent execution. Processor 1003 may include components for implementing a graphics pipeline (e.g., ...). Figure 4 Graphics pipeline 200, Figure 8 The hardware of the graphics pipeline 315. For example, the processors 1003 may include one or more GPU cores, vector processors, scaler processes, or hardware accelerators.

[0104] Memory 1006 may include one or more memory components. Memory 1006 is defined herein as including any one or both of volatile and non-volatile memory. Volatile memory components are those that do not retain information when power is off. Volatile memory may include, for example, random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), magnetic random access memory (MRAM), or other volatile memory structures. Volatile memory can be used to implement system memory (e.g., main memory, cache, etc.). System memory refers to fast memory that can temporarily store data or instructions for fast read and write access to assist processor(s)(s)1003. Memory 1006 may include Figure 4 memory 208 or Figure 8 The memory 303, or one or more other memory devices.

[0105] Non-volatile memory components are those components that retain information when power is off. Non-volatile memory includes read-only memory (ROM), hard disk drives, solid-state drives, USB flash drives, memory cards accessed via memory card readers, floppy disks accessed via associated floppy disk drives, optical disks accessed via optical disk drives, and magnetic tapes accessed via suitable magnetic tape drives. ROM can include, for example, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or other similar memory devices. Storage memory can be implemented using non-volatile memory to provide long-term retention of data and instructions.

[0106] Memory 1006 can refer to a combination of volatile and non-volatile memory used to store instructions and data. For example, data and instructions can be stored in non-volatile memory and loaded into volatile memory for processing by processor(s) 1003. Execution of instructions can include, for example, a compiler that translates the code into machine code in a format that can be loaded from non-volatile memory into volatile memory and then executed by processor 1003; source code converted to an appropriate format, such as object code that can be loaded into volatile memory for execution by processor 1003; or source code that is interpreted by another executable program to generate instructions in volatile memory and executed by processor 1003, etc. Instructions can be stored or loaded into any part or component of memory 1006, including, for example, RAM, ROM, system memory, memory, or any combination thereof.

[0107] Although memory 1006 is shown as separate from other components of the multi-view display system 1000, it should be understood that memory 1006 may be at least partially embedded or otherwise integrated into one or more components. For example, processor(s) 1003 may include onboard memory registers or caches to perform processing operations.

[0108] Multiple I / O components 1009 include, for example, a touchscreen, speaker, microphone, button, switch, dial pad, camera, sensor, accelerometer, or other components that receive user input or generate output directed to the user. Multiple I / O components 1009 may receive user input and convert it into data for storage in memory 1006 or for processing by multiple processors 1003. Multiple I / O components 1009 may receive data output from memory 1006 or multiple processors 1003 and convert it into a user-perceived format (e.g., sound, haptic response, visual information, etc.).

[0109] A specific type of I / O component 1009 is a display 1012. Display 1012 may include a multi-view display (e.g., multi-view display 112), a multi-view display combined with a 2D display, or any other display that presents images. A capacitive touchscreen layer serving as I / O component 1009 may be layered within the display to allow the user to provide input while simultaneously perceiving visual output. Multiple processors 1003 may generate data formatted for presentation on display 1012. Multiple processors 1003 may execute instructions to render images on the display for user perception.

[0110] Bus 1015 facilitates communication of instructions and data between (multiple) processors 1003, memory 1006, (multiple) I / O components 1009, display 1012, and any other components of the multi-view display system 1000. Bus 1015 may include address translators, address decoders, structures, conductive traces, conductive lines, ports, plugs, sockets, and other connectors to allow communication of data and instructions.

[0111] The instructions within memory 1006 can be embodied in various forms to implement at least a portion of the software stack. For example, the instructions can be embodied as operating system 1031, applications(s) 1034, device drivers (e.g., display driver 1037), firmware (e.g., display firmware 1040), or other software components. Operating system 1031 is a software platform that supports the basic functions of the multi-view display system 1000 (such as task scheduling, controlling I / O components 1009, providing access to hardware resources, managing power, and supporting applications 1034).

[0112] Multiple applications 1034 execute on operating system 1031 and can access the hardware resources of multi-view display system 1000 via operating system 1031. In this respect, the execution of multiple applications 1034 is at least partially controlled by operating system 1031. Multiple applications 1034 may be user-level software programs that provide advanced functions, services, and other capabilities to users. In some embodiments, application 1034 may be a dedicated "app" that is downloadable or otherwise accessible to the user on multi-view display system 1000. Users can launch multiple applications 1034 via a user interface provided by operating system 1031. Multiple applications 1034 may be developed by developers and defined in various source code formats. Application 1034 may be developed using several programming or scripting languages, such as C, C++, C#, Objective C, etc., as examples. Swift Perl, PHP, Visual Ruby, Go, or other programming languages. Applications 1034 (or more) can be compiled into object code by a compiler or interpreted by an interpreter for execution by processors 1003 (or more). Application 1034 can be... Figure 4 Application 205. An application can also be another application, which is used for creating... Figure 4 The developer of application 205 provides a user interface (e.g., user interface 244) as part of the configuration pattern.

[0113] Device drivers, such as display driver 1037, include instructions that allow operating system 1031 to communicate with various I / O components 1009. Each I / O component 1009 may have its own device driver. Device drivers can be installed, stored in memory, and loaded into system memory. For example, during installation, display driver 1037 translates high-level display instructions received from operating system 1031 into lower-level instructions implemented by display 1012 to display images.

[0114] Firmware (such as display firmware 1040 as an example) may include machine code or assembly code that allows I / O components 1009 or display 1012 to perform low-level operations. Firmware can translate electrical signals of specific components into higher-level instructions or data. For example, display firmware 1040 can control how display 1012 activates individual pixels at a low level by adjusting voltage or current signals. Firmware may be stored in and executed directly from non-volatile memory. For example, display firmware 1040 may be embodied in a ROM chip coupled to display 1012, such that the ROM chip is isolated from other memory and system memory of the multi-view display system 1000. Display 1012 may include processing circuitry for executing display firmware 1040.

[0115] Operating system 1031, applications(s) 1034, drivers (e.g., display driver 1037), firmware (e.g., display firmware 1040), and potentially other instruction sets may all include instructions executable by processor(s) 1003 or other processing circuitry of the multi-view display system 1000 to perform the functions and operations described above. Although the instructions described herein may be embodied in software or code executed by processor(s) 1003 as described above, alternatively, the instructions may also be embodied in dedicated hardware or a combination of software and dedicated hardware. For example, the functions and operations performed by the instructions discussed above may be implemented as circuits or state machines employing any one or a combination of several techniques. These techniques may include, but are not limited to, discrete logic circuits having logic gates for implementing various logical functions when one or more data signals are applied, application-specific integrated circuits (ASICs) with appropriate logic gates, field-programmable gate arrays (FPGAs), or other components.

[0116] In some embodiments, instructions for performing the functions and operations discussed above may be embodied in a non-transitory computer-readable storage medium. The computer-readable storage medium may or may not be part of the multi-view display system 1000. Instructions may include, for example, statements, code, or declarations that can be retrieved from the computer-readable medium and executed by processing circuitry (e.g., processors 1003). In the context of this discussion, a “non-transitory computer-readable medium” can be any medium capable of containing, storing, or maintaining the instructions described herein for use by or in conjunction with an instruction execution system (such as the multi-view display system 1000, as an example).

[0117] Non-transitory computer-readable media can include any of a number of physical media, such as magnetic, optical, or semiconductor media. More specific examples of suitable computer-readable media may include, but are not limited to, magnetic magnetic disks, magnetic hard disks, memory cards, solid-state drives, USB flash drives, or optical discs. Furthermore, computer-readable media can be random access memory (RAM), including, for example, static random access memory (SRAM), dynamic random access memory (DRAM), or magnetic random access memory (MRAM). Additionally, computer-readable media can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or other types of memory devices.

[0118] The multi-view display system 1000 can perform any operation or implement any function as described above. For example, the flowcharts and process flows discussed above can be executed by the multi-view display system 1000, which executes instructions and processes data. Although the multi-view display system 1000 is shown as a single device, the embodiments are not limited thereto. In some embodiments, the multi-view display system 1000 can offload instruction processing in a distributed manner, such that multiple multi-view display systems 1000 or other computing devices operate together to execute instructions that can be stored or loaded in a distributed manner. For example, at least some instructions or data can be stored, loaded, or executed in a cloud-based system that incorporates the operation of the multi-view display system 1000.

[0119] Therefore, examples and embodiments of tilting the convergence plane of a multi-view image have been described. For example, as the multi-view image is rendered for display, the convergence plane can be tilted in a real-time graphics pipeline. In this regard, the convergence plane can be tilted by applying a shear function to the different views of the multi-view image based on the relative position of each view. It should be understood that the above examples are merely illustrative of some of the many specific examples representing the principles described herein. Obviously, those skilled in the art can readily devise many other arrangements without departing from the scope defined by the appended claims.

Claims

1. A computer-implemented method for determining the convergence plane of a tilted multi-view image, the method comprising: The views of the multi-view image are loaded into memory, and the views are formatted as bitmaps defined by a pixel coordinate system; Identify the distance between the view and the central viewpoint; as well as The view is rendered as a sheared view in the graphics pipeline according to a shear function applied along the axes of the pixel coordinate system. The shear strength of the shear function is related to the distance, wherein the greater the distance, the greater the shear strength. The shear function tilts the convergence plane, and The shear function is a graphical operation that shifts pixels of the multi-view image along a direction according to the shear intensity. The shear intensity quantifies the amount of shear effect applied to the multi-view image by the shear function, and the direction of shifting pixels of the multi-view image depends on the positive or negative sign of the distance.

2. The method for tilting the convergence plane of the multi-view image according to claim 1, wherein, The shear function is configured to skew the view only along the horizontal axis of the pixel coordinate system.

3. The method for tilting the convergence plane of the multi-view image according to claim 2, wherein, The multi-view images include maps generated by navigation applications.

4. The method for tilting the convergence plane of the multi-view image according to claim 1, wherein, The distance between a view and the central viewpoint is identified by determining the ordered view number of the view in the multi-view image.

5. The method for tilting the convergence plane of the multi-view image according to claim 1, further comprising: Receive user input from the user interface; as well as The shear intensity is determined based on the user input.

6. The method for tilting the convergence plane of the multi-view image according to claim 1, the method further comprising: The shear intensity is automatically determined by calculating the disparity value at a common point between the view and another view of the multi-view image.

7. The method for tilting the convergence plane of the multi-view image according to claim 1, further comprising: Receive user input from the user interface; The range of disparity values ​​is determined based on the user input; as well as The shader is configured to operate on the pixel in response to the pixel of the view having a disparity value within the range of the disparity values.

8. The method of tilting the convergence plane of the multi-view image according to claim 7, wherein the shader is configured to perform at least one of a transparency operation and a depth-of-field operation.

9. A multi-view display system, the system comprising: processor; as well as The memory stores a plurality of instructions, which, when executed, cause the processor to: The views of the multi-view image are loaded into the memory, and the views are formatted as bitmaps defined by a pixel coordinate system; as well as Instructions are sent to the graphics pipeline to render the view as a sheared view based on a shear function applied along an axis of the pixel coordinate system, the shear strength of which is related to the position of the view relative to other views in the multi-view image. The graphics pipeline is configured to implement the shearing function as pixels of the bitmap are sampled by the graphics pipeline, and The shear function tilts the convergence plane in the graphics pipeline, and The shear function is a graphic operation that shifts the pixels of the multi-view image according to the shear intensity. The shear intensity quantifies the amount of shear effect applied to the multi-view image by the shear function. The positive or negative sign of the shear intensity controls the direction of the shear applied by the shear function.

10. The multi-view display system according to claim 9, wherein, The shear function is configured to skew the view only along the horizontal axis of the pixel coordinate system.

11. The multi-view display system according to claim 9, wherein, The multi-view images include maps generated by navigation applications.

12. The multi-view display system according to claim 9, wherein, When these instructions are executed, they also cause the processor to: Receive user input from the user interface; and The shear intensity is determined based on the user input.

13. The multi-view display system according to claim 9, wherein, When the plurality of instructions are executed, the processor also causes the processor to automatically determine the shear intensity by calculating the disparity value at a common point between the view and another view of the multi-view image.

14. The multi-view display system according to claim 9, wherein, When these instructions are executed, they also cause the processor to: Receive user input from the user interface; Determine the range of disparity values ​​based on the user input; and The shader is configured to operate on the pixel in response to the pixel of the view having a disparity value within the range of the disparity values.

15. The multi-view display system according to claim 14, wherein, The shader is configured to perform at least one of transparency and depth of field operations.

16. The multi-view display system according to claim 9, wherein, The multi-view display system is configured to provide wide-angle emitted light using wide-angle backlighting during 2D mode; The multi-view display system is configured to provide directional emission light during multi-view mode using a multi-view backlight with a multi-beam element array, the directional emission light comprising a plurality of directional beams provided by each multi-beam element in the multi-beam element array; The multi-view display system is configured to time-multiplex the 2D mode and the multi-view mode using a mode controller, sequentially activating the wide-angle backlight during a first sequential time interval corresponding to the 2D mode, and activating the multi-view backlight during a second sequential time interval corresponding to the multi-view mode; and The direction of the directional beams in the plurality of directional beams corresponds to the different view directions of the multi-view image.

17. The multi-view display system according to claim 16, wherein, The multi-view display system is configured to guide light as guided light in a light guide; and The multi-view display system is configured to use a portion of the guided light scattered by the multi-beam element in the multi-beam element array as the directional emission light, wherein each multi-beam element in the multi-beam element array includes one or more of a diffraction grating, a micro-refractive element, and a micro-reflective element.

18. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor of a computing system, implement tilting a convergence plane in a graphics pipeline, the tilting of the convergence plane in the graphics pipeline comprising: Multiple views are generated for a multi-view image, each view being formatted as a bitmap defined by a pixel coordinate system, the multiple views including a first view and a second view; Based on the first shear intensity of the shear function applied along the axis of the pixel coordinate system, the first view is rendered as a first-sheared view in the graphics pipeline; as well as Based on the second shear intensity of the shear function applied along the axis of the pixel coordinate system, the second view is rendered as a second-sheared view in the graphics pipeline. Wherein, the first shear intensity of the shear function is related to the position of the first view relative to other views in the multi-view image, and The second shear intensity of the shear function is related to the position of the second view relative to other views in the multi-view image. The shear function causes the convergence plane to tilt, and The shear function is a graphic operation that shifts the pixels of the multi-view image according to the shear intensity. The shear intensity quantifies the amount of shear effect applied to the multi-view image by the shear function. The positive or negative sign of the shear intensity controls the direction of the shear applied by the shear function.

19. The non-transitory computer-readable storage medium according to claim 18, wherein, The shear function is applied only along the horizontal axis of the pixel coordinate system.

20. The non-transitory computer-readable storage medium according to claim 18, wherein, The shear function is configured to skew the first view and the second view along the vertical axis of the pixel coordinate system.

21. The non-transitory computer-readable storage medium according to claim 18, wherein, The first shear intensity is a negative shear intensity, and the second shear intensity is a positive shear intensity.

22. The non-transitory computer-readable storage medium according to claim 18, wherein, The graphics pipeline is configured to implement the shearing function as pixels of the bitmap of the multi-view image are sampled by the graphics pipeline.

Images