Processing extended dimensional light field images

Abstract

In one embodiment, a method, medium, and system processes and displays light field images using a view function based on pixel location in the image and the viewer's distance from the display (observer's Z position). The view function can be an angular view function that specifies different angular views for different pixels in the light field image based on inputs that can include x or y pixel location in the image, the viewer's distance from the display, and the viewer's angle relative to the display. In one embodiment, light field metadata, such as angular range metadata and / or angular offset metadata, can be used to process and display the images. In one embodiment, color volume mapping metadata can be used to adjust the color volume mapping based on the determined angular view. The color volume mapping metadata can also be adjusted based on the angular offset metadata.

Description

[Technical field]

[0001] [Related Applications] This application claims priority to U.S. Provisional Application No. 63 / 121,372, filed December 4, 2020, and European Patent Application No. 20211870.9, both of which are incorporated by reference in their entireties.

[0002] [Technical field] The present disclosure relates to the field of image processing, and in particular to image processing of light field images. [Background technology]

[0003] A recent development in the field of image processing and display is called light field processing, which has the ability to display images from previously rendered volumetric content at different viewpoints, both horizontally and vertically. These different orientations are at different angles from the classical "straight on" viewing position, where the line between the viewer and the display is perpendicular to the surface of the display. This type of image processing and display is now called 4D light field imaging, because the imaging can be described as a function of four values. These values ​​are the pixel location (e.g., x,y) of the previously rendered image and the horizontal and vertical angles of the viewpoint. Further background information on 4D light field images is provided in: Light Field Image Processing: An Overview, by Gaochang Wu,et.al., IEEE Journal of Selected Topics in Signal Processing, Vol.11, No.7, October 2017, pages 926-954, and Light Field Rendering, by Marc Levoy and Pat Hanrahan, in Proc. of the 23d Annual Conf. on Computer Graphics and Interactive Techniques, 1996, pages 31-42. Summary of the Invention

[0004] This disclosure describes methods and apparatus for mapping light fields, such as 4D light fields. In one embodiment, the method, medium, and system process and display light field images using a view function based on pixel location in the image and viewer position (observer distance Z) from the display. The view function can be an angular view function that specifies different angular views for different pixels in the light field image based on inputs that can include x or y pixel location in the image, viewer distance from the display, and viewer position relative to the display. In one embodiment, light field metadata, such as angular range metadata and / or angular offset metadata, can be used to enhance image processing and display. In one embodiment, color volume mapping metadata can be used to adjust color volume mapping based on the determined angular view. Color volume mapping metadata can also be adjusted based on angular offset metadata.

[0005] In one embodiment, the method comprises the steps of: receiving, for each pixel of a plurality of pixels in an image, image data represented in a light field format including image data of different views, such as different reference views; receiving a selection of a desired viewpoint associated with an image; determining one or more views at each pixel in the plurality of pixels using a view function that determines one or more views based on spatial coordinates of each pixel of the plurality of pixels in the image, based on a desired viewpoint, and based on a distance between the desired viewpoint and the display; In one embodiment, the method may include the following additional operations: Rendering an image based on the determined view; displaying the rendered image in the determined view; Image data received and decoded in light field format can be referred to as a baseband light field representation that includes different reference views that can be used to construct additional views based on these different reference views. The baseband light field image format can be represented as either a) a planar format decoded as tiles, where each tile is one of the possible views, or b) an interleaved format.

[0006] In one embodiment, the baseband light field image is a 4D light field image previously rendered as volumetric content, and a selection of a desired viewpoint is received from a user to view the image at the desired viewpoint. The view function may be an angle view function including a horizontal angle view function and a vertical angle view function, where the horizontal angle view function may have inputs including a distance between the desired viewpoint and the display, a horizontal spatial coordinate of a pixel, and a horizontal component of the desired viewpoint, and the vertical angle view function may have inputs including a distance between the desired viewpoint and the display, a vertical spatial coordinate of a pixel, and a vertical component of the desired viewpoint.

[0007] In one embodiment, a view function is defined for a reference plane at a reference distance from the display, and the view function determines the same view for all pixels in the image for any one viewpoint within the reference plane. For viewpoints outside the reference plane, the view function can determine different views for different pixels in the image. In one embodiment, the desired viewpoint is selected based on an estimated viewer position.

[0008] In one embodiment, the method comprises the additional operation of: receiving color volume mapping metadata; applying color volume mapping based on the determined view and color volume mapping metadata; In one embodiment, the color volume mapping metadata is adjusted based on the desired viewpoint and the angular offset metadata. In one embodiment, the angular offset metadata may be interpolated based on the desired viewpoint. In one embodiment, the color volume mapping metadata may vary across different images on a scene-by-scene or image-by-image basis.

[0009] In one embodiment, the method comprises the following additional operations: It may also include the step of interpolating the determined view at the desired viewpoint from a set of closest available reference views in the image data, in one embodiment, the interpolation may use bilinear interpolation from a dense light field image that includes many reference views.

[0010] In one embodiment, the method comprises the following additional operations: and restricting potential desired viewpoints to a valid viewing zone. In one embodiment, the restricting can include either (a) hard clamping invalid viewpoints to viewpoints within the valid viewing zone, or (b) soft clamping invalid viewpoints to viewpoints within the valid viewing zone. In one embodiment, the method includes the additional operation: The method may also include receiving metadata including the angular ranges used to determine the valid viewing zone.

[0011] Aspects and embodiments described herein may include a non-transitory machine-readable medium capable of storing executable computer program instructions that, when executed, cause one or more data processing systems to perform the methods described herein. The instructions may be stored in a non-transitory machine-readable medium, such as a volatile memory, such as dynamic random access memory (DRAM), or a non-volatile memory, such as flash memory or other forms of memory. Correspondences and embodiments described herein may be in the form of a data processing system that is constructed or programmed to perform the methods. For example, the data processing system may be constructed with hardware logic that performs the methods, or may be programmed with a computer program to perform the methods.

[0012] Aspects and embodiments described herein may also include computer products and computer media, such as, for example: A computer program product including instructions that, when executed by a computer, cause the computer to perform any of the methods described in this disclosure, including the following exemplary embodiments, such as exemplary embodiments 1-15.

[0013] A computer readable [storage] medium containing instructions that, when executed by a computer, cause the computer to perform any of the methods described in this disclosure, including the following exemplary embodiments, such as exemplary embodiments 1-15.

[0014] The above summary does not include an exhaustive list of all embodiment aspects of the present disclosure, and all systems, media, and methods may be implemented from any suitable combination of the various aspects and embodiments summarized above and those disclosed in the detailed description that follows. [Brief description of the drawings]

[0015] The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference symbols indicate similar elements and in which:

[0016] [Figure 1] 1 shows an example of three different views of an image, such as a 4D light field image, from three different viewing positions or viewpoints.

[0017] [Diagram 2] We show an example of how a 4D light field image can be stored as a set of tiles (called the decoded planar format), where each tile is one possible view of the image, and each view corresponds to a particular viewpoint or viewing position.

[0018] [Figure 3A] 1 shows examples of viewing zones and reference viewing positions for a full high definition (FHD) image (having a pixel resolution of 1920x1080).

[0019] [Figure 3B] Another example of the viewing zone for an FHD image and a right-most viewing position is shown.

[0020] [Figure 3C] 13 shows another example of viewing zones and nearest viewing points for an FHD image.

[0021] [Figure 4A] 1 shows examples of different viewing zones for different angular ranges of a light field image. [Figure 4B] 1 shows examples of different viewing zones for different angular ranges of a light field image. [Figure 4C] 1 shows examples of different viewing zones for different angular ranges of a light field image. [Figure 4D] 1 shows examples of different viewing zones for different angular ranges of a light field image.

[0022] [Figure 5A]1 illustrates an example of how to convert an invalid viewpoint or viewing position to a valid viewpoint for a light field image.

[0023] [Figure 5B] 1 illustrates another example of a method for converting an invalid viewpoint or viewing position into a valid viewpoint for a light field image.

[0024] [Figure 5C] 1 shows an example of a soft clamping function that can be used to convert invalid viewpoints to valid viewpoints.

[0025] [Figure 5D] 13 shows an example of the use of a soft clamp function to convert an invalid viewpoint to a valid viewpoint.

[0026] [Figure 6A] 1 shows a flow chart illustrating a method according to one embodiment.

[0027] [Figure 6B] 4 shows a flow chart illustrating a method according to another embodiment.

[0028] [Figure 7] 1 is a block diagram illustrating an example of a data processing system that can be used to implement one or more embodiments described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Various embodiments and aspects will be described with reference to the details set forth below, and the various embodiments are illustrated in the accompanying drawings. The following description and drawings are illustrative and are not to be construed as limiting. Numerous specific details are set forth in order to provide a thorough understanding of the various embodiments. However, in certain instances, well-known or conventional details are not described in order to concisely describe the embodiments.

[0030] Reference herein to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment. The appearances of the phrase "in one embodiment" in various places in this specification do not necessarily all refer to the same embodiment. The processes illustrated in the following figures are performed by processing logic that is comprised of hardware (e.g., circuitry, dedicated logic, etc.), software, or a combination of both. Although the processes are described below with respect to some sequential operations, it should be recognized that some of the described operations may be performed in different orders. Additionally, some operations may be performed in parallel rather than sequentially.

[0031] This disclosure describes a method, a non-transitory machine-readable medium, and a data processing system that can map a light field, such as a 4D light field that includes a reference view, to different viewpoints of an image. This disclosure begins with an overview specific to 4D light fields and then describes a process for mapping such a light field to a particular viewpoint. Further aspects of using interpolation and using metadata are then described. It will be understood that the embodiments described herein can be combined in various different combinations that are intended to be covered by the following claims.

[0032] In one embodiment, the light field, such as a 4D light field, can be a complete representation of a volumetric scene behind a flat surface, such as a flat surface of a display screen of a display device. The display device can display images of the scene at different viewpoints, so that the viewer is presented with different images depending on the viewing position or viewpoint. The images can be pre-rendered volumetric content that is stored in a 4D light field format. FIG. 1 shows an example of a system 10 that displays different images at different viewpoints on a display 12, according to one embodiment. When the viewer is in a central position, the display 12 shows an image 14, which is a conventional image presented today in systems that do not use a light field. When the viewer moves to the left of the display, the system is shown an image 18 that shows how the scene looks from a viewpoint on the left of the display 12. When the viewer moves to the right of the display, the display 12 presents an image 16 that shows how the scene looks from a viewpoint on the right of the display 12.

[0033] 4D light fields allow the viewer to look around in a scene, showing slightly different perspectives from each viewing position, as if looking through a window at a real scene. In one embodiment, the slightly different perspectives can show different images that can include specular highlights that are only visible in one view and not in adjacent views, such as light shining on snow, water, metal, skin, eyes, etc. In one embodiment, the light field can include occlusions in the viewing zone that are revealed by slight movements of the viewer. These occlusions can be at the borders of the window or in the scene itself, where some objects may be partially hidden by closer objects. In one embodiment, the light field can support both optically captured content and rendered graphics rendered by a computer or other data processing system. The light field can actually allow the viewer to move or walk around in the scene and see different views of the scene by moving to different viewpoints that are valid in the scene. For example, in FIG. 1, by changing the viewer's viewpoint, it may be possible to walk around at least a portion of the car and see the front of the car, the right side of the car, and the left side of the car, or at least a portion of the right side of the car and at least a portion of the left side of the car.

[0034] A light field can be considered to have four dimensions, including pixel location (e.g., x,y) and angular information (u,v). Additionally, for each pixel location, there is color information that represents the color of the pixel in each possible view. Each pixel location can have multiple views (e.g., multiple color values, one for each view) that are selected based on the angular information. That is, the color information selected at a pixel location depends on the angular information that is derived from the viewpoint selected by the user or is determined by the system (e.g., the system estimates the user's position). A first viewpoint causes the selection of a first view, which causes the selection of a first color information (corresponding to the first view) at a particular pixel, and a second viewpoint causes the selection of a second view, which causes the selection of a second color information (corresponding to the second view) at the same pixel. A conventional image can be considered to have three dimensions, represented by pixel location (e.g., x,y) and color information "c" (so a conventional image can be represented by the notation Im(x,y,c)). The additional information in a 4D light field is the angular information (e.g., u,v) obtained from the viewpoint chosen to view the scene, so a 4D light field image can be represented by the notation Im(x,y,u,v,c).

[0035] The light field can have an angular resolution that can be defined as the number of views (also referred to as the number of angular views in some embodiments). In the example of FIG. 2, the horizontal angular resolution is 5 (5 different views) and the vertical angular resolution is 5 (5 different views). The light field can also be defined as having an angular range (e.g., in degrees) that can be defined as the maximum angle in both the vertical and horizontal directions. In one embodiment, there is a specified angular range supported by the content, which can be determined during the capture, creation, or rendering of the content, and can be stored and transmitted as angular range metadata associated with the content, where the angular range can be specified in degrees. In one embodiment, there are two angular range values, one for the maximum horizontal angle at which the light field can be accurately viewed, and one for the maximum vertical angle at which the light field can be accurately viewed. If the angular range is 0 for both horizontal and vertical, there are no alternative views in either direction, and the content can only be properly viewed from what is described herein as the reference viewing position, and a single view from this viewing position (reference viewing position) is the same as the existing content in a traditional 2D system, and is the same as the image 14 shown in FIG. 1. Another term that can be used to describe angular information is angular density, which can be expressed as the ratio (angular resolution) / (angular range), where angular resolution can be expressed in units of views / degree.

[0036] A 4D light field image can be conceptualized in two ways: a planar light field image (as shown in FIG. 2) and an interleaved light field image. The received and decoded 4D light field image is called a baseband light field image that includes a reference view, which can be used to create more views based on the reference view. In a planar light field image, such as the 4D light field image shown in FIG. 2, each tile or plane is an image that corresponds to a different reference viewing position or viewpoint, and each of these tiles or planes is one of the reference views. A planar 4D light field is suitable for spatial operations such as compression and resizing using existing image processing architectures described below. In the example of FIG. 2, the angular resolution is 5 views by 5 views. The central tile 14 (e.g., when u=2, v=2) is today's conventional 2D image and corresponds to the reference viewing position described below. The angular range of this example of FIG. 2 is 30° in both the horizontal and vertical directions. The left-most tile along the center row of tiles (v=2) corresponds to image 18 of FIG. 1 at a reference viewing distance, as described below, and the right-most tile along the center column of tiles corresponds to image 16 of FIG. 1 at a reference viewing distance, as described below. Other tiles in FIG. 2 correspond to different viewing positions at different horizontal angles 20 and vertical angles 22 at a reference viewing distance, as described below. For example, the tile in the upper left corner of FIG. 2 corresponds to a horizontal view of the left-most viewpoint and a vertical view of the vertically highest viewpoint. The tile in the upper right corner of FIG. 2 corresponds to a horizontal view of the right-most viewpoint and a vertical view of the vertically highest viewpoint. The tile in the lower left corner of FIG. 2 corresponds to a horizontal view of the left-most viewpoint and a vertical view of the vertically lowest viewpoint. The tile in the lower right corner of FIG. 2 corresponds to a horizontal view of the right-most viewpoint at a vertical view of the vertically lowest viewpoint. The horizontal view of pixels along the top row of tiles, from left to right in this representation of Figure 2, is (a) for the first tile (the tile in the upper left corner), x0u0, x1u0, x2u0, x3u0, ... (b) for the next tile (to the right of the first tile), x0u1, x1u1, x2u1, ...The horizontal views of the pixels along the top row of tiles, from left to right in this representation of Figure 2, are (a) for the first tile (the tile in the upper left corner), y0u0, y1u0, y2u0, y3u0, ... (b) for the next tile (to the right of the first tile), y0u1, y1u1, y2u1, .... Thus, at the reference viewing distance given below, all pixels in the upper left corner tile have the view shown in the upper left corner tile of Figure 2, and all pixels in the next tile (along the top row of tiles) have the view shown in the next tile of Figure 2.

[0037] A 4D light field can also be represented as an interleaved image. In this representation, the angular views are interleaved horizontally and vertically, so that adjacent pixels correspond to different view directions. This representation can facilitate angular processing such as interpolation between viewpoints, since only a small contiguous region of the 4D light field image is needed to select or interpolate a new viewpoint (described below). The views of the pixels along the top row of this representation are, from left to right, X0U0, X0U1, X0U2, X0U3, X0U4, X0U5, X0U6, X0U7, X0U8, X0U9, X0U10, X0U11, X0U12, X0U13, X0U14, X0U15, X0U16, X0U17, X0U18, X0U19, X0U20, X0U21, X0U22, X0U23, X0U24, X0U25, X0U26, X0U27, X0U28, X0U31, X0U32, X0U33, X0U34, X0U35, X0U36, X0U37, X0U38, X0U39, X0U40, X0U41, X0U42, X0U43, X0U44, X0U45, X0U46, X0U47, X0U48, X0U49, X0U51, X0U52, X0U53, X0U54, X0U55, X0U56, X0U57, X0U58, X0U59, X0U61, X0U62, X0U63, X0U64, X0U65, X0U65, X0U76, X0U87, X0U98, X0U109, X0U1109, X0 1、 X0U 2、 X0U 3、 X0U 4、 X1U 0、 X1U 1、 Generate ...

[0038] Planar 4D light fields share some similarities with planar color representations of images, where each color dimension is represented as a complete image (e.g., R,R,R,G,G,G,B,B,B). Alternatively, a color image can be represented in an interleaved format (e.g., R,G,B,R,G,B,R,G,B). Similarly, an interleaved 4D light field represents each view interleaved with the other views. By indexing into the appropriate portion of the system memory that stores the light field image, conversion between the planar and interleaved formats can be performed reversibly and efficiently.

[0039] In this disclosure, many figures and methods focus only on the horizontal axis (x, u) for clarity, it being understood that the same approach can be applied to the vertical axis (y, v) even if not explicitly stated.

[0040] Viewing zones and standard viewing positions A viewing zone can be defined as an area in which a valid viewpoint can be rendered. In one embodiment, this viewing zone can be used to constrain the view as described further below. The valid viewing zone is the viewing zone of an image constrained and defined by the image's angular range metadata, which specifies the range of angles at which the image may be viewed. A valid viewpoint is any viewpoint within the valid viewing zone. An invalid viewpoint is a viewpoint outside the valid viewing zone. This viewing zone can be divided into regions 59 and 61 shown in Figures 3A, 3B, and 3C, which show a horizontal top view 50 of the display 51 and the viewing zone including regions 59 and 61. In the example shown in Figures 3A, 3B, and 3C, the valid viewing zone is restricted to regions 59 and 61. Region 61 indicates the viewing zone in which the four-dimensional light field can be accurately viewed at or beyond the visual acuity of a normal observer, and this region 61 is beyond the reference viewing distance defined by reference viewing plane 56. That is, region 61 has an observer / viewer distance from display 51 that is greater than the distance between reference viewing plane 56 and display 51 (shown on y-axis 53). Region 59 shows a viewing zone where the 4D light field can still be seen accurately, but the observer can see individual pixels due to limited spatial resolution (as the observer is closer to display 51 than reference viewing plane 56). The viewing zone is determined by the angular range and spatial resolution, as explained in more detail below. Reference viewing plane 56 separates regions 59 and 61 and includes a reference viewpoint position 57 that is at the center of reference viewing plane 56. The position along x-axis 55 can be the (horizontal) position of the observer. It will be understood that the representation of the viewing zone in the vertical direction will be similar to that shown in Figures 3A, 3B and 3C.

[0041] A reference viewing position 57 at the center of the reference viewing plane 56 can be defined as being at a reference distance z0 when it is in the center of the screen (in horizontal and vertical directions) and the spatial resolution of the screen is 60.8 pixels / degree. For a full high definition (FHD) resolution image (e.g., a 1920*1080 image), this reference distance z0 is 3.2 picture heights (hence the distance between the reference plane 56 and the display 51 is 3.2 picture heights). As shown in Figures 3A, 3B, etc., the reference plane and the plane of the display surface are parallel to each other. This reference distance z0 separates the viewing zones 59 and 61, and the reference viewing plane 56 is located at a distance of z0 from the display 51. At larger distances, the visual acuity is reduced compared to the screen resolution, allowing images to be displayed with high visual fidelity. At closer distances, the visual acuity is greater than the screen resolution, and the individual pixels that make up the image can be seen. The reference viewing distance (z0, in units of screen height) can be calculated from the vertical spatial resolution of the panel in the vertical dimension (e.g. Y=1080, 2160, or 4320, etc.) by the following formula:

number

[0042] As also shown in Figure 3A, there are three angles positioned horizontally across the display: left edge, center, and right edge. These indicate the angles formed by the normal to the screen and the viewing position. These angles are calculated from the pixel position (x,y, in screen height units), observer position (ObX,ObY, in screen height units), and reference viewing distance (z0, in screen height units) using thetafun (in degrees), as explained below.

number

[0043] At the nominal viewing position, these angles (shown as angles 63 alongside the display 15) are -15.5 degrees, 0 degrees, and 15.5 degrees, respectively, assuming an image with an aspect ratio of 16:9. Also shown are three horizontal view 65 indices, indicating which angular views will be presented to the viewer. The angular views are calculated by the formula:

number

[0044] Viewing position on the reference viewing plane and the rightmost viewing position The viewing position along the line at z0 is referred to as the viewing position at the reference viewing plane 56. As the viewer moves laterally (at a constant Z distance relative to the display 51) along this line at the reference viewing plane 56, a new view of the scene from the 4D light field is displayed. If the viewpoint is on the reference viewing plane, then all pixels in the image will have the same view at that viewpoint, in one example. Figure 3B shows one viewpoint that is on the reference viewing plane, called the rightmost viewpoint 67. This is defined as the point that is in the content angular range from the center of the image.

[0045] As shown in Figure 3B, the angle between the screen normal and the right-most viewpoint is 30 degrees at the center of the screen, which is the angle range in this example. Also shown in Figure 3B, the angles between the screen normal and the right-most viewpoint are all different, ranging from 40.5 degrees at the left-most pixel of the image to 16.7 degrees at the right-most pixel. However, the angular view was defined earlier to be the same for all positions in the viewing plane. This is done by further defining the following equation:

number

number

[0046] Nearest listening position Figure 3C shows an additional viewpoint of interest called the "Closest Viewing Position", which is defined as the closest point within the viewing zone. As can be seen, the angle formed between the screen normal and the Closest Viewing Point is 40.5 degrees for the leftmost pixel of the image, and the same for the Rightmost Viewing Point. The angles between the screen normal and the closest viewing point are 0 and -40.5 degrees for the central and rightmost pixels, respectively. Their corresponding angular views are also labelled (see Views 65) and are -2, 0 and 2 respectively (-Umax,0,Umax). This means that the appropriate image for the closest viewing position is constructed from multiple views from the 4D light field: the leftmost pixel is taken from the rightmost view (Umax=2), the central pixel is taken from the central view (u=0) and the rightmost pixel is taken from the leftmost view (-Umax=-2). Intermediate pixels are either taken from intermediate views or are interpolated; interpolation is described below. Note that views from any viewing position not on the Reference Viewing Plane are similarly constructed from multiple views from the 4D light field. That is, in one embodiment, when the viewpoint is not on the Reference Viewing Plane 56, multiple views are used for a pixel in the image, and different pixels at different spatial locations use different views.

[0047] Given the aspect ratio (ar), the right viewing position (ObXmax, in screen height units), and the reference viewing distance (z0, in screen height units), the closest viewing point (ObZmin, in screen height units) is calculated:

number

[0048] Farthest listening position The farthest viewing position ObZmax (in units of screen height) is the farthest distance at which the 4D light field may be correctly perceived. In one embodiment, it is calculated from the aspect ratio (ar), the right-most viewing position (ObXmax, in units of screen height), and the reference viewing distance (z0, in units of screen height):

number

[0049] How angle range affects viewing range The angular range corresponds to the maximum angle from the center of the screen to the rightmost viewing position. Figures 4A, 4B, 4C, and 4D show examples comparing angular ranges of 0 degrees, 5 degrees, 15.5 degrees, and 30 degrees, respectively, and the rightmost viewing position for each of these examples.

[0050] At the 0 degree angular extent shown in Figure 4A, the viewing zone is only a single point corresponding to the reference viewpoint. This is the case for 2D images today, where the images are generated to be viewed in correct perspective from a single viewing position. At any other viewing position outside the viewing zone, the perspective becomes incorrect and the human visual system infers that the image is on a 2D plane, rather than the actual scene "behind" the 2D plane. For U=1 and V=1, the angular resolution must be 0, and the 4D light field is folded to be the same as a regular 2D image.

[0051] In the 5 degree angular range shown in Figure 4B, the viewing zone (including regions 59A and 61A) grows slightly and becomes diamond-shaped. Within this viewing zone, the viewpoint shifts slightly so that a perspective-correct image can be calculated and displayed, mimicking the experience of looking through a window. The right-most viewpoint 71 on the reference viewing plane 56 provides the same view (view 2) for all pixels in the image.

[0052] At the 15.5 degree angular range shown in Figure 4C, the viewing zone matches the size of the image (rightmost viewing position 73 corresponds to the width of the image) and allows viewing in correct perspective from any distance. The viewing zone (including regions 59B and 61B) increases as the angular range increases. At the 30 degree angular range shown in Figure 4D, the viewing zone (including regions 59C and 61C) is larger than the image at most locations in the viewing zone. The rightmost viewing position 75 extends beyond the rightmost pixel of display 51.

[0053] Angular View Functions Using the horizontal and vertical view functions, the angular view for a given viewing position can be calculated.

number

[0054] We can further specify the following constraints on the angular view function to ensure that each view of a planar 4D light field image corresponds to an image with the correct perspective on the reference viewing plane: Using these constraints, we can construct views of every viewpoint on the reference plane, and use these views as reference views for constructing views of viewpoints outside the reference viewing plane:

number

number

number

[0055] Angular view interpolation and dense 4D light fields For many viewing positions (especially viewpoints outside the reference viewing plane), the angular view function may return a fractional value. This means that the required view is somewhere between adjacent views, so the correct view for such a viewing position needs to be interpolated from existing views, such as the tiles and views shown in FIG. 2. The process of creating an image by interpolating between two or more images is known in the art. However, to perform such an interpolation, it is desirable to have a light field image with high angular density, which in this disclosure is called a dense 4D light field image. A dense 4D light field is defined as having an angular density high enough that the difference between adjacent views is imperceptible or barely perceptible to the eye at the reference viewing plane. This occurs when a lateral shift of the viewpoint (ObXmin=ar / X) of the same size as a single pixel size of the image causes the angular view to increase close to 1.0. This occurs at the reference viewing position when:

number

number

[0056] At viewing distances greater than the reference plane, the same lateral shift in ObXmin may result in an angular view increment less than 1.0, but this is expected due to the reduced visual acuity of the observer at larger viewing angles.

[0057] The simplest form of angular view interpolation is nearest neighbor, where the interpolated view (IV) is calculated according to:

number

[0058] A better (smoother) approach in one embodiment is to use bilinear interpolation where the closest two, three or more angular views are added based on the straight line distance to each view. Bilinear interpolation can first interpolate the horizontal view and then the vertical view, or vice versa. An example of bilinear interpolation using two views is as follows:

number

[0059] In dense 4D light fields with sufficiently high angular resolution, this interpolation can result in a smoother visual experience with viewpoint changes. Using more than two views for bilinear interpolation can result in more consistent sharpness across multiple viewpoints, since some level of interpolation or viewpoint combination is always applied.

[0060] Interleaved 4D light fields can be useful in some embodiments when interpolating between views, where for each spatial location x,y, the closest angular view (u,v) is stored in an adjacent memory location, making lookup efficient. For example, when interpolating between three adjacent views, the layout of the data in memory allows the system to read a sequence of consecutive address locations from memory for the three views to obtain the data needed to perform the interpolation.

[0061] More advanced forms of interpolation are also possible and are useful in some embodiments for less dense (or sparse) 4D light fields. One such interpolation is conventionally known for use in frame rate interpolation and is the family of motion estimation-motion compensation techniques. These attempt to create an interpolated view by aligning features between adjacent views and shifting or morphing the adjacent views. Such techniques are known in the art and are used in services such as Intel TrueView. These techniques can be used in the embodiments described herein to interpolate between views to obtain interpolated pixel color data.

[0062] Further improvements in the interpolation are also possible by considering 4D light field videos, which include multiple 4D light field images separated by some time t, such as a frame rate of 30 frames per second or a time based on some other frame rate.

[0063] Constraining the angular view to the effective viewing range For viewing positions outside the valid viewing range, the angular view function may return angular views beyond the range included in the image (e.g., Umax, -Umax). This can be resolved in one or more embodiments by restricting the observer positions ObX, ObY, and ObZ to the valid viewing range before determining the angular view. In one embodiment, the view closest to the observer position (within the valid viewing range) is selected as the view to be rendered, preventing abrupt changes in the angular view when the observer crosses the boundary between the valid viewing zone and outside the zone.

[0064] The angular views can be constrained as shown in Figures 5A and 5B. Referring now to Figure 5A, an actual viewer position 91 (ObX, ObY, ObZ) is outside the valid viewing zone that includes regions 59D and 61D, and this actual viewer position 91 is constrained to be within the valid viewing zone by selecting a constrained viewpoint 93 (ObXc, ObYc, ObZc) and determining the view to be provided using that constrained viewpoint 93. In the case of Figure 5B, an actual viewer position 95 is outside the valid viewing zone, and this actual viewer position 95 is constrained to be within the valid viewing zone by selecting a constrained viewpoint 97 (ObXc, ObYc, ObZc) and determining the view to be provided using that constrained viewpoint 97. Within the valid viewing range, the intended perspective is rendered, with the perspective shifting in a natural way with the viewer position, and outside the viewing zone, it is guaranteed that the angular view is the closest available view that is within the valid viewing range. This approach works, in one embodiment, by finding the intersection of a line formed between observer position 91 (at ObX, ObY, ObZ) and reference viewing position 57 with the line formed by the boundary of the valid viewing area. This intersection is shown in Figure 5A at constrained viewpoint 93 and in Figure 5B at constrained viewpoint 97. This intersection is calculated in one embodiment in two stages.

[0065] The first step is to determine the point on the boundary of the nearest valid viewing zone that is on a line between the observer and the reference viewing position:

number

[0066] The second step is to restrict the observer's position to within the valid viewing range, creating a constrained viewing position (also called a constrained viewpoint):

number

[0067] In another embodiment, the viewing position can be further modified to achieve a soft clamping to the boundaries of the valid viewing range. In one embodiment, this is achieved by smoothly "compressing" the viewing positions within the valid viewing zone towards the reference position. As the observer moves away from the reference viewing position, the viewpoint changes less as the boundaries of the reference viewing position are approached. This provides a less natural experience than viewing through a window, but avoids abrupt transitions at the edges of the valid viewing zone and allows for a larger size of the zone in which different perspectives can be observed based on the viewing position.

[0068] The operations in one embodiment for applying a soft clamp are as follows: 1) Determine a boundary of the effective viewing zone that is closer to the observer than any other boundary of the effective viewing zone, and determine a point on that boundary of the effective viewing zone that is on a line between the observer and the reference viewing position (same as before, but without the second clamping stage). 2) Compress the angular view of the area close to the effective viewing zone towards the reference position as shown below: a. Determine the distance of the observer from the reference viewing position:

number

number

number

number

number

number

[0069] The above function uses a cubic spline as the soft compression function, as shown in FIG. 5C. Other functions with similar shapes can be used instead. The black dashed line in FIG. 5C represents the clamping method described above (without soft clamping). Curve 103 shows the soft clamping function. Points 107 and 108 show the locations of c1 and c2, respectively, and point 105 shows how an observer position outside the valid viewing range is mapped to another viewpoint within the valid viewing range. FIG. 5D shows an example of using soft clamping to clamp viewpoint 115 outside the valid viewing range to viewpoint 117 within the valid viewing range. Note that viewpoint 117 is not at the edge of the valid viewing range, but is offset a short distance from it. Color Volume Mapping Once the correct angular view is obtained, color volume mapping can also be performed to map the dynamic range of the light field image to the capabilities of the display for the selected angular view. This can be done using techniques used in Dolby Vision, such as the color volume mapping process. This process can be guided by metadata as described in various existing Dolby patents (see, for example, U.S. Pat. No. 10,600,166). This metadata can be based on an image or set of images, such as a set of images in a scene that share the same overall color balance and color range and dynamic range. The metadata can vary from scene to scene or image to image, depending on the level of control required to render the images as accurately as possible. Different scenes may have different amounts of dynamic range and may also have different color balances and color ranges, so the metadata may change based on these different scenes. Similarly, different views in the same light field image may have different amounts of dynamic range and may have different color balances and color ranges. In one embodiment, a set of color volume mapping (CVM) metadata can be provided for the light field image, and this CVM metadata can be adjusted based on the selected view, rather than providing separate CVM metadata for each of the different possible views.

[0070] When applying color volume mapping to a view rendered from a 4D light field image, an embodiment may use a process similar to the Dolby Vision process described above. In one embodiment, the 4D light field may include additional metadata fields that allow for adjustments to the mapping based on viewing position. For example, from a reference viewing position, the image displayed on the display may be dark. However, as the observer moves to the right-most viewing position, a bright object such as a window or light source may appear, changing the characteristics of the image for optimal color volume mapping. This works similarly to human vision, where looking towards a window brightens the image on the retina and the visual system adjusts the exposure (also known as adaptation).

[0071] In one embodiment, metadata included in the 4D light field (e.g., CVM metadata) can be adjusted based on the angular view function by the following steps: 1) Load the metadata corresponding to the reference viewing position. 2) Load angular offset metadata corresponding to at least one additional viewing position. In the example, this includes nine offsets to the offset metadata related to the average luminance in frames corresponding to the extreme viewing angles (u=-Umax,0,Umax and v=-Vmax,0,Vmax). For example, the angular offset metadata might include a value indicating that the average image luminance from the rightmost viewing position is 0.1 brighter than the reference viewing position (corresponding to the bright window mentioned above). The resolution of the angular offset metadata may match the angular resolution of the 4D light field image, or it may be smaller. The angular range of the angular offset metadata must match the angular range of the 4D light field image, so the rightmost offset metadata is paired with the rightmost angular viewing position. The nine offsets in this example are:

number

[0072] Methodology examples and implementation considerations Two examples of methods using one or more embodiments described herein are provided with reference to FIGS. 6A and 6B. The method illustrated in FIG. 6A may begin at operation 201. At operation 201, a data processing system may receive a light field image, such as a 4D light field image, having multiple possible views of the image. In one embodiment, the data processing system may receive a series of light field images, such as animation content or a movie. The data processing system may also receive optional metadata related to the views in the image at operation 203. This optional metadata may include color volume mapping metadata, angle range metadata, and angle offset metadata. As previously discussed, the angle offset metadata may be used to adjust the color volume mapping metadata used to perform the color volume mapping dependent on the particular view selected. At operation 205, the data processing system may receive a selection of a desired viewpoint. For example, a user may select a particular viewpoint using a user interface provided by the data processing system. Alternatively, the data processing system may determine the desired viewpoint based on an estimate of the user's position in the environment surrounding the light field image. Next, in operation 207, the data processing system may determine one or more views at each pixel location using the following: a desired viewpoint, and a view function that determines the view as a function of the pixel location and the distance between the desired viewpoint and the display. Next, in operation 209, the data processing system may render an image based on the views determined in operation 207. If multiple light field images have been received, each may have a desired viewpoint, which is used to determine the view based on the view function. In operation 211, the data processing system may display the rendered image.

[0073] In the method illustrated in FIG. 6B, the data processing system may receive one or more light field images and associated metadata in operation 251. The data processing system may also receive a selection of a desired viewpoint, which may be a user selection or a selection performed by the data processing system based, for example, on an estimated position of the user. Next, in operation 253, the data processing system determines whether the desired viewpoint is a valid viewpoint based on the valid viewing range of the particular light field image. For example, the data processing system may determine that the viewpoint is outside the valid viewing range, in which case the data processing system performs operation 255 to clamp the desired viewpoint to a valid viewpoint using one or more embodiments described herein. If operation 253 determines that the viewpoint is valid or if operation 255 creates a valid viewpoint, processing may proceed to operation 257. In operation 257, the data processing system may use an angular view function as described herein against a current input of the desired viewpoint (which may be a constrained viewpoint) to determine a view to be rendered for each pixel location in the image. In certain cases, the view determined by the view function may be interpolated from existing views, such as adjacent or nearby views of the desired viewpoint. For example, in one embodiment, bilinear interpolation may be used to interpolate between the nearest views to derive the appropriate view, and therefore the pixel color value for each pixel location. Then, in operation 261, the data processing system may adjust the color volume mapping using the color volume mapping metadata and the angular offset metadata to adjust the color volume mapping metadata. For example, interpolation may be used to interpolate the angular offset metadata to derive the appropriate correction or adjustment to the color volume mapping metadata based on the desired viewpoint or the determined view. Then, in operation 263, the data processing system may render an image based on the determined view and the final metadata, and display the rendered image.

[0074] The angular view functions described herein can be implemented using a three-dimensional (3D) lookup table or using the functional form described herein. In an embodiment, the atan function can be replaced by a close approximation as known in the art. The soft compression function can be implemented as a one-dimensional lookup table or using the functional form described herein. For content with large angular resolution, the amount of data can be very large. Storing the entire light field image in DRAM memory may be impossible for some applications. In this case, rather than storing the entire image in DRAM memory, it may be desirable to store the light field image in an interleaved format, interpolate each angular view, and perform color volume mapping using a few pixels (stored in DRAM memory) at a time. It may also be desirable to compress the light field image, especially for distribution over a network or set of networks (such as the Internet). This compression can take advantage of the high degree of correlation between adjacent viewpoints and can be done using JPEG, JPEG2000, HEVC, AVC, VVC, etc., or alternatively using MPEG-I.

[0075] FIG. 7 illustrates an example of a data processing system 800 that can be used in one or more embodiments described herein. For example, the system 800 can be used to perform any of the methods or calculations described herein, such as those illustrated in FIGS. 6A and 6B. The data processing system can also create light field images with associated metadata for consumption by a client system. Note that while FIG. 7 illustrates various components of the device, it is not intended to represent a particular architecture or manner of interconnecting the components, as such details are not germane to the disclosure. It is also understood that networked computers or other data processing systems, or other consumer electronic devices, having fewer components or possibly more components, may also be used with the disclosed embodiments.

[0076] As shown in FIG. 7, device 800, which is one form of data processing system, includes a microprocessor 805 and a bus 803 coupled to a read only memory (ROM) 807, a volatile RAM 809, and a non-volatile memory 811. The microprocessor 805 can retrieve instructions from the memories 807, 809, and 811 and execute the instructions to perform the operations described above. The microprocessor 805 may include one or more processing cores. The bus 803 interconnects these various components and also interconnects these components 805, 807, 809, and 811 with peripheral devices such as a display control and display device 813 and input / output (I / O) devices 815, which may be touch screens, mice, keyboards, modems, network interfaces, printers, and other devices well known in the art. Typically, the input / output devices 815 are coupled to the system through an input / output controller 810. Volatile Random Access Memory (RAM) 809 is typically implemented as Dynamic RAM (DRAM), which requires continuous power to refresh or maintain the data in the memory.

[0077] Non-volatile memory 811 is typically a magnetic hard drive, a magnetic optical drive, an optical drive, a DVD RAM, flash memory, or other type of memory system that maintains data (e.g., large amounts of data) even after power is removed from the system. Typically, non-volatile memory 811 is also random access memory, although this is not required. While FIG. 7 illustrates non-volatile memory 811 as a local device directly coupled to the remaining components in the data processing system, it is understood that the disclosed embodiments may utilize non-volatile memory off the system, such as a network storage device coupled to the data processing system via a network interface, such as a modem, an Ethernet interface, or a wireless network. Bus 803 may include one or more buses interconnected through various bridges, controllers, and / or adapters, as is well known in the art.

[0078] Some of the above may be implemented with logic circuits, such as dedicated logic circuits, or with microcontrollers or other types of processing cores that execute program code instructions. Thus, the processes taught in the above discussion may be performed with program code, such as machine-executable instructions that cause a machine that executes those instructions to perform a particular function. In this context, a "machine" may be a machine that converts intermediate-form (or "abstract") instructions into processor-specific instructions (e.g., an abstract execution environment such as a "virtual machine" (e.g., Java Virtual Machine), an interpreter, a common language runtime, a high-level language virtual machine, etc.), and / or electronic circuits (e.g., "logic circuits" implemented with transistors) located on semiconductor chips designed to execute instructions, such as general-purpose processors and / or special-purpose processors. The processes taught in the above discussion may also be performed by (in place of or in combination with) electronic circuits designed to perform the process (or a portion thereof) without executing program code.

[0079] The present disclosure also relates to an apparatus for performing the operations described herein. The apparatus may be specially constructed for the required purposes or may comprise a general-purpose device selectively activated or reconfigured by a computer program stored on the device. Such computer programs may be stored on non-transitory computer-readable storage media, such as any type of disk including floppy disks, optical disks, CD-ROMs and magneto-optical disks, DRAM (volatile), flash memory, read-only memory (ROM), RAM, EPROM, EEPROM, magnetic or optical cards, or any type of medium suitable for storing electronic instructions, each coupled to a device bus.

[0080] A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, non-transitory machine-readable media include read-only memory ("ROM"), random access memory ("RAM"), magnetic disk storage media, optical storage media, flash memory devices, etc.

[0081] An article of manufacture may be used to store program code. The article of manufacture storing the program code may be embodied as, but is not limited to, one or more non-transitory memories (e.g., one or more flash memories, random access memories (static, dynamic, or other)), optical disks, CD-ROMs, DVD ROMs, EPROMs, EEPROMs, magnetic or optical cards, or other types of machine-readable media suitable for storing electronic instructions. The program code may also be downloaded from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a data signal embodied in a propagation medium (e.g., via a communications link (e.g., a network connection)) and stored in the non-transitory memory (e.g., DRAM or flash memory or both) of the client computer.

[0082] The preceding detailed descriptions have been presented in terms of algorithms and symbolic representations of operations on data bits within a device memory. These algorithmic descriptions and representations are the tools used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is herein, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. The operations require physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0083] It should be borne in mind, however, that these and similar terms are all associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise noted, as is clear from the above discussion, throughout the description, discussions utilizing terms such as "receive," "determine," "send," "end," "wait," "change," and the like, will be understood to refer to operations and processes of the device or similar electronic computing apparatus that manipulate data represented as physical (electronic) quantities in the device's registers and memories, and convert them into other data similarly represented as physical quantities in the device's memories or registers, or other storage, transmission, or display of such information.

[0084] The processes and displays presented herein are not inherently related to any particular apparatus or other devices. Various general purpose systems may be used with programs in accordance with the teachings presented herein, or it may prove convenient to construct more specialized apparatus to perform the operations described. The required structure for a variety of these systems will appear from the description that follows. Further, the disclosure has not been described with reference to any particular programming language. It will be understood that a variety of programming languages ​​may be used to implement the teachings of the disclosure as described herein.

[0085] Exemplary embodiments The following text presents numbered embodiments in a claim-like format, and it is understood that these embodiments may be presented as claims in one or more future applications, such as one or more continuation or divisional applications. Although separate embodiments are described in detail below, it will be appreciated that these embodiments may be combined or modified in part or in whole.

[0086] Exemplary embodiment 1 1. A method for processing data, comprising the steps of: receiving image data represented in a light field format, the image data including image data for different views of a plurality of pixels in an image; receiving angular range metadata for the image; receiving a selection of a desired viewpoint associated with the image; determining one or more views at each pixel of the plurality of pixels using a view function that determines one or more views, the view function being based on or having inputs including spatial coordinates of each pixel of the plurality of pixels in the image, the desired viewpoint, the received angular range metadata, and (1) a distance between the desired viewpoint and a display or (2) a desired zoom / magnification amount; The method includes:

[0087] Exemplary embodiment 2 The method of exemplary embodiment 1, wherein the image is a 4D light field image pre-rendered as volumetric content, a selection of the desired viewpoint is received from a user to view the image at the desired viewpoint, and the view function is an angle view function including a horizontal angle view function and a vertical angle view function, the horizontal angle view function having inputs including a distance between the desired viewpoint and the display, a horizontal spatial coordinate of a pixel, and a horizontal component of the desired viewpoint, and the vertical angle view function has inputs including a distance between the desired viewpoint and the display, a vertical spatial coordinate of a pixel, and a vertical component of the desired viewpoint.

[0088] Exemplary embodiment 3 The method of exemplary embodiment 1, wherein the view function is defined with respect to a reference plane at a reference distance from the display, and the view function enables the same view to be determined for all pixels in the image for any one viewpoint within the reference plane.

[0089] Exemplary embodiment 4 The method of exemplary embodiment 3, wherein for viewpoints outside the reference plane, the view function determines different views for different pixels in the image, and the desired viewpoint is selected based on an estimated viewer position or a user selected position.

[0090] Exemplary embodiment 5 The method comprises: rendering the image based on the determined one or more views; displaying the rendered image in the determined view; 2. The method of exemplary embodiment 1, further comprising:

[0091] Exemplary embodiment 6 The method of example embodiment 1, wherein the image is a 4D light field image previously rendered as volume content stored in either a) a decoded planar format as tiles, where each tile is one of the possible views, or b) an interleaved format.

[0092] Exemplary Embodiment 7 The method comprises: receiving color volume mapping metadata; applying color volume mapping based on the determined one or more views and the color volume mapping metadata; The method of any one of exemplary embodiments 1 to 6, further comprising:

[0093] Exemplary Embodiment 8 The method of exemplary embodiment 7, wherein the color volume mapping metadata is adjusted based on the desired viewpoint and angular offset metadata that specifies one or more adjustments to the color volume mapping metadata based on or as a function of the desired viewpoint.

[0094] Exemplary embodiment 9 The method comprises: interpolating the angular offset metadata based on the desired viewpoint; 9. The method of exemplary embodiment 8, further comprising:

[0095] Exemplary embodiment 10 9. The method of exemplary embodiment 8, wherein the color volume mapping metadata varies across a number of different images on a scene-by-scene or image-by-image basis.

[0096] Exemplary embodiment 11 The method comprises: interpolating the determined one or more views at the desired viewpoint from a set of closest available views in the image data; 2. The method of exemplary embodiment 1, further comprising:

[0097] Exemplary embodiment 12 The method of exemplary embodiment 11, wherein the interpolation uses bilinear interpolation from a dense light field image with sufficiently high angular density that the difference between adjacent views is imperceptible or barely perceptible to a viewer at a reference viewing plane.

[0098] Exemplary embodiment 13 The method of exemplary embodiment 1, further comprising restricting the possible desired viewpoints to a valid viewing zone, the valid viewing zone of an image being defined by angular range metadata (of the image) that specifies the range of angles at which the image can be accurately viewed.

[0099] Exemplary embodiment 14 The limiting step comprises: (a) hard clamping invalid viewpoints to viewpoints within the valid viewing zone; or (b) soft-clamping the invalid viewpoints to viewpoints within the valid viewing zone; Includes one of the following: The method of exemplary embodiment 13, wherein a hard clamp always selects points on the boundary of the valid viewing zone, and a soft clamp selects a set of points near but not on the boundary of the valid viewing zone.

[0100] Exemplary embodiment 15 The method includes the steps of receiving metadata including offset metadata associated with luminance metadata specifying a statistical (e.g., average or median) luminance value of the image, the offset metadata specifying an adjustment of the luminance metadata as a function of the viewpoint; 15. The method of exemplary embodiment 14, further comprising:

[0101] Exemplary embodiment 16. A data processing system, the data processing system being programmed or configured to execute the method according to any one of the exemplary embodiments 1-15.

[0102] Exemplary Embodiment 17 A non-transitory machine-readable medium storing executable program instructions that, when executed by a data processing system, cause the data processing system to perform the method according to any one of exemplary embodiments 1-15.

[0103] Exemplary Embodiment 18 The method comprises: receiving color volume mapping metadata; applying color volume mapping based on the determined one or more views and the color volume mapping metadata; The method according to any one of exemplary embodiments 1 to 6, 8 to 14, further comprising:

[0104] Exemplary Embodiment 19 The method comprises: receiving metadata including offset metadata associated with luminance metadata specifying a statistical luminance value of the image, the offset metadata being used to adjust the luminance metadata based on the viewpoint; The method of any one of exemplary embodiments 1 to 14, further comprising:

[0105] In the foregoing specification, specific exemplary embodiments have been described. It will be apparent that various changes and modifications may be made therein without departing from the broad spirit and scope of the following claims. The specification and drawings are, therefore, to be regarded in an illustrative rather than a restrictive sense.

Claims

1. 1. A method for processing data, comprising the steps of: receiving image data represented in a light field format, the image data including image data for different views of a plurality of pixels in an image; receiving angular range metadata for the image, the angular range metadata specifying a range of angles over which the image can be accurately viewed; receiving a selection of a desired viewpoint associated with the image; determining one or more views at each pixel of the plurality of pixels using a view function that determines one or more views, the view function having inputs including spatial coordinates of each pixel of the plurality of pixels in the image, the angular range metadata, the desired viewpoint, and a distance between the desired viewpoint and a display; Including, A method according to claim 1, wherein the view function is defined relative to a reference plane at a reference distance from the display, the view function determining the same view for all pixels in the image for any one viewpoint within the reference plane.

2. 2. The method of claim 1 , wherein the view functions are angular view functions including a horizontal angle view function and a vertical angle view function, the horizontal angle view function having inputs including a distance between the desired viewpoint and the display, a horizontal spatial coordinate of a pixel, and a horizontal component of the desired viewpoint, and the vertical angle view function having inputs including a distance between the desired viewpoint and the display, a vertical spatial coordinate of a pixel, and a vertical component of the desired viewpoint.

3. 2. The method of claim 1, wherein for viewpoints outside the reference plane, the view function determines different views for different pixels in the image, and the desired viewpoint is selected based on an estimated viewer position or a user selection of the desired viewpoint.

4. The method comprises: rendering the image based on the determined one or more views; displaying the rendered image in the determined view; The method according to any one of claims 1 to 3, further comprising:

5. 5. The method of claim 1, wherein the image is a previously rendered 4D light field image as volumetric content stored in either a) a decoded planar format as tiles, each tile being one of the possible views, or b) an interleaved format.

6. The method comprises: receiving color volume mapping metadata; applying color volume mapping based on the determined one or more views and the color volume mapping metadata; The method of any one of claims 1 to 5, further comprising:

7. 7. The method of claim 6, wherein the color volume mapping metadata is adjusted based on the desired viewpoint and angular offset metadata that specifies one or more adjustments to the color volume mapping metadata based on or as a function of the desired viewpoint.

8. The method comprises: interpolating the angular offset metadata based on the desired viewpoint; The method of claim 7 further comprising:

9. A method according to any one of claims 6 to 8, wherein the colour volume mapping metadata varies across different images on a scene-by-scene or image-by-image basis.

10. The method comprises: Interpolating the determined one or more views at the desired viewpoint from a set of closest available views in the image data; The method of any one of claims 1 to 9, further comprising:

11. The method of claim 10 , wherein the step of interpolating uses bilinear interpolation from a dense light field image with sufficient angular density.

12. The method comprises: restricting the viewpoint to a valid viewing zone, the valid viewing zone of the image being defined by said angular range metadata; The method of any one of claims 1 to 11, further comprising:

13. The limiting step comprises: (a) hard clamping invalid viewpoints to viewpoints within the valid viewing zone; or (b) soft-clamping the invalid viewpoints to viewpoints within the valid viewing zone; Any one of the following: The method of claim 12 , wherein a hard clamp always selects points on a boundary of the valid viewing zone, and a soft clamp selects a set of points near but not on the boundary of the valid viewing zone.

14. receiving metadata including offset metadata associated with luminance metadata specifying a statistical luminance value of the image, the offset metadata specifying an adjustment of the luminance metadata as a function of the viewpoint; The method of any one of claims 1 to 13, further comprising:

15. The method of any one of claims 1 to 14, wherein the image is a 4D light field image previously rendered as volumetric content, and a selection of the desired viewpoint is received from a user for viewing the image at the desired viewpoint.

16. A data processing system programmed or configured to carry out a method according to any one of claims 1 to 15.

17. A non-transitory machine-readable medium storing executable program instructions which, when executed by a data processing system, cause the data processing system to perform the method of any one of claims 1 to 14.

Images