Method, medium and apparatus for prediction of light field encoding and decoding

Abstract

A method, medium, and apparatus for prediction of light field encoding and decoding are disclosed. There is provided a method for predicting a component of a current pixel as part of an encoding or decoding method, the method comprising: selecting a first ray corresponding to the current pixel from two planar parameterization describing a ray field corresponding to a pixel of a sensor, wherein the current pixel belongs to a current sub-aperture image in a matrix of sub-aperture images captured by a sensor of the plenoptic camera, and wherein the distance d from the microlens array of the camera to the sensor is equal to the focal length f of the microlenses of the microlens array; determining an object point located at the depth Z of the first light; determining a second ray originating from the object point and passing through a single point corresponding to a waist of a ray bundle, the ray bundle corresponding to pixels belonging to the reference sub-aperture image; and determining the position # imgabs0 # of the point of intersection of the second light with the sensor on the sensor

Description

[0001] This application is a divisional application of the application with application number 201980045344.5, application date May 29, 2019, national entry date January 5, 2021, and invention name “Prediction of light field encoding and decoding”. Technical Field

[0002] The present disclosure generally relates to the field of Light Field (LF) video coding.

[0003] More specifically, the present disclosure relates to a method for predicting components of pixels belonging to a sub-aperture image captured by a Type I plenoptic camera.

[0004] The present disclosure may be of interest, for example, in the field of LF compression and / or decompression. Background Art

[0005] On the one hand, View Synthesis Prediction (VSP) has been standardized as part of the 3D-HEVC extension for multi-view video coding. More specifically, VSP includes warping texture data from a reference view to a current view using depth information so that a predictor for the current view can be generated (e.g., see S. Yea and A. Vetro, "View Synthesis Prediction for Multiview video coding", Signal Processing: Image Communication, Vol. 24, No. 1-2, pp. 89-100, January 2009).

[0006] On the other hand, LF data includes:

[0007] - multi-view video, i.e. in video where images belonging to an image matrix are captured simultaneously from multiple camera angles; or

[0008] - Plenoptic video, also called lenslet-based video, ie video in which the images belonging to the image matrix are sub-aperture images that are captured simultaneously from a single camera using a lens matrix in addition to the main lens system.

[0009] VSP assumes, among other things, a pinhole model of the camera used to capture the LF video. This prevents achieving high performance in LF video codecs, especially for LF captured by plenoptic cameras where the extension of the aperture cannot be ignored.

[0010] Therefore, there is a need for an improved method for predicting the components (e.g., chrominance or luminance) of pixels of a sub-aperture image in a LF captured by a plenoptic camera. Summary of the invention

[0011] The present disclosure relates to a method for predicting a component of a current pixel belonging to a current sub-aperture image in a matrix of sub-aperture images captured by a sensor of a type I plenoptic camera. The method comprises:

[0012] - Determine the position on the sensor based on at least the following conditions:

[0013] - the distance D from the exit pupil of the camera's main lens to the camera's microlens array;

[0014] - focal length F of the main lens;

[0015] - the focal length f of the microlenses of the microlens array; and

[0016] a set of parameters of a camera model which allows the derivation of a two-plane parameterization describing the light field corresponding to the pixels of the sensor;

[0017] - Predicting the components of the current pixel based on at least one reference pixel on the sensor that belongs to a reference sub-aperture image in the matrix and that is located in the vicinity of the location.

[0018] Another aspect of the present disclosure relates to a device for predicting a component of a current pixel of a current sub-aperture image in a matrix of sub-aperture images captured by a sensor of a type I plenoptic camera. Such a device includes a processor configured to:

[0019] - Determine the position on the sensor based on at least the following conditions:

[0020] - the distance D from the exit pupil of the camera's main lens (100l) to the camera's microlens array (100mla);

[0021] - focal length F of the main lens;

[0022] - the focal length f of the microlenses of the microlens array; and

[0023] a set of parameters of a camera model which allows the derivation of a two-plane parameterization describing the light field corresponding to the pixels of the sensor;

[0024] - Predicting the components of the current pixel based on at least one reference pixel on the sensor that belongs to a reference sub-aperture image in the matrix and that is located in the vicinity of the location.

[0025] Another aspect of the present disclosure relates to a bitstream carrying a light field captured by a sensor of a Type I plenoptic camera. Such a bitstream includes metadata, the metadata including:

[0026] - the distance D from the exit pupil of the camera's main lens to the camera's microlens array;

[0027] - focal length F of the main lens;

[0028] - the focal length f of the microlenses of the microlens array; and

[0029] A set of parameters of a model of the camera which allows the derivation of a two-plane parameterization describing the light field corresponding to the pixels of the sensor.

[0030] Furthermore, the present disclosure relates to a non-transitory computer-readable medium including a computer program product recorded thereon and capable of being executed by a processor, the computer program product including program code instructions including program code instructions for implementing the previously described method for predicting a component of a current pixel belonging to a current sub-aperture image in a matrix of sub-aperture images captured by a sensor of a type I plenoptic camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] - Figure 1 illustrates an encoder and a decoder for respectively encoding and decoding a current pixel of a sub-aperture image captured by a Type I plenoptic camera in accordance with at least one embodiment;

[0032] - Figure 2a and 2b The diagram shows Figure 1 The relationship between the micro-image captured by the sensor of the camera and the corresponding sub-aperture image;

[0033] - Figure 3a and 3b Pictured Figure 1 Optical properties of type I plenoptic camera;

[0034] - Figure 4 The present invention illustrates a method for Figure 1 Flowchart of a method for predicting components of a current pixel implemented in an encoder and a decoder of;

[0035] - Figure 5a Pictured Figure 4 The first and second light rays involved in the method;

[0036] - Figure 5b The diagram shows the Figure 4 The method is Figure 1 The position determined on the sensor of the Type I plenoptic camera;

[0037] - Figure 6 The diagram can be used to implement Figure 4 An exemplary system of the method. DETAILED DESCRIPTION

[0038] Throughout the drawings, the same reference numerals denote similar elements and steps.

[0039] A method for predicting a component (e.g., chrominance or luminance) of a current pixel belonging to a current sub-aperture image in a matrix of sub-aperture images captured by a sensor of a type I plenoptic camera is disclosed. The method comprises the step of determining a position on the sensor based on parameters of the camera. Based on the determined position, the component of the current pixel is predicted based on at least one reference pixel on the sensor belonging to a reference sub-aperture image in the matrix and located near the determined position.

[0040] The prediction depends only on the parameters of the camera, e.g. the derivation can be performed both on the encoding side and on the decoding side based on a small number of parameters to be shared between devices.

[0041] Figure 1 An encoder 120 and a decoder 130 are depicted for respectively encoding and decoding a current pixel of a sub-aperture image of an object 110 captured by a type-I plenoptic camera 100 in accordance with at least one embodiment.

[0042] like Figure 1 As shown in FIG. 1 , a type I plenoptic camera 100 includes a main lens 1001, a microlens array 100m1a, and a sensor 100s. A type I plenoptic camera (also referred to as a plenoptic camera 1.0) is a plenoptic camera in which the distance d from the microlens array (100m1a) to the sensor (100s) is equal to the focal length f of the microlens, i.e., f=d, as Figure 3a , 3b and further shown in 5a.

[0043] The three axes define the coordinate system (CS) of the camera. More specifically, when the camera 100 is pointed at the object 110 being captured:

[0044] - The z-axis of the CS is defined as the camera's entrance pupil originating from the scene with positive z (i.e., the optical image of the aperture stop when viewed from the front of the camera);

[0045] - the x-axis of the CS is defined as originating from the camera's entrance pupil, extending horizontally and pointing to the right; and

[0046] The y-axis of the -CS is defined as originating from the entrance pupil of the camera, extending vertically and pointing toward the bottom.

[0047] According to these definitions, the entrance pupil lies in the plane defined by z=0 in CS.

[0048] When capturing an object 110 at a given instant, the type I plenoptic camera 100 delivers a matrix 200sai of sub-aperture images belonging to the LF ( Figure 2b ).exist Figure 2b In the illustrated embodiment, the matrix 200sai of sub-aperture images includes four sub-aperture images, each sub-aperture image includes six pixels.

[0049] Each subaperture image consists of a matrix 200mi ( Figure 2a ), one micro-image is located in front of a corresponding micro-lens of the micro-lens array 100m1a. More specifically:

[0050] - the upper left pixel of the sub-aperture image 200sai1 corresponds to the upper left pixel 200mi1tl of the micro-image 200mi1;

[0051] - the top center pixel of the sub-aperture image 200sai1 corresponds to the top left pixel 200mi2tl of the micro-image 200mi2;

[0052] - the upper right pixel of the sub-aperture image 200sai1 corresponds to the upper left pixel 200mi3tl of the micro-image 200mi3;

[0053] - the lower left pixel of the sub-aperture image 200sai1 corresponds to the upper left pixel 200mi4tl of the micro-image 200mi4;

[0054] - the bottom center pixel of the sub-aperture image 200sai1 corresponds to the top left pixel 200mi5tl of the micro-image 200mi5; and

[0055] -The lower right pixel of the sub-aperture image 200sai1 corresponds to the upper left pixel 200mi6tl of the micro-image 200mi6.

[0056] The same is true for other sub-aperture images of the matrix 200sai. For example:

[0057] - the upper left pixel of the sub-aperture image 200sai2 corresponds to the upper right pixel 200mi1tr of the micro-image 200mi1;

[0058] - the top center pixel of the sub-aperture image 200sai2 corresponds to the top right pixel 200mi2tr of the micro-image 200mi2;

[0059] -etc.

[0060] Each microimage therefore has the same number of pixels, ie in the present embodiment four pixels, so as to be able to reconstruct four subaperture images of the matrix 200sai each having six pixels.

[0061] The matrix 200sai of sub-aperture images is delivered to the encoder 120. The encoder comprises a device 150 configured to Figure 4 The method shown is used to predict the components of the current pixel.

[0062] A prediction residual may be derived between the prediction of the components of the pixels of a given sub-aperture image belonging to the matrix 200sai and a significant value, the prediction being based on a reference sub-aperture image also belonging to the matrix 200sai. The prediction residual may then be encoded into a bitstream.

[0063] The encoding of the successive matrices of sub-aperture images captured by the camera 100 delivers a bitstream carrying the LF captured by the type I plenoptic camera 100. This bitstream may be transmitted to a decoder 130 which also comprises the same device 150 as the encoder 120.

[0064] The decoder 130 may predict the values ​​of the components of the pixels of a given sub-aperture image belonging to the matrix 200sai based at least on the reference sub-aperture image belonging to the matrix 200sai and on the parameters of the type I plenoptic camera 100 for capturing the LF.

[0065] It is assumed that the decoder 130, in particular the device 150, knows the parameters of the type I plenoptic camera 100. For example, the bitstream carrying the LF captured by the sensor 100s of the type I plenoptic camera 100 and transmitted by the encoder 120 to the decoder 130 includes metadata with said parameters. Figure 4 As discussed, in at least one embodiment, such parameters include the distance D from the exit pupil of the main lens 100l to the microlens array 100mla, the focal length F of the main lens 100l, the focal length f of the microlenses of the microlens array 100mla, and a parameter set of a model of the camera 100 (e.g., the pattern, position, and orientation of the microlens array), which allows the derivation of a two-plane parameterization describing the light field corresponding to the pixels of the sensor. The set of rays in the light field can be parameterized in a variety of ways. Among them, the most common is the two-plane parameterization.

[0066] Figure 3a and 3b The diagram illustrates the property of a Type I plenoptic camera 100 according to which all waists of light ray bundles corresponding to pixels belonging to the same sub-aperture image arrive at the same point in the object focal plane of the camera 100 .

[0067] More specifically, for any type I plenoptic camera, the sensor 100s is located in the image focal plane (i.e., in the plane of equation z=-Df in CS) of the microlens array 100mla (i.e., each pixel of the sensor is at a distance f of the microlens array 100mla). Therefore, the bundle of rays originating from a given pixel on the sensor and covering a given microlens of the microlens array 100mla produces parallel rays between the given microlens under consideration and the main lens 100l. The parallel rays are further collimated by the main lens 100l so that they all converge toward a given focus in the object focal plane of the main lens 100l (i.e., in the plane of equation z=F in CS).

[0068] For example, Figure 3a As shown in FIG. 1 , the bundle of rays corresponding to the upper left pixel 200mi1tl of the sub-aperture image 200sai1 (and therefore the same as described above with respect to Figure 2a and 2b The bundle of rays (corresponding to the upper left pixel 200mi1t1 of the micro-image 200mi1 in question) converges towards a single point 300sai1 in the object focal plane of the main lens 1001 (thus corresponding to the waist of the bundle of rays).

[0069] Furthermore, due to this particular arrangement of the optical system of the type I plenoptic camera, it occurs that all light bundles corresponding to pixels of the same sub-aperture image converge toward the same single point in the object focal plane of the main lens 1001. In other words, all light bundles corresponding to pixels of the same sub-aperture image share a common waist located in the object focal plane of the main lens 1001.

[0070] For example, it corresponds to the top center pixel 200mi2tl of the sub-aperture image 200sai1 (and thus corresponds to the top center pixel 200mi2tl of the sub-aperture image 200sai1). Figure 2a and 2b The light bundles corresponding to the upper left pixel 200mi2tl of the micro-image 200mi2 in question also converge toward the point 300sai1, as do the light bundles corresponding to the upper left pixel 200mi1tl. Therefore, the point 300sai1 in the object focal plane of the main lens 1001 is the common waist of the light bundles corresponding to the pixels belonging to the sub-aperture image 200sai1.

[0071] This is true for any pixel of the sub-aperture image of the matrix 200sai. For example, it corresponds to the top center pixel 200mi2tr of the sub-aperture image 200sai2 (and thus corresponds to the pixel 200mi2tr of the matrix 200sai2). Figure 2a and 2b The light beam corresponding to the upper right pixel 200mi2tr of the micro-image 200mi2 in question also converges toward the point 300sai2, as shown in FIG. Figure 3bTherefore, the point 300sai2 in the object focal plane of the main lens 1001 is the common waist of the light bundles corresponding to the pixels belonging to the sub-aperture image 200sai2.

[0072] Thus, a common waist of bundles of rays corresponding to pixels belonging to the same sub-aperture image may be uniquely defined for each sub-aperture image of the matrix 200sai of sub-aperture images.

[0073] Figure 4 1 shows a method for predicting the components of a current pixel according to at least one embodiment. Such a method may be implemented, for example, in the encoder 120 and / or the decoder 130. By referring to Figure 5a and 5b The examples discussed here describe the corresponding steps in detail.

[0074] As an example, consider that the current pixel under consideration is the pixel 200mi2tr belonging to the current sub-aperture image, which is assumed to be sub-aperture image 200sai2.

[0075] In step S400 , the position 500 is determined on the sensor 100 s based on the following parameters of the camera 100 :

[0076] - the distance D from the exit pupil of the main lens 1001 of the camera 100 to the microlens array 100m1a of the camera 100;

[0077] - focal length F of the main lens 1001;

[0078] - the focal length f of the microlenses of the microlens array 100mla; and

[0079] A set of parameters of the model of the camera 100 (eg the pattern, position and orientation of the microlens array) which allows the derivation of a two-plane parameterization describing the light field corresponding to the pixels of the sensor 100s.

[0080] More specifically, the expected position 500 is close to one or more reference pixels on the sensor 100s that belong to a reference sub-aperture image, which in this example is assumed to be sub-aperture image 200sai1, and that captures the same object point as the current pixel.

[0081] To do this, use the above Figure 3a and 3b The property in question depends on which it is possible to define uniquely for each sub-aperture image of the matrix 200sai of sub-aperture images a common waist of the bundles of rays corresponding to pixels belonging to the same sub-aperture image.

[0082] More specifically, if Figure 5a As shown in , an object point 550op located at a depth Z along a first ray 510 corresponding to a current pixel 200mi2tr is determined based on a parameter set of a model of the camera 100. Here, the depth Z is the depth associated with the current pixel 200mi2tr under consideration. For example, the depth Z is determined by processing a current sub-aperture image 200sai2 to which the current pixel 200mi2tr belongs. In that case, the object point belongs to an object 110 captured by the camera 100. In another variant, the depth Z is captured by a dedicated depth sensor of the camera 100 simultaneously with capturing the current sub-aperture image 200sai2. In still other variants, the depth Z is provided as a default value.

[0083] The position 500 is determined as the intersection of the sensor 100s with a second ray 520 originating from the object point 550op and passing through the common waist 300sai1 of the bundle of rays corresponding to the pixels belonging to the reference sub-aperture image 200sai1. Due to this restriction of passing through the common waist 300sai1, it can be expected that the second ray 520 matches the valid ray that reaches the reference pixel belonging to the reference sub-aperture image 200sai1 capturing the same object point 550op as the current pixel 200mi2tr under consideration. According to this, it can be expected that the position 500 coincides with a reference pixel or is at least close to (one or more) reference pixels belonging to the reference sub-aperture image 200sai1. Therefore, it makes sense to predict one or more components of the current pixel 200mi2tr based on (one or more) components of (one or more) reference pixels on the sensor 100s belonging to the reference sub-aperture image 200sai1 and located near the position 500.

[0084] return Figure 4 To determine the theoretical position 500, in step S400a the coordinates of the common waist 300sai2 of the light ray bundles corresponding to the pixels belonging to the current sub-aperture image 200sai2 are determined in the object focal plane of the main lens 1001. In fact, as mentioned above Figure 3a and 3b As discussed above, such a common waist 300sai2 can be uniquely determined in the object focal plane of the camera 100 (i.e., the plane of the equation z=F in CS). In the same way, the coordinates of the common waist 300sai1 of the bundle of rays corresponding to the pixels belonging to the reference sub-aperture image 200sai1 are determined in the object focal plane of the camera 100.

[0085] In the embodiment considered, the two-plane parameterization includes I0[i,j,u,v] and I1[i,j,u,v], which define the intersection points of the main ray with planes z=0 and z=1 in CS respectively, and the main ray corresponds to the pixel at position (u,v) in the sub-aperture image with index (i,j) in the matrix 200sai of the sub-aperture image.

[0086] In that case, the coordinates and Determined as:

[0087]

[0088] and

[0089]

[0090] in:

[0091] -F is the focal length of the main lens 100l;

[0092] -(i c ,j c ) is the index of the current sub-aperture image 200sai2 in the matrix 200sai of sub-aperture images;

[0093] -(u ca ,v ca ) are the coordinates of any pixel in the current sub-aperture image 200sai2;

[0094] -(i r ,j r ) is the index of the reference sub-aperture image 200sai1 in the matrix 200sai of sub-aperture images; and

[0095] -(u ra ,v ra ) are the coordinates of any pixel in the reference sub-aperture image 200sai1.

[0096] In other embodiments, the two-plane parameterization includes I respectively defining the intersection points of the principal ray with other planes (eg, z=z0 and z=z1) in CS except for planes z=0 and z=1. z0 [i,j,u,v] and I z1 [i,j,u,v]. In that case, the linear interpolation defined by (Equation 1a) and (Equation 1b) can be rewritten as:

[0097]

[0098] and

[0099]

[0100] In still other embodiments, a two-plane parameterization defines intersections of reference rays other than the principal ray with two different planes.

[0101] In step S400b, the first coordinates of the intersection point of the first light ray 510 corresponding to the current pixel 200mi2tr and the given plane are determined.

[0102] In this embodiment, the given plane is the plane z=0 in CS, and the first ray 510 is the principal ray corresponding to the current pixel 200mi2tr on the sensor 100s.

[0103] In that case, the first coordinate Determined as:

[0104]

[0105] in:

[0106] -(i c ,j c ) is the index of the current sub-aperture image 200sai2 in the matrix 200sai of sub-aperture images; and

[0107] -(u c ,v c ) are the coordinates of the current pixel 200mi2tr in the current sub-aperture image 200sai2.

[0108] In other embodiments, the given plane is the plane z=z0 in CS, and the parameterization I z0 [i,j,u,v] instead of I0[i,j,u,v] to rewrite (Equation 2).

[0109] In still other embodiments, a two-plane parameterization defines intersections of reference rays other than the principal ray with two different planes.

[0110] In step S400c, a second coordinate of the intersection of a second ray 520 with the same given plane as used to define the first coordinate is calculated, the second ray 520 coming from an object point 550op at a depth Z along the first ray 510 and passing through a common waist 300sai1 of ray bundles corresponding to pixels belonging to the reference sub-aperture image 200sai1.

[0111] In this embodiment, the given plane considered is the plane z=0 in CS, and the first ray 510 is the main ray corresponding to the current pixel 200mi2tr on the sensor 100s, and the coordinates of the object point 550op located at the depth Z along the first ray 510 in CS are It can be expressed as the coordinates of two points belonging to the first ray 510 (ie, two points corresponding to the intersections of the first ray 510 with the planes z=0 and z=F, respectively) and Linear interpolation of :

[0112]

[0113] Equivalently, the coordinates It can be expressed as the second coordinate and the coordinates of the two points belonging to the second ray 520 Center of gravity:

[0114]

[0115] By passing the coordinates By equating the two previous expressions, we can find the second coordinate Can be determined as:

[0116]

[0117] in:

[0118] - F is the focal length of the main lens 1001; and

[0119] -Z is the depth associated with the current pixel 200mi2tr.

[0120] In other embodiments, where the given plane is the plane z=z0 in CS, and where (Equation 2) can be parameterized using z0 [i,j,u,v] instead of I0[i,j,u,v], (Equation 3) can be rewritten as:

[0121]

[0122] in:

[0123] - is the first coordinate;

[0124] - is the second coordinate;

[0125] - F is the focal length of the main lens 1001; and

[0126] -Z is the depth associated with the current pixel 200mi2tr.

[0127] However, selecting a given plane as plane z=0 in CS allows for simpler derivation and fewer parameters to be shared between the encoder 120 and the decoder 130 .

[0128] Furthermore, choosing the given plane as the plane z=0 allows to calculate the coordinates of the common waist 300sai1 of the bundles of rays corresponding to the pixels belonging to the reference sub-aperture image 200sai1 based on And based on the second coordinate The coordinates of the position 500 in the plane z=-Df of the sensor are derived in a direct manner In practice, assuming that the deviation of the second ray 520 by the main lens 1101 and the microlenses of the array 100m1a remains small (assuming that the second ray 520 passes directly through the center of the lens or microlens under consideration without error), it can be written as follows:

[0129]

[0130] where θ x is the angle between the projection of the second light ray 520 in the plane (x, z) of CS and the z-axis, and θ y is the angle between the projection of the second light ray 520 in the plane (y, z) of CS and the z-axis.

[0131] In addition, it is assumed that the focal length F of the main lens 100l is much larger than the coordinate Any component of , then the tangent function can be approximated as:

[0132]

[0133] So finally:

[0134]

[0135] After the coordinates of the position 500 on the sensor 100s are determined, in step S410, at least one component of the current pixel 200mi2tr is predicted based on (one or more) components of one or more reference pixels on the sensor 100s that belong to the reference sub-aperture image 200sai1 and are located near the position 500.

[0136] To this end, in step S410a, at least two reference pixels on the sensor 100s that belong to the reference sub-aperture image 200sai1 in the matrix 200sai and are located near the position 500 are selected (eg, based on a Euclidean metric or the like).

[0137] In step S410b, the components of the at least two reference pixels are interpolated (e.g., linear or bilinear interpolation, bicubic interpolation, Lanczos interpolation, etc.) to deliver interpolated components. In that case, the predicted component of the current pixel 200mi2tr is a function of the interpolated component (e.g., proportional to it, etc.).

[0138] However, in other embodiments not shown, a single reference pixel on the sensor 100s is selected that belongs to the reference sub-aperture image 200sai1 and is located near the position 500. In that case, the components of the prediction of the current pixel 200mi2tr are directly derived from the components of the selected reference pixel (e.g., the values ​​of the components of the selected reference pixel are assigned to the current pixel 200mi2tr as the components of the prediction).

[0139] return Figure 4 In step S410b1, the coordinates of position 500 are For each considered reference pixel among the at least two reference pixels:

[0140] - in step S410b2, extracting the coefficients weighting the coordinates of the considered reference pixel in the centroid expression; and

[0141] - In step S410b3, the components of the considered reference pixel are weighted with the corresponding extracted coefficients.

[0142] Then, in step S410b4, a linear combination of the weighted components is performed to deliver the interpolated components.

[0143] For example, as an illustrative example, let us consider that the four reference pixels selected in the neighborhood of location 500 are pixels 200mi1tl, 200mi2tl, 200mi4tl, and 200mi5tl ( Figure 5b ). In that case, the coordinates of position 500 It can be expressed as a centroid expression:

[0144]

[0145] in:

[0146] - and

[0147] -

[0148] And among them, in CS:

[0149] -x l Represents the common abscissa of reference pixels 200mi1tl and 200mi4tl;

[0150] -x r Represents the common abscissa of reference pixels 200mi2tl and 200mi5tl;

[0151] -y trepresents the common ordinate of reference pixels 200mi1tl and 200mi2tl; and

[0152] -y b Represents the common vertical coordinate of reference pixels 200mi4tl and 200mi5tl.

[0153] Applying steps S410b2, S410b3 and S410b4 to the above centroid expression results in the predicted components for the current pixel 200mi2tr The following expression:

[0154]

[0155] and:

[0156] -Y tl is the component of the reference pixel 200mi1tl;

[0157] -Y tr is the component of the reference pixel 200mi2tl;

[0158] -Y bl is the component of reference pixel 200mi4tl; and

[0159] -Y br It is the component of the reference pixel 200mi5tl.

[0160] The prediction method can thus be part of the encoding method and / or the decoding method. To this end, on the encoder side, the component Y of the current pixel is then compared in the bitstream with the predicted component Y. The residual between them is encoded.

[0161] At the decoder side, the residual is decoded from the bitstream and the predicted component of the current pixel is converted into Added to the decoded residual to reconstruct component Y.

[0162] Figure 6 A block diagram of an exemplary system in which various aspects and exemplary embodiments are implemented is illustrated. System 1000 is a specific way to implement encoder 120 and / or decoder 130, and can be implemented as a device including various components described below and configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. System 1000 can be communicatively coupled to other similar systems and coupled to displays via communication channels, such as Figure 6 As shown in and known to those skilled in the art, to implement the various aspects described in this document.

[0163] The system 1000 may include at least one processor 1010 configured to execute instructions loaded therein to implement various aspects described in this document. The processor 1010 may include embedded memory, input-output interfaces, and various other circuits known in the art. The system 1000 may include at least one memory 1020 (e.g., a volatile memory device, a non-volatile memory device). The system 1000 may include a storage device 1040, which may include a non-volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, a disk drive, and / or an optical drive. As non-limiting examples, the storage device 1040 may include an internal storage device, an attached storage device, and / or a network accessible storage device. The system 1000 may include an encoder / decoder module 1030 configured to process data to provide encoded video or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 is configured to implement a prediction method according to at least one embodiment, and thus includes a processor configured to perform a prediction based on the prediction method. Figure 4 The method shown (according to any embodiment thereof) predicts a device 150 of the components of a current pixel. Such a device 150 may comprise its own processor and memory.

[0164] The encoder / decoder module 1030 represents a module(s) that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000, or may be incorporated into the processor 1010 as a combination of hardware and software, as known to those skilled in the art.

[0165] Program code to be loaded onto the processor 1010 to perform various aspects described in this document may be stored in the storage device 1040 and then loaded onto the memory 1020 for execution by the processor 1010. According to an exemplary embodiment, during execution of the processes described in this document, one or more of the processor(s) 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items including, but not limited to, input video, decoded video, bitstreams, equations, formulas, matrices, variables, operations, and operation logic.

[0166] The system 1000 may include a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data from the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel may be implemented in a wired and / or wireless medium. The various components of the system 1000 may all be connected or communicatively coupled together using various suitable connections, including, but not limited to, internal buses, wires, and printed circuit boards.

[0167] The exemplary embodiments may be implemented by computer software implemented by the processor 1010 or by hardware or by a combination of hardware and software. As a non-limiting example, the exemplary embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as, as a non-limiting example, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 1010 may be of any type suitable for the technical environment and, as a non-limiting example, may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0168] The embodiments and aspects described herein can be implemented, for example, in methods or processes, devices, software programs, data streams or signals. Even if only discussed in the context of a single form of embodiment (e.g., discussed only as a method), the embodiments of the features discussed can also be implemented in other forms (e.g., devices or programs). The device can be implemented, for example, with appropriate hardware, software and firmware. The method can be implemented in, for example, a device (such as, for example, a processor), and the processor generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit or a programmable logic device. The processor also includes a communication device, such as, for example, a computer, a mobile phone, a portable / personal digital assistant ("PDA") and other devices that contribute to information communication between end users.

[0169] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations throughout this document are not necessarily all referring to the same embodiment.

[0170] Additionally, this document may refer to "determining" various information. Determining information may include, for example, one or more of: estimating information, calculating information, predicting information, or retrieving information from a memory.

[0171] Additionally, this document may refer to "accessing" various information. Accessing information may include, for example, one or more of: receiving information, retrieving information (e.g., from a memory), storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0172] Additionally, this document may refer to "receiving" various information. Like "accessing," receiving is a broad term. Receiving information may include one or more of: for example, accessing information or retrieving information (e.g., from storage). Additionally, in one way or another, during operations such as, for example: storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is generally involved.

[0173] As will be apparent to one of ordinary skill in the art, embodiments may generate various signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing a method or data generated by one of the described embodiments. For example, a signal may be formatted to carry a bit stream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

[0174] According to one embodiment, a method for predicting a component of a current pixel belonging to a current sub-aperture image in a matrix of sub-aperture images captured by a sensor of a type I plenoptic camera is proposed. The method comprises:

[0175] - Determine the position on the sensor based on at least the following conditions:

[0176] - the distance D from the exit pupil of the camera's main lens to the camera's microlens array;

[0177] - focal length F of the main lens;

[0178] - the focal length f of the microlenses of the microlens array; and

[0179] a set of parameters of a camera model which allows the derivation of a two-plane parameterization describing the light field corresponding to the pixels of the sensor;

[0180] - Predicting the components of the current pixel based on at least one reference pixel on the sensor that belongs to a reference sub-aperture image in the matrix and that is located in the vicinity of the location.

[0181] Therefore, the present disclosure proposes a new and inventive solution for predicting a component (eg, chrominance or luminance) of a current pixel of a sub-aperture image in a light field captured by a Type-I plenoptic camera.

[0182] More specifically, such predictions rely inter alia on a two-plane parameterization which leads to improved prediction results compared to a simple pinhole model assumed, for example, in the VSP technique.

[0183] Furthermore, since only the camera's parameters are used, the derivation can be performed on both the encoding and decoding sides based on a small number of parameters to be shared between devices. Furthermore, the camera's physical parameters are static parameters and therefore do not need to be updated frequently, further minimizing the overhead cost on top of the light field data.

[0184] According to one embodiment, a device for predicting a component of a current pixel of a current sub-aperture image in a matrix of sub-aperture images captured by a sensor of a type I plenoptic camera is provided. Such a device comprises a processor configured to:

[0185] - Determine the position on the sensor based on at least the following conditions:

[0186] - the distance D from the exit pupil of the camera's main lens to the camera's microlens array;

[0187] - focal length F of the main lens;

[0188] - the focal length f of the microlenses of the microlens array; and

[0189] a set of parameters of a camera model which allows the derivation of a two-plane parameterization describing the light field corresponding to the pixels of the sensor;

[0190] - Predicting the components of the current pixel based on at least one reference pixel on the sensor that belongs to a reference sub-aperture image in the matrix and that is located in the vicinity of the location.

[0191] According to one embodiment, the position is determined as the intersection of a second ray with the sensor, the second ray originating from an object point located at depth Z along the first ray corresponding to the current pixel and passing through the common waist of the ray bundles corresponding to pixels belonging to the reference sub-aperture image in the matrix.

[0192] According to one embodiment, determining the location includes:

[0193] -determined in the object focal plane of the camera:

[0194] - the coordinates of the common waist of the bundles of rays corresponding to the pixels belonging to the current subaperture image

[0195] as well as

[0196] - the coordinates of the common waist of the bundles of rays corresponding to the pixels belonging to the reference subaperture image

[0197]

[0198] - determining first coordinates of the intersection of a first ray corresponding to a current pixel on the sensor with a given plane; and

[0199] -Based on at least the first coordinate, coordinate and coordinates to calculate a second coordinate of an intersection of a second ray with the given plane, the second ray corresponding to the position on the sensor and coming from a point of the object at the depth Z along the first ray;

[0200] and where the coordinates of the position in the plane of the sensor are is at least the second coordinate and the coordinate function.

[0201] Therefore, the property of a type I plenoptic camera allows a straightforward deduction based on optical geometry considerations, according to which bundles of rays corresponding to pixels belonging to the same sub-aperture image have a common waist in the object focal plane (i.e., the plane corresponding to z=F in the camera's z-coordinate system).

[0202] According to one embodiment, the two-plane parameterization includes defining the intersection points I0[i,j,u,v] and I1[i,j,u,v] of the principal ray with the planes z=0 and z=1 in the coordinate system of the camera, respectively, the principal ray corresponding to the pixel at the position (u,v) in the sub-aperture image indexed (i,j) in the matrix of the sub-aperture image. The object focal plane is the plane z=F in the coordinate system of the camera. Determine the coordinates and The following relationships are enforced respectively:

[0203] as well as

[0204]

[0205] in:

[0206] -F is the focal length of the main lens 100l;

[0207] -(i c , j c) is the index of the current sub-aperture image in the matrix of sub-aperture images;

[0208] -(u ca , v ca ) are the coordinates of any pixel in the current subaperture image;

[0209] -(i r , j r ) is the index of the reference sub-aperture image in the matrix of sub-aperture images; and

[0210] -(u ra , v ra ) are the coordinates of any pixel in the reference subaperture image.

[0211] In other words, the coordinates Independent of the coordinates of the considered pixel in the current subaperture image (u ca , v ca ). For the coordinates following the characteristics of the type I plenoptic camera mentioned above It also holds that, according to this, bundles of rays corresponding to pixels belonging to the same sub-aperture image have a common waist in the object focal plane.

[0212] According to one embodiment, the given plane is the plane z=0 in the coordinate system of the camera. Determining the first coordinate forces compliance with the relationship

[0213] in:

[0214] - is the first coordinate;

[0215] -(i c , j c ) is the index of the current sub-aperture image in the matrix of sub-aperture images; and

[0216] -(u c , v c ) are the coordinates of the current pixel in the current sub-aperture image.

[0217] According to one embodiment, the calculation of the second coordinate forces the equation

[0218] in:

[0219] - is the second coordinate;

[0220] - F is the focal length; and

[0221] -Z is the depth.

[0222] According to one embodiment, the plane of the sensor is the plane z=-Df in the coordinate system of the camera,

[0223] And the position determination is enforced to satisfy the equation in:

[0224] - are the coordinates of the position of the sensor in the plane;

[0225] -F is the focal length;

[0226] - f is the focal length of a microlens of the microlens array; and

[0227] -D is the distance from the exit pupil of the camera's main lens to the camera's microlens array.

[0228] According to one embodiment, predicting the components of the current pixel includes:

[0229] - selecting at least two reference pixels on the sensor that belong to the reference sub-aperture images in the matrix and that are located in close proximity to the location; and

[0230] - interpolating components of each of the at least two reference pixels to deliver the interpolated components.

[0231] The predicted components of the current pixel are functions of the components being interpolated (e.g., linear or bilinear interpolation, bicubic interpolation, Lanczos interpolation, etc.)

[0232] According to one embodiment, interpolation includes:

[0233] - expressing the position as the centroid of the coordinates of at least two reference pixels on the sensor; and for each considered reference pixel among the at least two reference pixels:

[0234] - extracting the coefficients that weight the coordinates of the considered reference pixel in the expression of the centroid;

[0235] - weighting the components of the considered reference pixel with the corresponding extracted coefficients; and - performing a linear combination of the weighted components to deliver the interpolated component.

[0236] Therefore, interpolation is both simple and robust.

[0237] According to one embodiment, a bitstream carrying a light field captured by a sensor of a type I plenoptic camera is provided. This bitstream includes metadata, including:

[0238] - the distance D from the exit pupil of the camera's main lens to the camera's microlens array;

[0239] - focal length F of the main lens;

[0240] - the focal length f of the microlenses of the microlens array; and

[0241] A set of parameters of a model of the camera which allows the derivation of a two-plane parameterization describing the light field corresponding to the pixels of the sensor.

[0242] According to one embodiment, a non-transitory computer-readable medium is proposed, which includes a computer program product recorded thereon and capable of being executed by a processor, the computer program product including program code instructions, which program code instructions include program code instructions for implementing the previously described method for predicting a component of a current pixel belonging to a current sub-aperture image in a matrix of sub-aperture images captured by a sensor of a type I plenoptic camera.

[0243] According to one embodiment, a coding method is proposed, which comprises a method as disclosed for predicting the component of a current pixel belonging to a current sub-aperture image in a matrix of sub-aperture images captured by a sensor of a type I plenoptic camera (in any of its embodiments).

[0244] According to one embodiment, a decoding method is proposed, which comprises the method as disclosed for predicting the component of a current pixel belonging to a current sub-aperture image in a matrix of sub-aperture images captured by a sensor of a type I plenoptic camera (in any of its embodiments).

[0245] According to one embodiment, an encoder is proposed comprising a device for predicting the component of a current pixel belonging to a current sub-aperture image in a matrix of sub-aperture images captured by a sensor of a type I plenoptic camera (in any of its embodiments).

[0246] According to one embodiment, a decoder is proposed comprising a device for predicting the component of a current pixel belonging to a current sub-aperture image in a matrix of sub-aperture images captured by a sensor of a type I plenoptic camera (in any of its embodiments).

Claims

1. A method, as part of an encoding or decoding method, for predicting a component of a current pixel, the method comprising: selecting a first ray corresponding to a current pixel from a two-plane parameterization describing a field of rays corresponding to a pixel of a sensor, wherein the current pixel belongs to a current sub-aperture image in a matrix of sub-aperture images captured by a sensor of a plenoptic camera, and wherein a distance d from a microlens array of the camera to the sensor is equal to a focal length f of a microlens of the microlens array; determining an object point located at a depth Z of the first ray; determining a second ray originating from the object point and passing through a single point corresponding to a waist of a ray bundle, the ray bundle corresponding to a pixel belonging to a reference sub-aperture image; as well as Determine the position of the intersection of the second light ray and the sensor on the sensor 2. The method of claim 1, wherein said determining said location comprises: Determine in the object focal plane of the camera: The coordinates of the point corresponding to the waist of the beam The bundle of rays corresponds to pixels belonging to the current sub-aperture image; as well as The coordinates of the point of the bundle of rays corresponding to the pixel belonging to the reference sub-aperture image determining first coordinates of an intersection of the first ray corresponding to a current pixel on the sensor and a given plane; as well as Based at least on the first coordinate, the coordinate and the coordinates to calculate second coordinates of an intersection of the second ray and the given plane, the second ray corresponding to the position on the sensor and coming from the object point at the depth Z along the first ray; where the coordinates of the position in the plane of the sensor are is at least the second coordinate and the coordinate function.

3. The method of claim 1 , wherein the two-plane parameterization comprises defining intersection points I0[i,j,u,v] and I1[i,j,u,v] of a principal ray with planes z=0 and z=1 in the coordinate system of the camera, respectively, the principal ray corresponding to a pixel at position (u,v) in the sub-aperture image indexed (i,j) in the matrix of the sub-aperture images, wherein the object focal plane is the plane z=F in the coordinate system of the camera, and wherein, Determining coordinates and The following relationships are enforced respectively: as well as in: F is the focal length of the main lens; (i c ,j c ) is the index of the current sub-aperture image in the matrix of sub-aperture images; (u ca ,v ca ) are the coordinates of any pixel in the current sub-aperture image; (i r ,j r ) is the index of the reference sub-aperture image in the matrix of sub-aperture images; and (u ra ,v ra ) are the coordinates of any pixel in the reference sub-aperture image.

4. The method according to claim 3, wherein the given plane is the plane z=0 in the coordinate system of the camera, and wherein determining the first coordinate forces compliance with the relationship: in: is the first coordinate; (i c ,j c ) is the index of the current sub-aperture image in the matrix of sub-aperture images; and (u c ,v c ) are the coordinates of the current pixel in the current sub-aperture image.

5. The method according to claim 4, wherein the calculation of the second coordinate is forced to comply with the equation: in: is the second coordinate; F is the focal length of the main lens; and Z is the depth.

6. The method according to claim 5, wherein the plane of the sensor is the plane z=-Df in the coordinate system of the camera, and wherein determining the position imposes compliance with the equation: in: are the coordinates of the position of the sensor in the plane; F is the focal length of the main lens; f is the focal length of the microlens of the microlens array; and D is the distance from the exit pupil of the main lens of the camera to the microlens array of the camera.

7. The method of claim 1 , wherein predicting the components of the current pixel comprises: selecting at least two reference pixels on the sensor that belong to the reference sub-aperture image in the matrix and are located in the neighborhood of the location; as well as Components of each of the at least two reference pixels are interpolated to deliver an interpolated component, the predicted component of the current pixel being a function of the interpolated component.

8. The method of claim 7, wherein the interpolation comprises: expressing the position as a centroid of coordinates of the at least two reference pixels on the sensor; as well as For each considered reference pixel among the at least two reference pixels: extracting the coefficients weighting the coordinates of the considered reference pixel in the centroid expression; weighting the components of the considered reference pixel by the corresponding extracted coefficients; as well as A linear combination of the weighted components is performed to deliver the interpolated components.

9. A coding method comprising the prediction method according to claim 1.

10. A decoding method, comprising the prediction method according to claim 1.

11. A non-transitory computer-readable storage medium comprising program code instructions for implementing the method of claim 1 when the program is executed on a computer or a processor.

12. A device, as part of an encoding or decoding device, for predicting a component of an electric current, wherein the device comprises a processor, the processor being configured to: selecting a first ray corresponding to a current pixel from a two-plane parameterization describing a field of rays corresponding to a pixel of the sensor, wherein the current pixel belongs to a current sub-aperture image in a matrix of sub-aperture images captured by a sensor of a plenoptic camera, and wherein a distance d from a microlens array of the camera to the sensor is equal to a focal length f of a microlens of the microlens array, determining the object point located at the depth Z of the first ray, determining a second ray originating from the object point and passing through a single point corresponding to the waist of a ray bundle, the ray bundle corresponding to a pixel belonging to the reference sub-aperture image, Determine the position of the intersection of the second light ray and the sensor on the sensor as well as The component of the current pixel is predicted based on at least one reference pixel on the sensor that belongs to a reference sub-aperture image in the matrix and is located in the vicinity of the position.

13. The apparatus of claim 12, wherein said determining said location comprises: Determine in the object focal plane of the camera: The coordinates of the point corresponding to the waist of the beam The bundle of rays corresponds to pixels belonging to the current sub-aperture image; as well as The coordinates of the point of the bundle of rays corresponding to the pixel belonging to the reference sub-aperture image determining first coordinates of an intersection of the first ray corresponding to a current pixel on the sensor and a given plane; as well as Based at least on the first coordinate, the coordinate and the coordinates to calculate second coordinates of an intersection of the second ray and the given plane, the second ray corresponding to the position on the sensor and coming from the object point at the depth Z along the first ray; where the coordinates of the position in the plane of the sensor are is at least the second coordinate and the coordinate function.

14. The apparatus of claim 12 , wherein the two-plane parameterization comprises defining intersection points I0[i,j,u,v] and I1[i,j,u,v] of a principal ray with planes z=0 and z=1 in the coordinate system of the camera, respectively, the principal ray corresponding to a pixel at position (u,v) in the sub-aperture image indexed (i,j) in the matrix of the sub-aperture images, wherein the object focal plane is a plane z=F in the coordinate system of the camera, and wherein, Determining coordinates and The following relationships are enforced respectively: as well as in: F is the focal length of the main lens (i c ,j c ) is the index of the current sub-aperture image in the matrix of sub-aperture images; (u ca ,v ca ) are the coordinates of any pixel in the current sub-aperture image; (i r ,j r ) is the index of the reference sub-aperture image in the matrix of sub-aperture images; and (u ra ,v ra ) are the coordinates of any pixel in the reference sub-aperture image.

15. The apparatus of claim 12, wherein predicting the components of the current pixel comprises: selecting at least two reference pixels on the sensor that belong to the reference sub-aperture image in the matrix and are located in the neighborhood of the location; as well as Components of each of the at least two reference pixels are interpolated to deliver an interpolated component, the predicted component of the current pixel being a function of the interpolated component.

16. The apparatus of claim 15, wherein the interpolation comprises: expressing the position as a centroid of coordinates of the at least two reference pixels on the sensor; And for each considered reference pixel among the at least two reference pixels: extracting the coefficients weighting the coordinates of the considered reference pixel in the centroid expression; weighting the components of the considered reference pixel by the corresponding extracted coefficients; as well as A linear combination of the weighted components is performed to deliver the interpolated components.

17. An encoding device comprising the prediction device according to claim 12.

Images