A device and method for generating a data representation of a pixel beam

Abstract

There are several types of plenoptic devices with their proprietary file formats. Currently, there are no standards supporting the acquisition and transmission of multi-dimensional information in order to have an exhaustive overview of the different parameters on which the light field depends. As a result, the acquired light field data for different cameras have a wide variety of formats. Therefore, the concept of pixel beams is introduced, which represents the volume occupied by a group of rays in the object space of the optical system of the camera. The method according to the invention makes it possible to provide data representing a set of pixel beams describing an unknown first optical system, since these data are obtained by imaging the set of pixel beams by a second optical system. Such data representing the set of pixel beams makes it possible to generate a parameterized output image from which post-processing is performed.

Description

[0001] This divisional application is a divisional application with application date of July 11, 2017, application number 201780054065.6, and invention name “A device and method for generating data representation of pixel light beams”. Technical Field

[0002] The present invention relates to the generation of data representing a light field and providing realism in rendered data. Background Art

[0003] The acquisition of four-dimensional or 4D light field data, which can be viewed as the sampling of a 4D light field (i.e., the recording of rays), is a busy research topic, as discussed in the article "Understanding Camera Trade-offs via Bayesian Analysis of Light Field Projections" published by Anat Levin et al. in the Proceedings of the ECCV 2008 Conference. Understanding camera trade-offs through a Bayesian analysis of light field projections )”.

[0004] Compared to classic two-dimensional or 2D images obtained from a camera, 4D light field data enables the user to take advantage of more post-processing features that enhance the rendering of the image and the interactivity with the user. For example, with 4D light field data, refocusing of the image can be performed with a freely selectable focus distance, which means that the position of the focal plane can be specified / selected a posteriori, as well as slightly changing the viewpoint in the scene of the image. In order to acquire 4D light field data, several techniques can be used. For example, a plenoptic camera is able to acquire 4D light field data. A plenoptic camera comprises a main lens, a microlens array comprising a plurality of microlenses arranged in a two-dimensional array, and an image sensor. Another way to acquire 4D light field data is to use a camera array comprising a lens array and an image sensor.

[0005] In the example of a plenoptic camera, a main lens receives light from objects in the object field of the main lens and passes the light through the image field of the main lens.

[0006] Finally, another way to collect 4D light fields is to use a conventional camera that is configured to capture a sequence of 2D images of the same scene at different focal planes. For example, J.-H. Park et al. published a paper titled “Capturing light fields using focal plane scanning and optical reconstruction using three-dimensional displays” in Optics Express, Vol. 22, No. 21, October 2014. Lightray field capture using focal plane sweeping and its optical reconstruction using 3D displays The technique described in “ )” can be used to collect 4D light field data with the aid of a conventional camera.

[0007] There are several ways to represent 4D light field data. In fact, in a July 2006 paper titled "Digital Light Field Photography" by Ren Ng, Digital Light Field Photography )", three different ways of representing 4D light field data are described. First, 4D light field data can be represented when recorded by a plenoptic camera through some microlens images. The 4D light field data in this representation are called raw images or raw 4D light field data. Second, the 4D light field data can be represented by a set of sub-aperture images when recorded by a plenoptic camera or a camera array. A sub-aperture image corresponds to an image of a scene captured from a viewpoint that is slightly different between two sub-aperture images. These sub-aperture images give information about the parallax and depth of the imaged scene. Third, the 4D light field data can be represented by a set of epipolar images, see for example the paper "Generation of an EPI representation of a 4D light field using a single-lens focusing plenoptic camera" by S. Wanner et al. in the conference proceedings of ISVC 2011 ( Generating EPI Representation of a4D Light Fields with a Single Lens Focused Plenoptic Camera )”.

[0008] Light field acquisition devices are extremely diverse. There are different types of light field cameras, such as plenoptic or camera arrays. Within each type there are many differences, such as different optical arrangements or microlenses of different focal lengths. Each camera has its own proprietary file format. Therefore, the acquired light field data for different cameras has a wide variety of formats.

[0009] In Ren Ng, "Fourier Slice Photography", ACM Transactions on Graphics (ToG), 24th edition, number 3, pages 735-744, 2005, the authors describe a theorem: In the Fourier domain, a photograph formed by a full aperture is a 2D slice in a 4D light field. This theorem is used to analyze the performance of digital refocusing, where photographs focused at different depths from a single light field are computed. Furthermore, the theorem is amenable to a Fourier domain algorithm for digital refocusing, where appropriate 2D slices of the light field are Fourier transformed and an inverse Fourier transform is performed. However, this solution requires a high computational load because it involves working in the Fourier domain.

[0010] The present invention has been devised in view of the foregoing. Summary of the invention

[0011] According to a first aspect of the invention, there is provided a computer-implemented method for generating data representing at least one volume in an object space of a first optical system, the volume being occupied by a group of rays passing through a pupil of the first optical system and conjugated to an image of at least one pixel of a sensor associated with the first optical system in the object space of the first optical system, the volume occupied by the group of rays being referred to as a pixel beam, comprising:

[0012] - calculating the conjugate of said at least one pixel beam through the second optical system, called the imaging pixel beam,

[0013] - calculating a parameter value representing the radiation transmitted by said imaging pixel beam, which is associated with at least one element of at least one grid defined on a projection surface located at a distance D from the second optical system and belonging to an intersection between said at least one imaging pixel beam and the projection surface.

[0014] This approach is able to create an image from a collection of pixel beams imaged in the focal volume of an optical system with their common definition, namely a set of image elements sampled on a regular grid.

[0015] In fact, the set of pixel beams imaged in the focal volume of the second optical system provides data representing the set of pixel beams, which describes an unknown first optical system, i.e., is independent of the camera embedded in the optical system, because these data are obtained by imaging the set of pixel beams by the second optical system.

[0016] However, this set of imaging pixel beams overlaps the focal volume depending on the structure and parameters of the second optical system, such as its orientation, position and focal length.

[0017] By calculating an image describing a set of pixel beams of a first optical system through a second optical system providing a representation of the object space of several optical systems embedded in various cameras in a dual and compact space, corresponding to the focal volumes of the optical systems and the radiance values ​​of the imaged pixel beams associated with the elements of a grid on a projection surface, it is possible to classify the set of imaged pixel beams in order to obtain an image that can be used in post-processing operations.

[0018] In an embodiment of the method according to the invention, the grid on the projection surface is a pixel grid.

[0019] When the grid is a pixel grid, the sets of pixel beams are classified based on the pixel grid and thus directly on the image.

[0020] In an embodiment of the method according to the invention, an oversampling grid is defined on the projection surface, the elements of which are smaller than the pixels of the pixel grid.

[0021] This enables the values ​​of the radiation pixel beams associated with the elements of the oversampled grid to be accumulated, thereby more accurately calculating the value of the radiation.

[0022] In an embodiment of the method according to the invention, a parameter value representative of the radiation transmitted by the imaging pixel beam is calculated for a pixel beam intersecting the projection surface.

[0023] When executing the method according to this embodiment of the method of the present invention, the computational load is reduced.

[0024] In an embodiment of the method according to the invention, the pixel beam for which the conjugation is to be calculated is selected by selecting a pixel beam whose cross section is comprised in a restitution pupil defined in the second optical system.

[0025] Such a pupil is defined by its shape (eg a circular shape defined by its radius) and its position relative to the optical center of the second optical system.The pupil reduces the number of pixel beams to be processed by allowing processing of pixel beams whose entire cross-section is inside the restoration pupil.

[0026] In an embodiment of the method according to the invention, parameter values ​​representative for radiation transmitted by the imaging pixel beam are calculated for pixel beams intersecting the projection surface and whose cross section is comprised in a restoration pupil defined in the second optical system.

[0027] When executing the method according to this embodiment of the method of the present invention, the computational load is reduced.

[0028] Another aspect of the invention is an apparatus for generating data representing at least one volume in an object space of a first optical system, the volume being occupied conjugately by a group of rays passing through a pupil of the first optical system and an image of at least one pixel of a sensor associated with the first optical system in the object space of the first optical system, the volume occupied by the group of rays being referred to as a pixel beam, the apparatus comprising a processor configured to:

[0029] - calculating the conjugate of said at least one pixel beam through the second optical system, called the imaging pixel beam,

[0030] - calculating a parameter value representing the radiation transmitted by said imaging pixel beam, which is associated with at least one element of at least one grid defined on the projection surface located at a distance D from the second optical system and belonging to the intersection between said at least one imaging pixel beam and the projection surface.

[0031] In an embodiment of the device according to the invention, the grid on the projection surface is a pixel grid.

[0032] In an embodiment of the device according to the invention, an oversampling grid is defined on the projection surface, the elements of which are smaller than the pixels of the pixel grid.

[0033] In an embodiment of the device according to the invention, for a pixel beam intersecting the projection surface, a parameter value representative of the radiation transmitted by the imaging pixel beam is calculated.

[0034] In an embodiment of the device according to the invention, the pixel beam for which the conjugation is to be calculated is selected by selecting a pixel beam whose cross section is comprised in a restoration pupil defined in the second optical system.

[0035] In an embodiment of the device according to the invention, parameter values ​​representative for radiation transmitted by the imaging pixel beam are calculated for pixel beams intersecting the projection surface and whose cross section is comprised in a restoration pupil defined in the second optical system.

[0036] According to another aspect of the present invention, a computer-implemented method is provided, comprising: receiving content captured by a first optical system, wherein the content comprises a first set of pixel beams, each pixel beam having at least one associated parameter; defining a projection surface having grid elements and a second optical system; reformatting the first set of pixel beams using the second optical system and the projection surface to determine a set of imaging pixel beams represented by a conjugate of the first set of pixel beams; determining intersections of the imaging pixel beams with the projection surface; accumulating parameter contributions associated with the imaging pixel beams based on intersections of the imaging pixel beams with the grid elements on the projection surface; and generating an image of pixel values ​​based on the accumulated parameter contributions, wherein the pixel values ​​are based on accumulated contributions from various imaging pixel beams to the grid elements of the projection surface.

[0037] In an embodiment of the computer-implemented method according to the invention, said parameter comprises a radiation or a color obtained by at least one sensor of said first optical system.

[0038] In an embodiment of the computer-implemented method according to the invention, said pixel beams consist of volume rays.

[0039] Some of the processes implemented by the elements of the present invention may be computer-implemented. Thus, the elements may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, native software, microcode, etc.), or an embodiment combining software and hardware aspects, which may all be generally referred to herein as "circuits," "modules," or "systems." Furthermore, the elements may take the form of a computer program product embodied in any tangible medium of expression having a computer usable program code embodied in the medium.

[0040] Since elements of the present invention may be implemented in software, the present invention may be embodied as computer readable code for providing to a programmable device on any suitable carrier medium. Tangible carrier media may include storage media such as floppy disks, CD-ROMs, hard disk drives, magnetic tape devices, or solid-state storage devices. Transient carrier media may include signals such as electrical, electronic, optical, acoustic, magnetic, or electromagnetic signals, for example microwave or radio frequency signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Embodiments of the present invention will now be described, by way of example only, with reference to the following drawings, in which:

[0042] Figure 1 shows the volume occupied by the set of rays passing through the pupil of the camera optical system,

[0043] Figure 2 FIG. 4 shows a hyperbolic surface representing a pixel beam according to an embodiment of the present invention.

[0044] Figure 3 shows a hyperboloid and its asymptotic cone,

[0045] Figure 4 shows a pixel beam represented by two coaxial / partially overlapping cones according to an embodiment of the present invention,

[0046] Figure 5 is a schematic block diagram illustrating an example of an apparatus for generating data representing a pixel light beam according to an embodiment of the present invention,

[0047] Figure 6 is a flowchart for illustrating a process for encoding an image captured by an optical acquisition system according to an embodiment of the present disclosure,

[0048] Figure 7 represents the conjugation of the pixel light beam and the pixel light beam through the second optical system according to an embodiment of the present invention,

[0049] Figure 8 are the projection surface and the restored pupil used in the method according to an embodiment of the present invention,

[0050] Fig. 9 represents the pixel grid and the oversampling grid defined on the restored surface,

[0051] Fig.10 represents a set of two pixel beams in the object space representing two light field acquisition systems,

[0052] Fig.11 is a flow chart of a hybrid method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] As will be appreciated by those skilled in the art, various aspects of the present principles may be embodied as systems, methods, or computer-readable media. Therefore, various aspects of the present principles may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, native software, microcode, etc.), or an embodiment of a combination of software and hardware aspects, which may be generally referred to herein as a "circuit," "module," or "system." In addition, various aspects of the present principles may take the form of a computer-readable storage medium. Any combination of one or more computer-readable storage media may be used.

[0054] For any optical acquisition system, whether plenoptic or not, it is interesting to obtain, in addition to the raw images or epipolar images representing the 4D light field data captured by the optical acquisition system, information related to the correspondence between the pixels of the sensors of the optical acquisition system and the object space of the optical acquisition system. Knowing in which part of the object space of the optical acquisition system the pixels of the sensors belonging to the optical acquisition system are sensing enables improved signal processing operations such as demultiplexing, de-mosaicking, refocusing, etc., and enables the mixing of images captured by different optical systems with different characteristics. Furthermore, the information related to the correspondence between the pixels of the sensors of the optical acquisition system and the object space of the optical acquisition system is independent of the optical acquisition system.

[0055] This disclosure introduces Figure 1 The concept of a pixel beam 10 is shown in FIG. 1 , which represents a pixel beam transmitted through a camera ( Figure 1 The volume occupied by the light beam group of the pupil of the optical system 11 (not shown), and the conjugate of the pixels of the sensor of the camera in the object space of the optical system in the direction perpendicular to the pupil surface.

[0056] The group of light rays is sensed by the pixels 12 of the sensor 13 of the camera through the pupil 14 of the optical system 11. The optical system 11 may be a combination of lenses suitable for a camera or video camera. The pupil of the optical system is defined as the image of the aperture stop seen through the optical system (i.e. the lens of the optical acquisition system, which is located in front of the aperture stop). The aperture stop is an opening that limits the amount of light passing through the optical system of the optical acquisition system. For example, an adjustable blade aperture located within a camera lens is an aperture stop for a lens. The amount of light allowed through the aperture is controlled by the diameter of the aperture opening, which can be adjusted according to the amount of light that the camera user wishes to allow in. For example, making the aperture smaller can reduce the amount of light allowed in through the aperture and at the same time increase the depth of focus. Due to the refractive effect of parts of the lens, the apparent size of the aperture can be larger or smaller than its physical size. Formally, the pupil is the image of the aperture stops of all the lenses of the optical acquisition system that are located between the physical aperture stop and the observation space.

[0057] A pixel beam 10 is defined as the bundle of light rays that arrive at a given pixel 12 when propagating through the optical system 11 via an entrance pupil 14. When light travels in a straight line in free space, the shape of such a pixel beam 10 can be defined by two cross sections, one being the conjugate 15 of the pixel 12 and the other being the entrance pupil 14. A pixel 12 is defined by its non-zero surface and its sensitivity map.

[0058] In a first embodiment of the present invention, the pixel beam 30 may be represented by a hyperboloid of one sheet supported by the following two elements: Figure 2 Shown: pupil 24 and conjugate 25 of pixel 12 in object space.

[0059] A hyperboloid is a regular surface that can support the concept of a bundle of rays and is compatible with the concept of "etendue" of a physical beam, which is related to the energy conservation through various cross-sections of a physical beam.

[0060] like Figure 3 As shown, a sheet of hyperboloid 30 is mostly identical to its asymptotic cones 31, 32, except in a fundamental region of its smallest cross section, called the beam waist 35, which corresponds to the conjugate 15 in object space. For a plenoptic system, such as a light field camera, this is the region where spatial sampling is performed by multipath light. A sampling space with a unique cone that degenerates to a point in this region is unsuitable, since the sensitivity of a pixel 12 is significant over a few tens of square microns on its surface, and cannot be represented by a mathematical point that would be an infinitesimal surface such as the tip of a cone.

[0061] In an embodiment of the present invention, each pixel beam 10, 20, 30 is defined by four independent parameters z and z that define the position and size of the pixel conjugate 15, 35 in front of the pupil 14. p ,θ x ,θ y , α, and the six pupil parameters x0, y0, z0, θ that define the position, orientation and radius of the pupils 14 and 24. x0 ,θ y0 When represented by a hyperboloid, these six pupil parameters are common to a set of pixel beams that share the same pupil 14, 24. In practice, a pixel beam represents the volume occupied by the set of rays in the object space of the optical system 11 sensed by the pixel 12 through the pupil 14, i.e., for a given pair of pixels 12 / pupil 14, 24, it corresponds to a unique pixel beam 10, 20, 30, but multiple different pixel beams can be supported by the same pupil 14, 24.

[0062] The origin O of the coordinate system (z, y, z) representing the parameters of a hyperboloid of the pixel beams 10, 20, 30 corresponds to Figure 1 , where the z-axis defines a direction perpendicular to the surface of the pupil 14 , 24 .

[0063] Parameter θ x ,θ y defines the direction of the incident principal ray relative to the center of the pupil 14. They depend on the optical elements of the optical system 11 and the position of the pixels 12 on the sensor 13. More precisely, the parameters θ x ,θ y φ represents the shear angle of the direction defining the conjugate 15 of the pixel 12 from the center of the pupil 14 .

[0064] Parameter z p represents the distance along the z-axis of the beam waist 35 of the pixel beam 10, 20, 30 or the conjugate 15 of the pixel 12.

[0065] The parameter α represents the radius of the beam waist 35 of the pixel light beam 10 , 20 , 30 .

[0066] For an optical system 11 that can model optical distortion and field curvature, the parameter z p and α can depend on the parameter θ via the parameter function x ,θ y .

[0067] Four independent parameters are associated with the pixel 12 and its conjugate 15 .

[0068] The six supplementary pupil parameters defining the pixel beams 10, 20, 30 are:

[0069] -r, which represents the radius of pupil 14, 24,

[0070] - x0, y0, z0, which represent the coordinates of the center of the pupil 14, 24 in the (x, y, z) coordinate system, and

[0071] -θ x0 ,θ y0 , which represents the orientation of the pupil 14, 24 in a reference (x, y, z) coordinate system.

[0072] These 6 pupil parameters are related to the pupil 14, 24. Another parameter c is defined. This parameter c depends on the parameter z associated with the pixel 12 and its conjugate 15 p and α, and depends on a parameter r associated with the pupil 14, 24. The parameter c defines the angular aperture α of the pixel beam 10, 20, 30 and is given by:

[0073]

[0074] Therefore, the expression for the parameter c is given by the following equation:

[0075]

[0076] The coordinates (x, y, z) of a point in object space belonging to the surface bounding the pixel beam 10, 20, 30 are functions of the above defined set of parameters associated with the pupil 14 and the conjugate 15 of the pixel. Thus, equation (2) capable of generating a hyperboloid representing the pixel beam 10, 20, 30 is:

[0077]

[0078] The parametric equation (3) for the same hyperboloid representing the pixel beams 10, 20, 30 is:

[0079]

[0080] Where v is an angle in the (x, y) plane such that the pixel beams 10, 20, 30 can be generated from a generating hyperbola, v varies in the [0, 2π] interval, and z∈[0,∞] is the z-axis coordinate along a direction defining a surface perpendicular to the pupils 14, 24. Equations (2) and (3) are written assuming that the cross section of the pixel 12 and its conjugate 15 are circular and that the cross section of the pupils 14, 24 is also circular.

[0081] When the pixel beam is represented by its parametric equation, the information related to the correspondence between the pixel of the sensor of the optical acquisition system and the object space of the optical acquisition system can take the form of four independent parameters z including the position and size of the pixel conjugate 15, 35 defined in front of the pupil 14, 24. p ,θ x ,θ y , α and the six pupil parameters x0, y0, z0, θ that define the position, orientation and radius of the pupils 14 and 24. x0 ,θ y0 .

[0082] Thus, in addition to the raw image or epipolar image representing the 4D light field data captured by the optical acquisition system, this set of parameters is also provided for use in processing the 4D light field data.

[0083] In a second embodiment of the present invention, the pixel beam 40 may be composed of two coaxial, partially overlapping cones (eg Figure 4 The front cone 41 shown F and rear cone 41 R ) indicates that the cone is supported by two elements: the pupil 44 and the conjugate 45 of the pixel 42 in object space, ie the surface in object space imaged on the pixel.

[0084] Nose cone 41 F is an image of a convex frustum defined by pixels 42 and pupil 44. The vertex of the convex frustum is located outside the sensor of the optical acquisition system. By construction, the front cone 41 F Converges in the object space of the optical collection system, and the front cone 41 F The vertex of the front cone 41 is located between the pupil 44 and the conjugate of the pixel 45 or the waist of the pixel beam 40. F The solid angle subtended by pupil 44 originates from pixel 42 .

[0085] Rear cone 41 R is the image of a cone defined by the pixels 42 and the pupil 44, whose apex is located between the pupil 44 and the sensor of the optical acquisition system. By construction, the back cone 41 R The vertex of the back cone 41 is located outside the beam waist 45 of the pupil 40. R It does not necessarily converge in the object space of the optical collection system. In some cases, it can degenerate into a cylinder or a diverging cone. In the latter case, the vertex of the diverging cone is located in the image space of the optical collection system, that is, before the pupil 44 is incident.

[0086] Nose cone 41 F and rear cone 41 R Share the same axis of rotation, which is the line connecting the center of the pupil 44 and the center of the beam waist 45 .

[0087] A cone is a regular surface that can support the concept of a bundle of rays, and when two cones are combined, is compatible with the concept of "etendue" of a physical beam, which is related to the energy conservation through each section of a physical beam. The intersection of a cone with a plane is a conic section, which can be characterized by multiple coefficients as for a hyperbola. Considering its vertex, a cone can be described by three angular parameters: the polar angle measured from the cone's axis of rotation to the vertex angle and the direction of the axis of rotation given by two angles.

[0088] Let xyz be the coordinate system of the optical collection system, let z denote the optical axis of the optical collection system with z>0 in the object space of the optical collection system, and the center of the pupil 44 be the origin of the coordinate system.

[0089] The optical system of the optical acquisition system images the object space of the optical acquisition system from the range z∈[2f;+∞] to the image space of the optical acquisition system z∈[-2f;-f], where f is the focal length of the optical system of the optical acquisition system. In the coordinate system xyz of the optical acquisition system, the positions of the pupil 44 and the beam waist 45 of the pixel beam 40 are known from the calibration of the optical acquisition system. It is assumed that the pupil 44 and the beam waist 45 are parallel and both perpendicular to the z-axis.

[0090] Let us call the chief ray of the pixel beam 40 as z'. The chief ray is the line connecting the center of the pupil 44 and the center of the waist 45 of the pixel beam 40. The chief ray is also the axis of rotation and the axis of symmetry of the pixel beam 40. Thus, in the coordinate system xyz', the pixel beam 40 is a rotating entity.

[0091] Front cone 41 F and rear cone 41 R both have their vertices located on the chief ray z' of the pixel beam 40. In the thin lens approximation, assuming that the sensor of the optical acquisition system is not located in the rear focal plane, the coordinates of these two vertices are calculated in the coordinate system xyz of the optical acquisition system as follows:

[0092]

[0093] That is:

[0094]

[0095] where P, z p , W and z w represent the diameter of the pupil 44 with P>0, its z coordinate, the diameter of the conjugate 45 of the pixel with 0<W<+∞, and its z coordinate (0<z w <+∞), respectively.

[0096] When the rear cone 41 R is a diverging cone, the z coordinate z rear of the vertex of the rear cone 41R can be positive, and when the rear cone 41R is a converging cone, the z coordinate z rear of the vertex of the rear cone 41R can be negative. If the pupil 44 and the conjugate 45 of the pixel of the pixel beam have the same size, it can also be infinite.

[0097] If the sensor of the optical acquisition system is located in the rear focal plane, then W = +00 and z w = +∞. Since their ratio is constant:

[0098]

[0099] Assume that the optical system of the optical acquisition system is a converging lens, where p and f represent the diameter of the pixel 42 with p>0 and the focal length f>0 of the optical system of the optical acquisition system, respectively.

[0100] The vertex angle is given by:

[0101]

[0102] Considering the vertices of each cone, whose union represents the pixel beam 40, rays can be defined by two angular parameters: the polar angle and the azimuthal angle ([0, 2π]) measured from the pixel beam's axis of rotation to the vertex angle.

[0103] Those information related to the pixel beams are metadata associated with a given optical acquisition system. They can be provided as data files stored, for example, on a CD-ROM or flash drive provided with the optical acquisition system. Data files containing additional information related to the pixel beams can also be downloaded from a server belonging to the manufacturer of the optical acquisition system. In an embodiment of the present invention, this additional information related to the pixel beams can also be embedded in the header of the image captured by the optical acquisition system.

[0104] Knowledge of this information related to the pixel beams enables processing of images captured by any optical acquisition system, independent of proprietary file formats and features of the optical acquisition system used to capture the images to be processed.

[0105] The knowledge of the information related to the pixel beams enables processing of images captured by any optical acquisition system, independent of proprietary file formats and characteristics of the optical acquisition system used to capture the images to be processed.

[0106] Figure 5 is a schematic block diagram illustrating an example of an apparatus for generating data representing a pixel light beam according to an embodiment of the present disclosure.

[0107] The apparatus 500 includes a processor 501, a storage unit 502, an input device 503, a display device 504, and an interface unit 505 connected via a bus 506. Of course, the constituent elements of the computer apparatus 500 may be connected via connections other than bus connections.

[0108] The processor 501 controls the operation of the device 500. The storage unit 502 stores at least one program capable of generating data representing pixel beams representing the object space of the first optical system when these pixel beams are imaged by the second optical system operated by the processor 501, and various data, including parameters related to the position of the pixel 12 on the sensor 13 or parameters related to the first optical system 11 and the second optical system of the optical acquisition system, parameters used by the calculations performed by the processor 501, intermediate data of the calculations performed by the processor 501, etc. The processor 501 can be formed by any known and suitable hardware, or software, or a combination of hardware and software. For example, the processor 501 can be formed by dedicated hardware such as a processing circuit, or by a programmable processing unit such as a CPU (central processing unit) that executes a program stored in its memory.

[0109] The storage unit 502 may be formed of any suitable storage or component capable of storing programs, data, etc. in a computer-readable manner. Examples of the storage unit 502 include a non-transitory computer-readable storage medium such as a semiconductor memory device, and a magnetic, optical, or magneto-optical recording medium loaded into a read and write unit. Figure 6 As described above, the program enables the processor 501 to perform a process for calculating data representing a pixel beam set according to an embodiment of the present disclosure.

[0110] The input device 503 may be formed by a keyboard, a pointing device such as a mouse, etc., for a user to input commands so that the user selects parameters for generating a parameter representation of a volume occupied by a group of rays in an object space of an optical system. The output device 604 may be formed by a display device to display an image generated according to an embodiment of the present disclosure, such as a graphical user interface (GUI). For example, the input device 503 and the output device 504 may be formed integrally by a touch screen panel.

[0111] The interface unit 505 provides an interface between the apparatus 500 and an external device. The interface unit 505 may communicate with the external device via wired or wireless communication. In an embodiment, the external device may be an optical acquisition system, such as an actual camera.

[0112] Figure 6 is a flow chart for illustrating a process of generating data representing a pixel light beam according to an embodiment of the present invention.

[0113] The processor 501 of the device 500 executes step 600A, such as Figure 7 As shown, at least one pixel beam 70 representing a set of pixel beams in the object space (not shown) of the first optical system is calculated from the conjugate 72 of the pixel beam 70 passing through the second optical system 71. In the following example, the pixel beams 70, 72 are represented by a hyperboloid 20.

[0114] Therefore, the processor 501 calculates the conjugate 72 of the pixel beam 70 through the optical system 71 (eg, a thin lens) to provide data representing the pixel beam 70 in the focal volume of the optical system 71 .

[0115] The conjugate 72 of the pixel beam 70 passing through the optical system 71 may be calculated as follows.

[0116] As reference Figures 1 to 3 As mentioned above, the pixel beam associated with the pupil of the optical system is defined by 6 external parameters:

[0117] r: entrance pupil radius of the first optical system,

[0118] x0, y0, z0: coordinates of the center of the entrance pupil of the first optical system in the reference coordinate system, and

[0119] ·θ x0 ,θ y0 : The direction of the entrance pupil of the first optical system in the reference coordinate system.

[0120] In the reference coordinate system, for simplicity, keep θ x0 =θ y0 = 0, the parametric equation of the hyperboloid representing the pixel beam is given by:

[0121]

[0122] or equivalently:

[0123]

[0124] In an embodiment of the invention, a value (such as an RGB value) representing the color acquired by a pixel of the sensor of the first optical acquisition system is associated with the corresponding pixel light beam 70 .

[0125] Parameters defining the plane representing the second optical system 71 are either defined when the second optical system 71 is a virtual optical system, or retrieved when the second optical system is a real optical system:

[0126] The coordinates of the optical center of the second optical system 71 are x C ,y C 、z C

[0127] The coordinate of the unit vector perpendicular to the surface of the optical system 71 is θ xC ,θ yC

[0128] • Focus f of the optical system 71 .

[0129] To simplify the description, the optical center of the second optical system 71 is located in the same plane as the pupil center of the first optical system: C = z0, and the optical system 71 should be parallel to the pupil, i.e. θ xC =θ yC =0.

[0130] The coordinates of point M relative to the center C of optical system 71 are expressed as: x, y, z. The coordinates of image M' of point M through optical system 71 are expressed as: x', y', z'. Each of these coordinates is an algebraic value.

[0131] The coordinates of point M′ are calculated using the Newtonian form of the “Lens Maker formula”, namely:

[0132] (|z|-f)(|z′|-f)=f 2 (10)

[0133] Since point M is located before the optical system 71, i.e. z<0, and point M′ is located after the optical system 71, i.e. z′>0, then:

[0134] (-zf)(z′-f)=f 2 (11)

[0135] Given

[0136]

[0137] When the optical system 71 is a thin lens, since light passing through its optical center does not deviate, the magnification is given by:

[0138]

[0139] Which gives:

[0140]

[0141] And conversely,

[0142]

[0143] When M is a point belonging to the surface of the pixel beam 70, the coordinates of a point M' located on the surface of the imaging pixel beam are:

[0144]

[0145] When the paraxial lens converts light (straight lines) in object space into light in image space, and when the pixel beam 70 is represented by a regular surface, the imaging pixel beam 72 is also represented by a regular surface, such as the image of a hyperbolic surface in object space is a hyperbolic surface in image space.

[0146] Let us determine the parameters representing the hyperboloid of the imaging pixel beam 72 from the parameters representing the hyperboloid of the pixel beam 70 and the parameters of the optical system 71 .

[0147] Consider the following equation:

[0148]

[0149] represents a pixel beam 70, where θ x =θ y =0, x0=y0=z0 and v=0 in order to reduce the pixel beams on the axis and the analysis of the x and z planes.

[0150] Then the parameter to be determined is z′ w , α′, which is z p , α, f, r, where z′ w is the z coordinate of the beam waist of the imaging pixel beam 72 (due to the beam aperture −α≠0, W′ is not the conjugate point of P through the imaging process) and α′ is the beam waist radius of the imaging pixel beam 72 .

[0151] Assume θ x =θ y = 0, x0 = y0 = z0 and v = 0, equation (16) applied to point M simplifies to:

[0152]

[0153] Using the paraxial lens transformation of equation (14) for M we obtain:

[0154]

[0155]

[0156]

[0157]

[0158] It is an equation representing the imaging pixel beam 72 in the focal volume of the optical system 71 .

[0159] Deriving equation (20) enables the calculation of the minimum and / or maximum of the hyperbola representing the pixel beam by finding the zero transitions of the derivatives.

[0160] Equation (20), when developed, gives:

[0161]

[0162] The derivative of equation (21) is given by:

[0163]

[0164]

[0165] Computing the derivative (23) with zero swaps gives:

[0166]

[0167]

[0168] Known but Each term of can be divided by Therefore, if zp >>f Then the approximation can be derived: Since α′ is the beam waist radius of the imaging pixel beam 72, it is also the minimum of the positive hyperbola of the imaging pixel beam 72, that is, it corresponds to the minimum value defined by Equation 20 for z′=z′ w The value of x′ is:

[0169] a′=x′(z′w) (27)

[0170] This gives:

[0171]

[0172]

[0173]

[0174] Let us consider a point x, y and its conjugate x', y' passing through the optical system 71. For a ray passing through the point x, y and the center of the optical system 71, the laws of optics give:

[0175] If the light is the incident light θ when it strikes the optical system 71 at a height x0 x ,So: And similarly:

[0176] therefore:

[0177]

[0178] Formula from lens manufacturer From this, we can see that:

[0179]

[0180] And similarly:

[0181]

[0182] Therefore, the equation representing the imaging pixel beam 72 is:

[0183]

[0184] The parameter z from the pixel beam 70 is p ,θ x ,θ y , α obtained parameter z′ w ,θ′ x ,θ′ y , a′ is expressed as:

[0185]

[0186] in,

[0187] In step 600, the processor 501 of the device 500 compares each of the imaging pixel beams 72 of the set of imaging pixel beams with the value of the radiation (denoted as l(p) 2 ), the radiation propagates in a pixel beam 72 of pixel imaging toward the second optical system 71. For example, the value of the radiation is given using RGB values.

[0188] The set of imaged pixel beams 72 is either obtained by calculating the conjugate 72 of the pixel beams 70 through the optical system 71 as explained in the previous step 600A, or the set of already imaged pixel beams is provided to the device 500 from an external source.

[0189] In step 601, Figure 8 The projection surface 80 represented in φ is defined in the focal volume of the second optical system 71. Such a projection surface is equivalent to a virtual sensor, for example.

[0190] Such a projection surface 80 is, for example, rectangular in shape, and is orthogonal to and centered on the optical axis of the second optical system 71. It intersects the optical axis of the second optical system 71 at a distance D from the second optical system 71.

[0191] In step 603 , the processor 501 of the apparatus 500 defines at least one grid on the projection surface 80 .

[0192] In an embodiment of the present invention, the grid defined on the projection surface 80 is a pixel grid.

[0193] In another embodiment of the present invention, two grids are defined on the projection surface 80, such as Fig. 9 As shown, a pixel grid 90 and an oversampling grid 91 including a plurality of sub-pixel elements.

[0194] A pixel of the pixel grid has two coordinates on the projection surface 80 and has an extension in two directions di, dj, where i∈[0, i max ],j∈[0,j max ], and (x, y, dx, dy) i，j =Rect(i, j) and Rect(i, j) are by default functions defined to calculate a rectangular area of ​​position x, y and dimensions dx, dy on the projection surface 80.

[0195] For optimal rendering compared to the available pixel beam information, dx, dy can be defined by half of the minimum overlap between pixel beam interfaces on the projection surface 80, which corresponds to the Shannon criterion. For situations where a higher resolution is required at the focal point of the projection surface 80, the minimum information sample has the size of the minimum overlap between pixel beam sections. Statistical estimates can be used to define dx and dy.

[0196] The sub-pixel elements of the oversampling grid 91 are mxn times smaller than the pixels of the pixel grid 90, i.e.

[0197] In step 602, a second optical system 71 is defined. Figure 8 A restitution pupil 81 is shown.

[0198] The restoration pupil 81 is, for example, circular, with a radius r RP The position of the restored pupil 81 relative to the optical system 71 is given by its center coordinate (x RP ,y RP ) is given.

[0199] In step 603, the processor 501 selects a first group S of imaging pixel beams 72 from the set of imaging pixel beams. RP The selected imaging pixel beam S RP is the imaging pixel beam whose section is completely located in the restoration pupil 81. This step 603 is optional and may not be performed in some embodiments of the present invention.

[0200] In step 604, processor 501 selects a second group S of imaging pixel beams 72 from the set of imaging pixel beams. PS The selected imaging pixel beam S PS is the imaging pixel beam that intercepts the projection surface 80. This step 604 is optional and may not be performed in certain embodiments of the present invention.

[0201] In another embodiment of the present invention, the first group S of imaging pixel light beams is RP A second group S of imaging pixel beams is selected PS Therefore, the second group S of imaging pixel beams PS This corresponds to the imaging pixel beam 72 having a cross section that is completely located in the restoration pupil 81 and intercepts both the projection surface 80 .

[0202] In step 605, two accumulation buffers V(i sub , j sub ), W(isub , j sub ).

[0203] In step 606, for the second group S PS For each imaging pixel light beam, the processor 501 calculates the cross section of the imaging pixel light beam intercepting the projection surface 80. Such a cross section is, for example, circular, elliptical or rectangular.

[0204] Then, in step 607, the value l(p) associated with the imaging pixel beam under consideration is accumulated at each element of the grid defined on the projection surface 80. Thus, for example, the value l(p) associated with the imaging pixel beam under consideration is accumulated at each sub-pixel element of the oversampled grid 91 intercepted by a cross section of the imaging pixel beam under consideration.

[0205] The accumulation consists in summing the values ​​l(p) of the considered sub-pixel elements of the oversampled grid 91 in a sub-pixel value register v embedded in the device 500 and summing the values ​​of the hits in a weight register w embedded in the device 500. If the image corresponds to a flat image, the value of the hit can be 1 or 0. If the radiance obtained on a uniform "white" source image is not uniform, for example due to vignetting, the weight w has a value between 0 and 1, which corresponds to the acquisition of a uniform "white" source image:

[0206]

[0207]

[0208] For the second group S PS Steps 606 and 607 are executed for each pixel beam.

[0209] In step 608, the value of l(p) corresponding to the considered cross section of the imaging pixel beam intercepting the oversampled grid 91 is calculated by the processor 501. This value is given by:

[0210] In step 609, the value of l(p) corresponding to a pixel of the pixel grid 90 is calculated by the processor 501. Such a value L(i,j) is obtained by scaling the value L(i,j) associated with the sub-pixel element belonging to the same pixel of the pixel grid 90 by a factor mxn using a method such as border scaling, sum subsampling or by another scaling algorithm. sub , j sub ) is obtained.

[0211] This method enables to take a single image in the focal volume of a known optical system as the distance D between the projection surface 80 and the optical system, the radius r of the restored pupil 81 RP and the center coordinates (xRP ,y RP ) function to calculate.

[0212] Calculate the radius r of the pupil 81 with different D values ​​and restoration RP Several images are provided with focus stacking.

[0213] Calculate the center coordinate values ​​(x RP ,y RP ) provides matrix focus stacking.

[0214] Methods according to different embodiments of the invention enable mixing light field data acquired by different types of light field acquisition systems, they may be acquired from tangible acquisition systems or computer graphics image (CGI) data that are fully or partially simulated by a computer for a given scene description.

[0215] Fig.10 1 and 110 represent two sets of pixel beams 100 and 110 representing the object space of two light field acquisition systems (not shown in the figure). Figure 6 The method described images the conjugate of the two sets 100 and 110 of pixel beams by means of a common optical system 120. Thus, two sets of imaged pixel beams 130 and 140 are obtained.

[0216] In a first embodiment of the invention, to address inaccurate calibration and positioning of a light field acquisition system intended to mix light field data, different sets of parameters representing the optical system 120 are provided to calculate the conjugation of the pixel beams of each set 100 and 110 of pixel beams.

[0217] That is, the conjugate of the pixel beams of the first set 100 of pixel beams is calculated using the first set of parameters representing the optical system 120 , and the conjugate of the pixel beams of the second set 110 of pixel beams is calculated using the first set of parameters representing the optical system 120 .

[0218] The parameters of the optical system 120 are defined in the object space in which the pixel beams of a given set of pixel beams are defined. The optical system 120 is defined by:

[0219] - its optical center x C ,y C 、z C location,

[0220] -Unit vector θ xc ,θ yc ,

[0221] - Its focal length f.

[0222] Adjusting these parameters for each set of pixel beams 100, 110 allows positioning and scaling the conjugates of the pixel beams of these respective sets. It also enables Figure 6 The described method of image generation and blending operations precedes the adjustment of the sets of two imaging pixel beams 130 and 140 relative to each other.

[0223] In the second embodiment of the present invention, when calculating the conjugate of the pixel light beam passing through the optical system 120, the intrinsic parameters of the light field acquisition system and the extrinsic parameters of the light field acquisition system are considered.

[0224] The intrinsic parameters of the light field acquisition system relate the coordinate system of the sensor pixels to the coordinate system of the light field acquisition system. These intrinsic parameters are, for example, the z-axis that defines the position and size of the pixel conjugate in front of the pupil of the optical system 120. p ,θ x ,θ y , α, and the six pupil parameters x0, y0, z0, θ that define the position, orientation, and radius of the pupil x0 ,θ y0 These six pupil parameters are common to a collection of pixel beams when represented by a hyperboloid.

[0225] The extrinsic parameters of the light field acquisition system relate the coordinate system of the light field acquisition system to the world coordinate system and define the position and orientation of the light field acquisition system.

[0226] For each light field acquisition system C i , the function transforms the point P in the world coordinate system w Transform its image into the coordinate system of the light field acquisition system Such a function is, for example, a combination of a translation and a rotation. In this case: in and Both are related to the light field acquisition system C i The associated rotation matrix and translation vector. In some embodiments of the present invention, the coordinate system of the light field acquisition system and the world coordinate system are the same. In this case, the rotation matrix is the identity matrix I, and the translation vector is the zero vector.

[0227] Fig.11 A flow chart showing a second embodiment of the present invention.

[0228] In step 1110, a first adjustment of the pixel beams of the first set of pixel beams 100 is calculated using an intrinsic parameter set of the light field acquisition system used to acquire the first set of pixel beams 100. The result of this step 1110 is a first adjusted set of pixel beams.

[0229] In step 1120, a second adjustment of the first adjusted set of pixel beams is calculated using the set of extrinsic parameters of the light field acquisition system used to acquire the first set of pixel beams 100. The result of this step 1120 is a second adjusted set of pixel beams.

[0230] In step 1130, according to the reference Figure 6 The described method calculates the conjugate of the second adjusted set of pixel beams.The result of this step 1130 is the first imaged set of pixel beams.

[0231] In step 1140, a first adjustment of the pixel beams of the first set of pixel beams 110 is calculated using an intrinsic parameter set of the light field acquisition system used to acquire the first set of pixel beams 110. The result of this step 1140 is a fourth adjusted set of pixel beams.

[0232] In step 1150, a second adjustment of the fourth adjusted set of pixel beams is calculated using the extrinsic parameters of the light field acquisition system used to acquire the first set of pixel beams 110. The result of step 1150 is a fifth adjusted set of pixel beams.

[0233] In step 1160, according to the reference Figure 6 The described method calculates the conjugate of the fifth adjusted set of pixel beams.The result of this step 1150 is a second imaged set of pixel beams.

[0234] In step 1170, the first imaged set of pixel beams and the second imaged set of pixel beams are mixed into a single image.

[0235] During steps 1120 and 1150, some creative editing may be performed while the second adjustment is performed using external parameters. This may also be the case during the mixing step 1170.

[0236] This approach enables special visual effects to be performed at the light field level.

[0237] Although the present invention has been described above with reference to a specific embodiment, the present invention is not limited to the specific embodiment and modifications will be apparent to those skilled in the art within the scope of the present invention.

[0238] Many further modifications and variations will suggest themselves to those skilled in the art upon reference to the foregoing illustrative embodiments, which are given by way of example only and are not intended to limit the scope of the invention, which is determined solely by the scope of the appended claims. In particular, different features from different embodiments may be interchanged where appropriate.

Claims

1. A computer-implemented method comprising: receiving first content captured by a first acquisition device, the first acquisition device comprising a plurality of pixels and having a first optical system, wherein the first content comprises a first set of pixel beams, each pixel beam having at least one first associated parameter; receiving second content captured by a second capture device, the second capture device comprising a plurality of pixels and having a second optical system, wherein the second content comprises a second set of pixel beams, each pixel beam having at least one second associated parameter; wherein a pixel beam is defined as the volume occupied by a set of rays emitted in the object space of an optical system; said rays traverse the conjugate of a pupil and a pixel of said optical system; determining a projection surface having mesh elements and an associated non-physical optical system, wherein the non-physical optical system is defined by intrinsic parameters and extrinsic parameters; transforming the first set of pixel beams using the intrinsic and extrinsic parameters and the projection surface format to determine a first set of imaging pixel beams represented by a conjugate of the first set of pixel beams in the non-physical optical system; transforming the second set of pixel beams using the intrinsic parameters and the extrinsic parameters and the projection surface format to determine a second set of imaging pixel beams in the non-physical optical system; wherein the first group of pixel light beams and the second group of pixel light beams are mixed together to form a unique group of pixel light beams; determining an intersection section of an imaging pixel beam from the first group of imaging pixel beams and the second group of imaging pixel beams with the projection surface; accumulating values ​​of parameters associated with the imaging pixel beams from the first group of imaging pixel beams and the second group of imaging pixel beams based on intersections of the imaging pixel beams with the grid elements on the projection surface, wherein the values ​​of the parameters include at least one of radiance or color; and An image of pixel values ​​is generated based on the accumulated parameter values.

2. The method of claim 1, wherein: The at least one first associated parameter is obtained by at least one sensor of the first acquisition device.

3. The method of claim 1, wherein: The grid elements on the projection surface are a pixel grid.

4. The method of any one of claims 1 to 3, wherein the projection surface is located in the image space of the non-physical optical system.

Images