Method and device for encoding a signal representing a light field content

Abstract

The present invention relates to the delivery of a set of data and metadata, and more particularly to the delivery of light field content. Light field data occupies a large amount of storage space, which makes storage troublesome and processing inefficient. In addition, the light field acquisition devices are very diverse, and each camera has its own proprietary file format. Since light field data acquired from different cameras has a variety of formats, complex processing is caused at the receiver side. To this end, a method of encoding a signal representative of the content of a light field is proposed, in which parameters of light rays sensed by different pixels of a sensor are mapped onto the sensor. A second set of encoded parameters is used to reconstruct light field content from parameters representing light sensed by different pixels of the sensor.

Description

[0001] This divisional application is a divisional application with a filing date of June 22, 2017, application number 201710481124.X, and invention name “Method and device for encoding a signal representing light field content”. Technical Field

[0002] The present invention relates to the delivery of collections of data and metadata, and more particularly to the delivery of light field content. Background Art

[0003] In the article published by Anat Levin et al. in the ECCV 2008 conference proceedings, Understandingcameratrade-offsthroughaBayesiananalysisoflightfieldprojectio ns " explains the acquisition of four-dimensional or 4D light field data, which can be regarded as a sampling of the 4D light field, that is, a record of light.

[0004] Compared to classic two-dimensional or 2D images obtained from a camera, 4D light field data enables the user to access more post-processing features that enhance the presentation of the image and interactivity with the user. For example, with 4D light field data, refocusing of the image can be performed with a freely selectable focus distance, meaning that the position of the focal plane can be specified / selected a posteriori, as well as slightly changing the viewpoint in the scene of the image.

[0005] There are several ways to represent 4D light field data. In fact, Ren Ng published a paper in July 2006 entitled " DigitalLightFieldPhotography " in Chapter 3.3 of his doctoral dissertation describes three different ways to represent 4D light field data. First, 4D light field data can be represented when recorded by a plenoptic camera through a collection of microlens images. The 4D light field data in this representation is called raw images or raw 4D light field data. Second, 4D light field data can be represented when recorded by a plenoptic camera or a camera array through a set of sub-aperture images. The sub-aperture images correspond to captured images of the scene from viewpoints that differ slightly between the two sub-aperture images. These sub-aperture images give information about the parallax and depth of the imaged scene. Third, 4D light field data can be represented by a set of epipolar images, see, for example, S. Wanner et al., in the ISVC 2011 conference proceedings, entitled " Generating EPIRepresentation of a 4D Light Fields with a SingleLensFocusedPlenopticCamera "Article.

[0006] Light field data takes up a lot of storage space, which makes storage cumbersome and processing inefficient. In addition, light field acquisition devices are very diverse. There are different types of light field cameras, such as plenoptic or camera arrays. Within each type, there are many differences, such as different optical arrangements or microlenses of different focal lengths. Each camera has its own proprietary file format. Currently, there is no standard that supports the acquisition and transmission of multi-dimensional information for a detailed overview of the different parameters that the light field depends on. Since the light field data acquired from different cameras has multiple formats, it causes complex processing on the receiver side.

[0007] The present invention has been designed in view of the foregoing. Summary of the invention

[0008] According to a first aspect of the present invention, there is provided a computer-implemented method of encoding a signal representing light field content, the method comprising:

[0009] - encoding, for at least one pixel of the sensor, a first set of parameters representative of the light sensed by said pixel,

[0010] - encoding a second set of parameters intended for reconstructing light field content from the first set of parameters.

[0011] The parameters transmitted according to the encoding method according to an embodiment of the invention are independent of the optical acquisition system used to acquire the light field content to be transmitted and processed by the receiving device.

[0012] In the method according to an embodiment of the present invention, the parameters representing the light sensed by different pixels of the sensor of the optical acquisition system, i.e., the parameters in the first parameter set, are mapped on the sensor. Therefore, these parameters can be considered as pictures. For example, when the light sensed by the pixels of the optical acquisition system is represented by four parameters, the parameters representing the light sensed by the pixels of the sensor of the optical acquisition system are grouped in four pictures.

[0013] Such pictures can be encoded and transmitted according to video standards such as MPEG-4 Part 10 AVC (also known as h264), h265 / HEVC or their possible successors h266, and transmitted in a connected video bitstream. The second coding set can be encoded using a supplemental enhancement information (SEI) message. The format defined in the method according to an embodiment of the present invention enables strong compression of the transmitted data without introducing any strong errors (lossless coding) or a limited amount of errors (lossy coding).

[0014] The method according to an embodiment of the present invention is not limited to light field data directly collected by an optical device. These data can be computer graphics images (CGI) that are fully or partially simulated by a computer for a given scene description. Another source of light field data can be modified post-production data, such as color grading, light field data obtained from an optical device or CGI. Now in the film industry, it is also common to have data that is a mixture of both data acquired using an optical acquisition device and CGI data. It should be understood that the pixels of a sensor can be simulated by a computer-generated scene system, and by extension the entire sensor can be simulated by the system. From this it should be understood that any reference to "pixels of a sensor" or "sensor" can be a physical object attached to an optical acquisition device or a simulated entity obtained by a computer-generated scene system.

[0015] According to an embodiment of the encoding method, the encoding method further comprises:

[0016] - encoding a third set of parameters representing color data associated with the light represented by the first set of parameters.

[0017] According to an embodiment of the encoding method, at least one parameter in the first parameter set represents the distance between the coordinates of the light ray and a plane that fits the coordinate set of multiple light rays sensed by multiple pixels of the optical acquisition system, and at least one parameter in the second parameter set represents the coordinates of the fitting plane.

[0018] Encoding the distance between the coordinates of a ray and a plane fitting the set of coordinates of a plurality of rays sensed by different pixels of the sensor enables compression of the data to be transmitted, since the amplitude between different values ​​of the calculated distance is generally lower than the amplitude between different values ​​of the coordinates.

[0019] According to an embodiment of the encoding method, at least one parameter in the first parameter set represents:

[0020] - a difference between a value representing the light sensed by said pixel and a value representing the light sensed by another pixel preceding said pixel in a row of sensors, or

[0021] - when the pixel is the first pixel of a row of sensors, the difference between the value representing the light sensed by the pixel and the value representing the light sensed by the last pixel of the row before the row to which the pixel belongs.

[0022] The value representing the light ray may be a coordinate representing the light ray or a distance between coordinates or a plane fitting a set of coordinates of a plurality of light rays sensed by different pixels of the sensor.

[0023] This makes it possible to compress the data by reducing the amplitude between different values ​​of the parameter to be transmitted.

[0024] According to an embodiment of the encoding method, the parameters in the first parameter set are encoded using independent codecs.

[0025] According to an embodiment of the encoding method, the second parameter set is only transmitted when the second parameter set comprises a parameter indicating that the first parameter set has not changed since a last transmission of the first parameter set.

[0026] This makes it possible to reduce the amount of data to be transmitted to the decoding device.

[0027] According to an embodiment of the encoding method, the light field data consists of a sequence of subsets of the light field data. For example, the subsets are indexed in time in order to represent a dynamic or moving scene. Typically, the elements (or subsets) of the sequence are called frames. In this case, the transmission rate of the signal representing the light content (typically characterized by, but not limited to, the number of frames per second) is higher than the transmission rate of the encoded first parameter set and the encoded second parameter set.

[0028] This makes it possible to reduce the amount of data to be transmitted to the decoding device.

[0029] Another object of the present invention relates to a device for encoding a signal representing light field content, comprising a processor, the processor being configured to:

[0030] - encoding, for at least one pixel of the sensor, a first set of parameters representative of the light sensed by said pixel,

[0031] - encoding a second set of parameters intended for reconstructing light field content from the first set of parameters.

[0032] Another aspect of the invention relates to a computer-implemented method of decoding a signal representing light field content, the method comprising:

[0033] - for at least one pixel of the sensor, decoding a first set of parameters representative of the light sensed by said pixel,

[0034] - decoding a second set of parameters intended for reconstructing light field content from the decoded first set of parameters,

[0035] - reconstructing the light field content based on the decoded first set of parameters and the decoded second set of parameters.

[0036] According to an embodiment of the decoding method, the decoding method further includes:

[0037] - decoding a third set of parameters representing color data associated with the light represented by the first set of parameters,

[0038] - reconstructing the light field content based on the decoded third parameter set together with the decoded first parameter set and the decoded second parameter set.

[0039] Another aspect of the invention relates to a device for decoding a signal representing light field content, comprising a processor configured to:

[0040] - for at least one pixel of the sensor, decoding a first set of parameters representative of the light sensed by said pixel,

[0041] - decoding a second set of parameters intended for reconstructing light field content from the first set of parameters,

[0042] - reconstructing the light field content based on the decoded first set of parameters and the decoded second set of parameters.

[0043] Another aspect of the invention relates to a signal transmitted by a device for encoding a signal representing light field content to a device for decoding said signal representing light field content, said signal carrying a message, said message comprising:

[0044] - a first set of parameters representing light sensed by at least one pixel of the sensor,

[0045] - a second set of parameters intended for reconstructing light field content from the first set of parameters,

[0046] - The reconstruction of the light field content by the decoding device is based on the decoded first set of parameters and the decoded second set of parameters.

[0047] Some of the processing implemented by the elements of the present invention can be computer-implemented. Therefore, such elements can take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or the form of an embodiment that combines software and hardware aspects (which can generally be referred to as "circuit", "module" or "system" herein). In addition, these elements can take the form of a computer program product contained in any tangible medium of expression, with a computer-usable program code contained in the medium.

[0048] Since elements of the present invention can be implemented in software, the present invention can be implemented as computer readable code for providing to a programmable device on any suitable carrier medium. Tangible carrier media may include storage media such as floppy disks, CD-ROMs, hard disk drives, magnetic tape devices, or solid-state storage devices. Transient carrier media may include signals such as electrical, electronic, optical, acoustic, magnetic, or electromagnetic signals, such as microwave or RF signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Embodiments of the present invention will now be described by way of example only and with reference to the following drawings, in which:

[0050] Figure 1 is a block diagram of a light field camera device according to an embodiment of the present invention;

[0051] Figure 2 is a block diagram illustrating a specific embodiment of a possible implementation of a light field data formatting module;

[0052] Figure 3 The diagram shows rays passing through two reference planes P1 and P2 used for parameterization;

[0053] Figure 4 Illustration of a pass through a known depth Reference plane at of light;

[0054] Figure 5 is a flow chart illustrating the steps of a method of formatting light field data according to an embodiment of the present invention;

[0055] Figure 6 is a flow chart illustrating the steps of a method of formatting light field data according to an embodiment of the present invention;

[0056] Figure 7 Indicates when four independent monochrome codecs are used to transmit to the receiver Mapping, Map or Mapping;

[0057] Figure 8 Represents when grouped in a single image Mapping, Map or Mapping. DETAILED DESCRIPTION

[0058] As will be appreciated by those skilled in the art, aspects of the present principles may be embodied as systems, methods, or computer-readable media. Thus, aspects of the present principles may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or a form of an embodiment combining software and hardware aspects (which may all be generally referred to herein as "circuits," "modules," or "systems"). In addition, aspects of the present principles may take the form of a computer-readable storage medium. Any combination of one or more computer-readable storage media may be used.

[0059] Embodiments of the present invention provide for formatting of light field data for further processing applications such as format conversion, refocusing, viewpoint change, etc. The provided formatting enables the light field data to be properly and easily reconstructed at the receiver side for processing thereof. The provided format has the advantage of being independent of the device used to acquire the light field data.

[0060] Figure 1 1 is a block diagram of a light field camera device according to an embodiment of the present invention. The light field camera includes an aperture / shutter 102, a main (objective) lens 101, a microlens array 110, and a light sensor array. In some embodiments, the light field camera includes a shutter release that is activated to capture a light field image of an object or scene.

[0061] The light sensor array 120 provides light field image data, which is acquired by the LF data acquisition module 140, used to generate a light field data format by the light field data formatting module 150 and / or processed by the light field data processor 155. According to embodiments of the present invention, after acquisition and after processing, the light field data may be stored in the memory 190 in a raw data format, as a sub-aperture image or a focal stack, or in the memory 190 in a light field data format.

[0062] In the illustrated example, the light field data formatting module 150 and the light field data processor 155 are arranged in the light field camera 100 or integrated into the light field camera 100. In other embodiments of the present invention, the light field data formatting module 150 and / or the light field data processor 155 may be provided in a separate component external to the light field capture camera. The separate component may be local or remote relative to the light field image capture device. It should be understood that any suitable wired or wireless protocol may be used to transmit the light field image data to the formatting module 150 or the light field data processor 155; for example, the light field data processor may be connected via the Internet, a cellular data network, a WiFi network, or a wireless network. A communication protocol and / or any other suitable means for transmitting the captured light field image data and / or other data.

[0063] According to an embodiment of the present invention, the light field data formatting module 150 is configured to generate data representing the acquired light field. The light field data formatting module 150 may be implemented in software, hardware, or a combination thereof.

[0064] According to an embodiment of the present invention, the light field data processor 155 is configured to operate on the raw light field image data received directly from the LF data acquisition module 140, for example, to generate formatted data and metadata. Output data may be generated, such as, for example, a still image of the captured scene, a 2D video stream, etc. The light field data processor may be implemented in software, hardware, or a combination thereof.

[0065] In at least one embodiment, the light field camera 100 may also include a user interface 160 for enabling a user to provide user input to control the operation of the camera 100 performed by the controller 170. Control of the camera may include one or more of the following: controlling optical parameters of the camera, such as shutter speed, or in the case of an adjustable light field camera, controlling the relative distance between the microlens array and the light sensor or the relative distance between the objective lens and the microlens array. In some embodiments, the relative distance between the optical elements of the light field camera can be manually adjusted. Control of the camera may also include controlling other light field data acquisition parameters, light field data formatting parameters, or light field processing parameters of the camera. The user interface 160 may include any suitable user input device, such as a touch screen, a button, a keyboard, a pointing device, and / or the like. In this manner, the input received through the user interface may be used to control and / or configure the LF data formatting module 150 for controlling data formatting, the LF data processor 155 for controlling the processing of the collected light field data, and the controller 170 for controlling the light field camera 100.

[0066] The light field camera includes a power source 180, such as one or more replaceable or rechargeable batteries. The light field camera includes a memory 190 for storing captured light field data and / or processed light field data, or other data such as software for implementing the methods of embodiments of the invention. The memory may include external and / or internal memory. In at least one embodiment, the memory may be provided at a device and / or location separate from the camera 100. In one embodiment, the memory includes a removable / exchangeable storage device, such as a memory stick.

[0067] The light field camera may also include a display unit 165 (e.g., an LCD screen) for viewing the scene in front of the camera before capturing and / or for viewing previously captured and / or rendered images. The screen 165 may also be used to display one or more menus or other information to the user. The light field camera may also include one or more I / O interfaces 195, such as a FireWire or Universal Serial Bus (USB) interface, or for connecting to the camera via the Internet, a cellular data network, a WiFi network, The I / O interface 195 may be used to transmit data, such as light field representative data generated by the LF data formatting module according to an embodiment of the present invention and light field data such as raw light field data or data processed by the LF data processor 155, to and from an external device (such as a computer system or a display unit) for use in a rendering application.

[0068] Figure 2 is a block diagram illustrating a specific embodiment of a possible implementation of the light field data formatting module 250 and the light field data processor 253 .

[0069] The circuit 200 includes a memory 290, a memory controller 245, and a processing circuit 240 including one or more processing units (CPUs). The one or more processing units 240 are configured to run various software programs and / or instruction sets stored in the memory 290 to perform various functions including light field data formatting and light field data processing. According to an embodiment of the present invention, the software components stored in the memory include a data formatting module (or instruction set) 250 for generating data representing the acquired light data, and a light field data processing module (or instruction set) 255 for processing the light field data. Other modules may be included in the memory for applications of the light field camera device, such as an operating system module 251 for controlling general system tasks (e.g., power management, memory management) and for facilitating communication between various hardware and software components of the device 200, and an interface module 252 for controlling and managing communication with other devices via I / O interface ports.

[0070] Embodiments of the present invention provide representations of light field data based on light sensed by pixels of a camera's sensor or light simulated by a computer-generated scene system. In practice, another source of light field data may be modified post-production data, such as color grading, light field data obtained from an optical device or CGI. It is also common in the film industry now to have data that is a mixture of both data acquired using an optical acquisition device and CGI data. It should be understood that the pixels of a sensor may be simulated by a computer-generated scene system, and by extension the entire sensor may be simulated by the system. It should be understood from this that any reference to "pixels of a sensor" or "sensor" may be a physical object attached to an optical acquisition device or a simulated entity obtained by a computer-generated scene system.

[0071] Knowing that whatever the type of acquisition system, for which pixels of its sensors correspond at least to linear light trajectories or rays in space outside the acquisition system, data representing rays in three-dimensional (or 3D) space are computed.

[0072] In the first embodiment, Figure 3 The diagram shows the paths through which the parameters are set parallel to each other and at known depths. and The light rays at the intersection point are At depth The first reference plane P1 at the intersection At depth In this way, given and Through four coordinates Therefore, the light field can be parameterized by a pair of reference planes P1, P2 (also referred to as parameterization planes herein) for parameterization, where each ray is represented as a point (x1, y1, x2, y2,)∈R in the 4D ray space 4 .

[0073] exist Figure 4 In the second embodiment shown, by The ray is parameterized at the intersection of the reference plane P3 and the ray. At depth The normalized vector v providing the direction of the light ray in space has the following coordinates: Because it will Assume that is positive, and it can be known that v x and v y, so we can just use its two first coordinates (v x , v y ) to describe the vector.

[0074] According to the second embodiment, the four coordinates (x3, y3, v x , v y ) to identify the light. Therefore, the light field can be parameterized by a reference plane P3 for parameterization (also referred to herein as the parameterization plane), where each light ray is represented as a point (x3, y3, v3) in the 4D light space. x , v y ,)∈R 4 .

[0075] Parameters representing rays in a 4D ray space are calculated by the light field data formatting module 150 . Figure 5 1 is a flow chart illustrating steps of a method for formatting light field data acquired by the camera 100 according to an embodiment of the present invention. The method is performed by the light field data formatting module 150.

[0076] When using the pinhole model to calibrate the light field acquisition system, the basic distortion-free projection model is given by:

[0077]

[0078] in

[0079] f is the focal length of the main lens of the camera 100,

[0080] ·c u and C v is the coordinate of the intersection of the optical axis of the camera 100 and the sensor,

[0081] ·(X c , Y c , Z c , 1) T is the position of a point in the space sensed by the camera in the camera coordinate system,

[0082] (u, v, 1) T The coordinates in the camera coordinate system are (X c , Y c , Z c , 1) T The coordinates of the projection of the point on the camera sensor in the sensor coordinate system.

[0083] In step 501, the light field data formatting module 150 calculates the light field data represented by the coordinates (u, v, 1) in the sensor coordinate system. TThe coordinates of the vector V in the direction of the light in the space sensed by the pixel of the sensor. In the sensor coordinate system, the coordinates of the vector V are:

[0084] (uc u , vc v , f) T .

[0085] In the pinhole model, the coordinates are (u, v, 1) T The coordinates of the intersection of the light rays sensed by the pixel with the plane placed from the pinhole at coordinate Z1 and parallel to the sensor plane are:

[0086]

[0087] and is calculated during step 502 .

[0088] In case of mixing several acquisitions, i.e. acquiring light field data by different types of cameras, a single coordinate system is used. In this case, the correction of the coordinates of points and vectors should be modified accordingly.

[0089] According to an embodiment of the invention, the set of coordinates defining the rays sensed by the pixels of the camera's sensor and calculated during steps 501 and 502 are reassembled in a map. In another embodiment, the rays are directly calculated by a computer-generated scene system simulating the propagation of rays.

[0090] In an embodiment of the invention, these mappings are associated with the color mapping of the light field data to be transmitted to the receiver. Thus, in this embodiment, for each pixel (u,v) of the camera's sensor, the parameters representing the color data are associated to the light sensed by the given pixel and a 4-tuple of floating point values ​​(χ1 χ2 χ3 χ4) corresponding to (x1,y1,x2,y2,) when the light field can thus be parameterized by a pair of reference planes P1,P2 for parameterization, or to (x3,y3,v4) when the light is parameterized by a normalized vector. x ,v y ,). In the following description, a 4-tuple of floating point values ​​(χ1 χ2 χ3 χ4) is given by:

[0091]

[0092] In another embodiment, the acquisition system is not calibrated using the pinhole model, so the parameterization through two planes is not recalculated from the model. Instead, the parameterization through two planes must be measured during the calibration operation of the camera. This may be the case, for example, of a plenoptic camera, which includes an array of microlenses between the camera's main lens and the sensor.

[0093] In yet another embodiment, these maps are simulated directly by a computer generated scene system, or post-produced from acquired data.

[0094] Since the light sensed by the pixel of the camera's sensor is represented by a floating point quaternion (χ1χ2χ3χ4), the four parameters can be put into four parameter maps, for example, a first map including parameters χ1 for each light sensed by the pixel of the camera's sensor, a second map including parameters χ2, a third map including parameters χ3, and a fourth map including parameters χ4. Each of the four above-mentioned maps (referred to as χ i Map) has the same size as the acquired light field image itself, but has floating point content.

[0095] These four maps can be compressed using similar tools as for video data, after some adaptations to take into account the strong correlations between parameters representing rays sensed by neighboring pixels and to rank groups of rays and therefore the parameters representing them.

[0096] In order to compress the floating point values ​​(χ1 χ2 χ3 χ4) and thus reduce the χ i The size of the mapping, the light field data formatting module 150 in step 503 for each x i Map calculations are included in the χ i The parameter x in the mapping i The parameter χ i The equation for the fitted plane is given by:

[0097]

[0098] where u and v are the coordinates of a given pixel of the camera's sensor.

[0099] In step 504, for each x i Mapping, calculation parameter α i ,β i ,γ i To minimize the error:

[0100]

[0101] The calculation result of step 504 is the parameter:

[0102]

[0103] It corresponds to the parameter χ i The value of the fitting parameter χ i The difference of the plane of the values ​​of i The values ​​in the map have a much lower range in magnitude.

[0104] By calculating in step 505 To compress the value

[0105] Then, in step 506, the previous parameter x can be calculated. i Value Make the parameter value Range from 0 to 2 N -1 (inclusive), where N is the number of bits selected corresponding to the capabilities of the encoder intended to be used to encode the light field data to be transmitted. The value of the parameter Given by:

[0106]

[0107] In step 507, the light field data formatting module 150 generates four mappings: Mapping, Mapping, Mapping and Mapping, corresponding to each of the parameters (χ1 χ2 χ3χ4) representing the light sensed by the pixels of the camera's sensor.

[0108] In step 508, the light field data formatting module 150 generates a SEI (supplemental enhancement information) message including a message intended to be used during reciprocal computation at the receiver side to retrieve the original The following fixed parameters are mapped These four parameters are considered metadata transmitted in the SEI message, the contents of which are given in the following table:

[0109] Table 1

[0110]

[0111] On the receiver side, it is possible to retrieve the original The reciprocal computation of the mapping is given by

[0112] i.

[0113] ii.

[0114] In step 509, The maps, color maps and SEI messages are transmitted to at least a receiver which processes these data in order to render the light field content.

[0115] The size of the maps representing the light field data may be further reduced before they are transmitted to the receiver. The following embodiments are complementary to the embodiment consisting of minimizing the error:

[0116]

[0117] exist Figure 6 In the first embodiment shown, due to The map contains values ​​with low spatial frequencies, so the derivatives of the signal along directions in space can be transmitted.

[0118] For example, given the parameter x associated with the pixel at coordinate (0,0) i Value The light field data formatting module 150 calculates the parameter x associated with the pixel at coordinate (1,0) in step 601. i Value The parameter x associated with the pixel at coordinate (0,0) i Value The difference between

[0119]

[0120] More generally, during step 601, the light field data formatting module 150 calculates the parameter x associated with a given pixel of the sensor. i The value of is associated with the parameter x of another pixel preceding a given pixel in a row of sensors of an optical acquisition system or a computer generated scene system. i The difference between the values ​​of:

[0121]

[0122] When a given pixel is the first pixel of a row of sensors, the light field data formatting module 150 calculates the parameter x associated with the given pixel. i The value of is the same as the parameter x associated with the first pixel of the row before the row to which the given pixel belongs. i The difference between the values ​​of:

[0123]

[0124] In step 602, The map, color map, and SEI message (generated during step 508) are transmitted to at least a receiver which processes these data in order to render the light field content.

[0125] In the second embodiment, since Mapping, Map or The map contains values ​​with very slow spatial frequencies, so The spatial downsampling is performed in the two dimensions of the mapping and then the Linear interpolation is performed between the mapped samples to recover the entire Mapping.

[0126] For example, we can reduce the size of the map from N_rows*M_columns to N_rows / 2*M_columns / 2. At the receiver, the map can be expanded to its original size; the resulting gaps can be filled by interpolation methods (or so-called upsampling). Simple bilinear interpolation is usually sufficient.

[0127]

[0128] exist Figure 7 In the third embodiment shown, four independent monochrome codecs (such as, for example, h265 / HEVC) can be used to convert each Mapping, Map or The map is transmitted to the receiver.

[0129] In the fourth embodiment, Mapping, Map or Maps can be grouped in a single image, such as Figure 8 To achieve this goal, one approach is to reduce the size of the map by a factor of 2 using a subsampling method, as in the second embodiment, and then Mapping, Map or The maps are each concatenated in a quadrant of the image that has the same size as the color map. This approach is often referred to as "frame packing" because it packs multiple frames into a single one. Sufficient metadata should be transmitted, for example in an SEI message, to signal that frame packing has been performed in such a way that a decoder can adequately unpack the frame. A single monochrome codec (such as, but not limited to, for example, h265 / HEVC) can then be used to transmit the packed maps in a single frame.

[0130] In this case, the SEI message as shown in Table 1 must also contain a flag indicating that the frame packing method has been used to pack the four maps into a single one (see Table 1b).

[0131] Table 1b

[0132]

[0133] When several cameras are grouped to form a rig, it is better and more consistent to define a single world coordinate system and 2 parameterized planes common to all cameras. The description message (e.g. SEI) can then contain the common information (representation type, z1 and z2) plus the 4 mappings for each camera ( Mapping, Map or ) are described in Table 2.

[0134] In this case, the use of a monochrome codec may be used to convert the video to a monochrome image, taking into account the multi-view aspect of a configuration such as MPEG Multi-view Video Coding (MVC) or MPEG Multi-view High Efficiency Video Coding (MV-HEVC). Mapping, Map or The map is transmitted to the receiver.

[0135] Table 2

[0136]

[0137] In the fifth embodiment, when the Mapping, Map or When the mapped modification is zero, these Mapping, Map or is marked as skipped and not transmitted to the receiver. In this case, the SEI message contains an indication to the receiver that Mapping, Map or The content of such SEI message is shown in Table 3:

[0138] Table 3

[0139]

[0140] In the sixth embodiment, since Mapping, Map or The corrections to the acquisition system parameters represented in are modified slowly in time, so it is worthwhile to transmit them to the receiver at a frame rate that is slower than the frame rate of the color mapping. Mapping, Map or The transmission frequency must be at least one of the IDR frames.

[0141] In a seventh embodiment, the color map uses a YUV or RGB format and is encoded with a video encoder such as MPEG-4 AVC, h265 / HEVC or h266 or an image encoder such as JPEG, JPEG2000, MJEG. When several cameras are used to capture light field content, the color can be coherently encoded using the MV-HEVC codec.

[0142] Although the present invention has been described above with reference to specific embodiments, the present invention is not limited to the specific embodiments, and modifications will be apparent to those skilled in the art, which are within the scope of the present invention.

[0143] Many additional modifications and variations will suggest themselves to those skilled in the art when referring to the foregoing illustrative embodiments, which are given by way of example only and are not intended to limit the scope of the invention which is determined solely by the appended claims. In particular, different features from different embodiments may be interchanged where appropriate.

Claims

1. A method implemented in a device having a processor and a memory, the method comprising: determining a plurality of light rays sensed by respective pixels of a sensor capturing a light field image, wherein the plurality of light rays have respective trajectories in three-dimensional space and associated color data measured by the respective pixels of the sensor; For each ray, determining a set of trajectory parameters representing the trajectory of the ray and a set of color parameters representing associated color data; arranging the set of trajectory parameters into a parameter map, wherein corresponding parameters in the set of trajectory parameters are arranged into a map in the parameter map; compressing the parameter map; arranging the set of color parameters into a color map; and Compresses the color map.

2. The method according to claim 1, wherein One of the parameter maps is compressed by a standard codec, such as High Efficiency Video Coding (HEVC) or Moving Picture Experts Group (MPEG) Multi-view HEVC (MV-HEVC).

3. The method according to claim 1, wherein One of the color maps is compressed by a standard codec, such as Joint Photographic Experts Group (JPEG) image coding, JPEG 2000 image coding, HEVC, or MV-HEVC.

4. The method according to claim 1, wherein One of the trajectory parameter sets includes two intersection points ((x1, y1) and (x2, y2)) of the corresponding ray and two corresponding reference planes (P1 and P2) in three-dimensional space.

5. The method according to claim 1, wherein One of the trajectory parameter sets includes an intersection point (x3, y3) of the corresponding ray and a reference plane (P3) in the three-dimensional space and a vector (v) indicating a direction of the corresponding ray in the three-dimensional space. x , v y ).

6. The method according to claim 1, wherein Arranging the trajectory parameter set into a parameter map, comprising: The trajectory parameter sets are arranged into a first map, a second map, a third map, and a fourth map, wherein: The first mapping includes first parameters respectively extracted from the trajectory parameter set, The second mapping includes second parameters respectively extracted from the trajectory parameter set, The third mapping includes third parameters respectively extracted from the trajectory parameter set, and The fourth mapping includes fourth parameters respectively extracted from the trajectory parameter sets.

7. The method according to claim 1, wherein The compression of the parameter map comprises: combining the parameter maps into a single image; and The single image is encoded.

8. The method according to claim 1, wherein The compression of the parameter map comprises: For each mapping in the parameter map: fitting a plane to the map, the fitted plane being defined by plane parameters determined by the fitting; encoding a difference between the mapping and the fitted plane; as well as A supplemental enhancement information message is generated including the plane parameters.

9. The method of claim 1, wherein: The sensor is a physical object attached to an optical acquisition device, and the light field image is obtained by the optical acquisition device.

10. The method of claim 1, wherein: The sensor is a simulated entity obtained by a computer-generated scene system, and the light field image is a computer-generated image.

11. The method of claim 1 , wherein: The sensor is partly a physical object attached to an optical acquisition device and partly a simulated entity obtained by a computer-generated scene system, and the light field image is partly obtained by the optical acquisition device and partly a computer-generated image.

12. A device comprising a processor configured to: determining a plurality of light rays sensed by respective pixels of a sensor capturing a light field image, wherein the plurality of light rays have respective trajectories in three-dimensional space and associated color data measured by the respective pixels of the sensor; For each ray, determining a set of trajectory parameters representing the trajectory of the ray and a set of color parameters representing associated color data; arranging the set of trajectory parameters into a parameter map, wherein corresponding parameters in the set of trajectory parameters are arranged into a map in the parameter map; compressing the parameter map; arranging the set of color parameters into a color map; and Compresses the color map.

13. The apparatus according to claim 12, wherein One of the parameter maps is compressed by a standard codec, such as High Efficiency Video Coding (HEVC) or MPEG Multi-view HEVC (MV-HEVC).

14. The apparatus according to claim 12, wherein One of the color maps is compressed by a standard codec, such as JPEG image coding, JPEG2000 image coding, HEVC or MV-HEVC.

15. The apparatus according to claim 12, wherein One of the trajectory parameter sets includes two intersection points of the corresponding ray and two corresponding reference planes in the three-dimensional space.

16. The apparatus according to claim 12, wherein One of the trajectory parameter sets includes an intersection point of the corresponding ray with a reference plane in the three-dimensional space and a vector indicating a direction of the corresponding ray in the three-dimensional space.

17. The apparatus according to claim 12, wherein: Arranging the trajectory parameter set into a parameter map, comprising: The trajectory parameter sets are arranged into a first map, a second map, a third map, and a fourth map, wherein: The first mapping includes first parameters respectively extracted from the trajectory parameter set, The second mapping includes second parameters respectively extracted from the trajectory parameter set, The third mapping includes third parameters respectively extracted from the trajectory parameter set, and The fourth mapping includes fourth parameters respectively extracted from the trajectory parameter sets.

18. The apparatus according to claim 12, wherein The compression of the parameter map comprises: combining the parameter maps into a single image; and The single image is encoded.

19. The apparatus according to claim 12, wherein The compression of the parameter map comprises: For each mapping in the parameter map: fitting a plane to the map, the fitted plane being defined by plane parameters determined by the fitting; encoding a difference between the mapping and the fitted plane; as well as A supplemental enhancement information message is generated including the plane parameters.

20. The apparatus of claim 12, wherein: The sensor is a physical object attached to an optical acquisition device, and the light field image is obtained by the optical acquisition device.

21. The apparatus of claim 12, wherein: The sensor is a simulated entity obtained by a computer-generated scene system, and the light field image is a computer-generated image.

22. The apparatus of claim 12, wherein: The sensor is partly a physical object attached to an optical acquisition device and partly a simulated entity obtained by a computer-generated scene system, and the light field image is partly obtained by the optical acquisition device and partly a computer-generated image.

Images