Method and apparatus for processing image content

Abstract

A method and system for processing image content is provided. The method includes receiving information about a content image captured by at least one camera. The content includes a multi-view representation of the image, including both distorted and undistorted regions. Camera parameters and image parameters are then obtained and used to determine which regions in the image are undistorted and which regions are distorted. This is used to calculate a depth map for the image using the determined undistorted and distorted information. A final stereoscopic image is then rendered using the calculations of the distorted and undistorted regions and the depth map.

Description

Technical Field

[0001] This embodiment generally relates to image processing, and more particularly to using a depth map within an image captured using transmitted camera parameters.

Background Art

[0002] Conventional cameras capture light from a three-dimensional scene on a two-dimensional sensor device sensitive to visible light. The photosensitive technology used in such imaging devices is often based on semiconductor technology that can convert photons into electrons, such as, for example, charge coupled device (CCD) or complementary metal oxide technology (CMOS). A digital image light sensor typically includes, for example, an array of photocells, each cell being configured to capture incident light. A 2D image providing spatial information is obtained from measurements of the total amount of light captured by each photocell of the image sensor device. A 2D image can provide information regarding the intensity of the light and the color of the light at the spatial points of the light sensor, but does not provide information about the direction of the incident light.

[0003] Since visual perception needs to be created afterwards, it is complex to generate 3D or 4D renderings from the captured 2D images. Two important considerations in creating accurate visual recognition need to be done using parallax estimation and depth map calculation. A depth map is an image or image channel that contains information about the distance of the surface of a scene object from the viewpoint. In other words, a depth map is a special image in which each pixel records the distance of the object observed at that position relative to the camera (or the reciprocal of the distance, or any information that is a function of the distance). A depth map can be calculated, for example, using several cameras observing the same field of view, and the depth can be inferred from the variation in parallax between viewpoints. In practice, the estimated depth map shows false pixels. Depth map estimation becomes difficult for many reasons. Some of these difficulties can include objects that are partially masked from one camera to the next, variations in reflected light from objects observed at different positions, surfaces that contain little or no texture that create difficulties in parallax estimation, and sensitivity variations between cameras.

[0004] Parallax estimation and concepts are important in visual perception and can be defined as the displacement or difference in the visual position of an object seen along two different lines of sight, and can be measured by the angle of inclination between those two lines. Each human eye has different and slightly overlapping lines of sight. This concept makes it possible to achieve the perception of depth. Parallax also affects optical devices that view an object from slightly different angles.

[0005] With images and streaming content, providing stereoscopic perception becomes even more complex. Multiple viewpoints of the same scene image captured at different angles may be provided to create appropriate parallax and depth maps. However, storage and processing become difficult because the relevant data is extensive. For example, to provide motion parallax, data regarding multi-viewpoint content is required. The information regarding the content must be dense enough to provide sufficient overlap between viewpoints but also be able to provide an effect at different viewing angles. This is one of the important factors that requires leveraging any compression algorithm to address in order to reduce the amount of data transmitted (taking into account each camera parameter as well). Unfortunately, in the prior art, currently, no easy and practical technology has been provided in this field. As a result, it is desirable to provide a technology that reduces the data captured and is used to provide three-dimensional and four-dimensional perception.

Summary of the Invention

[0006] A method and system for processing image content are provided. The method includes receiving information regarding a content image captured by at least one camera. The content includes a multi-viewpoint representation of an image that includes both distorted and non-distorted regions. Next, camera parameters and image parameters are obtained and used to determine which regions within the image are non-distorted and which regions are distorted. This is used to calculate a depth map of the image using the determined non-distorted and distorted information. Then, a final stereoscopic image is rendered using the calculation of the distorted and non-distorted regions and the depth map.

Brief Description of the Drawings

[0007] Here, by way of mere example, different embodiments will be described with reference to the following drawings.

[0008]

Figure 1

[0009]

Figure 2

[0010]

Figure 3

[0011]

Figure 4

[0012]

Figure 5

[0013]

Figure 6

[0014]

Figure 7

[0015]

Figure 8-1

Figure 8-2

[0016]

Figure 9

[0017]

Figure 10

[0018]

Figure 11

[0019]

Figure 12-1

Figure 12-2

[0020]

Figure 13-1

Figure 13-2

Figure 13-3

[0021]

Figure 14-1

Figure 14-2

Figure 14-3

[0022]

Figure 15-1

Figure 15-2

DETAILED DESCRIPTION OF THE INVENTION

[0023] Most image captures provide two-dimensional images. Different techniques can be used to create three- or four-dimensional renderings of these images. For example, two or more viewpoints of a scene can be captured from different angles for use in its reconstruction, such as when using calibrated or uncalibrated stereo pairs of cameras, or multiple images through a single camera, or when using a light field camera / plenoptic camera.

[0024] To enable the reproduction of multi-dimensional visual perception, the transmitted multi-viewpoint content needs to include relevant information such as valid depth information. When two or more cameras or angles are used, the depth maps of each camera are required in a well-defined MVD or multi-viewpoint and depth format. This information is often transmitted as input in a format used in the extension of technologies such as the High Efficiency Video Coding (HEVC) standard for video compression / decompression.

[0025] As described above, in order to provide motion parallax, multi-viewpoint content must have sufficient overlap between viewpoints but be dense enough to be effective at different viewing angles, which requires a lot of captured information, so compression algorithms are important for reducing the amount of data transmitted. Previously, inter-view prediction was introduced as an extension of the HEVC codec's 3D-HEVC and MV-HEVC. At that time, multi-viewpoint camera systems were mainly regarded as horizontal-only systems, and the prediction mechanism only effectively utilized the horizontal direction. Therefore, the interview difference was defined as the horizontal difference. Using this difference, it was possible to calculate the corresponding pixels at another viewpoint. Current camera arrays are no longer just horizontal, but are 2D or even in a 3D arrangement. Calculating the corresponding pixels of adjacent viewpoints requires more complex processing that must take into account each camera parameter. To address these problems and deficiencies, additional information for characterizing the camera, such as distortion information, should be provided. In one embodiment, a pair of camera modes are introduced that can represent a coefficient matrix for calculating the pixel positions of each viewpoint.

[0026] MPEG-I programs targeting the delivery of content (such as 6DoF content) can enable end-users to move within the content and perceive parallax. The rendered content on the client side needs to be adapted in real-time to the movement of the observer's head. To create this parallax, not only normal 2D content but also content corresponding to what may not be visible at the initial angle but may be visible from different angles when the viewer moves their head needs to be delivered. This content can typically be captured by an array of cameras, with each camera viewing the scene from slightly different angles and positions. The distance between the cameras roughly determines the amount of parallax that the system can provide. The data volume for transmitting multi-viewpoint content in such cases can be comprehensive. Furthermore, several depth maps need to be transmitted in association with the texture so that intermediate viewpoints can be synthesized to correctly render any viewing position. The MVD format has already been used in the past for delivering such content. For example, it has already been used as an input format for the 3D-HEVC extension of HEVC. In this standard, camera parameters were transmitted as SEI messages used on the decoder side.

[0027] In some cases, especially when the rendering is volumetrically comprehensive, camera parameters are essential for accurately calculating the corresponding positions of a given point within the space of any of the input viewpoints. For example, in 3D-HEVC, multi-viewpoint content is provided only from horizontally aligned cameras, which can then be adjusted later. This means that different viewpoints have been pre-processed to have their respective camera principal points on the same grid. This also means that for a given point in space, the distance between their positions in two different viewpoints corresponding to two different cameras is a difference represented only in the horizontal direction.

[0028] When multiple cameras that are not horizontally aligned are used, they are not adjusted without considering any preprocessing such as distortion correction. Some calibration may be desirable, and camera parameters become important. The necessary camera parameters include extrinsic parameters, intrinsic parameters, and distortion parameters.

[0029] Intrinsic parameters address the internal characteristics of the camera, such as its focal length, skew, distortion, and image center. On the other hand, extrinsic parameters describe their overall position and orientation. Knowing the intrinsic parameters makes it possible to estimate the structure of the scene in Euclidean space and is a very important first step for 3D computer vision to remove lens distortion that degrades accuracy. In geometric optics, distortion is the deviation from a straight-line projection, which is a projection where straight lines in the scene maintain straight lines in the image. It is a form of optical aberration.

[0030] FIG. 6 schematically shows a general overview of an encoding and decoding system according to one or more embodiments. The system of FIG. 6 is configured to perform one or more functions. A preprocessing module 30 may be provided to prepare content for encoding by an encoding device 40. The preprocessing module 30 can perform acquisition of multiple images and merging of the acquired multiple images into a common space. Depending on the acquired video data representation, the preprocessing module 30 can perform a mapping space transformation. After being encoded, the data, which can be encoded immersive video data or 3D CGI, can typically be implemented in any network interface present in, for example, a gateway. The data is then transmitted through a communication network such as the Internet, although any other network can also be assumed. The data is then received via a network interface 60 as needed. The network interface 60 can be implemented in a gateway, a television, a set-top box, a head-mounted display device, an immersive (projection) wall, or any immersive video rendering device. After reception, the data is sent to a decoding device 700. Next, the decoded data is processed by a component 80 that can be a player. The data is then prepared for a rendering device 90.

[0031] On the decoder side, camera parameters are extracted from the stream and calculations are performed to calculate the corresponding pixel positions for different viewpoints (for example, for viewpoint prediction in the decoding process). These calculations include matrix multiplications and inverse matrix calculations and can be very computationally intensive. To reduce the complexity of the decoder, it is possible to calculate these camera parameters in advance on the encoder side and send them to the bitstream in an improved way from the perspective of the decoder.

[0032] FIG. 1 provides an exemplary view of a multi-view camera 100. In this example, an array of 16 cameras 110 (a 4-camera by 4-camera base represented as 110), which can be associated with one or more depth maps, is provided. In one embodiment, the depth map may be able to use an 8-bit representation of depth, but this is not essential and may vary in alternative embodiments. Additionally, FIG. 1, however, is provided by way of example only, and other array arrangements with more or fewer camera members may be provided in alternative embodiments. In the example of FIG. 1 including a particular camera array, the overlap between the captured viewpoints is significant and requires important compression steps. In FIGS. 2a and 2b, for ease of understanding, a view of the image is shown in the associated depth maps (referred to as 210 and 220) of FIGS. 2a and 2b.

[0033] FIG. 8 shows a related table (Table 1) that provides an illustration of multi-view acquisition information. This table provides the SEI message syntax in HEVC. The SEI message describes the intrinsic and extrinsic parameters of the camera. Currently, the parameters are required by the decoder to accurately calculate the corresponding positions of a given point in space at any of the viewpoints. Further, the previous description does not include any distortion parameters associated with each camera. The camera model described in the HEVC SEI message only considers non-distorted cameras. It is important to provide an opportunity to describe distortion parameters in order to consider all types of content, which may or may not be distorted.

[0034] Another limitation presented by the prior art is the computational complexity required for these camera parameters to be used in the way they are described (section G.14.2.6 of the HEVC standard). Each value of each rotation or translation matrix is given in scientific notation. This corresponds to a sign (1 bit), an exponent (6 bits), and a mantissa (v bits). The intrinsic parameters (focal length, skew, and principal point) are also described using the same notation. This notation requires some calculations before being used on the decoder side. In an alternative embodiment, it is possible to simplify the decoder-side calculations by sending a parallel 32-bit fixed-point version of these parameters.

[0035] In one embodiment, simplifying the decoder-side calculations may involve removing some of the calculations performed when manipulating the camera parameters. In one embodiment, as will be described later, this entire calculation can be performed in a very accurate manner, and correspondingly, the position of a given point in space from one viewpoint to another can be presented. This enables the extraction of information to transform the position corresponding to one camera to the position corresponding to another camera. In one embodiment, pre-computed matrices can be provided, particularly to simplify the computational complexity required on the decoder side.

[0036] In another embodiment, when there are camera parameters associated with the acquisition of each viewpoint, · Integrating the camera distortion parameters to ensure the use of any type of content (whether distorted or not) · Simplifying the decoder's computational load by proposing pre-computed matrix products to handle the projection and non-projection of pixels from two groups of cameras · Techniques can be used that enable the transmission of camera parameters by proposing pre-computed warp maps and non-warp maps to simplify the calculation of distortion on the decoder side.

[0037] In addition, it is provided in the input format of the encoder to facilitate the understanding of the concepts presenting multi-viewpoint and depth formats. (Multi-viewpoint + depth means that for each viewpoint, RGB content is associated with a depth map at the same pixel resolution. This depth map can be generated by any means (calculation, measurement, etc.) as is known to those skilled in the art. In one embodiment, in order to correctly and effectively utilize such content from multiple cameras, a calibration step is required to determine the relative positions (external parameters) of the cameras, such as the focal length or principal point position, and the individual camera parameters (intrinsic parameters).

[0038] In one embodiment, this calibration step is performed before shooting using a specific test pattern and related software. In order to understand the technologies developed and used in conjunction with some of the embodiments used herein, it is necessary to explore some background materials regarding the compression of multi-viewpoint and depth content information. For this purpose, it is useful to explore examples using different viewpoints of different points in space and calculate the corresponding pixel positions at different viewpoints for at least one of these points in space. In one embodiment, as shown in FIG. 3, for one pixel of one viewpoint, the relevant camera positions can be calculated to determine the corresponding position as if it were obtained by another camera for this point. In this example, the position of this point is P(u, v) of camera c (referred to as 310), which also corresponds to the position P'(u', v') when obtained by camera c' as referred to at 320.

[0039] In this embodiment, when there is information about point P, the intrinsic and external parameters are used to enable the calculation of P'. Consider a camera calibrated as an ordinary pinhole. If its intrinsic matrix is

Number

Number

[0040] In one embodiment,

Number

Number

Number

Number

Number

Number

Number

Number

Number

[0041] For a given camera and the current viewpoint, let #c be its exponent.

Number

Number

[0042] Instead of transmitting the intrinsic and extrinsic matrices to pre-calculate the projection of one pixel onto another viewpoint, it is possible to transmit for each group of two cameras the necessary product of the matrices corresponding to Equation (1). Replace P with P = (R T) and Q with Q = (R^(-1) - R^(-1).T).

Number

[0043] Next, finally, it is described as "Error! Reference source not found." [Number] Regarding storage, the 2x2 camera parameter approach thus requires only a 3x3 matrix A_cc' and a 3x1 vector B_cc' per camera pair. [Number] In the formula, [Number]

[0044] Theoretically, any combination of camera pairs can be transmitted, which means n(n - 1) / 2 sets of information for n cameras. Nevertheless, in one embodiment, predictions of the viewpoints to be decoded in all combinations (using the already decoded viewpoints) are not required. According to the normal dependencies between the encoded viewpoints, only a given number of camera pairs are needed. The number of pairs to be transmitted is likely to be in the order of 2*n instead of n(n - 1) / 2 camera pairs. 2 Theoretically, any combination of camera pairs can be transmitted, which means n(n - 1) / 2 sets of information for n cameras. Nevertheless, in one embodiment, predictions of the viewpoints to be decoded in all combinations (using the already decoded viewpoints) are not required. According to the normal dependencies between the encoded viewpoints, only a given number of camera pairs are needed. The number of pairs to be transmitted is likely to be in the order of 2*n instead of n(n - 1) / 2 camera pairs. 2 The number of pairs to be transmitted is likely to be in the order of 2*n instead of n(n - 1) / 2 camera pairs.

[0045] FIG. 9 provides a table (Table 2) according to one embodiment having an example of 2x2 camera parameters. This table provides calculations and numbers in scientific notation. Also, as noted above, Equation (2) requires an implicit division by z to obtain these homogeneous coordinates. To simplify the calculations performed on the decoder side, this division can be approximated by a shift of a given number of bits (introducing a given rounding error). In this example, a / z is replaced by a / (floor(log2(z))). Embodiment 1b: 2x2 camera parameters, fixed-point representation of data

[0046] In this embodiment, also shown in the table of FIG. 10 (Table 3), instead of representing arbitrary values of both the Acc’ and Bcc’ matrices in scientific notation, it is possible to present such information in fixed-point notation. Thus, the components appearing in this table are modified to indicate the entries provided in this table. However, for the remainder of this document, for any of the remaining embodiments, it should be noted that if the parameters are described in scientific notation, it is possible to consider the specifications of the fixed-point representation of each of these parameters. Subsequently, similar embodiments that handle the same parameters in fixed-point notation may be proposed. Multi-viewpoint content presenting optical distortion.

[0047] The previous description was based on distortion-free content, meaning that the original content from the camera was modified to remove the distortion introduced by the optical system. Here, consider the content without correcting this distortion. The pinhole model cannot provide an exact correspondence due to the geometric distortion that occurs in an actual optical system. First,

Number

Number

Number

Number

[0048] Taking optical distortion into account, the image projection equation becomes as follows. [Number] [Number] represents the forward warping operator induced by distortion. W is usually a polynomial and is thus defined by a set of coefficients in floating - point format. {a k}} k≦N There are various distortion models in the literature. For example, Zhang considers only the first two terms of the radial distortion (Z. Zhang, "A flexible new technique for camera calibration", IEEE Trans. Pattern Analysis & Machine Intelligence, vol.22, no.11, pp.1330~1334, Nov. 2000): [Number] In the formula, [Number] represents the radius of the projection. On the other hand, in his well - known Matlab toolbox (http: / / www.vision.caltech.edu / bouguetj / calib_doc / ), Bouguet uses a higher - performance five - coefficient model that also considers tangential distortion and higher - order radial distortion. [Number] In the formula, [Number] Inverting such a polynomial model results in a rational fraction, which would induce meaningless computational complexity. Approximating the warping without distortion with a polynomial of the same degree is very simple. 1 1The expression "without distortion" corresponds, in the sense of "inverse distortion", to the warping that returns from the distorted light rays (reaching the image sensor of the optical system) to the non-distorted light rays of the entire object.

[0049] Currently, several embodiments for distorted content can be explored here. The first one requires polynomial calculations but restricts the metadata to the most compact form. Subsequently, it improves loop performance but requires pre-computing a warp map without distortion. Embodiment 2: Distortion parameters characterized by polynomial operations

[0050] In this embodiment, based on the model already applied, the number of parameters described by the distortion can vary. The first information to be transmitted is the model to be applied (from the list of known models). The number of parameters is inferred from the model. Both the distortion information and the information without distortion are sent to avoid calculating the coefficients without distortion on the decoding side. From a syntax perspective, the transmission of such information is reflected in Figure 11 (Table 4). Embodiment 3: Distortion parameters characterized by polynomial calculations combined with a 2×2 camera parameter representation

[0051] When considering the distortion formula (1), it is as follows.

Number

Number

Number

Number

Number

Number

Number

Number

Number

Number

[0052] This is shown in FIG. 12. Refer to Table 5 (Embodiment 3, distortion parameters combined with 2×2 camera pair parameters, scientific notation). Embodiment 4: Distorted content using a warp map without distortion combined with 2×2 camera parameter representation

[0053] FIG. 5 is a diagram of distorted content and related distorted mapping. In the previous set of equations, polynomial calculations

Number

Number

Number

[0054] Regarding storage, in this embodiment, for every two cameras, a 3×3 matrix [Number] and a 3×1 vector [Number] In addition, for every one camera, one polynomial W c , one distortion-free map [Number] and one 2×3 matrix [Number] (instead of two polynomials and two 2×3 matrices) are required.

[0055] Note that by pre - calculating the warp map without distortion, it is possible to save half of the polynomial calculations. The warp map can present a lower resolution than the input image. In that case, the warped position is interpolated from the pre - calculated nodes. The subsampling processing factor can be applied in both the horizontal and vertical directions to reduce the amount of information to be transmitted. This is further shown in FIG. 13 as referred to in Table 6.

[0056] In another embodiment, instead of defining the subsampling processing factor of the unwarp map (subsampling processing factor X and subsampling processing factor Y), the horizontal and vertical sizes of the map without distortion are directly transmitted. Table 6 in FIG. 13 is modified as shown in FIG. 14 as referred to as Table 7. It should also be noted that a similar embodiment can be proposed for Embodiment 5 by replacing the subsampling processing factor with the size of the map (for both the map without distortion and the distorted map). Embodiment 5: Distorted content using a distorted warp map and a warp map without distortion in combination with a 2×2 camera parameter representation Also, the warp map can be used to avoid the remaining polynomial calculations by defining

Number

Number

Number

Number

Number

Number

Number

Number

Number

Number

Number

[0057] FIG. 7 is a flowchart diagram of an embodiment. In FIG. 7, at step 700, information is received from a processor or the like around the content captured by at least one camera. Multiple cameras may be used, and the content may include multiple images or the same image from multiple angles. The received information includes, in one embodiment, camera parameters for non-distorted and distorted renderings of the content as shown at 710. Next, at step 720, a matrix is calculated for the camera. At step 730, distortion parameters are obtained to handle the distorted content. At step 740, calculations are performed on the matrix of the camera pair including the distortion parameters. At step 750, a warp map is calculated to simplify the calculation of distortion compensation, and then at step 760, the final image is rendered.

Claims

1. A method comprising: Receiving information regarding content images captured by at least a camera pair, wherein the content images include a multi-viewpoint representation of an image including both distorted content and non-distorted content; Obtaining at least one of camera parameters and image parameters; Using at least one of the camera parameters and the image parameters to obtain distortion information indicating which content within the multi-viewpoint representation is non-distorted and which content is distorted; Calculating a depth map of the image using the distortion information; Rendering a final stereoscopic image using the distortion information and the depth map.

2. The method according to claim 1, wherein the at least one of the camera parameters and the image parameters is used to provide a matrix for the at least camera pair.

3. The method according to claim 1 or 2, wherein the distortion information is provided to obtain distorted content.

4. The method according to claim 3, wherein the distortion information is provided to identify distorted content.

5. The method according to claim 1 or 2, wherein the matrix for the at least camera pair includes distortion parameters.

6. The method according to claim 5, wherein the distortion information is used to provide a distortion compensation value and is used to calculate a warp map.

7. The method according to claim 5, wherein the matrix for the at least camera pair is used to determine a warp map.

8. The method according to claim 7, wherein the warp map is further defined as a refinement of motion vectors.

9. The method according to claim 8, wherein the warp map is associated with a prediction mode (mvd).

10. An apparatus comprising: One or more processors, Receiving information regarding content images captured by at least a camera pair, wherein the content images include a multi-viewpoint representation of an image including both distorted content and non-distorted content; Obtaining at least one of camera parameters and image parameters; Using at least one of the camera parameters and the image parameters, obtain distortion information indicating which content in the multi-viewpoint representation is not distorted and which content is distorted. Using the distortion information, calculate a depth map of the image. Render a final stereoscopic image using the distortion information and the depth map. An apparatus comprising one or more processors configured to perform the above.

11. The apparatus according to claim 10, wherein at least one of the camera parameters and the image parameters is used to provide a matrix for the at least camera pair.

12. The apparatus according to claim 10 or 11, wherein the distortion information is provided to obtain distorted content.

13. The apparatus according to claim 12, wherein the distortion information is provided to identify distorted content.

14. The apparatus according to claim 10 or 11, wherein the matrix for the camera pair includes distortion parameters.

15. The apparatus according to claim 14, wherein the distortion information provides a distortion compensation value and is used to calculate a warp map.

16. The apparatus according to claim 14, wherein the matrix for the at least camera pair is used to determine a warp map.

17. The apparatus according to claim 16, wherein the warp map is further defined as a refinement of the motion vector.

18. The apparatus according to claim 17, wherein the warp map is associated with a prediction mode (mvd).

19. A non-transitory processor-readable medium storing internally instructions for causing a processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory storage medium storing instructions of program code for execution according to the method according to any one of claims 1 to 9. ​

Images