Method and apparatus for creating a atlas comprising a tiled representation of a virtual color cube

Abstract

The present disclosure provides a method and system for processing image content. In one embodiment, the method includes receiving a plurality of captured content showing the same scene with different focal lengths and depth maps as captured by one or more cameras: and generating a consensus cube by obtaining depth map estimates from the received content. Visibility of different objects is then analyzed to create a soft visibility cube that provides visibility information about each content. A color cube is then generated by using information from the consensus cube and the soft visibility cube. The color cube is then used to combine different received content and generate a single image for the received plurality of content.

Description

Technical Field

[0001] This embodiment generally relates to a method and apparatus for processing image content. One or more embodiments relate to compressing volumetric data in image processing, and more specifically to techniques for selecting a subset of data to reduce the size of the volumetric data. Background Technology

[0002] Conventional cameras capture light from a three-dimensional scene onto a two-dimensional sensor device sensitive to visible light. The photosensitive technology used in such imaging devices is typically based on semiconductor technology capable of converting photons into electrons, such as, for example, a CCD (Computer-Coordinated Device) or Complementary Metal-Oxide-Semiconductor (CMOS) technology. For instance, a digital image light sensor typically comprises an array of photosensitive units, each configured to capture incident light. A 2D image providing spatial information is obtained from a measurement of the total amount of light captured by each photosensitive unit of the image sensor device. While the 2D image can provide information about the intensity and color of the light at a spatial point on the light sensor, it does not provide information about the direction of the incident light.

[0003] Recently, other types of cameras have been developed, offering richer and more image-dense products. One such camera is the light field camera. Light field cameras allow for the capture of real-world content from a variety of viewpoints. Two main families of light field cameras are: camera matrices; or all-light cameras. A camera matrix can be replaced by a single camera used to perform multiple acquisitions from various viewpoints. Therefore, the captured light field is limited to a static scene. For an all-light camera, microlenses are positioned between the main lens and the sensor. The microlenses are generating micro-images corresponding to the various viewpoints. The matrix of micro-images collected by the sensor can be converted into so-called sub-aperture images, which are equivalent to the acquired images obtained with the camera matrix. The proposed invention is described in consideration of camera matrices, but the proposed invention is equally applicable to the set of sub-aperture images extracted from an all-light camera.

[0004] Image capture and processing typically involve the generation and storage of volumetric data, even when 2D images are involved. The amount of data increases due to many factors when the image provides more information and is generally of higher quality.

[0005] Therefore, there is a need to provide a technique that allows for the management of data composition to compute specific volumes of data. The goal is to reduce the amount of data while maintaining image quality. Summary of the Invention

[0006] This disclosure provides a method and system for processing image content. In one embodiment, the method includes: receiving a plurality of captured content, the plurality of captured content showing the same scene as a scene captured by one or more cameras with different focal lengths and depth maps; and obtaining a depth map estimate from the received content. In this embodiment, a consensus cube can be generated from the depth map estimate. The visibility of different objects can then be analyzed to provide visibility information about at least one piece of content. The analysis of the object's visibility can be used to create a soft visibility cube that provides the visibility information. The visibility information and the depth map estimate can be used to generate a color cube. The color cube can be generated using information from the consensus cube and the soft visibility cube. The color cube can then be used to combine different received content and generate a single image for the plurality of received content. Attached Figure Description

[0007] Different embodiments of the invention will now be described by way of example only and with reference to the following figures, in which:

[0008] Figure 1 This is a schematic diagram showing a perspective view of the epipolar line according to the embodiment;

[0009] Figure 2 It is an illustration of different continuous slices of the virtual color cube according to the implementation plan;

[0010] Figure 3 This is a diagram illustrating an application of a synthesis algorithm according to one implementation scheme;

[0011] Figure 4 This illustrates the merging of a virtual color cube into a virtual image;

[0012] Figure 5 It is a diagram of the tiling of virtual color cubes according to the implementation plan;

[0013] Figure 6 It is a diagram illustrating the consensus values ​​based on the implementation plan;

[0014] Figure 7 It is a diagram of different thresholds of a virtual image based on an implementation scheme.

[0015] Figure 8 This is a diagram of a table that provides information collected from multiple views.

[0016] Figure 9 It is a diagram showing the fractions of important blocks according to the implementation plan;

[0017] Figure 10 It is based on the flowchart method of the implementation plan; and

[0018] Figure 11It is a block diagram of a device in which one or more implementation schemes can be implemented. Detailed Implementation

[0019] Compared to traditional 2D images, light field imaging and video processing offer a wider range of image processing possibilities. However, capturing high-quality light fields is challenging because a vast amount of data must be captured and managed. Typically, many different views need to be combined, provided within a high dynamic range with excellent color and resolution. Furthermore, 2D images must be processed to project them onto a three-dimensional plane. In digital imaging, this involves providing a grid of planes representing pixels. For each visible point in space, a 2D image typically provides the intensity of one or more pixels. Additionally, other principles associated with stereoscopic image processing must be considered, such as providing two distinct views of a scene. This is because depth management is provided to the user's eye by offering slightly offset images (parallax) for the left and right eyes to create a depth impression. These requirements significantly enhance the visual experience, but they also dramatically increase the amount of data that must be captured, managed, stored, and recovered.

[0020] Figure 1 This is an illustration or stereoscopic image used for depth estimation between the left and right eyes. An epipolar line 100 is provided to provide this depth estimation. Prior to this, it can be fully understood that... Figure 1 These concepts, and those associated with the other figures to be discussed, may be helpful in defining some of the concepts that will be referenced and used when describing different associated embodiments.

[0021] Depth map estimation – Using a light field camera, a given object in a scene can be viewed multiple times under different parallaxes. Therefore, the distance of the object to all cameras can be estimated. A so-called depth map can be derived, where each pixel quantizes the distance of the object visible in the corresponding image captured by a given camera.

[0022] Multi-view Depth (MVD) – MVD specifies a set of images obtained from a camera matrix, plus a corresponding set of depth map images. A depth map is associated with an image, sharing the same spatial resolution and the same viewpoint.

[0023] Point cloud – A point cloud is a set of 3D points in a WCS (Web Center of Statistics). Each 3D point is associated with an RGB color. By feeding each RGB pixel into the WCS, understanding the camera calibration parameters and the corresponding depth, a point cloud can be easily obtained from MVD (Mountain View Depth).

[0024] Image-view compositing describes an algorithm that allows for the computation of an image from a scene observed from a location not yet captured by a camera matrix. The extrinsic and intrinsic parameters of the virtual camera can be freely defined; obviously, if the virtual camera shares the same intrinsic parameters as the real camera, and if the virtual camera is not too far from the real camera, the quality of the synthesized image will be very good.

[0025] Camera calibration – Camera calibration is a set of algorithms and specific images acquired to estimate so-called extrinsic and intrinsic parameters. Extrinsic parameters describe the camera's position in a real-world coordinate system (WCS): three translations characterizing the position of the primary lens pupil center; and three rotation angles characterizing the orientation of the camera's principal optical axis. Intrinsic parameters describe the internal properties of each camera, such as focal length, principal point, and pixel size. Intrinsic parameters may also include geometric distortion produced by the primary lens, which distorts the captured image compared to an ideal thin lens. Many calibration procedures rely on multiple views of the chessboard from different viewpoints.

[0026] Volumetric data is an image or video format that allows users to view realistic content from various positions and orientations. A wide variety of video or image formats process volumetric data. Currently, there is no universal volumetric data format, and it remains an active area of ​​research. To benefit from volumetric data, users are using head-mounted displays (HMDs) that track the head in space, allowing the head's position and orientation to control the position of a virtual camera. In contrast, 360-degree video provides a partial sense of immersion, where only the orientation of the virtual camera can be controlled. 360-degree video cannot reproduce the parallax variations captured by a light field camera. If the virtual camera can be freely positioned and oriented in space, volumetric data is called 6DOF (6 degrees of freedom). If the virtual camera's orientation is restricted to a window, volumetric video is called windowed 6DOF. Windowed 6DOF can also be viewed by a single user in front of a traditional screen. Webcams are being used to track the head to control the position and virtual camera. Vividly, the image displayed on the screen is calculated in real-time based on volumetric data. Depth management is an important concept in image processing.

[0027] Virtual Color Cube - The "virtual color cube" can also be called a multiplane image (MPI).

[0028] Calibrling N cameras is typically complex when using any multi-camera capture system. The N cameras are usually calibrated using, for example, a black and white checkerboard, which all cameras observe simultaneously. Several images are taken with the checkerboard positioned at different locations from the cameras. In each image, the 2D coordinates of the corner points defined by the two black squares and two white squares of the checkerboard are extracted. Based on an image, the 2D coordinates of the corner points are correlated with the 2D coordinates of the same corner points observed by the other cameras.

[0029] Given N 2D coordinates of corner points observed by N cameras and different exposures, the camera positions can be estimated using the World Coordinate System (WCS). In this system, the pupil center of the primary lens of camera i∈[1,N] is determined by the translation vector T. i = (X, Y, Z) t Positioned in space, and the orientation of the optical axis is determined by a 3D rotation matrix R. i Definition. The pose matrix of camera i is defined by P. i =(R i T i )∈R 3×4 Definition. The external matrix of camera i is defined by Q. i =(R i -1 -R i -1 .T i )∈R 3×4 Definitions. Intrinsic camera parameters: focal length; principal point; pixel size; simultaneous estimation of geometric distortion and extrinsic camera parameters.

[0030] Through camera calibration, for any distance z between camera i and a visible object at pixel (x, y), the 2D pixel coordinates (x, y) of camera i can be converted into 3D WCS coordinates (X, Y, Z). t It can also be derived from space (X, Y, Z). t Calculate the coordinates of any point at the pixel (x, y) of camera i.

[0031] return Figure 1 This allows for depth estimation between images simultaneously observed by two cameras. Let x... L (x, y) represents the pixels of the left-hand camera, which observes objects in space X(X, Y, Z). L This is the optical center of the left-hand camera. It lies on the straight line (O). L Any object X on X) i All are the same pixel x L Observed. On the right-hand camera, at coordinate x... r Observed object X iThese coordinates all lie on a line on the sensor, called the epipolar line 100. In one embodiment, the depth associated with a pixel can be estimated using conventional methods, which is performed using the epipolar line. A method according to an embodiment includes one or more of the following.

[0032] Reference camera pixel p ref (x, y) is defined at pixel coordinates (x, y).

[0033] pixel p ref By using the rotation and translation matrices associated with the reference camera, various distance candidates Z c Project the coordinates onto the world coordinate system. Obtain the candidate coordinates P. ref (X c Y c Z c The physical location of ) is determined by these coordinates, which are represented by pixels p. ref (x, y) are observed. For a good depth estimate, the number of candidate S is typically equal to 100. S is also called the number of slices because they define a number of planes that cut through 3D space in parallel slices of the estimated depth map.

[0034] Then, candidate P are selected based on external and internal camera parameters. ref Project the image onto the second camera. Derive the S-coordinate from the second camera. This all depends on the distance from candidate Z. c .

[0035] If p ref (x, y) and The most similar is at pixel p on the reference camera. ref The real physical object P observed at [location] ref distance Equal to distance from candidate Z c .

[0036] Similarity is calculated using various estimators. Two common similarity estimators are listed below:

[0037] The L1 norm between two pixels - Let the observed pixel p be a color pixel defined by three scalars, which correspond to the three color components red, green, and blue (p... R p G p B ). 2 pixels p ref (x, y) and The L1 norm between them is determined by In China, it is said to have The candidates for the smallest L1 norm are identical objects in the observation space. The corresponding Z...c It is related to pixel p ref Associated depth estimation.

[0038] The squared L2 norm between two pixels - similar to the previous norm, but the expected similarity measure is determined by... definition.

[0039] In practice, depth estimation is highly sensitive to noise if only the color component of a single pixel is used to estimate similarity. To overcome this limitation, patches of several surrounding pixels are used to calculate the similarity between two pixels. This technique involves cross-patch depth estimation. Clearly, it requires significantly more computation, as it requires P times more computation for a P×P pixel patch compared to the similarity between two pixels. 2 This requires more computation. This is a key point for real-time estimation, especially when embedded in mobile devices. The similarity operator described above can be used for patches around pixels.

[0040] L1 norm between two patches - let P ref，T (x, y) represents pixel p ref A P×P pixel patch around (x, y) is used, and each pixel is used for a different pixel. Surrounding patches The L1 norm between patches is determined by Definition. From S candidates In the middle, it is said to have p ref Candidates for the minimum L1 norm of (x, y) are identical objects in the observation space. The corresponding Z... c It is related to pixel p ref Depth estimation associated with (x, y).

[0041] In another implementation with a matrix of N cameras, N-1 depth maps are estimated for a given camera. These depth maps can be merged into a single depth map (by averaging, taking the closest data, etc.) so that one depth map is estimated for each camera. At the end of this procedure, N images obtained from the N cameras are associated with N depth maps. As discussed earlier, this data is often referred to as Multi-View Depth (MVD).

[0042] In one implementation, the above concepts can be applied to obtain view composition. View composition refers to the computation of images from a virtual camera located near a camera matrix whose MVD has been observed / computed. For example, in one implementation, the following techniques can be used to obtain view composition.

[0043] Figure 10An implementation scheme for this synthesis is provided. As indicated in step 1000, when multiple contents with different depth maps and focal lengths are received from the camera, as shown in step 1020, the consensus cube is first constructed using the contents, which may include one or more objects and images.

[0044] Step 1010 - Consensus Cube - Using this step, a cube is computed for each input image. This cube quantifies the degree of matching between all depth maps and the viewpoints of the selected input camera for many sampling depths.

[0045] Step 1020 - Soft Visibility Cube - This cube is computed by integrating the consensus cube. The soft visibility cube quantifies, for a camera viewpoint, how visible an object is from a given pixel. Visibility is called "soft" because depth map estimation is prone to error. As for the consensus cube, soft visibility is equivalent to probability.

[0046] Step 1030 - Virtual Color Cube Estimation - Understand the consensus cube and visibility cube of the input image, and estimate the virtual color cube from the virtual camera.

[0047] Step 1040 - Virtual Image Calculation from Virtual Color Cubes: Stacking virtual color cubes to form a single virtual image.

[0048] Some of these steps will now be discussed in more detail below.

[0049] In one implementation, the context of processing volumetric video content is considered, particularly when this content is represented in a format referred to as a virtual color cube. This virtual color cube is a large amount of data (number of slices multiplied by the image size). A prudent approach is to retain only the significant pixels of this virtual color cube in the form of tiles and store them in a tile atlas. As discussed, the method of selecting which tiles to store in the atlas using the concept of residual tiles is an important aspect. The virtual color cube is calculated based on a camera array arranged in a matrix. For example, the image in the technical description comes from a 4×4 camera matrix mounted in a 25cm×25cm setup. How the depth map, consensus cube, virtual color cube, or its conversion into a tile atlas is calculated is beyond the scope of this invention.

[0050] The purpose of a virtual color cube, or its atlas version, is to calculate a virtual view of a scene as seen by a virtual camera located near a real camera used to capture the scene. This involves stacking virtual color cubes into a single image, as described in the technical specification. Figure 4 As shown.

[0051] 1. Consensus Computation - Consensus represents the degree of agreement between a depth map and a given depth map. For a depth map consisting of (N... xN y Each input image I consists of ) pixels i and its corresponding depth map D i Calculate the consensus cube C i Cube C i By (N) x N y The slice consists of S) pixels, where S represents the number of slices. Each slice s∈[1,S] is associated with a distance z, which is related to z. min and z max They change inversely. The minimum and maximum distances are defined based on the scene content and are typically set to the same minimum and maximum distances used to calculate the depth map.

[0052] To define the consensus cube, an impulse function Π(a, b, c) is defined such that:

[0053]

[0054] In addition, the Heaviside H(a, b) function is defined as follows:

[0055]

[0056] At a distance z = D i At slice s associated with (x, y), the consensus value at pixel (x, y) of camera i is equal to:

[0057]

[0058] Where M is the set of cameras used to compute the consensus for camera i. For accurate computation, M is chosen to be equal to the set of all cameras except camera i. D k (x′ k y′ k ) is in pixel coordinates (x′ k y′ k The distance given by the depth map associated with camera k at point (x′) is given. k y′ k The coordinates are calculated using the following steps: 1 / Understand z = D i In the case of (x, y), backproject the pixel coordinates (x, y) from camera i to the WCS at (X, Y, Z); and 2 / project the WCS at (X, Y, Z) to coordinates (x′). k y′ k At camera k at position k, the projection and back projection are calculated using intrinsic and extrinsic camera parameters.

[0059] Consensus is defined as the distance between the object of agreement and the camera being z = D. iThe ratio of the number of cameras at (x, y) to the total number of cameras still visible at a distance of z. Consensus C i The calculation is noisy, especially when most images are occluded beyond a certain distance. In this case, the denominator of equation (3) tends to zero. One option is to set a minimum for the denominator. This minimum was experimentally set to N′ / 4, where N′ is the number of cameras that almost share the field of view. Smooth consensus C i To improve its signal-to-noise ratio, denoising is performed slice-by-slice using a so-called guided denoising algorithm. C is obtained from the consensus of slices s. i The pixels surrounding (x, y, s) and from the observed image I i The local smoothing kernel is calculated by taking the pixels around the pixel at (x, y).

[0060] Soft visibility computation - through consensus on its C according to the following equation i Integrating the slot slices to calculate the given image I i Soft visibility:

[0061]

[0062] The visibility of the first slice is equal to 1, and then decreases to 0. As the visibility decreases towards 0, it means that outside a given slice, image I... i In pixel I i The visible object at (x, y) is occluded. The max() function in equation (4) prevents the visibility from dropping below 0. This often happens because consensus is a protocol among all cameras that can see objects other than the occluded object from view i. It may be equal to the value used to calculate C. i The number of cameras, M.

[0063] Virtual color cube estimation - using observed image I k Set M' computes estimates of the virtual image seen from the virtual camera's location, such that k∈M'. Set M' can be simply defined as the four real cameras closest to the virtual camera. To estimate the virtual image seen from the virtual camera's location, a virtual color cube Color is initially computed. synth (x, y, z). A color cube is located in the coordinate system of a virtual camera, characterized by intrinsic and extrinsic camera parameters. Each slice of this virtual cube is calculated as the average of an M′ image weighted by the corresponding soft visibility.

[0064]

[0065] Similar to equation (3), (x′ k y′ k , z′ kLet represent the back-projected coordinates (x, y, z) from the virtual camera to the real camera k. The greatest advantage of this method is that the integer coordinates (x, y, z) from the virtual color cube are calculated using a reverse warp method, thanks to the cube's sampling of z. The virtual color cube is analogous to a focus stack, where only objects located on a given slice are visible, and foreground objects have been removed.

[0066] Virtual image computation is performed by stacking virtual color cubes—the virtual color cubes are then merged to form a unique virtual color image. First, a consensus cube (Consensus) associated with the virtual color image needs to be computed. synth (x, y, z) and the visibility cube SoftVis synth (x, y, z). Similar to equation (5), the calculation is performed by averaging over the initial consensus or visibility cube of M':

[0067] Consensus synth (x, y, z) = ∑ k∈M′ C k (x′ k y k ′,z′ k (6)

[0068] SoftVis synth (x, y, z) = ∑ k∈M′ SoftVis k (x′ k y′ k , z′ k (7)

[0069] The two cubes defined above are combined to form CC(x, y, z).

[0070] CC(x, y, z) = min(Consensus) synth (x, y, z), SoftVis synth (x, y, z)) (8)

[0071] Virtual Color Cube Estimation - CC is a probability that varies between 0 and 1. Typical values ​​are:

[0072] • If CC(x, y, z) is equal to 1, it means that all cameras agree that the object is located at a distance z from the virtual camera and is visible at coordinates (x, y) within the virtual camera.

[0073] A high value of CC > 50% is rare. It corresponds to an object (texture region) with accurate depth estimation and is precisely located on the slice of the virtual camera and is very close to the slice of the real camera.

[0074] • CC values ​​are mostly equal to 0 because many slices do not match any objects.

[0075] For objects with limited detail, the depth maps extracted from the original images are inconsistent, resulting in low original consensus, which can be as low as 1 / N, where N is the number of cameras. In this case, the consensus coefficient (CC) is also low, approximately 1 / N.

[0076] • For objects located between two slices, the CC value can be less than 1 / N. Therefore, CC values ​​of several percentage points are common.

[0077] Then, the color slices are weighted by consensus and accumulated until the light visibility reaches zero:

[0078]

[0079] In practice, the virtual color cube stores pixels with four values: red-green-blue and alpha (RGBA). RGB encodes the color calculated via equation (5). Alpha encodes the CC(x, y, z) components calculated via equation (8), which illustrates the algorithm applied to images captured using a 4×4 camera matrix. Four consensus cubes and a visibility cube are calculated using 128 slices from four central cameras. All depth maps contribute to the calculation of the consensus and visibility cubes: set M consists of 15 cameras. A composite color cube is calculated using four central cameras: set M' consists of four cameras, showing a detailed view of the four original images (four images on the left) and the composite image (image on the right).

[0080] In such Figure 3 One implementation provided uses a scene composed of complex occlusions to obtain more accurate results. The M' consensus cube and visibility cube require significant memory. Memory consumption can be reduced by applying the full process slice-by-slice. However, caution must be exercised because slices of the virtual color cube will intersect with multiple slices of the consensus cube and visibility cube associated with the original image. Slice-by-slice computation is not feasible for the camera matrix because the cameras are not approximately on the same plane and pointing in the same orientation.

[0081] A matrix of 4×4 cameras, each consisting of 2MPixes; 200 slices for computing the depth map consensus cube and the visibility cube; the computation of a synthetic image is completed in 5 seconds on the GPU, requiring 8GB of memory.

[0082] In the final step of the view compositing algorithm, the virtual color cubes are merged into a single virtual image based on some weights. The left side shows how this merging is done based on 2D coordinates. This step is simple and compatible with real-time rendering. In fact, most of the computation time is spent on the first three steps of the algorithm.

[0083] Using a virtual color cube defined for a given virtual camera position, any other virtual view can be approximated. The strategy is to merge the virtual color cube with any "second" projection, as shown in the right figure. The second projection controls the viewpoint and camera position of the second virtual camera in the final composite image. Therefore, two virtual camera positions are defined, the first virtual camera position P... c Dedicated to calculating the virtual color cube, and the second virtual camera position P i Specifically designed to merge virtual color cubes into freely chosen virtual camera positions. In effect, the virtual camera P... c Located in the middle of the real camera, the virtual camera P i It is controlled by the user's head position to achieve an immersive experience.

[0084] Equation (10) is obtained by using a 4×4 projection matrix P i Modify the projection of the 3D coordinates (x, y, z):

[0085]

[0086] Where [x] p y p , z p ,1]=P i ×[x, y, z, 1]. Projected coordinates (x p y p , z p () is a non-integer, value Color synth (x p y p , z p Estimation is performed through interpolation.

[0087] Equation (10) is modified using pro. The virtual color cube is merged with the tilted projection to produce the result relative to P. c The virtual image is of slightly lower quality compared to the complete computational algorithm. This is in Figure 4 As shown in [the image]. Figure 4In the diagram, left-side rendering 402 provides a virtual image that shares the same virtual camera position as the virtual color cube. Right-side rendering 404 provides an image freely computed from the virtual color cube. However, this implementation allows the computation of the first three steps of the algorithm, up to the computation of the virtual color cube, to be separated from the stacking of the cube into the virtual image. Therefore, real-time rendering can be achieved through the recorded content and some pre-computation of the virtual color cube.

[0088] Next, we can explore how to obtain optimized real-time rendering topics to achieve a new volumetric data format. Virtual color cubes are inherently large. For example, for a camera setup with 4×4 cameras each having 2048×1088 pixels, a virtual color cube typically consists of 128×2048×1088 pixels for 128 slices. Virtual color cubes are also filled with zeros because most slices in the cube do not match the detail of the real content in the scene; note that 90% of the pixels are empty or contribute negligibly: CC(x, y, z) < 10%. As shown, the stacking of color cubes is computed using recent GPU cards at 4Hz. To improve speed by 10x, the virtual color cube is transformed into a new structure with few or no zero pixels. A basic approach is to divide the virtual color cube into fixed-size [T] slices. x T y The tile T in the image i This is in Figure 5 Provided by China.

[0089] exist Figure 5 The image shows the tiling of the virtual color cube to save all tiles into a 2D image. Selecting all tiles of the same size makes the segmentation of the virtual color cube easier. This translates to saving only the important tiles in a single 2D image. Components are saved pixel-by-pixel, RGB color components plus alpha (A) components, indicated by the CC value as described by equation (8). The set of important tiles is named the atlas. Tile T i By [T] x T y Small images composed of pixels and virtual color cubes (x i y i s i The 3D position within ) is used to represent it, where s i In pixel coordinates (x i y i ) and (x i +T x y i +T y The image slices were extracted from the [T] section. As shown in the figure, the [T] section of the image slices... x T yThe pixels are saved into a 2D image. Additionally, a virtual color cube (xi) is created to describe the color. i y i s i The atlas contains indexes of tile positions within a 2D image and their corresponding indexes, representing a subset of the complete virtual color cube.

[0090] The next topic to explore is the tile selection process. The main challenge here is that creating an atlas involves describing the importance of tiles to determine whether they are saved or discarded. A simple method has been implemented: if at least one pixel's CC component is greater than CC... s Threshold coefficient, then patch T i It is important. If CC s Setting it to 0.0 will save all tiles that have at least one pixel and a non-empty CC.

[0091] Figure 6 Important tiles and threshold coefficient CC are shown. s The compression ratio is determined by the following conditions: an setup consisting of N=16 cameras with images of 2048×1088 pixels; a depth map calculated for 200 slices; tiles of [32, 32] pixels; and the scene being the so-called "toy train". The score of the important tiles defines the compression ratio. C r The number of important tiles is divided by the total number of tiles within the virtual color cube. Clearly, the image quality of the virtual image increases with the compression ratio C. r And degenerate.

[0092] - All tiles with non-empty pixels account for 45% of all tiles.

[0093] The desired compression ratio is 15%, resulting in an atlas size of 64MB, comparable to the size of Multi-View Data (MVD) when retaining both the original data (one unsigned character value per pixel) and the depth map (one unsigned character per pixel). This compression ratio is based on CC... s The image was obtained at a resolution of 0.1. The image quality is acceptable, but there is reasonable degradation.

[0094] With a compression ratio of 15%, the computation time for virtual images in the atlas is achieved at a frame rate greater than 30Hz. This frame rate enables real-time computation by controlling the virtual camera position through head tracking.

[0095] The impact of this on image quality can be referenced. Figure 7 To understand. In Figure 7 middle, Figure 1 The virtual image calculated using various thresholds is scaled as follows: A / CC s =0; B / CC s=0.17 and C r =12%; D / CC s =0.3 and C r =9.7%. This allows for tile selection regarding image quality. The main visible effect is that objects appear transparent, especially in areas with poor texture where depth map estimation is less reliable. Figure 7 middle, Figure 1 The following virtual images are shown with three different settings: 1 / Virtual image without tile selection; 2 / Virtual image with CC applied. s The virtual image when selecting tiles with a value of 0.17 corresponds to a compression ratio C. r =12%; and the last 3 / applied CC s =0.3 and compression ratio C r =9.7% of the tiles selected were virtual images. The looseness of the tiles made textureless foreground objects appear partially transparent. In fact, the depth map was inaccurate for these objects, resulting in poor consensus and a low CC factor. Discarded tiles were lost to remove the transparency. The proposed solution provides an innovative approach to eliminating this effect.

[0096] In one implementation, selecting tiles to reduce (lighten) the size of the atlas of virtual color cubes describing the scene, making foreground objects appear partially transparent, may provide the best solution. To address this issue, in one implementation, the solution may retain unselected tiles in the residual image. This approach includes one or more of the following steps:

[0097] 1. Calculate the virtual color cube.

[0098] 2. Set the residual image with the original image size to 0.

[0099] 3. From z min The first slice closer to the camera corresponds to z. max The last slice is used to analyze the slice.

[0100] 4. Divide slice s into [T] x T y A pixel tile, the tile is located at coordinates (t) of the tile grid. x , t y The location corresponds to the pixel coordinates (t) of the virtual color cube. x T x , t y T y ,s). Each tile T i Each one was analyzed separately.

[0101] 5. Block T i The content is equal to that located in (t)x T x , t y T y ,s) and (t) x T x +T x , t y T y +T y The pixels of the virtual color cube between (s) plus the pixels located in (t) x T x , t y T y ) and (t x T x +T x , t y T y +T y The residual image pixels between () and (). If the residual is empty, the tile being analyzed is equal to the pixels of the virtual color cube.

[0102] 6. If block T i At least one pixel within has a CC value greater than CC. s If the condition is met, the tile is saved to the atlas. The corresponding pixel on the residual image is set to 0. Otherwise, tile T... i It is saved to the residual image, replacing the previous pixel values.

[0103] Then, the last three steps (4, 5, and 6) can be repeated until all slices and tiles are resolved. This shows the C# of the retained tiles. r With threshold coefficient CC s The ratio. It can be noted that even for the threshold coefficient CC... s =1, the residual method is also able to preserve some tiles because they will be accumulated until they reach CC=1.

[0104] Figure 9 The virtual image portions computed from the atlas at two different compression ratios were compared using both the commonly used method and the proposed method. It can be noted that, for the same compression ratio, the virtual image computed using the proposed method shows almost no transparency around red objects. The virtual image computed using the residual tiles is nearly identical to the virtual image computed using all tiles (see [link to original text]). Figure 7 , Figure 1 -A).

[0105] Figure 11 This is a diagram illustrating an apparatus in which one or more embodiments of the present disclosure may be implemented. Although in Figure 11The device 5 is depicted to include a camera 1 (such as a light field camera 1 (or 1A, which will be explained in a later section of this specification)), but the light field camera 1 may be configured separately from the device 5. The device 5 may be any device, such as, for example, a desktop or personal computer, a smartphone, a smartwatch, a tablet, a mobile phone, a portable / personal digital assistant (“PDA”), and other devices that facilitate information communication between the end user and the light field camera 1. The light field camera 1 may also have an equivalent hardware configuration to that of the device 5.

[0106] Device 5 includes the following components connected to each other via an address and data bus 54 (which also transmits clock signals): a processor 51 (or CPU), a non-volatile memory of type ROM (read-only memory) 52, a random access memory or RAM 53, a radio interface (RX) 56, an interface 55 (TX) suitable for transmitting data, a light field camera 1, and an MMI (human-machine interface) 58 (I / F appli) suitable for displaying information to the user and / or inputting data or parameters.

[0107] It should be noted that the terms “register” or “storage” used in the description of memories 52 and 53 specify both low-capacity and high-capacity memory areas (such as enabling the entire program to be stored in such memory or representing all or part of the data received and decoded for such memory) in each of the memories mentioned.

[0108] ROM 52 includes a program "prog". One or more steps of the method for implementing embodiments of this disclosure and the algorithms described below are stored in ROM 52 memory and associated with the device 5 that implements these steps. Upon power-up, processor 51 loads and runs the instructions for these algorithms. RAM 53 includes, in registers and / or memory, the following items: an operating program for processor 51 responsible for turning on device 5; receive parameters (e.g., parameters for frame modulation, encoding, MIMO (Multiple-Input Multiple-Output), and recurrence); transmit parameters (e.g., parameters for frame modulation, encoding, MIMO, and recurrence); incoming data corresponding to data received and decoded by radio interface 56; decoded data, formed for transmission at the interface to application program 58; parameters of the main lens 10; and / or information representing the center of the micro-image formed by the microlenses of the microlens array. (Except for information regarding...) Figure 11Beyond the structures described, other structures of device 5 are compatible with this disclosure. Specifically, according to various alternative embodiments, device 5 may be implemented in pure hardware, for example as a dedicated component (e.g., in an ASIC (Application-Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or VLSI (Very Large Scale Integration)) or as several electronic components embedded in the device, or even as a hybrid of hardware and software components. Radio interfaces 56 and 55 are adapted to receive and transmit signals according to one or more telecommunications standards, such as IEEE 802.11 (Wi-Fi), standards conforming to the IMT-2000 specification (also known as 3G), 3GPP LTE (also known as 4G), and IEEE 802.15.1 (also known as Bluetooth). According to alternative embodiments, device 5 does not include any ROM, but only RAM, in which algorithms implementing the steps of the methods specific to this disclosure are stored.

[0109] One or more features of the implementation scheme may be implemented in software, hardware, or a combination thereof. One or more steps of the method according to the invention may be implemented by a processor. One embodiment relates to a computer program product comprising instructions that, when executed by a processor, cause the processor to perform one or more steps of the method of any of the embodiments.

[0110] Although this embodiment has been described above with reference to specific embodiments, this disclosure is not limited to the specific embodiments, and modifications falling within the scope of the claims will be apparent to those skilled in the art.

[0111] When referring to the foregoing exemplary embodiments, many further modifications and variations will arise in those skilled in the art. These exemplary embodiments are given by way of example only and are not intended to limit the scope of the invention, which is defined solely by the appended claims. Specifically, different features from different embodiments may be interchanged where appropriate.

Claims

1. A method for creating an atlas comprising a tiled representation of virtual color cubes, the method comprising: A virtual color cube representing a volumetric scene is obtained from the viewpoint of a virtual camera. The virtual color cube includes slices of the volumetric scene at different depths, each slice including a 2D pixel array, wherein each pixel in each slice includes a color specification and an alpha value. Each slice of the virtual color cube is divided into multiple tiles, and each tile includes a rectangular subset of the pixels of the slice; as well as Construct an atlas of important tiles, where a tile is significant if at least one pixel in the tile has an alpha value greater than a threshold alpha value.

2. The method of claim 1, wherein the atlas further includes an index for each tile indicating the 3D position of the tile within a virtual color cube.

3. The method of claim 2, wherein for each tile in each slice of the virtual color cube, the atlas for constructing important tiles comprises: The alpha value of the pixels in the image patch is compared with the threshold alpha value; as well as The comparison determines whether a tile is included in the atlas or excluded.

4. The method according to claim 3, wherein for excluded tiles, the atlas of important tiles further comprises: a. Store the pixels of the excluded tiles in the residual image at positions corresponding to the positions occupied by the excluded tiles in the original slice of the virtual color cube; b. When processing subsequent slices of the virtual color cube, the stored pixels of the excluded tiles are added to the pixels of the corresponding tiles in the subsequent slice to produce modified pixels, and c. When performing the comparison and determination relative to the corresponding tile in a subsequent slice, the modified pixels are used.

5. The method of claim 1, wherein the atlas for constructing important tiles further comprises processing the slices of the virtual color cube in order of their depth values.

6. The method according to claim 1, wherein the color specification includes R, G and B color values.

7. The method of claim 1, wherein the alpha value of a pixel of a slice reflects the confidence that the pixel represents a portion of the volumetric scene and is accurately located at a depth corresponding to the slice in which the pixel resides.

8. The method of claim 7, wherein the alpha value of the pixel represents a probability.

9. The method of claim 7, wherein the alpha value of the pixel is a value in the range of 0 to 1.

10. The method of claim 1, wherein the alpha value of a pixel within a slice reflects the extent to which the surface of a plurality of cameras capturing the volumetric scene exists within the pixel position of a pixel in the slice.

11. The method of claim 1, wherein each slice of the virtual color cube is partitioned using the same tile pattern.

12. An apparatus for creating an atlas comprising a tiled representation of virtual color cubes, the apparatus comprising: A processor configured to perform the method of any one of claims 1-11.

Images