Reducing Volume Data while Maintaining Visual Fidelity

Abstract

Management body data, including: defining a view volume in a spatial volume, wherein the volume data has multiple points in the spatial volume, and at least one point is in the view volume and at least one point is not in the view volume; defining a grid with multiple cells in the spatial volume and dividing the spatial volume into individual cells, wherein each point has a corresponding cell in the grid, and each cell in the grid has zero or more corresponding points; and for the cells in the grid, reducing the number of points when the cell is outside the view volume.

Description

Technical Field

[0001] The present disclosure relates to volumetric data, and more particularly to reducing volumetric data while maintaining visual fidelity. Background Art

[0002] Volumetric data can be very large, in some cases, nearly hundreds of gigabytes of memory and billions of unique points. Loading and rendering such a large amount of data can be very problematic for real-time performance, especially for virtual production in movies and television, games, and virtual and augmented reality experiences. Summary of the Invention

[0003] The present invention provides a method for reducing volumetric data while maintaining visual fidelity.

[0004] In one implementation, a method for managing volumetric data is disclosed. The method includes: defining a frustum in a spatial volume, wherein the volumetric data has a plurality of points in the spatial volume, and at least one point is in the frustum and at least one point is not in the frustum; defining a grid having a plurality of cells in the spatial volume and dividing the spatial volume into corresponding cells, wherein each point has a corresponding cell in the grid, and each cell in the grid has zero or more corresponding points; and for the cells in the grid, reducing the number of points when the cell is outside the frustum.

[0005] In one implementation, the method further includes keeping the number of points unchanged for the cells inside the frustum. In one implementation, the frustum is a three-dimensional (3-D) box. In one implementation, reducing the number of points for a cell includes merging and spatially filtering the volumetric data to replace a first number of points with a second number of points, wherein the first number is greater than the second number. In one implementation, each point in the second number of points uses a locally averaged position, color, and size. In one implementation, the method further includes defining two or more sub-cells for a cell in the grid, each sub-cell being within the cell.

[0006] In another implementation, a system for managing volumetric data is disclosed. The system includes: a view volume definer for defining a view volume in a spatial volume, where the volumetric data has multiple points in the spatial volume, and at least one point is in the view volume and at least one point is not in the view volume; a grid definer for defining a grid in the spatial volume, the grid having multiple cells, where the spatial volume is divided into corresponding cells, each point has a corresponding cell in the grid, and each cell in the grid has zero or more corresponding points; a processor for receiving the view volume from the view volume definer and the grid from the grid definer; and a point reducer for receiving the view volume and the grid from the processor to reduce the number of points of the volumetric data for the cell when the cell in the grid is outside the view volume, where once the point reducer completes its operation, the processor displays the point-reduced volumetric data.

[0007] In one implementation, the system is a head-mounted virtual reality (VR) device worn by a user, where the VR device is configured to process and display volumetric data for the user to view. In one implementation, the view volume is a 3-D box. In one implementation, the system further includes a combiner and a spatial filter for combining and spatially filtering the volumetric data to replace a first number of points with a second number of points, where the first number is greater than the second number. In one implementation, the combiner and the spatial filter also locally average the position, color, and size of each point in the second number of points. In one implementation, the system further includes a sub-cell definer for defining two or more sub-cells for a cell in the grid, where each sub-cell is within the cell. In one implementation, the sub-cell definer uses a box filter in three dimensions to define the position, color, and size of each point. In one implementation, the sub-cell definer uses a Gaussian filter in three dimensions to define the position, color, and size of each point.

[0008] In another implementation, a non-transitory computer-readable storage medium storing a computer program for managing volumetric data is disclosed. The computer program includes executable instructions that cause a computer to perform the following operations: define a view volume in a spatial volume, where the volumetric data has multiple points in the spatial volume, and at least one point is in the view volume and at least one point is not in the view volume; define a grid having multiple cells in the spatial volume and divide the spatial volume into corresponding cells, where each point has a corresponding cell in the grid, and each cell in the grid has zero or more corresponding points; and reduce the number of points for the cell when the cell in the grid is outside the view volume.

[0009] In one implementation, the computer program further includes executable instructions that cause the computer to perform the following operations: keeping the number of points for cells within the view frustum constant. In one implementation, the view frustum is a three-dimensional (3-D) box. In one implementation, the executable instructions that cause the computer to reduce the number of points corresponding to the cells include executable instructions that cause the computer to perform the following operations: merging and spatially filtering the volume data to replace a first number of points with a second number of points, where the first number is greater than the second number. In one implementation, each of the second number of points uses a locally averaged position, color, and size. In one implementation, the computer program further includes executable instructions that cause the computer to perform the following operations: defining two or more sub-cells for a cell in the grid, each sub-cell being within the cell.

[0010] Other features and advantages should be apparent from this specification, which illustrates aspects of the present disclosure by way of example. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Details of the present disclosure, its structure and operation, can be partly gathered by studying the drawings, in which like reference numerals refer to like parts, and in which:

[0012] Figure 1A is a flowchart of a method for managing volume data according to one embodiment of the present disclosure;

[0013] Figure 1B is a schematic diagram of steps for reducing the number of points for a cell in the grid;

[0014] Figure 2 is a block diagram of a system for managing volume data according to one embodiment of the present disclosure;

[0015] Figure 3A is a representation of a computer system and a user according to an embodiment of the present disclosure; and

[0016] Figure 3B is a functional block diagram of a computer system showing a hosted video application according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] As mentioned above, volume data can be very large. Thus, loading and rendering such large amounts of data can be very problematic for real-time performance, especially for virtual production in movies and television, games, and virtual and augmented reality experiences.

[0018] Some embodiments of the present disclosure provide systems and methods for implementing techniques for processing video data. In one embodiment, a video system creates and manages volume data. The system designates a limited view volume of the volume data. The system uses the limited view volume to reduce the overall volume point count of the data without loss of rendered visual fidelity / quality at any position and orientation within the inner view volume. This reduction allows for reduced data loading times and faster data streaming, as well as faster rendering (visual display) due to processing fewer points. These speed improvements are useful for virtual reality (VR) applications due to the performance requirements of head-mounted displays and for virtual production in film and television, gaming, and virtual and augmented reality experiences.

[0019] After reading the following description, it will become apparent how the present disclosure may be implemented in various embodiments and applications. Although various embodiments of the present disclosure will be described herein, it should be understood that these embodiments are presented by way of example and not limitation. Accordingly, the detailed description of the various embodiments should not be construed as limiting the scope or breadth of the present disclosure.

[0020] In one embodiment, a particular view volume defines a potentially visible region of interest, where visual fidelity decreases as the distance from the inner view volume to any position in the volume space (e.g., a scene in a movie or game) increases. In this embodiment, the input property "minimum point size" can be used to set a constant level of detail for all points within the inner view volume, while for positions outside the inner view volume (i.e., the outer view volume), a varying level of detail can be set by projecting the minimum inner point size based on the distance from the view volume to the outer position away from the view volume boundary. Thus, by defining a particular volume of space that will limit where the user can view the data, various methods can be employed to combine and reduce the data in a way that has the least impact on visual fidelity.

[0021] In one embodiment, the volume data is merged and spatially filtered to replace many points with fewer points having locally averaged positions, colors, and sizes. In other embodiments, any property associated with a point can be filtered at any position in space.

[0022] In one embodiment of a system for managing volume data, a particular spatial view volume is designated as a 3-D box in space at any position, size, and orientation where the volume data is expected to be viewed. In another embodiment, a particular view volume is designated as another shape, such as a 3-D frustum of a hemisphere or rectangular pyramid. Optional settings can be provided for both the inner and outer data of the volume. For example, the minimum point size of the inner data and the resolution of the sample grid of the outer data.

[0023] In one embodiment, a spatial data structure is used to accelerate processing and spatially subdivide a volume to gather locally adjacent points at specific locations in space. The spatial data structure can include a deeper spatial subdivision of points or point locations with attributes, thereby efficiently processing a varying number of highly detailed point clouds with limited memory.

[0024] For example, in a video system, a Uniform Grid is used as the spatial data structure, and the system subdivides a large number of points into "grid cells" for fast searching. In one embodiment, this is used for 3-D filtering of adjacent points. In this embodiment, each 3-D point location is quickly mapped to a cell using a single multiplication and addition operation, and if a cell is subdivided, each cell can have a list of points or sub-grids. Additionally, to avoid having large lists of points in a single cell, the cells can be recursively subdivided and pre-classified to improve performance. In one embodiment, the system defines a specified grid resolution to effectively manage the maximum number of points in each cell. The system subdivides each cell according to the specified grid resolution. Thus, in one embodiment, only adjacent cells are considered for filtering. However, in other embodiments, the system allows the use of filters of any size across multiple cells and grids, for example, to improve quality. The output of the filtered data is a single large list of points, which is volumetrically segmented by the final uniform grid at a lower resolution. The resulting points are then divided into continuous level-of-detail (LOD) data structures for rendering.

[0025] Therefore, in one embodiment, the video system uses the following process to manage volume data. For each point, the following steps are taken: (1) Calculate cell properties at the point, such as minimum, maximum, center; (2) Calculate the distance from the point to the boundary of the nearest view frustum rectangle; (3) If the point is within the box, set the sub-cell to the minimum cell size, otherwise, project the sub-cell outward from the boundary by that distance to get the projected size; (4) Load the sub-cell with data from the main grid; (5) Calculate the list of points in each sub-cell; (6) Calculate the color and size of the final points using a box filter based on the list of points in the final sub-cell; and calculate the list of points in each sub-cell.

[0026] In an alternative embodiment, instead of using a uniform grid for classification, any spatial data structure (e.g., KD-tree, BST, octree, etc.) can be used according to the requirements for runtime performance and memory constraints. Additionally, using different spatial classification systems together can generally provide efficiency improvements compared to using only a single type of classification. For example, it is often beneficial to first classify points into a "coarse" classification using a KD-tree and then perform a "fine" classification using a uniform grid. However, generally speaking, a uniform grid runs quickly because it has good CPU cache coherence and can minimize CPU execution stalls in a multi-threaded environment.

[0027] When calculating the final point size and color in a sub-cell, the system uses any filter type, including a box filter or a Gaussian filter in three dimensions. Non-uniform filters like a Gaussian distribution filter kernel emphasize local properties such as the position, size, and color of the final single-output point attributes, which can improve sharpness at the cost of increasing noise. A three-dimensional filter kernel is used to blend the color and size of points to preserve visual fidelity and appearance. Thus, various filter kernels and sizes can be used to resolve the accurate attributes of a point cloud dataset at any position in three-dimensional space, such as a 3x3x3 box, Gaussian, etc.

[0028] Yet another embodiment includes customizable sampling rates and data filtering and settings for both the internal view volume and the external view volume. In this embodiment, a minimum point size is set for the internal view volume, and the sampling rate is defined according to the uniform grid resolution. The average position of local point clusters is used to retain the sub-voxel positions for each sampled sub-cell. Thus, retaining the sub-voxel positions of the filtered output points in each sub-cell reduces the visual noise artifacts associated with the sampled data on the uniform grid of positions. This improves the visual quality of an animated dataset with slow motion.

[0029] Figure 1A is a flowchart of a method 100 for managing volume data according to an embodiment of the present disclosure. In Figure 1A In the illustrated embodiment, the method includes defining a view volume in a spatial volume at step 110. Thus, in one embodiment, the view volume defines the spatial volume around the location where a player is in a game or movie. In one embodiment, the view volume is a box. In another embodiment, the view volume is a hemisphere. Additionally, the volume data has multiple points in the spatial volume, and at least one point is in the view volume and at least one point is not in the view volume.

[0030] In step 120, a grid is defined in a volume as having a plurality of cells. The volume is divided into individual cells, and each point has a corresponding cell in the grid. Each cell in the grid has zero or more corresponding points. Then in step 130, when a cell is outside the view frustum, the number of points corresponding to that cell in the grid is reduced. Thus, in this embodiment, the count of volume points within the view frustum remains constant while the count of volume points outside the view frustum is reduced. See Figure 1B , which shows step 150 of reducing the number of points corresponding to a cell in the grid.

[0031] In one embodiment, the number of points used for the cell is reduced by merging and spatially filtering the data to replace a first number of points with a second number of points, where the first number is greater than the second number. Each of the second number of points uses a locally averaged position, color, and size. In one embodiment, in step 140, two or more sub-cells are defined for a cell in the grid, where each sub-cell is within the cell.

[0032] As described above, when calculating the size and color of the final point in a sub-cell, the system uses any filter type, including a three-dimensional box filter or a Gaussian filter. A non-uniform filter such as a Gaussian distribution filter kernel emphasizes local attributes such as the position, size, and color of the final single output point attribute, which can improve sharpness at the cost of increased noise. A three-dimensional filter kernel is used to blend the color and size of points to preserve visual fidelity and appearance. Thus, various filter kernels and sizes can be used to resolve the accurate attributes of a point cloud data set at any position in three-dimensional space, such as a 3x3x3 box, Gaussian, etc.

[0033] Figure 2 is a block diagram of a system 200 for managing volume data according to one embodiment of the present disclosure. In one embodiment, the system 200 is a head-mounted virtual reality (VR) device worn by a user, where the VR device is configured to process and display volume data for the user to view. In Figure 2 the illustrated embodiment, the system 200 includes a processor 210 that communicates with a view frustum definer 220, a grid definer 230, a point reducer 240, a display 250, and a sub-cell definer 260. In one embodiment, the sub-cell definer 260 defines two or more sub-cells for a cell in the grid, where each sub-cell is within the cell.

[0034] In one implementation, the view volume definer 220 is configured to define a view volume within the volume space. The defined view volume is transmitted to the processor 210. In one implementation, the view volume is a box. In another implementation, the view volume is a hemisphere. Further, the volume data has a plurality of points within the volume space, with at least one point within the view volume and at least one point outside the view volume.

[0035] In one implementation, the grid definer 230 is configured to define a grid within the volume space as having a plurality of cells. The volume space is divided into individual cells, and each point has a corresponding cell within the grid. Each cell within the grid has zero or more corresponding points. The defined grid is transmitted to the processor 210.

[0036] In one implementation, the point reducer 240 is configured to reduce the number of points corresponding to a cell when the cell within the grid is outside the view volume. Thus, as described above, the count of the volume points within the view volume remains constant while the count of the volume points outside the view volume is reduced. Therefore, this selective "quantity reduction" of this implementation is lossy compared to compression, but retains visual (including size and color) fidelity. The point reducer 240 transmits the result of reducing the number of points corresponding to the cells in the grid to the processor 210.

[0037] In one implementation, the point reducer 240 uses a combiner 242 and a spatial filter 244 to combine and spatially filter the data to replace a first quantity of points with a second quantity of points to reduce the number of points corresponding to the cell, where the first quantity is greater than the second quantity. Each point in the second quantity of points uses local average position, color, and size.

[0038] In one implementation, only adjacent cells are considered for filtering. However, in other implementations, the system 200 allows any filter size across multiple cells and grids, for example to improve quality. The output of the filtered data is a single large list of points, which is volumetrically segmented by a final uniform grid at a lower resolution, and then the resulting points are divided into a continuous level-of-detail (LOD) data structure for rendering.

[0039] As described above, when calculating the size and color of the final points in the sub-cells, the system 200 uses any filter type, including a three-dimensional box filter or a Gaussian filter. Non-uniform filters such as a Gaussian distribution filter kernel emphasize local attributes such as position, size, and color of the final single output point attributes, which can improve sharpness at the cost of increasing noise. The three-dimensional filter kernel is used to blend point colors and sizes to retain visual fidelity and appearance. Thus, various filter kernels and sizes can be used to resolve the accurate attributes of a point cloud data set at any position in three-dimensional space, such as 3x3x3 box, Gaussian, etc.

[0040] Once the dot reducer 240 finishes its operation, the processor 210 displays the volume data with reduced dots on the display 250.

[0041] Figure 3A is a representation of a computer system 300 and a user 302 according to an embodiment of the present disclosure. The user 302 uses the computer system 300 to implement a video application 390 for managing volume data, such as illustrated and described in Figure 1A and Figure 2 the method 100 and system 200 in.

[0042] The computer system 300 stores and executes Figure 3B the video application 390. In addition, the computer system 300 can communicate with a software program 304. The software program 304 can include software code for the video application 390. The software program 304 can be loaded on an external medium such as a CD, DVD, or storage drive, as will be further explained below.

[0043] In addition, the computer system 300 can be connected to a network 380. The network 380 can be connected in various different architectures, for example, a client-server architecture, a peer-to-peer network architecture, or other types of architectures. For example, the network 380 can communicate with a server 385 that coordinates engines and data used within the video application 390. In addition, the network can be different types of networks. For example, the network 380 can be the Internet, a local area network, or any variant of a local area network, a wide area network, a metropolitan area network, an intranet or an extranet, or a wireless network.

[0044] Figure 3B is a functional block diagram of a computer system 300 that hosts a video application 390 according to an embodiment of the present disclosure. The controller 310 is a programmable processor and controls the operation of the computer system 300 and its components. The controller 310 loads instructions (e.g., in the form of a computer program) from the memory 320 or an embedded controller memory (not shown) and executes these instructions to control the system. In its execution, the controller 310 provides a software system to the video application 390 so that, for example, it can create a device group and use a task queue to transmit device setting data in parallel. Alternatively, the service can be implemented as a separate hardware component in the controller 310 or the computer system 300.

[0045] The memory 320 temporarily stores data for use by other components of the computer system 300. In one embodiment, the memory 320 is implemented as RAM. In one embodiment, the memory 320 also includes long-term or permanent memory, such as flash memory and / or ROM.

[0046] The storage device 330 stores data temporarily or long - term for use by other components of the computer system 300. For example, the storage device 330 stores data used by the video application 390. In one embodiment, the storage device 330 is a hard disk drive.

[0047] The media device 340 receives removable media and reads and / or writes data to the inserted media. In one embodiment, for example, the media device 340 is a CD drive.

[0048] The user interface 350 includes components for accepting user input from the user of the computer system 300 and presenting information to the user 302. In one embodiment, the user interface 350 includes a keyboard, a mouse, audio speakers, and a display. The controller 310 uses the input from the user 302 to adjust the operation of the computer system 300.

[0049] The I / O interface 360 includes one or more I / O ports to connect to corresponding I / O devices, such as external memory or auxiliary devices (e.g., printers or PDAs). In one embodiment, the ports of the I / O interface 360 include ports such as: USB ports, PCMCIA ports, serial ports, and / or parallel ports. In another embodiment, the I / O interface 360 includes a wireless interface for wireless communication with external devices.

[0050] The network interface 370 includes wired and / or wireless network connections, such as an RJ - 45 or “Wi - Fi” interface (including but not limited to 802.11) that supports Ethernet connections.

[0051] The computer system 300 includes additional hardware and software of a typical computer system (e.g., power supply, cooling, operating system), although these components are not specifically shown in Figure 3B for simplicity. In other embodiments, different configurations of the computer system (e.g., different bus or storage configurations or multi - processor configurations) may be used.

[0052] A description of the disclosed embodiments is provided herein so that any person skilled in the art can make or use the present disclosure. Many modifications to these embodiments will be apparent to those skilled in the art, and the principles defined herein can be applied to other embodiments without departing from the spirit or scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein, but rather to embrace the widest scope consistent with the primary and novel features disclosed herein.

[0053] In certain embodiments of the present disclosure, all features of each of the examples discussed above are not necessarily required. In addition, it should be understood that the description and drawings presented herein represent the subject matter broadly covered by the present disclosure. It should also be understood that the scope of the present disclosure fully encompasses other embodiments that may become apparent to those skilled in the art, and the scope of the present disclosure is thus limited only by the appended claims.

Claims

1. A method for managing volumetric data, comprising: Defining a view volume in a volume space, wherein the volumetric data has a plurality of points in the volume space, and at least one point is within the view volume and at least one point is outside the view volume; Defining a grid having a plurality of cells in the volume space and dividing the volume space into corresponding cells, wherein each point has a corresponding cell in the grid, and each cell in the grid has zero or more corresponding points; And When a cell in the grid is outside the view volume, reducing the number of points in the cell by merging and spatially filtering the volumetric data to replace a first number of points with a second number of points, while the number of points in the cells within the view volume remains constant, wherein the first number is greater than the second number, wherein the number of points in the cells outside the view volume is reduced by using the local average position, color, and size of each point in the cells outside the view volume.

2. The method according to claim 1, wherein the view volume is a three-dimensional (3-D) box.

3. The method according to claim 1, further comprising Defining two or more sub-cells for a cell in the grid, each sub-cell being within the cell.

4. A system for managing volumetric data, the system comprising: A view volume definer for defining a view volume in a volume space, wherein the volumetric data has a plurality of points in the volume space, and at least one point is within the view volume and at least one point is outside the view volume; A grid definer for defining a grid in the volume space, the grid having a plurality of cells, wherein The volume space is divided into corresponding cells, Each point has a corresponding cell in the grid, and Each cell in the grid has zero or more corresponding points; A processor for receiving the view volume from the view volume definer and the grid from the grid definer; And A point reducer for receiving the view volume and the grid from the processor to reduce the number of points in a cell of the grid of the volumetric data by merging and spatially filtering the volumetric data to replace a first number of points with a second number of points when the cell is outside the view volume, while the number of points in the cells within the view volume remains constant, wherein the first number is greater than the second number, wherein the number of points in the cells outside the view volume is reduced by using the local average position, color, and size of each point in the cells outside the view volume, wherein once the point reducer has completed its operation, the processor displays the volumetric data with reduced points.

5. The system according to claim 4, wherein the system is a head-mounted virtual reality (VR) device worn by a user, and the VR device is configured to process and display the volumetric data for the user to view.

6. The system according to claim 4, wherein the view volume is a 3-D box.

7. The system according to claim 4, further comprising A sub-cell definer for defining two or more sub-cells for a cell in the grid, wherein each sub-cell is within the cell.

8. The system according to claim 7, wherein the sub-cell definer uses a box filter in three dimensions to define the position, color, and size of each point.

9. The system according to claim 7, wherein the sub-cell definer uses a Gaussian filter in three dimensions to define the position, color, and size of each point.

10. A non-transitory computer-readable storage medium storing a computer program for managing volume data, the computer program including executable instructions that cause a computer to perform the following operations: Define a view frustum in a spatial volume, where, The volume data has a plurality of points in a spatial volume, and at least one point is in the viewing volume and at least one point is not in the viewing volume; Define a grid having a plurality of cells in the spatial volume and divide the spatial volume into corresponding cells, wherein each point has a corresponding cell in the grid and each cell in the grid has zero or more corresponding points; And When a cell in the grid is outside the viewing volume, reduce the number of points in the cell by merging and spatially filtering the volume data to replace a first number of points with a second number of points, while the number of points in the cells within the viewing volume remains constant, wherein the first number is greater than the second number, wherein the number of points in the cells outside the viewing volume is reduced by using the local average position, color, and size of each point in the cells outside the viewing volume.

11. The non-transitory computer-readable storage medium according to claim 10, wherein the viewing volume is a three-dimensional (3-D) box.

12. The non-transitory computer-readable storage medium according to claim 10, further including executable instructions that cause a computer to perform the following operation: Define two or more sub-cells for a cell in the grid, each sub-cell being within the cell.

Images