Real-time volume visualization of 2-d images

Abstract

Capturing and visualizing video, including: capturing video data using a plurality of cameras; sending the captured video data to a first shader; computing depth information at the first shader using the captured video data; generating a three-dimensional (3-D) point cloud using the depth information; and rendering a visualization image using the 3-D point cloud.

Description

Technical Field

[0001] This disclosure relates to processing video data, and more specifically, to capturing video data of a subject and environment, and to presenting visualizations of the captured video data while the data is being captured. Background Technology

[0002] Video systems can reconstruct 3D volumetric datasets of objects from captured 2D data using multispectral imaging (e.g., a combination of 2D infrared and color cameras). Traditionally, this reconstruction process is done "offline," and the system does not present images or visualizations to view or verify the final result until the reconstruction is complete, which can be a very time-consuming task. Summary of the Invention

[0003] This disclosure provides for capturing video data and visualizing the captured data while it is being captured.

[0004] In one embodiment, a method for capturing and visualizing video is disclosed. The method includes: capturing video data using multiple cameras; sending the captured video data to a first shader; calculating depth information at the first shader using the captured video data; generating a three-dimensional (3-D) point cloud using the depth information; and rendering a visualization image using the 3-D point cloud.

[0005] In one embodiment, rendering the visualization image includes presenting the visualization image in a 3-D environment. In one embodiment, calculating depth information includes using the distance between two cameras from a plurality of cameras. In one embodiment, the method further includes calculating camera lens distortion to account for lens distortion. In one embodiment, the first shader is a computational shader. In one embodiment, the method further includes applying color information to a 3-D point cloud. In one embodiment, the plurality of cameras includes at least one IR camera and at least one color camera. In one embodiment, the captured video data includes a plurality of IR images. In one embodiment, the method further includes inputting depth information and at least one color image into a second shader. In one embodiment, the method further includes the second shader asynchronously applying color information from at least one color image to associated points in the 3-D point cloud.

[0006] In another embodiment, a system for capturing and visualizing video is disclosed. The system includes: a plurality of 2-D cameras for capturing video data; a first computational shader for receiving the video data and calculating depth information using the received video data, the first computational shader for generating a 2-D depth buffer using the depth information; and a renderer for rendering a visualization image of a 3-D point cloud using the 2-D depth buffer.

[0007] In one embodiment, the system further includes a display for presenting a visualized image in a 3-D environment. In one embodiment, the system further includes a color shader for applying color to a 3-D point cloud. In one embodiment, the plurality of 2-D cameras include at least one IR camera and at least one color camera. In one embodiment, the captured video data includes a plurality of IR images. In one embodiment, the system further includes a second computational shader for receiving a generated 2-D depth buffer and at least one color image, generating a 3-D point cloud from the 2-D depth buffer, and asynchronously applying color from the at least one color image to associated points in the 3-D point cloud.

[0008] In another embodiment, a non-transitory computer-readable storage medium is disclosed that stores a computer program for capturing and visualizing video. The computer program includes executable instructions that cause a computer to perform the following operations: capture video data using multiple cameras; send the captured video data to a first shader; calculate depth information at the first shader using the captured video data; generate a 3D point cloud using the depth information; and render a visualized image using the 3D point cloud.

[0009] In one embodiment, the executable instructions for rendering a visualization image by a computer include executable instructions for rendering the visualization image in a 3-D environment by a computer. In one embodiment, the executable instructions for calculating depth information by a computer include executable instructions for using the distance between two of a plurality of cameras. In one embodiment, the program further includes executable instructions for applying color information to a 3-D point cloud by a computer.

[0010] Other features and advantages will become apparent from this description, which illustrates various aspects of the disclosure by way of example. Attached Figure Description

[0011] Details regarding the structure and operation of this disclosure can be partially gathered by studying the accompanying drawings, wherein the same reference numerals denote the same parts, and wherein:

[0012] Figure 1A This is a flowchart of a method for video capture and visualization according to one embodiment of the present disclosure;

[0013] Figure 1B This is a diagram illustrating the process of calculating depth information from multiple 2-D cameras;

[0014] Figure 1C This is an illustration showing the process of combining 2D images from two IR cameras to generate a 3D image;

[0015] Figure 1DThis is a schematic diagram illustrating the process of combining a 3-D image with a color image to generate a 3-D color image;

[0016] Figure 2 This is a block diagram of a system 200 for video capture and visualization according to one embodiment of the present disclosure;

[0017] Figure 3A This refers to the computer system and user representation according to embodiments of this disclosure; and

[0018] Figure 3B This is a functional block diagram illustrating a managed video capture and visualization application according to an embodiment of the present disclosure. Detailed Implementation

[0019] As mentioned above, the offline reconstruction process of reconstructing 3D data of an object from captured 2D data using multispectral imaging equipment can be a very time-consuming task. Furthermore, the offline process does not present images or visualizations to view or verify the final result until the reconstruction is complete.

[0020] Certain embodiments of this disclosure provide systems and methods for implementing techniques to present reconstructed results or versions of results in real-time or near real-time for better understanding of the captured result data. This technique enables final adjustments before capture to ensure the resulting data is as good as possible.

[0021] After reading the following description, it will become apparent how this disclosure can be implemented in various ways and applications. While various embodiments of this disclosure will be described herein, it should be understood that these embodiments are presented by way of example only and not as a limitation. Therefore, the detailed description of the various embodiments should not be construed as limiting the scope or breadth of this disclosure.

[0022] In one implementation, the video system uses a graphics processing unit (GPU) “computation shader” to sample the image stream at set intervals. This enables the system to construct a three-dimensional (3-D) volumetric point cloud from the captured 2-D images and apply color to the point cloud. The system then renders the point cloud in a 3-D environment. In one implementation, the 3-D environment is then navigated around the visualized dataset using a virtual camera. In one implementation, the video system is used in a video production or studio environment and includes one or more cameras for image capture and one or more computers for processing the camera data.

[0023] In other embodiments, the system is configured to: (1) present a 3-D scene using a virtual camera to facilitate verification of the constructed data; (2) generate a depth map from multiple 2-D images (e.g., using multiple cameras, including an IR camera and a color camera); (3) construct a 3-D point cloud from the generated depth map; and (4) apply colors from the color camera to the 3-D point cloud.

[0024] Figure 1A This is a flowchart of a method 100 for video capture and visualization according to one embodiment of the present disclosure. Figure 1A In the illustrated embodiment, at step 110, video data is captured using multiple cameras (e.g., 2-D cameras), and at step 120, the captured video data is sent to the shader. Then, at step 130, the shader uses the captured video data to calculate depth information.

[0025] In one implementation, such as Figure 1B As shown, method 100 calibrates in Figure 1B The diagram illustrates 2-D cameras (e.g., two IR cameras A and C) whose optical centers are separated by a distance (y) from each other on a line called the baseline, used to calculate depth information from 2-D images (e.g., from two or more images or even just one image). In other embodiments, two color cameras may be used. In one embodiment, camera lens distortion is also accounted for. Method 100 also determines a spatial point for which depth is to be calculated. For both cameras, this point is "visible" in 2-D (e.g., point x for camera A and point x' for camera b). Each camera also has a calibrated focal length (f).

[0026] In one implementation, method 100 first calculates the depth (d) as follows: Where t represents the distance between the two cameras A and C, f represents the focal length, and Z represents the distance from the baseline to the focal point (q) of object 170. Therefore, depth is calculated as the distance between points x and x', and is proportional to the product of t and f, and inversely proportional to the distance Z.

[0027] In one implementation, the above method 100 is used for fast computation in an asynchronous computation shader. Therefore, in situations such as... Figure 1CIn this embodiment shown, method 100 provides two images 180, 184 (e.g., one from IR camera A and the other from IR camera C) as input to an asynchronous computation shader (i.e., a first computation shader). The first computation shader then computes a depth buffer and outputs the result as a 2-D depth buffer. Once the depth information is computed in the form of a 2-D depth buffer at step 130, a 3-D point cloud 186 is generated using the depth information at step 140.

[0028] In one implementation, the generation of the 3-D point cloud involves calculating the three axes (x, y, z) of the point cloud (3-D point cloud) using a depth buffer. The z-position is calculated as follows:

[0029] z = (t*f) / (d*p), where

[0030] d = depth

[0031] t = the distance between two cameras A and C

[0032] f = focal length, and

[0033] p = the point where the z-component is calculated.

[0034] The calculation of x and y positions depends on the camera's focal length (i.e., horizontal field of view (H) and vertical field of view (V)) and resolution (i.e., resolution width (R)). x ) and resolution height (R y The x and y positions are calculated as follows: (x, y)

[0035] x = z / tan(x²), where

[0036] x2 = x1 + p x *(H / R x ),

[0037] x1 = (π – H) / 2,

[0038] y = z * tan(y²) * -1, where

[0039] y2=y1+p y *(V / R y ),

[0040] y1 = 2*π – (V / 2).

[0041] Then, at step 150, a 3-D point cloud rendering visualization image is used, and at step 160, the visualization image is presented in a 3-D environment. In one implementation, such as Figure 1DAs shown, color is applied to a 3-D point cloud 186 using a color image 182 captured by color camera B to produce a color 3-D image 190. By presenting a visualized image in a 3-D environment, users can use a virtual camera to navigate around the resulting volumetric data to examine the scene and data more closely.

[0042] In alternative implementations, the following changes are possible. For example, the computed depth buffer is rendered in a 2-D view (such as a grayscale image) for further visualization and verification. In another example, offline, high-performance cloud processing is used to perform depth construction “near real-time.” In this example, images are captured from a camera, the data is sent to a cloud system for processing, and the resulting point cloud is sent back to the host computer for rendering.

[0043] Figure 2 This is a block diagram of a system 200 for video capture and visualization according to one embodiment of the present disclosure. Figure 2 In the embodiment shown, system 200 includes a plurality of 2-D cameras 210, 212, 214 (including IR cameras 210, 212 and color camera 214), a first shader 220, a second shader 222, and a renderer 240.

[0044] In one implementation, multiple 2-D IR cameras 210, 212 capture video data (i.e., 2-D images) and transmit them to a first shader 220. The first shader 220 then uses the received video data to calculate depth information. Thus, in this implementation, the first shader 220 receives two images (e.g., one from IR camera 210 and the other from IR camera 212) as input.

[0045] In one implementation, the first shader 220 is calibrated by its optical center distance from each other (in... Figure 1B 2-D IR cameras 210 and 212, positioned at a distance (y) from the baseline shown, calculate depth information from 2-D images (e.g., from two or more images or even just one image). In other embodiments, two color cameras may be used. In one embodiment, camera lens distortion is also accounted for. The first shader 220 also determines a point in space where the depth is to be calculated. This point is “visible” in 2-D for both cameras 210 and 212. Each camera also has a calibrated focal length (f). In one embodiment, the first shader 220 is configured as an asynchronous computation shader for fast computation.

[0046] In one implementation, the first shader 220 first calculates the depth (d) as follows: Where t represents the distance between the two cameras 210 and 212, f represents the focal length, and Z represents the distance from the baseline to the focal point (q) of object 170. The first shader 220 then calculates the depth buffer and outputs the result as a 2-D depth buffer to the second shader.

[0047] In one implementation, a 2-D depth buffer output by a first shader 220 and a color image from a color camera 214 are input to a second shader 222. The second shader 222 asynchronously applies colors from the color image to associated points computed from the depth buffer. The output of the second shader 222 is a visualization image, which is a 3-D point cloud with color data.

[0048] In one implementation, renderer 240 then renders the visualization image onto display 250 in a 3-D environment. By rendering the visualization image in a 3-D environment, the user can navigate around the generated volumetric data using a virtual camera to examine the scene and data more closely. Thus, in one implementation, display 250 is coupled to a computing device that includes a processor to process the visualization image and perform the processes necessary to run the virtual camera.

[0049] In an alternative implementation, the camera captures an image and transmits it to a cloud system for processing, and the resulting point cloud is sent back to system 200 by the cloud system for rendering.

[0050] Figure 3A This is a representation of a computer system 300 and a user 302 according to an embodiment of the present disclosure. The user 302 uses the computer system 300 to implement a video capture and visualization application 390 for video capture and visualization, as illustrated and described with respect to method 100 and system 200 in Figures 1 and 2.

[0051] Computer system 300 Store and Execute Figure 3B The video capture and visualization application 390 is described above. Furthermore, the computer system 300 can communicate with software program 304. Software program 304 may include software code for the video capture and visualization application 390. Software program 304 may be loaded onto external media such as a CD, DVD, or storage drive, as will be explained further below.

[0052] Furthermore, computer system 300 can connect to network 380. Network 380 can be connected in various different architectures, such as client-server architecture, peer-to-peer network architecture, or other types of architecture. For example, network 380 can communicate with server 385, which coordinates the engine and data used within video capture and visualization application 390. Moreover, the network can be of different types. For example, network 380 can be the Internet, a local area network (LAN) or any variant of a LAN, a wide area network (WAN), a metropolitan area network (MAN), an intranet or extranet, or a wireless network.

[0053] Figure 3B This is a functional block diagram of a computer system 300 for a managed video capture and visualization application 390 according to an embodiment of the present disclosure. Controller 310 is a programmable processor and controls the operation of the computer system 300 and its components. Controller 310 loads instructions (e.g., in the form of computer programs) from memory 320 or an embedded controller memory (not shown) and executes these instructions to control the system, such as providing data processing to build depth and rendering data to present visualizations. In its execution, controller 310 provides software systems to the video capture and visualization application 390, such as enabling the parallel creation of groups of devices and the transmission of device setup data using task queues. Alternatively, this service may be implemented as a separate hardware component within controller 310 or computer system 300.

[0054] The memory 320 temporarily stores data for use by other components of the computer system 300. In one embodiment, the memory 320 is implemented as RAM. In another embodiment, the memory 320 also includes long-term or permanent memory, such as flash memory and / or ROM.

[0055] Storage device 330 temporarily or permanently stores data for use by other components of computer system 300. For example, memory 330 stores data used by video capture and visualization application 390. In one embodiment, storage device 330 is a hard disk drive.

[0056] Media device 340 receives removable media and reads and / or writes data to inserted media. In one embodiment, for example, media device 340 is an optical disc drive.

[0057] User interface 350 includes components for accepting user input from a user of computer system 300 and presenting information to user 302. In one embodiment, user interface 350 includes a keyboard, mouse, audio speakers, and a display. Controller 310 uses input from user 302 to adjust the operation of computer system 300.

[0058] I / O interface 360 ​​includes one or more I / O ports for connecting to corresponding I / O devices, such as external storage devices or supplementary devices (e.g., printers or PDAs). In one embodiment, the ports of I / O interface 360 ​​include ports such as USB ports, PCMCIA ports, serial ports, and / or parallel ports. In another embodiment, I / O interface 360 ​​includes a wireless interface for wireless communication with external devices.

[0059] Network interface 370 includes wired and / or wireless network connections, such as an RJ-45 or “Wi-Fi” interface (including but not limited to 802.11) that supports Ethernet connectivity.

[0060] Computer system 300 includes typical additional hardware and software of a computer system (e.g., power supply, cooling, operating system), but for simplicity, in Figure 3B These components are not specifically shown. In other implementations, different configurations of the computer system may be used (e.g., different bus or storage configurations or multiprocessor configurations).

[0061] This document provides a description of the disclosed embodiments to enable any person skilled in the art to make or use this disclosure. Many modifications to these embodiments will be apparent to those skilled in the art, and the principles defined herein can be applied to other embodiments without departing from the spirit or scope of this disclosure. For example, embodiments of the systems and methods can be applied and adapted to other applications, such as virtual production (e.g., virtual reality environments) or medical imaging, in addition to video production for film or television. Therefore, this disclosure is not intended to be limited to the embodiments shown herein, but is accorded the widest scope consistent with the principal and novel features disclosed herein.

[0062] In certain embodiments of this disclosure, not all features of each of the examples discussed above are necessarily required. Furthermore, it should be understood that the descriptions and figures presented herein represent a broad range of subjects covered by this disclosure. It should also be understood that the scope of this disclosure fully covers other embodiments that may become apparent to those skilled in the art, and the scope of this disclosure is accordingly limited only by the appended claims.

Claims

1. A method for capturing and visualizing video, comprising: Video data is captured using multiple cameras, each including an optical center and a focal length; Send the captured video data to the first shader; Depth information is calculated using the captured video data at the first shader; Input depth information and at least one color image into the second shader; A 3D point cloud is generated using depth information by a second shader; A second shader asynchronously applies color information from the at least one color image to associated points in the 3D point cloud to generate a 3D point cloud with color data; and Simultaneously with the capture of the video data, a visualization image is rendered using a 3D point cloud containing color data. The depth information is calculated in the following way: Establish a baseline between the optical centers of two of the plurality of cameras; Calculate the first distance between the optical centers of the two cameras; Determine the second distance from the baseline to the object's focal point; and Take the product of the first distance and the focal length, and divide the product by the second distance.

2. The method of claim 1, wherein rendering the visualization image includes presenting the visualization image in a 3-D environment.

3. The method of claim 1, further comprising: Calculate camera lens distortion to account for lens distortion.

4. The method of claim 1, wherein the first shader is a computation shader.

5. The method of claim 1, wherein the plurality of cameras includes at least one IR camera and at least one color camera.

6. The method of claim 1, wherein the captured video data comprises a plurality of IR images.

7. A system for capturing and visualizing video, the system comprising: Multiple 2-D cameras are used to capture video data, each 2-D camera including an optical center and a focal length; The first computation shader receives video data and uses the received video data to calculate depth information. The first computation shader uses depth information to generate a 2D depth buffer; A second computational shader is configured to receive a generated 2-D depth buffer and at least one color image, generate a 3-D point cloud from the 2-D depth buffer, and asynchronously apply colors from the at least one color image to associated points in the 3-D point cloud to generate a 3-D point cloud with color data. as well as A renderer is used to render a visualization image using a 3-D point cloud with color data while the video data is being captured. The depth information is calculated in the following way: Establish a baseline between the optical centers of two of the plurality of 2D cameras; Calculate the first distance between the optical centers of the two 2D cameras; Determine the second distance from the baseline to the object's focal point; and Take the product of the first distance and the focal length, and divide the product by the second distance.

8. The system of claim 7, further comprising: A display used to present visualized images in a 3D environment.

9. The system of claim 7, wherein the plurality of 2-D cameras includes at least one IR camera and at least one color camera.

10. The system of claim 7, wherein the captured video data comprises a plurality of IR images.

11. A non-transitory computer-readable storage medium storing a computer program for capturing and visualizing video, the computer program comprising executable instructions that cause a computer to perform the following operations: Video data is captured using multiple cameras, each including an optical center and a focal length; Send the captured video data to the first shader; Depth information is calculated using the captured video data at the first shader; Input depth information and at least one color image into the second shader; A 3D point cloud is generated using depth information by a second shader; A second shader asynchronously applies color information from the at least one color image to associated points in the 3D point cloud to generate a 3D point cloud with color data; and Simultaneously with the capture of the video data, a visualization image is rendered using a 3D point cloud containing color data. The depth information is calculated in the following way: Establish a baseline between the optical centers of two of the plurality of cameras; Calculate the first distance between the optical centers of the two cameras; Determine the second distance from the baseline to the object's focal point; and Take the product of the first distance and the focal length, and divide the product by the second distance.

12. The non-transitory computer-readable storage medium of claim 11, wherein the executable instructions for rendering a visualization image by a computer include executable instructions for rendering the visualization image in a 3-D environment by a computer.

Images