View-independent multi-camera volumetric capture system

Abstract

Reducing data used during capture in the physical capture volume by selectively activating image capture devices from the virtual view includes setting up a virtual camera to receive information about the physical capture volume and multiple image capture devices in the virtual view and visualize the physical capture volume and multiple image capture devices, providing the virtual camera with a virtual view of the physical capture volume including the ability to move around the physical capture volume and the ability to activate or deactivate each of the multiple image capture devices, calculating a view frustum, which is a region of 3D space within the physical capture volume that appears on the virtual camera's view screen, and defining the virtual camera's view frustum that intersects with the multiple image capture devices defined in the virtual view.

Description

Technical Field

[0004]

[0001]

[0001] This disclosure relates to the ability to process and capture video data and reframe the performance already captured without the need to completely capture new data.

Background Art

[0002]

[0002] Volumetric capture systems generally require the use of multiple cameras to capture a subject with complete coverage. Thus, the system may potentially capture a lot of redundant data depending on how the captured subject is to be used. Further, any change in camera settings such as focal length may require recapturing the entire performance associated with each camera and all the data.

Summary of the Invention

Problems to be Solved by the Invention

[0003] This disclosure provides for reducing the data used during capture by selectively activating cameras from virtual views and enabling reframing of data already captured without the need to recapture new data. <\(0000017\)>

Means for Solving the Problems

[0004] One implementation discloses a system for reducing the data used during capture using a virtual camera. The system includes a visual display for displaying a visual representation of the virtual camera, a processor for controlling the visual representation of the virtual camera, and a volumetric capture system that communicates with the processor and the visual display, the volumetric capture system including a plurality of image capture devices, the virtual camera providing a bird's-eye view and the ability to activate or deactivate each of the plurality of image capture devices, the virtual camera receiving information about the volumetric capture system and the plurality of image capture devices in a virtual world and visualizing the volumetric capture system and the plurality of image capture devices.

[0005]

[0005] In one implementation, the plurality of image capture devices are arranged in a dome-shaped structure. In one implementation, the virtual camera further provides the ability to move around the dome-shaped structure. In one implementation, the virtual camera has several settings associated with it, the settings including aperture, focal length, focus, and other device parameters. In one implementation, the processor includes data for each of the plurality of image capture devices. In one implementation, the processor is configured to calculate a view frustum, which is a region of 3D space within the volumetric capture system that appears on the view screen.

[0006]

[0006] Another implementation discloses a method for reducing the data used during capture in a physical capture volume by selectively activating image capture devices from a virtual view. The method includes: setting up a virtual camera to receive information about the physical capture volume and a plurality of image capture devices in the virtual view and to visualize the physical capture volume and the plurality of image capture devices; providing the virtual camera with the ability to move around the physical capture volume and to activate or deactivate each of the plurality of image capture devices; calculating a viewing frustum which is a region of 3D space within the physical capture volume that appears on the view screen of the virtual camera; and defining the viewing frustum of the virtual camera that intersects with the plurality of image capture devices as defined in the virtual view.

[0007]

[0007] In one implementation, the method further includes the step of arranging the plurality of image capture devices in a dome-shaped structure surrounding the physical capture volume. In one implementation, the method further includes the step of arranging the plurality of image capture devices sequentially on a post. In one implementation, the virtual camera has several associated settings, the several associated settings include aperture, focal length, focus, and other device parameters. In one implementation, the step of defining the view frustum includes the step of defining the view frustum using the several associated settings relating to the virtual camera. In one implementation, the method further includes the step of activating only the image capture devices of the plurality that are intersected by the virtual view frustum. In one implementation, the method further includes the step of deactivating the image capture devices of the plurality that are not intersected by the virtual view frustum.

[0008]

[0008] Further implementations disclose a non-temporary computer-readable storage medium that stores a computer program for reducing the data used during capture in a physical capture volume by selectively activating image capture devices from a virtual view. The computer program includes executable instructions which cause the computer to set up a virtual camera to receive information about the physical capture volume and a plurality of image capture devices in the virtual view and to visualize the physical capture volume and the plurality of image capture devices; to provide the virtual camera with the ability to move around the physical capture volume and to activate or deactivate each of the plurality of image capture devices; to calculate a viewing frustum which is a region of 3D space in the physical capture volume that appears on the view screen of the virtual camera; and to define the viewing frustum of the virtual camera that intersects with the plurality of image capture devices defined in the virtual view.

[0009]

[0009] In one implementation, the non-temporary computer-readable storage medium further includes an executable instruction causing the computer to arrange the plurality of image capture devices in a dome-shaped structure surrounding the physical capture volume. In one implementation, the non-temporary computer-readable storage medium further includes an executable instruction causing the computer to arrange the plurality of image capture devices sequentially on a post. In one implementation, the virtual camera has several associated settings, the associated settings including aperture, focal length, focus, and other device parameters. In one implementation, the executable instruction causing the computer to define the viewing frustum includes an executable instruction causing the computer to define the viewing frustum using the associated settings relating to the virtual camera. In one implementation, the non-temporary computer-readable storage medium further includes an executable instruction causing the computer to activate only the image capture devices among the plurality of image capture devices that are intersected by the virtual view frustum. In one implementation, the non-temporary computer-readable storage medium further includes an executable instruction causing the computer to deactivate the image capture devices among the plurality of image capture devices that are not intersected by the virtual view frustum.

[0010]

[0010] Other features and advantages will also become apparent from this specification, which illustrates aspects of the present disclosure as examples.

[0011]

[0011] By examining the attached drawings in which the same parts are shown by the same reference numerals, the details of this disclosure can be partially gathered with respect to both its structure and operation. [Brief explanation of the drawing]

[0012] [Figure 1A] This is a figure of a volumetric capture system 100 including multiple image capture devices, according to one implementation of the present disclosure. [Figure 1B]This is a diagram of a volumetric capture system communicating with a virtual camera, based on one implementation of the present disclosure. [Figure 1C] This figure shows a physical capture space (e.g., a dome) represented in a virtual 3D world viewport (e.g., in a game) by one implementation of the present disclosure. [Figure 1D] This diagram shows a virtual camera intersecting with a camera defined in the virtual space (through which the user views the scene). [Figure 1E] This diagram shows a view of a virtual camera and the area where the cameras intersect. [Figure 2] This is a block diagram of a system for reducing the data used during capture, as implemented in one of the disclosures. [Figure 3] This is a flowchart of a method, one implementation of the present disclosure, for selectively activating an image capture device from a virtual view, thereby reducing the data used during capture on a physical capture volume and enabling reframing of already captured data without the need to recapture new data. [Figure 4A] This is a diagram of a computer system and user as implemented in this disclosure. [Figure 4B] This is a functional block diagram showing a computer system that hosts a virtual camera application as implemented in this disclosure. [Modes for carrying out the invention]

[0013]

[0021] As described above, conventional volumetric capture systems with multiple cameras can capture a lot of duplicate data, and any change in camera settings may require the recapture of the overall performance and all data associated with each camera.

[0014]

[0022] Certain implementations of the present disclosure provide a system and method that reduce the data used during capture by selectively activating cameras from a virtual view and enable reframing of already captured data without the need to recapture new data.

[0015]

[0023] After reading the following description, the implementation methods of the present disclosure in various implementations and applications will become clear. Although various implementations of the present disclosure are described herein, it should be understood that these implementations are presented only as examples and not by way of limitation. Therefore, the detailed description of various implementations should not be construed as limiting the scope or extension of the present disclosure.

[0016]

[0024] FIG. 1A is a diagram of a volumetric capture system 100 including a plurality of image capture devices 102-118 according to one implementation of the present disclosure. In the implementation shown in FIG. 1A, the plurality of image capture devices 102-118 are arranged in a dome-shaped structure having a capture volume therein. In another implementation, the plurality of image capture devices 102-118 are sequentially arranged on posts. In further implementations, any and all combinations of the arrangements of the plurality of image capture devices 102-118 are used. In one implementation, the image capture device is a video camera having a lens.

[0017]

[0025] FIG. 1B is a diagram of a volumetric capture system 100 communicating with a virtual camera 120 according to one implementation of the present disclosure. In the implementation shown in FIG. 1B, the virtual camera 120 is configured in a personal computer 122 or a similar device (such as a tablet computer), and these are used to control the virtual camera 120.

[0018]

[0026] In one implementation, the virtual camera 120 provides an aerial view of the volumetric capture system 100 that includes the ability to move around the system 100 and the ability to activate or deactivate each of the plurality of image capture devices. The virtual camera 120 has several settings associated with it, and some of the settings include aperture, focal length, focus, and other device parameters (such as lens distortion and image sensor format).

[0019]

[0027] In the implementation shown in FIG. 1B, the virtual camera 120 receives information about the physical volumetric capture system 100 in the virtual world and visualizes the capture system 100 and the physical cameras. To do this, the computer 122 (which controls the virtual camera 120) communicates with the capture system 100 and has data about each of the plurality of image capture devices within the system 100. In one implementation, the computer 122 first calculates a frustum, which is a region of 3D space within the capture volume that appears on the view screen. Thus, the frustum includes only the region that appears on the view screen (such as for a game, movie, etc.). Regions that do not appear on the view screen are not included in the frustum.

[0020]

[0028] In one implementation, the settings for the virtual camera 120 define the frustum of the camera 120 that intersects the physical cameras defined in the virtual scene. In one example, only the physical cameras 124, 126, 128 intersected by the virtual frustum are activated for capture. Thus, in this example, the other physical cameras are deactivated for capture. This can significantly reduce the amount of data captured by significantly reducing the number of physical cameras actually used while obtaining sufficient coverage depending on the type of capture being performed.

[0021]

[0029] Figure 1C shows a physical capture space (e.g., a dome) represented in a virtual 3D world viewport 130 (e.g., in a game) according to one implementation of the present disclosure. In this virtual 3D viewport 130, there exists a virtual camera with a viewing frustum of a specific size, depending on lens parameters (e.g., focal length). This viewing frustum intersects with virtual representations of physical image capture devices 132-142, depending on the viewing direction of the virtual camera. These intersections determine which of the physical image capture devices 132-142 (e.g., image capture devices 134, 138, 140, 142) actually captures any given image data.

[0022]

[0030] In one implementation, each of the physical image capture devices (e.g., image capture devices 132-142 in Figure 1C) includes an image sensor large enough to capture the entire scene of the capture system (e.g., a full-frame sensor) and a lens wide enough. Thus, the physical image capture devices (each having a large image sensor and a wide lens) provide the ability to digitally crop images to different focal lengths. For example, the scene can be cropped from 24mm to 35mm or 50mm while retaining sufficient pixel information so as not to degrade image quality. The data is always captured at full resolution using the widest lens setting. Therefore, by using the virtual camera 120 and changing the camera and lens parameters accordingly, already captured data can be reprocessed to use different parameters. For example, the data can be reprocessed to change the framing of already captured data using a different focal length. Thus, by moving the virtual camera 120 to intersect with other physical cameras from the original capture, it is possible to effectively "re-shoot" using a different viewpoint than that used in the original capture without physically recapturing the physical scene.

[0023]

[0031] Figure 1D shows a virtual camera 150 (through which the user views the scene) that intersects with a camera defined in the virtual space. Cameras 152-158 represent cameras that operate for capture in the physical capture space represented in the virtual 3D world viewport 130.

[0024]

[0032] Figure 1E shows the view 160 of the intersection of the virtual camera 150 and camera 150. This is effectively viewed through the lens of the virtual camera 150 in the 3D viewport 130 of the application controlling the system.

[0025]

[0033] Figure 2 is a block diagram of a system 200 for reducing data used during capture, according to one implementation of the present disclosure. In the implementation shown in Figure 2, system 200 includes a volumetric capture system 210 that communicates with a processor 220 and a visual display 230. In one implementation, the visual display 230 displays a visual representation of a virtual camera 232. In one implementation, the display of the virtual camera 232 is controlled by a processor 220 residing in a personal computer or similar device (such as a tablet computer).

[0026]

[0034] In one implementation, the volumetric capture system 210 includes multiple image capture devices arranged in a dome-shaped structure. In one implementation, a virtual camera 232 provides a bird's-eye view of the volumetric capture system 210, including the ability to move around the system 210 and to activate or deactivate each of the multiple image capture devices. The virtual camera 232 has several settings associated with it, some of which include aperture, focal length, focus, and other device parameters.

[0027]

[0035] In the implementation shown in Figure 2, the virtual camera 232 receives information about the physical volumetric capture system 210 in the virtual world and visualizes the capture system 210 and the physical camera. To do this, the processor 220 (which controls the display of the virtual camera 232 on the visual display 230) communicates with the capture system 210 and has data about each of the multiple image capture devices within the system 210. In one implementation, the processor 220 calculates a viewing frustum, which is the region of 3D space within the capture volume that appears on the view screen. Thus, the viewing frustum includes only the region that appears on the view screen (such as in a game or movie). Regions that do not appear on the view screen are not included in the viewing frustum.

[0028]

[0036] In one implementation, the settings for virtual camera 232 define the viewing frustum of camera 232 that intersects with the physical cameras defined in the virtual scene. In one example, only the physical cameras intersected by the virtual viewing frustum are activated and captured. Therefore, in this example, other physical cameras are stopped and captured. This can significantly reduce the amount of data captured by greatly reducing the number of physical cameras actually used, while still achieving sufficient coverage depending on the type of capture being performed.

[0029]

[0037] In one implementation, System 200 is a system comprised entirely of hardware including one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate / logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. In another implementation, System 200 consists of a combination of hardware and software.

[0030]

[0038] Figure 3 is a flowchart of Method 300, one implementation of the present disclosure, for reducing the data used during capture on a physical capture volume and enabling reframing of already captured data without the need to recapture new data, by selectively activating image capture devices from a virtual view. To do this, multiple image capture devices are arranged in a dome-shaped structure surrounding a physical capture volume. In another implementation, multiple image capture devices are arranged sequentially on a post. Further implementations use any and all combinations of the arrangement of multiple image capture devices. In one implementation, the image capture device is a video camera with a lens.

[0031]

[0039] In the implementation shown in Figure 3, step 310 sets up a virtual camera to receive information about the physical capture volume and multiple image capture devices in a virtual view and visualize the physical capture volume and multiple image capture devices. In one implementation, step 320 provides the virtual camera with a virtual view of the physical capture volume (e.g., a bird's-eye view) which includes the ability to move around the volume and to activate or deactivate each of the multiple image capture devices. The virtual camera has several settings associated with it, some of which include aperture, focal length, focus, and other device parameters.

[0032]

[0040] In one implementation, step 330 calculates the viewing frustum, which is the region of 3D space within the capture volume that appears on the virtual camera's view screen. Therefore, the viewing frustum includes only the region that appears on the view screen (such as in a game or movie). Regions that do not appear on the view screen are not included in the viewing frustum. In step 340, the settings for the virtual camera are used to define the viewing frustum of the virtual camera that intersects with the physical image capture devices defined in the virtual view. In one example, only the physical cameras intersected by the virtual viewing frustum are activated to perform the capture. Therefore, in this example, other physical cameras are stopped to perform the capture. This can significantly reduce the amount of data captured by greatly reducing the number of physical cameras actually used, while still achieving sufficient coverage depending on the type of capture being performed.

[0033]

[0041] Figure 4A shows a computer system 400 and user 402 according to an implementation of the present disclosure. User 402 uses the computer system 400 to implement a virtual camera application 490 for reducing the data used during capture, as illustrated and described with respect to system 200 in Figure 2 and method 300 in Figure 3.

[0034]

[0042] Computer system 400 stores and executes the virtual camera application 490 shown in Figure 4B. Furthermore, computer system 400 can communicate with software program 404. Software program 404 may contain software code for the virtual camera application 490. Software program 404 can be loaded onto an external medium such as a CD, DVD, or storage drive, as will be further described below.

[0035]

[0043] Furthermore, the computer system 400 can be connected to the network 480. The network 480 can be connected in various different architectures, such as a client-server architecture, a peer-to-peer network architecture, or other types of architectures. For example, the network 480 can communicate with a server 485 that coordinates the engine and data used within the virtual camera application 490. Also, the network can be of different types. For example, the network 480 can be the Internet, a local area network or any variation of a local area network, a wide area network, a metropolitan area network, an intranet or extranet, or a wireless network.

[0036]

[0044] Figure 4B is a functional block diagram showing a computer system 400 hosting a virtual camera application 490 according to an implementation of the present disclosure. The controller 410 is a programmable processor that controls the operation of the computer system 400 and its components. The controller 410 loads instructions (for example, in the form of computer programs) from memory 420 or built-in controller memory (not shown), executes these instructions to control the system, and provides, for example, data processing. In its execution, the controller 410 provides a software system to the virtual camera application 490. Alternatively, this service can be implemented as a separate hardware component in the controller 410 or the computer system 400.

[0037]

[0045] Memory 420 temporarily stores data for use by other components of the computer system 400. In one implementation, memory 420 is implemented as RAM. In one implementation, memory 420 also includes long-term or permanent memory such as flash memory and / or ROM.

[0038]

[0046] The storage 430 stores data temporarily or for extended periods for use by other components of the computer system 400. For example, the storage 430 stores data used by the virtual camera application 490. In one implementation, the storage 430 is a hard disk drive.

[0039]

[0047] The media device 440 accepts removable media and reads and / or writes data to the inserted media. In one implementation, for example, the media device 440 is an optical disc drive.

[0040]

[0048] The user interface 450 includes components for receiving user input from the user of the computer system 400 and presenting information to the user 402. In one implementation, the user interface 450 includes a keyboard, mouse, audio speaker, and display. The controller 410 uses the input from the user 402 to coordinate the operation of the computer system 400.

[0041]

[0049] The I / O interface 460 includes one or more I / O ports and connects to corresponding I / O devices such as external storage or supplementary devices (e.g., printers or PDAs). In one implementation, the ports of the I / O interface 460 include ports such as USB ports, PCMCIA ports, serial ports, and / or parallel ports. In another implementation, the I / O interface 460 includes a wireless interface for wireless communication with external devices.

[0042]

[0050] Network interface 470 includes wired and / or wireless network connections such as RJ-45 or “Wi-Fi” interfaces (including, but not limited to, 802.11) that support Ethernet connectivity.

[0043]

[0051] Computer system 400 includes additional hardware and software typical of a computer system (e.g., power, cooling, operating system), but these components are not specifically shown in Figure 4B for simplification. Other implementations may use different configurations of the computer system (e.g., different bus or storage configurations or multiprocessor configurations).

[0044]

[0052] The descriptions herein of the disclosed implementations are made so that a person skilled in the art can implement or utilize this disclosure. A number of modifications to these implementations will be readily apparent to a person skilled in the art, and the principles defined herein can be applied to other implementations without departing from the spirit or scope of this disclosure.

[0045]

[0053] Not all features of each of the embodiments described above are necessarily required in any particular implementation of the Disclosure. Furthermore, the descriptions and drawings presented herein should be understood to represent the subject matter broadly intended by this Disclosure. Moreover, the scope of this Disclosure fully includes other implementations that may be apparent to those skilled in the art, and therefore, the scope of this Disclosure should not be limited by anything other than the appended claims. [Explanation of Symbols]

[0046] 100 Volumetric Capture Systems 102-118 Multiple image capture devices 120 virtual cameras 122 Personal Computers 124,126,128 Physical Cameras 130 Virtual 3D World Viewport 132-142 Physical Image Capture Devices 150 virtual cameras 152~158 Camera 160 views 200 Systems 210 Volumetric Capture System 220 processors 230 Visual Display 232 Virtual Cameras 300 ways 310 Set up a virtual camera to receive information about the physical capture volume and physical camera in the virtual view, and visualize the physical capture volume and physical camera. 320 virtual cameras provide a virtual view of the physical capture volume, including the ability to move around and the ability to activate or deactivate each of the image capture devices. Calculate the viewing frustum, which is the 3D spatial region within the capture volume that appears on the 330 view screen. 340 Use the settings related to the virtual camera to define the view frustum of the camera that intersects with the physical camera defined in the virtual scene. 400 Computer Systems 402 User 404 Software Program 410 Controller 420 memory 430 storage 440 media devices 450 User Interfaces 460 I / O interfaces 470 Network Interfaces 480 Network 485 Servers 490 Virtual Camera Applications

Claims

1. A system for reducing data used during capture using a virtual camera by selectively activating an image capture device from a virtual view, wherein the system: A visual display for displaying the visual representation of the virtual camera, A processor for controlling the visual representation of the virtual camera, A volumetric capture system that communicates with the processor and the visual display, Includes, The volumetric capture system includes a plurality of image capture devices, The aforementioned multiple image capture devices are arranged in a dome-shaped structure, The virtual camera provides a bird's-eye view and the ability to activate or deactivate each of the plurality of image capture devices. The virtual camera receives information about the volumetric capture system and the plurality of image capture devices in the virtual world, visualizes the volumetric capture system and the plurality of image capture devices, The view frustum, which is a region in 3D space within the volumetric capture system that appears on the view screen, is calculated. Of the aforementioned multiple devices, only the image capture device intersected by the viewing frustum is activated. A system characterized by the following features.

2. The system according to claim 1, wherein the virtual camera further provides the ability to move around the dome-shaped structure.

3. The system according to claim 1, wherein the virtual camera has several settings associated with it, the settings including aperture, focal length, focus, and other device parameters.

4. The system according to claim 1, characterized in that the processor includes data for each of the plurality of image capture devices.

5. A method for reducing data used during capture in a physical capture volume by selectively activating an image capture device from a virtual view, wherein the method is: The steps include arranging the plurality of image capture devices in a dome-shaped structure surrounding the physical capture volume, The steps include setting up a virtual camera, receiving information about the physical capture volume and multiple image capture devices in the virtual view, and visualizing the physical capture volume and the multiple image capture devices, The steps include providing the virtual camera with a virtual view of the physical capture volume, which includes the ability to move around the physical capture volume and the ability to activate or deactivate each of the plurality of image capture devices, The steps include calculating a viewing frustum, which is a region of 3D space within the physical capture volume that appears on the view screen of the virtual camera, The steps include defining the viewing frustum of the virtual camera that intersects with the plurality of image capture devices defined in the virtual view, The steps include: activating only the image capture device that is intersected by the viewing frustum among the plurality of devices; A method characterized by including the following.

6. Steps include sequentially arranging the multiple image capture devices on the post, The method according to claim 5, further comprising:

7. The method according to claim 5, characterized in that the virtual camera has several associated settings, the several associated settings include aperture, focal length, focus, and other device parameters.

8. The step of determining the frustum of view is: The step of defining the viewing frustum using the aforementioned associated settings for the virtual camera, including, The method according to claim 7, characterized in that

9. The step of stopping the image capture device among the plurality of devices that is not intersected by the viewing frustum, The method according to claim 5, further comprising:

10. A non-temporary computer-readable storage medium that stores a computer program for reducing the data used during capture on a physical capture volume by selectively activating an image capture device from a virtual view, wherein the computer program includes executable instructions, and the executable instructions are transmitted to the computer. The multiple devices are arranged in a dome-shaped structure surrounding the physical capture volume, The virtual camera is set up to receive information about the physical capture volume and multiple image capture devices in the virtual view, and the physical capture volume and multiple image capture devices are visualized. The virtual camera is provided with a virtual view of the physical capture volume, including the ability to move around the physical capture volume and the ability to activate or deactivate each of the plurality of image capture devices. The process involves calculating the viewing frustum, which is a region in the 3D space within the physical capture volume that appears on the view screen of the virtual camera, To define the viewing frustum of the virtual camera that intersects with the plurality of image capture devices defined in the virtual view, To activate only the image capture device that intersects with the viewing frustum among the aforementioned multiple devices, A non-temporary computer-readable storage medium characterized by the following features.

11. To sequentially place the aforementioned multiple devices on the post, Further includes an executable instruction that causes the computer to perform the following: A computer-readable storage medium according to claim 10, characterized in that...

12. The computer-readable storage medium according to claim 10, characterized in that the virtual camera has several associated settings, the several associated settings include aperture, focal length, focus, and other device parameters.

13. The executable instruction that causes the computer to determine the frustum of view is: Using the aforementioned associated settings for the virtual camera, the viewing frustum is defined. Includes an executable instruction that causes the computer to perform the following: A computer-readable storage medium according to claim 12, characterized in that...

14. To stop the image capture device among the plurality of devices that is not intersected by the viewing frustum, Further includes an executable instruction that causes the computer to perform the following: A computer-readable storage medium according to claim 10, characterized in that...

Images