Progressive morphological lens parameter encoding

Abstract

Automatic transfer between virtual lens parameters and optical lens parameters for an optical camera positioned within a physical scene includes: providing the optical camera with spatial awareness of the physical scene, including the position and orientation of the display screen; creating a feedback loop between the optical camera and the lens to enable the lens to transfer lens settings to scene registration; determining when the lens's focus exceeds the optical focus limit based on the lens settings; and enabling the lens to automatically transfer from using optical lens parameters to using virtual lens parameters via scene registration when the lens's focus moves beyond the optical focus limit.

Description

Technical Field

[0001] This disclosure relates to the processing and capture of video data for a video capture system, and more specifically, to the automatic switching between optical lens parameters and virtual lens parameters. Background Technology

[0002] In a virtual production environment where optical cameras / lenses and combinations of virtual cameras / lenses are used in a virtual scene, along with a physical LED display wall used to display the virtual scene, there are differences in how the optical lens parameters relate to the virtual scene. Therefore, it is difficult to effectively perform optical zoom or focus on objects “inside” the virtual scene because the optical parameters can zoom and focus beyond the flat LED display showing the virtual screen. Summary of the Invention

[0003] This disclosure provides a technique for automatically switching between optical lens parameters and virtual lens parameters based on the physical position of a camera relative to a defined fixed-focus target.

[0004] In one embodiment, a method is disclosed for transferring between virtual lens parameters and optical lens parameters of an optical camera positioned within a physical scene. The method includes: providing the optical camera with spatial awareness of the physical scene, the spatial awareness including the position and orientation of a display screen; creating a feedback loop between the optical camera and a lens to enable the lens to transfer lens settings to scene registration; determining, based on the lens settings, when the lens's focus exceeds the optical focus limit; and, when the lens's focus movement exceeds the optical focus limit, enabling the lens to automatically transfer from using optical lens parameters to using virtual lens parameters via scene registration.

[0005] In one embodiment, spatial perception of the physical scene includes the orientation of the optical camera relative to the display screen. In one embodiment, spatial perception of the physical scene is managed by a processor and storage unit in conjunction with the display screen. In one embodiment, spatial perception of the physical scene includes an indoor positioning system. In one embodiment, the method further includes transmitting virtual lens parameters to a virtual camera system to render a virtual scene on the display screen. In one embodiment, even when using optical lens parameters, virtual lens parameters are continuously transmitted to the virtual camera system so that the virtual camera system can adjust settings for real-time operation. In one embodiment, the virtual camera system provides a virtual camera and lens having the same characteristics as the optical camera. In one embodiment, the method further includes transmitting the position and orientation of the optical camera to the virtual camera system. In one embodiment, the virtual camera system includes an algorithm that converts the optical lens parameters and orientation of the optical camera into converted virtual lens parameters. In one embodiment, the method further includes calculating and displaying at least one of depth of field and automatic exposure using the converted virtual lens parameters.

[0006] In another embodiment, a system for transferring between virtual lens parameters and optical lens parameters is disclosed. The system includes: an optical camera and a lens positioned within a physical scene; a display screen positioned within the physical scene; and a processor for providing the optical camera with spatial awareness of the physical scene, the spatial awareness including the position and orientation of the display screen; creating a feedback loop between the optical camera and the lens to receive lens settings from the lens; determining, based on the lens settings, when the lens's focus exceeds the optical focus limit; and enabling the lens to automatically switch from using optical lens parameters to using virtual lens parameters when the lens's focus moves beyond the optical focus limit.

[0007] In one embodiment, spatial perception of the physical scene includes the orientation of an optical camera relative to the display screen. In one embodiment, a processor manages spatial perception of the physical scene in conjunction with the display screen and storage units. In one embodiment, the system further includes an indoor positioning system to provide spatial perception of the physical scene to the processor. In one embodiment, the system further includes a virtual camera system to render a virtual scene on the display screen using virtual lens parameters. In one embodiment, the virtual camera system provides a virtual camera and lens having the same characteristics as the optical camera. In one embodiment, the virtual camera system includes a conversion function that maps parameters from the optical camera to the virtual camera. In one embodiment, the virtual camera system includes an algorithm that converts optical lens parameters and the orientation of the optical camera into converted virtual lens parameters. In one embodiment, the processor uses the converted virtual lens parameters to calculate and display at least one of depth of field and automatic exposure.

[0008] Other features and advantages will be apparent from this description, which illustrates various aspects of the disclosure by way of example. Attached Figure Description

[0009] Details regarding the structure and operation of this disclosure can be obtained in part from studying the accompanying drawings, wherein the same reference numerals refer to the same parts, and wherein:

[0010] Figure 1 This is a diagram showing an arrangement in which the camera and lens are not optically focused beyond the optical focal plane / focus limit defined by the surface of the display screen;

[0011] Figure 2A This is a block diagram of a video capture system according to one embodiment of the present disclosure;

[0012] Figure 2B This is a diagram illustrating a new setup for seamless virtual and optical lens parameter transfer according to one embodiment of the present disclosure;

[0013] Figure 3 This is a flowchart illustrating a method for seamless virtual and optical lens parameter transfer of a camera positioned in a physical scene according to one embodiment of the present disclosure.

[0014] Figure 4A This refers to the computer system and user representation according to embodiments of this disclosure; and

[0015] Figure 4B This is a functional block diagram illustrating a computer system for a managed lens parameter transfer application according to an embodiment of the present disclosure. Detailed Implementation

[0016] As mentioned above, in a typical virtual production environment using a combination of optical cameras / lenses and virtual cameras / lenses in a virtual scene, along with a physical LED display wall used to display the virtual scene, there are differences in how the optical lens parameters relate to the virtual scene. Therefore, it is difficult to effectively perform optical zoom or focus on objects "inside" the virtual scene. That is, the optical parameters can zoom and focus beyond the flat LED display showing the virtual screen.

[0017] Figure 1 This diagram illustrates setup 100, in which the optical focus of camera 110 and lens 120 does not exceed the optical focal plane / focus limit 140 defined by the surface of display screen 130. In this setup 100, the area of ​​acceptable optical focus 150 can be defined as a predetermined distance range from the optical focus limit 140.

[0018] Some embodiments of this disclosure provide a technique for automatically switching between optical lens parameters and virtual lens parameters based on the physical position of the camera relative to a defined fixed-focus target.

[0019] After reading the following description, it will become apparent how this disclosure can be implemented in various embodiments and applications. While various embodiments of this disclosure will be described herein, it should be understood that these embodiments are given by way of example only and not by way of limitation. Therefore, this detailed description of the various embodiments should not be construed as limiting the scope or breadth of this disclosure.

[0020] The features provided in the implementation may include, but are not limited to, one or more of the following: (a) real-time communication with the camera and the control computer, wherein the computer determines the parameters to be used;

[0021] (b) Built-in scene representation for the camera and / or lens, and the system makes the determination there;

[0022] (c) An indoor positioning system (IPS) for camera position tracking; and (d) an external position tracking system and a communication and / or control processor with the camera and lens.

[0023] Figure 2A This is a block diagram of a video capture system 200 according to one embodiment of the present disclosure. Figure 2A In the embodiment shown, the video capture system 200 includes a camera 210 and lens 220, a display screen 230, and other components for video capture such as a processor 240 and a storage unit 242.

[0024] In one embodiment, as indicated above, camera 210 includes optical lens parameters, virtual lens parameters, and physical scene perception aspects. Camera 210 manages the physical scene perception aspects in conjunction with display 230, processor 240, and storage unit 242. In one embodiment, system 200 also includes an indoor positioning system (IPS) 244 and an automated approach using appropriate parameters (i.e., optical or virtual) in conjunction with processor 240 and storage unit 242.

[0025] Regarding physical scene perception using IPS 244 in conjunction with processor 240, camera 210 is configured to sense its own position relative to physical space and determine a focal target (i.e., the physical location of display screen 130) that cannot be exceeded by focusing and zooming in the scene. When optical zoom or focus on the lens changes and reaches the focal target, the lens automatically shuts down optical operation and works effectively through virtual zoom and focus (while still operating the optical controls). Camera 210 then sends digital parameters (e.g., virtual lens parameters) to system 200, which updates the virtual camera with the corresponding values. Therefore, by using seamless automatic optics and virtual lens parameters, it is possible to focus and zoom in a virtual 3-D scene.

[0026] Figure 2B The diagram shows a new setting 250, which illustrates seamless virtual and optical lens parameter transfer according to one embodiment of the present disclosure. Figure 2B The illustrated implementation includes a camera 210, a lens 220, and a display screen 230. The new setup 250 also includes a processor 240, a storage unit 242, and an IPS 244. Figure 2B In the new 250 setting, seamless virtual and optical lens parameter transfer enables zooming and focusing that goes "out of the screen" but remains "within volume".

[0027] exist Figure 2B In the illustrated embodiment, lens 120 operates similarly to optical-to-digital focusing and zoom in a consumer point-and-shoot camera. However, this embodiment is configured such that the digital version forwards optical lens parameters, rather than a single magnification value. Camera 210 stores a representation of the physical scene. In one embodiment, the representation is stored as "scene registration" as part of its firmware. In another embodiment, the representation is stored in an external storage device, such as in a small hardware unit attached to camera 210, or as part of a completely independent system using high-speed network communication.

[0028] exist Figure 2BIn the illustrated embodiment, as mentioned above, camera 210 (including lens 220) has spatial awareness of the physical scene and the position and orientation of display screen 230, as well as its own orientation relative to display screen 230 and physical space. Changes in lens parameters are transmitted to a control system (on or outside the camera). Additionally, a feedback loop (supported by processor 240, storage unit 242, and IPS 244) is created between camera 210 and lens 220 to enable lens 220 to transmit its settings to scene registration, which in turn transmits parameters for whether optical zoom and focus 280 should be used. The feedback loop system provides progressive morphological lens parameter exchange. When this feedback loop system determines that optical lens parameters can no longer be used because the focus of lens 220 exceeds the optical focus limit 140, lens 220 automatically switches to virtual zoom and focus 270 parameters. These virtual parameters are transmitted to an external virtual camera system that renders a virtual scene 260 on display screen 230. In one implementation, these virtual parameters can always be sent, even when using optical parameters, allowing the virtual camera system to adjust settings in real time as needed.

[0029] In one implementation, the virtual camera system provides a virtual camera and lens with the same characteristics as the optical version. Alternatively, the virtual camera system may include a conversion function that maps parameters from the optical system to the virtual system. Furthermore, the position and orientation of the optical camera are transmitted to the virtual camera system. The virtual camera system includes an algorithm that converts the optical lens parameters and optical camera spatial information into a virtual equivalent. This optical-to-virtual conversion is used, but not limited to, to calculate and display effects such as depth of field (DoF), automatic exposure, and other camera effects. Therefore, when the lens parameters and spatial information of the optical lens and camera are combined with the described virtual equivalent, the virtual camera system can zoom and focus on a virtual 3D object in ways that are impossible or difficult to achieve using only optical solutions.

[0030] Figure 3 This is a flowchart illustrating a method 300 for seamless virtual and optical lens parameter transfer for a camera positioned within a physical scene, according to one embodiment of this disclosure. Figure 3 In the embodiments shown, seamless virtual and optical lens parameter transfer enables zooming and focusing that can operate "outside the screen" but remain "within volume".

[0031] exist Figure 3In the illustrated embodiment, at step 310, spatial perception of the physical scene, including the position and orientation of the display screen 230, is provided to the camera 210 (including lens 220). In one embodiment, spatial perception also includes the orientation of the camera 210 relative to the display screen 230 and the physical space. As described above, the camera 210 manages aspects of physical scene perception in conjunction with the display screen 230, processor 240, and storage unit 242. In one embodiment, the system 200 also includes an indoor positioning system (IPS) 244 and an automated approach using appropriate parameters (i.e., optical or virtual) in conjunction with the processor 240 and storage unit 242.

[0032] In one implementation, a feedback loop is created at step 320 between camera 210 and lens 220 to enable lens 220 to transmit its settings to scene registration. In one implementation, the feedback loop system provides progressive morphological lens parameter exchange by determining at step 330 when optical lens parameters can no longer be used because the focus of lens 220 exceeds the optical focus limit 140. Therefore, when the feedback loop system determines (at step 330) that the focus of lens 220 exceeds the optical focus limit 140, lens 220 automatically switches from using optical lens parameters to using virtual lens parameters using virtual zoom and focus 270 at step 340. These virtual parameters are transmitted to an external virtual camera system that renders a virtual scene 260 on display screen 230. In one implementation, these virtual parameters can always be sent, even when using optical parameters, so the virtual camera system can adjust its settings in real time as needed.

[0033] In one implementation, the virtual camera system provides a virtual camera and lens with the same characteristics as the optical version. Alternatively, the virtual camera system may include a conversion function that maps parameters from the optical system to the virtual system. Furthermore, the position and orientation of the optical camera are transmitted to the virtual camera system. The virtual camera system includes an algorithm that converts optical lens parameters and optical camera spatial information into a virtual equivalent. This optical-to-virtual conversion is used, but is not limited to, calculating and displaying effects such as depth of field (DoF), automatic exposure, and other camera effects. Therefore, when the lens parameters and spatial information of the optical lens and camera are combined with the described virtual equivalent, the virtual camera system can zoom and focus on a virtual 3D object in ways that are impossible or difficult to achieve using only optical solutions.

[0034] Figure 4A This is a representation of a computer system 400 and a user 402 according to an embodiment of this disclosure. User 402 uses the computer system 400 to implement an application 490 for seamless virtualization and optical lens parameter transfer, as per [the relevant documentation / concept]. Figure 2A The video capture system 200 and Figure 3The method 300 for seamless virtual and optical lens parameter transfer is shown and described in the literature.

[0035] Computer system 400 Store and Execute Figure 4B The lens parameter transfer application 490 is described above. Furthermore, the computer system 400 can communicate with the software program 404. The software program 404 may include software code for the lens parameter transfer application 490. The software program 404 can be loaded onto external media such as a CD, DVD, or storage drive, which will be explained further below.

[0036] Furthermore, computer system 400 can be connected to network 480. Network 480 can be connected in various different architectures, such as client-server architecture, peer-to-peer network architecture, or other types of architecture. For example, network 480 can communicate with server 485, which coordinates the engine and data used within lens parameter transfer application 490. Similarly, the network can be of different types. For example, network 480 can be any variant of the Internet, LAN or WAN, WAN, MAN, intranet or extranet, or wireless network.

[0037] Figure 4B This is a functional block diagram of a computer system 400 illustrating a managed lens parameter transfer application 490 according to an embodiment of the present disclosure. Controller 410 is a programmable processor and controls the operation of the computer system 400 and its components. Controller 410 loads instructions (e.g., in the form of a computer program) from memory 420 or an embedded controller memory (not shown) and executes these instructions to control the system, such as providing data processing. In its execution, controller 410 provides software systems for the lens parameter transfer application 490, such as performing a mask generation process to extract a subject from the environment without requiring a green screen. Alternatively, this service can be implemented as a separate hardware component within controller 410 or computer system 400.

[0038] Memory 420 temporarily stores data for use by other components of computer system 400. In one embodiment, memory 420 is implemented as RAM. In another embodiment, memory 420 also includes long-term or permanent memory, such as flash memory and / or ROM.

[0039] Storage device 430 stores data temporarily or for extended periods for use by other components of computer system 400. For example, storage device 430 stores data used by lens parameter transfer application 490. In one embodiment, storage device 430 is a hard disk drive.

[0040] Media device 440 receives removable media and reads data from and / or writes data to inserted media. In one embodiment, media device 440 is, for example, an optical disc drive.

[0041] User interface 450 includes components for accepting user input from a user of computer system 400 and presenting information to user 402. In one embodiment, user interface 450 includes a keyboard, mouse, audio speakers, and a display. Controller 410 uses input from user 402 to adjust the operation of computer system 400.

[0042] I / O interface 460 includes one or more I / O ports for connecting to corresponding I / O devices, such as external storage devices or auxiliary devices (e.g., printers or PDAs). In one embodiment, the ports of I / O interface 460 include ports such as USB ports, PCMCIA ports, serial ports, and / or parallel ports. In another embodiment, I / O interface 460 includes a wireless interface for wireless communication with external devices.

[0043] Network interface 470 includes wired and / or wireless network connections that support Ethernet connectivity, such as RJ-45 or “Wi-Fi” interfaces (including but not limited to 802.11).

[0044] Computer system 400 includes typical additional hardware and software of a computer system (e.g., power supply, cooling, operating system), but for simplicity... Figure 4B These components are not specifically shown. In other embodiments, different configurations of computer systems may be used (e.g., different bus or storage configurations or multiprocessor configurations).

[0045] The description of the implementations disclosed herein is provided to enable any person skilled in the art to make or use this disclosure. Many modifications to these implementations will be apparent to those skilled in the art, and the principles defined herein can be applied to other implementations without departing from the spirit or scope of this disclosure. While the above description includes systems and methods for seamless virtual and optical lens parameter transfer in video capture, the systems and methods described are applicable to other fields such as medical imaging.

[0046] In certain embodiments of this disclosure, not all features of every example described above are necessarily required. Furthermore, it should be understood that the descriptions and figures presented herein represent the broad subject matter contemplated by this disclosure. It should also be understood that the scope of this disclosure fully encompasses other embodiments that will become apparent to those skilled in the art, and the scope of this disclosure is therefore limited only by the appended claims.

[0047] One implementation includes one or more programmable processors and corresponding computer system components to store and execute computer instructions, such as providing virtual camera system and exchanging and encoding lens parameters for gradient morphology.

[0048] Additional variations and implementations are also possible. For example, multiple camera systems can be used together, such as in a production setup with equipment equipped with more than one optical camera or with multiple camera rigs. In one such example, the system adjusts which set of optical parameters to use based on the selection of the active camera.

Claims

1. A method of automatically switching between transmitting virtual lens parameters to an external virtual camera system and transmitting optical lens parameters to an optical camera having an optical lens positioned within a physical scene, the method comprising: providing spatial information of the physical scene to the optical camera, the spatial information including a position and orientation of the optical camera relative to a display screen; creating a feedback loop between the optical camera and the optical lens to enable the optical lens to transmit lens settings to scene registration; determining when a focus of the optical lens exceeds an optical focus limit defined by a surface of the display screen according to the lens settings; and automatically switching, by the scene registration, the optical lens from transmitting optical lens parameters to the optical camera for optical operations to transmitting virtual lens parameters and virtual spatial information to an external virtual camera system to render a virtual scene on the display screen when the focus of the lens moves beyond the optical focus limit, wherein the virtual camera system converts the optical lens parameters and orientation of the optical camera to virtual lens parameters and virtual spatial information for the external virtual camera system to update the virtual camera with values corresponding to the virtual lens parameters and virtual spatial information.

2. The method of claim 1, wherein the spatial information of the physical scene includes an orientation of the optical camera relative to the display screen.

3. The method of claim 1, wherein the spatial information of the physical scene is managed by a processor and a storage unit in conjunction with the display screen.

4. The method of claim 1, wherein the spatial information of the physical scene is provided by an indoor location system.

5. The method of claim 1, wherein the virtual lens parameters are continuously transmitted to the virtual camera system even when the optical lens parameters are in use so that the virtual camera system can adjust settings for real-time operations.

6. The method of claim 1, wherein the virtual camera system provides a virtual camera and lens having the same characteristics as the optical camera.

7. The method of claim 6, wherein the virtual camera system includes a conversion function that maps parameters from the optical camera to the virtual camera.

8. The method of claim 1, further comprising calculating and displaying at least one of a depth of field and auto exposure using the converted virtual lens parameters.

9. A system of automatically switching between using virtual lens parameters and using optical lens parameters, the system comprising: an optical camera and an optical lens positioned within a physical scene; a display screen positioned within the physical scene; and a virtual camera system. ​ a processor to provide spatial information of a physical scene to an optical camera, the spatial information including a position and orientation of the optical camera relative to a display screen, to create a feedback loop between the optical camera and the optical lens to receive lens settings from the optical lens, to determine when a focus of the optical lens moves beyond an optical focus limit defined by a surface of the display screen based on the lens settings, and to enable the optical lens to automatically switch from using optical lens parameters for optical operations by the optical camera to using virtual lens parameters and virtual spatial information to render a virtual scene on the display screen when the focus of the lens moves beyond the optical focus limit, wherein the system converts the optical lens parameters and orientation of the optical camera to the virtual lens parameters and virtual spatial information to update a virtual camera with values corresponding to the virtual lens parameters and virtual spatial information.

10. The system of claim 9, wherein the processor manages the spatial information of the physical scene in conjunction with the display screen and the storage unit.

11. The system of claim 9, further comprising an indoor location system to provide the spatial information of the physical scene to the processor.

12. The system of claim 9, further comprising a virtual camera system to render the virtual scene on the display screen using the virtual lens parameters.

13. The system of claim 12, wherein the virtual camera system provides a virtual camera and lens having the same characteristics as the optical camera.

14. The system of claim 13, wherein the virtual camera system includes a conversion function to map parameters from the optical camera to the virtual camera.

15. The system of claim 9, wherein the processor calculates and displays at least one of a depth of field and auto exposure using the converted virtual lens parameters.

Images