Progressive morphological lens parameter coding

Abstract

The invention relates to progressive morphological lens parameter coding. Automatically transferring between virtual lens parameters and optical lens parameters for positioning an optical camera within a physical scene, including: providing spatial perception of the physical scene to the optical camera, including the position and orientation of the display screen; creating a feedback loop between the optical camera and the lens to enable the lens to communicate lens settings to a scene registration; determining when a focus of the lens exceeds an optical focus limit according to the lens setting; and enabling the lens to automatically transition from using optical lens parameters to using virtual lens parameters through scene registration when the focus of the lens moves beyond the optical focus limit.

Description

[0001] This application is a divisional application of Chinese patent application 202180058252.8, filed on January 31, 2023, with the invention name “Progressive Morphological Lens Parameter Coding”. Technical Field

[0002] The present disclosure relates to processing and capturing of video data for a video capture system, and more particularly, to automatically switching between optical lens parameters and virtual lens parameters. Background Art

[0003] In a virtual production environment that uses a combination of optical cameras / lenses and virtual cameras / lenses in a virtual scene together with a physical LED display wall for displaying the virtual scene, there is a difference between the optical lens parameters and how the parameters relate to the virtual scene. Therefore, it is difficult to effectively perform optical zoom or focus on objects "inside" the virtual scene because the optical parameters can zoom and focus beyond the flat LED display showing the virtual screen. Summary of the invention

[0004] The present disclosure provides for implementing a technique for automatically transferring between optical lens parameters and virtual lens parameters based on the physical position of a camera relative to a defined fixed focus target.

[0005] In one embodiment, a method of transferring between virtual lens parameters and optical lens parameters of an optical camera positioned within a physical scene is disclosed. The method includes: providing spatial awareness of the physical scene to the optical camera, the spatial awareness including the position and orientation of a display screen; creating a feedback loop between the optical camera and the lens to enable the lens to communicate lens settings to scene registration; determining when the focus of the lens exceeds the optical focus limit based on the lens setting; and enabling the lens to automatically transfer from using the optical lens parameters to using the virtual lens parameters through scene registration when the focus of the lens moves beyond the optical focus limit.

[0006] In one embodiment, the spatial perception of the physical scene includes the orientation of the optical camera relative to the display screen. In one embodiment, the spatial perception of the physical scene is managed by a processor and a storage unit in conjunction with the display screen. In one embodiment, the spatial perception of the physical scene includes an indoor position system. In one embodiment, the method also includes transmitting virtual lens parameters to the virtual camera system to render the virtual scene on the display screen. In one embodiment, the virtual lens parameters are continuously transmitted to the virtual camera system even when the optical lens parameters are used so that the virtual camera system can adjust the settings for real-time operation. In one embodiment, the virtual camera system provides a virtual camera and lens with the same characteristics as the optical camera. In one embodiment, the method also includes transmitting the position and orientation of the optical camera to the virtual camera system. In one embodiment, the virtual camera system includes an algorithm for converting the optical lens parameters and orientation of the optical camera into converted virtual lens parameters. In one embodiment, the method also includes calculating and displaying at least one of depth of field and automatic exposure using the converted virtual lens parameters.

[0007] In another embodiment, a system for transferring between virtual lens parameters and optical lens parameters is disclosed. The system includes: an optical camera and a lens positioned within a physical scene; a display screen positioned within the physical scene; and a processor for providing spatial awareness of the physical scene to the optical camera, the spatial awareness including the position and orientation of the display screen, creating a feedback loop between the optical camera and the lens to receive lens settings from the lens, determining when the focus of the lens exceeds the optical focus limit based on the lens settings, and enabling the lens to automatically transfer from using the optical lens parameters to using the virtual lens parameters when the focus of the lens moves beyond the optical focus limit.

[0008] In one embodiment, the spatial perception of the physical scene includes the orientation of the optical camera relative to the display screen. In one embodiment, the processor manages the spatial perception of the physical scene in conjunction with the display screen and the storage unit. In one embodiment, the system also includes an indoor position system to provide the processor with spatial perception of the physical scene. In one embodiment, the system also includes a virtual camera system to render the virtual scene on the display screen using virtual lens parameters. In one embodiment, the virtual camera system provides a virtual camera and lens with the same characteristics as the optical camera. In one embodiment, the virtual camera system includes a conversion function to map parameters from the optical camera to the virtual camera. In one embodiment, the virtual camera system includes an algorithm to convert the optical lens parameters and the orientation of the optical camera into converted virtual lens parameters. In one embodiment, the processor uses the converted virtual lens parameters to calculate and display at least one of depth of field and automatic exposure.

[0009] Other features and advantages should be apparent from the description, which illustrates various aspects of the disclosure by way of example. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Details regarding the structure and operation of the present disclosure may be obtained, in part, by studying the accompanying drawings, in which like reference numerals refer to like parts, and in which:

[0011] Figure 1 is a diagram showing a setup in which the camera and lens are not optically focused beyond the optical focal plane / focus limit defined by the surface of the display screen;

[0012] Figure 2A is a block diagram of a video capture system according to one embodiment of the present disclosure;

[0013] Figure 2B is a diagram showing a new setup illustrating seamless virtual and optical lens parameter transfer according to one embodiment of the present disclosure;

[0014] Figure 3 is a flow chart illustrating a method for seamless virtual and optical lens parameter transfer of a camera positioned within a physical scene according to one embodiment of the present disclosure;

[0015] Figure 4A is a representation of a computer system and a user according to an embodiment of the present disclosure; and

[0016] Figure 4B is a functional block diagram illustrating a computer system hosting a lens parameter transfer application according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] As described above, in a conventional virtual production environment that uses a combination of an optical camera / lens and a virtual camera / lens in a virtual scene together with a physical LED display wall for displaying the virtual scene, there is a difference between the optical lens parameters and how the parameters are related to the virtual scene. Therefore, it is difficult to effectively perform optical zoom or focus on objects "inside" the virtual scene. That is, the optical parameters can zoom and focus beyond the flat LED display showing the virtual screen.

[0018] Figure 1 1 is a diagram illustrating a setup 100 in which the optical focus of the camera 110 and lens 120 does not exceed an optical focal plane / focus limit 140 defined by the surface of the display screen 130. In this setup 100, an area of ​​acceptable optical focus 150 may be defined as a predetermined distance range from the optical focus limit 140.

[0019] Certain embodiments of the present disclosure provide for implementing a technique for automatically switching between optical lens parameters and virtual lens parameters based on the physical position of a camera relative to a defined fixed focus target.

[0020] After reading the following description, it will become apparent how to implement the present disclosure in various embodiments and applications. Although various embodiments of the present disclosure will be described herein, it should be understood that these embodiments are given only by way of example and not limitation. Thus, this detailed description of the various embodiments should not be interpreted as limiting the scope or breadth of the present disclosure.

[0021] Features provided in embodiments may include, but are not limited to, one or more of the following: (a) real-time communication with the camera and a control computer, where the computer determines the parameters to be used; (b) a built-in scene representation for the camera and / or lens and the system makes the determination there; (c) an indoor positioning system (IPS) for camera position tracking; and (d) an external position tracking system and communication with the camera and lens, and / or a control processor.

[0022] Figure 2A is a block diagram of a video capture system 200 according to one embodiment of the present disclosure. Figure 2A In the embodiment shown in , the video capture system 200 includes a camera 210 and a lens 220 , a display screen 230 , and other components for video capture such as a processor 240 and a storage unit 242 .

[0023] In one embodiment, as indicated above, the camera 210 includes optical lens parameters, virtual lens parameters, and physical scene perception aspects. The camera 210 manages the physical scene perception aspects in conjunction with the display screen 230, the processor 240, and the storage unit 242. In one embodiment, the system 200 also includes an indoor position system (IPS) 244 and an automatic way to use the appropriate parameters (i.e., optical or virtual) in conjunction with the processor 240 and the storage unit 242.

[0024] For the physical scene perception aspect using IPS 244 in conjunction with processor 240, camera 210 is configured to sense its own position relative to physical space and determine a focus target in the scene that cannot be exceeded by focus and zoom (i.e., where display screen 130 is physically located). When the optical zoom or focus on the lens changes and reaches the focus target, the lens automatically shuts down optical operation and effectively works through virtual zoom and focus (while still operating the optical controls). Camera 210 then sends digital parameters (e.g., virtual lens parameters) to system 200, which updates the virtual camera with corresponding values. Thus, by using seamless automatic optical and virtual lens parameters, it is possible to focus and zoom in a virtual 3-D scene.

[0025] Figure 2B is a diagram showing a new setup 250 illustrating seamless virtual and optical lens parameter transfer according to one embodiment of the present disclosure. Figure 2B The embodiment shown in includes a camera 210, a lens 220, and a display screen 230. The new arrangement 250 also includes a processor 240, a storage unit 242, and an IPS 244. Figure 2B In the new setting 250, seamless virtual and optical lens parameter transfer enables "out of screen" operation while still maintaining "in volume" zoom and focus.

[0026] exist Figure 2B In the embodiment shown in , the operation of lens 120 is similar to the optical to digital focus and zoom in consumer point-and-shoot cameras. However, this embodiment is configured so that the digital version forwards the optical lens parameters, rather than a single magnification value. The camera 210 stores a representation of the physical scene. In one embodiment, the representation is stored as part of its firmware as a "scene registration". In another embodiment, the representation is stored in a storage device located externally, such as in a small hardware unit attached to the camera 210, or as part of a completely independent system using high-speed network communications.

[0027] exist Figure 2B In the embodiment shown in , as mentioned above, the camera 210 (including the lens 220) has a spatial awareness of the physical scene and the position and orientation of the display screen 230, as well as its own orientation relative to the display screen 230 and the physical space. Changes in the lens parameters are communicated to a control system (on-camera or off-camera). In addition, a feedback loop (supported by the processor 240, the storage unit 242, and the IPS 244) between the camera 210 and the lens 220 is created to enable the lens 220 to communicate its settings to the scene registration, which communicates back whether the optical zoom and focus 280 parameters are to be used. The feedback loop system provides a progressive morphological lens parameter exchange. When this feedback loop system determines that the optical lens parameters can no longer be used because the focus of the lens 220 exceeds the optical focus limit 140, the lens 220 automatically switches to the parameters of the virtual zoom and focus 270. These virtual parameters are communicated to the external virtual camera system that renders the virtual scene 260 on the display screen 230. In one embodiment, these virtual parameters may always be sent, even when optical parameters are used, so that the virtual camera system can adjust settings in real time as needed.

[0028] In one embodiment, a virtual camera system provides a virtual camera and lens having the same characteristics as the optical version. Alternatively, the virtual camera system may include a conversion function that maps parameters from the optical system to the virtual system. In addition, the position and orientation of the optical camera are transmitted to the virtual camera system. The virtual camera system includes an algorithm that converts optical lens parameters and optical camera spatial information into virtual equivalents. This optical to virtual conversion is used for, but not limited to, calculating and displaying effects such as depth of field (DoF), automatic exposure, and other camera effects. Therefore, when the lens parameters and spatial information of the optical lens and camera are combined with the described virtual equivalents, the virtual camera system can zoom and focus on virtual 3D objects in a way that is impossible or difficult using only optical solutions.

[0029] Figure 3 is a flow chart illustrating a method 300 for seamless virtual and optical lens parameter transfer of a camera positioned within a physical scene according to one embodiment of the present disclosure. Figure 3 In the embodiment shown in , seamless virtual and optical lens parameter transfer enables "out of screen" operation while still maintaining "in volume" zoom and focus.

[0030] exist Figure 3 In the embodiment shown in , at step 310, spatial awareness of the physical scene, including the position and orientation of the display screen 230, is provided to the camera 210 (including the lens 220). In one embodiment, the spatial awareness also includes the orientation of the camera 210 relative to the display screen 230 and the physical space. As described above, the camera 210 manages the physical scene awareness aspects in conjunction with the display screen 230, the processor 240, and the storage unit 242. In one embodiment, the system 200 also includes an indoor position system (IPS) 244 and an automated manner using appropriate parameters (i.e., optical or virtual) in conjunction with the processor 240 and the storage unit 242.

[0031] In one embodiment, a feedback loop is created between the camera 210 and the lens 220 at step 320 to enable the lens 220 to communicate its settings to the scene registration. In one embodiment, the feedback loop system provides progressive morphological lens parameter exchange by determining at step 330 when the optical lens parameters can no longer be used because the focus of the lens 220 exceeds the optical focus limit 140. Thus, when the feedback loop system determines (at step 330) that the focus of the lens 220 exceeds the optical focus limit 140, the lens 220 automatically transitions from using the optical lens parameters to using the virtual lens parameters of the virtual zoom and focus 270 at step 340. These virtual parameters are communicated to the external virtual camera system, which renders the virtual scene 260 on the display screen 230. In one embodiment, these virtual parameters may always be sent, even when the optical parameters are used, so the virtual camera system can adjust the settings in real time as needed.

[0032] In one embodiment, the virtual camera system provides a virtual camera and lens with the same characteristics as the optical version. Alternatively, the virtual camera system may include a conversion function that maps parameters from the optical system to the virtual system. In addition, the position and orientation of the optical camera are transmitted to the virtual camera system. The virtual camera system includes an algorithm that converts optical lens parameters and optical camera spatial information into virtual equivalents. This optical to virtual conversion is used for, but not limited to, calculating and displaying effects such as depth of field (DoF), automatic exposure, and other camera effects. Therefore, when the lens parameters and spatial information of the optical lens and camera are combined with the described virtual equivalents, the virtual camera system can zoom and focus on virtual 3D objects in a way that is impossible or difficult using only optical solutions.

[0033] Figure 4A 4 is a representation of a computer system 400 and a user 402 according to an embodiment of the present disclosure. The user 402 uses the computer system 400 to implement an application 490 for seamless virtual and optical lens parameter transfer, as described with respect to Figure 2A The video capture system 200 and Figure 3 The method 300 for seamless virtual and optical lens parameter transfer is shown and described in FIG.

[0034] Computer system 400 stores and executes Figure 4B The computer system 400 may also include a lens parameter transfer application 490. In addition, the computer system 400 may be in communication with a software program 404. The software program 404 may include software code for the lens parameter transfer application 490. The software program 404 may be loaded on an external medium such as a CD, DVD, or storage drive, as will be further explained below.

[0035] In addition, computer system 400 can be connected to network 480. Network 480 can be connected in a variety of different architectures, such as a client-server architecture, a peer-to-peer network architecture, or other types of architectures. For example, network 480 can communicate with server 485, which coordinates engines and data used within lens parameter transfer application 490. Likewise, the network can be different types of networks. For example, network 480 can be the Internet, a local area network or a local area network, a wide area network, a metropolitan area network, an intranet or extranet, or any variation of a wireless network.

[0036] Figure 4B 4 is a functional block diagram illustrating a computer system 400 hosting a lens parameter transfer application 490 according to an embodiment of the present disclosure. The controller 410 is a programmable processor and controls the operation of the computer system 400 and its components. The controller 410 loads instructions (e.g., in the form of a computer program) from the memory 420 or an embedded controller memory (not shown) and executes these instructions to control the system, such as to provide data processing. In its execution, the controller 410 provides a software system for the lens parameter transfer application 490, such as to perform a mask generation process to extract a subject from an environment without the need for a green screen. Alternatively, the service may be implemented as a separate hardware component in the controller 410 or the computer system 400.

[0037] Memory 420 temporarily stores data for use by other components of computer system 400. In one implementation, memory 420 is implemented as RAM. In one implementation, memory 420 also includes long-term or permanent memory, such as flash memory and / or ROM.

[0038] The storage device 430 stores data temporarily or long term for use by other components of the computer system 400. For example, the storage device 430 stores data used by the lens parameter transfer application 490. In one embodiment, the storage device 430 is a hard drive.

[0039] The media device 440 receives removable media and reads data from and / or writes data to the inserted media. In one implementation, for example, the media device 440 is an optical disk drive.

[0040] The user interface 450 includes components for accepting user input from a user of the computer system 400 and presenting information to the user 402. In one embodiment, the user interface 450 includes a keyboard, a mouse, audio speakers, and a display. The controller 410 uses the input from the user 402 to adjust the operation of the computer system 400.

[0041] The I / O interface 460 includes one or more I / O ports to connect to corresponding I / O devices, such as external storage devices or auxiliary devices (e.g., printers or PDAs). In one embodiment, the ports of the I / O interface 460 include ports such as USB ports, PCMCIA ports, serial ports, and / or parallel ports. In another embodiment, the I / O interface 460 includes a wireless interface for wireless communication with external devices.

[0042] Network interface 470 includes a wired and / or wireless network connection supporting an Ethernet connection, such as an RJ-45 or "Wi-Fi" interface (including but not limited to 802.11).

[0043] Computer system 400 includes additional hardware and software typical of computer systems (e.g., power, cooling, operating system), but is not shown for simplicity. Figure 4B These components are not specifically shown in the drawings. In other implementations, a differently configured computer system (eg, a different bus or storage configuration or a multi-processor configuration) may be used.

[0044] The description of the implementations disclosed herein is provided to enable any person skilled in the art to make or use the present disclosure. Many modifications to these implementations will be apparent to those skilled in the art, and the principles defined herein may be applied to other implementations without departing from the spirit or scope of the present disclosure. While the above description includes systems and methods for seamless virtual and optical lens parameter transfer in video capture, the described systems and methods are applicable to other fields such as medical imaging.

[0045] In the specific implementation of the present disclosure, not all features of each of the above examples are required. In addition, it should be understood that the description and drawings presented herein represent the subject matter widely expected by the present disclosure. It should also be understood that the scope of the present disclosure fully encompasses other embodiments that will become apparent to those skilled in the art, and the scope of the present disclosure is therefore limited only by the appended claims.

[0046] One embodiment includes one or more programmable processors and corresponding computer system components to store and execute computer instructions, such as to provide virtual camera systems and lens parameter exchange and encoding of progressive morphologies.

[0047] Additional variations and implementations are also possible. For example, multiple camera systems can be used together, such as using equipment with more than one optical camera installed or a production setup with multiple camera equipment. In one such example, the system adjusts which set of optical parameters to use based on the camera that selects the activity.

Claims

1. A method for enabling focus outside a display screen, the method comprising: Providing spatial perception information of a physical scene, the spatial perception information including a position and orientation of an optical camera (210) within the physical scene relative to a display screen; Creating a feedback loop between the optical camera (210) and the optical lens (220) to enable the optical lens (220) to communicate lens settings to the scene registration; determining when the focus of the optical lens (220) exceeds an optical focus limit defined by a surface of the display screen; as well as When the focus of the lens moves beyond the optical focus limit, automatically switching from focusing by the optical lens (220) using optical lens parameters for optical operation based on scene registration to virtual focusing by the virtual camera system using virtual lens parameters and virtual space information to render a virtual scene on a display screen, wherein The method further includes converting optical lens parameters and orientation of the optical camera (210) into virtual lens parameters and virtual space information of the virtual camera system, so as to update the virtual camera with values ​​corresponding to the virtual lens parameters and the virtual space information.

2. The method of claim 1, wherein the spatial perception information of the physical scene comprises an orientation of the optical camera relative to the display screen.

3. A method according to claim 1 or 2, wherein the spatial perception information of the physical scene is managed by a processor and a storage unit in combination with a display screen.

4. The method of claim 1, wherein the spatial perception information of the physical scene comprises an indoor location system.

5. The method according to claim 1, wherein: Even when the optical lens parameters are used, the virtual lens parameters are continuously transmitted to the virtual camera system so that the virtual camera system can adjust the settings for real-time operation. The method of claim 1 , wherein the virtual camera system provides a virtual camera and lens having the same characteristics as an optical camera.

7. The method according to claim 1, further comprising At least one of depth of field and automatic exposure is calculated and displayed using the converted virtual lens parameters.

8. The method according to claim 1, further comprising When the focus of the lens moves beyond the optical focus limit, the zooming performed by the optical lens (220) using the optical lens parameters for optical operation based on the scene alignment is automatically switched to the virtual zooming performed by the virtual camera system using the virtual lens parameters and virtual space information to render the virtual scene on the display screen.

9. A system for enabling focus outside of a display screen within a physical scene, the system comprising: an optical camera (210) and an optical lens (220) positioned within the physical scene; as well as Processor for: Providing spatial perception information of a physical scene to an optical camera (210), the spatial perception information including a position and orientation of the optical camera (210) relative to a display screen; creating a feedback loop between the optical camera (210) and the optical lens (220) to receive lens settings from the optical lens (220), enabling the optical lens (220) to use optical lens parameters for optical operation, determining when the focus of the optical lens exceeds the optical focus limit defined by the surface of the display screen, and When the focus of the lens moves beyond the optical focus limit, a virtual scene is rendered on a display screen by using virtual lens parameters and virtual space information so that the virtual camera and the virtual lens can focus on the virtual 3D object, wherein Optical lens parameters and orientation of the optical camera (210) are provided to a virtual camera system including a virtual camera and a virtual lens for conversion into virtual lens parameters and virtual space information, thereby updating the virtual camera with values ​​corresponding to the virtual lens parameters and virtual space information.

10. The system of claim 9, further comprising a display screen (230).

11. The system according to claim 9 or 10, further comprising Indoor location system, which provides spatial awareness information of the physical scene to the processor.

12. A computer program product storing instructions, which when executed are used to cause a computer to perform the following steps: Providing spatial perception information of a physical scene, the spatial perception information including a position and orientation of an optical camera (210) within the physical scene relative to a display screen; receiving lens parameters from a feedback loop between an optical camera (210) and an optical lens (220); and A virtual scene is rendered on a display screen using virtual lens parameters and virtual space information to be virtually focused when the focus of the optical lens (220) exceeds an optical focus limit defined by a surface of the display screen, wherein the optical lens parameters are used for optical operation of the optical lens (220), The instructions also cause the computer to perform the following steps: converting optical lens parameters and orientation of the optical camera (210) into virtual lens parameters and virtual space information of the virtual camera system, so as to update the virtual camera with values ​​corresponding to the virtual lens parameters and virtual space information.

13. The computer program product of claim 12, wherein: Even when the optical lens parameters are used, the virtual lens parameters are continuously transmitted to the virtual camera system so that the virtual camera system can adjust the settings for real-time operation.

14. A computer program product according to claim 12 or 13, wherein: The virtual camera system provides a virtual camera and lens with the same characteristics as an optical camera.

15. The computer program product according to claim 12 or 13, wherein: The virtual camera system includes an algorithm for converting optical lens parameters and orientation of an optical camera (210) into virtual lens parameters and virtual space information.

Images