A real-time shooting system and method for digital animation
By combining Unreal Engine and motion capture system with virtual camera technology, real-time display and synchronized camera movement are achieved, solving the problem of effect display delay in traditional 3D animation production and realizing efficient real-time shooting and adjustment of digital animation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 杭州玄机科技股份有限公司
- Filing Date
- 2024-09-11
- Publication Date
- 2026-08-04
AI Technical Summary
In traditional 3D animation production, the display of effects is delayed, and the shooting or recording effects cannot be obtained in real time, resulting in low efficiency and delays in modification, especially in terms of multiple characters and software compatibility.
By redeveloping the Unreal Engine and combining it with a motion capture system, virtual camera technology is used to achieve real-time preview and synchronized camera movement. By combining the motion capture module, the Unreal Engine module, and the screen-sharing module, changes in virtual content are displayed in real time, simulating the depth of field and focal length changes of a real camera.
It enables real-time shooting of digital animation, allowing for timely adjustments to camera effects and actor performance quality, improving production efficiency, and solving the problem of delayed effect display in existing technologies.
Smart Images
Figure CN119228953B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of shooting and producing digital film and television works, and specifically to a real-time shooting system and method for digital animation. Background Technology
[0002] Traditional 3D animation production typically begins with animators creating character assets based on the storyline. Then, relevant production staff create motion capture data and bind it to the characters before recording in a dedicated animation engine. Throughout this process, it's clear that the animator's skill and experience directly impact the animation quality. The recording or filming process also significantly affects the final visual effect. On one hand, time constraints limit the ability to capture detailed animation, and the manual frame-by-frame production method is inefficient. On the other hand, while traditional motion capture technology is widespread, it only provides basic motion data, not the complete performance effect. The director's judgment of the shot's effect, the actors' performance quality, shot composition, and camera movement cannot be shown in real-time on set; only basic data collection is possible.
[0003] In summary, the current 3D animation production process suffers from severe effect lag (i.e., the inability to capture the effects of shooting or recording in real time). Once the effects are displayed, repeated modifications are often required, resulting in extremely low efficiency. Therefore, there is an urgent need to provide a complete real-time virtual shooting system to overcome the bottlenecks of traditional 3D production and motion capture workflows, and to resolve delays caused by multiple characters and software compatibility issues based on existing workflows. Summary of the Invention
[0004] To address the aforementioned shortcomings of traditional 3D animation production processes, this invention provides a technical solution capable of real-time display of digital animation shooting / recording effects. This invention achieves the above objectives by further developing the Unreal Engine, fully utilizing its virtual camera functionality in conjunction with a motion capture system. The virtual camera reflects the movement of the real camera onto the virtual camera by locating the camera's position and angle, and previews changes in the virtual content in real time. As long as the camera can be tracked, lens movement and correct perspective relationships can be obtained in a real-time rendering environment, perfectly synchronized with the real camera. Furthermore, the virtual camera system can easily adjust the focal length and simulate the depth-of-field changes of the real camera, ensuring accurate and realistic reproduction of the real camera's shooting effects.
[0005] The technical solution provided in this application is, on the one hand, a real-time shooting system for digital animation, including: a motion capture module, a UE Unreal Engine module, and a screen-sharing module.
[0006] The motion capture module is used to generate several first virtual cameras by marking motion capture points of several physical cameras, and to place the several first virtual cameras in a virtual motion capture area; the several physical cameras and the several first virtual cameras correspond one-to-one.
[0007] Preferably, the motion capture module is Vicon Shogun Live software running on a computer device.
[0008] The Unreal Engine (UE) is used to receive data captured by the motion capture module in real time through the built-in Live Link plugin; and to construct several second virtual cameras with controllable parameters, wherein the real-time shooting images of the second virtual cameras and the first virtual camera are bound one-to-one.
[0009] Furthermore, the plurality of second virtual cameras are constructed based on the BP_Switch data structure of the virtual camera built-in plugin in the Unreal Engine. By selecting the corresponding second virtual camera in the BP_Switch data structure in the VCS of the Unreal Engine, and setting its details to add a real-time connection node for props, the real-time shooting images of the corresponding second virtual camera and the corresponding first virtual camera are bound together.
[0010] The screen mirroring module connects to the Unreal Engine module via the Virtual Camera in the remote session plugin of the Unreal Engine to display the real-time shooting footage of the corresponding second virtual camera in the Unreal Engine module on the same screen.
[0011] Furthermore, the screen-sharing display module can be one or more, and each screen-sharing display module displays a different real-time captured image from the second virtual camera.
[0012] Preferably, the screen-sharing module is the application software Live Link Vcam running on an iOS device.
[0013] Accordingly, the technical solution provided in this application is, in another aspect, a method for real-time shooting of digital animation. This real-time shooting method includes:
[0014] A number of physical cameras are marked with motion capture points to construct a number of first virtual cameras, and the number of first virtual cameras are placed in a virtual motion capture field; the number of physical cameras and the number of first virtual cameras correspond one-to-one.
[0015] The Unreal Engine UE is configured to receive data captured by the motion capture system in real time through the built-in Live Link plugin, construct several second virtual cameras with controllable parameters, and bind the real-time shooting images of the several second virtual cameras to the several first virtual cameras in a one-to-one correspondence.
[0016] The display module is connected via the Virtual Camera remote session plugin in Unreal Engine to display the real-time shooting footage from the corresponding second virtual camera in the Unreal Engine module on the same screen.
[0017] Furthermore, the plurality of second virtual cameras are constructed based on the BP_Switch data structure of the virtual camera built-in plugin in the Unreal Engine. By selecting the corresponding second virtual camera in the BP_Switch data structure in the VCS of the Unreal Engine, and setting its details to add a real-time connection node for props, the real-time shooting images of the corresponding second virtual camera and the corresponding first virtual camera are bound together.
[0018] The technical solution provided in this application enables the shooting of digital animation, which allows for real-time and comprehensive display of the images captured by each camera. This facilitates the production / shooting personnel to promptly grasp the lens effects and the quality of the actors' performances, so as to adjust character resources, lens composition, and camera movement methods in a timely manner.
[0019] The technical solution provided in this application can effectively solve the problems of delayed effect display, excessive delay in modifying shooting / recording effects (especially delays caused by multiple characters and software compatibility), and low efficiency in the existing 3D animation production process. Attached Figure Description
[0020] Figure 1 A schematic diagram of one embodiment of the real-time shooting system for digital animation provided by the present invention. Detailed Implementation
[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0022] like Figure 1 As shown, the present invention provides a real-time shooting system for digital animation, including: a motion capture module 100, a UE Unreal Engine 200, and a screen display module 300.
[0023] The motion capture module 100 is used to generate a plurality of first virtual cameras by marking motion capture points of a plurality of physical cameras (i.e., physical virtual cameras), and to place the plurality of first virtual cameras in a virtual motion capture field (virtual motion capture scene); the plurality of physical cameras and the plurality of first virtual cameras correspond one-to-one.
[0024] The screen mirroring module connects to the Unreal Engine module via the VirtualCamera in the remote session plugin of the Unreal Engine to display the real-time captured images of the corresponding second virtual camera in the Unreal Engine module on the same screen.
[0025] exist Figure 1 In the illustrated embodiment, the motion capture module 100 is the Vi conShogun Live software running on a computer device. The process of constructing the first virtual camera in the Vi conShogun Live software is as follows:
[0026] S10. Open the Vicon Shogun Live software, create several physical cameras according to the storyboard requirements, and place them in the motion capture area.
[0027] S11. Mark motion capture points for several physical cameras (the motion trajectory of the motion capture points can be tracked by infrared light). At this time, the marked physical cameras are displayed as points on the software screen.
[0028] S12. Connect the motion capture points to generate the first virtual camera skeleton. Obviously, the first virtual camera corresponds one-to-one with the physical camera.
[0029] The Unreal Engine 200 is used to receive data captured by the motion capture module in real time through the built-in LiveLink plugin. It uses the BP_Switch data structure of the built-in virtual camera plugin of the Unreal Engine 200 to construct several second virtual cameras with controllable parameters. The real-time shooting images of the several second virtual cameras are bound one-to-one with those of the first virtual camera.
[0030] The data structure BP_Switch is used to adjust the focal length, depth of field, and offset parameters (top, bottom, left, right, etc.) of the image captured by the second virtual camera (which is bound to the first virtual camera), as well as the blueprint for switching shooting views. The second virtual camera built in Unreal Engine can fully utilize Unreal Engine's powerful image simulation and inference capabilities, obtaining more comprehensive and detailed image information by setting the relevant parameters in BP_Switch.
[0031] exist Figure 1 In the illustrated embodiment, the operation process related to the Unreal Engine 200 and the screen-sharing module 300 includes:
[0032] S20. In Unreal Engine, enable the Live Link plugin and the Vicon real-time plugin;
[0033] S21. Receive data from the Shogun Live auto-capture module via the LiveLink plugin, add and modify the real-time IP address, set the Port Number to the 801 band, and set the channels of the computer performing real-time motion capture and the computer running Unreal Engine to the same signal for linking and transmitting real-time animation (otherwise, real-time capture is not possible).
[0034] S22. Place the BP_Switch data structure from the virtual camera's built-in plugin within the scene, and then click "Build Camera" in the detailed settings interface to generate a second virtual camera with controllable parameters. Repeat this step to build multiple second virtual cameras with controllable parameters.
[0035] S23. By selecting the corresponding second virtual camera in the Bp_Switch data structure in the VCS of the UE Unreal Engine, and setting its details to add the real-time connection node of the prop, the real-time shooting screen of the corresponding first virtual camera is bound to the corresponding second virtual camera.
[0036] S24. Click Settings - Plugins to open the Plugins menu, search for Virtual Camera, and enable LiveLink and Remote Session plugins. Virtual Camera is a plugin that links a virtual object to a physical camera and displays it in the outline. LiveLink refers to linking the signal of the real-time motion capture module so that the signal captured by the motion capture module can be transmitted to the computer of Unreal Engine in real time. Enabling Remote Session plugin refers to the plugin that links the signal of the screen mirroring module (Live Link Vcam) to Unreal Engine.
[0037] S25. Open Project Settings, set the computer's IP address and port in UDP; end the static endpoint list connection device and add the device's IP address. This step ensures that the computer running Unreal Engine acts as the signal source to the camera, and that the IP address of the computer running Unreal Engine and the camera's IP address are connected under the same signal.
[0038] S26. In the default settings, set the pixel frame buffer format to 8-bit RGBA and restart the editor.
[0039] exist Figure 1In the illustrated embodiment, the screen mirroring module is a Live Link Vcam running on an iOS device. Of course, any application software or device that can interconnect with UE4 to achieve real-time display can serve as the screen mirroring module in this application; no limitation is made here. The following example only uses a Live Link Vcam running on an iOS device.
[0040] S27. Launch the Live Link Vcam app on your iOS device and enter the IP address of the computer running Unreal Engine UE.
[0041] S28. In the Unreal Engine panel, click and drag the second virtual camera into the scene. Placing the second virtual camera plugin into the scene displays the scene being captured, facilitating subsequent scene shooting.
[0042] S29. Connect your iPhone or iPad via Live Link in Unreal Engine (green indicates a successful connection). The signal source icon will automatically appear after the plugin is added; the signal source is the iPhone app signal. Yellow indicates standby and not connected, while green indicates connected.
[0043] After connecting the S30 and the iPhone app, further settings are made on the computer running Unreal Engine to ensure it accurately connects to the Live Link Vcam app on the iOS device for screen mirroring.
[0044] Furthermore, the screen-sharing display module 300 can be one or more, and each display module shows a different real-time shot from the second virtual camera. In this way, multiple second virtual cameras can easily adapt to the needs of multi-character performances and comprehensive, multi-angle real-time shooting.
[0045] Corresponding to the aforementioned real-time shooting system for digital animation, this application also provides a real-time shooting method for digital animation. This real-time shooting method includes:
[0046] A number of physical cameras are marked with motion capture points to construct a number of first virtual cameras, and the number of first virtual cameras are placed in a virtual motion capture field; the number of physical cameras and the number of first virtual cameras correspond one-to-one.
[0047] The Unreal Engine UE is configured to receive data captured by the motion capture system in real time through the built-in Live Link plugin, construct several second virtual cameras with controllable parameters, and bind the real-time shooting images of the several second virtual cameras to the several first virtual cameras in a one-to-one correspondence.
[0048] The display module is connected via the Virtual Camera remote session plugin in Unreal Engine to display the real-time shooting footage from the corresponding second virtual camera in the Unreal Engine module on the same screen.
[0049] Furthermore, the plurality of second virtual cameras are constructed based on the BP_Switch data structure of the virtual camera built-in plugin in the Unreal Engine. By selecting the corresponding second virtual camera in the BP_Switch data structure in the VCS of the Unreal Engine, and setting its details to add a real-time connection node for props, the real-time shooting images of the corresponding second virtual camera and the corresponding first virtual camera are bound together.
[0050] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Those skilled in the art will recognize that the present invention can have various modifications and variations. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A real-time shooting system for digital animation, characterized in that, The real-time shooting system includes: a motion capture module, a UE Unreal Engine module, and several screen-sharing modules; The motion capture module is used to generate several first virtual cameras by marking motion capture points of several physical cameras, and to place the several first virtual cameras in a virtual motion capture field; the several physical cameras and the several first virtual cameras correspond one-to-one. The Unreal Engine (UE) is used to receive data captured by the motion capture module in real time via the built-in LiveLink plugin. Several second virtual cameras with controllable parameters are constructed using the BP_Switch data structure of the UE's built-in virtual camera plugin. The BP_Switch data structure is used to adjust the focal length, depth of field, top, bottom, left, and right offset parameters of the second virtual camera's image, as well as the blueprint for switching shooting views. By selecting the corresponding second virtual camera in the BP_Switch data structure within the UE's VCS, and setting its details to add real-time connection nodes for props, the real-time shooting images of the corresponding second virtual camera and the corresponding first virtual camera are bound together. The screen mirroring module is used to connect to the Unreal Engine module via the VirtualCamera in the remote session plugin of the Unreal Engine UE, so as to display the real-time shooting screen of the corresponding second virtual camera in the Unreal Engine module on the same screen.
2. The real-time shooting system as described in claim 1, characterized in that, The motion capture module is Vicon Shogun Live software running on a computer device.
3. The real-time shooting system as described in claim 2, characterized in that, The screen-sharing display module can be one or more, and each screen-sharing display module displays a different real-time captured image from the second virtual camera.
4. The real-time shooting system as described in claim 3, characterized in that, The screen mirroring module is Live Link Vcam, an application software running on iOS devices.
5. A method for real-time shooting of digital animation, characterized in that, The real-time shooting method includes: A number of physical cameras are marked with motion capture points to construct a number of first virtual cameras, and the first virtual cameras are placed in a virtual motion capture field; there are multiple physical cameras and one-to-one correspondences between them; In Unreal Engine UE, data captured by the motion capture system is received in real time through the built-in LiveLink plugin. Several second virtual cameras with controllable parameters are constructed using the data structure BP_Switch of the virtual camera built-in plugin of Unreal Engine UE. The data structure BP_Switch is used to adjust the focal length, depth of field, top, bottom, left, and right offset parameters of the image captured by the second virtual cameras, as well as the blueprint for switching the shooting scene. By selecting the corresponding second virtual camera in the data structure BP_Switch in Unreal Engine UE's VCS, and setting the real-time connection node of the props in its details, the real-time shooting scene of the corresponding second virtual camera and the corresponding first virtual camera can be bound together. The display module is connected via the VirtualCamera remote session plugin in Unreal Engine to display the real-time shooting footage from the corresponding second virtual camera in the Unreal Engine module on the same screen.