Virtual preview method and system based on camera robot and Unreal Engine

Abstract

The invention provides a virtual rehearsal method and a virtual rehearsal system based on a camera robot and Unreal Engine in order to solve the problem that a lens is perfectly duplicated during split rehearsal during actual field shooting and improve the accuracy and the shooting efficiency of virtual rehearsal, so that a split mirror is designed in the Unreal Engine in a split mirror rehearsal stage, and the rehearsal accuracy and the shooting efficiency of the virtual rehearsal are improved. In the actual shooting stage, the camera shooting robot executes the lens track of the sub-lens, synchronizes the virtual camera in the Unreal Engine in real time, and outputs a virtual-real composite picture.

Description

Technical Field

[0001] The present invention relates to the technical fields of virtual reality technology, film shooting and production, and robot control systems, and in particular to a virtual preview method and system based on a camera robot and Unreal Engine. Background Art

[0002] In the existing entertainment, advertising, film and other fields, the application of virtual pre-production systems is becoming more and more widespread. These systems provide an immersive experience by simulating real scenes, which can save time and cost in the design and creation process. At the same time, in the field of film and television shooting, the control of cameras is becoming more and more precise. Among them, camera robots, as camera motion control systems, can achieve accurate and stable shooting effects.

[0003] In the existing virtual preview system, there is no good combination with the camera robot, so there are some unavoidable defects. In the actual shooting process, it is very difficult for the photographer to accurately simulate complex lens movements and photography techniques, such as specific camera angles, lens focal lengths, and movement methods. This difficulty may cause the photographer to waste time repeatedly adjusting the lens movement during shooting, and even cause the final film to be significantly different from the expectations, thereby increasing the cost of post-production modification or reshooting. However, by using a camera robot, it is easy to achieve accurate reproduction of the lens trajectory, thereby solving this problem.

[0004] There is a problem with traditional camera robots. They cannot synchronize with the virtual camera in the virtual scene in real time during the actual shooting process, which makes it impossible for the director to monitor the synthesis effect of the shooting picture and the virtual scene in real time, and may require additional modifications in the later stage. To solve this problem, one way is to add an additional camera tracking system, but this system is usually expensive and will increase the cost of shooting. Summary of the invention

[0005] The present invention aims to solve the defects of the prior art, and provides a virtual preview method and system based on the camera robot and Unreal Engine by combining the camera robot and Unreal Engine, so that the lens trajectory during the storyboard preview can be perfectly reproduced, and the camera robot is synchronized with the virtual camera in the virtual scene of Unreal Engine in real time during the actual shooting. In this way, the accuracy of virtual preview and shooting efficiency can be significantly improved.

[0006] Implementation steps: Step 1: Create a 3D model of the camera robot in Unreal Engine, including the camera, gimbal, cage, robotic arm, robot base, and track. Step 2: Perform forward kinematics and inverse kinematics binding on the joints and tracks of the robot arm in the camera robot model, so that the user can control the entire camera robot system by simply adjusting the position of the camera in the three-dimensional scene; Step 3: Add constraints to the camera robot, including the limit motion angle, limit speed and limit acceleration of each joint. These limit values ​​depend on the design and mechanical structure of the robot. Step 4. Set the virtual camera internal parameters in Unreal Engine to the parameters required for the storyboard, including camera sensor size, focal length, focus distance, and aperture; Step 5: Create an animation sequence in Unreal Engine to make a shot storyboard. The animation sequence includes the camera's movement trajectory, the lens's change trajectory, time length, special effect triggering, etc. Step 6. Convert the animation sequence of Unreal Engine into a trajectory file executable by the camera robot. Resample the animation sequence curve to meet the frequency requirement of 4ms per frame of the camera robot. According to the camera's mirror movement trajectory, calculate the motion trajectory of each robot joint and track through inverse kinematic binding. The final trajectory file contains the time of each frame, the value of each robot joint angle, the value of the track, the value of the focus distance, the value of the focal length and the value of the aperture; Step 7: To ensure that the real camera robot can accurately reproduce the motion trajectory of the camera robot in the virtual environment, the following legitimacy check is performed: Check whether the joint angle, velocity, acceleration and trajectory values ​​of each frame exceed the preset threshold.

[0007] If the trajectory fails the validity check, the system will provide a specific time point and out-of-range value to indicate the location of the illegal value. The user needs to modify the animation sequence according to the prompt to ensure that the trajectory can pass the validity check.

[0008] Step 8: Export the trajectory file executable by the camera robot and send it to the camera robot for execution during on-site shooting.

[0009] Step 9. Establish a connection between Unreal Engine and the camera robot through UDP communication. Obtain the end position of the camera robot and the internal parameter data of the camera in real time. Through calculation, the world coordinates, focal length, focus distance, optical center and other parameters of the camera will be obtained. In this way, the camera in the virtual scene in Unreal Engine can be synchronized with the camera in the real world in real time. By directly monitoring the screen, the user can observe the final synthesis result in real time.

[0010] The beneficial effects of the present invention are: Design storyboard previews in Unreal Engine, where powerful rendering capabilities provide more realistic visual effects; real-time preview and modification save time and costs; The camera robot is used to reproduce the lens trajectory of the storyboard, which improves the accuracy of the preview and saves the photographer's time cost in adjusting and executing the storyboard; The storyboard track includes the triggering of virtual special effects and on-site physical devices, which improves the accuracy of pre-performance; The camera controlled by the camera robot and the virtual camera in Unreal Engine are synchronized to provide real-time composite image monitoring, reducing the risk of post-production adjustments and avoiding the cost of real-time tracking equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings: Attached Figure 1 The present invention is an embodiment of a virtual pre-production method and system based on a camera robot and Unreal Engine.

[0012] Attached Figure 2 The present invention is an on-site shooting flow chart of a virtual pre-performance method and system based on a camera robot and Unreal Engine according to an embodiment of the present invention.

[0013] Attached Figure 3 The present invention is a block diagram of a virtual preview method and system based on a camera robot and Unreal Engine according to an embodiment of the present invention. Implementation

[0014] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0015] The idea of ​​the present invention is to provide a virtual preview method and system based on the camera robot and Unreal Engine by combining the camera robot and Unreal Engine, so that the lens trajectory during the storyboard preview can be perfectly reproduced, and the camera robot is synchronized with the virtual camera in the Unreal Engine virtual scene in real time during the actual shooting. The present invention includes the following steps: Step 1: Create a 3D model of the camera robot in Unreal Engine, including the camera, gimbal, cage, robotic arm, robot base, and track.

[0016] Step 2: Perform forward kinematics and inverse kinematics binding on the joints and tracks of the robotic arm in the camera robot model, so that the user can control the entire camera robot system by simply adjusting the position of the camera in the three-dimensional scene.

[0017] Step 3: Add constraints to the camera robot, including the limit motion angle, limit speed and limit acceleration of each joint. These limit values ​​depend on the design and mechanical structure of the robot.

[0018] Step 4. Set the virtual camera internal parameters in Unreal Engine to the parameters required for the storyboard, including camera sensor size, focal length, focus distance, and aperture.

[0019] Step 5: Create an animation sequence in Unreal Engine to make a shot storyboard. The animation sequence includes the camera's movement trajectory, the lens's change trajectory, time length, special effect triggering, etc.

[0020] Step 6. Convert the animation sequence of Unreal Engine into a trajectory file executable by the camera robot. Resample the animation sequence curve to meet the camera robot's 4ms frequency requirement per frame. According to the camera's mirror trajectory, calculate the motion trajectory of each robot joint and track through inverse kinematic binding. The final trajectory file contains the time of each frame, the values ​​of each robot joint angle, the track value, the focus distance value, the focal length value, and the aperture value.

[0021] Step 7: To ensure that the real camera robot can accurately reproduce the motion trajectory of the camera robot in the virtual environment, the following legitimacy check is performed: Check whether the joint angle, velocity, acceleration and trajectory values ​​of each frame exceed the preset threshold.

[0022] If the trajectory fails the validity check, the system will provide a specific time point and out-of-range value to indicate the location of the illegal value. The user needs to modify the animation sequence according to the prompt to ensure that the trajectory can pass the validity check.

[0023] Step 8: Export the trajectory file executable by the camera robot and send it to the camera robot for execution during on-site shooting.

[0024] Step 9: Establish a connection between Unreal Engine and the camera robot through UDP communication. Obtain the end position of the camera robot and the internal parameter data of the camera in real time.

[0025] Step 10: Through calculation, the world coordinates, focal length, focus distance, optical center and other parameters of the camera are obtained. In this way, the camera in the virtual scene in Unreal Engine can be synchronized with the camera in the real world in real time. By directly monitoring the screen, the user can observe the final synthesis result in real time.

[0026] like Figure 3 A virtual pre-performance method and system based on a camera robot and Unreal Engine is shown. The system is modularly designed and includes a camera robot construction module, a storyboard animation sequence module, a trajectory generation module, and a camera robot real-time synchronization module, wherein: Camera robot construction module: The 3D model construction module is used to construct a 3D 1:1 model of the camera robot; the kinematics construction module is used to construct the robot's kinematic model through the robot's DH parameters; the camera construction module is used to construct a virtual camera model through the camera's internal reference data; Storyboard animation sequence module: used to design storyboard previews to form an animation sequence; Trajectory generation module: The trajectory conversion module is used to convert the animation sequence into a trajectory executable by the camera robot; the trajectory verification module is used to verify whether the trajectory can be correctly executed by the camera robot; the trajectory file generation module is used to generate a trajectory file in a format readable by the camera robot; Camera robot real-time synchronization module: used to synchronize the camera controlled by the camera robot with the virtual camera in UnrealEngine in real time.

[0027] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A virtual preview method and system based on a camera robot and Unreal Engine, characterized in that: The following steps are involved: S1. Build a 3D model of the camera robot in Unreal Engine, including the camera, gimbal, cage, robotic arm, robot base, and track; S2. Perform forward kinematics and inverse kinematics binding on the joints and tracks of the robot arm in the camera robot model, so that the user can control the entire camera robot system by simply adjusting the position of the camera in the three-dimensional scene; S3, adding constraints to the camera robot, including the limit motion angle, limit speed and limit acceleration of each joint, these limit values ​​depend on the design and mechanical structure of the robot; S4. Set the virtual camera internal parameters in Unreal Engine to the parameters required for the storyboard, including camera sensor size, focal length, focus distance, and aperture; S5. Create animation sequences in Unreal Engine for storyboarding. S6. Convert the animation sequence of Unreal Engine into a trajectory file executable by the camera robot; S7. To ensure that the real camera robot can accurately reproduce the motion trajectory of the camera robot in the virtual environment, perform trajectory legitimacy verification; S8, exporting a trajectory file executable by the camera robot, and sending it to the camera robot during on-site shooting; S9, executing the trajectory file, the camera robot controls the camera to execute the storyboard trajectory, and synchronously triggers special effects and on-site physical devices; S10. Establish a connection between Unreal Engine and the camera robot through UDP communication. Obtain the end position of the camera robot and the internal parameter data of the camera in real time; S11. Output the virtual-real composite image to the monitoring device in real time through Unreal Engine.

2. A virtual pre-performance method and system based on a camera robot and Unreal Engine according to claim 1, characterized in that: The camera robot three-dimensional model described in step 1 includes: a camera, a pan / tilt, a cage, a robotic arm, a robot base, and a track.

3. A virtual pre-performance method and system based on a camera robot and Unreal Engine according to claim 2, characterized in that: The shot storyboard animation sequence described in step 5 specifically includes: the camera's movement trajectory, the lens's change trajectory, the time length, the virtual special effect triggering node, and the on-site physical device triggering node.

4. A virtual pre-performance method and system based on a camera robot and Unreal Engine according to claim 3, characterized in that: The specific steps for converting the Unreal Engine animation sequence described in step 6 into a trajectory file executable by the camera robot are as follows: S61, resampling all curves of the animation sequence to one frame per 4 ms to meet the camera robot frequency requirement; S62, according to the camera's movement trajectory, calculate the movement trajectory of each mechanical arm joint and track of the camera robot through inverse kinematics binding; S63. The finally generated trajectory file includes the time of each frame, the values ​​of the angles of each robot arm joint, the values ​​of the track, the values ​​of the focus distance, the values ​​of the focal length and the values ​​of the aperture.

5. A virtual pre-performance method and system based on a camera robot and Unreal Engine according to claim 4, characterized in that: The specific steps of the trajectory legitimacy check described in step 7 are as follows: S71, checking whether the joint angle, velocity, acceleration and trajectory values ​​of each frame exceed a preset threshold; S72. If the trajectory fails the legality check, the system will provide a specific time point and an out-of-range value to indicate the location of the illegal value.

6. A virtual pre-performance method and system based on a camera robot and Unreal Engine according to claim 5, characterized in that: The specific steps of outputting the virtual-real composite image in step 11 are as follows: S111, processing the camera robot terminal position and camera memory data, calculating the camera's world coordinates, focal length, focus distance and optical center, and synchronizing the virtual camera in Unreal Engine; S112, rendering an image of a virtual camera viewport through Unreal Engine; S113, collecting images taken by a real camera through a capture card, and performing image matting processing; S114: The synthetically rendered virtual image and the real shot image are output to a monitoring device via a capture card.

Images