Virtual and real camera posture correction method, device, storage medium and electronic equipment

Abstract

The present invention provides a method, device, storage medium and electronic device for correcting the posture of a virtual and real camera, and relates to the technical field of virtual production. The method first obtains a camera extrinsic parameter matrix, and then calculates a rotation vector R and a translation vector T according to the camera extrinsic parameter matrix. Then, a real camera coordinate P is generated by the rotation matrix RM and the translation vector T. Then, the real camera coordinate P and the rotation vector R are transformed to generate virtual camera coordinates P_oc and a virtual camera rotation vector RA respectively. Finally, the virtual camera posture is automatically corrected according to the virtual camera coordinates P_oc and the virtual camera rotation vector RA, so as to realize real-time automatic correction of the virtual camera posture according to the picture taken by the real camera, and ensure that the picture taken by the virtual and real camera has no deviation. Compared with the existing manual correction, the present invention greatly reduces the operation time and complexity of the virtual and real camera position correction in virtual production, and brings a better experience to virtual production.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual filming, and in particular to a virtual-real camera posture correction method, device, storage medium and electronic equipment. Background Art

[0002] Virtual production is a new production method that combines virtual computer images with real actors' performances and can visualize the special effects in real time on the set. In virtual production, the shooter does not have to endure the green screen and does not have to wait for weeks or even months for post-production. All key decision makers collaborate on-site to generate the final image in real time, which can be obtained immediately and modified at any time. Unreal Engine is not only widely used in the field of game production, but also an important tool in the virtual production process.

[0003] In virtual production, it is necessary to combine the scene images captured by the virtual camera in the Unreal Engine with the real images actually shot by the real camera for synthesis. In virtual production shooting, in order to ensure the consistency of the real camera and virtual camera images, it is necessary to correct the posture of the virtual and real cameras, that is, adjust the location (Location) and rotation (Rotation) of the virtual camera so that the virtual camera image can be synthesized with the image shot by the real camera without deviation in real time. The posture correction is a necessary but not sufficient condition for achieving the consistency of the virtual and real camera images. In order to achieve the consistency of the virtual and real camera images, a series of subsequent work such as automatic correction of lens distortion, automatic color matching, and automatic matching of lens focal length curves is also required.

[0004] Currently, manual correction of virtual and real camera poses is generally performed, which requires constant adjustment of relevant parameters and corresponding comparison, so it has the major drawback of being time-consuming and labor-intensive. Summary of the invention

[0005] The purpose of the present invention is to solve the above-mentioned problems mentioned in the background technology and to propose a virtual-real camera posture correction method, device, storage medium and electronic device.

[0006] To achieve the above object, the present invention first proposes a method for automatically correcting the virtual and real camera posture, comprising the following steps:

[0007] Get the camera extrinsic matrix;

[0008] Obtain the rotation vector R and the translation vector T in the camera extrinsic parameter matrix;

[0009] Transform the rotation vector R into a rotation matrix RM;

[0010] Generate the real camera coordinates P according to the formula P=RM^(-1)*T, where the real camera coordinates P are the position of the camera in the world coordinate system, RM is the rotation matrix, and T is the translation vector;

[0011] Performing coordinate transformation on the real camera coordinates P to generate virtual camera coordinates P_oc;

[0012] Performing coordinate transformation on the rotation vector R to generate a virtual camera rotation vector RA;

[0013] Changing the position of the camera in the Unreal Engine coordinate system according to the virtual camera coordinate P_oc;

[0014] Use the virtual camera rotation vector RA as the rotation axis and ||RA||*180 / π as the rotation angle to change the camera's posture in the Unreal Engine coordinate system.

[0015] Optionally, the Unreal Engine coordinate system is a UE coordinate system.

[0016] Optionally, the initial posture of the camera in the UE coordinate system is: the direction of the lens facing horizontally downward is the camera X axis, the direction of the lens facing horizontally right is the camera Y axis, and the direction of the lens facing the calibration plate is the camera Z axis.

[0017] Optionally, the coordinate transformation is a coordinate transformation from a right-handed coordinate system to a left-handed coordinate system.

[0018] Optionally, obtain the camera extrinsic matrix through checkerboard calibration.

[0019] Optionally, the rotation vector R is transformed into a rotation matrix RM by using the Rodrigues equation.

[0020] The present invention also proposes a device for automatically correcting the virtual and real camera posture, comprising:

[0021] A first acquisition module is configured to acquire a camera extrinsic parameter matrix;

[0022] A second acquisition module is configured to acquire a rotation vector R and a translation vector T in the camera extrinsic parameter matrix;

[0023] A first transformation module, configured to transform the rotation vector R into a rotation matrix RM;

[0024] A first coordinate generation module is configured to generate a real camera coordinate P according to a formula P=RM^(-1)*T, wherein the real camera coordinate P is a position of the camera in a world coordinate system, wherein RM is a rotation matrix and T is a translation vector;

[0025] A second coordinate generation module is configured to perform coordinate transformation on the real camera coordinate P to generate a virtual camera coordinate P_oc;

[0026] A second transformation module is configured to perform coordinate transformation on the rotation vector R to generate a virtual camera rotation vector RA;

[0027] A position module, configured to change the position of the camera in the Unreal Engine coordinate system according to the virtual camera coordinate P_oc;

[0028] The posture module is configured to change the posture of the camera in the Unreal Engine coordinate system by using the virtual camera rotation vector RA as the rotation axis and ||RA||*180 / π as the rotation angle.

[0029] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for automatically correcting the virtual and real camera posture is implemented.

[0030] The present invention also proposes an electronic device, an electronic device comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method of automatically correcting the virtual and real camera posture.

[0031] Beneficial effects of the present invention:

[0032] In a method for automatically correcting the virtual and real camera posture provided by an embodiment of the present invention, a camera extrinsic parameter matrix is ​​first obtained, and then a rotation vector R and a translation vector T are calculated according to the camera extrinsic parameter matrix, and then the real camera coordinates P are generated by the rotation matrix RM and the translation vector T, and then the real camera coordinates P and the rotation vector R are transformed to generate virtual camera coordinates P_oc and virtual camera rotation vector RA respectively, and finally the position and posture of the camera in the Unreal Engine coordinate system are changed according to the virtual camera coordinates P_oc and the virtual camera rotation vector RA respectively, so as to realize real-time automatic correction of the virtual camera posture according to the picture taken by the real camera, ensuring that the pictures taken by the virtual and real cameras are consistent and without deviation. Compared with the existing manual correction that requires continuous adjustment of relevant parameters and corresponding comparison, the embodiment of the present invention greatly reduces the operation time and complexity of virtual and real camera position correction in virtual production, and brings a better experience to virtual production.

[0033] Furthermore, a method for automatically correcting the virtual-real camera posture in an embodiment of the present invention uses a virtual camera rotation vector RA as the rotation axis and ||RA||*180 / π as the rotation angle to change the posture of the camera in the Unreal Engine coordinate system, thereby solving the problem of gimbal deadlock caused when adjusting the virtual-real camera posture with the help of Euler angles.

[0034] The features and advantages of the present invention will be described in detail through embodiments in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of a flow chart of a method for automatically correcting virtual and real camera postures according to an embodiment of the present invention;

[0036] Figure 2 The present invention is a structural block diagram of a device for automatically correcting virtual and real camera postures according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to facilitate understanding by those skilled in the art, the present invention will be further described in detail below in conjunction with specific embodiments.

[0038] Figure 1 The following is a schematic flow chart of a method for automatically correcting virtual and real camera postures according to an embodiment of the present invention. Figure 1 As shown, the method for automatically correcting the virtual and real camera posture includes steps S10 to S60:

[0039] S10, obtaining the camera extrinsic matrix;

[0040] S20, obtaining a rotation vector R and a translation vector T in the camera extrinsic parameter matrix;

[0041] S30, transforming the rotation vector R into a rotation matrix RM;

[0042] S40, generating a real camera coordinate P according to a formula P=RM^(-1)*T, wherein the real camera coordinate P is the position of the camera in the world coordinate system, wherein RM is a rotation matrix and T is a translation vector;

[0043] S50, performing coordinate transformation on the real camera coordinates P to generate virtual camera coordinates P_oc;

[0044] S60, performing coordinate transformation on the rotation vector R to generate a virtual camera rotation vector RA;

[0045] S70, changing the position of the camera in the Unreal Engine coordinate system according to the virtual camera coordinate P_oc;

[0046] S80, using the virtual camera rotation vector RA as the rotation axis and ||RA||*180 / π as the rotation angle to change the posture of the camera in the Unreal Engine coordinate system.

[0047] In a method for automatically correcting the virtual and real camera postures of an embodiment of the present invention, a camera extrinsic parameter matrix is ​​first obtained, and then a rotation vector R and a translation vector T are calculated according to the camera extrinsic parameter matrix, and then the real camera coordinates P are generated by the rotation matrix RM and the translation vector T, and then the real camera coordinates P and the rotation vector R are transformed to generate virtual camera coordinates P_oc and virtual camera rotation vector RA respectively, and finally the position and posture of the camera in the Unreal Engine coordinate system are changed according to the virtual camera coordinates P_oc and the virtual camera rotation vector RA respectively, so as to realize real-time automatic correction of the virtual camera posture according to the pictures taken by the real camera, ensuring that the pictures taken by the virtual and real cameras are consistent and without deviation. Compared with the existing manual correction that requires continuous adjustment of relevant parameters and corresponding comparison, the present invention greatly reduces the operation time and complexity of virtual and real camera position correction in virtual production, and brings a better experience to virtual production.

[0048] In the following, each step of a method for automatically correcting virtual and real camera postures in an embodiment of the present invention will be described in more detail with reference to the accompanying drawings and embodiments.

[0049] In step S10, the camera extrinsic parameter matrix is ​​obtained.

[0050] In one embodiment of the present invention, the camera extrinsic parameter matrix is ​​obtained through chessboard calibration.

[0051] It should be noted that in the general process of camera calibration, the camera calibration needs to be calibrated with the help of a checkerboard calibration plate. This algorithm (A Flexible New Technique for Camera Calibration) was proposed by Zhang Zhengyou in 1998 as a general algorithm for solving the camera's intrinsic parameter matrix and extrinsic parameter matrix. The real camera can be calibrated by photographing the calibration plate from different angles to obtain the camera's extrinsic parameter matrix and intrinsic parameter matrix. Because the checkerboard calibration algorithm is a commonly used method in camera calibration, it will not be repeated here. In this embodiment, only the camera extrinsic parameter matrix is ​​obtained, where the camera extrinsic parameter matrix contains the relative position information of the camera coordinate system and the world coordinate system. In other embodiments, the camera extrinsic parameter matrix and the camera intrinsic parameter matrix can be obtained at the same time.

[0052] Step S20, obtaining the rotation vector R and the translation vector T in the camera extrinsic parameter matrix.

[0053] In one embodiment, the PNP (perspective-n-point) algorithm is used to calculate the intrinsic parameter matrix and the extrinsic parameter matrix to obtain the rotation vector R and the translation vector T. In other embodiments, other known methods may be used to obtain the rotation vector R and the translation vector T in the camera extrinsic parameter matrix, so they are not described in detail here.

[0054] Step S30, transforming the rotation vector R into a rotation matrix RM.

[0055] When dealing with three-dimensional rotation problems, both the rotation vector and the rotation matrix can represent the rotation transformation, but since the rotation matrix is ​​needed to calculate the position of the camera in the world coordinate system, in order to simplify the calculation, in step S30, the rotation vector R is first transformed into the rotation matrix RM. In one embodiment of the present invention, the rotation vector R can be transformed into the rotation matrix RM by using the Rodriguez equation.

[0056] Step S40, generating real camera coordinates P according to the formula P=RM^(-1)*T, wherein the real camera coordinates P are the position of the camera in the world coordinate system, wherein RM is a rotation matrix and T is a translation vector.

[0057] When calculating the absolute position of the camera in the Unreal Engine coordinate system, it is necessary to consider the conversion relationship between the Unreal Engine coordinate system, the world coordinate system, and the camera coordinate system. In one embodiment of the present invention, the Unreal Engine is a UE4 engine. In other embodiments, the Unreal Engine may also be a Unity 3D engine or an engine developed by itself.

[0058] In one embodiment, assuming that the position of the first object in the camera coordinate system is Pcam, the position of the first object in the world coordinate system is Pworld, and Pcam and Pworld satisfy the following formula:

[0059] Pcam=RM*Pworld+T. (1)

[0060] According to formula (1), the real camera coordinates P = RM^(-1)*T are generated. The real camera coordinates P can be expressed as (P1, P2, P3).

[0061] Step S50, performing coordinate transformation on the real camera coordinates P to generate virtual camera coordinates P_oc.

[0062] In one embodiment, since the real camera coordinates P are calculated with the help of OpenCV library, OpenCV is a right-handed coordinate system, and UE4 uses DirectX for rendering, so UE4 is a left-handed coordinate system. In step S50, the transformation between the left-handed coordinate system and the right-handed coordinate system needs to be considered, and the virtual camera coordinates P_oc=(-P1, P2, P3) are finally generated by negatively taking the value on the X-axis of the real camera coordinates P. In other embodiments, Unity3D is also a left-handed coordinate system, and the same method can be used for processing, so it will not be repeated here.

[0063] Step S60, performing coordinate transformation on the rotation vector R to generate a virtual camera rotation vector RA.

[0064] Since any rotation can be represented by a rotation axis and a rotation angle, the rotation vector is a three-dimensional vector whose direction is consistent with the rotation axis and whose length is equal to the rotation angle. Such a three-dimensional vector can describe the rotation. Considering the difference between the left-handed and right-handed coordinate systems, the value on the X axis of the rotation vector R is negative to generate the virtual camera rotation vector RA, that is, RA = (-R1, R2, R3).

[0065] Step S70: changing the position of the camera in the Unreal Engine coordinate system according to the virtual camera coordinate P_oc.

[0066] In one embodiment, since a method for automatically correcting the virtual and real camera pose is derived from Zhang Zhengyou's calibration algorithm, the position of the camera in the Unreal Engine coordinate system is the relative position of the camera and the calibration plate, that is, the position in the coordinate system with the coordinates of the first corner point of the calibration plate as the origin. In actual operation, this offset needs to be taken into account, and the position of the first corner point of the calibration plate can be set as the origin of the UE coordinate system.

[0067] Step S80, using the virtual camera rotation vector RA as the rotation axis and ||RA||*180 / π as the rotation angle to change the posture of the camera in the Unreal Engine coordinate system.

[0068] After generating the rotation vector R, it can be rotated with the help of Euler angles. Since the rotation angle calculated by Euler angles depends on the rotation order between the three axes, and in some specific cases, it will cause the gimbal deadlock, thereby losing one dimension of freedom. The embodiment of the present invention uses the virtual camera rotation vector RA as the rotation axis and ||RA||*180 / π as the rotation angle to change the posture of the camera in the Unreal Engine coordinate system, thereby solving the above-mentioned problem of gimbal deadlock in certain specific cases. In addition, the initial posture of the camera in the UE coordinate system is: the horizontal downward direction of the lens is the camera X-axis, the horizontal right direction of the lens is the camera Y-axis, and the direction of the lens toward the calibration plate is the camera Z-axis.

[0069] Based on the above method for automatically correcting the virtual and real camera posture, the embodiment of the present invention also provides a device for automatically correcting the virtual and real camera posture, such as Figure 2 As shown, the device includes the following modules:

[0070] A first acquisition module 100 is configured to acquire a camera extrinsic parameter matrix;

[0071] A second acquisition module 200 is configured to acquire a rotation vector R and a translation vector T in the camera extrinsic parameter matrix;

[0072] A first transformation module 300, configured to transform the rotation vector R into a rotation matrix RM;

[0073] The first coordinate generation module 400 is configured to generate a real camera coordinate P according to the formula P=RM^(-1)*T, wherein the real camera coordinate P is the position of the camera in the world coordinate system, where RM is a rotation matrix and T is a translation vector;

[0074] The second coordinate generation module 500 is configured to perform coordinate transformation on the real camera coordinate P to generate a virtual camera coordinate P_oc;

[0075] The second transformation module 600 is configured to perform coordinate transformation on the rotation vector R to generate a virtual camera rotation vector RA;

[0076] A position module 700, configured to change the position of the camera in the Unreal Engine coordinate system according to the virtual camera coordinate P_oc;

[0077] The posture module 800 is configured to change the posture of the camera in the Unreal Engine coordinate system by using the virtual camera rotation vector RA as the rotation axis and ||RA||*180 / π as the rotation angle.

[0078] In summary, the apparatus for automatically correcting the virtual and real camera posture according to the embodiment of the present invention can be implemented in the form of a program and run on a computer device. The memory of the computer device can store various program modules constituting the apparatus for automatically correcting the virtual and real camera posture, for example, Figure 2 The first acquisition module 100, the second acquisition module 200, the first transformation module 300, the first coordinate generation module 400, the second coordinate generation module 500, the second transformation module 600, the position module 700, and the posture module 800 are shown. The program composed of each program module enables the processor to execute the steps of the method for automatically correcting the virtual and real camera posture of each embodiment of the present application described in this specification.

[0079] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the steps in a method for automatically correcting virtual and real camera postures in each embodiment of the present application are implemented.

[0080] An embodiment of the present invention also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the steps of a method for automatically correcting the virtual and real camera posture of each embodiment of the present application.

[0081] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0082] The above embodiments are explanations of the present invention, not limitations of the present invention. Any solution that is simply modified from the present invention belongs to the protection scope of the present invention. The above is only a preferred embodiment of the present invention. The protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A method for automatically correcting virtual and real camera posture, characterized in that: The following steps are involved: Get the camera extrinsic matrix; Obtain the rotation vector R and the translation vector T in the camera extrinsic parameter matrix; Transform the rotation vector R into a rotation matrix RM; Generate the real camera coordinates P according to the formula P=RM^(-1)*T, where the real camera coordinates P are the position of the camera in the world coordinate system, RM is the rotation matrix, and T is the translation vector; Performing coordinate transformation on the real camera coordinates P to generate virtual camera coordinates P_oc; Performing coordinate transformation on the rotation vector R to generate a virtual camera rotation vector RA; Changing the position of the camera in the Unreal Engine coordinate system according to the virtual camera coordinate P_oc; Use the virtual camera rotation vector RA as the rotation axis and ||RA||*180 / π as the rotation angle to change the camera's posture in the Unreal Engine coordinate system.

2. The method for automatically correcting virtual and real camera posture according to claim 1, characterized in that: The Unreal Engine coordinate system is the UE coordinate system.

3. The method for automatically correcting virtual and real camera posture according to claim 2, characterized in that: The initial posture of the camera in the UE coordinate system is: the camera X-axis is the direction of the lens pointing horizontally downward, the camera Y-axis is the direction of the lens pointing horizontally to the right, and the camera Z-axis is the direction of the lens pointing toward the calibration plate.

4. The method for automatically correcting virtual and real camera posture according to claim 1, characterized in that: The coordinate transformation is a coordinate transformation from a right-handed coordinate system to a left-handed coordinate system.

5. The method for automatically correcting virtual and real camera posture according to claim 1, characterized in that: The camera extrinsic matrix is ​​obtained through checkerboard calibration.

6. The method for automatically correcting virtual and real camera posture according to claim 1, characterized in that: The rotation vector R is transformed into a rotation matrix RM by the Rodrigues equation.

7. A device for automatically correcting virtual and real camera posture, characterized in that: include: A first acquisition module is configured to acquire a camera extrinsic parameter matrix; A second acquisition module is configured to acquire a rotation vector R and a translation vector T in the camera extrinsic parameter matrix; A first transformation module, configured to transform the rotation vector R into a rotation matrix RM; A first coordinate generation module is configured to generate a real camera coordinate P according to a formula P=RM^(-1)*T, wherein the real camera coordinate P is a position of the camera in a world coordinate system, wherein RM is a rotation matrix and T is a translation vector; A second coordinate generation module is configured to perform coordinate transformation on the real camera coordinate P to generate a virtual camera coordinate P_oc; A second transformation module is configured to perform coordinate transformation on the rotation vector R to generate a virtual camera rotation vector RA; A position module, configured to change the position of the camera in the Unreal Engine coordinate system according to the virtual camera coordinate P_oc; The posture module is configured to change the posture of the camera in the Unreal Engine coordinate system by using the virtual camera rotation vector RA as the rotation axis and ||RA||*180 / π as the rotation angle.

8. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for automatically correcting the virtual and real camera posture according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the method for automatically correcting the virtual and real camera posture according to any one of claims 1 to 6.

Images