Method and device for aggregating camera tracking data and lens data in virtual shooting

Abstract

The invention discloses an intelligent recombination method and device for multi-path camera tracking metadata in virtual shooting, and belongs to the field of virtual shooting, and the method specifically comprises the steps: obtaining pose metadata packets collected by at least two paths of camera tracking systems in real time; resolving the frame rate, the equipment feature code and the pose data of the multi-path camera in real time; judging the stability of the pose data of the multi-path camera; judging the quantity and coupling of the stably transmitted multi-path pose data of the camera tracking system, calculating the finally output pose data in real time according to a preset strategy and a coupling threshold, and packaging the finally output pose data into a real-time metadata packet; and sending the real-time metadata packet to a terminal carrying a virtual production engine in a UDP (User Datagram Protocol) communication protocol so as to control a virtual camera in the virtual production engine. The problems that a current single camera tracking system is unstable and data transmission is prone to packet loss are solved, and reliable guarantee is provided for creation stability of a virtual shooting site.

Description

Technical Field

[0001] The present application relates to the technical field of film and television virtual shooting, and in particular to a method and device for aggregating camera tracking data and lens data in virtual shooting. Background Art

[0002] In recent years, with the advancement of film and television production technology, film and television shooting and production methods have also undergone tremendous changes and updates, and virtual shooting technology has gradually become the mainstream shooting method in current film and television production. In the current VP virtual shooting, most of the pose data of the camera tracking system and the lens data of the focus system are transmitted independently and enter the virtual engine system separately. They cannot be effectively combined into a virtual camera object, which often causes the inner cone to fail to render correctly and cannot meet the shooting requirements. In XR virtual shooting, most mainstream XR fusion systems currently only support a single device to provide camera data, and do not support the aggregation of data solutions from different devices.

[0003] How to receive camera pose and lens data from different devices and aggregate data from different devices and solutions has become an urgent problem to be solved. Summary of the invention

[0004] In order to solve at least one of the above technical problems, the present application provides a method and device for aggregating camera tracking data and lens data in virtual shooting.

[0005] In a first aspect, the present application provides a method for aggregating camera tracking data and lens data in virtual shooting, which adopts the following technical solution:

[0006] A method for aggregating camera tracking data and lens data in virtual shooting, the method being executed in a server, comprising:

[0007] S1: acquiring posture data collected by a camera tracking system in real time, the posture data including three-axis translation data and three-axis rotation data, wherein the camera tracking system is installed in a constructed virtual shooting studio;

[0008] S2: acquiring lens data collected by a camera focus system in real time, the lens data including focus, focal length and aperture data of the camera lens, wherein the camera focus system is installed on a physical camera lens in a constructed virtual shooting studio;

[0009] S3: Solving, processing and aggregating the camera pose data obtained in step S1 and the camera lens data obtained in step S2 to obtain a real-time metadata package suitable for virtual shooting;

[0010] S4: Send the real-time metadata packet to the terminal equipped with the virtual production engine via the UDP communication protocol to control the virtual camera in the virtual production engine.

[0011] Optionally, models of the camera tracking system include HTC Mars Camtrack, ViconVero, Mo-SysStarTracker Max, Zeiss CineCraft, Stype RedSpy, Optitack Prime41 and Qingtong MC4000.

[0012] Optionally, models of the camera focus system include Arri UMC-4 and MovcamXRport.

[0013] Optionally, step S3 specifically includes:

[0014] S31, through the camera posture transmission data protocol, the posture data P transmitted by the camera tracking system is calculated, P = {θ Pitch ,θ yaw ,θ roll , t z , t y , t x}, where -180°≤θ Pitch ,θ yaw ,θ roll <180°, representing the rotation value of the camera around x, y, and z, respectively, t z , t y , t x Represents the values ​​of the camera translation along z, y, and x respectively;

[0015] S32, through the camera lens transmission data protocol, the lens data L transmitted by the camera focus system is obtained by solving, L = {μ zoom , μ focus , μ iris},μ zoom , μ focus , μ iris Respectively represent the focal length, focus and aperture value of the lens;

[0016] S33, solving and processing the frame rates of the tracking system and the focus tracking system, and calculating the output frame rate f of the metadata packet;

[0017] S34, reorganizing and aggregating the obtained posture data P and lens data L based on the output frame rate f and the preset output strategy to obtain the real-time metadata package data.

[0018] Further optionally, the camera posture transmission data protocol includes a FreeD protocol, a VirtualRealityPeripheral Network (VRPN) protocol, a Vicon Datastream protocol and an Ncam protocol.

[0019] Further optionally, the camera lens transmission data protocol includes FreeD protocol, Ncam protocol, Arrimetadata protocol and Movcam metadata protocol.

[0020] Further optionally, step S33 specifically includes:

[0021] S331, using formula (1) to calculate the frame rate of the tracking system:

[0022] f p =1 / N p (2)

[0023] Among them, N p Indicates the number of times the tracking system sends tracking data in one second;

[0024] S332, using formula (2) to calculate the frame rate of the focus tracking system:

[0025] f l =1 / N l (2)

[0026] Wherein, Nl represents the number of times the focus system sends lens data packets within one second;

[0027] S333, using formula (3) to calculate the output frame rate of the metadata packet:

[0028]

[0029] Further optionally, the preset output strategy is represented by a preset strategy identifier, the preset strategy identifier is 2 bytes in total, wherein 01 represents direct tracking data, 02 represents direct lens data, and 03 represents reorganized tracking data and lens data.

[0030] Further optionally, step S34 specifically includes:

[0031] S341, when the preset strategy identifier is 01, output data O=P, and the output frame rate is the tracking system frame rate f p ;

[0032] S342, when the preset strategy identifier is 02, the output data O=L, the output frame rate is the tracking system frame rate f l ;

[0033] S343, when the preset strategy identifier is 03, output data The output frame rate is the output frame rate f calculated in S333, where It means to normalize and combine two sets of data.

[0034] Optionally, the real-time metadata data packet is encoded using the FreeD data protocol through the UDP communication protocol and then sent to a terminal equipped with a virtual production engine.

[0035] In a second aspect, the present application provides a device for aggregating camera tracking data and lens data in virtual shooting, which adopts the following technical solution:

[0036] A device for aggregating camera tracking data and lens data in virtual shooting, applied to a server, comprising:

[0037] A data acquisition module, used to acquire in real time the posture data collected by the camera tracking system and the lens data collected by the camera focus system, wherein the camera tracking system is installed in the constructed virtual shooting studio, and the camera focus system is installed on the physical camera lens in the constructed virtual shooting studio;

[0038] A data solving module is used to solve the acquired posture data, lens data and output frame rate, wherein the posture data includes three-axis translation data and three-axis rotation data, and the lens data includes the focus, focal length and aperture data of the camera lens;

[0039] A data aggregation module is used to calculate and aggregate the output metadata package data according to the calculated posture data, lens data, output frame rate and preset strategy;

[0040] The real-time data sending module is used to send the real-time metadata package to a terminal equipped with a virtual production engine to control a virtual camera in the virtual production engine.

[0041] The beneficial technical effects of the present invention are:

[0042] The present application provides a method and device for aggregating camera tracking data and lens data in virtual shooting. By acquiring the camera posture data collected by the camera tracking system and the camera lens data collected by the camera focus system, and solving, processing and aggregating the corresponding data, a real-time metadata package suitable for virtual shooting is obtained. This solves the problem that the current camera posture data and lens metadata are independently connected to the virtual shooting system, cannot be split, aggregated, and cannot be hot-switched between different schemes. This provides a reliable guarantee for reducing the cost of virtual shooting equipment, improving the efficiency of virtual shooting, and increasing the flexibility of shooting scene creation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of a method for aggregating camera tracking data and lens data in virtual shooting according to the present invention;

[0044] Figure 2It is a module architecture block diagram of a device for aggregating camera tracking data and lens data in virtual shooting provided by one embodiment of the present invention; DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0047] Reference Figure 1 The embodiment of the present invention discloses a method for aggregating camera tracking data and lens data in virtual shooting. The method is executed in a server and includes the following steps:

[0048] S1: acquiring posture data collected by a camera tracking system in real time, the posture data including three-axis translation data and three-axis rotation data, wherein the camera tracking system is installed in a constructed virtual shooting studio.

[0049] Build a virtual filming studio, set up the camera, install the camera tracking system in the studio, and add a virtual camera object to the virtual filming scene. The available tracking system models include but are not limited to HTC Mars Camtrack, Vicon Vero, Mo-Sys StarTracker Max, ZeissCineCraft, Stype RedSpy, Optitack Prime41 and Qingtong MC4000.

[0050] S2: acquiring lens data collected by a camera focus system in real time, the lens data including focus, focal length and aperture data of the camera lens, wherein the camera focus system is installed on a physical camera lens in a constructed virtual shooting studio.

[0051] Install the camera focus system on the camera lens. Available focus system models include but are not limited to Arri UMC-4 and Movcam XRport.

[0052] S3: Solve, process and aggregate the camera pose data obtained in step S1 and the camera lens data obtained in step S2 to obtain a real-time metadata package suitable for virtual shooting.

[0053] Step S3 specifically includes:

[0054] S31, through the camera posture transmission data protocol, the posture data P transmitted by the camera tracking system is calculated, P = {θ Pitch ,θyaw ,θ roll , t z , t y , t x}, where -180°≤θ Pitch ,θ yaw ,θ roll <180°, representing the rotation value of the camera around x, y, and z, respectively, t z , t y , t x Represents the value of the camera translation along z, y, and x respectively.

[0055] S32, through the camera lens transmission data protocol, the lens data L transmitted by the camera focus system is obtained by solving, L = {μ zoom , μ focus , μ iris},μ zoom , μ focus , μ iris They represent the focal length, focus and aperture values ​​of the lens respectively.

[0056] S33, solving and processing the frame rates of the tracking system and the focus tracking system, and calculating the output frame rate f of the metadata package.

[0057] Among them, in this embodiment, the camera posture transmission data protocol includes the FreeD protocol, the VirtualRealityPeripheral Network (VRPN) protocol, the Vicon Datastream protocol and the Ncam protocol; the camera lens transmission data protocol includes the FreeD protocol, the Ncam protocol, the Arri metadata protocol and the Movcam metadata protocol.

[0058] Step S33 specifically includes:

[0059] S331, using formula (1) to calculate the frame rate of the tracking system:

[0060] f p =1 / N p (3)

[0061] Among them, N p Indicates the number of times the tracking system sends tracking data in one second.

[0062] S332, using formula (2) to calculate the frame rate of the focus tracking system:

[0063] f l =1 / N l (2)

[0064] Among them, N lIndicates the number of times the focus system sends lens data packets in one second.

[0065] S333, using formula (3) to calculate the output frame rate of the metadata packet:

[0066]

[0067] S34, reorganizing and aggregating the obtained posture data P and lens data L based on the output frame rate f and the preset output strategy to obtain the final output real-time metadata package data.

[0068] In this embodiment, the preset output strategy is represented by a preset strategy identifier, which has 2 bytes in total, where 01 represents direct tracking data, 02 represents direct lens data, and 03 represents reorganized tracking data and lens data. Step S34 specifically includes:

[0069] S341, when the preset strategy identifier is 01, output data O=P, and the output frame rate is the tracking system frame rate f p .

[0070] S342, when the preset strategy identifier is 02, the output data O=L, the output frame rate is the tracking system frame rate f l .

[0071] S343, when the preset strategy identifier is 03, output data The output frame rate is the output frame rate f calculated in S333, where It means to normalize and combine two sets of data.

[0072] S4: Send the real-time metadata packet to the terminal equipped with the virtual production engine via the UDP communication protocol to control the virtual camera in the virtual production engine.

[0073] In this embodiment, the real-time metadata packet for virtual shooting is encoded by FreeD data protocol through UDP communication protocol and sent to the terminal equipped with the virtual production engine to control the virtual camera in the virtual production engine.

[0074] In one embodiment, Figure 2 As shown, a device for aggregating camera tracking data and lens data in virtual shooting is provided, including the following program modules:

[0075] The data acquisition module 201 is used to acquire in real time the posture data collected by the camera tracking system and the lens data collected by the camera focus system, wherein the camera tracking system is installed in the constructed virtual shooting studio, and the camera focus system is installed on the physical camera lens in the constructed virtual shooting studio;

[0076] A data solving module 202 is used to solve the acquired posture data, lens data and output frame rate, wherein the posture data includes three-axis translation data and three-axis rotation data, and the lens data includes the focus, focal length and aperture data of the camera lens;

[0077] A data aggregation module 203 is used to calculate and aggregate output metadata package data according to the calculated posture data, lens data, output frame rate and preset strategy;

[0078] The real-time data sending module 204 is used to send the real-time metadata package to a terminal equipped with a virtual production engine to control a virtual camera in the virtual production engine.

[0079] The above are preferred embodiments of the present application, and the protection scope of the present application is not limited thereby. Therefore, any equivalent changes made according to the methods and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for aggregating camera tracking data and lens data in virtual shooting, characterized in that: The method is executed in a server and includes: S1: acquiring posture data collected by a camera tracking system in real time, the posture data including three-axis translation data and three-axis rotation data, wherein the camera tracking system is installed in a constructed virtual shooting studio; S2: acquiring lens data collected by a camera focus system in real time, the lens data including focus, focal length and aperture data of the camera lens, wherein the camera focus system is installed on a physical camera lens in a constructed virtual shooting studio; S3: Solving, processing and aggregating the camera pose data obtained in step S1 and the camera lens data obtained in step S2 to obtain a real-time metadata package suitable for virtual shooting; S4: Send the real-time metadata packet to the terminal equipped with the virtual production engine via the UDP communication protocol to control the virtual camera in the virtual production engine.

2. The method for aggregating camera tracking data and lens data in virtual shooting according to claim 1, characterized in that: The models of camera tracking systems include HTC Mars Camtrack, Vicon Vero, Mo-SysStarTracker Max, Zeiss CineCraft, Stype RedSpy, Optitack Prime 41 and Qingtong MC4000.

3. The method for aggregating camera tracking data and lens data in virtual shooting according to claim 1, characterized in that: The models of the camera follow focus systems include Arri UMC-4 and Movcam XRport.

4. The method for aggregating camera tracking data and lens data in virtual shooting according to claim 1, characterized in that: Step S3 specifically includes: S31, through the camera posture transmission data protocol, the posture data P transmitted by the camera tracking system is calculated, P = {θ Pitch ,θ yaw ,θ roll ,t z ,t y ,t x }, where -180°≤θ Pitch ,θ yaw ,θ roll <180°, respectively represents the rotation value of the camera around x, y, and z, t z ,t y ,t x Represents the values ​​of the camera translation along z, y, and x respectively; S32, through the camera lens transmission data protocol, the lens data L transmitted by the camera focus system is obtained by solving, L = {μ zoom ,μ focus ,μ iris },μ zoom ,μ focus ,μ iris Respectively represent the focal length, focus and aperture value of the lens; S33, solving and processing the frame rates of the tracking system and the focus tracking system, and calculating the output frame rate f of the metadata packet; S34, reorganizing and aggregating the obtained posture data P and lens data L based on the output frame rate f and the preset output strategy to obtain the real-time metadata package data.

5. The method for aggregating camera tracking data and lens data in virtual shooting according to claim 4, characterized in that: The camera posture transmission data protocols include FreeD protocol, Virtual Reality Peripheral Network (VRPN) protocol, Vicon Datastream protocol and Ncam protocol.

6. The method for aggregating camera tracking data and lens data in virtual shooting according to claim 4, characterized in that: The camera lens transmission data protocols include FreeD protocol, Ncam protocol, Arri metadata protocol and Movcam metadata protocol.

7. The method for aggregating camera tracking data and lens data in virtual shooting according to claim 4, characterized in that: Step S33 specifically includes: S331, using formula (1) to calculate the frame rate of the tracking system: f p =1 / N p (1) Among them, N p Indicates the number of times the tracking system sends tracking data in one second; S332, using formula (2) to calculate the frame rate of the focus tracking system: f l =1 / N l (2) Among them, N l Indicates the number of times the focus system sends lens data packets within one second; S333, using formula (3) to calculate the output frame rate of the metadata packet:

8. The method for aggregating camera tracking data and lens data in virtual shooting according to claim 4, characterized in that: The preset output strategy is represented by a preset strategy identifier, which has 2 bytes in total, where 01 represents direct tracking data, 02 represents direct lens data, and 03 represents reorganized tracking data and lens data.

9. The method for aggregating camera tracking data and lens data in virtual shooting according to claim 4, characterized in that: Step S34 specifically includes: S341, when the preset strategy identifier is 01, output data O=P, and the output frame rate is the tracking system frame rate f p ; S342, when the preset strategy identifier is 02, the output data O=L, the output frame rate is the tracking system frame rate f l ; S343, when the preset strategy identifier is 03, output data The output frame rate is the output frame rate f calculated in S333, where It means to normalize and combine two sets of data.

10. The method for aggregating camera tracking data and lens data in virtual shooting according to claim 1, characterized in that: The real-time metadata data packet is encoded by using the FreeD data protocol through the UDP communication protocol and then sent to a terminal equipped with a virtual production engine.

11. A device for aggregating camera tracking data and lens data in virtual shooting, characterized in that: Applied to a server, the device comprises: A data acquisition module, used to acquire in real time the posture data collected by the camera tracking system and the lens data collected by the camera focus system, wherein the camera tracking system is installed in the constructed virtual shooting studio, and the camera focus system is installed on the physical camera lens in the constructed virtual shooting studio; A data solving module is used to solve the acquired posture data, lens data and output frame rate, wherein the posture data includes three-axis translation data and three-axis rotation data, and the lens data includes the focus, focal length and aperture data of the camera lens; A data aggregation module is used to calculate and aggregate the output metadata package data according to the calculated posture data, lens data, output frame rate and preset strategy; The real-time data sending module is used to send the real-time metadata package to a terminal equipped with a virtual production engine to control a virtual camera in the virtual production engine.

Images