Intelligent recombination method and device for multi-path camera tracking metadata in virtual shooting

Abstract

The invention discloses an intelligent recombination method and device for multi-path camera tracking metadata in virtual shooting, and belongs to the field of virtual shooting, and the method specifically comprises the steps: obtaining pose metadata packets collected by at least two paths of camera tracking systems in real time; resolving the frame rate, the equipment feature code and the pose data of the multi-path camera in real time; judging the stability of the pose data of the multi-path camera; judging the quantity and coupling of the stably transmitted multi-path pose data of the camera tracking system, calculating the finally output pose data in real time according to a preset strategy and a coupling threshold, and packaging the finally output pose data into a real-time metadata packet; and sending the real-time metadata packet to a terminal carrying a virtual production engine in a UDP (User Datagram Protocol) communication protocol so as to control a virtual camera in the virtual production engine. The problems that a current single camera tracking system is unstable and data transmission is prone to packet loss are solved, and reliable guarantee is provided for creation stability of a virtual shooting site.

Description

Technical Field

[0001] The present application relates to the technical field of film and television virtual shooting, and in particular to a method and device for intelligently reorganizing metadata of multi-channel camera tracking in virtual shooting. Background Art

[0002] Virtual shooting is an important method for filming movies and TV shows, but it has many problems. On the set, in order to improve the tracking accuracy and the anti-interference ability of the tracking system, multiple tracking systems with different principles are often deployed to track the same camera. Once the tracking system is switched, the rendering needs to be ended and the editing mode needs to be returned to for modification, which will affect the creative efficiency of the director and the main creator. In addition, the multi-channel tracking system cannot automatically switch in real time and intelligently reorganize, which greatly limits the on-site creation and shooting creativity.

[0003] How to receive the posture data transmitted by the multi-channel camera tracking system and filter and reorganize the multi-channel data has become an urgent problem to be solved. Summary of the invention

[0004] In order to solve at least one of the above technical problems, the present application provides a method and device for intelligently reorganizing metadata of multi-channel camera tracking in virtual shooting.

[0005] In the first aspect, the present application provides a method for intelligently reorganizing metadata of multi-channel camera tracking in virtual shooting, which adopts the following technical solution:

[0006] A method for intelligently reorganizing metadata of multi-channel camera tracking in virtual shooting, the method being executed in a server, comprising:

[0007] S1: acquiring, in real time, posture metadata packets collected by at least two camera tracking systems, wherein the camera tracking systems are installed in a constructed virtual shooting studio or on a physical camera in the studio;

[0008] S2: Calculate the frame rate, device signature and posture data of multiple cameras in real time through the camera posture metadata packet transmission data protocol, wherein the posture data includes three-axis translation data and three-axis rotation data;

[0009] S3: determining the stability of the posture data of the multiple cameras, eliminating the posture data of the camera tracking system that cannot be stably transmitted, thereby selecting the posture data of the camera tracking system that can be stably transmitted;

[0010] S4: determining the quantity and coupling of the multi-channel pose data of the stably transmitted camera tracking system, and calculating the pose data outputted in real time according to a preset strategy and coupling threshold, and encapsulating the pose data into a real-time metadata packet;

[0011] S5: Send the real-time metadata packet to a terminal equipped with a virtual production engine using a UDP communication protocol to control a virtual camera in the virtual production engine.

[0012] Optionally, models of the camera tracking system include HTC Mars Camtrack, Vicon Vero, Mo-Sys StarTracker Max, Zeiss CineCraft, Stype RedSpy, Optitack Prime41 and Qingtong MC4000.

[0013] Optionally, the camera posture transmission data protocol includes a FreeD protocol, a Virtual Reality Peripheral Network (VRPN) protocol, a Vicon Datastream protocol and an Ncam protocol.

[0014] Optionally, the stability of the multi-channel pose data is determined by setting a timeout threshold σ (σ<3s), and the pose data receiving time δ of each camera tracking system is counted. If δ>σ, the pose data of the channel is determined to be unstable and is eliminated, thereby screening out the pose data of the remaining camera tracking system with stable transmission, wherein the value range of the timeout threshold is σ<3s, and δ is calculated using formula (1):

[0015] δ i =∈ i2 -∈ i1 (2)

[0016] Among them, ∈ i2 Indicates the time when the current frame of the i-th pose data is received, ∈ i1 Indicates the time before the i-th pose data is received.

[0017] Optionally, step S4 specifically includes:

[0018] S41, judging the number of the screened multi-channel pose data, if the number of camera tracking systems with stable transmission remaining after screening is greater than 1, proceeding to step S42, if the number of camera tracking systems with stable transmission remaining after screening is equal to 1, proceeding to step S43, if the number of camera tracking systems with stable transmission remaining after screening is less than 1, proceeding to step S44;

[0019] S42, according to the coupling tightness between the camera tracking systems that remain to be stably transmitted after screening, calculate, reorganize and encode based on the posture data provided by the remaining tracking systems to obtain an output real-time metadata package;

[0020] S43, encoding the posture data provided by the camera tracking system that is still stably transmitted after screening to obtain an output real-time metadata packet;

[0021] S44, using the data packet data outputted in the previous frame as the output metadata data packet data.

[0022] Further optionally, step S42 specifically includes:

[0023] S421, determine the coupling between the camera tracking systems with stable transmission remaining after screening, set the coupling threshold Δ (0<Δ<1) in advance, and calculate the coupling value ξ between the multiple camera tracking systems i，j , and compare it with the coupling threshold Δ, and select the coupling value ξ i，j The two-way camera tracking system data with the smallest value that is less than the coupling threshold Δ executes step S422. If all coupling values ​​are greater than the coupling threshold Δ, then step S423 is executed. i，j Use formula (2) to formula (8) to calculate:

[0024]

[0025]

[0026] in, They represent the rotation values ​​of the camera around x, y, and z obtained by the i-th camera tracking system, respectively. Respectively represent the values ​​of the camera translation along z, y, and x obtained by the i-th camera tracking system;

[0027] S422, for the coupling value ξ selected in step S421 i，j The three-axis translation data and the three-axis rotation data in the posture data transmitted by the two-way camera tracking system that are the smallest and less than the coupling threshold Δ are averaged respectively, and the six average values ​​are used as the final output real-time metadata package;

[0028] S423, obtain the posture data P′={θ′ Pitch ,θ′ Roll ,θ′ Yaw , t′ z , t′ y , t′ x}, respectively using the acquired multi-channel camera tracking system pose data Calculate the coupling with P′ and obtain the coupling value ξ k And compare, select a tracking system that is closest to the output pose data of the previous frame, and use the pose data of the camera tracking system to calculate and reorganize to obtain the final output real-time metadata package.

[0029] Further optionally, step S423 specifically includes:

[0030] Step S4231, obtaining the pose data of the camera tracking system k closest to the output pose data of the previous frame selected in step S423 And get the pose data of the previous frame of system k

[0031] Step S4232, combining the posture data P′={θ′ Pitch ,θ′ Roll ,θ′ Yaw , t′ z , t′ y , t′ x}, and use formula (9) to formula (13) to calculate the output posture data P = {θ Pitch ,θ Roll ,θ Yaw , t z , t y , t x}:

[0032]

[0033] Optionally, the real-time metadata data packet is encoded using the FreeD data protocol through the UDP communication protocol and then sent to a terminal equipped with a virtual production engine.

[0034] In a second aspect, the present application provides a device for intelligently reorganizing metadata for tracking multiple cameras in virtual shooting, which adopts the following technical solution:

[0035] A device for intelligently reorganizing metadata of multiple camera tracking in virtual shooting, applied to a server, comprising:

[0036] A data acquisition module, used to acquire in real time the posture data collected by at least two camera tracking systems, wherein the camera tracking systems are installed in a constructed virtual shooting studio or on a physical camera in the studio;

[0037] A data calculation module is used to calculate the frame rate, device feature code and posture data of multiple cameras, wherein the posture data includes three-axis translation data and three-axis rotation data;

[0038] The data screening module is used to eliminate camera tracking systems that cannot stably transmit posture data and select systems that can stably transmit;

[0039] The data reorganization module is used to calculate and reorganize the output metadata package data according to the screened tracking system data and the preset strategy and coupling threshold;

[0040] The real-time data sending module is used to send the real-time metadata package to a terminal equipped with a virtual production engine to control a virtual camera in the virtual production engine.

[0041] The beneficial technical effects of the present invention are:

[0042] The present application provides a method and device for intelligently reorganizing metadata of multi-channel camera tracking in virtual shooting. After acquiring the camera posture data collected by the multi-channel camera tracking system, the camera tracking system data that can be stably transmitted is screened out, and the corresponding data is calculated and reorganized to obtain a real-time metadata package suitable for virtual shooting. This solves the problems of instability of the current single camera tracking system and easy packet loss in data transmission, and provides a reliable guarantee for improving the stability and efficiency of the virtual shooting scene and increasing the creative flexibility of the shooting scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of a method for intelligently reorganizing metadata of multi-channel camera tracking in virtual shooting of the present invention;

[0044] Figure 2 It is a block diagram of the module architecture of a device for intelligently reorganizing metadata for tracking multiple cameras in virtual shooting provided by one embodiment of the present invention; DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0047] Reference Figure 1 The embodiment of the present invention discloses a method for intelligently reorganizing metadata of multi-channel camera tracking in virtual shooting. The method is executed in a server and includes the following steps:

[0048] S1: acquiring in real time posture metadata packets collected by at least two camera tracking systems, where the camera tracking systems are installed in a constructed virtual shooting studio or on a physical camera in the studio.

[0049] Build a virtual filming studio, set up the camera, install the camera tracking system in the studio, and add a virtual camera object to the virtual filming scene. The available tracking system models include but are not limited to HTC Mars Camtrack, Vicon Vero, Mo-Sys StarTracker Max, ZeissCineCraft, Stype RedSpy, Optitack Prime41 and Qingtong MC4000.

[0050] S2: Calculate the frame rate, device signature and posture data of multiple cameras in real time through the camera posture metadata packet transmission protocol, wherein the posture data includes three-axis translation data and three-axis rotation data.

[0051] In this embodiment, the camera posture transmission data protocol includes FreeD protocol, Virtual Reality Peripheral Network (VRPN) protocol, Vicon Datastream protocol and Ncam protocol.

[0052] S3: Determine the stability of the pose data of the multiple cameras, and eliminate the pose data of the camera tracking system that cannot be stably transmitted, so as to select the pose data of the camera tracking system that can be stably transmitted; when determining the stability of the multiple pose data, set a timeout threshold σ, and perform statistics on the pose data receiving time δ of each camera tracking system. If δ>σ, the pose data of this channel is determined to be unstable and eliminated, wherein the value range of the timeout threshold is σ<3s. δ is calculated using formula (1):

[0053] δ i =∈ i2 -∈ i1 (3)

[0054] Among them, ∈ i2 Indicates the time when the current frame of the i-th data channel is received, ∈ i1 Indicates the time before the i-th data frame is received.

[0055] S4: Determine the quantity and coupling of the multi-channel pose data of the stably transmitted camera tracking system, calculate the final output pose data in real time according to the preset strategy and coupling threshold, and encapsulate it into a real-time metadata package:

[0056] Step S4 specifically includes:

[0057] S41, judge the number of multi-channel posture data screened out. If the number of camera tracking systems with stable transmission remaining after screening is greater than 1, proceed to step S42. If the number of camera tracking systems with stable transmission remaining after screening is equal to 1, proceed to step S43. If the number of camera tracking systems with stable transmission remaining after screening is less than 1, proceed to step S44.

[0058] S42, according to the degree of coupling between the tracking systems with the remaining stable transmission after screening, the posture data provided by the camera tracking systems with the remaining stable transmission are calculated, reorganized and encoded to obtain output real-time metadata package data.

[0059] Step S42 specifically includes:

[0060] S421, determine the coupling between the camera tracking systems with stable transmission remaining after screening, set the coupling threshold Δ (0<Δ<1) in advance, and calculate the coupling value ξ between the multi-channel systems i，j , and compare it with the coupling threshold Δ, and select the coupling value ξ i，j If the two-way system data with the smallest value and less than the coupling threshold Δ execute step S422, if all coupling values ​​are greater than the coupling threshold Δ, execute step S423, ξ i，j Use formula (2) to formula (8) to calculate:

[0061]

[0062] in, They represent the rotation values ​​of the camera around x, y, and z obtained by the i-th camera tracking system, respectively. They represent the values ​​of the camera translation along z, y, and x obtained by the i-th camera tracking system.

[0063] S422, for the coupling value ξ selected in step S421 i，j The three-axis translation data and the three-axis rotation data in the pose data transmitted by the two-way camera tracking system that are the smallest and less than the coupling threshold Δ are averaged respectively, and the six average values ​​are used as the real-time metadata package data finally output.

[0064] S423, obtain the posture data P′={θ′ Pitch ,θ′ Roll ,θ′ Yaw , t′ z , t′ y , t′ x}, respectively using the acquired multi-channel camera tracking system pose data Calculate the coupling with P′ and obtain the coupling value ξ kAnd compare, select a camera tracking system that is closest to the output pose data of the previous frame, and use the pose data of the camera tracking system to calculate and reorganize to obtain the final output real-time metadata package data.

[0065] Step S423 specifically includes:

[0066] Step S4231, obtaining the pose data of the camera tracking system k closest to the output pose data of the previous frame selected in step S423 And get the pose data of the previous frame of system k

[0067] Step S4232, combining the posture data P′={θ′ Pitch ,θ′ Roll ,θ′ Yaw , t′ z , t′ y , t′ x}, and use formula (9) to formula (13) to calculate the output posture data P = {θ Pitch ,θ Roll ,θ Yaw , t z , t y , t x}:

[0068]

[0069]

[0070] S43, encoding the posture data provided by the camera tracking system that is still stably transmitted after screening to obtain output real-time metadata package data.

[0071] S44, using the data packet data outputted in the previous frame as the outputted real-time metadata data packet data.

[0072] S5: Send the real-time metadata packet to the terminal equipped with the virtual production engine via the UDP communication protocol to control the virtual camera in the virtual production engine.

[0073] In this embodiment, the real-time metadata packet for virtual shooting is encoded by FreeD data protocol through UDP communication protocol and sent to the terminal equipped with the virtual production engine to control the virtual camera in the virtual production engine.

[0074] In one embodiment, Figure 2 As shown, a device for intelligently reorganizing metadata of multiple camera tracking in virtual shooting is provided, comprising the following program modules:

[0075] The data acquisition module 201 is used to acquire the posture data collected by the multi-channel camera tracking system in real time, and the camera tracking system is installed in a virtual shooting studio or on a physical camera in the studio;

[0076] The data calculation module 202 is used to calculate the frame rate, device feature code and posture data of the acquired multi-channel cameras, wherein the posture data includes three-axis translation data and three-axis rotation data;

[0077] A data screening module 203 is used to eliminate camera tracking systems that cannot stably transmit posture data, and screen out camera tracking systems that can stably transmit;

[0078] The data reorganization module 204 is used to calculate and reorganize the data of the camera tracking system with stable transmission according to the preset strategy and coupling threshold to obtain an output real-time metadata package;

[0079] The real-time data sending module 205 is used to send the real-time metadata package to a terminal equipped with a virtual production engine to control a virtual camera in the virtual production engine.

[0080] The above are preferred embodiments of the present application, and the protection scope of the present application is not limited thereby. Therefore, any equivalent changes made according to the methods and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for intelligently reorganizing metadata of multi-channel camera tracking in virtual shooting, characterized in that: The method is executed in a server and includes: S1: acquiring, in real time, posture metadata packets collected by at least two camera tracking systems, wherein the camera tracking systems are installed in a constructed virtual shooting studio or on a physical camera in the studio; S2: Calculate the frame rate, device signature and posture data of multiple cameras in real time through the camera posture metadata packet transmission data protocol, wherein the posture data includes three-axis translation data and three-axis rotation data; S3: determining the stability of the posture data of the multiple cameras, eliminating the posture data of the camera tracking system that cannot be stably transmitted, thereby selecting the posture data of the camera tracking system that can be stably transmitted; S4: determining the quantity and coupling of the multi-channel pose data of the stably transmitted camera tracking system that has been screened out, and calculating the final output pose data in real time according to a preset strategy and coupling threshold, and encapsulating the final output pose data into a real-time metadata package; S5: Send the real-time metadata packet to a terminal equipped with a virtual production engine using a UDP communication protocol to control a virtual camera in the virtual production engine.

2. The method for intelligently reorganizing metadata of multiple camera tracking in virtual shooting according to claim 1, characterized in that: The models of camera tracking systems include HTC Mars Camtrack, Vicon Vero, Mo-SysStarTracker Max, Zeiss CineCraft, Stype RedSpy, Optitack Prime 41 and Qingtong MC4000.

3. The method for intelligently reorganizing metadata of multi-channel camera tracking in virtual shooting according to claim 1, characterized in that: The camera posture transmission data protocols include FreeD protocol, Virtual Reality Peripheral Network (VRPN) protocol, Vicon Datastream protocol and Ncam protocol.

4. The method for intelligently reorganizing metadata of multiple camera tracking in virtual shooting according to claim 1, characterized in that: The stability of the multi-channel pose data is determined by setting a timeout threshold σ, and the pose data receiving time δ of each camera tracking system is counted. If δ>σ, the pose data of the channel is determined to be unstable and is eliminated, thereby screening out the pose data of the remaining camera tracking system with stable transmission, wherein the value range of the timeout threshold is σ<3s, and δ is calculated using formula (1): d i =∈ i2 -∈ i1 (1) Among them, ∈ i2 Indicates the time when the current frame of the i-th pose data is received, ∈ i1 Indicates the time before the i-th pose data is received.

5. The method for intelligently reorganizing metadata of multiple camera tracking in virtual shooting according to claim 1, characterized in that: Step S4 specifically includes: S41, judging the number of the screened multi-channel pose data, if the number of camera tracking systems with stable transmission remaining after screening is greater than 1, proceeding to step S42, if the number of camera tracking systems with stable transmission remaining after screening is equal to 1, proceeding to step S43, if the number of camera tracking systems with stable transmission remaining after screening is less than 1, proceeding to step S44; S42, according to the degree of coupling between the camera tracking systems with stable transmission remaining after screening, calculate, reorganize and encode the posture data provided by the camera tracking systems with stable transmission remaining, and obtain an output real-time metadata packet; S43, encoding the posture data provided by the camera tracking system that is still stably transmitted after screening to obtain an output real-time metadata packet; S44, using the data packet data outputted in the previous frame as the outputted real-time metadata data packet.

6. The method for intelligently reorganizing metadata of multiple camera tracking in virtual shooting according to claim 5, characterized in that: Step S42 specifically includes: S421, determine the coupling between the camera tracking systems with stable transmission remaining after screening, set the coupling threshold Δ in advance, and calculate the coupling value ξ between the multiple camera tracking systems i,j , and compare it with the coupling threshold Δ, and select the coupling value ξ i,j The two-way camera tracking system data with the smallest value that is less than the coupling threshold Δ executes step S422. If all coupling values ​​are greater than the coupling threshold Δ, then step S423 is executed. i,j Use formula (2) to formula (8) to calculate: Among them, the coupling threshold range is 0<Δ<1, They represent the rotation values ​​of the camera around x, y, and z obtained by the i-th camera tracking system, respectively. Respectively represent the values ​​of the camera translation along z, y, and x obtained by the i-th camera tracking system; S422, for the coupling value ξ selected in step S421 i,j The three-axis translation data and the three-axis rotation data in the pose data transmitted by the two-way camera tracking system with the smallest coupling Δ and less than the threshold value are averaged respectively, and the six average values ​​are used as the final output real-time metadata package; S423, obtain the posture data P′={θ′ Pitch ,θ′ Yoll ,θ′ Yaw ,t′ z ,t′ y ,t′ x }, respectively using the acquired multi-channel camera tracking system pose data Calculate the coupling with P′ and obtain the coupling value ξ k And compare, select a tracking system that is closest to the output pose data of the previous frame, and use the pose data of the camera tracking system to calculate and reorganize to obtain the final output real-time metadata package.

7. The method for intelligently reorganizing metadata of multiple camera tracking in virtual shooting according to claim 6, characterized in that: Step S423 specifically includes: Step S4231, obtaining the pose data of the camera tracking system k closest to the output pose data of the previous frame selected in step S423 And get the pose data of the previous frame of system k Step S4232, combining the posture data P′={θ′ Pitch ,θ′ Roll ,θ′ Yaw ,t′ z ,t′ y ,t′ x }, and use formula (9) to formula (13) to calculate the output posture data P = {θ Pitch ,θ Roll ,θ Yaw ,t z ,t y ,t x }:

8. The method for intelligently reorganizing metadata of multiple camera tracking in virtual shooting according to claim 1, characterized in that: The real-time metadata data packet is encoded by using the FreeD data protocol through the UDP communication protocol and then sent to a terminal equipped with a virtual production engine.

9. A device for intelligently reorganizing metadata of multiple camera tracking in virtual shooting, characterized in that: Applied to a server, the device comprises: A data acquisition module, used to acquire in real time the posture data collected by at least two camera tracking systems, wherein the camera tracking systems are installed in a constructed virtual shooting studio or on a physical camera in the studio; A data calculation module is used to calculate the frame rate, device feature code and posture data of multiple cameras, wherein the posture data includes three-axis translation data and three-axis rotation data; The data screening module is used to eliminate the camera tracking systems that cannot stably transmit the pose data and screen out the camera tracking systems that can stably transmit. The data reorganization module is used to calculate and reorganize the output real-time metadata package based on the filtered tracking system data according to the preset strategy and coupling threshold; The real-time data sending module is used to send the real-time metadata package to a terminal equipped with a virtual production engine to control a virtual camera in the virtual production engine.

Images