Method for calculating motion trail data of real prop from motion process of virtual camera

Abstract

The invention discloses a method for calculating real prop movement track data from a virtual camera movement process, and belongs to the field of virtuality and reality combined stage performance. The method comprises the following steps: step 1, performing virtual scene layout, and determining a film object and a virtual item; 2, designing a camera motion track and time length, and setting camera parameters of a virtual camera; 3, designing a simulation scene, a simulation prop and a simulation screen; 4, simulating a virtual screen picture, and playing a shot picture on a simulation screen; 5, setting a corresponding relation between the virtual scene and the simulation scene; step 6, calculating virtual item coordinates; 7, calculating the spatial position of the virtual item; 8, calculating the space angle of the virtual prop; step 9, acquiring the coordinate and angle of each frame of virtual item on the simulation screen; and step 10, mapping real orbit data. According to the method, the real prop moving trajectory data can be reversely calculated according to the virtual camera, the subsequent field debugging time of the program is shortened, and the programming efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of virtual-real combined stage performance, and particularly relates to a method for calculating real prop motion trajectory data from virtual camera motion process. BACKGROUND

[0002] For a virtual-real combined program of stage performance, real performance needs to be fused with video according to virtual shooting video, and the position of real props is not only related to the lens animation of virtual shooting video, but also related to the motion of virtual props in the video, in addition to the screen shape and the vehicle track position. It is difficult to ensure that the position and angle of real props are consistent with virtual shooting video in reality, and a basic satisfactory effect can be achieved only by continuous testing, but there are problems such as long debugging time and the effect obtained is not necessarily the best.

[0003] Therefore, how to provide a method for calculating real prop motion trajectory data according to virtual props in virtual shooting video, so as to better fuse real performance with video, shorten program debugging time and improve program arrangement efficiency is a problem to be solved.

[0004] Therefore, the present application is proposed. SUMMARY

[0005] The present application provides a method for calculating real prop motion trajectory data from virtual camera motion process, which can calculate real prop motion trajectory data according to virtual props in virtual shooting video, so as to better fuse real performance with video, shorten program debugging time and improve program arrangement efficiency, thereby solving the above technical problems in the prior art.

[0006] The present application is achieved by the following technical scheme: A method for calculating real prop motion trajectory data from virtual camera motion process, comprising: Step 1: designing a virtual scene in a three-dimensional development engine according to the content of a script, and labeling virtual props in the virtual scene, wherein the virtual props are props that do not need to be rendered into a video and are performed by real props on a real stage in front of a real screen subsequently; Step 2: setting the motion trajectory and time length of a virtual camera for shooting a video in the three-dimensional development engine, and setting camera parameters of the virtual camera according to the real screen; Step 3: creating a simulation screen corresponding to the real screen and simulation props corresponding to real props performing on the real stage in front of the real screen in a simulation scene through the three-dimensional development engine according to the real stage and the real screen of actual performance; Step 4, in the three-dimensional development engine, the video taken by the virtual camera is obtained through the camera parameters of the virtual camera, and the video is played on the corresponding simulation screen; Step 5, the correspondence between the virtual scene and the simulation scene is set; Step 6, the coordinate value of the virtual prop in the video picture taken by the virtual camera is calculated through the camera parameters; Step 7, the position of the virtual prop picture displayed on the simulation screen space when the simulation screen displays the video picture of the virtual camera is calculated through the size of the simulation screen, the vertex position of the simulation screen, the coordinate value of each vertex and the coordinate value calculated in step 6; Step 8, the angle of the virtual prop picture displayed on the simulation screen space when the simulation screen displays the virtual prop is calculated through the position of the virtual camera and the virtual prop, the vertex position of the simulation screen and the normal information of each vertex and the coordinate value calculated in step 6; Step 9, the virtual camera motion trajectory set in step 2 is played, and when the virtual camera takes the virtual prop, the coordinate and angle of each frame of the virtual prop picture on the simulation screen are continuously obtained according to the calculation method of step 7 and step 8, so as to obtain the continuous space coordinates of the virtual prop picture on the simulation screen when the virtual camera takes the virtual prop; Step 10, according to the real track data, the continuous space coordinates of each frame of the virtual prop picture on the simulation screen are mapped to the real track data to form the motion trajectory data of the real prop on the real stage.

[0007] Compared with the prior art, the method for calculating the motion trajectory data of the real prop from the virtual camera motion process has the following beneficial effects:

[0008] Through the camera animation when the virtual scene is taken, the screen shape and the real track information are fused, the running trajectory data of the real prop on the real stage is reversely calculated, the time of subsequent live debugging of the program is greatly reduced, and the program arrangement efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0010] Figure 1 The flow chart of the method for calculating the motion trajectory data of the real prop from the virtual camera motion process provided by the embodiments of the present application.

[0011] Figure 2The specific flowchart of the method for calculating real prop motion trajectory data from virtual camera motion process provided by the embodiment of the present application is shown in the figure.

[0012] Figure 3 The schematic diagram of the relationship between three scene elements in the method provided by the embodiment of the present application is shown in the figure.

[0013] Figure 4 The flowchart of steps 6-10 in the method provided by the embodiment of the present application is shown in the figure.

[0014] Figure 5 The virtual scene schematic diagram in the method provided by the embodiment of the present application is shown in the figure.

[0015] Figure 6 The simulation scene schematic diagram in the method provided by the embodiment of the present application is shown in the figure.

[0016] Figure 7 The real scene schematic diagram in the method provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application are clearly and completely described below in combination with the specific contents of the present application; obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments, which do not constitute a limitation to the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative efforts fall within the protection scope of the present application.

[0018] Firstly, the terms possibly used in the present text are explained as follows: The term “and / or” means either of the two or both at the same time, for example, X and / or Y means three cases including “X” or “Y” or “X and Y”.

[0019] The terms “include”, “contain”, “have”, “possess” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example: including a technical feature element (such as raw materials, components, ingredients, carriers, dosage forms, materials, sizes, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or articles, etc.) should be interpreted as not only including the explicitly listed technical feature element, but also including other technical feature elements not explicitly listed in the art.

[0020] The term "consisting of' shall be used in the claims to mean excluding any element not specified in the claim. If the term "consisting of' is used in any claim that issue from this specification including the claims of this application filed on even date, then the term shall be interpreted in accordance with the interpretation given to the term "consisting of' under 35 U.S.C. § 112, paragraph 6. If this term appears in any claim that issues from this specification including the claims of this application filed on even date, then it shall be interpreted in accordance with the interpretation given to the term "consisting of' under 35 U.S.C. § 112, paragraph 6.

[0021] Unless otherwise defined, the terms "mounting", "connected", "connecting", "fixed", and the like are to be construed in a broad sense, for example: can be fixed connection, can also be detachable connection, or integral connection; can be mechanical connection, can also be electrical connection; can be directly connected, can also be indirectly connected through an intermediate medium, can be internal communication of two elements. The specific meaning of the above terms in this paper can be understood according to the specific situation by those skilled in the art.

[0022] The terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, which is only for the convenience of description and simplification of description, and does not mean that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation herein.

[0023] The schemes provided by the present application will be described in detail below. The contents not described in detail in the embodiments of the present application belong to the prior art known to those skilled in the art. If the specific conditions are not specified in the embodiments of the present application, the conventional conditions or the conditions recommended by the manufacturer are used. If the manufacturer of the reagent or instrument used in the embodiments of the present application is not specified, it is a conventional product that can be purchased on the market.

[0024] As shown in Figure 1 , Figure 2 and Figure 4 The embodiments of the present application provide a method for calculating real prop motion trajectory data from virtual camera motion process, comprising: Step 1: design a virtual scene in a three-dimensional development engine according to the content of the script, and label virtual props in the virtual scene. The virtual prop is a prop that does not need to be rendered into a video and will be performed by a real prop in front of a real stage later. Preferably, the three-dimensional development engine used is UnrealEngine; Step 2, set the motion track and time length of the virtual camera shooting the video in the three-dimensional development engine, and set the camera parameters of the virtual camera according to the real screen; since the screen is arc-shaped, two virtual cameras are currently used for shooting, and the motion track and time length of the two virtual cameras need to be set at the same time; Step 3, according to the real stage and real screen of the actual performance, create a simulation screen corresponding to the real screen and a simulation prop corresponding to the real prop performing in front of the real screen on the real stage in the simulation scene through the three-dimensional development engine; due to the arc-shaped screen, the simulation screen model also needs to be divided into two parts, corresponding to the two camera pictures shot respectively; Step 4, in the three-dimensional development engine, obtain the video shot by the virtual camera through the camera parameters of the virtual camera, and play the video on the corresponding simulation screen; Step 5, set the correspondence between the virtual scene and the simulation scene; Step 6, calculate the coordinate value of the virtual prop in the virtual camera shooting video picture through the camera parameters; Step 7, calculate the position of the virtual prop picture displayed on the simulation screen space when the simulation screen displays the video picture shot by the virtual camera through the size of the simulation screen, the vertex position of the simulation screen, the coordinate value of each vertex and the coordinate value calculated in step 6; Step 8, calculate the angle of the virtual prop picture displayed on the simulation screen space when the simulation screen displays the virtual prop through the position of the virtual camera and the virtual prop, the vertex position of the simulation screen and the normal information of each vertex, and the coordinate value calculated in step 6; Step 9, play the virtual camera motion track set in step 2, when the virtual camera shoots the virtual prop, continuously obtain the coordinate and angle of each frame of virtual prop picture on the simulation screen according to the calculation method of step 7 and step 8, to obtain the continuous space coordinates of the virtual prop picture on the simulation screen when the virtual camera shoots the virtual prop; Step 10, according to the real track data, map the continuous space coordinates of each frame of virtual prop picture on the simulation screen to the real track data to form the motion track data of the real prop on the real stage.

[0025] Reference Figure 3 , Figure 5-Figure 7 The technical names involved in the above method are explained as follows: Virtual scene 10: a virtual three-dimensional scene created for shooting the film to be played in the program (see Figure 5 ).

[0026] Virtual camera 30: a camera for shooting in the virtual scene, which can capture the virtual scene picture shot according to the camera parameters, and can output video, usually with three-dimensional software.

[0027] Virtual prop 20: a prop in the virtual scene, which needs to be interacted with the actor in front of the real screen later, used to assist the virtual camera positioning.

[0028] Simulation scene: a virtual scene simulating the real stage, which is different from the previous virtual scene in that this scene is only used to simulate the scene for the audience to watch, not for the film to be shot (see Figure 6 ).

[0029] Simulation screen 40: placed in the simulation scene, simulating the real screen and the content played.

[0030] Simulation prop 60: placed in the simulation scene, simulating the state of the real prop.

[0031] Simulation track 50: placed in the simulation scene, recording the position of the real track.

[0032] The parts of the real scene are as follows, see Figure 7 , including: Real stage: a performance venue and audience viewing area constructed on site according to CAD drawings.

[0033] Real screen 80: a huge arc-shaped screen composed of multiple LED screens.

[0034] Real prop 70: a movable prop designed from a vehicle chassis and decorations, which can interact with the actor and can input time-driven motion trajectory data. In front of the real prop 70 is the audience A.

[0035] Real track 90: positioning information for controlling the running track of the vehicle on the ground according to the CAD drawings.

[0036] Preferably, in step 4 of the above method, in the three-dimensional development engine, the video taken by the virtual camera is obtained through the camera parameters of the virtual camera and played on the corresponding simulation screen in the following manner, including: Step 41: setting the virtual camera parameters in the three-dimensional development engine; Step 42: creating a render target and a render target material; Step 43: setting the display content of the simulation screen to the created render target material, realizing the video playing on the corresponding simulation screen.

[0037] Preferably, in step 5 of the above method, the correspondence between the virtual scene and the simulation scene is set in the following manner, including: Step 51: setting the two virtual cameras to correspond to the simulation screens respectively; Step 52: setting the virtual prop to correspond to the simulation prop.

[0038] Preferably, in step 6 of the above method, the coordinate value of the virtual prop in the video picture taken by the virtual camera is calculated by the camera parameters in the following way, comprising: Step 61, obtaining the viewport matrix of the virtual camera through the camera parameters; Step 62, obtaining the projection matrix of the virtual camera through the camera parameters; Step 63, combining the viewport matrix obtained in step 61 and the projection matrix obtained in step 62 to calculate the uv coordinate of the virtual prop on the projection plane of the virtual camera.

[0039] Preferably, in step 7 of the above method, the position of the virtual prop picture displayed on the simulation screen when the simulation screen displays the video picture of the virtual camera is calculated by the size of the simulation screen, the vertex position of the simulation screen, the coordinate value of each vertex and the coordinate value calculated in step 6 in the following way, comprising: Step 71, obtaining the mesh model of the simulation screen; Step 72, obtaining the triangle list of the mesh model; Step 73, traversing the triangle list to obtain the uv attribute of each point of the triangle; Step 74, using the barycentric coordinate method to judge whether the uv coordinate of the virtual prop is within the current triangle; Step 75, if the uv coordinate of the virtual prop is within the triangle, calculating the distance between the uv coordinate of the virtual prop and each vertex of the triangle by using the weighted average method; Step 76, according to the calculated distance, calculating the three-dimensional position of the virtual prop in the simulation screen space by using the spatial coordinates of the three vertices of the triangle through weighted average.

[0040] Preferably, in step 8 of the above method, the angle of the virtual prop picture displayed on the simulation screen when the simulation screen displays the virtual prop is calculated by the position of the virtual camera and the virtual prop, the vertex position and normal information of each vertex of the simulation screen and the coordinate value calculated in step 6 in the following way, comprising: Step 81, obtaining the position of the virtual prop and the angle of the virtual prop; Step 82, obtaining the position and angle of each virtual camera and calculating the average angle of the two virtual cameras; Step 83, calculating the angle direction of the virtual prop within the view angle of the virtual camera through the average angle; Step 84, obtaining the mesh model of the simulation screen; Step 85, obtaining the triangle list of the mesh model; Step 86, traversing the triangle list to obtain the normal attribute of each point of the triangle; Step 87, according to the distance calculated in step 75 of step 7, the uv coordinate point of the virtual prop in the triangle is calculated by the weighted average method; Step 88: according to the weighted normal direction and the virtual prop angle calculated in step 83, the superimposed angle is calculated, that is, the angle of the virtual prop picture in the simulation screen space.

[0041] Preferably, in step 10 of the above method, the continuous spatial coordinates of each frame of virtual prop picture on the simulation screen are mapped to the real track data according to the real track data to form the motion trajectory data of the real prop on the real stage, including: Step 101, get the real track data of CAD design, import it into the three-dimensional development engine, and generate simulation track data; Step 102, divide the simulation track data into entry segment, performance segment and exit segment; Step 103, according to the performance segment coordinate data of the simulation track data, the continuous spatial coordinates of the virtual prop picture obtained in step 9 are interpolated and corrected according to time, and the corrected data is used as the performance segment animation of the simulation prop; Step 104, calculate the entry segment of the simulation track data according to the fixed time interpolation, and the calculated data is used as the entry segment animation of the simulation prop; Step 105, calculate the exit segment of the simulation track data according to the fixed time interpolation, and the calculated data is used as the exit segment animation of the simulation prop; Step 106, integrate the entry segment animation, performance segment animation and exit segment animation data of the simulation prop, and perform smoothing filtering to obtain the overall animation of the simulation prop track line, that is, the motion trajectory data of the virtual prop corresponding to the real prop on the real stage.

[0042] Through the above steps, the entire running data of a single simulation prop is obtained, which can be converted into the running data of the real prop.

[0043] Preferably, the above method further comprises: Step 11, play the video of step 4 and the real track data of step 10 in the simulation screen to verify the running of the simulation prop, and after verification, send the motion trajectory data of the simulation prop to the ros system by using the TCP network library; Step 12, in the ros system, drive the vehicle chassis of the real prop according to the motion trajectory data of the simulation prop, and play the film on the real screen to test the actual performance scene.

[0044] In summary, the method of the embodiment of the present application can realize simulation of a shooting scene, simulation of a playing screen picture and simulation of a simulation prop. The state of each frame of the virtual prop can be viewed in real time, the position of the simulation prop can be calculated, and the real prop can be controlled through a network component, thereby reducing the number of times and time of on-site testing.

[0045] In order to more clearly show the technical solutions provided by the present application and the technical effects generated, the schemes provided by the embodiments of the present application are described in detail below with specific examples.

[0046] Embodiment 1 As shown in Figure 1 , Figure 2 and Figure 4 , the embodiment of the present application provides a method for calculating real prop motion trajectory data from a virtual camera motion process, which relates to reverse calculation of camera motion trajectory, calculation of actual space coordinates of playing content when playing a screen video, import and export and merging of real vehicle trajectory, etc. The development environment used is based on a three-dimensional development engine UnrealEngine (hereinafter referred to as UE) development, which includes simulation of a 3d shooting scene, simulation of a playing screen picture and simulation of a simulation prop. By using the animation sequence frame controller of UE, the state of each frame of the virtual prop can be viewed in real time, in addition, through a custom plug-in, the position of the simulation prop which needs to be combined with the real prop in front of the playing screen can be calculated, and finally, through a network component and a serialization component, information can be synchronized to the ros environment in real time to control the real prop.

[0047] The main algorithms applied include 3d projection algorithm, uv tiling interpolation calculation method, 3d space position transformation calculation and polynomial interpolation algorithm, in addition, some network communication and data serialization libraries are used.

[0048] The method includes the following steps: Step 1: A film designer designs a virtual scene layout in a three-dimensional development engine according to the content of a script, determines which props in the virtual scene are objects that need to be rendered into a film, which props are virtual props, the virtual props do not need to be rendered into the official film, but are props that are subsequently performed by real props in front of a real screen on a real stage, the virtual props are marked with a special color in the scene and are rendered into a test film, which facilitates subsequent testing in a real scene; Step 2: Based on the script, the filmmaker designs the virtual camera's motion trajectory and duration in the 3D development engine. Since the screen is curved, two virtual cameras are currently used for filming. The photographer needs to set the motion position and angle animation data for both virtual cameras simultaneously. The camera's motion speed and acceleration must match the performance of the real prop vehicle. Furthermore, the virtual camera's parameters must be set based on the screen's curvature. Step 3: Design the simulation scene and props. Based on the actual stage and screen size and position of the actual performance, the 3D modeler creates a 3D model of the real screen in the virtual scene, as well as a 3D model of the real-life props that will be used on the real stage in front of the real screen. Also, because of the curved screen, the simulation screen model needs to be divided into two parts, one for each camera view. Step 4: Virtual screen image simulation: Use the UE Capture2D component to capture the images shot by the two virtual cameras through the parameters of the virtual camera, and play the two captured images on the two simulation screens. The specific method is as follows: Step 41: Set the parameters of the two Capture2D components to the virtual shooting camera parameters; Step 42: Create two Capture2D rendering targets and create two corresponding materials based on the rendering targets; Step 43: Set the display contents of the two simulation screens to the two created materials; In this way, the content displayed on the two simulated screens will be consistent with the video content shot by the two virtual cameras. Through model splicing, the simulation of the real screen is completed.

[0049] Step 5, scene information setting, setting the correspondence between the virtual scene and the simulation scene, which can be achieved by developing a plug-in for the 3D development engine according to the following specific methods: Step 51: Setting the simulation screens corresponding to the two virtual cameras; Step 52: Setting the simulation prop model corresponding to the virtual prop model; Configure the corresponding relationship for subsequent calculation.

[0050] Step 6: Calculate the UV coordinates of the virtual props in the camera image using the Capture2D camera parameter information. Figure 6 , the specific method is: Step 61: Get the camera viewport matrix through the camera parameters.

[0051] Step 62: Get the camera projection matrix through the camera parameters.

[0052] Step 63: Obtain the virtual prop coordinate, calculate with the viewport matrix and the projection matrix, and obtain the uv coordinate of the prop on the camera projection plane.

[0053] Step 7, by simulating the size of the screen model, the vertex position of the simulation screen, the uv information of each vertex, and the coordinate value calculated in step 6, the position of the virtual prop picture displayed in the video picture when the simulation screen displays the video picture of the virtual camera in the simulation screen space is calculated, and the specific method is: Step 71: Obtain the mesh model of the simulation screen; Step 72: Obtain the triangle list of the mesh model; Step 73: Traverse the triangle list to obtain the uv attribute of each point of the triangle; Step 74: Determine whether the virtual prop uv coordinate is in the current triangle, and the determination method is to use the barycentric coordinate method to determine; Step 75: If the virtual prop uv is in the triangle, the distance between the prop uv and each vertex of the triangle is calculated using the weighted average method; Step 76: According to the calculated distance, the spatial coordinates of the three vertices of the triangle are used, and the 3d position of the virtual prop picture in the simulation screen space is obtained by using the weighted average calculation.

[0054] Through the above steps, the virtual prop picture taken by the virtual camera at a certain moment can be obtained, and the three-dimensional coordinates of the virtual prop picture when the simulation screen is played.

[0055] Step 8, by the position information of the virtual camera and the virtual prop, the vertex position of the simulation screen, the normal information of each vertex, and the coordinate value calculated in step 6, the angle of the virtual prop picture displayed in the simulation screen space when the simulation screen displays the virtual prop is calculated, and the specific method is: Step 81: Obtain the virtual prop position and the angle of the virtual prop; Step 82: Obtain the position and angle of each virtual camera, and calculate the average angle of the two virtual cameras; Step 83: Calculate the angle of the virtual prop in the virtual camera view by the average angle; Step 84: Obtain the mesh model of the simulation screen; Step 85: Obtain the triangle list of the mesh model; Step 86: Traverse the triangle list to obtain the normal attribute of each point of the triangle; Step 87: According to the distance calculated in step 7 step 75, the weighted normal direction of the point in the triangle is calculated using the weighted average method; Step 88: Calculate the superimposed angle according to the weighted normal direction and the prop angle orientation calculated in step 83.

[0056] Through the above steps, the virtual prop picture at a certain moment can be obtained, and the rotation angle of the virtual prop picture when playing on the simulation screen.

[0057] Step 9: According to the frame number of the film, the virtual camera motion animation set in step 2 is played to obtain the coordinates and angles of each frame of the virtual prop picture on the simulation screen. In this way, the spatial coordinates of the virtual prop on the simulation screen when the virtual camera is shooting the virtual prop are obtained, and only when the virtual camera shoots the virtual prop is generated.

[0058] Step 10: According to the data of the real track, the coordinates of each frame of the virtual prop on the simulation screen are mapped to the real track data, and the specific method is: Step 101: Obtain the real track data designed by CAD, import it into UE, and generate a simulation track; Step 102: According to the actual situation, the simulation track data is divided into an entry section, a performance section, and an exit section; Step 103: According to the performance section coordinate data of the simulation track, the virtual prop picture coordinate obtained in step 9 is interpolated and corrected according to time, and the corrected data is used as the performance section animation of the simulation prop; Step 104: The entry section of the simulation track is calculated according to the fixed time interpolation, and the calculated data is used as the entry section animation of the simulation prop; Step 105: The exit section of the simulation track is calculated according to the fixed time interpolation, and the calculated data is used as the exit section animation of the simulation prop; Step 106: The entry section, performance section, and exit section animation data of the simulation prop are integrated and smoothed to obtain the entire track line animation of the simulation prop.

[0059] Through the above steps, the entire running data of a single simulation prop is obtained, which can be converted into the running data of the real prop.

[0060] Further, the above method further comprises: Step 11: Play the film in the simulation screen and verify the running of the simulation prop. After verification, use the TCP network library to send the simulation prop motion data to the ros system.

[0061] Step 12: In the ros system, drive the real prop vehicle chassis according to the motion data, and play the film on the real screen to test the actual performance scene.

[0062] Through the above steps, the camera animation during virtual scene shooting is completed, the screen shape and real track information are fused, and the running data of the real stage prop is reversely calculated, so that the subsequent on-site debugging time is greatly reduced, and the program arrangement efficiency is increased.

[0063] As can be seen from the above, the method of the embodiment of the present application can realize simulation of a shooting scene, simulation of a playing screen picture, and simulation of a simulation prop. The state of each frame of the virtual prop can be viewed in real time, the position of the simulation prop can be calculated, and finally the real prop is controlled through a network component, so that the number and time of on-site tests are reduced.

[0064] The method of the present application is applied to a large prop performance stage project and is suitable for a project combining virtual and real performances. First, an animation film file is generated by shooting in a virtual scene, and then the entity prop is moved in front of the arc-shaped screen to cooperate with the film. Since the shooting scene is the movement of the camera cooperating with the movement of the virtual prop, in the actual performance, the real prop moves while the audience is stationary. How to obtain the position and direction of the real prop at a certain moment in the actual scene according to the shooting film is an important factor affecting the performance of virtual and real integration. The present application provides a calculation process to obtain the position and direction of the prop, provides a reference for performance equipment, and improves the program arrangement efficiency.

[0065] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by programs instructing related hardware. The programs can be stored in a computer-readable storage medium, and when the programs are executed, the processes of the above-mentioned embodiments of the methods can be included. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), etc.

[0066] The above description is only a preferred embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in the present application can be easily thought of by those skilled in the art, and should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims. The information disclosed in the background section of the present application is only intended to deepen the understanding of the general background of the present application, and should not be regarded as acknowledging or implying in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A method for calculating the motion trajectory data of a real prop from the motion process of a virtual camera, characterized in that: include: Step 1: Design a virtual scene in a 3D development engine based on the script content and label the virtual props in the virtual scene. Virtual props are props that do not need to be rendered into the video and are subsequently performed by real props on a real stage in front of a real screen. Step 2: Set the motion trajectory and duration of the virtual camera for shooting the video in the 3D development engine, and set the camera parameters of the virtual camera according to the real screen; Step 3, based on the real stage and real screen of the actual performance, create a simulation screen corresponding to the real screen and simulation props corresponding to the real props performed on the real stage in front of the real screen in the simulation scene through the three-dimensional development engine; Step 4: In the 3D development engine, obtain the video shot by the virtual camera through the camera parameters of the virtual camera, and play the video on the corresponding simulation screen; Step 5: Set the corresponding relationship between the virtual scene and the simulation scene; Step 6: Calculate the coordinates of the virtual props in the video image captured by the virtual camera using the camera parameters; Step 7, calculating the position of the virtual prop image displayed in the simulation screen space when the simulation screen displays the video image of the virtual camera by using the size of the simulation screen, the vertex position of the simulation screen, the coordinate value of each vertex, and the coordinate value calculated in step 6; Step 8, calculating the angle of the virtual prop image displayed in the simulation screen space when the virtual prop is displayed on the simulation screen based on the positions of the virtual camera and the virtual prop, the vertex positions of the simulation screen and the normal information of each vertex, and the coordinate values ​​calculated in step 6; Step 9: Play the virtual camera motion trajectory set in step 2. When the virtual camera captures the virtual prop, the coordinates and angles of each frame of the virtual prop on the simulation screen are continuously acquired according to the calculation method in steps 7 and 8, thereby obtaining the continuous spatial coordinates of the virtual prop on the simulation screen when the virtual camera captures the virtual prop. Step 10: Based on the real track data, the continuous spatial coordinates of each frame of the virtual prop image on the simulation screen are mapped to the real track data to form the motion trajectory data of the real prop on the real stage.

2. The method for calculating the motion trajectory data of a real prop from the motion process of a virtual camera according to claim 1, characterized in that: In the method, the three-dimensional development engine used is: UnrealEngine.

3. The method for calculating the motion trajectory data of a real prop from the motion process of a virtual camera according to claim 1 or 2, characterized in that: In step 4, in the 3D development engine, the video shot by the virtual camera is obtained through the camera parameters of the virtual camera in the following manner, and the video is played on the corresponding simulation screen, including: Step 41, setting virtual camera parameters in the 3D development engine; Step 42, creating a render target and a render target material; Step 43: Set the display content of the simulation screen to the created rendering target material, so that the video can be played on the corresponding simulation screen.

4. The method for calculating the motion trajectory data of a real prop from the motion process of a virtual camera according to claim 1 or 2, characterized in that: In step 5, the correspondence between the virtual scene and the simulation scene is set in the following manner, including: Step 51, setting the simulation screens corresponding to the two virtual cameras respectively; Step 52: Set the simulation props corresponding to the virtual props.

5. The method for calculating the real prop motion trajectory data from the virtual camera motion process according to claim 1 or 2, characterized in that: In step 6, the coordinate values ​​of the virtual props in the video frame shot by the virtual camera are calculated using the camera parameters in the following manner, including: Step 61, obtaining the viewport matrix of the virtual camera through the camera parameters; Step 62, obtaining the projection matrix of the virtual camera through the camera parameters; In step 63, the viewport matrix obtained in step 61 and the projection matrix obtained in step 62 are combined to perform calculations to obtain the UV coordinates of the virtual prop on the virtual camera projection plane.

6. The method for calculating real prop motion trajectory data from a virtual camera motion process according to claim 1 or 2, characterized in that: In step 7, the position of the virtual prop image displayed in the simulation screen space when the simulation screen displays the video image of the virtual camera is calculated by using the size of the simulation screen, the vertex position of the simulation screen, the coordinate value of each vertex, and the coordinate value calculated in step 6 in the following manner, including: Step 71, obtaining a grid model of the simulation screen; Step 72, obtaining a triangle list of the mesh model; Step 73, traverse the triangle list and obtain the UV attribute of each point of the triangle; Step 74, using the barycentric coordinate method to determine whether the UV coordinates of the virtual prop are within the current triangle; Step 75: If the UV coordinate of the virtual prop is within the triangle, use the weighted average method to calculate the distance between the UV coordinate of the virtual prop and each vertex of the triangle; Step 76 , based on the calculated distance, the spatial coordinates of the three vertices of the triangle are used to calculate the three-dimensional position of the virtual prop in the simulation screen space through weighted average calculation.

7. The method for calculating the real prop motion trajectory data from the virtual camera motion process according to claim 6, characterized in that: In step 8, the angle of the virtual prop image displayed in the simulation screen space when the virtual prop is displayed on the simulation screen is calculated using the positions of the virtual camera and the virtual prop, the vertex positions of the simulation screen and the normal information of each vertex, and the coordinate values ​​calculated in step 6 in the following manner, including: Step 81, obtaining the position and angle of the virtual prop; Step 82, obtaining the position and angle of each virtual camera, and calculating the average angle of the two virtual cameras; Step 83, calculating the angular orientation of the virtual prop within the virtual camera's viewing angle by averaging the angles; Step 84, obtaining a grid model of the simulation screen; Step 85, obtaining a triangle list of the mesh model; Step 86, traverse the triangle list and obtain the normal attribute of each point of the triangle; Step 87, using the weighted average method to calculate the weighted normal direction of the UV coordinate point of the virtual prop in the triangle based on the distance calculated in step 75 of step 7; Step 88: Based on the weighted normal direction and the virtual prop angle orientation calculated in step 83, the superimposed angle is calculated, which is the angle of the virtual prop image in the simulation screen space.

8. The method for calculating real prop motion trajectory data from a virtual camera motion process according to claim 1 or 2, characterized in that: In step 10, the continuous spatial coordinates of each frame of the virtual prop image on the simulation screen are mapped to the real track data according to the real track data to form the motion trajectory data of the real prop on the real stage, including: Step 101: Acquire the real track data designed by CAD, import it into the 3D development engine, and generate simulated track data; Step 102, dividing the simulation track data into an entrance segment, a performance segment, and an exit segment; Step 103, interpolating and correcting the continuous spatial coordinates of the virtual prop screen obtained in step 9 according to the performance segment coordinate data of the simulation track data according to time, and using the corrected data as the performance segment animation of the simulation prop; Step 104, the entrance segment of the simulation track data is interpolated and calculated according to a fixed time, and the calculated data is used as the entrance segment animation of the simulation prop; Step 105, the exit segment of the simulation track data is interpolated and calculated according to a fixed time, and the calculated data is used as the exit segment animation of the simulation prop; Step 106 , integrating the entrance, performance, and exit animation data of the simulated props, and performing smooth filtering to obtain the overall animation of the track line of the simulated props, that is, the motion trajectory data of the real props corresponding to the virtual props on the real stage.

9. The method for calculating the real prop motion trajectory data from the virtual camera motion process according to claim 8, characterized in that: The method further comprises: Step 11: Play the video of step 4 and the real track data of step 10 on the simulation screen to verify the operation of the simulation props. After the verification is passed, use the TCP network library to send the motion trajectory data of the simulation props to the ROS system; Step 12: In the ROS system, the vehicle chassis of the real props is driven according to the motion trajectory data of the simulated props, and the movie is played on the real screen to conduct actual performance scene testing.

Images