Mixed reality virtual preview shooting system

Abstract

The present invention relates to the field of motion capture technology, and discloses a mixed reality virtual preview shooting system, which is used to improve the efficiency of mixed reality virtual video shooting and reduce production costs. The mixed reality virtual preview shooting system includes a real shooting system, a virtual shooting system, an optical positioning and motion capture system, a central control system and a display system; the real shooting system includes a film and television camera and a video capture card; the virtual shooting system includes a virtual camera, an inertial measurement device and a wireless transmission device; the optical positioning and motion capture system includes a light capture processing server and multiple optical motion capture cameras, and the light capture processing server is connected to multiple optical motion capture cameras and wireless transmission devices; the central control system includes a central management server, and the central management server is connected to the video capture card, the wireless transmission device and the light capture processing server respectively; the display system includes a display device, and the display device is connected to the central management server network.

Description

Technical Field

[0001] The present invention relates to the field of motion capture technology, and in particular to a mixed reality virtual preview shooting system. Background Art

[0002] Mixed reality technology is a further development of virtual reality technology. By presenting virtual scene information in real scenes, an interactive feedback information loop is established between the real world, the virtual world and the user to enhance the realism of the user experience.

[0003] In traditional film and television visual design, traditional virtual studios support relatively few cameras and lenses, most of which are fixed or cannot be moved freely, the docking rendering engine is single or old, and the synthesis relationship between characters and visual design CG is mostly a single front and back layer superposition, without complex mixed virtual and real interactive parts. At the same time, in order to avoid being exposed, the camera position needs to be completely fixed at a fixed angle, which is difficult to adjust temporarily on the spot. It relies solely on visual illusions to improve the sense of integration and cannot respond to light.

[0004] In the prior art, starting from obtaining the storyboard from the director and crew, an offline method is adopted, relying on 3D animators to produce shots and pre-show the storyboards, which has a long production cycle. Therefore, the hybrid virtual-reality shooting system lacks on-site storyboard pre-shows, resulting in low efficiency in the later computer-synthesized 3D video. Summary of the invention

[0005] The main purpose of the present invention is to solve the problem that mixed virtual and real shooting cannot be previewed on site, resulting in low efficiency of later computer production.

[0006] The first aspect of the present invention provides a mixed reality virtual pre-production shooting system, which includes a real shooting system, a virtual shooting system, an optical positioning and motion capture system, a central control system and a display system connected in communication; wherein:

[0007] The real shooting system includes a video camera and a video capture card, and the video camera and the video capture card are in communication connection;

[0008] The virtual shooting system includes a virtual camera, an inertial measurement device and a wireless transmission device, wherein the inertial measurement device is installed on the virtual camera, and the inertial measurement device is communicatively connected with the wireless transmission device;

[0009] The optical positioning and motion capture system comprises an optical capture processing server and a plurality of optical motion capture cameras, wherein the optical capture processing server is respectively connected to the plurality of optical motion capture cameras and the wireless transmission device for communication;

[0010] The central control system includes a central management server, which is respectively connected to the video capture card, the wireless transmission device and the light capture processing server for communication;

[0011] The display system comprises a display device, and the display device is connected to the central management server through a network.

[0012] Optionally, the video camera is used to shoot the object to be captured in an actual shooting scene to obtain real-shot video data; and the video capture card is used to send the real-shot video data to the central management server.

[0013] Optionally, the object to be captured wears a motion capture suit with a plurality of optical marking points attached thereto in advance, and the plurality of optical marking points are used to locate the positions of various joints of the object to be captured.

[0014] Optionally, the video camera and the virtual camera are respectively installed with a preset number of optical marking points;

[0015] The inertial measurement device is used to collect inertial navigation data corresponding to the virtual camera in mobile shooting through a nine-axis inertial sensor;

[0016] The wireless transmission device is used to send the corresponding inertial navigation data to the light capture processing server.

[0017] Optionally, the optical capture processing server further includes a three-dimensional motion capture processing module and an optical inertial fusion module, the three-dimensional motion capture processing module is communicatively connected to the multiple optical motion capture cameras, and the optical inertial fusion module is communicatively connected to the central management server and the wireless transmission device respectively;

[0018] The multiple optical motion capture cameras are used to locate and photograph the optical marker points respectively to obtain the corresponding two-dimensional image data;

[0019] The three-dimensional motion capture processing module is used to obtain the three-dimensional coordinate data corresponding to each other and the motion posture information of the object to be captured according to the two-dimensional image data corresponding to each other;

[0020] The optical inertial fusion module is used to perform coordinate system calibration and posture fusion processing in sequence according to the three-dimensional coordinate data corresponding to each optical marker point on the virtual camera and the corresponding inertial navigation data to obtain camera posture data, and send the corresponding three-dimensional coordinate data, the camera posture data and the motion posture information to the central management server.

[0021] Optionally, the light capture processing server also includes an inertial navigation setting module, which is communicatively connected to the optical inertial fusion module and is used to install the nine-axis inertial sensor and the preset number of optical marker points on the virtual camera according to a preset position relationship using a preset link port number and a preset rigid body name.

[0022] Optionally, the central management server includes a virtual rehearsal shooting and synthesis module and a rendering and synthesis module, the virtual rehearsal shooting and synthesis module is respectively connected to the video acquisition card and the optical inertial fusion module in communication, and the rendering and synthesis module is respectively connected to the virtual rehearsal shooting and synthesis module and the wireless transmission device in communication;

[0023] The virtual preview shooting synthesis module is used to perform real-time matting on the real-shot video data so as to generate a virtual image in the virtual scene and adjust the virtual scene angle in real time;

[0024] The rendering and synthesis module is used to convert the matte image information and the adjusted virtual scene into a three-dimensional virtual reality mixed video stream according to the camera posture data, the corresponding three-dimensional coordinate data and the motion posture information, and send the three-dimensional virtual reality mixed video stream to the virtual camera and the display device respectively.

[0025] Optionally, the display device is used to receive and synchronously display the three-dimensional virtual reality mixed video stream, so that the target person can preview the three-dimensional virtual reality mixed video stream during the displacement of the video camera and the virtual camera, and adjust the shooting action and shooting angle of the object to be captured in real time.

[0026] Optionally, the optical positioning and motion capture system further includes a calibration device, wherein the calibration device is used to calibrate the positions of the multiple optical motion capture cameras in the actual shooting scene through a calibration rod.

[0027] Optionally, the mixed reality virtual rehearsal shooting system also includes a camera setting system, which is communicatively connected to the central control system, and is used to obtain site parameters in the actual shooting scene, and determine the number of cameras and the corresponding camera installation positions corresponding to the optical motion capture cameras according to the site parameters, so that each optical marker point can be captured by any three optical motion capture cameras, and the site parameters include large space site length information and width information.

[0028] In the technical solution provided by the present invention, the mixed reality virtual rehearsal shooting system includes a real shooting system, a virtual shooting system, an optical positioning and motion capture system, a central control system and a display system that are connected in communication; wherein the real shooting system includes a film and television camera and a video capture card, and the film and television camera is connected in communication with the video capture card; the virtual shooting system includes a virtual camera, an inertial measurement device and a wireless transmission device, and the inertial measurement device is installed on the virtual camera, and the inertial measurement device is connected in communication with the wireless transmission device; the optical positioning and motion capture system includes a light capture processing server and a plurality of optical motion capture cameras, and the light capture processing server is respectively connected in communication with the plurality of optical motion capture cameras and the wireless transmission device; the central control system includes a central management server, and the central management server is respectively connected in communication with the video capture card, the wireless transmission device and the light capture processing server; the display system includes a display device, and the display device is connected in network with the central management server. In the embodiment of the present invention, motion trajectory recording, data export fusion, redundant target shielding and real-time preview of imaging are realized through the real shooting system, the virtual shooting system, the optical positioning and motion capture system, the central control system and the display system. Compared with traditional green screen matting, the present invention obtains the three-dimensional motion trajectory information of the object to be captured in real time, realizes the superposition of scenes and characters in three-dimensional space, and has the synthetic processing effect of three-dimensional depth information; at the same time, through virtual reality interaction, the camera movement visual effect of the filmed video can be previewed in real time, thereby improving the efficiency of mixed reality virtual video shooting and reducing production costs and production cycles. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of the structure of a mixed reality virtual preview shooting system according to an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of the structure of an optical positioning and motion capture system in an embodiment of the present invention;

[0031] Figure 3 A schematic diagram of the structure of a central management server in an embodiment of the present invention;

[0032] Figure 4 Another structural schematic diagram of the mixed reality virtual pre-performance shooting system according to an embodiment of the present invention;

[0033] Figure 5 Schematic diagram of an application scenario of a mixed reality virtual pre-show shooting system in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The embodiment of the present invention provides a mixed reality virtual preview shooting system, which previews the camera movement visual effects of a filmed film in real time through virtual reality interaction, improves the efficiency of mixed reality virtual video shooting, and reduces production costs and production cycles.

[0035] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that shown or described herein.

[0036] For ease of understanding, the following describes an embodiment of the present invention. Figure 1 In the embodiment of the present invention, the mixed reality virtual preview shooting system includes a real shooting system 1, a virtual shooting system 2, an optical positioning and motion capture system 3, a central control system 4 and a display system 5 connected in communication; wherein,

[0037] The real shooting system 1 includes a video camera 11 and a video capture card 12, and the video camera 11 and the video capture card 12 are in communication connection;

[0038] The virtual shooting system 2 includes a virtual camera 21, an inertial measurement device 22 and a wireless transmission device 23. The inertial measurement device 22 is installed on the virtual camera 21, and the inertial measurement device 22 is connected to the wireless transmission device 23 for communication.

[0039] The optical positioning and motion capture system 3 includes an optical capture processing server 31 and a plurality of optical motion capture cameras 32, and the optical capture processing server 31 is respectively connected to the plurality of optical motion capture cameras 32 and the wireless transmission device 23 for communication;

[0040] The central control system 4 includes a central management server 41, which is respectively connected to the video capture card 12, the wireless transmission device 23 and the light capture processing server 31 for communication;

[0041] The display system 5 includes a display device 51 , and the display device 51 is connected to the central management server 41 through a network.

[0042] Specifically, the real shooting system 1 is used to receive a preset shooting instruction, and based on the preset shooting instruction, the film and television camera 11 is used to shoot a video of the object to be captured in the actual shooting scene to obtain real-shot video data. Further, the video capture card 12 is used to upload the real-shot video data to the central management server 41, so that the central management server 41 can store and process the real-shot video data. Among them, the object to be captured is a moving object, including a person and a prop, and the real-shot video data is used to indicate the motion picture information of the object to be captured in the actual shooting scene. Optionally, the actual shooting scene includes a green screen and a truss, and the truss is used to layout multiple optical motion capture cameras 32, wherein the background of the actual shooting scene can also be a blue screen, which is not specifically limited here. It should be noted that the film and television camera 11 is also used to support the lens translation, up and down movement, and zooming in and out of the fixed pan and tilt, and supports the camera movement of a large rocker arm in a large range, for example, the film and television camera 11 is used in an actual shooting scene of 10 to 2000 square meters.

[0043] Specifically, the virtual camera 21 in the virtual shooting system 2 is a handheld six-degree-of-freedom camera, and the virtual camera 21 is used to move and shoot the preset virtual scene corresponding to the actual shooting scene, wherein the six degrees of freedom refer to that an object has six degrees of freedom in three-dimensional space, that is, the degrees of freedom of movement along the three rectangular coordinate axes of x, y, and z and the degrees of freedom of rotation around the three coordinate axes of x, y, and z. Therefore, the virtual camera 21 can be used to adjust the shooting angle of the preset virtual scene, and can also be used to adjust the focal length and aperture value, and to control the shooting start and end time of the preset virtual scene. For example, when the start button in the virtual camera 21 is activated, the virtual camera 21 is used to receive a start recording instruction and start shooting the preset virtual scene. When the pause button or stop button of the virtual camera 21 is activated, the virtual camera 21 is used to receive a pause recording instruction or a stop recording instruction and stop shooting the preset virtual scene. And it is used to synchronously display the three-dimensional virtual reality mixed video stream sent by the central management server 41. It should be noted that when the virtual camera 21 moves, it shoots the object to be captured in the preset virtual scene, and there are changes in shooting angle and moving speed. Therefore, the inertial measurement device 22 is used to collect motion posture information of the virtual camera 21, and the wireless transmission device 23 is used to upload the collected motion posture information to the optical capture processing server 31 in the optical positioning and motion capture system 3. Among them, the wireless transmission device 23 can be wireless Bluetooth, and also has wireless Bluetooth image transmission function, or it can be wireless WIFI.

[0044] Specifically, the optical positioning and motion capture system 3 includes multiple optical motion capture cameras 32, which are used to identify optical markers bound to different parts of the object to be captured. The optical capture processing server 31 is used to obtain the position and orientation of each optical marker in the actual shooting scene, thereby determining the motion trajectory of the object to be captured in the actual shooting scene, and importing it into the central management server 41 in real time. Optionally, the object to be captured wears a motion capture suit with multiple optical markers attached in advance, and the multiple optical markers are used to locate the joint positions of the object to be captured, wherein the optical markers include reflective markers and active markers, and the reflective markers can be reflective balls. The active markers are suitable for situations where the reflective markers are difficult to locate and track under ambient lighting conditions, so that the optical positioning and motion capture system 3 can be used both indoors and outdoors. Furthermore, the video camera 11 and the virtual camera 21 are also pre-set with a preset number of optical markers, respectively. The preset number of optical markers combined with multiple optical motion capture cameras 32 are used to locate the spatial position information of the video camera 11 and the virtual camera 21, wherein the preset number is a positive integer. For example, 3, 4 or 5, the specific number is not limited here.

[0045] For example, different reflective marking points (reflective balls) are arranged on the head, hands, feet and lower back of the object to be captured, and multiple optical motion capture cameras 32 are used to track the reflective marking points on the object to be captured, and accurately locate the spatial position and orientation of the reflective marking points, so as to obtain the extremity position of the object to be captured. The optical capture processing server 31 is used to determine the motion posture information of the object to be captured according to the extremity position of the object to be captured.

[0046] Specifically, the central management server 41 in the central control system 4 is first used to perform matte processing on the real-shot video data captured by the film and television camera 11 to obtain matte image information, wherein the matte image information includes character model information, and secondly, to synthesize the matte image information into a preset virtual scene (a virtual scene corresponding to the actual shooting scene) through three-dimensional synthesis software or a preset engine, or to obtain a virtual character corresponding to the character model information, merge the corresponding virtual character into the corresponding preset virtual scene, and then drive the character model information or virtual character movement through forward kinematics and motion posture information. For example, when a motion capture actor (object to be captured) wearing a green motion capture suit performs an action in a green screen shooting space, the optical capture processing server 31 and multiple optical motion capture cameras 32 are used to obtain all the motion posture information of the motion capture actor. The central management server 41 is used to perform overall picture calculation on the real-shot video data shot by the real shooting system 1, obtain the matte image information containing the motion capture actor, and the virtual character corresponding to the matte image information, and synthesize the corresponding virtual character into the preset virtual scene actually required through the three-dimensional synthesis software or the preset engine, and use the motion posture information to drive the pre-set virtual character, the virtual character is a virtual object dancing with the music, and obtain a three-dimensional virtual reality mixed video stream. Further, the central management server 41 is used to send the three-dimensional virtual reality mixed video stream to the virtual camera 21 and the display device 51. Optionally, in addition to being used to superimpose and synthesize the preset virtual scene with the virtual character, the central management server 41 can also be used to perform three-dimensional depth synthesis processing on the real character and the preset virtual scene, and the preset real scene and the virtual character, which is not limited here. It should be noted that the central control system 4 also includes input devices and output devices, such as a mouse, a keyboard and a display.

[0047] Specifically, the display system 5 includes a display device 51, which is used to receive and synchronously display the three-dimensional virtual reality mixed video stream sent by the central management server 41, so that the target personnel can adjust the shooting action and shooting angle of the object to be captured in real time, and preview the visual effect of the camera movement of the filmed film during the displacement of the film camera 11 and the virtual camera 21, wherein the target personnel include the director and the photographer. Therefore, the display device 51 is used to provide a real-time scheduling preview of the lens picture composition, lens movement, and synthesis effect, so that the target personnel can adjust and modify the creativity and picture composition in time. This avoids the problems of high production costs and long production cycles in the early and late stages of traditional film and television production, and provides real-time visual shooting effects through virtual preview shooting.

[0048] In an embodiment of the present invention, a mixed display virtual preview shooting system is provided, which realizes motion trajectory recording, data export fusion, redundant target (objects other than the object to be captured) shielding and real-time preview of imaging through a real shooting system, a virtual shooting system, an optical positioning and motion capture system, a central control system and a display system. Compared with the traditional green screen matting, the present invention obtains the three-dimensional motion trajectory information of the object to be captured in real time, realizes the superposition of scenes and characters in three-dimensional space, and has the synthesis processing effect of three-dimensional depth information; at the same time, through virtual reality interaction, the visual effect of the camera movement of the filmed film is previewed in real time, which improves the efficiency of mixed reality virtual video shooting and reduces production costs and production cycles.

[0049] See also Figure 1 and Figure 2 The optical capture processing server 31 further includes a three-dimensional motion capture processing module 311 and an optical inertial fusion module 312. The three-dimensional motion capture processing module 311 is communicatively connected to a plurality of optical motion capture cameras 32. The optical inertial fusion module 312 is communicatively connected to the central management server 41 and the wireless transmission device 23 respectively.

[0050] A plurality of optical motion capture cameras 32 are used to locate and photograph each optical marker point to obtain two-dimensional image data corresponding to each point;

[0051] A three-dimensional motion capture processing module 311 is used to obtain the three-dimensional coordinate data and the motion posture information of the object to be captured according to the two-dimensional image data;

[0052] The optical inertial fusion module 312 is used to perform coordinate system calibration and posture fusion processing in sequence according to the motion posture information corresponding to each optical marker point on the virtual camera 21 and the corresponding inertial navigation data, obtain camera posture data, and send the corresponding three-dimensional coordinate data, camera posture data and motion posture information to the central management server 41.

[0053] The multiple optical motion capture cameras 32 can be connected to the optical capture processing server 31 by wire or wirelessly, for example, by a POE switch. Specifically, the multiple optical motion capture cameras 32 are used to collect the two-dimensional image information corresponding to the video camera 11, the virtual camera 21 and the multiple optical markers, that is, the video camera 11, the virtual camera 21 and each optical marker in the object to be captured are photographed by at least two or more optical motion capture cameras 32 at the same time; the three-dimensional motion capture processing module 311 is used to convert the two-dimensional image data corresponding to each into the three-dimensional coordinate data corresponding to each, that is, the spatial position information, and the motion posture information of the object to be captured, wherein the three-dimensional coordinate data corresponding to each is used to indicate the position and orientation of each optical marker in the world coordinate system, so as to realize the positioning and tracking of the moving object; and the optical inertial fusion module 312 is used to receive the inertial navigation data corresponding to the virtual camera 21, and perform coordinate system calibration and posture fusion calculation based on the three-dimensional coordinate data corresponding to each optical marker on the virtual camera 21 and the corresponding inertial navigation data, obtain the camera posture data, and send the three-dimensional coordinate data, camera posture data and motion posture information corresponding to each to the central management server 41. Optionally, the optical positioning and motion capture system 3 further includes a calibration device 33, which is used to calibrate the positions of the multiple optical motion capture cameras 32 in the actual shooting scene through a calibration rod. For example, the calibration rod may be in a T-shaped structure, and the dimensions of the calibration rod are a horizontal rod length of 418 mm and a vertical rod length of 578 mm.

[0054] Furthermore, the inertial measurement device 22 is used to collect the inertial navigation data corresponding to the virtual camera 21 and multiple optical markers during mobile shooting through a nine-axis inertial sensor, and send the corresponding inertial navigation data to the wireless transmission device 23, and the inertial measurement device 22 is an inertial measurement unit (IMU); the wireless transmission device 23 is used to send the corresponding inertial navigation data to the optical inertial fusion module 312. Optionally, the light capture processing server 31 also includes an inertial navigation setting module 313, which is connected to the optical inertial fusion module 312 for communication, and is used to install the nine-axis inertial sensor and a preset number of optical markers on the virtual camera 21 according to a preset position relationship using a preset link port number and a preset rigid body name. Among them, the nine-axis inertial sensor includes a three-axis angular velocity sensor (gyroscope), a three-axis acceleration sensor, and a three-axis magnetic induction sensor. The nine-axis inertial sensor can ensure high-precision calculation of the rigid body rotation angle, and its error range is within 0.05 degrees, so that the synthesized three-dimensional virtual reality mixed video stream is more stable and can interact in real time with low latency. For example, a 200 Hz angle data refresh rate is used to update the three-dimensional virtual reality mixed video stream. It is understandable that the preset number of optical markers can use at least 3 reflective balls. The reflective ball is a rigid body and also a motion tracker, and its motion trajectory represents the change in spatial coordinates. In order to obtain a more accurate posture, it is generally used in conjunction with a nine-axis inertial sensor (inertial measurement device 22). The spatial coordinates of the rigid body and the nine-axis inertial sensor are unified to finally obtain the precise posture data of the rigid body. The purpose of calibrating the rigid body and the nine-axis inertial sensor is to unify the spatial coordinate system. The motion posture information of the rigid body is the position and posture in the custom world coordinate system. The posture of the nine-axis inertial sensor (the corresponding inertial navigation data) is the posture change relative to the hardware itself when it is started. The coordinate system alignment processing of the rigid body and the nine-axis inertial sensor, as well as the posture fusion processing, can improve the calculation accuracy of the rigid body's posture.

[0055] It should be noted that the reflective ball (optical marker) is flexibly bound to the nine-axis inertial sensor, and the preset rigid body name and the preset link port number are a one-to-one mapping relationship. Therefore, the inertial navigation setting module 313 is used to update the configuration of the preset rigid body name and the preset link port number and clear the existing configuration, so that the reflective ball (optical marker) and the nine-axis inertial sensor can be bound and unbound. At the same time, the nine-axis inertial sensor reduces the circuit board and the overall space, and the corresponding hardware volume is small, which is suitable for portable installation on a moving object that accurately captures rotation data, and improves the calculation accuracy of a single sensor in locating the spatial position and the direction of movement. It can be understood that the inertial navigation setting module 313 is not only used to determine the motion posture data of the virtual camera 21, but also can be used to determine the motion posture data of the object to be captured, and the association relationship between the nine-axis inertial sensor and the optical marker is determined and set according to the actual shooting scene, thereby achieving high-precision motion detection.

[0056] See also Figure 1 and Figure 3 Optionally, the central management server 41 includes a virtual rehearsal shooting synthesis module 411 and a rendering synthesis module 412, the virtual rehearsal shooting synthesis module 411 is respectively connected to the video capture card 12 and the optical inertial fusion module 312 for communication, and the rendering synthesis module 412 is respectively connected to the virtual rehearsal shooting synthesis module 411 and the wireless transmission device 23 for communication;

[0057] The virtual preview shooting synthesis module 411 is used to perform real-time matting of the real-shot video data so as to generate a virtual image in the virtual scene and adjust the angle of the virtual scene in real time;

[0058] The rendering synthesis module 412 is used to convert the matte image information and the adjusted virtual scene into a three-dimensional virtual reality mixed video stream according to the camera posture data, the corresponding three-dimensional coordinate data and the motion posture information, and send the three-dimensional virtual reality mixed video stream to the virtual camera 21 and the display device 51 respectively.

[0059] The rendering synthesis module 412 receives the matte image information sent by the virtual preview shooting synthesis module 411, wherein the matte image information is the image information containing the object to be captured, and the rendering synthesis module 412 is used to use the Unreal Engine UE4 to perform three-dimensional animation character setting, sound simulation and lighting rendering. Specifically, the virtual preview shooting synthesis module 411 is used to set the virtual character corresponding to the matte image information and adjust the virtual scene angle in real time; the rendering synthesis module 412 is used to synchronously set the motion posture of the virtual character according to the camera posture data, the three-dimensional coordinate data corresponding to each, and the motion posture information through the Unreal Engine UE4; at the same time, the preset voice information is added to the preset virtual scene to obtain a three-dimensional virtual reality mixed video stream. Further, the wireless transmission device 23 is used to send the three-dimensional virtual reality mixed video stream to the virtual shooting machine 21 and the display device 51 respectively, so that the target person can preview the shooting effect in real time. Among them, the display device 51 can use a large screen projection method to display the storyboard lens, so as to achieve the visualization of the storyboard.

[0060] Optionally, the central management server 41 is also used to store camera posture data, corresponding three-dimensional coordinate data and motion posture information, and three-dimensional virtual reality mixed video streams to facilitate post-production and synthesis.

[0061] Optionally, the central management server 41 is also used to align the adjusted virtual scene and the actual shooting scene in the three-dimensional space coordinate system according to the camera posture data and the corresponding three-dimensional coordinate data before performing video synthesis, so that the output three-dimensional virtual reality mixed video stream has high frame rate, stability, and delay-free spatial positioning and virtual-reality synchronization effects.

[0062] It can be understood that the virtual rehearsal shooting and synthesis module 411 is used to indicate the color keying processing, that is, to define the transparency according to the specific color value or brightness value of the image in the real video data. When a certain value is keyed out, all pixels with similar color or brightness value will become transparent. Furthermore, the virtual rehearsal shooting and synthesis module 411 is used to obtain the keying image information from the real video data.

[0063] See also Figure 4 Optionally, the mixed reality virtual rehearsal shooting system also includes a camera setting system 6, which is communicated with the central control system 4 and is used to obtain the site parameters in the actual shooting scene, and determine the number of cameras and the corresponding camera installation positions corresponding to the optical motion capture cameras 32 according to the site parameters, so that each optical marker point is captured by any three optical motion capture cameras 32, and the site parameters include large space site length information and width information.

[0064] In the mixed reality virtual preview shooting system, it is necessary to locate and track the motion trajectory of multiple optical markers. In order to improve the accuracy of positioning, each optical marker needs to be captured by any three optical motion capture cameras 32, and then the spatial position information of each optical marker is determined. However, too many cameras will lead to the problem of too high construction cost. Therefore, the camera setting system 6 is used to arrange the optical motion capture cameras 32, please refer to Figure 4 and Figure 5 , the actual shooting scene is an open space with a height of 3.5 meters to 7 meters, and is equipped with professional film and television lighting, trusses, green boxes or blue boxes, wherein the trusses include single-layer trusses, two-layer trusses and multi-layer trusses, and the site parameters of the actual shooting scene include large space site length information of 20 meters and width information of 10 meters. Further, the camera setting system 6 is used to calculate the camera configuration cost value in the actual shooting scene according to the site parameters of the actual shooting scene, and determine the number of cameras corresponding to the optical motion capture camera 32 and the corresponding camera installation position through the camera configuration cost value. Multiple optical motion capture cameras 32 are arranged at different positions of the truss in the actual shooting scene, and the visual range of each optical motion capture camera 32 covers the entire actual shooting scene. The film and television camera 11 and the virtual camera 21 are used to shoot the object to be captured.

[0065] It is understandable that the hybrid display virtual preview shooting system combines multiple functions such as real-time preview, virtual broadcast, and motion capture. On the basis of traditional matting technology, it uses spatial positioning motion capture technology to synchronize the positions of the film and television cameras 11, virtual cameras 21 and all virtual and real objects (for example, props) in the space, so that they can interact and preview the synthesis effect in real time in the Unreal Engine. In the smooth picture rendering and realistic three-dimensional fusion picture, the virtual and real combination effect is achieved, which improves the storytelling and visual expression of the film and television shooting techniques. Not only can the target personnel (director) interact on site to clarify the needs, obtain the picture angle of the preset composition according to the needs, and preview the shooting effect in real time, but also determine the effects and elements of each lens in real time through the interactive visualization lens, the shooting process is in place in one step, and the post-production has determined the direction in the shooting stage. At the same time, for real-life actors, virtual actors, and virtual characters interacting in real time, the interaction between characters and CG is no longer a simple front and back superposition, and the position of objects has 3D depth information. For example, real people walk into the virtual ancient city, real people walk around the CG car, real people and virtual characters hug and fight with complex and natural performance effects. The camera position is recorded directly on the set and seamlessly imported into the illusion Maya 3D software for rendering and synthesis, reducing the cost of repeated communication and modification from pre-production to post-production. Moreover, under any shooting conditions, the virtual character scene can be accurately tracked in the real environment, and the 11-position displacement lens of the film and television camera can be zoomed in and out, and the virtual and real scenes can be fully presented in real time. This realizes the animation production of film and television, television, animation and games, as well as the virtual reality preview shooting in studios, studios and large exhibition halls.

[0066] In the embodiment of the present invention, a mixed display virtual preview shooting system is provided, which realizes motion trajectory recording, data export fusion, redundant target shielding and real-time preview of imaging through a real shooting system, a virtual shooting system, an optical positioning and motion capture system, a central control system and a display system. Compared with the traditional green screen matting, the three-dimensional motion trajectory information of the object to be captured is obtained in real time, and the superposition of scenes and characters in three-dimensional space is realized, with the synthesis processing effect of three-dimensional depth information; at the same time, through virtual reality interaction, the visual effect of the camera movement of the filmed film can be previewed in real time, which improves the efficiency of mixed reality virtual video shooting and reduces production costs and production cycles.

[0067] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A mixed reality virtual preview shooting system, characterized in that: The mixed reality virtual preview shooting system includes a real shooting system, a virtual shooting system, an optical positioning and motion capture system, a central control system and a display system connected in communication; wherein, The real shooting system includes a video camera and a video capture card, and the video camera and the video capture card are in communication connection; The virtual shooting system includes a virtual camera, an inertial measurement device and a wireless transmission device, wherein the inertial measurement device is installed on the virtual camera, and the inertial measurement device is communicatively connected with the wireless transmission device; wherein the virtual camera is used to adjust the shooting angle of a preset virtual scene, or adjust the focal length and aperture value, or control the shooting start and end time of the preset virtual scene; The optical positioning and motion capture system comprises an optical capture processing server and a plurality of optical motion capture cameras, wherein the optical capture processing server is respectively connected to the plurality of optical motion capture cameras and the wireless transmission device for communication, wherein the optical motion capture cameras are used to identify optical marker points bound to different parts of the object to be captured, and the optical capture processing server is used to obtain the position and orientation of each optical marker point in the actual shooting scene, thereby determining the motion trajectory of the object to be captured in the actual shooting scene; The central control system includes a central management server, which is respectively connected to the video capture card, the wireless transmission device and the light capture processing server for communication; The display system includes a display device, which is connected to the central management server network and is used to provide real-time scheduling preview of lens image composition, lens movement, and synthesis effect.

2. The mixed reality virtual pre-performance shooting system according to claim 1, characterized in that: The video camera is used to shoot the object to be captured in the actual shooting scene to obtain real-shot video data; the video acquisition card is used to send the real-shot video data to the central management server.

3. The mixed reality virtual pre-performance shooting system according to claim 2, characterized in that: The object to be captured wears a motion capture suit with a plurality of optical marking points attached in advance, and the plurality of optical marking points are used to locate the positions of the joints of the object to be captured.

4. The mixed reality virtual pre-performance shooting system according to claim 2, characterized in that: The video camera and the virtual camera are respectively equipped with a preset number of optical marking points; The inertial measurement device is used to collect inertial navigation data corresponding to the virtual camera in mobile shooting through a nine-axis inertial sensor; The wireless transmission device is used to send the corresponding inertial navigation data to the light capture processing server.

5. The mixed reality virtual pre-performance shooting system according to claim 4, characterized in that: The optical capture processing server further includes a three-dimensional motion capture processing module and an optical inertial fusion module, wherein the three-dimensional motion capture processing module is communicatively connected to the plurality of optical motion capture cameras, and the optical inertial fusion module is communicatively connected to the central management server and the wireless transmission device respectively; The multiple optical motion capture cameras are used to locate and photograph the optical marker points respectively to obtain the corresponding two-dimensional image data; The three-dimensional motion capture processing module is used to obtain the three-dimensional coordinate data corresponding to each other and the motion posture information of the object to be captured according to the two-dimensional image data corresponding to each other; The optical inertial fusion module is used to perform coordinate system calibration and posture fusion processing in sequence according to the three-dimensional coordinate data corresponding to each optical marker point on the virtual camera and the corresponding inertial navigation data to obtain camera posture data, and send the corresponding three-dimensional coordinate data, the camera posture data and the motion posture information to the central management server.

6. The mixed reality virtual pre-performance shooting system according to claim 5, characterized in that: The light capture processing server also includes an inertial navigation setting module, which is communicatively connected to the optical inertial fusion module and is used to install the nine-axis inertial sensor and the preset number of optical markers on the virtual camera according to a preset position relationship using a preset link port number and a preset rigid body name.

7. The mixed reality virtual pre-performance shooting system according to claim 5, characterized in that: The central management server includes a virtual rehearsal shooting synthesis module and a rendering synthesis module, wherein the virtual rehearsal shooting synthesis module is respectively connected to the video acquisition card and the optical inertial fusion module in communication, and the rendering synthesis module is respectively connected to the virtual rehearsal shooting synthesis module and the wireless transmission device in communication; The virtual preview shooting synthesis module is used to perform real-time matting on the real-shot video data so as to generate a virtual image in the virtual scene and adjust the virtual scene angle in real time; The rendering and synthesis module is used to convert the matte image information and the adjusted virtual scene into a three-dimensional virtual reality mixed video stream according to the camera posture data, the corresponding three-dimensional coordinate data and the motion posture information, and send the three-dimensional virtual reality mixed video stream to the virtual camera and the display device respectively.

8. The mixed reality virtual pre-performance shooting system according to claim 7, characterized in that: The display device is used to receive and synchronously display the three-dimensional virtual reality mixed video stream, so that the target person can preview the three-dimensional virtual reality mixed video stream during the displacement of the video camera and the virtual camera, and adjust the shooting action and shooting angle of the object to be captured in real time.

9. The mixed reality virtual pre-performance shooting system according to claim 2, characterized in that: The optical positioning and motion capture system also includes a calibration device, which is used to calibrate the positions of the multiple optical motion capture cameras in the actual shooting scene through a calibration rod.

10. The mixed reality virtual pre-performance shooting system according to any one of claims 1 to 9, characterized in that: The mixed reality virtual rehearsal shooting system also includes a camera setting system, which is communicated with the central control system and is used to obtain site parameters in the actual shooting scene, and determine the number of cameras and the corresponding camera installation positions corresponding to the optical motion capture cameras according to the site parameters, so that each optical marker point is captured by any three optical motion capture cameras, and the site parameters include large space site length information and width information.

Images