Lightweight virtual manufacturing system

Abstract

The lightweight virtual production system comprises the following steps: step 1, in an early stage of a virtual production process, a virtual scene and a real scene are included, the designed real scene is submitted to a real scene making team to make the real scene, and the designed virtual scene is submitted to a virtual group to make the virtual scene; 2, in the shooting stage, a lightweight virtual shooting unit is adopted in the new scheme; and step 3, providing automatic sheet lens manufacturing in a later-stage manufacturing stage, and realizing automatic lens manufacturing by adopting a plurality of script tools. According to the method, after live-action scanning and collection, manual CG model manufacturing and surface material mapping processing are not needed any more, the 3D Gaussian virtual scene is obtained through training of collected data, the workload of one scene manufacturing is generally one fifth of that of a PBR technology, the manufacturing cost is one fifth of that of the PBR technology, and in other words, the original scene manufacturing cost is one hundred thousand yuan. According to the invention, only 20,000 yuan is needed, and the delivery time is shorter.

Description

Technical Field

[0001] The present invention belongs to the field of film and television production, and in particular relates to a lightweight virtual production system. Background Art

[0002] "Virtual Production" has been the hottest new technology in the film and television production industry over the past five years. Inspired by the success of numerous international projects, domestic film and television studios and major production companies have also begun experimenting with it. To utilize virtual production, a studio must be equipped with specialized virtual filming equipment. This type of studio is known as a virtual studio. Currently, the mainstream virtual studio is the LED virtual studio, equipped with large LED screens. Since the second half of 2020, dozens, if not hundreds, of LED virtual studios have been built across the country.

[0003] Despite being the hottest technology, its adoption and promotion have been slow. By 2024, these LED virtual studios will generally be well-received but not commercially successful, with adoption rates falling short of expectations. Some film and television projects that have experimented with LED virtual studios have found the overall costs to be high and below expectations, thus discouraging many more projects from pursuing them.

[0004] Our goal in developing a "lightweight virtual production system" is to address the challenges facing the industry in applying and promoting virtual production technology. This difficulty stems from three main factors:

[0005] 1) The cost of virtual art set design is high, usually 100,000 yuan per scene.

[0006] 2) The cost of using a virtual studio is high, usually 150,000 yuan per day for the rental of the studio and virtual shooting system, and 15,000 yuan per day for supporting service fees, which means that shooting in a virtual studio costs 165,000 yuan per day.

[0007] 3) Post-production costs: Some of the shots taken in the LED virtual studio still require post-processing, and the cost is 10,000 to 20,000 yuan per minute of film.

[0008] Crews that have experimented with virtual production have found that overall costs haven't been significantly reduced compared to traditional methods. Furthermore, the number of high-budget film and television projects starting production between 2023 and 2024 has significantly decreased, with most projects being low-budget, short-form dramas. This further undermines the use of LED virtual studios. Summary of the Invention

[0009] The present invention provides a lightweight virtual production system to solve the defects in the prior art.

[0010] The present invention is achieved through the following technical solutions:

[0011] The lightweight virtual production system includes the following steps:

[0012] Step 1: In the early stages of the virtual production process, the crew's art team will design all the scenes to be filmed, including virtual scenes and real scenes. The designed real scenes will be handed over to the real scene set design team for real scenes, and the designed virtual scenes will be handed over to the virtual team for virtual scene production;

[0013] Step 2: During the filming phase, the new solution uses a lightweight virtual photography unit. Lightweight means that the virtual photography unit's equipment and facilities have been significantly reduced. In terms of facilities, a green screen replaces a large LED screen as the background for virtual shooting. The combination of virtual and real can achieve a seamless connection between virtual scenes / virtual props and local real scenes / physical props, and seamless synthesis;

[0014] Step 3: Provide automated shot production in the post-production stage, using multiple scripting tools to achieve automated shot production.

[0015] In the lightweight virtual production system described above, the production of the virtual scene in step 1 adopts the PBR (physically based rendering) process in the existing industry process.

[0016] In the lightweight virtual production system described above, the specific operations of virtual scene production in step 1 are as follows:

[0017] Step (1): Scanning and collecting real scenes, using a variety of methods to scan and collect real scenes, including using LiDAR laser scanners for 3D laser scanning, using laser total stations for calibration of LiDAR data stitching; using SLR cameras to take photos of surface materials; using drones for high-angle scanning and photography;

[0018] Step (2): Post-processing of the scanned data: HDR synthesis, color adjustment, image alignment, ghost removal, and noise reduction of the photos; stitching of LiDAR scan data, and alignment of the photos with the camera pose;

[0019] Step (3): Use the data processed in step (2) to train the model and obtain a virtual scene.

[0020] In the lightweight virtual production system described above, the virtual photography unit in step 2 includes a camera tracking unit and an on-site synthesis preview unit.

[0021] In the lightweight virtual production system described above, the virtual photography unit in step 2 is divided into a virtual photography unit H suitable for large scenes and a virtual photography unit J suitable for small scenes according to the requirements of different film and television projects for the size of the studio space. When the scenes built for the shooting project include both large and small scenes, the virtual photography unit H and the virtual photography unit J are used together.

[0022] The lightweight virtual production system described above, in which the virtual photography unit H is described, is suitable for virtual shooting in large studios. Large studios are typically over 10 meters high, with shooting areas covering hundreds or even thousands of square meters. This requires the camera tracker to be able to maintain high-precision tracking within a large activity space. The Halide tracker is used in hardware, and the UE engine is used in conjunction with the VRPN software tool developed specifically for the Halide tracker as software.

[0023] The lightweight virtual production system described above, with its virtual photography unit J, is suitable for virtual filming projects in small studios. It utilizes a solution based on Apple's AR Kit technology, with the tracker hardware being a recent iPhone Pro (15 or later) or iPad Pro (with an M2 chip or later). The real-time compositing system hardware consists of these iOS devices and a backend PC workstation.

[0024] In the lightweight virtual production system described above, the hardware of the camera tracker also includes a video capture box and a time code synchronizer. The video capture box and the time code synchronizer are connected and fixed to the camera body together with the iPhone Pro to form a complete camera tracker system.

[0025] In the lightweight virtual production system described above, the virtual photography unit J can realize all conventional virtual shooting functions, including setting the virtual space coordinate system in the studio (setting the origin position and coordinate axis direction), on-site real-time preview of keying synthesis, setting and adjusting blue and green screen keying parameters, setting AI keying, and recording Take. In the lightweight virtual production system described above, the specific operations of step three are as follows:

[0026] Step 1: Convert camera footage to sequence frames. Automatically convert the native format footage (MXF, BRAW, Pro Res, MP4) on the camera's memory card to sequence frames in EXR or PNG format.

[0027] Step 2: Color space conversion: During the sequence frame conversion process, use the script tool to convert the color space of the material that needs color space conversion;

[0028] Step 3: Batch conforming of 2D and 3D data. Using scripting tools, using timecode, each frame's tracking data, lens calibration data, scene metadata, and live composite preview reference are automatically conformed.

[0029] Step 4: AI matting processing (AI mask production): Use three AI matting tools, MODNET, PPMatting, and INSPYRENET, to perform AI automatic matting on the image (generate mask sequence frames);

[0030] Step 5: Automated post-rendering of virtual assets. Through scripts and template tools, automatically set the parameters for post-rendering and directly send the rendering task to Blender or UE engine.

[0031] The advantages of the present invention are:

[0032] After real-scene scanning and acquisition, the present invention no longer requires manual CG model production and surface material mapping. Instead, the collected data is used to train and obtain a 3D Gaussian virtual scene. The workload of producing a scene is usually one-fifth of that of the PBR process, and the production cost is also one-fifth of the PBR process. That is, the production of a scene that originally cost 100,000 yuan can now be reduced to only 20,000 yuan with the present invention, and the delivery time is also faster.

[0033] This invention has made virtual photography a lightweight process, eliminating the use of large LED screens and returning to a virtual studio using green screens. The virtual shooting system consists of a green screen, camera tracking equipment, and real-time rendering and synthesis equipment. Without the asset-heavy equipment facilities such as large LED screens, the overall cost of the virtual shooting system equipment is greatly reduced. The total equipment cost of the green screen virtual studio is only 3 million yuan, and the rental fee for shooting is 30,000 yuan per day.

[0034] The present invention adopts multiple automated tools in the post-production stage of film-finishing lens production, saving manual workload and merging the manual production seats of several different stages in conventional post-production visual effects production into one manual seat, thereby saving the labor cost of post-production and saving a large amount of software and hardware equipment costs for the production company. The cost of post-production film-finishing production of the present invention can be reduced to less than 10,000 yuan per minute. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0036] Figure 1 is a flow chart of the present invention;

[0037] Figure 2 It is a comparison diagram of the processing diagram of the virtual scene production of the present invention and the processing diagram of the traditional method;

[0038] Figure 3 It is a flow chart of virtual scene production of the present invention;

[0039] Figure 4 is a schematic structural diagram of a virtual photography unit of the present invention;

[0040] Figure 5 is a device connection diagram of the virtual photography unit H of the present invention;

[0041] Figure 6 is a schematic structural diagram of the virtual photography unit J of the present invention;

[0042] Figure 7 It is a structural schematic diagram of a camera tracker of the present invention;

[0043] Figure 8 This is a schematic diagram of the AI ​​matting process in step three of the present invention;

[0044] Figure 9 is a schematic diagram of the post-automatic rendering of virtual assets in step three of the present invention;

[0045] Figure 10 This is a schematic diagram of automatically generating 2D and 3D fused shot synthesis in step three of the present invention. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0047] like Figure 1 As shown, the lightweight virtual production system includes the following steps:

[0048] Step 1: In the early stages of the virtual production process, the crew's art team will design all the scenes to be filmed, including virtual scenes and real scenes. The designed real scenes will be handed over to the real scene set design team for real scenes, and the designed virtual scenes will be handed over to the virtual team for virtual scene production;

[0049] Step 2: During the filming phase, the new solution uses a lightweight virtual photography unit. Lightweight means that the virtual photography unit's equipment and facilities have been significantly reduced. In terms of facilities, a green screen replaces a large LED screen as the background for virtual shooting. The combination of virtual and real can achieve a seamless connection between virtual scenes / virtual props and local real scenes / physical props, and seamless synthesis;

[0050] Step 3: Provide automated shot production in the post-production stage, using multiple scripting tools to achieve automated shot production.

[0051] Preferably, the production of the virtual scene in step 1 described in this embodiment adopts the PBR (physically based rendering) process in the existing industry process.

[0052] PBR technology is widely used in video game production, achieving photorealism. The creation of a scene requires a scanning team to scan and collect data from the scene prototype using a LiDAR laser scanner for 3D laser scanning and capture surface texture maps using drones and DSLR cameras. The scanned data is then post-processed by the game asset production department, with a game art team using 3D CG animation tools to create computer CG models and surface texture maps. Because virtual scenes for film and television production require a higher level of realism than typical game scenes, their production requires a significant amount of effort. The virtual scene on the left in the image below was created by our team, with a man-day workload of 30 people. The production team's costs were approximately 100,000 yuan.

[0053] The virtual scene production of the present invention greatly reduces the workload of manual labor. Figure 2 As shown, taking the virtual scene on the right as an example, this scene is more complex than the one on the left (including interior scenes) and more realistic, but it only requires 6 people / day and costs 20,000 yuan, which is one-fifth of the traditional method.

[0054] like Figure 3 As shown, preferably, the specific operations of creating the virtual scene in step 1 of this embodiment are:

[0055] Step (1): Scanning and collecting real scenes, using a variety of methods to scan and collect real scenes, including using LiDAR laser scanners for 3D laser scanning, using laser total stations for calibration of LiDAR data stitching; using SLR cameras to take photos of surface materials; using drones for high-angle scanning and photography;

[0056] Step (2): Post-processing of the scanned data: HDR synthesis, color adjustment, image alignment, ghost removal, and noise reduction of the photos; stitching of LiDAR scan data, and alignment of the photos with the camera pose;

[0057] Step (3): Use the data processed in step (2) to train the model and obtain a virtual scene.

[0058] like Figure 4 As shown, preferably, the virtual photography unit in step 2 of this embodiment includes a camera tracking unit and a scene synthesis preview unit.

[0059] Preferably, the virtual photography unit in step 2 described in this embodiment is divided into a virtual photography unit H suitable for large scenes and a virtual photography unit J suitable for small scenes according to the requirements of different film and television projects for the size of the studio space. When the scenes built for the shooting project include both large scenes and small scenes, the virtual photography unit H and the virtual photography unit J are used together.

[0060] Preferably, the virtual photography unit H described in this embodiment is suitable for virtual shooting in large studios. The height of a large studio is more than 10 meters, and the shooting area in the studio is several hundred square meters or even thousands of square meters. The camera tracker needs to be able to maintain high-precision tracking in a large activity space. In terms of hardware, the Halide tracker is used, and the software uses the UE engine combined with the VRPN software tool we developed specifically for the Halide tracker. Various mainstream brands and models of movie cameras can be used for virtual shooting. The virtual shooting system does not include movie cameras, which need to be prepared by the user. The device connection diagram is shown in the figure below. Figure 5 shown.

[0061] like Figure 6 As shown, the virtual photography unit J described in this embodiment is preferably suitable for virtual photography projects in small studios. A solution based on Apple's AR Kit technology is used, where the tracker hardware is a recent iPhone Pro (15 or later) or iPad Pro (M2 chip or later). The hardware of the on-site real-time synthesis system consists of the above iOS devices and a backend PC workstation.

[0062] like Figure 7 As shown, preferably, the hardware of the camera tracker described in this embodiment also includes a video capture box and a time code synchronizer. The video capture box and the time code synchronizer are connected and fixed to the camera body together with the iPhone Pro to form a complete camera tracker system.

[0063] Preferably, the virtual photography unit J described in this embodiment can implement all conventional virtual photography functions, including setting the virtual space coordinate system within the studio (setting the origin position and coordinate axis direction), on-site real-time preview of keying synthesis, setting and adjusting blue and green screen keying parameters, setting AI keying, and recording Take. The lightweight virtual photography system can also implement advanced virtual photography functions, including lens optical calibration and on-site geometric survey, for more sophisticated post-processing and improved synthesis quality of the finished film.

[0064] Preferably, the specific operations of step three described in this embodiment are:

[0065] Step 1: Convert camera footage to sequence frames. Automatically convert the native format footage (MXF, BRAW, Pro Res, MP4) on the camera's memory card to sequence frames in EXR or PNG format.

[0066] Step 2: Color space conversion: During the sequence frame conversion process, use the script tool to convert the color space of the material that needs color space conversion;

[0067] Step 3: Batch conforming of 2D and 3D data. Using scripting tools, using timecode, each frame's tracking data, lens calibration data, scene metadata, and live composite preview reference are automatically conformed.

[0068] Step 4: AI matting processing (AI mask production): Use three AI matting tools, MODNET, PPMatting, and INSPYRENET, to perform AI automatic matting on the image (generate mask sequence frames), such as Figure 8 As shown;

[0069] Step 5: Automated post-rendering of virtual assets. Use scripts and template tools to automatically set the parameters for post-rendering and directly send rendering tasks to Blender or UE engines. Figure 9 As shown;

[0070] Preferably, step three of this embodiment can automatically generate a 2D and 3D fusion lens synthesis engineering file, and can adjust the three-dimensional virtual assets in the 2D+3D synthesis environment to achieve the best synthesis perspective effect with the 2D real-shot layer, such as Figure 10 As shown, you can adjust the Z depth of the real-life layer in the composite environment and adjust the camera tracking trajectory.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. Lightweight virtual production system, characterized by: The steps include: Step 1: In the early stages of the virtual production process, the crew's art team will design all the scenes to be filmed, including virtual scenes and real scenes. The designed real scenes will be handed over to the real scene set design team for real scenes, and the designed virtual scenes will be handed over to the virtual team for virtual scene production; Step 2: During the filming phase, the new solution uses a lightweight virtual photography unit. Lightweight means that the virtual photography unit's equipment and facilities have been significantly reduced. In terms of facilities, a green screen replaces a large LED screen as the background for virtual shooting. The combination of virtual and real can achieve a seamless connection between virtual scenes / virtual props and local real scenes / physical props, and seamless synthesis; Step 3: Provide automated shot production in the post-production stage, using multiple scripting tools to achieve automated shot production.

2. The lightweight virtual production system according to claim 1, characterized in that: In the existing industry process, the production of the virtual scene in step 1 adopts the PBR (physically based rendering) process.

3. The lightweight virtual production system according to claim 1, characterized in that: The specific operations of the virtual scene production in step 1 are as follows: Step (1): Scanning and collecting real scenes, using a variety of methods to scan and collect real scenes, including using LiDAR laser scanners for 3D laser scanning, using laser total stations for calibration of LiDAR data stitching; using SLR cameras to take photos of surface materials; using drones for high-angle scanning and photography; Step (2): Post-processing of the scanned data: HDR synthesis, color adjustment, image alignment, ghost removal, and noise reduction of the photos; stitching of LiDAR scan data, and alignment of the photos with the camera pose; Step (3): Use the data processed in step (2) to train the model and obtain a virtual scene.

4. The lightweight virtual production system according to claim 1, characterized in that: The virtual photography unit in step 2 includes a camera tracking unit and an on-site synthesis preview unit.

5. The lightweight virtual production system according to claim 1, characterized in that: In the step 2, the virtual photography unit is divided into a virtual photography unit H suitable for large scenes and a virtual photography unit J suitable for small scenes according to the requirements of different film and television projects for the size of the studio space. When the scenes built for the shooting project include both large and small scenes, the virtual photography unit H and the virtual photography unit J are used together.

6. The lightweight virtual production system according to claim 5, characterized in that: The virtual photography unit H is suitable for virtual shooting in large studios. It requires the camera tracker to maintain high-precision tracking in a large activity space. The hardware uses the Halide tracker, and the software uses the UE engine combined with the VRPN software tool we developed specifically for the Halide tracker.

7. The lightweight virtual production system according to claim 5, characterized in that: The virtual photography unit J is suitable for virtual photography projects in small studios. It uses a solution based on Apple's AR Kit technology. The tracker hardware is a recent iPhone Pro (15 or later) or iPad Pro (with an M2 chip or later). The real-time compositing system hardware consists of these iOS devices and a backend PC workstation.

8. The lightweight virtual production system according to claim 7, characterized in that: The hardware of the camera tracker also includes a video capture box and a time code synchronizer. The video capture box and the time code synchronizer are connected and fixed to the camera body together with the iPhone Pro to form a complete camera tracker system.

9. The lightweight virtual production system according to claim 7, characterized in that: The virtual photography unit J can realize all conventional virtual shooting functions, including setting the coordinate system of the virtual space in the studio, on-site real-time preview of the matte synthesis, setting and adjusting the blue and green screen matte parameters, setting AI matte, and recording Take.

10. The lightweight virtual production system according to claim 1, characterized in that: The specific operations of step three are: Step 1: Convert camera footage to sequence frames. Automatically convert the native format footage on the camera's memory card to sequence frames in EXR or PNG format. Step 2: Color space conversion: During the sequence frame conversion process, use the script tool to convert the color space of the material that needs color space conversion; Step 3: Batch conforming of 2D and 3D data. Using scripting tools, using timecode, each frame's tracking data, lens calibration data, scene metadata, and live composite preview reference are automatically conformed. Step 4: AI matting processing: Use three AI matting tools, MODNET, PPMatting, and INSPYRENET, to perform AI automatic matting on the image; Step 5: Automated post-rendering of virtual assets. Through scripts and template tools, automatically set the parameters for post-rendering and directly send the rendering task to Blender or UE engine.

Images