A method for constructing a film and television scene based on virtual reality

Abstract

The present invention belongs to the field of scene construction, and specifically refers to a method for constructing a film and television scene based on virtual reality. The method includes scene creation, distributed computing, device addition, physical simulation, state optimization, graphics rendering, and preview playback. This solution creatively proposes a distributed simulation rendering method, which speeds up the operation speed of graphics rendering through physical engine decoupling; creatively adopts a rendering method combining point-by-point and surface-by-surface, reducing the film and television production cost and providing greater creative freedom; creatively adopts a lens state generation method that combines the optimization of the lens state and the virtual camera motion state. Through hierarchical finite state machine and natural language processing technologies, it identifies the areas in the virtual film and television scene that are prone to VR motion sickness, reduces the discomfort of users, enhances the user experience, and realizes the facilitation of virtual scene creation work.

Description

Technical Field

[0001] The present invention relates to the field of scene construction, and specifically refers to a method for constructing a film and television scene based on virtual reality. Background Art

[0002] The method for constructing a scene based on virtual reality refers to using virtual reality technology to create and simulate a realistic three-dimensional film and television scene, enabling producers to freely design, adjust, and experience the scene in a virtual environment. This method can significantly improve the efficiency and creative space of film and television production, reduce the cost of live shooting, and at the same time provide an immersive visual experience for the audience.

[0003] In existing similar solutions, for example, in CN117456130A, a method for constructing a scene model, this method focuses on the realism and better display effect after the construction of the scene model. However, the scene modeling with high precision inevitably has the problem that the large amount of calculation leads to high difficulty in real-time rendering.

[0004] Distributed computing provides good technical support for solving the problem of high difficulty in real-time rendering. However, in existing similar solutions applying distributed computing, for example, in CN104183023A, a method for constructing multiple scene graphs in a distributed virtual environment, existing solutions mostly focus on the classification creation and organizational management optimization of multiple scene graphs in the virtual scene. Although this optimizes the efficiency of real-time modeling, it also sacrifices the interactive experience of scene construction. The virtual environment with fast rendering may instead interfere with the user's viewing point, thereby bringing an unsatisfactory user experience.

[0005] Therefore, the existing methods for constructing virtual reality film and television scenes often have technical problems such as insufficient scene realism, lack of interactivity, and complex construction processes; the traditional shadow rendering method realizes the way of calculating the shadow on the surface of each object by simulating the propagation process of light in the scene, and has the technical problem of large calculation amount and difficulty in real-time rendering; there is a technical problem that the three-dimensional virtual environment interferes with the user's viewing point, easily causes VR motion sickness, and may lead to nausea, discomfort, and disorientation of the user. Summary of the Invention

[0006] In view of the above situation, to overcome the defects of the prior art, the present invention provides a method for constructing a film and television scene based on virtual reality. Aiming at the technical problems that the existing methods for constructing virtual reality film and television scenes often have a complex scene construction process and a large amount of rendering calculations, this solution creatively proposes a distributed simulation rendering method, which uses a game engine for scene simulation and rendering. Through physical engine decoupling, complex tasks with large calculations are handed over to the edge server to be realized, reducing the computing power cost of the preview monitor device on the user side in physical simulation and accelerating the operation speed of graphics rendering. Aiming at the technical problem that the traditional shadow rendering method has a large difficulty in real-time rendering by simulating the propagation process of light in the scene to calculate the shadow on each object surface, this solution creatively adopts a rendering method combining point-by-point and surface-by-surface. According to the color, surface material, texture and illumination of the object itself, referring to the relative position and motion state of the virtual object and the virtual camera, the scene is colored and rendered in the order of first vertex by vertex and then surface by surface, reducing the film and television production cost. The scene can be adjusted at any time according to the film and television requirements, providing greater creative freedom, and having the characteristics of high efficiency, strong immersion, low cost and good flexibility, with high application prospects and market value. Aiming at the technical problem that the three-dimensional virtual environment interferes with the user's viewing point, easily causes VR motion sickness, and may cause the user to feel nauseous, uncomfortable and disoriented, this solution creatively adopts a lens state generation method optimized by combining the lens state and the motion state of the virtual camera. Through a hierarchical finite state machine (HFSM, Hierarchical Finite State Machine) and natural language processing technology, the lens state is initialized, the motion state of the virtual camera is estimated, the areas in the virtual film and television scene that are likely to cause VR motion sickness are identified, and a heuristic method is used to adjust the movement path of the virtual camera in the virtual film and television scene, optimize the picture quality, reduce the discomfort of the user, enhance the user experience, and realize the simplification of virtual scene creation work;

[0007] The technical solution adopted by the present invention is as follows: A method for constructing a film and television scene based on virtual reality provided by the present invention includes the following steps:

[0008] Step S1: Scene creation, which is used to create a virtual film and television scene. Specifically, it is to collect real physical environment data, construct a physical scene model, adjust the details of the physical scene model according to the film and television production requirements, deploy virtual objects on the physical scene model, and create a virtual film and television scene;

[0009] Step S2: Distributed calculation, which is used to implement complex tasks with large calculations through the edge server. Specifically, a game engine is used to perform graphics rendering and physical simulation operations on the virtual film and television scene, and the green screen technology is used to separate the actor from the green screen background and replace the green screen background with the virtual film and television scene;

[0010] Step S3: Device addition, which is used to build the basic hardware conditions for physical simulation and graphics rendering. Specifically, a preview monitor is adopted to display the shooting effect in real time, and a virtual camera is used to simulate the shooting effect of a real camera in a virtual film and television scene. During the real-time display and simulation of the shooting effect, one preview monitor is responsible for one virtual camera, and one virtual camera shoots one perspective of the virtual film and television scene;

[0011] Step S4: Physical simulation, which is used to simulate physical phenomena in virtual reality without performing graphics rendering. Specifically, a physical engine is adopted to simulate physical phenomena caused by collision, gravity, and friction in the virtual film and television scene, and collision detection is performed;

[0012] Step S5: State optimization, which is used to build the shooting method of the virtual camera. Specifically, through a hierarchical finite state machine, the lens state of the virtual camera is initialized, the motion state of the virtual camera is calculated, and a heuristic method is adopted to optimize the lens state and fine-tune the actor's position;

[0013] Step S6: Graphics rendering, which is used to render the film and television scene in real time. Specifically, a graphics pipeline is adopted to centrally host the computing tasks of drawing virtual objects on an edge server, and the computing results are sent to the preview monitor;

[0014] The computing results specifically refer to the virtual film and television scene after drawing virtual objects;

[0015] The edge server and the preview monitor are specifically in a one-to-many relationship, and the preview monitors in the same virtual film and television scene will share the same computing results from the edge server with each other;

[0016] Step S7: Preview playback, which is used to preview the computing results. Specifically, a graphics pipeline deployed on the preview monitor receives the computing results from the edge server, fuses the actor and the computing results to obtain the final fused image, and converts the final fused image into a 2D image for real-time display on the preview monitor.

[0017] Further, in Step S1, the collection of the scene model specifically includes the following steps:

[0018] Step S11: Preprocessing, collecting physical scene images. Specifically, the resolution of the physical scene images is adjusted, color correction and denoising processing are performed, the physical scene images are converted into physical three-dimensional scenes, and the physical three-dimensional scenes are imported into VR scene construction software;

[0019] Step S12: Generation of virtual 3D scene. Specifically, use 3D modeling tools to optimize the physical 3D scene, arrange corresponding virtual objects according to film and television requirements to generate a virtual 3D scene, and adopt a generative adversarial network to enhance the realism of the virtual 3D scene to generate a virtual film and television scene. The virtual objects include buildings, props, and background characters.

[0020] Further, in step S2, the distributed computing specifically includes the following steps:

[0021] Step S21: Separation of the physics engine. Specifically, decouple the physics engine from the game engine, use Docker containerization technology to encapsulate the physics engine into a Docker container image, split the game engine into a graphics pipeline and a physics engine, deploy the graphics pipeline to run independently on a preview monitor, and deploy the physics engine to run independently on an edge server through the Docker container image.

[0022] Step S22: Arrangement of edge servers. Specifically, arrange edge servers with low transmission latency at the edge position close to the preview monitor in the transmission network. One edge server deploying the physics engine serves multiple preview monitors in the same virtual film and television scene.

[0023] Step S23: Interface design. Specifically, design an interface for communication between the graphics pipeline and the physics engine, and achieve data synchronization between the graphics pipeline and the physics engine through the interface.

[0024] Step S24: Network communication synchronization. Specifically, use sockets for network communication between the preview monitor and the edge server.

[0025] Further, in step S4, the physical simulation specifically includes the following steps:

[0026] Step S41: Simulation limb generation. Specifically, model the limbs of the actor, and through the modeling, virtually simulate the limbs to obtain actor virtual limb data, which is used for collision detection.

[0027] Step S42: Collision detection. Specifically, when the virtually simulated limb enters the collision space range of the virtual object in the virtual film and television scene, it is detected that the two collide, and the virtually simulated limb and the virtual object are mutually misaligned to eliminate penetration.

[0028] Step S43: Consistent display. Specifically, the edge server equipped with the physics engine executes steps S41 to S42. The physics engine updates the position and actions of the virtually simulated limb, and transmits the results of the physical simulation to all preview monitors in the same virtual film and television scene in a streaming manner in real time. The preview monitor displays the results of the physical simulation of the actor and the virtual film and television scene.

[0029] Further, in step S5, the state optimization specifically includes the following steps:

[0030] Step S51: Disassemble the lens, which is used to initialize the lens state of the virtual camera. Specifically, a hierarchical finite state machine is adopted to decompose the dialogue scene to obtain the lens state of the virtual camera. Disassembling the lens includes the following steps:

[0031] Step S511: Analyze the film and television script using natural language processing algorithms to obtain a condition list;

[0032] The condition list specifically refers to the conditions for virtual film and television scene conversion and lens state conversion;

[0033] The lens state specifically refers to the type, position, and attitude of the lens;

[0034] Step S512: The hierarchical finite state machine receives the condition list, decomposes the complex dialogue scene between actors into multiple levels, with one level corresponding to processing one virtual film and television scene. According to the condition list, it triggers the lens state conversion, and decomposes the complex dialogue scene between actors in one virtual film and television scene into multiple lens states;

[0035] Step S52: Calculate the motion state of the virtual camera, which specifically includes the following steps:

[0036] Step S521: Calculate the relative camera pose matrix. Specifically, perform matrix multiplication on the OpenGL model view matrix of the previous frame and the camera pose matrix of the current frame to calculate the relative camera pose matrix. The formula used is as follows:

[0037] ;

[0038] In the formula, represents the relative camera pose matrix, is the OpenGL model view matrix of the previous frame, is the camera pose matrix of the current frame;

[0039] Step S522: Angular velocity conversion. Specifically, convert the angular velocity of the virtual camera into frame rate representation. The formula used is as follows:

[0040] ;

[0041] In the formula, represents the frame rate, represents the time required for the virtual camera to rotate one week, represents the angular velocity;

[0042] Step S523: Linear velocity calculation, specifically, extract the displacements of the virtual camera along the X, Y, and Z axes from the relative camera pose matrix respectively, multiply the displacements of the X, Y, and Z axes by the frame rate respectively to obtain the linear velocity of each frame of the virtual camera along the X axis, Y axis, and Z axis. The formula used is as follows:

[0043] ;

[0044] ;

[0045] ;

[0046] In the formula, , , represent the linear velocity of each frame of the virtual camera along the X, Y, and Z axes respectively, , , represent the displacements of the virtual camera along the X, Y, and Z axes respectively, represents the frame rate;

[0047] Step S524: Extract the change amounts, specifically, calculate the yaw angle, pitch angle, and roll angle in the order of the Y axis, X axis, and Z axis. The yaw angle, pitch angle, and roll angle are the angles describing the rotation of the virtual camera around the Y, X, and Z axes respectively;

[0048] Extract the yaw angle change amount, pitch angle change amount, and roll angle change amount of the virtual camera from the relative camera pose matrix respectively, multiply them by the frame rate respectively to obtain the yaw angular velocity, pitch angular velocity, and roll angular velocity. The formula used is as follows:

[0049] ;

[0050] ;

[0051] ;

[0052] In the formula, , , represent the yaw angle change amount, pitch angle change amount, and roll angle change amount of each frame respectively, , , represent the yaw angular velocity, pitch angular velocity, and roll angular velocity of the camera per second respectively, represents the frame rate;

[0053] Step S53: Optimize the lens state, specifically, refer to the line of interest, adopt a heuristic method to optimize the lens state and fine-tune the actor's position. Among them, the constraints of the heuristic method include smooth movement, picture occlusion, and picture clarity;

[0054] The line of interest specifically refers to the line connecting the actors, the line along the direction of the actors' movements, and the line pointing in the direction the actors are facing.

[0055] The smooth movement is used to maintain the coherence between consecutive shots and avoid camera shakes and jump cuts. Specifically, it refers to detecting whether the movement state of the virtual camera is consistent with the direction of the line of interest.

[0056] The scene occlusion is used to prevent actors from blocking each other. Specifically, it refers to using frame edge detection to count the occluded actors.

[0057] The clear image is used to avoid image blurring caused by high-speed movement. Specifically, it refers to detecting and controlling the movement state of the virtual camera.

[0058] Step S54: Visualize the movement state. Specifically, color is applied according to the values of linear velocity, yaw angular velocity, pitch angular velocity, and roll angular velocity to visualize the movement state of the virtual camera.

[0059] Step S55: Optimize the movement path. Specifically, according to the movement state of the virtual camera, in the artificially recognized areas in the virtual film and television scene that are prone to causing VR motion sickness, optimize the movement path of the virtual camera in the virtual film and television scene.

[0060] Furthermore, in step S6, the computing task of rendering the virtual object specifically includes the following steps:

[0061] Step S61: Process each vertex. The specific operations are as follows:

[0062] Step S611: Calculate the relative position of each vertex of the virtual object on the screen of the virtual camera and the normal direction of the surface where the vertex is located.

[0063] Step S612: Calculate the color A of each vertex according to the movement state of the virtual camera and the distance from the vertex to the virtual camera.

[0064] Step S62: Process each surface. Specifically, collect the color B representing the original color and surface material texture of the virtual object itself.

[0065] Step S63: Extract the grayscale value. Specifically, render the virtual film and television scene multiple times under the lighting conditions at different time points and output the grayscale C.

[0066] Step S64: Color mixing. Specifically, mix color A, color B, and grayscale C in a weighted multiplication manner to output the final color of each pixel of the virtual object.

[0067] The present invention provides a method for constructing a film and television scene based on virtual reality. The beneficial effects achieved by the present invention using the above solution are as follows:

[0068] (1) Aiming at the technical problems that the existing methods for constructing virtual reality film and television scenes often have a complex scene construction process and a large amount of rendering calculations, this solution creatively proposes a distributed simulation rendering method. The game engine is used for scene simulation and rendering. Through physical engine decoupling, complex tasks with large calculations are handed over to the edge server to achieve, reducing the computing power cost of the preview monitor device on the user side in physical simulation and accelerating the operation speed of graphics rendering;

[0069] (2) Aiming at the technical problem that the traditional shadow rendering method has a large difficulty in real-time rendering by simulating the propagation process of light in the scene to calculate the shadow on each object surface, this solution creatively adopts a rendering method combining point-by-point and surface-by-surface. According to the color, surface material, texture and light of the object itself, referring to the relative position and motion state of the virtual object and the virtual camera, the scene is colored and rendered in the order of first vertex by vertex and then surface by surface, reducing the film and television production cost. The scene can be adjusted at any time according to the film and television requirements, providing greater creative freedom, and having the characteristics of high efficiency, strong immersion, low cost and good flexibility, with high application prospects and market value;

[0070] (3) Aiming at the technical problem that the three-dimensional virtual environment interferes with the user's viewing point, easily causes VR motion sickness, and may cause the user to feel nauseous, uncomfortable and disoriented, this solution creatively adopts a lens state generation method optimized by combining the lens state and the motion state of the virtual camera. Through the hierarchical finite state machine (HFSM, Hierarchical Finite State Machine) and natural language processing technology, the lens state is initialized, the motion state of the virtual camera is estimated, the areas in the virtual film and television scene that are likely to cause VR motion sickness are identified, and a heuristic method is used to adjust the moving path of the virtual camera in the virtual film and television scene, optimize the picture quality, reduce the discomfort of the user, enhance the user experience, and realize the simplification of virtual scene creation work. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a schematic flowchart of a method for constructing a film and television scene based on virtual reality provided by the present invention.

[0072] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention, and do not constitute a limitation to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0073] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0074] Embodiment 1. Refer to Figure 1 , a method for constructing a film and television scene based on virtual reality provided by the present invention, the method includes the following steps:

[0075] Step S1: Scene creation, which is used to create a virtual film and television scene. Specifically, it is to collect real physical environment data, construct a physical scene model, adjust the details of the physical scene model according to the requirements of film and television production, deploy virtual objects on the physical scene model, and create a virtual film and television scene;

[0076] Step S2: Distributed computing, which is used to implement complex tasks with large amounts of computation through edge servers. Specifically, it is to use a game engine to perform graphics rendering and physical simulation operations on the virtual film and television scene, and use green screen technology to separate the actor from the green screen background and replace the green screen background with the virtual film and television scene;

[0077] Step S3: Device addition, which is used to construct the basic hardware conditions for physical simulation and graphics rendering. Specifically, it is to use a preview monitor to display the shooting effect in real time, and use a virtual camera to simulate the shooting effect of a real camera in the virtual film and television scene. Among them, one preview monitor is responsible for one virtual camera, and one virtual camera shoots one perspective of the virtual film and television scene;

[0078] Step S4: Physical simulation, which is used to simulate physical phenomena in virtual reality without performing graphics rendering. Specifically, it is to use a physics engine to simulate physical phenomena caused by collision, gravity, and friction in the virtual film and television scene and perform collision detection;

[0079] Step S5: State optimization, which is used to construct the shooting method of the virtual camera. Specifically, it is to initialize the lens state of the virtual camera through a hierarchical finite state machine, calculate the motion state of the virtual camera, use a heuristic method to optimize the lens state, and fine-tune the actor's position;

[0080] Step S6: Graphics rendering, which is used to render the film and television scene in real time. Specifically, it is to use a graphics pipeline to centrally host the computing tasks of drawing virtual objects on an edge server and send the calculation results to the preview monitor;

[0081] The calculation result specifically refers to the virtual film and television scene after drawing virtual objects;

[0082] The edge server and the preview monitor are in a one-to-many relationship specifically. Preview monitors in the same virtual film and television scene will share the same calculation results from the edge server with each other;

[0083] Step S7: Preview playback, which is used to preview the calculation results. Specifically, it uses the graphics pipeline deployed on the preview monitor to receive the calculation results from the edge server, fuse the actors and the calculation results to obtain the final fused image, and convert the final fused image into a 2D image for real-time display on the preview monitor.

[0084] Embodiment 2, refer to Figure 1 , based on the above embodiment, in step S1, the collection of the scene model specifically includes the following steps:

[0085] Step S11: Preprocessing, collecting physical scene images. Specifically, it adjusts the resolution of the physical scene images, performs color correction and denoising processing, converts the physical scene images into physical three-dimensional scenes, and imports the physical three-dimensional scenes into VR scene construction software;

[0086] Step S12: Generation of virtual three-dimensional scenes. Specifically, it uses 3D modeling tools to optimize the physical three-dimensional scenes, arranges corresponding virtual objects according to film and television requirements to generate virtual three-dimensional scenes, and uses generative adversarial networks to enhance the realism of the virtual three-dimensional scenes to generate virtual film and television scenes. The virtual objects include buildings, props, and background characters.

[0087] Embodiment 3, refer to Figure 1 , based on the above embodiment, in step S2, the distributed computing specifically includes the following steps:

[0088] In step S2, the distributed computing specifically includes the following steps:

[0089] Step S21: Separating the physics engine. Specifically, it decouples the physics engine from the game engine, uses Docker containerization technology to package the physics engine into a Docker container image, splits the game engine into a graphics pipeline and a physics engine, deploys the graphics pipeline on the preview monitor to run independently, and deploys the physics engine on the edge server to run independently through the Docker container image;

[0090] Step S22: Arranging edge servers. Specifically, it arranges edge servers with low transmission latency at the edge positions in the transmission network close to the preview monitors. One edge server deploying the physics engine serves multiple preview monitors in the same virtual film and television scene;

[0091] Step S23: Interface design, specifically for designing the interface for communication between the graphics pipeline and the physics engine, and realizing data synchronization between the graphics pipeline and the physics engine through the interface;

[0092] Step S24: Network communication synchronization, specifically for using sockets for network communication between the preview monitor and the edge server.

[0093] Example 4, refer to Figure 1 , based on the above example, in step S4, the physical simulation specifically includes the following steps:

[0094] Step S41: Simulation limb generation, specifically for modeling the limbs of the actor, creating virtual simulation limbs based on the modeling results. The virtual simulation limbs will serve as the actor's avatar to detect whether a collision occurs between the actor and the virtual object;

[0095] Step S42: Collision detection, specifically for detecting a collision when the virtual simulation limb enters the collision space range of the virtual object in the virtual film and television scene, and mutually offsetting and eliminating penetration between the virtual simulation limb and the virtual object;

[0096] Step S43: Consistent display, specifically for the edge server equipped with the physics engine to execute steps S41 to S42. The physics engine updates the position and actions of the virtual simulation limbs, and transmits the results of the physical simulation to all preview monitors in the same virtual film and television scene in a streaming manner in real time. The preview monitors display the results of the physical simulation of the actor and the virtual film and television scene.

[0097] Example 5, refer to Figure 1 , based on the above example, in step S5, the state optimization specifically includes the following steps:

[0098] Step S51: Disassembling the shot, used to initialize the lens state of the virtual camera. Specifically, a hierarchical finite state machine is used to decompose the dialogue scene to obtain the lens state of the virtual camera. The steps of disassembling the shot include the following:

[0099] Step S511: Using natural language processing algorithms to analyze the film and television script to obtain a list of conditions;

[0100] The list of conditions specifically refers to the conditions for virtual film and television scene conversion and lens state conversion;

[0101] The lens state specifically refers to the type, position, and attitude of the lens;

[0102] Step S512: The hierarchical finite state machine receives the condition list, decomposes the complex dialogue scenarios among the actors into multiple levels, with one level corresponding to processing one virtual film and television scene. According to the condition list, it triggers the lens state conversion, and decomposes the complex dialogue scenarios among the actors in one virtual film and television scene into multiple lens states;

[0103] Step S52: Calculate the motion state of the virtual camera, which specifically includes the following steps:

[0104] Step S521: Calculate the relative camera pose matrix. Specifically, perform matrix multiplication on the OpenGL model view matrix of the previous frame and the camera pose matrix of the current frame to calculate the relative camera pose matrix. The formula used is as follows:

[0105] ;

[0106] In the formula, represents the relative camera pose matrix, is the OpenGL model view matrix of the previous frame, is the camera pose matrix of the current frame;

[0107] Step S522: Angular velocity conversion. Specifically, convert the angular velocity of the virtual camera into frame rate representation. The formula used is as follows:

[0108] ;

[0109] In the formula, represents the frame rate, represents the time required for the virtual camera to rotate one week, represents the angular velocity;

[0110] Step S523: Linear velocity calculation. Specifically, extract the displacements of the virtual camera along the X, Y, and Z axes from the relative camera pose matrix respectively, multiply the displacements of the X, Y, and Z axes by the frame rate respectively, and obtain the linear velocities of the virtual camera along the X, Y, and Z axes for each frame. The formula used is as follows:

[0111] ;

[0112] ;

[0113] ;

[0114] In the formula, , , respectively represent the linear velocities of the virtual camera along the X, Y, and Z axes for each frame, , , respectively represent the displacements of the virtual camera along the X, Y, and Z axes, represents the frame rate;

[0115] Step S524: Extract the change amounts, specifically calculate the yaw angle, pitch angle, and roll angle in the order of the Y-axis, X-axis, and Z-axis. The yaw angle, pitch angle, and roll angle are respectively the angles describing the rotation of the virtual camera around the Y, X, and Z axes;

[0116] Extract the yaw angle change amount, pitch angle change amount, and roll angle change amount of the virtual camera from the relative camera pose matrix respectively, and multiply them by the frame rate to obtain the yaw angular velocity, pitch angular velocity, and roll angular velocity. The used formulas are as follows:

[0117] ;

[0118] ;

[0119] ;

[0120] In the formula, , , respectively represent the yaw angle change amount, pitch angle change amount, and roll angle change amount per frame, , , respectively represent the yaw angular velocity, pitch angular velocity, and roll angular velocity of the camera per second, represents the frame rate;

[0121] Step S53: Optimize the lens state, specifically refer to the line of interest, and use a heuristic method to optimize the lens state and fine-tune the actor's position. Among them, the constraints of the heuristic method include smooth movement, picture occlusion, and picture clarity;

[0122] The line of interest specifically refers to the line connecting the actors, the line along the direction of the actor's movement, and the line pointing to the direction the actor is facing;

[0123] The smooth movement is used to maintain the coherence between consecutive shots and avoid camera shakes and jump cuts. Specifically, it refers to detecting whether the movement state of the virtual camera is consistent with the direction of the line of interest;

[0124] The picture occlusion is used to avoid actors blocking each other. Specifically, it refers to using frame edge detection to count the occluded actors;

[0125] The picture clarity is used to avoid picture blurring caused by high-speed movement. Specifically, it refers to detecting and controlling the movement state of the virtual camera;

[0126] Step S54: Visualize the motion state, specifically, color according to the values of linear velocity, yaw angular velocity, pitch angular velocity, and roll angular velocity to visualize the motion state of the virtual camera;

[0127] Step S55: Optimize the movement path, specifically, according to the motion state of the virtual camera, manually identify the areas in the virtual film and television scene that are prone to causing VR motion sickness, and optimize the movement path of the virtual camera in the virtual film and television scene.

[0128] Example Six, refer to Figure 1 , based on the above example, in step S6, the computing task of rendering the virtual object specifically includes the following steps:

[0129] Step S61: Process vertex by vertex, and the specific operations are as follows:

[0130] Step S611: Calculate the relative position of each vertex of the virtual object on the screen of the virtual camera and the normal direction of the surface where the vertex is located;

[0131] Step S612: Calculate the color A of each vertex according to the motion state of the virtual camera and the distance from the vertex to the virtual camera;

[0132] Step S62: Process surface by surface, specifically, collect the color B representing the original color and surface material texture of the virtual object itself;

[0133] Step S63: Extract the grayscale value, specifically, render the virtual film and television scene multiple times under the illumination conditions at different time points, and output the grayscale C;

[0134] Step S64: Color mixing, specifically, mix color A, color B, and grayscale C in a weighted multiplication manner to output the final color of each pixel of the virtual object.

[0135] Example Seven, refer to Figure 1 , based on the above example, in step S12, the 3D modeling tool uses Blender 3D modeling software.

[0136] Example Eight, refer to Figure 1 , based on the above example, in steps S612 and S64, set the color according to the relative speed between the virtual camera and the virtual object. The faster the virtual object moves, the larger the value of the R channel (red) of the pixel, and the slower the virtual object moves, the larger the value of the B channel (blue) of the pixel.

[0137] Example Nine, refer to Figure 1, based on the above embodiment, in step S63, the different time points specifically refer to different moments of a day (morning, noon, and evening), rendering the parts in the virtual film and television scene that are often irradiated by sunlight into warm colors and the parts covered by shadows into cold colors.

[0138] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device.

[0139] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

[0140] The above describes the present invention and its implementation manners. Such description is not restrictive. What is shown in the drawings is only one of the implementation manners of the present invention, and the actual structure is not limited thereto. Generally speaking, if those of ordinary skill in the art are inspired by it and design similar structural manners and embodiments without creative efforts without departing from the purpose of the present invention's creation, they shall fall within the protection scope of the present invention.

Claims

1. A method for constructing a film and television scene based on virtual reality, characterized in that: The method includes the following steps: Step S1: Scene creation, which is used to create a virtual film and television scene. Specifically, it involves collecting real physical environment data, constructing a physical scene model, adjusting the details of the physical scene model according to film and television production requirements, arranging virtual objects on the physical scene model, and creating a virtual film and television scene; Step S2: Distributed computing, which is used to implement complex tasks with large amounts of computation through edge servers. Specifically, it uses a game engine to perform graphic rendering and physical simulation operations on the virtual film and television scene, and uses the green screen technology to separate the actor from the green screen background and replace the green screen background with the virtual film and television scene; Step S3: Device addition, which is used to build the basic hardware conditions for physical simulation and graphic rendering. Specifically, it uses a preview monitor to display the shooting effect in real time, and uses a virtual camera to simulate the shooting effect of a real camera in the virtual film and television scene. In the real-time display and simulation of the shooting effect, one preview monitor is responsible for one virtual camera, and one virtual camera shoots one perspective of the virtual film and television scene; Step S4: Physical simulation, which is used to simulate physical phenomena in virtual reality without graphic rendering. Specifically, it uses a physical engine to simulate physical phenomena caused by collision, gravity, and friction in the virtual film and television scene and perform collision detection; Step S5: State optimization, which is used to construct the shooting method of the virtual camera. Specifically, through a hierarchical finite state machine, it initializes the lens state of the virtual camera, calculates the motion state of the virtual camera, and uses a heuristic method to optimize the lens state and fine-tune the actor's position; Step S6: Graphic rendering, which is used to render the film and television scene in real time. Specifically, it uses a graphics pipeline to centrally host the computing task of drawing virtual objects on the edge server and send the computing result to the preview monitor; The computing result specifically refers to the virtual film and television scene after drawing virtual objects; The edge server and the preview monitor are specifically in a one-to-many relationship, and preview monitors in the same virtual film and television scene will share the same computing result from the edge server with each other; Step S7: Preview playback, which is used to preview the computing result. Specifically, it uses the graphics pipeline deployed on the preview monitor to receive the computing result from the edge server, fuse the actor and the computing result to obtain the final fused image, and convert the final fused image into a 2D image for real-time display on the preview monitor; In step S5, the state optimization specifically includes the following steps: Step S51: Shot breakdown, which is used to initialize the lens state of the virtual camera. Specifically, it uses a hierarchical finite state machine to decompose the dialogue scene to obtain the lens state of the virtual camera. The shot breakdown includes the following steps: Step S511: Using a natural language processing algorithm to analyze the film and television script to obtain a condition list; The condition list specifically refers to the conditions for virtual film and television scene conversion and lens state conversion; The lens state specifically refers to the type, position, and attitude of the lens; Step S512: The hierarchical finite state machine receives a list of conditions, decomposes the complex dialogue scenarios among the actors into multiple levels, with each level corresponding to processing a virtual film and television scene. According to the list of conditions, it triggers the transition of the shot state, and decomposes the complex dialogue scenarios among the actors in a virtual film and television scene into multiple shot states; Step S52: Calculate the motion state of the virtual camera; Step S53: Optimize the shot state. Specifically, with reference to the lines of interest, a heuristic method is used to optimize the shot state and fine-tune the positions of the actors. Among them, the constraints of the heuristic method include smooth motion, occlusion in the frame, and clear picture; The line of interest specifically refers to the line connecting the actors, the line along the direction of the actor's movement, and the line pointing to the direction the actor is facing; The smooth motion is used to maintain the coherence between consecutive shots and avoid shot jitter and jump cuts. Specifically, it refers to detecting whether the motion state of the virtual camera is consistent with the direction of the line of interest; The occlusion in the frame is used to avoid the actors blocking each other. Specifically, it refers to using frame edge detection to count the occluded actors; The clear picture is used to avoid the picture blurring caused by high-speed movement. Specifically, it refers to detecting and controlling the motion state of the virtual camera; Step S54: Visualize the motion state. Specifically, color is used according to the values of linear velocity, yaw angular velocity, pitch angular velocity, and roll angular velocity to visualize the motion state of the virtual camera; Step S55: Optimize the movement path. Specifically, according to the motion state of the virtual camera, in the area in the virtual film and television scene that is easily prone to VR motion sickness identified manually, optimize the movement path of the virtual camera in the virtual film and television scene.

2. A method for constructing a film and television scene based on virtual reality according to claim 1, wherein: In step S2, the distributed computing specifically includes the following steps: Step S21: Separate the physics engine. Specifically, decouple the physics engine from the game engine, encapsulate the physics engine into a Docker container image using Docker containerization technology, split the game engine into a graphics pipeline and a physics engine, deploy the graphics pipeline to run independently on the preview monitor, and deploy the physics engine to run independently on the edge server through the Docker container image; Step S22: Arrange the edge servers. Specifically, arrange the edge servers with low transmission latency at the edge position in the transmission network close to the preview monitor. One edge server deploying the physics engine serves multiple preview monitors in the same virtual film and television scene; Step S23: Interface design. Specifically, design the interface for communication between the graphics pipeline and the physics engine, and realize data synchronization between the graphics pipeline and the physics engine through the interface; Step S24: Network communication synchronization. Specifically, use sockets for network communication between the preview monitor and the edge server.

3. A method for constructing a film and television scene based on virtual reality according to claim 2, wherein: In step S4, the physical simulation specifically includes the following steps: Step S41: Simulate limb generation, specifically by modeling the limbs of the actor, and through the modeling, virtually simulate the limbs to obtain the virtual limb data of the actor, and the virtual data of the actor is used for collision detection; Step S42: Collision detection, specifically when the virtually simulated limb enters the collision space range of the virtual object in the virtual film and television scene, it is detected that the two collide, and the virtually simulated limb and the virtual object are mutually misaligned to eliminate penetration; Step S43: Consistent display, specifically the edge server equipped with the physics engine executes Steps S41 to S42, the physics engine updates the position and actions of the virtually simulated limb, and transmits the results of the physical simulation to all preview monitors in the same virtual film and television scene in a streaming manner in real time, and the preview monitor displays the results of the physical simulation of the actor and the virtual film and television scene.

4. A method for constructing a film and television scene based on virtual reality according to claim 3, wherein: In step S6, the computing task of rendering the virtual object specifically includes the following steps: Step S61: Vertex-by-vertex processing, and the specific operations are as follows: Step S611: Calculate the relative position of each vertex of the virtual object on the screen of the virtual camera, and the normal direction of the surface where the vertex is located; Step S612: Calculate the color A of each vertex according to the motion state of the virtual camera and the distance from the vertex to the virtual camera; Step S62: Surface-by-surface processing, specifically by collecting the color B representing the original color and surface material texture of the virtual object itself; Step S63: Gray value extraction, specifically by rendering the virtual film and television scene multiple times under the illumination conditions at different time points, and outputting the gray scale C; Step S64: Color mixing, specifically by mixing the color A, the color B, and the gray scale C in a weighted multiplication manner to output the final color of each pixel of the virtual object.

Images