Video scene construction method based on virtual reality

Through the technology of distributed simulation rendering and point-by-side rendering combined with layered finite state machine, the problem of inaccurate scenes, insufficient interaction, and difficult shadow rendering in the construction of virtual reality film and television scenes is solved, and the virtual camera movement is optimized to reduce the This has caused user discomfort and achieved efficient and low-cost film and television production and immersive experience.

CN119068094AActive Publication Date: 2024-12-03EPOCH TIMES (HENAN) INTERNATIONAL CULTURE MEDIA CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411200689.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-12-03
Estimated Expiration
2044-08-29

AI Technical Summary

Technical Problem

The existing virtual reality film and television scene construction methods have problems such as inaccurate scenes, insufficient interaction, complex construction process, and high computational volume of traditional shadow rendering and high difficulty in real-time rendering. The three-dimensional virtual environment may cause VR motion sickness, resulting in user discomfort and Disoriented.

Method used

The distributed simulation rendering method is adopted to decouple the game engine and the physics engine, and tasks with high computational volume are handed over to the edge server for processing. The rendering method, which combines point-by-face, and layered finite state machine combined with natural language processing technology are optimized. Lens status and virtual camera movement reduce user discomfort.

Benefits of technology

It improves the efficiency and immersion of film and television production, reduces costs, provides greater creative freedom, reduces user discomfort, and enhances the creation ease and user experience of virtual scenes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068094A_ABST
    Figure CN119068094A_ABST
Patent Text Reader

Abstract

The invention belongs to the field of scene construction, and particularly relates to a virtual reality-based movie and television scene construction method, which comprises the steps of scene creation, distributed calculation, equipment addition, physical simulation, state optimization, graphic rendering and preview playing. According to the scheme, a distributed simulation rendering method is creatively provided, and the operation speed of graphic rendering is increased through physical engine decoupling; a point-by-point and face-by-face combined rendering method is creatively adopted, so that the film and television production cost is reduced, and a greater creation freedom degree is provided; a lens state generation method combining lens state and virtual camera motion state optimization is creatively adopted, through a hierarchical finite state machine and a natural language processing technology, an area which easily causes VR motion sickness in a virtual film and television scene is identified, the discomfort of a user is reduced, the user experience is enhanced, and the virtual scene creation work is easy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of scene construction, and in particular to a method for constructing film and television scenes based on virtual reality. Background Art

[0002] VR-based scene construction uses virtual reality technology to create and simulate realistic three-dimensional film and television scenes, allowing filmmakers to freely design, adjust, and experience scenes in a virtual environment. This method can significantly improve the efficiency and creative space of film and television production, reduce the cost of real-life filming, and provide audiences with an immersive visual experience.

[0003] Among the existing approximate solutions, for example, CN117456130A is a scene model construction method. This method focuses on the realism and better display effect after the scene model is constructed. However, the high-precision scene modeling inevitably has the problem of large computational complexity and high difficulty in real-time rendering.

[0004] Distributed computing provides good technical support for solving the high difficulty of real-time rendering. However, in existing approximate solutions for applying distributed computing, for example, CN104183023A, a method for constructing multiple scene graphs in a distributed virtual environment, existing solutions focus on classifying, creating, and optimizing the organization and management of multiple scene graphs in virtual scenes. Although this improves the efficiency of real-time modeling, it also sacrifices the interactive experience of scene construction. The rapidly rendered virtual environment may interfere with the user's viewpoint, resulting in a poor user experience.

[0005] Therefore, existing methods for constructing virtual reality film and television scenes often have technical problems such as the scenes are not realistic enough, lack interactivity, and have a complex construction process; traditional shadow rendering methods simulate the propagation of light in the scene to calculate the shadow of each object surface, but have technical problems such as large computational complexity and difficulty in real-time rendering; there are technical problems such as the three-dimensional virtual environment interfering with the user's viewpoint, easily inducing VR motion sickness, and possibly causing nausea, discomfort, and disorientation in users. Summary of the Invention

[0006] In view of the above situation, in order to overcome the defects of the existing technology, the present invention provides a method for constructing film and television scenes based on virtual reality; in view of the technical problems that the existing virtual reality film and television scene construction methods often have a complex scene construction process and a large amount of rendering calculation, this solution creatively proposes a distributed simulation rendering method, which uses a game engine to simulate and render the scene, decouples the physical engine, and assigns the complex tasks with large amounts of calculation to the edge server for implementation, thereby reducing the computing power cost of the user's preview monitor device in physical simulation and speeding up the operation speed of graphics rendering; in view of the technical problem that the traditional shadow rendering method calculates the shadow of each object surface by simulating the propagation process of light in the scene, and has the difficulty of real-time rendering, this solution creatively adopts a rendering method that combines point-by-point and surface-by-surface rendering. The rendering method uses the object's color, surface material, texture, and lighting, and refers to the relative position and motion state of the virtual object and the virtual camera to render the scene in a vertex-by-vertex and then surface-by-surface order. This reduces film and television production costs and allows the scene to be adjusted at any time according to film and television needs, providing greater creative freedom. It has the characteristics of high efficiency, strong immersion, low cost, and good flexibility, and has high application prospects and market value. To address the technical problem that three-dimensional virtual environments interfere with the user's viewpoint and easily cause VR motion sickness, which may cause nausea, discomfort, and disorientation in users, this solution creatively adopts a lens state generation method that combines lens state and virtual camera motion state optimization. Through hierarchical finite state machine (HFSM) and natural language processing technology, the lens state is initialized, the motion state of the virtual camera is estimated, and areas in the virtual film and television scene that are prone to VR motion sickness are identified. A heuristic method is used to adjust the movement path of the virtual camera in the virtual film and television scene, optimize image quality, reduce user discomfort, enhance user experience, and make virtual scene creation easier.

[0007] The technical solution adopted by the present invention is as follows: The present invention provides a method for constructing a film and television scene based on virtual reality, the method comprising the following steps:

[0008] Step S1: scene creation, used to create a virtual film and television scene, specifically collecting real physical environment data, building a physical scene model, adjusting the details of the physical scene model according to film and television production requirements, and placing virtual objects on the physical scene model to create a virtual film and television scene;

[0009] Step S2: Distributed computing is used to implement complex tasks with large computational loads through edge servers. Specifically, the game engine is used to perform graphics rendering and physical simulation operations on the virtual film and television scenes. Green screen technology is used to separate the actors from the green screen background and replace the green screen background with the virtual film and television scenes.

[0010] Step S3: Adding equipment to build the basic hardware conditions for physical simulation and graphics rendering. Specifically, a preview monitor is used to display the shooting effect in real time, and a virtual camera is used to simulate the shooting effect of a real camera in the virtual film and television scene. In the real-time display and simulation of the shooting effect, one preview monitor is responsible for one virtual camera, and one virtual camera captures one perspective of the virtual film and television scene.

[0011] Step S4: physical simulation, which is used to simulate physical phenomena in virtual reality without graphics rendering. Specifically, a physical engine is used to simulate physical phenomena caused by collision, gravity, and friction in virtual film and television scenes, and to perform collision detection.

[0012] Step S5: state optimization, which is used to construct the shooting mode of the virtual camera. Specifically, the lens state of the virtual camera is initialized through a hierarchical finite state machine, the motion state of the virtual camera is calculated, and the lens state is optimized using a heuristic method to fine-tune the actor position.

[0013] Step S6: Graphics rendering, used to render film and television scenes in real time. Specifically, the graphics pipeline is used to centrally host the computational tasks of drawing virtual objects on the edge server, and the computational results are sent to the preview monitor.

[0014] The calculation result specifically refers to the virtual film and television scene after the virtual object is drawn;

[0015] The edge server and the preview monitor are specifically in a one-to-many relationship, and the preview monitors in the same virtual film and television scene will share the same calculation results from the edge server;

[0016] Step S7: Preview playback is used to preview the calculation results. Specifically, the graphics pipeline deployed on the preview monitor is used to receive the calculation results from the edge server, and the actors and the calculation results are fused to obtain the final fused image. The final fused image is converted into a 2D image and displayed in real time on the preview monitor.

[0017] Furthermore, in step S1, collecting the scene model specifically includes the following steps:

[0018] Step S11: Preprocessing, collecting physical scene images, specifically adjusting the resolution of the physical scene images, performing color correction and denoising, converting the physical scene images into physical three-dimensional scenes, and importing the physical three-dimensional scenes into VR scene construction software;

[0019] Step S12: Virtual 3D scene generation, specifically using 3D modeling tools to optimize the physical 3D scene, arranging corresponding virtual objects according to film and television requirements, generating a virtual 3D scene, and using a generative adversarial network to improve the realism of the virtual 3D scene to generate a virtual film and television scene. The virtual objects include buildings, props and background characters.

[0020] Furthermore, in step S2, the distributed computing specifically includes the following steps:

[0021] Step S21: Separating the physics engine, specifically decoupling the physics engine from the game engine, encapsulating the physics engine into a Docker container image using Docker containerization technology, splitting the game engine into a graphics pipeline and a physics engine, deploying the graphics pipeline on the preview monitor to run independently, and deploying the physics engine on the edge server through the Docker container image to run independently;

[0022] Step S22: deploying edge servers, specifically deploying edge servers with low transmission latency at the edge of the transmission network close to the preview monitors. One edge server deploying a physics engine serves multiple preview monitors in the same virtual film and television scene.

[0023] Step S23: Interface design, specifically designing an interface for communication between the graphics pipeline and the physics engine, and achieving data synchronization between the graphics pipeline and the physics engine through the interface;

[0024] Step S24: Network communication synchronization, specifically using sockets to perform network communication between the preview monitor and the edge server.

[0025] Furthermore, in step S4, the physical simulation specifically includes the following steps:

[0026] Step S41: generating simulated limbs, specifically modeling the actor's limbs, and through the modeling, virtually simulating the limbs to obtain actor virtual limb data, the actor virtual data is used for collision detection;

[0027] Step S42: Collision detection, specifically, when the virtual simulated limb enters the collision space range of the virtual object in the virtual film and television scene, it is detected that the two have collided, and the virtual simulated limb and the virtual object are displaced to eliminate penetration;

[0028] Step S43: Consistency display, specifically, the edge server equipped with the physics engine executes steps S41 to S42, the physics engine updates the position and movement of the virtual simulated limbs, and transmits the results of the physics simulation in real time in a streaming manner to all preview monitors in the same virtual film and television scene. The preview monitor displays the results of the physics simulation of the actors and the virtual film and television scene.

[0029] Furthermore, in step S5, the state optimization specifically includes the following steps:

[0030] Step S51: disassembling the lens to initialize the lens state of the virtual camera. Specifically, a hierarchical finite state machine is used to decompose the dialogue scene to obtain the lens state of the virtual camera. Disassembling the lens includes the following steps:

[0031] Step S511: using a natural language processing algorithm to analyze the film and television script and obtain a condition list;

[0032] The condition list specifically refers to the conditions for virtual film and television scene conversion and lens state conversion;

[0033] The lens status specifically refers to the type, position and posture of the lens;

[0034] Step S512: The hierarchical finite state machine receives the condition list and decomposes the complex dialogue scene between the actors into multiple layers, with each layer corresponding to a virtual film and television scene. According to the condition list, the lens state transition is triggered, and the complex dialogue scene between the actors in the virtual film and television scene is decomposed into multiple lens states.

[0035] Step S52: Calculating the motion state of the virtual camera, specifically including the following steps:

[0036] Step S521: Calculate the relative camera pose matrix. Specifically, perform matrix multiplication on the OpenGL model view matrix of the previous frame and the camera pose matrix of the current frame to calculate the relative camera pose matrix. The formula used is as follows:

[0037] ;

[0038] Where, represents the relative camera pose matrix, is the OpenGL modelview matrix of the previous frame, is the camera pose matrix of the current frame;

[0039] Step S522: angular velocity conversion, specifically converting the angular velocity of the virtual camera into a frame rate representation, using the following formula: ;

[0040] Where, Indicates the frame rate, Indicates the time required for the virtual camera to rotate one circle. represents angular velocity;

[0041] Step S523: Linear velocity calculation, specifically extracting the displacement of the virtual camera along the X, Y, and Z axes from the relative camera pose matrix, multiplying the displacement of the X, Y, and Z axes by the frame rate, and obtaining the linear velocity of the virtual camera along the X, Y, and Z axes for each frame. The formula used is as follows: ; ; ;

[0042] Where, 、 、 Respectively represent the linear speed of each frame of the virtual camera along the X, Y, and Z axes, 、 、 Respectively represent the displacement of the virtual camera along the X, Y, and Z axes, Indicates the frame rate;

[0043] Step S524: extracting the variation, specifically calculating the yaw angle, pitch angle, and roll angle in the order of the Y axis, X axis, and Z axis, where the yaw angle, pitch angle, and roll angle respectively describe the angles at which the virtual camera rotates around the Y, X, and Z axes;

[0044] Extract the yaw angle change, pitch angle change, and roll angle change of the virtual camera from the relative camera posture matrix, and multiply them by the frame rate to obtain the yaw angular velocity, pitch angular velocity, and roll angular velocity. The formula used is as follows: ; ; ;

[0045] Where, 、 、 Respectively represent the yaw angle change, pitch angle change, and roll angle change of each frame, 、 、 Respectively represent the camera's yaw angular velocity, pitch angular velocity, and roll angular velocity per second, Indicates the frame rate;

[0046] Step S53: Optimizing the lens state, specifically, using a heuristic method with reference to the line of interest to optimize the lens state and fine-tune the actor's position, wherein the constraints of the heuristic method include smooth motion, image occlusion, and image clarity;

[0047] The lines of interest specifically refer to lines connecting actors, lines along the direction of actors' movements, and lines pointing in the direction the actors are facing;

[0048] The smooth motion is used to maintain the continuity between consecutive shots and avoid shot oscillation and jump cuts. Specifically, it refers to detecting whether the motion state of the virtual camera is consistent with the direction of the line of interest;

[0049] The screen occlusion is used to prevent actors from blocking each other, specifically by using frame edge detection to count the occluded actors;

[0050] The image clarity is used to avoid image blur caused by high-speed motion, specifically referring to detecting and controlling the motion state of the virtual camera;

[0051] Step S54: visualizing the motion state, specifically coloring according to the values ​​of linear velocity, yaw angular velocity, pitch angular velocity, and roll angular velocity to visualize the motion state of the virtual camera;

[0052] Step S55: Optimizing the movement path, specifically, optimizing the movement path of the virtual camera in the virtual film and television scene by manually identifying areas in the virtual film and television scene that are prone to causing VR motion sickness based on the movement state of the virtual camera.

[0053] Furthermore, in step S6, the computing task of drawing the virtual object specifically includes the following steps:

[0054] Step S61: Vertex-by-vertex processing. The specific operations are as follows:

[0055] Step S611: Calculate the relative position of each vertex of the virtual object on the screen of the virtual camera and the normal direction of the surface where the vertex is located;

[0056] Step S612: Calculate the color A of each vertex according to the motion state of the virtual camera and the distance from the vertex to the virtual camera;

[0057] Step S62: Surface-by-surface processing, specifically collecting the color B representing the original color of the virtual object itself and the surface material texture;

[0058] Step S63: grayscale value extraction, specifically, rendering the virtual film and television scene multiple times under lighting conditions at different time points, and outputting grayscale C;

[0059] Step S64: color mixing, specifically mixing color A, color B, and grayscale C in a weighted multiplication manner to output the final color of each pixel of the virtual object.

[0060] The present invention provides a method for constructing a film and television scene based on virtual reality. The beneficial effects achieved by the present invention using the above scheme are as follows:

[0061] (1) In view of the technical problems that the existing virtual reality film and television scene construction methods often have complex scene construction processes and large rendering computational load, this solution creatively proposes a distributed simulation rendering method that uses a game engine for scene simulation and rendering. By decoupling the physics engine, the complex tasks with large computational load are handed over to the edge server for implementation, reducing the computing power cost of the user's preview monitor device in physical simulation and accelerating the computing speed of graphics rendering.

[0062] (2) The traditional shadow rendering method simulates the propagation of light in the scene to calculate the shadow of each object surface, which has the technical problem of difficult real-time rendering. This solution creatively adopts a rendering method that combines point-by-point and surface-by-surface. According to the color, surface material, texture and lighting of the object itself, and with reference to the relative position and motion state of the virtual object and the virtual camera, the scene is shaded and rendered in the order of vertex first and surface later, which reduces the cost of film and television production. The scene can be adjusted at any time according to the needs of film and television, providing greater creative freedom. It has the characteristics of high efficiency, strong immersion, low cost and good flexibility, and has high application prospects and market value.

[0063] (3) In order to solve the technical problem that the three-dimensional virtual environment interferes with the user's viewpoint and easily causes VR motion sickness, which may cause nausea, discomfort and disorientation in users, this solution creatively adopts a lens state generation method that combines the optimization of lens state and virtual camera motion state. Through the hierarchical finite state machine (HFSM) and natural language processing technology, the lens state is initialized, the motion state of the virtual camera is estimated, and the areas in the virtual film and television scene that are prone to VR motion sickness are identified. Using a heuristic method, the movement path of the virtual camera in the virtual film and television scene is adjusted to optimize the picture quality, reduce the user's discomfort, enhance the user experience, and make the virtual scene creation work easier. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A flowchart of a method for constructing a film and television scene based on virtual reality provided by the present invention.

[0065] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0067] Example 1, see Figure 1 The present invention provides a method for constructing a film and television scene based on virtual reality, which includes the following steps:

[0068] Step S1: scene creation, used to create a virtual film and television scene, specifically collecting real physical environment data, building a physical scene model, adjusting the details of the physical scene model according to film and television production requirements, and placing virtual objects on the physical scene model to create a virtual film and television scene;

[0069] Step S2: Distributed computing is used to implement complex tasks with large computational loads through edge servers. Specifically, the game engine is used to perform graphics rendering and physical simulation operations on the virtual film and television scenes. Green screen technology is used to separate the actors from the green screen background and replace the green screen background with the virtual film and television scenes.

[0070] Step S3: Adding equipment to build the basic hardware conditions for physical simulation and graphics rendering. Specifically, a preview monitor is used to display the shooting effect in real time, and a virtual camera is used to simulate the shooting effect of a real camera in the virtual film and television scene. One preview monitor is responsible for one virtual camera, and one virtual camera captures one perspective of the virtual film and television scene.

[0071] Step S4: physical simulation, which is used to simulate physical phenomena in virtual reality without graphics rendering. Specifically, a physical engine is used to simulate physical phenomena caused by collision, gravity, and friction in virtual film and television scenes, and to perform collision detection.

[0072] Step S5: state optimization, which is used to construct the shooting mode of the virtual camera. Specifically, the lens state of the virtual camera is initialized through a hierarchical finite state machine, the motion state of the virtual camera is calculated, and the lens state is optimized using a heuristic method to fine-tune the actor position.

[0073] Step S6: Graphics rendering, used to render film and television scenes in real time. Specifically, the graphics pipeline is used to centrally host the computational tasks of drawing virtual objects on the edge server, and the computational results are sent to the preview monitor.

[0074] The calculation result specifically refers to the virtual film and television scene after the virtual object is drawn;

[0075] The edge server and the preview monitor are specifically in a one-to-many relationship, and the preview monitors in the same virtual film and television scene will share the same calculation results from the edge server;

[0076] Step S7: Preview playback is used to preview the calculation results. Specifically, the graphics pipeline deployed on the preview monitor is used to receive the calculation results from the edge server, and the actors and the calculation results are fused to obtain the final fused image. The final fused image is converted into a 2D image and displayed in real time on the preview monitor.

[0077] Example 2, see Figure 1 This embodiment is based on the above embodiment. In step S1, collecting the scene model specifically includes the following steps:

[0078] Step S11: Preprocessing, collecting physical scene images, specifically adjusting the resolution of the physical scene images, performing color correction and denoising, converting the physical scene images into physical three-dimensional scenes, and importing the physical three-dimensional scenes into VR scene construction software;

[0079] Step S12: Virtual 3D scene generation, specifically using 3D modeling tools to optimize the physical 3D scene, arranging corresponding virtual objects according to film and television requirements, generating a virtual 3D scene, and using a generative adversarial network to improve the realism of the virtual 3D scene to generate a virtual film and television scene. The virtual objects include buildings, props and background characters.

[0080] Example 3, see Figure 1 This embodiment is based on the above embodiment. In step S2, the distributed computing specifically includes the following steps:

[0081] In step S2, the distributed computing specifically includes the following steps:

[0082] Step S21: Separating the physics engine, specifically decoupling the physics engine from the game engine, encapsulating the physics engine into a Docker container image using Docker containerization technology, splitting the game engine into a graphics pipeline and a physics engine, deploying the graphics pipeline on the preview monitor to run independently, and deploying the physics engine on the edge server through the Docker container image to run independently;

[0083] Step S22: deploying edge servers, specifically deploying edge servers with low transmission latency at the edge of the transmission network close to the preview monitors. One edge server deploying a physics engine serves multiple preview monitors in the same virtual film and television scene.

[0084] Step S23: Interface design, specifically designing an interface for communication between the graphics pipeline and the physics engine, and achieving data synchronization between the graphics pipeline and the physics engine through the interface;

[0085] Step S24: Network communication synchronization, specifically using sockets to perform network communication between the preview monitor and the edge server.

[0086] Example 4, see Figure 1 This embodiment is based on the above embodiment. In step S4, the physical simulation specifically includes the following steps:

[0087] Step S41: generating simulated limbs, specifically modeling the actor's limbs and creating virtual simulated limbs based on the modeling results. The virtual simulated limbs will serve as the actor's avatar and detect collisions between the actor and virtual objects.

[0088] Step S42: Collision detection, specifically, when the virtual simulated limb enters the collision space range of the virtual object in the virtual film and television scene, it is detected that the two have collided, and the virtual simulated limb and the virtual object are displaced to eliminate penetration;

[0089] Step S43: Consistency display, specifically, the edge server equipped with the physics engine executes steps S41 to S42, the physics engine updates the position and movement of the virtual simulated limbs, and transmits the results of the physics simulation in real time in a streaming manner to all preview monitors in the same virtual film and television scene. The preview monitor displays the results of the physics simulation of the actors and the virtual film and television scene.

[0090] Example 5, see Figure 1 This embodiment is based on the above embodiment. In step S5, the state optimization specifically includes the following steps:

[0091] Step S51: disassembling the lens to initialize the lens state of the virtual camera. Specifically, a hierarchical finite state machine is used to decompose the dialogue scene to obtain the lens state of the virtual camera. Disassembling the lens includes the following steps:

[0092] Step S511: using a natural language processing algorithm to analyze the film and television script and obtain a condition list;

[0093] The condition list specifically refers to the conditions for virtual film and television scene conversion and lens state conversion;

[0094] The lens status specifically refers to the type, position and posture of the lens;

[0095] Step S512: The hierarchical finite state machine receives the condition list and decomposes the complex dialogue scene between the actors into multiple layers, with each layer corresponding to a virtual film and television scene. According to the condition list, the lens state transition is triggered, and the complex dialogue scene between the actors in the virtual film and television scene is decomposed into multiple lens states.

[0096] Step S52: Calculating the motion state of the virtual camera, specifically including the following steps:

[0097] Step S521: Calculate the relative camera pose matrix. Specifically, perform matrix multiplication on the OpenGL model view matrix of the previous frame and the camera pose matrix of the current frame to calculate the relative camera pose matrix. The formula used is as follows: ;

[0098] Where, represents the relative camera pose matrix, is the OpenGL modelview matrix of the previous frame, is the camera pose matrix of the current frame;

[0099] Step S522: angular velocity conversion, specifically converting the angular velocity of the virtual camera into a frame rate representation, using the following formula: ;

[0100] Where, Indicates the frame rate, Indicates the time required for the virtual camera to rotate one circle. represents angular velocity;

[0101] Step S523: Linear velocity calculation, specifically extracting the displacement of the virtual camera along the X, Y, and Z axes from the relative camera pose matrix, multiplying the displacement of the X, Y, and Z axes by the frame rate, and obtaining the linear velocity of the virtual camera along the X, Y, and Z axes for each frame. The formula used is as follows: ; ; ;

[0102] Where, 、 、 Respectively represent the linear speed of each frame of the virtual camera along the X, Y, and Z axes, 、 、 Respectively represent the displacement of the virtual camera along the X, Y, and Z axes, Indicates the frame rate;

[0103] Step S524: extracting the variation, specifically calculating the yaw angle, pitch angle, and roll angle in the order of the Y axis, X axis, and Z axis, where the yaw angle, pitch angle, and roll angle respectively describe the angles at which the virtual camera rotates around the Y, X, and Z axes;

[0104] Extract the yaw angle change, pitch angle change, and roll angle change of the virtual camera from the relative camera posture matrix, and multiply them by the frame rate to obtain the yaw angular velocity, pitch angular velocity, and roll angular velocity. The formula used is as follows: ; ; ;

[0105] Where, 、 、 Respectively represent the yaw angle change, pitch angle change, and roll angle change of each frame, 、 、 Respectively represent the camera's yaw angular velocity, pitch angular velocity, and roll angular velocity per second, Indicates the frame rate;

[0106] Step S53: Optimizing the lens state, specifically, using a heuristic method with reference to the line of interest to optimize the lens state and fine-tune the actor's position, wherein the constraints of the heuristic method include smooth motion, image occlusion, and image clarity;

[0107] The lines of interest specifically refer to lines connecting actors, lines along the direction of actors' movements, and lines pointing in the direction the actors are facing;

[0108] The smooth motion is used to maintain the continuity between consecutive shots and avoid shot oscillation and jump cuts. Specifically, it refers to detecting whether the motion state of the virtual camera is consistent with the direction of the line of interest;

[0109] The screen occlusion is used to prevent actors from blocking each other, specifically by using frame edge detection to count the occluded actors;

[0110] The image clarity is used to avoid image blur caused by high-speed motion, specifically referring to detecting and controlling the motion state of the virtual camera;

[0111] Step S54: visualizing the motion state, specifically coloring according to the values ​​of linear velocity, yaw angular velocity, pitch angular velocity, and roll angular velocity to visualize the motion state of the virtual camera;

[0112] Step S55: Optimizing the movement path, specifically, optimizing the movement path of the virtual camera in the virtual film and television scene by manually identifying areas in the virtual film and television scene that are prone to causing VR motion sickness based on the movement state of the virtual camera.

[0113] Example 6, see Figure 1 This embodiment is based on the above embodiment. In step S6, the computing task of drawing the virtual object specifically includes the following steps:

[0114] Step S61: Vertex-by-vertex processing. The specific operations are as follows:

[0115] Step S611: Calculate the relative position of each vertex of the virtual object on the screen of the virtual camera and the normal direction of the surface where the vertex is located;

[0116] Step S612: Calculate the color A of each vertex according to the motion state of the virtual camera and the distance from the vertex to the virtual camera;

[0117] Step S62: Surface-by-surface processing, specifically collecting the color B representing the original color of the virtual object itself and the surface material texture;

[0118] Step S63: grayscale value extraction, specifically, rendering the virtual film and television scene multiple times under lighting conditions at different time points, and outputting grayscale C;

[0119] Step S64: color mixing, specifically mixing color A, color B, and grayscale C in a weighted multiplication manner to output the final color of each pixel of the virtual object.

[0120] Example 7, see Figure 1 This embodiment is based on the above embodiment. In step S12, the three-dimensional modeling tool adopts Blender 3D modeling software.

[0121] Example 8, see Figure 1 This embodiment is based on the above embodiment. In step S612 and step S64, the color is set according to the relative speed between the virtual camera and the virtual object. The faster the speed of the virtual object, the larger the R channel (red) value of the pixel, and the slower the speed of the virtual object, the larger the B channel (blue) value of the pixel.

[0122] Example 9, see Figure 1 This embodiment is based on the above embodiment. In step S63, at different time points, specifically different times of the day (morning, noon, and evening), the parts of the virtual film and television scene that are often illuminated by sunlight are rendered in warm colors, and the parts covered by shadows are rendered in cold colors.

[0123] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0124] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

[0125] The present invention and its embodiments are described above. This description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if a person skilled in the art is inspired by this and, without departing from the purpose of the present invention, designs structures and embodiments similar to this technical solution without inventiveness, they shall fall within the scope of protection of the present invention.

Claims

1. A method for constructing a film and television scene based on virtual reality, characterized in that: The method comprises the following steps: Step S1: scene creation, which is used to create a virtual film and television scene, specifically to collect real physical environment data, build a physical scene model, adjust the details of the physical scene model according to the film and television production requirements, and arrange virtual objects on the physical scene model to create a virtual film and television scene; Step S2: Distributed computing is used to implement complex tasks with large computing volume through edge servers, specifically using a game engine to perform graphics rendering and physical simulation operations on virtual film and television scenes, and using green screen technology to separate the actors from the green screen background, and replace the green screen background with the virtual film and television scene; Step S3: adding equipment, which is used to build basic hardware conditions for physical simulation and graphics rendering, specifically using a preview monitor to display the shooting effect in real time, and using a virtual camera to simulate the shooting effect of a real camera in a virtual film and television scene. In the real-time display and simulation of the shooting effect, one preview monitor is responsible for one virtual camera, and one virtual camera shoots one perspective of the virtual film and television scene; Step S4: physical simulation, which is used to simulate physical phenomena in virtual reality without graphics rendering, specifically using a physical engine to simulate physical phenomena caused by collision, gravity, and friction in virtual film and television scenes, and perform collision detection; Step S5: state optimization, which is used to construct the shooting mode of the virtual camera, specifically, by using a hierarchical finite state machine to initialize the lens state of the virtual camera, calculate the motion state of the virtual camera, use a heuristic method to optimize the lens state, and fine-tune the actor position; Step S6: Graphics rendering, which is used to render the film and television scene in real time. Specifically, the graphics pipeline is used to centrally host the computing tasks of drawing virtual objects on the edge server, and the computing results are sent to the preview monitor; The calculation result specifically refers to the virtual film and television scene after drawing the virtual object; The edge server and the preview monitor are specifically in a one-to-many relationship, and the preview monitors in the same virtual video scene will share the same calculation results from the edge server; Step S7: preview playback, used to preview the calculation results, specifically using the graphics pipeline deployed on the preview monitor to receive the calculation results from the edge server, and fuse the actors and the calculation results to obtain the final fused image, and convert the final fused image into a 2D image and display it in real time on the preview monitor.

2. A method for constructing a film and television scene based on virtual reality according to claim 1, characterized in that: In step S2, the distributed computing specifically includes the following steps: Step S21: separating the physics engine, specifically decoupling the physics engine from the game engine, encapsulating the physics engine into a Docker container image using Docker containerization technology, splitting the game engine into a graphics pipeline and a physics engine, deploying the graphics pipeline on a preview monitor to run independently, and deploying the physics engine on an edge server through a Docker container image to run independently; Step S22: deploying edge servers, specifically deploying edge servers with low transmission delay at edge locations close to preview monitors in the transmission network, and one edge server deploying a physical engine serves multiple preview monitors in the same virtual film and television scene; Step S23: interface design, specifically designing an interface for communication between the graphics pipeline and the physics engine, and realizing data synchronization between the graphics pipeline and the physics engine through the interface; Step S24: Network communication synchronization, specifically using a socket to perform network communication between the preview monitor and the edge server.

3. The method for constructing a film and television scene based on virtual reality according to claim 2, characterized in that: In step S4, the physical simulation specifically includes the following steps: Step S41: generating simulated limbs, specifically modeling the actor's limbs, and through the modeling, virtually simulating the limbs to obtain the actor's virtual limb data, and the actor's virtual data is used for collision detection; Step S42: collision detection, specifically, when the virtual simulated limb enters the collision space range of the virtual object in the virtual video scene, it is detected that the two have collided, and the virtual simulated limb and the virtual object are mutually displaced to eliminate penetration; Step S43: Consistent display, specifically, the edge server equipped with the physics engine executes steps S41 to S42, the physics engine updates the position and movement of the virtual simulated limbs, and transmits the results of the physics simulation in real time in a streaming manner to all preview monitors in the same virtual film and television scene. The preview monitor displays the results of the physics simulation of the actors and the virtual film and television scene.

4. The method for constructing a film and television scene based on virtual reality according to claim 3, characterized in that: In step S5, the state optimization specifically includes the following steps: Step S51: disassembling the lens, which is used to initialize the lens state of the virtual camera. Specifically, a hierarchical finite state machine is used to decompose the dialogue scene to obtain the lens state of the virtual camera. Disassembling the lens includes the following steps: Step S511: using a natural language processing algorithm to analyze the film and television script to obtain a condition list; The condition list specifically refers to the conditions for virtual video scene conversion and lens state conversion; The lens state specifically refers to the type, position and posture of the lens; Step S512: The hierarchical finite state machine receives the condition list, decomposes the complex dialogue scene between the actors into multiple levels, and each level corresponds to processing a virtual film and television scene. According to the condition list, the lens state transition is triggered, and the complex dialogue scene between the actors in a virtual film and television scene is decomposed into multiple lens states; Step S52: Calculate the motion state of the virtual camera; Step S53: optimizing the lens state, specifically, referencing the line of interest, using a heuristic method to optimize the lens state and fine-tune the actor's position, wherein the constraints of the heuristic method include smooth motion, screen occlusion, and clear screen; The interest lines specifically refer to lines connecting actors, lines along the direction of the actors' movements, and lines pointing to the direction the actors are facing; The smooth motion is used to maintain the continuity between consecutive shots and avoid shot oscillation and jump cuts. Specifically, it refers to detecting whether the motion state of the virtual camera is consistent with the direction of the interest line; The screen occlusion is used to prevent actors from occluding each other, specifically by using frame edge detection to count the occluded actors; The image clarity is used to avoid image blur caused by high-speed motion, specifically refers to detecting and controlling the motion state of the virtual camera; Step S54: visualizing the motion state, specifically coloring according to the values ​​of the linear velocity, the yaw angular velocity, the pitch angular velocity and the roll angular velocity to visualize the motion state of the virtual camera; Step S55: Optimizing the moving path, specifically, optimizing the moving path of the virtual camera in the virtual film and television scene by manually identifying the area in the virtual film and television scene that is likely to cause VR motion sickness according to the motion state of the virtual camera.

5. The method for constructing a film and television scene based on virtual reality according to claim 4, characterized in that: In step S6, the computing task of drawing the virtual object specifically includes the following steps: Step S61: Vertex-by-vertex processing, the specific operations are as follows: Step S611: Calculate the relative position of each vertex of the virtual object on the screen of the virtual camera and the normal direction of the surface where the vertex is located; Step S612: Calculate the color A of each vertex according to the motion state of the virtual camera and the distance from the vertex to the virtual camera; Step S62: Surface processing, specifically collecting the color B representing the original color of the virtual object itself and the surface material texture; Step S63: grayscale value extraction, specifically, rendering the virtual video scene multiple times under lighting conditions at different time points, and outputting grayscale C; Step S64: color mixing, specifically mixing color A, color B and grayscale C in a weighted multiplication manner to output the final color of each pixel of the virtual object.

Citation Information

Patent Citations

  • Multi-scene graph construction method in distributed virtual environment

    CN104183023A

  • Method and system for carrying out space virtual shooting based on virtual engine

    CN117596349A

  • Performance test method for game engine

    CN117982897A

  • Video Game Engine Assisted Virtual Studio Production Process

    US20230077552A1

  • Pose estimation virtual data set generation method based on physical engine and collision entity

    WO2024045273A1