Unreal engine-based real scene video and three-dimensional virtual model virtual-real fusion method

Abstract

The application provides a real and virtual fusion method based on Unreal Engine real scene video and three-dimensional virtual model, comprising the following steps: step one: Unreal Engine accesses real scene video; step two: Unreal Engine accesses the completed three-dimensional virtual model; step three: Unreal Engine realizes data registration of real scene video and three-dimensional virtual model; step four: based on Unreal Engine, the position of real scene video and three-dimensional virtual model viewport is synchronized, and real scene video and three-dimensional virtual model are realized real and virtual fusion; the application uses the video actually shot in the physical world to replace the virtual scene in the virtual world, realizes data registration of real scene video and three-dimensional virtual model based on Unreal Engine, realizes position synchronization of real scene video and three-dimensional virtual model viewport, realizes 360-degree full-view depth fusion of real scene video and three-dimensional virtual model in virtual space, greatly improves the reality of virtual scene, and brings good immersive experience.

Description

Technical Field

[0001] This invention relates to the field of video fusion technology, and in particular to a method for fusion of real-world video and 3D virtual models based on Unreal Engine. Background Technology

[0002] Video fusion technology is a branch of virtual reality (VR) technology, or rather, a stage in the development of VR. 3D video fusion technology refers to matching and fusing one or more video image sequences from cameras with associated 3D virtual scenes to generate a new dynamic virtual scene or model, achieving the fusion of virtual scenes and real-time video—that is, virtual-real fusion. Virtual-real fusion precisely blends and displays video footage at the corresponding real-world geographic coordinates of the 3D model. This fusion does not produce misalignment due to tilting or rotating the 3D model, fully leveraging the intuitive characteristics of the 3D scene. 3D video fusion technology can rely on a standalone 3D engine to achieve the fusion of small-scale or localized 3D scenes and video resources, or it can rely on a 3D Geographic Information System (3D GIS) to achieve global 3D geographic information visualization and virtual-real fusion applications.

[0003] Unreal Engine (UE) is a game engine developed by Epic Games. With its powerful real-time rendering capabilities, highly realistic image quality, developer-friendly editing tools, and comprehensive functional modules, Unreal Engine is one of the top game engines. It not only holds a dominant market share in game development but is also widely used in high-precision simulation, industrial design, virtual reality, film production, and many other industries and fields.

[0004] This application aims to provide a method for virtual-real fusion of real-scene video and 3D model data based on Unreal Engine. Before writing this application, a detailed search of existing technologies was conducted, and the closest existing technologies found are as follows:

[0005] Prior art 1: A three-dimensional video fusion and stitching method according to application number 201910976157.0, which mentions a technical solution including the following steps:

[0006] 3D model modeling: Create a precise 3D model of the real image as the basic data for the 3D video fusion and stitching system. It supports multiple inputs and can display data including image data, elevation data, vector data, and 3D model data to achieve real-time rendering of realistic 3D scenes.

[0007] Video point planning: Drag the camera position and posture arbitrarily in the 3D model to clearly display the camera coverage area and accurately define the blind zone area, and scientifically calculate the camera coverage angle and visualize static data information;

[0008] Massive video access and associated scheduling: Integrate all surveillance video resources within the management area, obtain segmented video streams from multiple video management platforms, deeply fuse fragmented segmented video information in time and space, and associate and schedule them in the form of visual targets in a three-dimensional model to achieve unified and efficient management of video resources;

[0009] 3D video fusion monitoring of key areas: Real-time fusion of fragmented surveillance videos from different locations into a 3D scene to achieve real-time global monitoring of the entire monitoring area.

[0010] Key vehicle control: In the event of an emergency in a key area, command personnel will quickly identify the target of interest and select the best real-time video view to obtain important information, and make a judgment and response as soon as possible.

[0011] This application addresses the problem of being unable to "see all" and "understand" massive amounts of discrete video by using a 3D video fusion and stitching method. It integrates and stitches together the massive amount of video from the monitored area into a 3D model, forming a 360-degree panoramic and transparent monitoring system. This meets the core security requirement of "seeing high and low, seeing everything and everything in detail" for the monitored area. At the same time, it can be combined with the deployment and simulation of security and emergency response plans to effectively improve the efficiency of comprehensive command and dispatch, and enable timely, accurate and unified command and handling of emergencies.

[0012] The application involves fusing and stitching massive amounts of video footage from the monitored area into a 3D model to create a 360-degree panoramic and transparent monitoring system. However, the solution fails to register the video and 3D model data, synchronize their viewport positions, or optimize the fusion effect. Consequently, the fusion is poor, the video images are distorted to some extent, and the deep integration of the real physical world and the virtual digital world is not achieved, thus reducing the realism of the virtual scene.

[0013] Therefore, a new technical solution is needed to solve the above-mentioned technical problems. Summary of the Invention

[0014] This application provides a method for virtual-real fusion based on Unreal Engine, which integrates real-world video with 3D virtual models, including the following steps:

[0015] Step 1: Connect Unreal Engine to live video to complete the construction of the live video portion in the virtual-real fusion process;

[0016] Step 2: Connect the completed 3D virtual model to the Unreal Engine to complete the construction of the virtual part in the virtual-real fusion process;

[0017] Step 3: Unreal Engine performs data registration between the live video and the 3D virtual model;

[0018] Step 4: Based on Unreal Engine, synchronize the viewport positions of the live video and the 3D virtual model to achieve virtual-real fusion.

[0019] As a preferred option, step five is also included: optimizing the effect of virtual-real fusion using Unreal Engine.

[0020] As a preferred embodiment, step one includes the following steps:

[0021] Step 1.1: Unreal Engine creates different media sources based on different real-world videos. The video sources are played through Unreal Engine's Media Player Component, and corresponding media textures are generated at the same time.

[0022] Step 1.2: Create a new material using media textures. Simultaneously, place a static mesh in the virtual scene world of the Unreal Engine Editor viewport. Adjust the size parameters of the static mesh to cover the viewport in the Unreal Engine Editor. Then, add the new material to the Screen Position node and apply the new material to the static mesh to ensure that the live video always faces the viewport in the Unreal Engine Editor, thus completing the construction of the live part in the video's virtual-real integration.

[0023] As a preferred embodiment, step two includes the following steps:

[0024] Step 2.1: Import the 3D virtual model into the Unreal Engine editor and place the 3D virtual model into the Unreal Engine virtual scene world, ensuring that the live video and the 3D virtual model can be seen through the virtual scene world viewport;

[0025] Step 2.2: Create relevant materials for the 3D virtual model in the Unreal Engine editor to complete the construction of the virtual part in the video's virtual-real integration.

[0026] As a preferred embodiment, the data registration of the real-scene video and the 3D virtual model includes the creation of the virtual scene camera viewport and the calibration of the real-scene video coordinates. The data registration of the real-scene video and the 3D virtual model is achieved based on the creation of the virtual scene camera viewport and the calibration of the real-scene video coordinates.

[0027] As a preferred embodiment, the process of creating the virtual scene camera viewport includes the following steps:

[0028] Step 3.1: Place a virtual scene camera in the virtual scene world, and face the static mesh of the virtual scene world in Step 1.2;

[0029] Step 3.2: Bind the static mesh to the virtual scene camera using the empty actor object in the Unreal Engine editor. The static mesh will move with the virtual scene camera, ensuring that the live video displayed by the static mesh is always within the viewport of the virtual scene camera.

[0030] As a preferred embodiment, the real-scene video coordinate calibration includes the following steps:

[0031] Step 3.3: Solve for the parameters within the real-scene video acquisition device using an intrinsic parameter calibration algorithm;

[0032] Step 3.4: Obtain the relationship between the coordinates of the three-dimensional virtual model and the real-scene video by using the world coordinates of the points in the three-dimensional virtual model and the texture matrix formula (1), and obtain the shooting angle and position information of the real-scene video by using the parameters in the real-scene video acquisition device;

[0033]

[0034] Where (X0, Y0, Z0, W0) are the world coordinates of a point D in the 3D virtual model; (s, t, r, q) are the texture coordinates d of the texel corresponding to point D;

[0035] ModelviewMatrix defines the position and orientation of the virtual scene camera in the world coordinate system. It is a product of a rotation matrix and a translation matrix.

[0036] The Projective Matrix defines a perspective transformation. In 3D rendering, perspective transformation is represented intuitively by a view volume. A view volume can be defined using the function glFrustum(left, right, bottom, top, near, far). The Scale and Bias Matrix is ​​used to transform the calculated coordinates [-1, 1] to the texture coordinates [0, 1].

[0037] As a preferred embodiment, the specific process for data registration between the real-scene video and the 3D virtual model is as follows:

[0038] By combining the relationship between the 3D virtual model and the coordinates of the real-world video obtained in step 3.4, as well as the shooting angle and position information of the real-world video, the 3D coordinates of a certain point in the real-world video corresponding to the 3D virtual model are calculated. By matching the 3D virtual model with these 3D coordinates, the matching of the real-world video with the virtual scene camera viewport of the virtual scene world is completed.

[0039] As a preferred embodiment, step four specifically includes the following steps:

[0040] Step 4.1: Obtain the shooting angle and position information of each frame of the real-scene video through the parameters in the real-scene video acquisition device, and link the shooting angle and position information of each frame together using Unreal Engine to form a continuous motion trajectory of the camera position and angle in the virtual scene.

[0041] Step 4.2: Import the above motion trajectory data into the virtual scene camera viewport position control parameters to achieve synchronization between the virtual scene camera viewport and the virtual scene world viewport, synchronization between the virtual 3D viewport and the real scene 2D viewport angles, and fusion of real scene video and 3D virtual model.

[0042] As a preferred embodiment, the effect optimization includes processing the foreground and background occlusion relationship between the real-scene video and the 3D virtual model, as well as the restoration and simulation of the real weather environment.

[0043] As a preferred embodiment, the processing of the front and rear occlusion relationship includes the following steps:

[0044] Step 5.1a: Determine the geographical location of the 3D virtual model in the live video, obtain the foreground-background relationship between the 3D virtual model and the objects in the live video, and determine the foreground objects in the live video that occlude the 3D virtual model. Obtain the mask of the foreground objects in the live video using any one of the foreground method, midground method, or background method, and export the mask video file synchronized with the live video timeline.

[0045] Step 5.2a: Import the mask video file into the virtual scene world using the methods in Steps 1.1 and 1.2 to generate a mask static mesh with occlusion effect. Then, bind the mask static mesh to the virtual scene camera using the method in Step 3.2. The mask static mesh will follow the movement and angle of the virtual scene camera to ensure that the video displayed by the mask static mesh is always within the viewport of the virtual scene camera.

[0046] Step 5.3a: Set the position of the mask static mesh in the virtual scene world. Place the mask static mesh at any position between the 3D virtual model and the virtual scene camera to realize the foreground occlusion relationship between the 3D virtual model and the foreground objects in the real scene video during video fusion.

[0047] As a preferred embodiment, the restoration and simulation of the real weather environment includes the following steps:

[0048] Step 5.1b: Download the plugin Ultra Dynamic Weather and import it into Unreal Engine. Place the Unreal Engine Editor Blueprint Actor that controls the weather in Ultra Dynamic Weather into the virtual scene world, and implement weather switching in the control panel of the Blueprint in the Unreal Engine Editor.

[0049] Step 5.2b: Add the Ultra dynamicweather material function to the material node of the 3D virtual model edited in step 2.2, and connect it to the material node of the 3D virtual model itself to realize the interaction of weather effects on the 3D object.

[0050] This application replaces virtual scenes with videos actually filmed in the physical world. It uses Unreal Engine to achieve data registration between the real-world video and the 3D virtual model, and synchronizes the viewport positions of the real-world video and the 3D virtual model based on Unreal Engine, realizing the fusion of the real-world video and the 3D virtual model. It achieves 360° full-view fusion of the real-world video and the 3D virtual model in the virtual space. At the same time, it solves a series of difficulties such as real-world video coordinate calibration, 3D virtual model data registration, video image distortion, handling the front and back occlusion relationship between real-world video and 3D virtual model data, and weather simulation. Ultimately, it achieves a deep integration of the real physical world and the virtual digital world, greatly enhancing the realism of the virtual scene and bringing a good immersive experience. Attached Figure Description

[0051] Figure 1 The flowchart of this application;

[0052] Figure 2 This is a schematic diagram of the calibration plate of this application;

[0053] Figure 3 This is a schematic diagram of an embodiment of this application;

[0054] Figure 4 This is a schematic diagram of video and 3D model viewport registration in this application; Detailed Implementation

[0055] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application. That is, the specific implementation methods described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0056] Example 1:

[0057] In this embodiment, the two-dimensional real-scene video can be drone aerial video or surveillance video. Figure 3 This is the technical model for this case;

[0058] The following is an application example based on drone-captured live video—a method for fusion of 2D live video of a building construction project with a 3D building model using Unreal Engine:

[0059] Step 1: Connect Unreal Engine to 2D real-world video captured by drones;

[0060] Specifically: Step 1.1: Import the 2D real-world video into the Unreal Engine editor, create a video source in the Unreal Engine editor, and then specify the video source location as the pre-prepared 2D real-world construction video. The 2D real-world construction video is played through the Unreal Engine's built-in Media Player Component while generating the corresponding Media texture.

[0061] Step 1.2: Viewport Setup for the 2D Construction Video: Create a new material A using the media texture generated in Step 1.1. Simultaneously, place a static mesh in the viewport of the Unreal Engine Editor and attach material A. Adjust the size parameters of the static mesh to 16:9 (since the drone video in this example is 16:9, the specific parameters can be changed according to the actual project), and ensure that the size of the static mesh can cover the viewport in the Unreal Engine Editor. Finally, add a Screen Position node to the material A of the static mesh to ensure that the 2D construction video always faces the viewport in the Unreal Engine Editor from any angle.

[0062] Step 2: Building the 3D virtual model. In this embodiment, the 3D virtual model is a 3D architectural model of the construction site.

[0063] Specifically: Step 2.1: Build a 3D building model of the construction building using 3D software, import the 3D building model of the construction building into the Unreal Engine editor, and put the 3D building model of the construction building into the virtual scene world, ensuring that the 2D real-scene video of the construction and the 3D building model of the construction building can be seen through the viewport of the virtual scene world.

[0064] Step 2.2: Create relevant material B for the 3D architectural model of the construction building in the Unreal Engine editor to complete the construction of the virtual part in the video virtual-real fusion;

[0065] Step 3: Viewport registration of the 2D construction video and the 3D building model of the construction site. This mainly includes the creation of the virtual scene camera viewport and the calibration of the 2D construction video coordinates. Based on the creation of the virtual scene camera viewport and the calibration of the 2D construction video coordinates, the viewport registration of the 2D construction video and the 3D building model of the construction site is achieved.

[0066] Specifically, the process of creating the virtual scene camera viewport includes the following steps:

[0067] Step 3.1: Create a virtual scene camera viewport: A virtual scene camera needs to be placed in the virtual scene world, and the virtual scene camera viewport should face the static mesh in Step 1;

[0068] Step 3.2: Bind the static mesh to the virtual scene camera using the empty actor object in the Unreal Engine editor. The static mesh will move with the viewport of the virtual scene camera, ensuring that the 2D real-world construction video displayed by the static mesh is always within the viewport of the virtual scene camera, no matter the angle.

[0069] Specifically, the coordinate calibration of the two-dimensional real-scene video during construction includes the following steps:

[0070] Step 3.3: Solve for the UAV parameters using an intrinsic parameter calibration algorithm. Intrinsic parameter calibration is a necessary step for visual sensors to eliminate lens distortion. Especially for UAV video lenses, to obtain a larger field of view and greater depth of field, the lenses are often made more curved, introducing severe barrel distortion. Therefore, these distortions must be corrected to fully utilize the acquired images. Zhang Zhengyou's calibration method is used to extract images of the calibration board from different directions. The calibration board is shown below. Figure 2 As shown, corner points are extracted from each image using the Harris corner extraction algorithm, and the extracted corner points are used to solve the equations to calculate the intrinsic parameters of the drone's camera equipment.

[0071] Step 3.4: Select a point in the 2D real-scene construction video that needs to be matched with the 3D building model of the construction building. The texture coordinates of the point in the 3D building model of the construction building are d(s, t, r, q). Obtain the shooting angle and position information of the 2D real-scene construction video through the parameters of the drone's real-scene lens. Determine the virtual scene camera matrix based on the shooting angle and position information of the 2D real-scene construction video.

[0072]

[0073] Where (X0, Y0, Z0, W0) are the world coordinates of a point D in the 3D virtual model; (s, t, r, q) are the texture coordinates d of the texel corresponding to point D; it can be seen from equation (1) that the texture matrix is ​​composed of three matrices connected in series; ModelviewMatrix defines the position and orientation of the virtual scene camera in the world coordinate system, which is a product of a rotation matrix and a translation matrix; Projective(Frustum)Matrix defines a perspective transformation. In 3D rendering, the perspective transformation can be represented intuitively by a view volume. A view volume can be defined by the function glFrustum(left, right, bottom, top, near, far); Scale and Bias Matrix is ​​used to transform the calculated coordinates [-1, 1] to the range of texture coordinates [0, 1].

[0074] Step 3.5: Obtain the projection matrix (Projective(Frustum)Matrix) using formulas (2)(3)(4)(5)(6).

[0075]

[0076]

[0077]

[0078]

[0079]

[0080] Among them: (2) is a formula obtained based on the relationship of similar triangles. From formula (2), we can obtain formulas (3) and (4). Similarly, we can obtain formulas (5) and (6).

[0081] The height and width of the 2D real-scene video image of the construction are height and width, respectively. A line segment with length top is mapped to a line segment with length v0, while a line segment with length -bottom is mapped to a line segment with length height-v0. This represents the ratio of the distance to the near-view plane of the scene to the distance to the camera of the drone. This represents the aspect ratio between the near-view plane distance and the lens width of the drone's camera.

[0082] Substituting the parameters obtained in steps 3.4 and 3.5 into texture matrix equation 1, we can inversely calculate the 3D coordinates D(X0, Y0, Z0, W0) corresponding to a point d in the 2D construction video of the 3D building model. Similarly, repeating steps 3.3-3.4, we determine the coordinates of another 3D building model point D2 (corresponding to point d2 in the 2D video). Then, we can determine the position of the 3D building model in the virtual scene world that matches the 2D construction video using these two points. Figure 4 As shown;

[0083] Step 4: Synchronize the viewport positions of the 2D construction video and the 3D architectural model of the building under construction:

[0084] Specifically: Step 4.1: Obtain the shooting angle and position information of each frame of the construction 2D real-scene video through the parameters in the drone's real-scene lens, and link the shooting angle and position information of each frame through the Sequencer level sequence function in Unreal Engine to form a continuous motion trajectory of the virtual scene camera position and angle.

[0085] Step 4.2: Import the motion trajectory data formed above into the virtual scene camera viewport position control parameters to achieve synchronization between the virtual scene camera viewport and the virtual scene world viewport, that is, to achieve synchronization between the virtual 3D viewport and the real scene 2D viewport angle. At this time, the motion trajectory of the 3D virtual scene camera is consistent with the real camera's perspective and position during the real scene video shooting, thus completing the virtual-real fusion of the construction 2D real scene video and the 3D building model.

[0086] Example 2:

[0087] This embodiment provides an optimization scheme to improve the effect of virtual-real fusion, specifically:

[0088] It also includes step five: Unreal Engine optimizes the virtual-real fusion effect of the construction 2D real-scene video and the 3D building model, mainly including handling the foreground and background occlusion relationship in video fusion, as well as the restoration and simulation of real weather environment.

[0089] Handling front and rear occlusion relationships in video fusion:

[0090] Step 5.1a: Determine the geographical location of the 3D building model of the construction building in the 2D live video of the construction site. This will reveal the foreground / background relationship between the 3D building model and objects in the 2D live video of the construction site. It will also identify several construction tower cranes as foreground objects that may occlude the 3D building model of the construction site. The mask of the construction tower cranes in the live video can be obtained using any of the foreground, midground, or background methods. The mask can then be exported as a mask video file that is synchronized with the timeline of the 2D live video of the construction site.

[0091] Step 5.2a: Import the mask video file into the virtual scene world using the methods in steps 1.1 and 1.2 to generate a mask static mesh with occlusion effect. Then, bind the mask static mesh to the virtual scene camera using the method in step 3.2. At this time, the mask static mesh will follow the position and angle changes of the virtual scene camera.

[0092] Step 5.3a: Set the position of the mask patch static mesh in the virtual scene world. Place the tower crane mask patch static mesh at any position between the 3D building model of the construction building and the virtual scene camera to realize the front and back occlusion relationship between the 3D building model of the construction building and the construction tower crane in the 2D real-world construction video in the video fusion.

[0093] Realistic weather environment restoration and simulation:

[0094] Step 5.1b: Realize and simulate the real weather environment in the virtual scene world through the Ultra dynamic weather plugin system. Specifically, place the Unreal Engine Editor Blueprint Actor that controls the weather in the Ultra dynamic weather plugin into the virtual scene world, and switch the weather in the control panel of the Blueprint in the Unreal Engine Editor.

[0095] Step 5.2b: Add the Ultradynamic weather material function to the material node of the 3D building model of the construction building edited in step 2.2, and connect it to the material node of the 3D building model of the construction building itself.

[0096] This embodiment is based on a virtual-real fusion method using Unreal Engine to fuse a 2D real-scene video of a building construction with a 3D construction building model. After the 2D real-scene video of the construction is fused with the 3D construction building model, the building becomes a 3D building model, the construction scene becomes the video, and the tower crane in front of the building becomes the foreground object in the video. Rotating different viewpoints, the fused image does not undergo distortion, and the fusion effect is good.

[0097] This application replaces virtual scenes with videos actually filmed in the physical world. It uses Unreal Engine to achieve data registration between the real-world video and the 3D virtual model, and synchronizes the viewport positions of the real-world video and the 3D virtual model based on Unreal Engine, realizing the fusion of the real-world video and the 3D virtual model. It achieves 360° full-view fusion of the real-world video and the 3D virtual model in the virtual space. At the same time, it solves a series of difficulties such as real-world video coordinate calibration, 3D virtual model data registration, video image distortion, handling the front and back occlusion relationship between real-world video and 3D virtual model data, and weather simulation. Ultimately, it achieves a deep integration of the real physical world and the virtual digital world, greatly enhancing the realism of the virtual scene and bringing a good immersive experience.

[0098] It should be noted that the components or steps in the above embodiments can be interchanged, substituted, added, or deleted. Therefore, the combinations formed by these reasonable permutations and transformations should also fall within the protection scope of this invention, and the protection scope of this invention should not be limited to the above embodiments.

[0099] The above are exemplary embodiments disclosed in this invention. The order of the disclosed embodiments is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. However, it should be noted that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the disclosed embodiments of this invention (including the claims) is limited to these examples. Various changes and modifications can be made without departing from the scope defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular.

[0100] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of the different aspects of the invention as described above exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A method for fusion of real-world video and 3D virtual models based on Unreal Engine, characterized in that, Includes the following steps: Step 1: Connect Unreal Engine to real-world video. The real-world video can be drone aerial footage or surveillance video, which is used to build the real-world portion of the virtual-real fusion process. Step 1.1: Unreal Engine creates different video sources based on different live-action videos. The video sources are played through Unreal Engine's media playback component, and corresponding media textures are generated at the same time. Step 1.2: Create a new material using media textures. At the same time, place a static mesh in the viewport of the Unreal Engine Editor, adjust the size parameters of the static mesh to cover the viewport of the Unreal Engine Editor, add the new material to the screen position node, and apply the new material to the static mesh to ensure that the live video always faces the viewport of the Unreal Engine Editor, thus completing the construction of the live part in the video's virtual-real combination. Step 2: Connect the completed 3D virtual model to the Unreal Engine to complete the construction of the virtual part in the virtual-real fusion process; Step 2.1: Import the pre-built 3D virtual model into Unreal Engine and place the imported 3D virtual model into the virtual scene world of Unreal Engine. You can see the live video and the 3D virtual model through the virtual scene world viewport. Step 2.2: Create relevant materials for the 3D virtual model in the Unreal Engine editor to complete the construction of the virtual part in the video's virtual-real integration; Step 3: Unreal Engine performs data registration between the live video and the 3D virtual model; Step 3.1: Place a virtual scene camera in the virtual scene world, and face the virtual scene camera viewport toward the static mesh in step 1.2; Step 3.2: Bind the static mesh to the virtual scene camera using the empty actor object in the Unreal Engine editor. The static mesh will move with the virtual scene camera, ensuring that the real-world video displayed by the static mesh is always within the viewport of the virtual scene camera. Step 4: Based on Unreal Engine, synchronize the viewport positions of the live video and the 3D virtual model to achieve virtual-real fusion of the live video and the 3D virtual model; Step 4.1: Obtain the shooting angle and position information of each frame of the real-scene video through the parameters in the real-scene video acquisition device, and link the shooting angle and position information of each frame together using Unreal Engine to form a continuous motion trajectory of the camera position and angle in the virtual scene. Step 4.2: Import the above motion trajectory data into the viewport position control parameters of the virtual scene camera to synchronize the viewport of the virtual scene camera with the viewport of the virtual scene world, and realize the virtual-real fusion of real scene video and 3D virtual model; Step 5: Optimize the effect of virtual-real fusion using Unreal Engine.

2. The method for virtual-real fusion of real-world video and 3D virtual model based on Unreal Engine according to claim 1, characterized in that, The data registration of the real-scene video and the 3D virtual model includes the creation of the virtual scene camera viewport and the calibration of the real-scene video coordinates. Based on the creation of the virtual scene camera viewport and the calibration of the real-scene video coordinates, the data registration of the real-scene video and the 3D virtual model is realized.

3. The method for virtual-real fusion of real-world video and 3D virtual model based on Unreal Engine according to claim 2, characterized in that, The real-scene video coordinate calibration includes the following steps: Step 3.3: Solve for the parameters within the real-scene video acquisition device using an intrinsic parameter calibration algorithm; Step 3.4: Obtain the relationship between the coordinates of the 3D virtual model and the real-scene video by using the world coordinates of the points in the 3D virtual model and the texture matrix formula, and obtain the shooting angle and position information of the real-scene video by using the parameters of the real-scene video acquisition device.

4. The method for virtual-real fusion of real-world video and 3D virtual model based on Unreal Engine according to claim 1, characterized in that, The effect optimization includes handling the foreground-background occlusion relationship between the real-scene video and the 3D virtual model, specifically including the following steps: Step 5.1a: Determine the geographical location of the 3D virtual model in the live video, obtain the foreground-background relationship between the 3D virtual model and the objects in the live video, and determine the foreground objects in the live video that occlude the 3D virtual model. Obtain the mask of the foreground objects in the live video using any one of the foreground method, midground method, or background method, and export the mask video file synchronized with the live video timeline. Step 5.2a: Import the mask video file into the virtual scene world using the methods in Steps 1.1 and 1.2 to generate a mask static mesh with occlusion effect. Then, bind the mask static mesh to the virtual scene camera using the method in Step 3.

2. The mask static mesh will follow the movement and angle of the virtual scene camera to ensure that the video displayed by the mask static mesh is always within the viewport of the virtual scene camera. Step 5.3a: Set the position of the mask static mesh in the virtual scene world. Place the mask static mesh at any position between the 3D virtual model and the virtual scene camera to realize the foreground occlusion relationship between the 3D virtual model and the foreground objects in the real scene video during video fusion.

5. The method for fusion of real-world video and 3D virtual model based on Unreal Engine according to claim 1, characterized in that, The effect optimization includes the restoration and simulation of realistic weather conditions, specifically including the following steps: Step 5.1b: Download the plugin Ultra Dynamic Weather and import it into Unreal Engine. Place the Unreal Engine Editor Blueprint Actor that controls the weather in Ultra Dynamic Weather into the virtual scene world, and implement weather switching in the control panel of the Blueprint in the Unreal Engine Editor. Step 5.2b: Add the Ultra dynamic weather material function to the material node of the 3D virtual model edited in step 2.2, and connect it to the material node of the 3D virtual model itself to realize the interaction of weather effects on the 3D object.

Images