Virtual-real cockpit fusion method based on SRWorks video see-through technology

Abstract

The application discloses a virtual-real cockpit fusion method based on SRWorks video perspective technology, and comprises the following steps: a virtual training scene is built; a real object image is collected; a cockpit cover viewport is created; a cockpit cover model is established; a dynamic material is created; a map located in a human eye field of view is intercepted from the virtual training scene; a dynamic material instance is created; the map is made into a dynamic material and is given to the cockpit cover model; a virtual-real fusion step comprises the following steps: a Scene Capture 2D class-based blueprint is established under Unreal Engine (UE4); a captured picture is drawn on a Render Target canvas; texture parameterization is carried out; an Actor class blueprint CockpitCover is established to store the cockpit cover model; and a fusion scene of the virtual training scene and a cockpit instrument panel is displayed in CockpitCover. The virtual-real cockpit fusion method based on the SRWorks video perspective technology is used to superimpose a real scene cockpit on a virtual environment under a natural light environment through the video perspective technology, so that the virtual-real cockpit fusion is realized, and a way of integrating the real scene into the virtual environment is provided.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of virtual-real fusion, and particularly relates to a virtual-real cabin fusion method based on SRWorks video perspective technology. BACKGROUND

[0002] Through analysis of the development of virtual-real fusion technologies at home and abroad, there is not much systematic research in the aspect of video perspective type virtual-real fusion. The existing researches are mainly concentrated in the field of augmented reality (AR) technology, which is also a kind of mixed reality (MR) technology. However, this technology is to superimpose a virtual environment on a real scene through optical perspective principle. At present, the method of virtual-real fusion applied in the field of simulation training is less, and the simulation training system is generally constructed based on a green screen environment. This method first needs to build a bounding box type green screen environment, and also needs to establish an accurate lighting system to realize the fusion of virtual-real scenes by relying on green screen image extraction algorithm.

[0003] At present, the virtual-real fusion technology used for simulation training mostly adopts desktop level and VR task trainers. The desktop level and VR task trainers have poor immersion, and have little significance when applied to training and teaching. Full-function simulators use six-degree-of-freedom motion platforms and spherical visual projection systems, which are expensive and need special team resources to maintain and support. The occupied area is as large as hundreds of square meters, so the simulator is too large. Due to the limitations of site, funds and personnel, it is difficult to guarantee and use and maintain, which causes the technology to be difficult to popularize and apply on a large scale, and the training tasks that can be simulated are also very limited. On the other hand, a visual system is formed by using a simulation cabin and a multi-channel liquid crystal screen (such as a typical three-channel and four-channel), but the field of view angle is small and the immersion is poor. Even if the number of channels is increased to six, it is also difficult to achieve the natural field of view angle of the human eye. Therefore, the tactical training system developed by this technology scheme has the problems of small field of view angle and poor immersion.

[0004] With the gradual maturity of commercial head-mounted MR display production, the technologies of tracking registration, virtual-real fusion and human-computer interaction are increasingly applied in the industry and have better effects. However, MR is not much applied in the military, especially in the field of flight simulation training. The current application mainly adopts an optical perspective type head-mounted display to realize virtual maintenance and disassembly training. The optical lens group in front allows light from the real and virtual environments to pass through at the same time, realizes the superposition of virtual guide information on the real equipment, and realizes the virtual-real fusion effect. In this way, the computer-generated virtual object cannot completely block the object in the real environment, so that the registered virtual object presents a semi-transparent state, which destroys the realism of the fusion of the real scene and the virtual scene and the depth information provided to people due to the occlusion. This method cannot be applied to flight simulation training systems, tactical simulation training systems and the like, and can only be used in the field of virtual maintenance.

[0005] With the development of MR display technology, video see-through technology has also been tried to be applied to simulation training at home and abroad, but all of them are simulation training systems constructed in a green screen environment. This method first needs to build a green screen environment with a bounding box, and also needs to establish an accurate lighting system to realize the fusion of virtual and real scenes by relying on green screen image extraction algorithm. This method not only occupies the site, but also increases the construction cost, and is particularly sensitive to lighting effects.

[0006] The above information disclosed in the background technology is only used to increase the understanding of the background technology of the present application, and therefore, it can include prior art known by those skilled in the art. SUMMARY

[0007] The present application aims at the technical problems of small field of view and poor immersion of virtual-real fusion technology of desktop and VR task trainers in the prior art, and the technical problem of the need to build a green screen environment for video see-through virtual-real fusion. A virtual-real cockpit fusion method based on SRWorks video see-through technology is proposed, which can solve the above problems.

[0008] To achieve the above application purposes, the present application adopts the following technical solutions:

[0009] A virtual-real cockpit fusion method based on SRWorks video see-through technology, comprising:

[0010] A virtual training scene building step to obtain terrain data of the training scene;

[0011] A cockpit canopy viewport creation step to establish a cockpit canopy model;

[0012] A real object image acquisition step to acquire images of a real object cockpit by a real camera;

[0013] A virtual-real fusion step, including a dynamic material creation step and a fusion scene display step;

[0014] The dynamic material creation step includes:

[0015] Real-time acquisition of the eye pose of a user to determine the field of view of the human eye, and cutting a texture map located in the field of view of the human eye from the training scene;

[0016] Creating a dynamic material instance, making the texture map as a dynamic material, and assigning the created dynamic material instance as a material to the cockpit canopy model;

[0017] The fusion scene display step includes:

[0018] Creating a material instance;

[0019] Real-time acquisition of images of the real object cockpit and parameters thereof;

[0020] assigning images of the real cockpit to a map, and passing the map to the material instance;

[0021] adjusting parameters of a camera in a virtual scene in Unreal Engine (UE4) to be consistent with parameters of a real camera;

[0022] assigning the material instance to the cockpit cover model, and displaying the material instance and dynamic material instance in the cockpit cover model as a display window.

[0023] In some embodiments of the present application, the step of building a virtual training scene includes:

[0024] determining a geographical range of the virtual training scene;

[0025] downloading a terrain data file of the determined geographical range from a map database;

[0026] converting the terrain data file into an HFZ format;

[0027] using a terrain mapping software World Machine to detail the terrain data file in the HFZ format;

[0028] creating a new map in a blueprint of Unreal Engine (UE4) and assigning an initial material, importing the terrain data file processed by the terrain mapping software World Machine and a height map, a normal map, and a grayscale map into the blueprint of Unreal Engine (UE4) in a file, and constructing a terrain model; Figure One

[0029] processing the initial material in Unreal Engine (UE4), including at least using multiple textures for picture rendering through map blending, optimizing light slow reflection through normal blending, calculating and adjusting terrain sizes in the height map and the normal map, and grayscale altitudes, and restoring real topography through picture rendering.

[0030] In some embodiments of the present application, the Unreal Engine (UE4) further includes segmenting the imported terrain data file to generate multiple terrain components, and loading the terrain data file in units of the terrain components.

[0031] In some embodiments of the present application, the step of creating a cockpit cover viewport includes:

[0032] establishing a cockpit cover model matching the size of the real cockpit;

[0033] defining a dashboard area in the cockpit cover model, establishing an orthographic projection model of the dashboard, and displaying the orthographic projection model in the dashboard area of the cockpit cover model;

[0034] ​The front projection model of the instrument panel is processed by using Boolean operation and is imported into the Unreal Engine UE4.

[0035] In some embodiments of the present application, in the real object image acquisition step, the real camera is a binocular camera, one of which is used to acquire and render a left eye image, and the other is used to acquire and render a right eye image.

[0036] The depth information of the image of the real cockpit is also acquired to generate a depth grid.

[0037] The acquired image is subjected to three-dimensional reconstruction to obtain geometric information of the real cockpit, and a self-adaptive grid composed of triangular subdivision fitting surface curvature is generated.

[0038] In some embodiments of the present application, in the dynamic material creation step, a Scene Capture 2D-based blueprint is established in the Unreal Engine UE4, the blueprint inherits a camera for capturing a picture from a parent class, the captured picture is drawn on a Render Target canvas through the camera, and the Render Target canvas serves as a basic texture of the dynamic material.

[0039] The angle of the camera is adjusted to ensure that the field of view of the camera is consistent with the field of view of the human eye, and the map is intercepted from the training scene according to the field of view of the camera.

[0040] The dynamic material function of the Unreal Engine UE4 is used to realize continuous switching of the material basic texture, a material is created as a parent material, and the map is parameterized.

[0041] An Actor class blueprint CockpitCover is established to store the cockpit cover model and realize the logic of display function, a Static Mesh component is added in the Actor class blueprint CockpitCover, and the imported cockpit cover model is assigned as a Mesh to the Static Mesh component.

[0042] In some embodiments of the present application, in the fusion scene display step, the fusion scene is displayed in the Actor class blueprint CockpitCover, that is, the real-time display of the virtual-real fusion scene on the cockpit cover model is realized.

[0043] Compared with the prior art, the present application has the following advantages and positive effects:

[0044] The virtual-real cockpit fusion method based on the SRWorks video see-through technology of the application is a method for realizing the fusion of virtual and real cockpits by superimposing the cockpit of a real scene into a virtual environment through the video see-through technology in a natural light environment, and providing a way for the real scene to be integrated into the virtual environment.

[0045] Other features and advantages of the present application will become more apparent from the following detailed description when read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings described in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort based on these drawings.

[0047] Figure 1 is a principle block diagram of an embodiment of the virtual-real cockpit fusion method based on the SRWorks video see-through technology proposed by the present application;

[0048] Figure 2 is a flowchart of the fusion of virtual and real cockpits in an embodiment of the virtual-real cockpit fusion method based on the SRWorks video see-through technology proposed by the present application;

[0049] Figure 3 is an effect of the virtual cockpit cover imported into UE4 in an embodiment of the virtual-real cockpit fusion method based on the SRWorks video see-through technology proposed by the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be described clearly and completely in the following by combining the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort belong to the protection scope of the present application.

[0051] It should be noted that in the description of the present application, the terms indicating the direction or position relationship of "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like are based on the direction or position relationship shown in the drawings, which is only for the convenience of description, and does not indicate or imply that the device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified and limited.

[0052] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting", "fixing" and the like should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0053] The head-mounted display device of the HTC VIVE Pro Eye type has the tool of SRWork, and the implementation of the present solution needs to be based on the SRWork tool of the HTC VIVE Pro Eye.

[0054] The basis for the carrier-based aircraft simulator to realize the virtual-real fusion technology lies in the implementation of the video see-through technology. In the Unreal Engine UE4, the real physical simulation cockpit photographed by the HTC VIVE Pro Eye camera is fused with the virtual training scene made by the Unreal Engine UE engine, and the fused picture is displayed through the HTC VIVE Pro Eye head-mounted display.

[0055] The design of the carrier-based aircraft flight simulation system based on video see-through needs to combine the requirements of carrier-based aircraft simulation training, meet the demand of realizing the basic subject of flight training, and the fidelity of the training scene, the real-time alignment of the virtual cockpit model and the real physical simulation cockpit, and the implementation effect of the video see-through are all very critical. Combined with the HTC VIVE Pro Eye, according to the modularization idea, the video see-through technology based on SRWorks is realized from three aspects of training scene module, cockpit cover viewport module and video see-through management module.

[0056] In the following, a specific embodiment will be described.

[0057] Embodiment one

[0058] This embodiment proposes a virtual-real cockpit fusion method based on SRWorks video see-through technology, as shown inFigure 1 As shown, comprising:

[0059] The virtual training scene building step acquires the terrain data of the training scene;

[0060] The cockpit canopy viewport creation step establishes a cockpit canopy model;

[0061] The real object image acquisition step acquires the image of the real object cockpit by a real camera;

[0062] The virtual-real fusion step includes a dynamic material creation step and a fusion scene display step.

[0063] The dynamic material creation step includes:

[0064] The eye pose of the user is acquired in real time to determine the human eye field of view, and the map located in the human eye field of view is intercepted from the training scene;

[0065] The dynamic material instance is created, the map is made into a dynamic material, and the created dynamic material instance is given to the cockpit canopy model as a material.

[0066] The fusion scene display step includes:

[0067] The material instance is created;

[0068] The image of the real object cockpit and its parameters are acquired in real time;

[0069] The image of the real object cockpit is assigned to the map, and the map is passed to the material instance;

[0070] The parameters of the camera in the virtual scene in the Unreal Engine UE4 are adjusted to be consistent with the parameters of the real camera;

[0071] The material instance is given to the cockpit canopy model, and the cockpit canopy model is used as a display window to display the material instance and the dynamic material instance.

[0072] The virtual-real cockpit fusion method based on the SRWorks video see-through technology of the embodiment is to superimpose the cockpit of the real scene into the virtual environment by the video see-through technology in the natural light environment, realize the fusion of the virtual-real cockpit, and provide a way to integrate the real scene into the virtual environment. The method uses a commercial head-mounted display and its API development library, designs a cockpit canopy viewport module based on the Unreal Engine UE4 engine to realize the fusion of the virtual-real cockpit, does not rely on other development platforms and plug-ins, green screen environment and image extraction algorithms, and has obvious advantages in integration method.

[0073] The virtual-real fusion step further comprises: establishing a blueprint based on a Scene Capture 2D class under Unreal Engine (UE4), drawing the captured picture to a Render Target canvas, and performing texture parameterization on the canvas as a base texture map of a dynamic material; and establishing an Actor class blueprint CockpitCover to store a cockpit cover model, wherein the CockpitCover displays a fusion scene of a virtual training scene and a cockpit instrument panel.

[0074] The method can be applied to a virtual-real fusion simulation training system, has the advantages of convenient deployment, strong pertinence, strong immersion, simple use, flexible implementation of multi-person cooperative technical and tactical confrontation training, and the like, and can meet the requirements of individual soldier training, team-level training and squadron-level training, and can be applied to a plurality of training subjects such as aircraft carrier deck take-off and landing, complex condition take-off and landing, in-flight refueling, formation flight and free air combat.

[0075] The virtual cockpit cover viewport mode is adopted to realize fusion of a real simulation cockpit and a virtual cockpit and a scene, a trainer can see the virtual cockpit cover on the real cockpit in a head-mounted display by wearing an HTC VIVE Pro Eye, and can see a virtual scene through the virtual cockpit cover, and fusion of the real cockpit and the virtual cockpit is realized.

[0076] The traditional flight training system is made by using a simulation terrain, has long time, low efficiency and simple mode, and is designed by using a Vega Prime, a Mantis, a BlueSky or the like to drive a large scene, but does not support VR training application, and therefore has problems of insufficient immersion, limited field of view angle and unreal training scene, and cannot be applied to a virtual-real fusion simulation training field. The UE4 engine is selected to restore a real terrain, VR / MR technology development is supported, cost is low, technology is advanced, and compatibility is good, the virtual-real fusion simulation training system can effectively improve the virtual reality scene and the immersion, the real training terrain is quickly and efficiently modeled, and scene loading is not stuck.

[0077] In some embodiments of the virtual-real fusion simulation training system, the virtual training scene establishment step comprises:

[0078] determining a geographical range of the virtual training scene;

[0079] downloading a terrain data file of the determined geographical range from a map database;

[0080] converting the terrain data file into an HFZ format;

[0081] performing detailed depiction on the terrain data file in the HFZ format by using a terrain map software World Machine;

[0082] In Unreal Engine UE4, a new map is created in Blueprint, and an initial material is assigned, and the terrain data file and height map, normal map processed by the terrain software WorldMachine are imported into the Unreal Engine UE4 in Blueprint to build a terrain model. Figure One

[0083] In Unreal Engine UE4, the initial material is processed, at least including using multiple textures for picture rendering through texture blending, optimizing light slow reflection through normal blending, calculating and adjusting the terrain size in the height map and the normal map, and restoring the real topography through picture rendering.

[0084] According to the requirements of the scene range of the carrier-based aircraft driving training, a 450km*450km super-large real training scene is restored in UE4, and the design is mainly divided into three steps: real terrain data acquisition and preprocessing, terrain data import into UE4 engine, and terrain level division.

[0085] In some embodiments of the application, the terrain data file imported into Unreal Engine UE4 is also segmented to generate multiple terrain components, and the terrain data file is loaded in units of the terrain components.

[0086] (1) First, according to the range of the required training real terrain, taking the Liaodong Peninsula and Shandong Peninsula as an example. The terrain data file containing the above-mentioned region is selected and downloaded through Google Maps; the downloaded DEM data file is read through GlobalMapper to confirm the required region and converted into HFZ format for World Machine to recognize and operate, and the area of the region is about 921600km2, the highest elevation is 2749m, and the lowest elevation is 17m. In World Machine, the terrain file is detailed, in order to ensure the clarity of the modeling, the preview resolution is adjusted to 1024*1024, the RGB channel is separated into a gray map, and the height map, normal map Figure One The file is imported into the UE4 Blueprint to build the final terrain model.

[0087] (2) A new map is created in UE4 Blueprint, and an initial material is assigned. Three layers are added in the texture detail panel to facilitate the connection of different gray maps later. The file generated in World Machine is imported into the created terrain, and the created material is connected. The material in UE4 is processed in a complex way, multiple textures are used for picture rendering through texture blending, light slow reflection is optimized through normal blending, etc. To realize 1:1 restoration of the training landscape and the real scene, and improve the terrain precision. UE4 uses values between -256 and 255.992 to represent the height, so the height map replacement formula is as follows: ​

[0088]

[0089] The displacement of the height map is introduced into UE4, the terrain size, gray scale map altitude, etc. are adjusted through the formula, and the real topography is restored as much as possible through picture rendering.

[0090] (3) According to the principle of saving display resources, the terrain imported from the outside needs to be divided in the UE4 terrain editor, and a plurality of terrain components are generated, and after the whole large terrain is divided, the work of subsequent level division is facilitated. The training terrain range to be realized by the system is too large, and the resolution requirement is high. In order to avoid the loading of super large terrain and the freezing phenomenon caused by processing all terrain data, the training terrain is divided into 30*30 terrain levels, and according to the conversion relationship between the terrain and the pixel area, the range of each terrain level is about 15km*15km. Among them, the airport located in the terrain of each small terrain level is refined, which is composed of 8*8 small terrain levels, and the range of each small level is about 1.875km*1.875km. The division ensures the clarity of the terrain level of the airport before take-off and the loading effect of the distant scene.

[0091] The cockpit cover viewport CockpitCover is the display window of the virtual scene, and is the most important factor to complete the virtual-real fusion video perspective technology. On the one hand, the virtual cockpit cover should be aligned with the real cockpit in a 1:1 manner to reduce the edge through phenomenon after fusion; on the other hand, the trainer sitting in the real cockpit can see the virtual cockpit cover on the real cockpit by wearing the HTC VIVE Pro Eye in the head-mounted display, and can see the virtual scene through the virtual cockpit cover, realizing the fusion of virtual and real cockpits.

[0092] Through the cockpit cover modeling and processing steps, the characteristic of this modeling method is that the FBX model of the virtual cockpit cover is established according to the real cockpit Catia model, which can accurately match the shape of the real cockpit and be consistent with the shape of the real cockpit cover. The cockpit cover model is used as the display window of the virtual scene, and the Scene Capture function of Unreal 4 engine is used to capture the virtual scene picture, which can greatly improve the immersion of the flight personnel in the training process.

[0093] As shown in Figure 3 The effect picture of the virtual cockpit cover imported into UE4.

[0094] In some embodiments of the application, in the cockpit cover viewport creation step, the method for establishing the cockpit cover model is:

[0095] According to the size of the real cockpit, a cockpit cover model matched with the real cockpit is established;

[0096] An instrument panel area is demarcated in the canopy model, a front projection model of the instrument panel is established and displayed in the instrument panel area of the canopy model;

[0097] The front projection model of the instrument panel is processed by using Boolean operation and is imported into the Unreal Engine UE4.

[0098] The characteristic of the modeling method is that the FBX model of the virtual canopy is established according to the physical cabin Catia model, the shape can be accurately matched with the physical cabin, and is consistent with the shape of the real machine canopy, the canopy model is used as a display window of the virtual scene, the Scene Capture function of the Unreal Engine 4 is used to complete the capture of the virtual scene picture, and the immersion of the flight personnel in the training process can be greatly improved.

[0099] The physical aircraft cabin canopy model is used for virtual-real fusion experiment, and the modeling method is as shown in Figure 2 The cabin canopy model file exported by Catia has a.STEP format and can be directly imported into 3Dmax.

[0100] The model is a complete canopy model, but when combined with the physical cabin, part of the physical cabin instrument panel will be blocked due to the realization of the video perspective function, so local processing of the canopy model is needed. A front projection model of the instrument panel is established. Then the model is trimmed by using "composite object-Boolean-difference set" in 3DMax.

[0101] The model processed by the Boolean operation is exported, and the exported file format is.FBX, which can be directly imported into UE4.

[0102] In some embodiments of the application, in the real image acquisition step, the real camera is a binocular camera, one camera is used to acquire and render the left eye image, and the other camera is used to acquire and render the right eye image;

[0103] It also includes obtaining the depth information of the image of the physical cabin to generate a depth grid;

[0104] The acquired image is three-dimensionally reconstructed to obtain the geometric information of the physical cabin, and a self-adaptive grid composed of triangular subdivision of the fitted surface curvature is generated.

[0105] The HTC VIVE Pro Eye can open the camera in SteamVR and test it, and in the case that the head-mounted camera is turned on, the imported virtual cabin model is displayed by using blue print programming, the virtual cabin is adjusted to align with the physical cabin, the imported training scene is assigned in the form of a map inside the canopy, and the scene picture is updated in real time to realize the video perspective effect.

[0106] The SRWorks video perspective method test based on the SRWorks video perspective method test includes:

[0107] (a) Pass-through module ensures the normal operation of the binocular camera, accepts the rendering level setting of the image, one of the cameras is responsible for rendering the left eye image, and the other is responsible for rendering the right eye image.

[0108] (b) Depth module is responsible for providing depth image and related information, ensuring that the depth image can be displayed in real time, and the provided depth information can be used to generate a depth grid for interaction with the collider.

[0109] (c) 3DReconstruction module is responsible for obtaining the geometric information of the real world scene, and generating a triangular subdivision fitting surface curvature to form an adaptive grid.

[0110] (d) Import the SRWorks Experience in Unreal project into UE4, and the Persistent Level contains a series of ViveSR actors containing SRWorks functions, providing references and references for video perspective implementation.

[0111] In some embodiments of the application, the dynamic material creation step further comprises establishing a Scene Capture 2D class-based blueprint in Unreal Engine UE4, the blueprint inherits a camera for capturing pictures from the parent class, and draws the captured picture to the Render Target canvas through the camera, and the Render Target canvas is used as the base texture of the dynamic material.

[0112] Adjust the angle of the camera to ensure that the field of view of the camera is consistent with the field of view of the human eye, and cut the map from the training scene according to the field of view of the camera. By adjusting the angle of the camera, it is ensured that the camera can capture the virtual training scene built from the perspective of the human eye in the cockpit. The captured Render Target is essentially a picture that can be used as the base texture of the material.

[0113] The dynamic material function of Unreal Engine UE4 is used to realize the continuous switching of the base texture of the material, and a material is created as a parent material to parameterize the map. The dynamic material function of UE4 is used to realize the continuous switching of the base texture of the material, a material is created as a parent material, and the base texture map is parameterized, which is convenient for later production of dynamic materials.

[0114] An Actor class blueprint CockpitCover is established for storing the cockpit cover model and implementing the display function logic, a Static Mesh component is added in the Actor class blueprint CockpitCover, and the imported cockpit cover model is assigned as a Mesh to the Static Mesh component. A Static Mesh component is added in the blueprint, and the cockpit cover model imported in 3DS MAX is assigned as a Mesh to the Static Mesh component.

[0115] A dynamic material instance is created, the parent class selects the newly created material, and the base texture parameter is set as the drawn Render Target. The created dynamic material instance is assigned as a material to the cockpit cover model, which realizes the requirement of displaying the virtual scene picture through the cockpit cover.

[0116] The display method of the virtual cockpit CockpitCover for the virtual scene VREnviroment is mainly implemented in the blueprint CockpitCover_Manager, therefore, in some embodiments of the present application, the fusion scene is displayed in the Actor class blueprint CockpitCover in the fusion scene display step. The method is as follows:

[0117] (a) Firstly, a function Creat VRMode CC MID is created, a material instance is created through Create Dynamic MaterialInstance, and is named as CC_MID.

[0118] (b) Three parameters are input to Set VRCapture Component: 1, CC Capture_VRWorld, the parameter content is the picture captured by the camera in UE4, that is, the picture snapshot; 2, screen size; 3, material instance CC_MID.

[0119] (c) A function Update Render Texture is created, in the Update Render Texture, a 2D map is created through CreateRender Target 2D, the picture captured by the camera, that is, the picture snapshot CC Capture_VRWorld is assigned to the 2D map, and then it is transmitted to the created material instance CC_MID.

[0120] (d) The position of the camera in UE4 corresponds to the camera position of the HTC VIVE Pro Eye in the world. Adjust the parameters of the camera in Scene Capture 2D. Adjust the range of the virtual scene that the user sees through the canopy cover by adjusting the size of the picture and the FOV of the camera. As much as possible, restore the human eye perspective, and realize the consistency of the virtual and real picture perspectives.

[0121] (e) Write a Set This Actor Owner Can See function in the blueprint to set the canopy cover visible in the MR mode, and the virtual scene cannot be seen through the camera. The blueprint CockpitCover_Manager corresponds to the virtual canopy cover variable CockpitCover in the scene. Assign the CC_MID material to the canopy cover model, and then set the real-time update material picture through the Update VR_CockpitcoverCapture function to realize the display effect that the virtual scene can only be seen through the canopy cover.

[0122] Using the HTC VIVE Pro Eye requires fixing two locators. When the environment in the locator range changes each time, the room needs to be set again, and the relative position of the head-mounted display needs to be corrected. Therefore, the display position of the virtual canopy cover in the real scene will be affected by the locator and is difficult to fix. In addition, since the head-mounted display perspective is not straight forward when sitting in the real cockpit, the edge part is difficult to align with the real cockpit through simple unit adjustment of the canopy cover. Therefore, the application designs an adjustment key to realize the manual correction function in this environment.

[0123] The S and W keyboard keys can adjust the position of the canopy cover model left and right, the A and D keyboard keys can adjust the canopy cover forward and backward, and the Q and E keyboard keys can adjust the canopy cover up and down. The R keyboard key combined with the Ctrl key can adjust the roll angle of the canopy cover. Similarly, the F and V keys adjust the yaw and pitch angles of the canopy cover, respectively.

[0124] The specific implementation method of realizing the virtual and real cockpit fusion based on the SRWorks video perspective method in the UE4 environment, and the adjustment key is designed to realize the function of manually correcting the relative position of the head-mounted display, adjusting the position and attitude angle of the canopy cover model, which facilitates manual adjustment of the fusion effect.

[0125] The method of the embodiment can realize the following functions:

[0126] (1) In a natural light environment, the virtual canopy cover and the real cockpit are fused and displayed based on the SRWorks video perspective technology, and the virtual scene can be fused and displayed in the virtual canopy cover. It does not need to build a green screen environment, and avoids the application mode of occupying the site and special environment.

[0127] (2) The application realizes the fusion of the physical simulation cockpit and the virtual environment, changes the construction mode of the traditional simulation training visual system, and no longer needs to splice the visual scene through a liquid crystal screen or to build the visual scene through a multi-channel projection system, thereby reducing the cost, avoiding the occupation of the site, and greatly improving the field of view and the sense of immersion.

[0128] (3) The design and implementation of the large landscape restore the real training terrain, replace the original simple and unreliable virtual training terrain, and through the UE4 engine and the means of rapid loading of the terrain, the real training large terrain model is put into the virtual training system and the corresponding picture optimization is carried out, the game engine efficiently loads and renders the large landscape, changes the traditional visual engine application mode, and improves the reality and effectiveness of the simulation training.

[0129] (4) The cockpit cover viewport module is based on the real machine cockpit cover and instrument panel model, and the instrument panel shielding problem in the application process is solved through operation processing in the modeling software, and good simulation effect is obtained after being imported into UE4.

[0130] (5) The video perspective management module is based on the functions of each module provided by SRWorks, and through blue print programming, the training scene is loaded in the form of a map on the material of the cockpit cover model in real time, the visibility of the training scene in the MR mode is set through a function, and the effect of seeing the training scene through the cockpit cover in the MR mode is realized.

[0131] (6) By adding the keys for correcting the position and the turning angle of the cockpit cover model and the head-mounted relative position adjustment keys, the effect of manually realizing the fusion of the virtual and the real through the keyboard is realized.

[0132] The terms in the embodiment are explained as follows:

[0133] UE4: UNREAL ENGINEE4 development engine.

[0134] SRWorks: SRWorks SDK development module provided by HTC VIVE Pro Eye.

[0135] MR: mixed reality, the display of the real world and the virtual world.

[0136] Catia: modeling tool, the model established by the tool is also called Catia model.

[0137] SteamVR: VR scene driving platform, used for driving the VR head-mounted display.

[0138] HTC VIVE Pro Eye: HTC provides a commercial head-mounted display model.

[0139] The above examples are only used to illustrate the technical solutions of the present application, but not to limit the present application; although the present application has been described in detail with reference to the foregoing examples, the technical solutions recorded in the foregoing examples can still be modified by those of ordinary skill in the art, or some technical features thereof can be replaced by equivalent replacements; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions claimed by the present application.

Claims

1. A method for fusing virtual and real cockpits based on SRWorks video perspective technology, characterized in that, include: The steps to build a virtual training scene include obtaining terrain data for the training scene; Steps for creating the cockpit canopy viewport: Build the cockpit canopy model; The physical image acquisition process involves using a real camera to capture images of the actual cockpit. The virtual-real fusion process includes the dynamic material creation process and the scene display process. The dynamic material creation steps include: The user's eye posture is acquired in real time to determine the field of view of the human eye, and the texture located within the field of view of the human eye is extracted from the training scene; Create a dynamic material instance, turn the texture into a dynamic material, and assign the created dynamic material instance as a material to the cockpit canopy model; The dynamic material creation step also includes creating a blueprint based on the Scene Capture 2D class in Unreal Engine 4. The blueprint inherits a camera for capturing the scene from the parent class. The captured scene is drawn onto the Render Target canvas through the camera. The Render Target canvas serves as the base texture of the dynamic material. Adjust the camera angle to ensure that the camera's field of view is consistent with the human eye's field of view, and extract textures from the training scene based on the camera's field of view; The dynamic material feature of Unreal Engine 4 is used to achieve continuous switching of the basic texture of the material. A material is created as the parent material, and the texture is parameterized. Create an Actor class blueprint CockpitCover to store the cockpit canopy model and the logic for implementing the display function. Add a Static Mesh component to the Actor class blueprint CockpitCover and assign the imported cockpit canopy model as a Mesh to the Static Mesh component. The fusion scene display steps include: Create a material instance; Real-time acquisition of images and parameters of the physical cockpit; Assign an image of the physical cockpit to a texture, and then pass the texture to the material instance; Adjust the camera parameters in Unreal Engine 4 within the virtual scene to match those of a real camera. The material instance is assigned to the canopy model, and the canopy model is used as a display window to display the material instance and the dynamic material instance.

2. The virtual-real cockpit fusion method according to claim 1, characterized in that, The steps for setting up a virtual training scenario include: Determine the geographical scope of the virtual training scenario to be implemented; Download the terrain data file for the determined geographic area from the map database; Convert the terrain data file into HFZ format; World Machine, a topographic mapping software, performs detailed rendering of HFZ format topographic data files; Create a new map in the Unreal Engine UE4 Blueprint and assign initial materials. Import the terrain data file processed by the terrain mapping software WorldMachine, along with the height map and normal map, into the Unreal Engine UE4 Blueprint to build the terrain model. In Unreal Engine 4, the initial materials are processed, including at least the use of multiple textures through texture blending for image rendering, optimization of slow reflection of light through normal blending, calculation and adjustment of terrain size and grayscale elevation in height map and normal map, and restoration of realistic terrain through image rendering.

3. The virtual-real cockpit fusion method according to claim 2, characterized in that, Unreal Engine 4 also includes splitting the imported terrain data file to generate multiple terrain components, and loading the terrain data file in units of the terrain components.

4. The virtual-real cockpit fusion method according to claim 1, characterized in that, The method for creating the canopy model in the canopy viewport creation steps is as follows: Based on the dimensions of the actual cockpit, a cockpit canopy model matching it was created; Define the instrument panel area in the canopy model, create an orthographic projection model of the instrument panel, and display it in the instrument panel area of ​​the canopy model. The orthographic projection model of the dashboard was processed using Boolean operations and then imported into Unreal Engine UE4.

5. The virtual-real cockpit fusion method according to any one of claims 1-4, characterized in that, In the physical image acquisition step, the real camera is a binocular camera, one of which is used to acquire and render the left eye image, and the other camera is used to acquire and render the right eye image. It also includes acquiring depth information from images of the actual cockpit to generate a depth mesh; The acquired images are reconstructed in three dimensions to obtain the geometric information of the actual cockpit, and an adaptive mesh is generated by triangulation of the fitted surface curvature.

6. The virtual-real cockpit fusion method according to claim 1, characterized in that, In the fusion scene display step, the fusion scene is displayed in the Actor class blueprint CockpitCover, which realizes the real-time display of the virtual-real fusion scene on the cockpit canopy model.

Images