Virtual and real scene construction method based on video perspective

Through video perspective technology, virtual and real scene construction is achieved in flight simulation training, solving the problem of low immersion in traditional equipment and virtual reality technology that cannot combine with real scenes, reducing costs and dangers, and improving immersion and experience effects.

CN119068149APending Publication Date: 2024-12-03QINGDAO VIRTUAL REALITY RES INST CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411106616.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing flight simulation training equipment has low immersion, traditional virtual reality technology cannot achieve a true combination of virtual and real, and video streaming green screen keying technology has high requirements for lighting and color, resulting in poor immersion effect.

Method used

Using a virtual and real scene construction method based on video perspective, virtual instrument images are generated through simulation computers to collect control data and simulation data, and the visual computer constructs and renders the scene-side scene, and generates and outputs the fusion image to the MR headset through network communication and fusion algorithm. , achieve the integration of virtual and real.

Benefits of technology

It reduces training costs and dangers, improves immersion and experience effects, and meets basic training requirements while enhancing immersion and experience effects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068149A_ABST
    Figure CN119068149A_ABST
Patent Text Reader

Abstract

A virtual and real scene construction method based on video perspective relates to the technical field of virtual and real fusion, and comprises the following steps: a simulation computer acquires control data of a simulation cabin and calculates simulation data of each subsystem of an airplane through simulation end software, and generates a corresponding virtual instrument image and a flight attitude; the visual computer constructs a visual end scene including a simulation cabin through visual end software; the visual computer performs scene rendering on the visual end scene through the visual end software; the simulation computer is in network communication with the scene computer, and the scene computer receives the flight attitude and performs scene driving on the scene of the scene end; the visual computer carries out virtual-real fusion through visual end software by adopting a fusion algorithm to generate a fused image; and the visual computer outputs the fused image to the MR head-mounted display. According to the invention, the training cost and danger can be reduced; and on the premise of meeting basic training requirements, the immersion and experience effects are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of virtual-reality fusion, and in particular to a virtual-reality scene construction method based on video perspective. Background Art

[0002] At present, the simulation training equipment used by pilots in daily flight simulation training is mostly based on the visual scheme of semi-physical aircraft simulator cockpit and trifold screen (or spherical screen). Although this type of simulation training equipment meets the daily learning and training needs of pilots to a certain extent, its low immersion problem is also extremely prominent. This is because the visual scheme constructed by the trifold screen or spherical screen cannot make the pilots truly immersed in the virtual scene.

[0003] Of course, there is also integrated development based on virtual reality technology combined with traditional aircraft simulators, but this integrated development method is mostly reflected in the VR state, that is, the pilot can only see the content in the virtual scene, but cannot see the equipment in the real scene. This method cannot achieve a true combination of virtual and real; another method is based on video stream green screen keying technology for integration. Although this method can enable pilots to achieve a certain degree of virtual and real fusion on the basis of wearing virtual reality head-mounted display devices combined with the green screen environment, that is, they can see the scene after the combination of virtual and real, but the video stream green screen keying technology has certain requirements on the lighting, color, etc. of the on-site environment, which makes it easy for pilots to encounter incomplete keying and poor immersion effects during the experience. Summary of the invention

[0004] In view of this, the technical problem to be solved by the present invention is: to provide a method for constructing virtual and real scenes based on video perspective, which can reduce the cost and danger of training; and improve the immersion and experience effect while meeting basic training requirements.

[0005] In order to solve the above technical problems, the technical solution of the present invention is:

[0006] A method for constructing a virtual-real scene based on video perspective, the method comprising the following steps:

[0007] S1. The simulation computer collects control data of the simulation cockpit and calculates simulation data of each subsystem of the aircraft through the simulation terminal software, and generates corresponding virtual instrument images and flight attitudes based on the control data and simulation data;

[0008] S2. The visual computer constructs a visual end scene including the simulated cockpit through the visual end software, wherein the visual end scene includes a model, audio, texture and material;

[0009] S3, the visual computer performs scene rendering on the visual end scene through the visual end software, and the scene rendering includes rendering of special effects, lighting, mapping and viewpoints;

[0010] S4, the simulation computer communicates with the vision computer through a network, the vision computer receives the flight attitude and performs scene driving on the vision end scene, wherein the scene driving includes updating the current vision end scene based on the change of the flight attitude;

[0011] S5, the vision computer uses a fusion algorithm to fuse the current surrounding environment, the simulated cockpit and the vision end scene through the vision end software to generate a fused image;

[0012] S6. The visual computer outputs the fused image to the MR head display.

[0013] Preferably, S1 includes the following steps:

[0014] S11, the simulation computer receives the control data;

[0015] S12, the simulation computer generates a simulation system, wherein the simulation system includes a radar system, a navigation system, an avionics system, and a fire control system;

[0016] S13. The simulation computer generates and updates the virtual instrument image based on the control data and the simulation system. The virtual instrument image includes an altimeter, an MFD, an airspeed indicator, and an attitude indicator.

[0017] Preferably, the simulated cockpit includes a control element, an interactive element and a display element; the control data includes control data acquired based on a state change of the control element and interactive data acquired based on a state change of the interactive element; the display element is used to display a virtual instrument image;

[0018] The control elements include a joystick, a throttle lever and a pedal; the interactive elements include switches, buttons and knobs; and the display element includes a liquid crystal screen.

[0019] Preferably, in S2, the visual computer constructs the visual end scene based on Unreal Engine tools;

[0020] The model includes airports, airplanes, buildings, oceans, ships, missiles, vehicles, and plants; the audio includes airplane engine audio, interactive element toggle audio, explosion audio, and warning audio; the texture includes maps and pictures involved in the visual end scene; and the material includes the appearance material required for the model.

[0021] Preferably, in S4, the simulation computer and the vision computer communicate via a network, and the network communication is used for two-way communication between the flight attitude and the vision end scene;

[0022] The visual computer is provided with interface communication, event communication and projection communication, and the interface communication, event communication and projection communication are used for blueprint data communication inside the visual end scene;

[0023] Updating the current visual end scene includes changing the state of the model, audio, texture and material according to the flight posture. The state changes include the display and disappearance of the model, the display and disappearance of particle effects, the change of the environment, and the change of the sound.

[0024] Preferably, in S5, the fusion algorithm includes a viewport synchronization algorithm, a posture tracking algorithm, a virtual-real registration algorithm, and a virtual-real superposition algorithm.

[0025] Preferably, it also includes a locator installed on the simulation cabin, and at least two positioning base stations installed outside the simulation cabin; the positioning base station is connected to the simulation computer, and the positioning base station is used to locate and track the locator and the MR head display.

[0026] Preferably, it also includes a large-screen display, which is connected to the visual computer and is used to display the fused image.

[0027] After adopting the above technical solution, the beneficial effects of the present invention are:

[0028] In the present application, the virtual-reality scene construction method includes: the simulation computer collects control data of the simulation cockpit and calculates simulation data of each subsystem of the aircraft through the simulation end software, and generates corresponding virtual instrument images and flight attitudes based on the control data and simulation data; the vision computer constructs a vision end scene including a simulation cockpit through the vision end software, and the vision end scene includes models, audio, textures and materials; the vision computer performs scene rendering on the vision end scene through the vision end software, and the scene rendering includes rendering of special effects, lighting, mapping and viewpoints; the simulation computer and the vision computer communicate over a network, the vision computer receives the flight attitude and performs scene driving on the vision end scene, and the scene driving includes updating the current vision end scene based on changes in the flight attitude; the vision computer uses a fusion algorithm to perform virtual-reality fusion on the current surrounding environment, the simulation cockpit and the vision end scene through the vision end software to generate a fused image; the vision computer outputs the fused image to the MR headset. After the simulation computer generates the virtual instrument image and flight attitude, and the vision computer generates the vision-side scene, the vision computer can drive the vision-side scene based on the flight attitude, update the current vision-side scene, and fuse the simulation cockpit and the vision-side scene so that both the simulation cockpit (including the virtual instrument image) and the vision-side scene can be displayed in the MR head display, which changes the original vision scheme using a semi-physical aircraft simulator cockpit and a trifold screen (or a spherical screen), as well as the virtual-reality fusion scheme based on video stream green screen matting technology, reduces the cost and danger of training; and improves the immersion and experience effect while meeting basic training requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0030] Figure 1 is a flow chart of a method for constructing a virtual-real scene based on video perspective according to an embodiment of the present invention;

[0031] Figure 2 is a framework diagram of an embodiment of the present invention;

[0032] Figure 3 It is a framework diagram of a simulation computer in an embodiment of the present invention generating a virtual instrument image based on control data;

[0033] Figure 4 It is a framework diagram of the visual computer in the embodiment of the present invention constructing the visual end scene and connecting with the MR avatar. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0035] Mixed reality (MR) is the further development of virtual reality technology. It introduces real scene information into the virtual environment, builds a bridge of interactive feedback information between the virtual world, the real world and the user, and thus enhances the realism of the user experience. The key point of mixed reality technology is to interact with the real world and obtain information in a timely manner, so its implementation needs to be in an environment that can interact with various things in the real world. By using the latest mixed reality technology, the MR head display device is combined with the traditional semi-physical aircraft simulator simulation cockpit equipment to construct a virtual-real fusion flight simulation training experience, thereby greatly improving the sense of immersion.

[0036] like Figures 1 to 4 As shown collectively, the present invention comprises the following steps:

[0037] S1. The simulation computer collects control data of the simulation cockpit and calculates simulation data of each subsystem of the aircraft through the simulation terminal software, and generates corresponding virtual instrument images and flight attitudes based on the control data and simulation data;

[0038] S2. The visual computer constructs a visual scene including a simulated cockpit through the visual software. The visual scene includes models, audio, textures and materials.

[0039] S3, the visual computer performs scene rendering on the visual end scene through the visual end software, and the scene rendering includes rendering of special effects, lighting, mapping and viewpoints;

[0040] S4, network communication between the simulation computer and the vision computer, the vision computer receives the flight attitude and performs scene driving on the vision end scene, the scene driving includes updating the current vision end scene based on the change of the flight attitude;

[0041] S5. The vision computer uses a fusion algorithm to fuse the current surrounding environment, the simulated cockpit and the vision end scene through the vision end software to generate a fused image;

[0042] S6. The visual computer outputs the fused image to the MR headset.

[0043] In S1, data collection, construction, transmission and reception of the simulation end are realized. The hardware design of the simulation end includes a simulation cockpit and a simulation computer, which are connected through video cables and data cables. Among them, the simulation cockpit, as a direct hardware device for interacting with the user, includes control elements, interactive elements and display elements; control data includes control data obtained based on the state change of the control element and interactive data obtained based on the state change of the interactive element; the display element is used to display the virtual instrument image; the control element includes a joystick, a throttle lever and a pedal; the interactive element includes switches, buttons and knobs; the display element includes an LCD screen.

[0044] As the main hardware device for running the simulation end software, the simulation computer is connected to the LCD screen and other display elements of the simulation cockpit through a video cable, and the virtual instrument screen drawn in the simulation end software is displayed on the LCD screen in real time; on the other hand, the control elements and interactive elements in the simulation cockpit will transmit the relevant data to the simulation computer through the data cable through the data acquisition board built into the simulation cockpit, and the simulation end software in the simulation computer will parse through the communication protocol to obtain the relevant data. In addition, the simulation computer is connected to the vision computer through a network cable, and the simulation end software can communicate the relevant data with the vision end software running in the vision computer through the network cable in the form of network communication.

[0045] S1 includes the following steps:

[0046] S11, the simulation computer receives control data;

[0047] S12. A simulation computer generates a simulation system, which includes a radar system, a navigation system, an avionics system, and a fire control system;

[0048] S13. The simulation computer generates and updates a virtual instrument image based on the control data and the simulation system. The virtual instrument image includes an altimeter, an MFD, an airspeed indicator, and an attitude indicator.

[0049] Among them, the simulation software is mainly used to design and develop the relevant functions of a certain type of aircraft through simulation, so that users can also experience the relevant functions of the real aircraft in the simulated cockpit and conduct simulation exercises of related training business.

[0050] The simulation software mainly includes two parts: the instrument layer and the system layer. Among them, the instrument layer mainly involves the drawing of the altimeter, MFD, airspeed indicator, and attitude indicator. The virtual instrument image developed by the instrument layer can be displayed on the LCD screen of the relevant instruments in the simulation cockpit after the simulation software is run; the system layer mainly simulates and develops the radar system, navigation system, avionics system, and fire control system of the real aircraft, so that the simulation software has functions close to the relevant systems of the real aircraft; the communication bus mainly simulates the common data bus on the current real aircraft. The virtual instrument image designed and developed by the instrument layer and the relevant simulation system developed by the system layer will communicate data through the communication bus.

[0051] The data collection of interactive components is based on the serial communication technology under the Windows platform. The serial communication module is developed through the Visual Studio development tool and the Win32 API related to serial communication to achieve the acquisition and analysis of the status data of the interactive components. The data collection of the operating components is based on the relevant API in DirectInput under the Windows platform, and the data collection function is developed based on the Visual Studio development tool. The production of virtual instruments is based on the human-machine interface development tool GLStudio. GLStudio has the functions of drawing virtual instruments and developing related functional interfaces and functional functions, and is fully capable of drawing and developing the required virtual instruments. The development of the simulation system is based on the Visual Studio development tool using C++ programming, and functional simulation is performed according to the business logic and functional description of each simulation system, so that it has a functional experience close to the real machine related system.

[0052] The visual end software is mainly developed based on the virtual reality development engine (Unreal Engine). Its function is to build virtual training scenes for users. At the same time, it combines relevant virtual-reality fusion algorithms to immerse users in the virtual-reality fusion scenes and enhance the realism of the user's training experience.

[0053] The visual end software mainly includes five parts: scene construction, scene rendering, data communication, scene drive, and virtual-reality fusion. Among them. Scene construction is mainly based on user needs by building a virtual visual end scene; scene rendering is mainly to adjust the relevant elements of the visual end scene to make the visual end scene more realistic in visual effect; data communication is mainly responsible for the data communication between the visual end software and the simulation end software and the data communication of the relevant functional modules within the visual end software; scene drive is mainly to drive the state of the model, image, and special effects in the visual end scene after the visual end software receives the relevant data; video perspective is mainly to achieve the purpose of matching and fusion of the semi-physical simulation cockpit and the virtual scene constructed by the visual end software through the relevant virtual-reality fusion algorithm.

[0054] In S2, the visual computer constructs the visual side scene based on the Unreal Engine tool; the models include airports, airplanes, buildings, oceans, ships, missiles, vehicles, and plants; the audio includes aircraft engine audio, interactive component toggle audio, explosion audio, and warning audio; the texture includes the maps and pictures involved in the visual side scene; the material includes the appearance material required for the model.

[0055] In S4, the vision computers communicate with each other over the network, and the network communication is used for two-way communication between the flight attitude and the vision end scene; the network communication is to build the UDP communication function in Unreal Engine, which is to realize the data communication between the vision end software and the simulation end software, that is, the flight, attitude and other data of the simulation end software can be sent to the vision end software through the network communication function, and the relevant data of the vision end software can also be sent to the simulation end software through the network communication function.

[0056] Interface communication, event communication and projection communication are set up in the visual computer. Interface communication, event communication and projection communication are used for blueprint class data communication within the visual end scene; interface communication, event communication and projection communication belong to the communication method between blueprint classes that come with UnrealEngine.

[0057] Updating the current visual scene includes changing the state of models, audio, textures, and materials according to the flight posture. State changes include the display and disappearance of models, the display and disappearance of particle effects, changes in the environment, and changes in sounds.

[0058] In S5, the fusion algorithms include viewport synchronization algorithm, posture tracking algorithm, virtual-real registration algorithm and virtual-real superposition algorithm. The simulated cockpit in the real scene is fused with the aircraft model and virtual environment in the virtual scene to form a virtual-real fusion scene. In layman's terms, the aircraft model in the virtual scene only has the part other than the simulated cockpit, and it and the simulated cockpit in the real scene together form a complete aircraft. When the simulation-end software and the visual-end software are running, the user wears the MR headset and immerses himself in the three-dimensional scene. He can see the simulated cockpit in the real scene and the virtual environment (sky, airport, ocean, plants, buildings, etc.).

[0059] In the present application, it also includes a locator installed on the simulation cockpit, and at least two positioning base stations installed on the outside of the simulation cockpit; the positioning base station is connected to the simulation computer, and the positioning base station is used to locate and track the locator and the MR headset and obtain their posture information; the locator is used to track the position of the semi-physical simulation cockpit.

[0060] The large-screen display is also included, and the large-screen display is connected to the visual computer, and the large-screen display is used to display the fused image. The large-screen display is the main carrier for displaying the image information presented in the MR head display, and the spectator can watch the image in the MR head display through the large-screen display.

[0061] After the simulation computer generates the virtual instrument image and flight attitude, and the vision computer generates the vision-side scene, the vision computer can drive the vision-side scene based on the flight attitude, update the current vision-side scene, and fuse the simulation cockpit and the vision-side scene so that both the simulation cockpit (including the virtual instrument image) and the vision-side scene can be displayed in the MR head display, which changes the original vision scheme using a semi-physical aircraft simulator cockpit and a trifold screen (or a spherical screen), as well as the virtual-reality fusion scheme based on video stream green screen matting technology, reduces the cost and danger of training; and improves the immersion and experience effect while meeting basic training requirements.

[0062] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a virtual-real scene based on video perspective, characterized in that: The virtual-real scene construction method comprises the following steps: S1. The simulation computer collects control data of the simulation cockpit and calculates simulation data of each subsystem of the aircraft through the simulation terminal software, and generates corresponding virtual instrument images and flight attitudes based on the control data and simulation data; S2. The visual computer constructs a visual end scene including the simulated cockpit through the visual end software, wherein the visual end scene includes a model, audio, texture and material; S3, the visual computer performs scene rendering on the visual end scene through the visual end software, and the scene rendering includes rendering of special effects, lighting, mapping and viewpoints; S4, the simulation computer communicates with the vision computer through a network, the vision computer receives the flight attitude and performs scene driving on the vision end scene, wherein the scene driving includes updating the current vision end scene based on the change of the flight attitude; S5, the vision computer uses a fusion algorithm to fuse the current surrounding environment, the simulated cockpit and the vision end scene through the vision end software to generate a fused image; S6. The visual computer outputs the fused image to the MR head display.

2. The method for constructing a virtual-real scene based on video perspective according to claim 1, characterized in that: S1 includes the following steps: S11, the simulation computer receives the control data; S12, the simulation computer generates a simulation system, wherein the simulation system includes a radar system, a navigation system, an avionics system, and a fire control system; S13. The simulation computer generates and updates the virtual instrument image based on the control data and the simulation system. The virtual instrument image includes an altimeter, an MFD, an airspeed indicator, and an attitude indicator.

3. The method for constructing a virtual-real scene based on video perspective according to claim 1, characterized in that: The simulated cockpit includes a manipulation element, an interaction element and a display element; the control data includes manipulation data acquired based on a state change of the manipulation element and interaction data acquired based on a state change of the interaction element; the display element is used to display a virtual instrument image; The control elements include a joystick, a throttle lever and a pedal; the interactive elements include switches, buttons and knobs; and the display elements include a liquid crystal screen.

4. The method for constructing a virtual-real scene based on video perspective as claimed in claim 3, characterized in that: In S2, the visual computer constructs the visual end scene based on the Unreal Engine tool; The model includes airports, airplanes, buildings, oceans, ships, missiles, vehicles, and plants; the audio includes airplane engine audio, interactive element toggle audio, explosion audio, and warning audio; the texture includes maps and pictures involved in the visual end scene; and the material includes the appearance material required for the model.

5. The method for constructing a virtual-real scene based on video perspective according to claim 1, characterized in that: In S4, the simulation computer and the vision computer communicate with each other through a network, and the network communication is used for two-way communication between the flight attitude and the vision end scene; The visual computer is provided with interface communication, event communication and projection communication, and the interface communication, event communication and projection communication are used for blueprint data communication inside the visual end scene; Updating the current visual end scene includes changing the state of the model, audio, texture and material according to the flight posture. The state changes include the display and disappearance of the model, the display and disappearance of particle effects, the change of the environment, and the change of the sound.

6. The method for constructing a virtual-real scene based on video perspective according to claim 1, characterized in that: In S5, the fusion algorithms include viewport synchronization algorithm, pose tracking algorithm, virtual-real registration algorithm and virtual-real superposition algorithm.

7. The method for constructing a virtual-real scene based on video perspective according to claim 1, characterized in that: It also includes a locator installed on the simulation cabin, and at least two positioning base stations installed outside the simulation cabin; the positioning base station is connected to the simulation computer, and the positioning base station is used to locate and track the locator and the MR head display.

8. The method for constructing a virtual-real scene based on video perspective according to claim 1, characterized in that: It also includes a large-screen display, which is connected to the visual computer and is used to display the fused image.