A low-latency mixed reality method and device for simulated flight mixed reality helmet

By using signal processing boards for parallel fusion processing in simulated flight hybrid reality systems, the high load and delay problems of virtual and real fusion are solved, and the low-latency simulated flight hybrid reality effect is achieved, simplifying the development and maintenance process.

CN118675395BActive Publication Date: 2024-11-26CHENGDU CHENGDIAN GUANGXIN TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411143362.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-26
Estimated Expiration
2044-08-20

AI Technical Summary

Technical Problem

In the existing mixed reality technology of simulated flight, virtual and real are integrated into high load and high computing power tasks, resulting in large system computing overhead, high latency, and difficult development and maintenance, which cannot meet the low-latency simulation requirements in the simulated flight field.

Method used

The signal processing board is used to merge the simulated flight image and the real-life camera image in parallel, and use FPGA hardware to acquire and decode the real-life image to reduce computing overhead, and decouple the flight image generation from mixed reality, so that it can be executed in parallel.

Benefits of technology

It significantly reduces the time delay of mixed reality in simulated flights, improves the efficiency and stability of the system, simplifies the development and maintenance process, and meets the low-latency simulation requirements in the field of simulated flights.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118675395B_ABST
    Figure CN118675395B_ABST
Patent Text Reader

Abstract

The present invention discloses a low-latency mixed reality method and device for a simulated flight mixed reality helmet, which belongs to the field of simulated flight training, and includes the steps of: S1, constructing a mixed reality system; S2, using the signal processing board in a PC to simultaneously receive a simulated flight picture and a real-scene camera picture, and outputting them after fusion processing; wherein the simulated flight picture is generated by the PC and output to the signal processing board through the video port of the graphics card of the PC; the real-scene camera picture is generated by a camera in the mixed reality helmet and transmitted to the signal processing board; the mixed reality picture is obtained by the signal processing board by mixing the simulated flight picture with the real-scene camera picture, and output to the mixed reality helmet for viewing by the user. The present invention reduces the computational overhead of the simulated flight computer, decouples the generation of flight pictures from the mixed reality, allows them to be executed in parallel, and further reduces the time delay of the simulated flight mixed reality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of simulated flight training, and more specifically, to a low-latency mixed reality method and device for a simulated flight mixed reality helmet. Background Art

[0002] The existing simulated flight mixed reality is implemented using PC software, that is, the real scene is captured by the PC software and merged and superimposed with the simulated flight virtual scene. Since the PC software needs to capture the camera image, generate the simulated flight virtual scene, and perform virtual and real fusion.

[0003] The use of the above method has the following technical problems: a) Virtual-reality fusion is a high-load and high-computing-power task. The software needs to detect each pixel of the picture and perform fusion processing, which leads to the consumption of computing power of the visual computer hardware, which is not conducive to the efficient and stable operation of the system; b) The real scene in the virtual-reality fusion is collected by the visual computer, and there are many collection links, involving USB, operating system, memory, and graphics card, which leads to a large delay in the real scene, which is not conducive to the requirement of low-latency simulation in the field of simulated flight; c) The virtual-reality fusion code needs to be run additionally on the visual computer, which is difficult to develop, maintain and debug. Compared with the use of dedicated equipment, it is not conducive to the modular expansion of various functions in the field of simulated flight. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a low-latency mixed reality method and device for a simulated flight mixed reality helmet, which reduces the computing overhead of the simulated flight computer and decouples flight picture generation from mixed reality so that they can be executed in parallel, further reducing the time delay of the simulated flight mixed reality.

[0005] The object of the present invention is achieved through the following solutions:

[0006] A low-latency mixed reality method for simulating a flight mixed reality helmet comprises the following steps:

[0007] S1, building a mixed reality system: connecting the mixed reality helmet to a PC, which has a signal processing board, and the signal processing board is connected to a graphics card in the PC;

[0008] S2, using the signal processing board in the PC to simultaneously receive the simulated flight picture and the real-scene camera picture, and output them after fusion processing; wherein the simulated flight picture is generated by the PC and output to the signal processing board through the video port of the graphics card of the PC; the real-scene camera picture is generated by the camera in the mixed reality helmet and transmitted to the signal processing board; the mixed reality picture is obtained by the signal processing board by mixing the simulated flight picture and the real-scene camera picture, and output to the mixed reality helmet for the user to watch;

[0009] In step S2, the real-scene camera image is collected, which specifically includes sub-steps: the signal processing board collects and decodes the camera image in parallel based on the FPGA hardware; the signal processing board outputs the decoded camera image to the display;

[0010] In step S2, the simulated flight picture is collected, which specifically includes the sub-steps of: receiving the simulated flight picture transmitted by the graphics card in real time, caching it by line during the receiving process, and superimposing, fusing and outputting the simulated flight picture and the real scene picture by line, so that the delay is the delay of one line of video cached by the hardware.

[0011] Furthermore, in step S2, the fusion process specifically includes the following sub-steps:

[0012] S21, detecting the pixel value of a pixel in the simulated flight picture, and if the pixel value of the pixel is RBG000, replacing it with the camera picture;

[0013] S22, when the flight simulation software is generating the simulated flight picture, it will perform corresponding processing on the picture to be fused, and set the fused aircraft parts to black. When the signal processing board performs virtual and real fusion, it will detect the simulated flight picture pixel by pixel in parallel, and replace its black pixel part with the real camera picture.

[0014] Furthermore, the image acquisition, image fusion and image output of the signal processing board are all implemented using FPGA.

[0015] Furthermore, for the simulated flight picture acquisition, the signal processing board uses the DP interface to receive the picture.

[0016] Furthermore, for real-scene camera image acquisition, the signal processing board acquires the image through a USB interface.

[0017] Furthermore, for mixed reality screen output, the signal processing board uses a DP interface for output.

[0018] Furthermore, the real-scene camera image is generated by the camera in the mixed reality helmet and is transmitted to the signal processing board through the UVC protocol.

[0019] A low-latency mixed reality device for a simulated flight mixed reality helmet comprises a processor and a memory, wherein a program is stored in the memory, and when the program is loaded by the processor, the low-latency mixed reality method for a simulated flight mixed reality helmet as described in any one of the above items is executed.

[0020] The beneficial effects of the present invention include:

[0021] The present invention physically separates the visual rendering in simulated flight from mixed reality. The simulated flight computer is only responsible for generating simulated flight images, and a new mixed reality signal processing board is added for visual camera acquisition and superposition. The present invention reduces the computational overhead of the simulated flight computer, and decouples flight image generation from mixed reality, allowing them to be executed in parallel, further reducing the time delay of simulated flight mixed reality. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0023] Figure 1 A data flow diagram of an embodiment of the present invention;

[0024] Figure 2 is an overall block diagram of an embodiment of the present invention;

[0025] Figure 3 This is a flowchart of the steps of the traditional real-scene image acquisition method;

[0026] Figure 4 This is a schematic diagram of a USB direct acquisition and hardware decoding method using a hardware signal processing board in an embodiment of the present invention;

[0027] Figure 5 It is a fusion schematic diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] All features disclosed in all embodiments in this specification, or steps in all methods or processes implicitly disclosed, except for mutually exclusive features and / or steps, can be combined and / or expanded or replaced in any manner.

[0029] The specific implementation process of the technical solution of the present invention is as follows:

[0030] In the embodiment of the present invention, the signal processing board simultaneously receives the simulated flight image and the real-scene camera image, fuses them and outputs them. The data stream is as follows: Figure 1As shown. There are three video data streams, namely: a) simulated flight screen; b) real-life camera screen; c) mixed reality screen. For the simulated flight screen, it is generated by the PC and output through the PC's graphics card video port, and the output destination is the signal processing board. For the real-life camera screen, it is generated by the camera in the mixed reality helmet and transmitted to the signal processing board through the UVC protocol. For the mixed reality screen, it is obtained by the signal processing board mixing the simulated flight screen and the real-life camera screen, and the output destination is the mixed reality helmet for users to watch.

[0031] The image acquisition, image fusion, and image output of the signal processing board are all implemented using FPGA. For simulated flight image acquisition, the signal board uses the DP interface to receive images. The schematic diagram is as follows: Figure 2 As shown in the figure, for real-time camera image acquisition, the signal processing board collects images through the USB interface.

[0032] After receiving the simulated flight image and the camera image, the signal processing board will fuse the two images. The technical solution of the embodiment of the present invention fuses the image overlay solution. The FPGA will detect the pixel value of the pixel point in the simulated flight image. If the pixel value is RBG000, it will be replaced with the camera image. For the mixed image output, the signal processing board uses the DP interface.

[0033] When the flight simulation software generates the simulated flight images, it will perform special processing on the images to be fused, and set the fused aircraft parts, such as the aircraft cockpit, to black (000000h). When the signal processing board performs virtual-real fusion, it will detect the simulated flight images pixel by pixel in parallel, and replace the black (000000h) pixel part with the real-scene camera image, such as Figure 5 shown.

[0034] For real scene image acquisition, such as Figure 3 As shown, the traditional acquisition method acquires data through the operating system USB driver and decodes it inside the operating system. The data flow is as follows:

[0035] a) The operating system USB driver collects the camera image, taking a 60Hz 1080P resolution image as an example, which takes about 16 milliseconds;

[0036] b) The video decoding software decodes the captured images, which takes about 10 milliseconds;

[0037] c) The decoded camera image is transmitted to the graphics card for virtual and real fusion, which takes 5 milliseconds;

[0038] d) The graphics card outputs the virtual and real fusion image to the display, which takes about 16 milliseconds.

[0039] The total time for the traditional method is 47 milliseconds.

[0040] like Figure 4 As shown, the present invention uses a hardware signal processing board USB direct acquisition and hardware decoding method to greatly compress the real scene acquisition and display time, and its data flow is as follows:

[0041] a) The signal processing board FPGA hardware collects and decodes the camera images in parallel, which takes 16 milliseconds;

[0042] b) The signal processing board outputs the decoded camera image to the display, which takes about 16 milliseconds.

[0043] The total time of the method in the embodiment of the present invention is 32 milliseconds. Compared with the traditional real scene acquisition method, the present invention reduces the delay time by about one third.

[0044] For the acquisition of simulated flight images, the present invention receives the simulated flight images transmitted by the graphics card in real time, and caches them by line during the receiving process, and superimposes, fuses and outputs the simulated flight images and real-scene images by line. The delay is the delay of the hardware caching one line of video, and the delay time is about 0.05 milliseconds.

[0045] In summary, compared with the traditional virtual-reality fusion method, the simulated flight screen display delay of the present invention increases by 0.05 milliseconds, which is negligible, and the real scene display delay is reduced by 15 milliseconds, greatly reducing the time overhead of virtual-reality fusion.

[0046] It should be noted that within the scope of protection defined in the claims of the present invention, the following embodiments can be combined and / or expanded or replaced in any logical way from the above specific implementation methods, such as disclosed technical principles, disclosed technical features or implicitly disclosed technical features.

[0047] Example 1

[0048] A low-latency mixed reality method for simulating a flight mixed reality helmet comprises the following steps:

[0049] S1, building a mixed reality system: connecting the mixed reality helmet to a PC, which has a signal processing board, and the signal processing board is connected to a graphics card in the PC;

[0050] S2, using the signal processing board in the PC to simultaneously receive the simulated flight picture and the real-scene camera picture, and output them after fusion processing; wherein the simulated flight picture is generated by the PC and output to the signal processing board through the video port of the graphics card of the PC; the real-scene camera picture is generated by the camera in the mixed reality helmet and transmitted to the signal processing board; the mixed reality picture is obtained by the signal processing board by mixing the simulated flight picture and the real-scene camera picture, and output to the mixed reality helmet for the user to watch;

[0051] In step S2, the real-scene camera image is collected, which specifically includes sub-steps: the signal processing board collects and decodes the camera image in parallel based on the FPGA hardware; the signal processing board outputs the decoded camera image to the display;

[0052] In step S2, the simulated flight picture is collected, which specifically includes the sub-steps of: receiving the simulated flight picture transmitted by the graphics card in real time, caching it by line during the receiving process, and superimposing, fusing and outputting the simulated flight picture and the real scene picture by line, so that the delay is the delay of one line of video cached by the hardware.

[0053] Example 2

[0054] On the basis of Example 1, in step S2, the fusion process specifically includes the following sub-steps:

[0055] S21, detecting the pixel value of a pixel in the simulated flight picture, and if the pixel value of the pixel is RBG000, replacing it with the camera picture;

[0056] S22, when the flight simulation software is generating the simulated flight picture, it will perform corresponding processing on the picture to be fused, and set the fused aircraft parts to black. When the signal processing board performs virtual and real fusion, it will detect the simulated flight picture pixel by pixel in parallel, and replace its black pixel part with the real camera picture.

[0057] Example 3

[0058] Based on Example 2, the image acquisition, image fusion and image output of the signal processing board are all implemented using FPGA.

[0059] Example 4

[0060] Based on Example 2, for the simulated flight image acquisition, the signal processing board uses the DP interface to receive the image.

[0061] Example 5

[0062] On the basis of Example 2, for real-scene camera image acquisition, the signal processing board acquires the image through a USB interface.

[0063] Example 6

[0064] Based on Example 2, for mixed reality screen output, the signal processing board uses the DP interface output.

[0065] Example 7

[0066] Based on Example 2, the real-scene camera image is generated by the camera in the mixed reality helmet and is transmitted to the signal processing board through the UVC protocol.

[0067] Example 8

[0068] A low-latency mixed reality device for a simulated flight mixed reality helmet comprises a processor and a memory, wherein a program is stored in the memory, and when the program is loaded by the processor, a low-latency mixed reality method for a simulated flight mixed reality helmet as described in any one of Examples 1 to 7 is executed.

[0069] The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves.

[0070] According to one aspect of an embodiment of the present invention, a computer program product or a computer program is provided, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in the above various optional implementations.

[0071] As another aspect, an embodiment of the present invention further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

Claims

1. A low-latency mixed reality method for a simulated flight mixed reality helmet, characterized in that: The following steps are involved: S1, building a mixed reality system: connecting the mixed reality helmet to a PC, which has a signal processing board, and the signal processing board is connected to a graphics card in the PC; S2, using the signal processing board in the PC to simultaneously receive the simulated flight picture and the real-scene camera picture, perform fusion processing and then output; wherein the simulated flight picture is generated by the PC and output to the signal processing board through the video port of the graphics card of the PC; the real-scene camera picture is generated by the camera in the mixed reality helmet and transmitted to the signal processing board; the mixed reality picture is obtained by the signal processing board by mixing the simulated flight picture and the real-scene camera picture, and output to the mixed reality helmet for the user to watch; In step S2, the real-scene camera image is collected, which specifically includes sub-steps: the signal processing board collects and decodes the camera image in parallel based on the FPGA hardware; the signal processing board outputs the decoded camera image to the display; In step S2, the simulated flight picture is collected, which specifically includes the sub-steps of: receiving the simulated flight picture transmitted by the graphics card in real time, caching it by line during the receiving process, and superimposing, fusing and outputting the simulated flight picture and the real scene picture by line, so that the delay is the delay of one line of video cached by the hardware; In step S2, the fusion process specifically includes the following sub-steps: S21, detecting the pixel value of a pixel in the simulated flight picture, and if the pixel value of the pixel is RBG000, replacing it with the camera picture; S22, when the flight simulation software is generating the simulated flight picture, it performs corresponding processing on the picture to be fused, and sets the fused aircraft parts to black. When the signal processing board performs virtual-real fusion, it detects the simulated flight picture pixel by pixel in parallel, and replaces its black pixel part with the real-scene camera picture.

2. The low-latency mixed reality method for a simulated flight mixed reality helmet according to claim 1, characterized in that: The image acquisition, image fusion and image output of the signal processing board are all implemented using FPGA.

3. The low-latency mixed reality method for a simulated flight mixed reality helmet according to claim 1, characterized in that: For the simulated flight picture acquisition, the signal processing board uses the DP interface to receive the pictures.

4. The low-latency mixed reality method for a simulated flight mixed reality helmet according to claim 1, characterized in that: For real-scene camera image acquisition, the signal processing board acquires images through a USB interface.

5. The low-latency mixed reality method for a simulated flight mixed reality helmet according to claim 1, characterized in that: For mixed reality screen output, the signal processing board uses the DP interface for output.

6. The low-latency mixed reality method for a simulated flight mixed reality helmet according to claim 1, characterized in that: The real-scene camera image is generated by the camera in the mixed reality helmet and is transmitted to the signal processing board through the UVC protocol.

7. A low-latency mixed reality device for a simulated flight mixed reality helmet, characterized in that: The invention comprises a processor and a memory, wherein a program is stored in the memory, and when the program is loaded by the processor, the low-latency mixed reality method for a simulated flight mixed reality helmet according to any one of claims 1 to 6 is executed.