A cross-platform flight display control video real-time injection method based on hardware video direct connection and asynchronous three-buffering
By using hardware video direct connection and asynchronous triple buffering, the problems of screen tearing and latency in flight simulators were solved, achieving efficient cross-platform video stream rendering, reducing CPU load and supporting multi-platform deployment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- AVIC AVIATION SIMULATION SYST CO LTD
- Filing Date
- 2026-03-16
- Publication Date
- 2026-06-12
AI Technical Summary
In flight simulators, existing technologies suffer from issues such as screen tearing and delays caused by asynchronous simulation clocks, CPU bandwidth consumption during high-definition video stream conversion, and system fragmentation caused by underlying graphics API calls during cross-platform deployment.
By employing a hardware video direct connection and asynchronous triple buffering approach, raw stream data is intercepted at the hardware driver layer, a cross-process shared memory structure is constructed, and the portable interface of the cross-platform graphics engine is used to perform lock-free synchronization and rendering of the video stream, avoiding direct calls to the underlying graphics API.
It enables real-time injection of high-fidelity, tear-free video streams, frees up CPU resources, supports seamless cross-platform deployment, and reduces system maintenance costs.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention relates to the field of flight simulator and virtual reality simulation technology, and in particular to a method and system for integrating high-resolution video streams with extremely low latency and tear-free real-time performance across processes using a hardware video capture card in a flight simulator display and control simulation system. Background Technology
[0002] In the development of modern flight simulators and integrated tactical training systems, the realism of the cockpit multifunction display (MFD) simulation directly determines the training effect. In order to simulate the electro-optical pods (EO / IR), synthetic vision systems (SVS), or high-resolution radar images of real fighter jets with high fidelity, the simulation system usually needs to use a dedicated video capture card to input the dynamic high-definition video stream from the external image generator (IG) or sensor simulation into the on-board display and control simulation software (such as the instrument interface developed based on vector engines such as VAPS XT).
[0003] The following technical bottlenecks were encountered during this integration process:
[0004] Screen tearing and latency caused by simulation clock asynchrony: Hardware capture cards trigger interrupt callbacks at extremely high fixed frequencies (e.g., 60Hz), while the main loop of the display and control simulation software is often affected by other simulation logic, resulting in frame rate fluctuations. Traditional cross-process buffering mechanisms are prone to causing callback blocking at the capture end, leading to significant screen tearing or transmission latency, disrupting the immersive experience of flight simulation, and even causing simulator sickness.
[0005] The CPU bandwidth bottleneck of high-fidelity raw stream conversion: The raw YUYV format output by the capture card must be converted in real time to the RGBA format required by the rendering engine. Real-time conversion at 1920×1080 resolution will greatly consume CPU bus bandwidth, resulting in insufficient computing power left for the logic calculation of other simulation systems.
[0006] State machine pollution and cross-platform fragmentation caused by low-level graphics APIs: In the flight simulator field, to meet the diverse delivery needs of different projects and customers, display and control simulation software must have the ability to be flexibly deployed across heterogeneous operating systems (such as Windows and Linux computer nodes). Current technology, which directly hardcodes calls to platform-specific low-level graphics APIs (such as Windows-specific DirectX or Linux-specific OpenGL instructions) for video texture uploading in the display and control components, not only easily disrupts the complex simulation graphics engine's own context state machine, leading to abnormal rendering of other instrument layers, but also causes the display and control code to completely lose cross-platform portability due to the significant differences in low-level graphics library standards between heterogeneous operating systems. This forces development teams to rewrite and independently maintain multiple sets of low-level rendering logic for different operating system delivery environments, resulting in severe system code fragmentation and greatly increasing the costs of customized development, project implementation, and subsequent maintenance of flight simulator systems. Summary of the Invention
[0007] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a cross-platform real-time injection method for flight display and control video based on direct hardware video connection and asynchronous triple buffering. This method establishes a complete link from the interception of the underlying hardware video stream to the asynchronous shared memory transmission, and completely replaces the underlying graphics API calls with the Portability Layer built into the cross-platform graphics engine, achieving smooth, high-fidelity, and cross-platform injection of multi-channel video streams in heterogeneous simulation nodes.
[0008] A cross-platform real-time flight display and control video injection method based on hardware video direct connection and asynchronous triple buffering mainly includes the following steps:
[0009] Step 1: Simulate low-latency interception of raw video stream at the hardware layer.
[0010] During the video capture process, the video capture card hardware is initialized, and the resolution is set to match the target emulation device. Raw stream data callback functions are directly registered at the hardware driver layer, bypassing the operating system's standard multimedia framework to capture externally generated raw YUYV video frame data with zero abstraction overhead.
[0011] Step 2: Construct an asynchronous triple-buffered shared memory structure to prevent false sharing across multiple cores.
[0012] In the operating system, cross-process shared memory is allocated, with its header defined as a packet header containing synchronization control and metadata, followed by three consecutive image data buffers of equal size. The packet header includes: read index record bits, write index record bits, image width and height and data size, specific status feature code, and padding bits used to align CPU cache lines to prevent false sharing of cache when the distributed simulation system is running concurrently with multiple cores.
[0013] Step 3: Parallel color conversion and zero-copy injection based on pre-computed lookup tables.
[0014] When the acquisition process triggers the raw stream callback: the current target buffer is selected based on the read / write index record bits using a lock-free exclusion method; using a static three-dimensional high-precision lookup table pre-generated based on color space conversion formulas and floating-point precision, a macro-pixel merging strategy (extracting 4 bytes of YUYV data each time) is adopted in a single memory traversal loop to retrieve and complete RGBA data in parallel, and directly write it sequentially to the target buffer; then the write index record bits are atomically updated.
[0015] Step 4: The display and control simulation process performs platform-independent resource initialization based on feature codes.
[0016] After the display control simulation process starts, the shared memory is dynamically mapped and the packet header is read during the drawing callback cycle of the graphics component. Dynamic allocation of the local pixel cache space is triggered only when the feature code in the packet header matches a preset value (indicating that the underlying hardware channel is ready) and the image size is consistent. During this process, direct calls to the underlying graphics API are prohibited. Instead, the cross-platform graphics engine's own memory update object and video memory allocation object are instantiated, and the platform porting layer of the graphics engine automatically completes the internal video memory alignment and packaging.
[0017] Step 5: Perform lock-free reading and cross-platform push rendering of video memory.
[0018] In the subsequent rendering process, lock-free comparison of read and write indices is used to lock the latest video frame and copy it to the local cache. Then, the cross-platform graphics engine's natively encapsulated memory push interface is called to asynchronously upload the data to the simulation graphics context, and the graphics engine's native graphics geometry interface is called to perform image rendering.
[0019] The beneficial effects of this invention are as follows:
[0020] 1) Eliminate tearing and latency: Through pure lock-free asynchronous triple buffering synchronization and direct connection to the underlying raw stream, the hardware acquisition clock and display control rendering clock are completely decoupled.
[0021] 2) Free up CPU computing power: The use of static floating-point lookup tables and macro-pixel merging processing significantly reduces the memory bandwidth consumption of real-time decoding of high-definition video streams.
[0022] 3) Achieve "write once, simulate on multiple platforms": Force the use of the cross-platform graphics engine's built-in portable abstract interface to take over video memory resources and data upload, completely avoiding the state machine pollution of the graphics engine caused by directly calling the underlying graphics API, so that the same code can be seamlessly deployed on various heterogeneous systems.
[0023] 4) Highly robust error prevention mechanism: The rendering function uses a lazy initialization probe design based on the feature code to avoid system-level memory crashes caused by sudden changes in external hardware status or misalignment of startup timing. Attached Figure Description
[0024] Figure 1 is a schematic diagram of the overall architecture and data flow of a cross-platform flight display and control video real-time injection method based on hardware video direct connection and asynchronous triple buffering provided by an embodiment of the present invention;
[0025] Figure 2 is a core execution flowchart of a cross-platform flight display and control video real-time injection method based on hardware video direct connection and asynchronous triple buffering provided by an embodiment of the present invention;
[0026] Figure 3 is a schematic diagram of the physical structure layout and lock-free anti-counterfeiting shared read / write mechanism of the asynchronous triple-buffered shared memory provided in an embodiment of the present invention;
[0027] Figure 4 is a schematic diagram of the parallel color conversion principle based on macro-pixel extraction and static three-dimensional floating-point lookup table provided in an embodiment of the present invention;
[0028] Figure 5 is a timing diagram of the display control simulation process based on feature code to perform lazy initialization and cross-platform video memory push according to the embodiment of the present invention;
[0029] Figure 6 is a physical hardware deployment topology diagram of the real-time injection method for flight simulator display and control video provided in an embodiment of the present invention. Detailed Implementation
[0030] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.
[0031] This embodiment uses a cross-platform graphics engine (such as VAPS XT) as the development basis for the display and control simulation software, and combines video acquisition hardware to explain in detail the execution process of the cross-platform flight display and control video real-time injection method.
[0032] I. System Physical Deployment and Overall Architecture
[0033] As shown in Figures 1 and 6, the cross-platform flight display and control video real-time injection method based on hardware video direct connection and asynchronous triple buffering provided by the embodiments of the present invention is physically deployed in the display and control simulation computer node of the flight simulator.
[0034] The dynamic high-definition video stream generated by the external visual or sensor imaging simulation computer is transmitted via physical cables to a hardware video capture card plugged into a high-speed bus (such as a PCIe) slot on the motherboard of the display and control simulation computer. The multi-core CPU of the display and control simulation computer runs the "video capture process" and the "cross-platform display and control simulation process" concurrently, and the two interact with each other without locks through "asynchronous triple-buffered shared memory" residing in the system main memory (RAM). Finally, the display and control simulation process calls the portable native interface of the cross-platform graphics engine to push the data to the video memory of the display and control simulation computer's discrete graphics card (GPU), and outputs the rendered image on the cockpit display (such as MFD).
[0035] II. Detailed Explanation of Core Method Execution Steps
[0036] As shown in Figure 2, the real-time injection method in this embodiment strictly follows steps one through five in a loop:
[0037] Step 1: Low-latency interception of raw video stream at the simulation hardware layer
[0038] When the video capture process starts, the video capture card initialization interface is called first to complete the initialization of the hardware and the capture thread.
[0039] To avoid data copying and abstraction layer latency caused by the standard multimedia framework of the operating system, this embodiment directly calls the underlying hardware control interface in the acquisition thread to force the physical channel to open. During this process, the system can dynamically set the target resolution (e.g., 1920×1080) and frame rate (e.g., 60Hz) according to the actual output or configuration file of the external video source.
[0040] Subsequently, the system calls the hardware driver-level callback registration interface (such as registering a Raw Direct Callback function) to directly register the raw stream data callback function to the direct memory access (DMA) controller in the hardware driver layer. Thus, when the video capture card hardware collects a frame of raw YUYV (4:2:2 format) video stream, it triggers a low-level callback via DMA, directly pushing the raw data pointer to the application layer, achieving zero-copy capture.
[0041] Step 2: Construct an asynchronous triple-buffered shared memory structure to prevent spoofing.
[0042] As shown in Figure 3, in order to achieve high-frequency lock-free communication across processes, the video acquisition process allocates an asynchronous shared memory that is visible to the entire system through the system API. The total size of this shared memory is calculated based on the currently dynamically configured resolution: Total size = Packet header size + (Image width × Image height × 4 bytes × 3 data buffers).
[0043] The header of the shared memory is strictly defined as a packet header structure (such as a Header). This packet header contains:
[0044] 1) Read / write synchronization flags: These include the index of the currently being read (cur_reading_idx) and the index of the most recently written item (cur_latest_idx). Both are declared as volatile to ensure memory visibility of variable modifications in multi-core CPU architectures.
[0045] 2) Image attribute metadata: This includes image width, image height, and the size of a single frame. This data size is dynamically assigned based on the actual initialized resolution.
[0046] 3) Status Feature Code: A specific feature verification code (such as magic feature code) is defined and a specific security value (such as 0x2026) is assigned when the acquisition is ready, which serves as a security verification credential for the subsequent display and control probe.
[0047] 4) Anti-fake shared byte padding: To prevent false cache sharing in multi-core concurrent distributed simulation systems, a placeholder padding bit is explicitly added to the end of the packet header (e.g., declaring padding bytes). Its core purpose is to pad the total number of bytes of the entire control body, so that it is strictly aligned to the typical cache line boundary of modern CPUs in physical memory.
[0048] Step 3: Parallel Color Conversion and Lock-Free Writing Based on Pre-computed Lookup Tables
[0049] Referring to Figure 4, after obtaining the YUYV raw stream by triggering the underlying raw stream callback, the system performs parallel conversion and atomic writing:
[0050] Sub-step 3.1 (Target Selection): Obtain the current read / write index from the packet header, execute the lock-free exclusion method, traverse the three index pools 0, 1, and 2, remove the index that is being read or the one that has just been written, and select the remaining single free index as the target write buffer for this loop.
[0051] Sub-step 3.2 (Pre-calculation lookup table): In order to overcome the bottleneck of CPU floating-point operation bandwidth, the system pre-generates a static three-dimensional floating-point lookup table (such as YUV2RGB_LUT) with saturation truncation accuracy based on the color space conversion formula during the initialization phase.
[0052] Sub-step 3.3 (Macro-pixel parallel conversion): During memory traversal, the algorithm extracts data in units of macro-pixels, setting a loop step size. Each time, it continuously extracts the shared U and V components and the independent Y0 and Y1 components. Subsequently, it uses parallel addressing of the static lookup table to obtain the RGB components of two pixels at once, appends an Alpha channel constant, synthesizes them into RGBA data, and then directly and sequentially appends it to the selected free area of the target buffer. After writing, it atomically updates the cur_latest_idx in the packet header to the current target buffer index, completing the lock-free submission of the new frame.
[0053] Step 4: Platform-independent lazy initialization of the display and control simulation process
[0054] Referring to Figure 5, the consumer-side component of this invention is a display and control simulation component developed based on a cross-platform graphics engine. To prevent the underlying state machine from crashing due to system startup timing errors, this component employs a lazy-safe initialization mechanism.
[0055] In the periodic graphics drawing callbacks triggered by the display and simulation component in each frame (such as the vDraw cycle in VAPS XT):
[0056] The system first maps shared memory and checks the validity of its pointers. If valid, it extracts the packet header for double security checks: 1) checking if the feature code verification passes (e.g., checking if the magic code matches); 2) checking if the metadata such as width and height recorded in the packet header matches the expected physical dimensions of the simulated instrument. If either condition is not met, it indicates that the underlying hardware link is not yet ready, and the component actively suspends the current callback cycle to avoid triggering null pointer exceptions or memory out-of-bounds exceptions.
[0057] The system takes over graphics resource allocation only when the verification passes and the component has not yet been initialized: it strictly avoids directly calling low-level (such as OpenGL / DirectX) APIs (such as the texture generation instruction glGenTextures), and instead instantiates the cross-platform graphics engine's built-in memory image definition object (such as instantiating the vxtRTImageUpdate_Memory object), calls its portability initialization method (such as vInit) to perform internal memory alignment and pixel wrapping; at the same time, it calls the cross-platform graphics engine's safe allocation interface (such as vAllocate) to dynamically allocate texture space in the graphics card, and sets the initialization success flag to true.
[0058] Step 5: Perform lock-free read and cross-platform memory push rendering
[0059] In subsequent rendering loops after initialization, the display control process compares the write index with the read index. When the two are inconsistent (indicating that the hardware has pushed in a new frame), the latest frame is locked and copied, and the single-frame data is zero-copied to the local pixel buffer.
[0060] During the actual rendering phase, the graphics control component strictly prohibits the use of platform-dependent low-level texture update instructions (such as glTexSubImage2D). Instead, it forces the use of native cross-platform graphics engine memory push interfaces (such as calling the vPush interface) to trigger asynchronous memory upload. Subsequently, considering the rotation properties of the image itself: if there is a rotation angle, the native API interfaces are called in sequence (such as calling vPushTrans to push onto the stack, vTranslate to translate, and vRotate to rotate in sequence), and finally the native rendering interface (such as vDrawImage) is called to hardware-accelerate the drawing of the video texture onto the screen surface. Then, the pop interface (such as vPopTrans) is called to restore the rendering state machine.
[0061] Through the complete takeover of the aforementioned Portability Layer, the display and control simulation code completely shields the underlying call differences for heterogeneous graphics standards, enabling unmodified cross-platform deployment in any operating system environment.
[0062] Based on the same inventive concept as the aforementioned cross-platform flight display and control video real-time injection method, this embodiment of the invention also provides a virtual device / system corresponding to the method. The functional modules in this system correspond to the steps in the aforementioned method embodiments. Specifically, the system includes:
[0063] The hardware video direct-connection acquisition module is used to initialize the video capture card hardware, register the underlying raw stream data callback function to capture external raw YUYV video frames, and convert them into RGBA data in parallel during memory traversal and write them to the target buffer.
[0064] An asynchronous triple-buffered shared memory module includes a header and three equally sized image data buffers. The header uses read / write index record bits to achieve lock-free synchronization between processes, uses status feature codes to provide hardware readiness probes, and includes alignment padding bits to prevent false sharing among multiple cores.
[0065] The cross-platform graphics control injection rendering module is used to dynamically map the asynchronous triple-buffered shared memory module during the graphics drawing callback cycle, perform lazy and safe initialization of graphics resources based on the status feature code, and call the native portability interface of the cross-platform graphics engine to complete local pixel buffer copying, asynchronous video memory push and hardware accelerated rendering.
[0066] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned cross-platform flight display and control video real-time injection method based on hardware video direct connection and asynchronous triple buffering.
[0067] This invention also provides an electronic device, including: a processor; and a memory for storing a computer program; wherein, when the processor executes the computer program, it implements the above-described cross-platform flight display and control video real-time injection method based on hardware video direct connection and asynchronous triple buffering.
[0068] The specific implementation principles and technical details of each module within the aforementioned system / device and equipment are completely consistent with those of the aforementioned method embodiments, possessing the same beneficial technical effects. To avoid repetition, they will not be described again here. The above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A cross-platform real-time injection method for flight display and control video based on hardware video direct connection and asynchronous triple buffering, characterized in that, Includes the following steps: Step 1: The video capture process initializes the video capture card hardware and directly registers a raw stream data callback function in the hardware driver layer to intercept the raw YUYV video frame stream data passed in from the outside; Step 2: The video acquisition process constructs a cross-process asynchronous triple-buffered shared memory in the operating system. The header of the shared memory is a packet header containing synchronization control and verification metadata. Three image data buffers of equal size are allocated consecutively after the packet header. Step 3: When the raw stream data callback function is triggered, the video acquisition process selects the target buffer from the three image data buffers based on the lock-free exclusion method; using a pre-computed static three-dimensional floating-point lookup table, the original YUYV video frame stream data is extracted in memory traversal and converted into RGBA data in parallel, and written sequentially to the target buffer; then the write index record bit in the packet header is atomically updated. Step 4: After the display and control simulation process starts, the shared memory is dynamically mapped and the packet header is read during the drawing callback cycle of the cross-platform graphics engine; if and only if the status feature code in the packet header matches the preset check value and the image size attribute is consistent with the expectation of the display and control simulation process, the native portable memory update interface and video memory allocation interface of the cross-platform graphics engine are called to perform dynamic allocation and initialization of local pixel buffer and video memory texture; Step 5: The display and control simulation process compares the write index record bit and the read index record bit in the packet header; if they are inconsistent, the read index record bit is updated to lock the latest video frame, the latest video frame is copied to the local pixel buffer, and the native portable video memory push interface and graphics drawing interface of the cross-platform graphics engine are called to perform cross-platform rendering of the display and control screen.
2. The method according to claim 1, characterized in that, The packet header constructed in step two includes: The read index record bit is used to indicate the buffer index that the display and control simulation process is currently reading; The write index record bit is used to indicate the latest buffer index that the video acquisition process has completed writing; Image width and image height attributes; Single-frame RGBA data size attribute; Status feature code, used to characterize the hardware probes that indicate the underlying video capture hardware and shared memory channels are initialized and ready; And byte padding bits, which are used to strictly align the total size of the packet header to an integer multiple of the CPU cache line size, in order to prevent false cache sharing in a distributed multi-core simulation concurrent architecture.
3. The method according to claim 1, characterized in that, The specific process of extraction, conversion, and writing in step three is as follows: During the initialization phase of the video acquisition process, a static three-dimensional floating-point lookup table for color conversion is established. The lookup table is generated in advance based on the color space conversion formula, combined with floating-point operations and saturation truncation logic. In the raw stream data callback function, memory traversal is performed, and each loop extracts 4 bytes of macro-pixel raw YUYV data in parallel. The 4 bytes contain independent Y components of two adjacent pixels and shared U and V components. The static three-dimensional floating-point lookup table is accessed twice consecutively to obtain the R, G, and B components corresponding to the two adjacent pixels. A fixed Alpha channel transparency value is appended, and the resulting 8 bytes of RGBA data are directly written into the consecutive physical addresses of the target buffer.
4. The method according to claim 1, characterized in that, The rule for selecting the target buffer in step three based on the lock-free exclusion method is as follows: Obtain the read index record bits and write index record bits from the packet header; From the three preset buffer indices, remove the indices that have the same record bit value as the read index and remove the indices that have the same record bit value as the write index, and select the remaining unique index as the target buffer for this raw stream data writing.
5. The method according to claim 1, characterized in that, The initialization process during the drawing callback cycle in step four includes: Each time the graphics drawing callback function is triggered, the validity of the shared memory mapping pointer is checked; If the status feature code does not match the preset verification value, or the externally input image size attribute does not match the expected size of the display and control simulation process, the display and control simulation process actively suspends the video texture allocation operation of the current frame and exits the current callback cycle to prevent cross-platform graphics engine context pollution and illegal memory access. Only when the double check passes and initialization is not yet complete, the native abstract memory image object of the cross-platform graphics engine is instantiated to complete the video memory alignment and pixel wrapping, and the initialization success flag is marked.
6. The method according to claim 1, characterized in that, The specific limitations of calling the portable video memory push interface and the graphics drawing interface in step five are as follows: During the rendering process of uploading the local pixel buffer to the video memory texture and performing geometric mapping, the video control simulation process is prohibited from directly calling the underlying OpenGL or DirectX graphics application interfaces of the operating system. By forcibly calling the native abstract push interface of the cross-platform graphics engine to trigger asynchronous video memory upload, the underlying graphics state machine is isolated, enabling the display control simulation software code to be compiled and deployed across platforms without modification in Windows systems and Linux or real-time operating system environments.
7. A cross-platform real-time injection method for flight display and control video based on hardware video direct connection and asynchronous triple buffering, characterized in that, include: The hardware video direct-connection acquisition module is used to initialize the video capture card hardware, register the underlying raw stream data callback function to capture external raw YUYV video frames, and convert them into RGBA data in parallel during memory traversal and write them to the target buffer. An asynchronous triple-buffered shared memory module includes a header and three equally sized image data buffers. The header uses read / write index record bits to achieve lock-free synchronization between processes, uses status feature codes to provide hardware readiness probes, and includes alignment padding bits to prevent false sharing among multiple cores. The cross-platform graphics control injection rendering module is used to dynamically map the asynchronous triple-buffered shared memory module during the graphics drawing callback cycle, perform lazy and safe initialization of graphics resources based on the status feature code, and call the native portability interface of the cross-platform graphics engine to complete local pixel buffer copying, asynchronous video memory push and hardware accelerated rendering.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the cross-platform flight display and control video real-time injection method based on hardware video direct connection and asynchronous triple buffering as described in any one of claims 1 to 6.
9. An electronic device, characterized in that, include: Processor; and Memory, used to store computer programs; Wherein, when the processor executes the computer program, it implements the cross-platform flight display and control video real-time injection method based on hardware video direct connection and asynchronous triple buffering as described in any one of claims 1 to 6.