FPGA-OCuLink-based real-time audio plug-in processing system

Abstract

The present invention relates to an ultra-low latency real-time audio plug-in processing system and method using an FPGA-based PCI Express OCuLink interface and Direct Memory Access (DMA). The present invention includes a multi-channel audio input unit (100), an FPGA (200), a host interface unit (300), an OCuLink interface unit (400), a DMA control unit (500), a host computer (600), a dedicated ASIO driver (700), a real-time audio plug-in host (800), and an audio output unit (900). The FPGA (200) receives multi-channel audio data and performs channel routing, data alignment, frame construction, synchronization, and TDM-based data processing, and transmits the processed audio data to a host computer (600) through a PCI Express-based OCuLink interface (400). The DMA control unit (500) writes the audio data directly to a memory area of ​​the host computer (600), and the memory area may be configured as a Ring Buffer structure. The host computer (600) transmits the multi-channel audio data to the real-time audio plugin host (800) via a dedicated ASIO driver (700), and the real-time audio plugin host (800) performs real-time audio signal processing by running a VST or VST3-based audio plugin. The real-time audio plugin host (800) may include Live Professor 2 as an example. In addition, the present invention can be coupled with a MADI-based external multi-channel audio device and can process local multi-channel audio inputs and MADI inputs selectively or simultaneously. Accordingly, the present invention improves the processing efficiency of multi-channel audio data by utilizing a PCI Express-based OCuLink direct connection and a DMA-based memory transfer structure between an FPGA and a host computer, and enables ultra-low latency multi-channel audio processing by utilizing an ASIO-based real-time audio plugin environment running on a host computer.

Description

Technology Field

[0001] The present invention relates to multi-channel digital audio processing technology, and specifically to real-time audio processing technology based on FPGA, PCIe, DMA, ASIO, VST, and VST3. Background Technology

[0002] Existing Structure: Audio I / O → USB / Network → OS → Driver → Buffer → CPU → Plugin

[0003] problem:

[0004] Increase transmission path

[0005] Buffering

[0006] OS / Driver Overhead

[0007] Network packet processing

[0008] Possibility of Dropout

[0009] Increased latency due to increased channels The problem to be solved

[0010] The present invention aims to improve the transmission delay, increased delay due to buffering, data transfer bottlenecks between a host computer and an audio processing device, and limitations in real-time audio plugin processing performance that occur in existing network-based audio transmission methods, in an audio system that processes multi-channel audio signals in real time.

[0011] In particular, the present invention has a technical objective of providing a structure that can directly receive and process multi-channel audio data in a Field Programmable Gate Array (FPGA), transmit the processed audio data at high speed to a host computer through a PCI Express-based OCuLink interface, and directly write audio data to a memory area of ​​a host computer using a Direct Memory Access (DMA) controller.

[0012] In addition, the technical task is to transmit multi-channel audio data recorded on the host computer to a real-time audio plugin host via a dedicated ASIO (Audio Stream Input / Output) driver, and to enable the execution of VST or VST3-based audio plugins on the real-time audio plugin host to perform EQ, compressor, reverb, delay, and other real-time audio processing.

[0013] In addition, the technical objective is to provide an open audio processing architecture that utilizes the computational resources of a general-purpose host computer and a real-time audio plugin environment without relying on a separate, expensive dedicated DSP audio console, by using FPGA-based multi-channel audio processing and PCI Express-based OCuLink and DMA data transfer structures.

[0014] In addition, the technical objective is to provide a scalable real-time audio processing system capable of responding to various audio input / output environments and channel configurations by enabling Time Division Multiplexing (TDM) processing for multi-channel audio inputs, Ring Buffer-based data management, and interoperability with external multi-channel audio interfaces based on Multichannel Audio Digital Interface (MADI). means of solving the problem

[0015] Reducing audio data transmission paths and unnecessary processing overhead by utilizing FPGAs, PCIe / OCuLink, and DMA to directly transfer multi-channel audio data to PC memory and allowing the PC to process VST / VST3 plugins in real-time, instead of replacing the existing PC CPU or using an expensive DSP console. Effects of the invention

[0016] To solve the problem, an ultra-low latency real-time audio plug-in processing system according to one embodiment of the present invention may include a multi-channel audio input unit, a Field Programmable Gate Array (FPGA), a PCI Express-based host interface unit, an OCuLink interface unit, a Direct Memory Access (DMA) control unit, a host computer, and a real-time audio plug-in host.

[0017] The multi-channel audio input unit can convert multiple analog audio input signals into digital audio data and provide them to the FPGA. The multi-channel audio input unit may include multiple channels of Analog-to-Digital Converters (ADCs), and in one embodiment, it may be configured to support 64 or more channels of multi-channel audio input.

[0018] The FPGA can receive digital audio data input from the multi-channel audio input and perform channel routing, data alignment, frame processing, and transmission data configuration for multiple audio channels. Additionally, the FPGA can process audio data in Time Division Multiplexing (TDM) mode or convert it into data corresponding to an external audio transmission standard as needed.

[0019] The host interface unit is connected to the FPGA via a PCI Express bus and can be configured to transmit multi-channel audio data processed by the FPGA to a host computer. The host interface unit can use a PCI Express Gen3 x4 or higher interface and can use an OCuLink interface for external connection.

[0020] The DMA control unit can be configured to write audio data generated in the FPGA directly to the memory area of ​​the host computer, while minimizing the process of the processor of the host computer directly copying or processing each audio data.

[0021] The host computer can transfer audio data written to memory by the DMA controller to an audio processing application through a dedicated audio driver. The dedicated audio driver can be configured to transfer the audio data to a real-time audio processing environment using an ASIO (Audio Stream Input / Output) interface.

[0022] The real-time audio plugin host receives audio data delivered through its dedicated audio driver and can perform effect processing, mixing, routing, or other real-time audio signal processing on the audio data using one or more audio plugins in VST or VST3 format. In one embodiment, the real-time audio plugin host may include Live Professor 2, but is not limited thereto, and may be implemented as another audio plugin host that supports VST or VST3 plugins.

[0023] In addition, the FPGA can receive output audio data transmitted from the host computer and route it to multiple output channels, and can output the output audio data through a Digital-to-Analog Converter (DAC) or a subsequent digital audio interface.

[0024] The present invention can reduce data transmission and processing paths that may occur in network-based audio streaming methods or general-purpose USB audio transmission methods by hardware-processing multi-channel audio data generated from an audio input device in an FPGA using such a configuration, directly connecting to a host computer using PCI Express and OCuLink interfaces, and transmitting the data to a memory area of ​​the host computer using DMA.

[0025] In addition, by transmitting audio data delivered via the DMA method to the real-time audio plugin host through a dedicated ASIO driver, the CPU computational resources of the existing general-purpose PC can be concentrated on the real-time audio signal processing of VST or VST3 plugins.

[0026] Accordingly, the present invention combines multi-channel audio input, FPGA-based hardware processing, PCI Express / OCuLink-based high-speed data transmission, DMA-based memory transfer, and VST / VST3-based real-time audio processing into a single system, thereby providing low latency and real-time processing performance required in a multi-channel live audio environment. Brief explanation of the drawing

[0027] Fig. 1 is according to one embodiment of the present invention Overall configuration diagram of an FPGA-PCIe / OCuLink-based ultra-low latency real-time audio plugin processing system am. Fig. 2 The digital audio data input from the multi-channel audio input section Channel routing, alignment, and frame configuration in FPGA This is a structural diagram that is transmitted to the host interface section via [this]. Fig. 3 between the FPGA and the host computer High-speed data transmission architecture based on PCI Express and OCuLink am. Fig. 4 The multi-channel audio data processed in the FPGA Data flow diagram of data directly transferred to the host computer's memory area using the DMA method am. Fig. 5 is the host computer's memory area and Audio data processing architecture between a dedicated ASIO driver and a real-time audio plugin host am. Fig. 6 The audio data delivered through the dedicated ASIO driver Structure diagram processed in real-time by Live Professor 2 and VST / VST3 plugin chainsam. Fig. 7 It starts from multi-channel digital audio input The entire signal flow diagram from FPGA (200), PCIe / OCuLink (300 / 400), DMA (500), ASIO (700), VST / VST3 plugin processing (800), and audio output am. Fig. 8 The multi-channel audio data input from external MADI equipment External extension architecture diagram of data received and routed from the FPGA and transmitted to the host computer via PCIe / OCuLink and DMA am. Fig. 9 It uses local audio input and external MADI-based audio input Hybrid audio processing architecture diagram that processes by inputting into an FPGA selectively or together am. Specific details for implementing the invention

[0028] The best embodiment of the ultra-low latency real-time audio plug-in processing system according to the present invention may be configured to include a multi-channel audio input unit (100), an FPGA (200), a host interface unit (300), an OCuLink interface unit (400), a DMA control unit (500), a host computer (600), a dedicated ASIO driver (700), a real-time audio plug-in host (800), and an audio output unit (900).

[0029] In the best embodiment, the FPGA (200) receives audio data of multiple channels input through the multi-channel audio input unit (100), and the FPGA (200) performs at least one of channel routing, data alignment, frame configuration, synchronization, and multi-channel data packing on the input audio data.

[0030] The FPGA (200) can process multi-channel audio data using Time Division Multiplexing (TDM) and can organize input multi-channel audio data into fixed data frame units and transmit them to the data transmission unit at the end.

[0031] Multichannel audio data processed by the FPGA (200) is transmitted to the host computer (600) through the host interface unit (300) and the OCuLink interface unit (400). The OCuLink interface unit (400) can be configured to transmit multichannel audio data between the FPGA (200) and the host computer (600) using a PCI Express-based high-speed serial interface.

[0032] At this time, the DMA control unit (500) controls the multi-channel audio data generated in the FPGA (200) to be written directly to a pre-allocated memory area of ​​the host computer (600) using the DMA (Direct Memory Access) method, rather than the CPU of the host computer (600) copying the data directly.

[0033] The memory area of ​​the host computer (600) may be configured as a Ring Buffer or Circular Buffer structure, and the DMA controller (500) sequentially writes audio data transmitted from the FPGA (200) to the corresponding Ring Buffer. The host computer (600) can continuously process the input audio data and output audio data while managing the write position and read position of the corresponding Ring Buffer.

[0034] These DMA and Ring Buffer structures reduce unnecessary memory copying and CPU intervention of audio data, and enable continuous real-time processing of multi-channel audio data.

[0035] Audio data recorded on the host computer (600) is transmitted to the real-time audio plugin host (800) by a dedicated ASIO driver (700). The dedicated ASIO driver (700) can manage the input and output of audio data between the audio processing environment of the host computer (600) and the FPGA (200)-based hardware data path.

[0036] The real-time audio plugin host (800) can run one or more VST or VST3-based audio plugins and can perform EQ, compressor, reverb, delay, gate, limiter and other real-time audio signal processing using the plugins.

[0037] In one embodiment of the present invention, Live Professor 2 can be used as the corresponding real-time audio plugin host (800). Live Professor 2 receives multi-channel audio data transmitted through a dedicated ASIO driver (700) and can process audio signals by running one or more VST or VST3 plugins in real time.

[0038] The audio data processed by the plugin can be transmitted again to the output data path of the host computer (600) through the dedicated ASIO driver (700), and can be transmitted to the audio output section (900) via the OCuLink interface section (400) and FPGA (200).

[0039] Accordingly, in the best embodiment of the present invention, the following data processing path is formed.

[0040] Multichannel audio input section (100) --> FPGA (200) --> PCI Express-based OCuLink interface section (400) --> DMA control section (500) --> Host computer memory / Ring Buffer

[0041] --> Dedicated ASIO driver(700) --> Live Professor 2 / VST / VST3(800)

[0042] --> Dedicated ASIO driver (700) --> Host computer (600)

[0043] PCI Express-based OCuLink interface section (400)

[0044] --> FPGA(200) --> Audio output section(900)

[0045] In addition, in the best embodiment of the present invention, a MADI interface may be additionally configured for connection with an external multi-channel audio device. In this case, multi-channel audio data input from an external MADI device may be received and processed by an FPGA (200) and then transferred to a memory area of ​​a host computer (600) through an OCuLink interface unit (400) and a DMA control unit (500).

[0046] In addition, by configuring local audio inputs and MADI inputs to be used selectively or simultaneously, local multi-channel audio inputs and external MADI-based multi-channel audio inputs can be processed in an integrated manner within a single FPGA-based processing structure.

[0047] According to the best embodiment as described above, preprocessing and data alignment of multi-channel audio data can be performed on the FPGA (200), a high-speed direct connection can be made with the host computer (600) via a PCI Express-based OCuLink interface, and audio data can be written directly to the host memory using the DMA controller (500). Accordingly, unnecessary data copying and CPU intervention that may occur during the transmission of multi-channel audio data can be reduced, and real-time audio processing can be performed using an ASIO-based audio plugin environment running on the host computer.

[0048] Furthermore, the present invention is not limited to a specific audio plugin host or a specific VST / VST3 plugin, and can be extended to various real-time audio plugin hosts and audio plugins that support the same ASIO interface.

[0050] Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms.

[0051]

[0052]

[0053] 1. The ultra-low latency real-time audio plugin processing system according to the overall system configuration example may include a multi-channel audio input unit (100), an FPGA (200), a host interface unit (300), an OCuLink interface unit (400), a DMA control unit (500), a host computer (600), a dedicated ASIO driver (700), a real-time audio plugin host (800), and an audio output unit (900).

[0055]

[0056] 2. According to the embodiment of the multi-channel audio input section and the FPGA internal processing structure, the multi-channel audio input section (100) can receive a plurality of analog audio input signals and convert them into digital audio data and provide them to the FPGA (200).

[0057] The multi-channel audio input section (100) may include a plurality of ADCs and may be configured to support 64 or more channels of multi-channel audio input. In addition, different sampling frequencies and bit depths may be used depending on the required sound quality and processing environment of the input audio.

[0058] The FPGA (200) can receive digital audio data provided by the multi-channel audio input unit (100) and perform routing of audio channels, channel alignment, data frame configuration, and generation of transmission data.

[0059]

[0061] 3. The FPGA (200) of the PCI Express / OCuLink embodiment between the FPGA and the host computer can process multiple channels of audio data in parallel in hardware and can be configured as PCI Express transmission data to transmit the input audio data to the host computer (600).

[0062] The FPGA (200) receives multi-channel audio data input via TDM and, if necessary, converts it into an external digital audio transmission standard such as MADI or processes MADI data received from the outside within the FPGA.

[0063] The host interface section (300) is connected to the FPGA (200) via a PCI Express bus and provides an interface for transmitting audio data generated by the FPGA (200) to the host computer (600).

[0064] The OCuLink interface section (400) can connect the FPGA (200) and the host computer (600) via an external cable using a PCI Express-based high-speed serial interface.

[0065]

[0066] 4. In an embodiment of data flow that is directly transferred to host memory using the DMA method, a PCI Express Gen3 x4 interface may be used, and accordingly, a high-speed data transmission path for multi-channel audio data may be provided between the FPGA (200) and the host computer (600).

[0067] The DMA control unit (500) may be configured to directly transfer audio data generated in the FPGA (200) to the memory area of ​​the host computer (600).

[0068] More specifically, the FPGA (200) generates audio data in fixed audio frame units, and the DMA controller (500) can write the corresponding audio data to a pre-allocated memory area of ​​the host computer (600).

[0069] The memory area may be composed of one or more ring buffers or circular buffer structures, and the FPGA (200) sequentially writes input audio data via DMA transfer, and the audio processing software of the host computer (600) can sequentially read the audio data written to the memory area.

[0070]

[0071] 5. According to an embodiment of the structure between a dedicated ASIO driver and a real-time audio plugin host, the central processing unit of the host computer (600) can reduce the process of directly copying each audio sample or audio frame for the transmission of audio data.

[0072] The host computer (600) may run on a Windows-based operating system, and in one embodiment, may be configured to operate in a Windows 11 IoT Enterprise LTSC environment.

[0073] A dedicated ASIO driver (700) can be installed on the host computer (600).

[0074] The dedicated ASIO driver (700) can transmit audio data written to the memory of the host computer (600) by the DMA controller (500) to an audio processing application.

[0075] The dedicated ASIO driver (700) can manage audio input buffers and audio output buffers and perform audio data transfer between the FPGA (200) and the real-time audio plugin host (800).

[0076]

[0077] 6. In the embodiment of the LIVE PROFESSOR 2 and VST / VST3 plugin chain processing structure, the real-time audio plugin host (800) can receive audio data transmitted through a dedicated ASIO driver (700) and perform real-time audio signal processing using one or more VST or VST3 plugins.

[0078] The real-time audio plugin host (800) may include Live Professor 2.

[0079] Live Professor 2 receives multi-channel audio data through a dedicated ASIO driver (700) and can perform EQ, compressor, reverb, delay, other effect processing or mixing processing using one or more VST or VST3 plugin chains configured by the user.

[0080] However, the real-time audio plugin host (800) is not limited to Live Professor 2 and can be implemented as another real-time audio processing host that supports VST or VST3 plugins.

[0081] Additionally, audio data processed by the real-time audio plugin host (800) can be transmitted to the output buffer of the host computer (600) via a dedicated ASIO driver (700) and to the FPGA (200) via a PCI Express / OCuLink interface.

[0082] The FPGA (200) routes the received output audio data to multiple output channels and, if necessary, outputs it through a DAC or an external digital audio output interface.

[0083] In such a configuration, the entire audio data path of the present embodiment can be formed as follows.

[0084] Multichannel audio input section (100)

[0085] → FPGA(200)

[0086] →PCI Express / OCuLink(400)

[0087] → DMA control unit (500)

[0088] → Host computer memory

[0089] → Dedicated ASIO driver (700)

[0090] → Real-time audio plugin host (800)

[0091] → VST / VST3 plugin

[0092] → Dedicated ASIO driver (700)

[0093] → PCI Express / OCuLink(400)

[0094] → FPGA(200)

[0095] → Audio output section (900).

[0096] In this embodiment, a PCI Express-based direct connection structure can be used instead of using network-based audio streaming in the transmission path of the audio data.

[0097] Accordingly, the processing burden that may occur during the process of configuring multi-channel audio data into network packets and performing network address and network transmission processing can be reduced.

[0098] In addition, by transferring audio data between the FPGA (200) and the host computer (600) based on DMA, the data copying burden of the central processing unit is reduced, and the CPU resources of the host computer (600) can be concentrated on real-time audio computation of the VST / VST3 plugin.

[0099] In the above embodiment, an audio buffer size of 32 samples or 64 samples can be used, and the buffer size can be changed depending on the processing performance and stability of the system.

[0100]

[0101] 8. In an embodiment of a MADI-based external expansion audio input configuration, an external MADI input can be connected to the FPGA (200). Multichannel digital audio data input through the external MADI input can be received and routed by the FPGA (200) and then transmitted to a host computer (600) via a PCI Express / OCuLink interface.

[0102] Accordingly, local audio inputs and far-stage inputs can be integrated into a single FPGA-based audio processing system.

[0103] In addition, the system of the present invention is not limited to 64 channels and can be expanded to support 32, 64, or 128 or more audio channels depending on the processing capabilities of the FPGA and PCI Express interface.

[0104]

[0105] 9. Hybrid configuration: Local input + MADI input (optional / simultaneous use) In the embodiment, Live Professor 2 corresponds to an example of a real-time audio plugin host, and the technical concept of the present invention is not limited to specific software.

[0106] Therefore, even if separate proprietary audio plugin host software is implemented in the future, the same

[0107] An audio data transfer structure based on FPGA-PCIe / OCuLink-DMA-ASIO can be applied.

Claims

Claim 1 In an ultra-low latency audio plugin processing system for processing multi-channel audio signals in real time, the system comprises: a multi-channel audio input unit for receiving digital audio data from multiple audio input channels; a Field Programmable Gate Array (FPGA) that performs channel routing, data alignment, and audio frame processing on the digital audio data received from the multi-channel audio input unit; a host interface unit connected to the FPGA via a PCI Express bus and transmitting the multi-channel audio data processed by the FPGA to a host computer; an OCuLink interface unit for performing PCI Express-based data transmission between the host interface unit and the host computer; a DMA controller unit that controls the direct writing of the multi-channel audio data processed by the FPGA to a memory area of ​​the host computer using a Direct Memory Access (DMA) method; and a dedicated audio driver that transmits the multi-channel audio data written to the memory area of ​​the host computer to a real-time audio plugin host via an Audio Stream Input / Output (ASIO) interface. A real-time audio plugin processing system comprising: a real-time audio plugin host that processes multi-channel audio data transmitted through a dedicated audio driver in real time using one or more VST or VST3 audio plugins; wherein multi-channel audio data processed in an FPGA is transmitted to a host computer via a PCI Express-based OCuLink interface and configured to be written directly to a memory area of ​​the host computer by a DMA controller. Claim 2 A system according to claim 1, wherein the FPGA selectively or simultaneously receives at least one of a local multi-channel audio input, a Time Division Multiplexing (TDM) input, and an external Multichannel Audio Digital Interface (MADI) audio input to perform channel routing, data alignment, frame construction, and synchronization processing. Claim 3 A system according to claim 1, wherein data transfer between the FPGA and the host computer is performed via a PCI Express Gen3 ×4-based OCuLink interface, and the DMA controller directly writes audio data processed by the FPGA to a plurality of ring buffers or a circular buffer area of ​​the host computer memory. Claim 4 A system according to claim 1, wherein a dedicated audio driver transmits audio data written in a memory area to a real-time audio plugin host via an ASIO interface, and the real-time audio plugin host performs real-time audio processing using one or more VST or VST3 plugin chains. Claim 5 A method for ultra-low latency audio plugin processing for processing multi-channel audio signals in real time, comprising: a first step of receiving audio data of multiple channels; a second step of performing channel routing, data alignment, and audio frame processing on the input audio data in an FPGA; a third step of transmitting the multi-channel audio data processed in the FPGA to a host computer via a PCI Express-based OCuLink interface; a fourth step of directly writing the multi-channel audio data to a memory area of ​​the host computer using a DMA method; a fifth step of transmitting the audio data written to the memory area to a real-time audio plugin host via a dedicated ASIO driver; a sixth step of processing the audio data in real time using one or more VST or VST3 plugins in the real-time audio plugin host; and a seventh step of transmitting the processed audio data to an audio output unit via an ASIO driver, a PCI Express-based OCuLink interface, and an FPGA; characterized by comprising: an ultra-low latency real-time audio plugin processing method. Claim 6 A method according to claim 5, wherein the first step of receiving audio data receives at least one of a local audio input and an external MADI audio input and transmits it to a common audio processing path, and the fourth step of directly writing in a DMA manner writes audio data by circulating read and write pointers in a circular buffer structure of a host computer memory.

Images