Method and device for processing multi-task parallel data stream based on FPGA (Field Programmable Gate Array)

Abstract

The invention discloses a multi-task parallel data stream processing method and device based on an FPGA (Field Programmable Gate Array), which are applied to the technical field of data processing. Carrying out normalization processing on the decoded data to generate a data stream in a unified format; performing coding and ID identification code embedding processing on the normalized data to generate to-be-stored data with an identifier; carrying out FIFO (First In First Out) buffering and write-in priority judgment on the coded data to generate priority queue data; dDR writing and BANK scheduling processing are carried out on the priority queue data, and stably stored parallel data are generated; carrying out reading and output coordination processing on DDR storage data to generate a continuous output data stream, and triggering a reading application by decoding ID identification code positioning data, reading FIFO according to depth scheduling and monitoring data volume by channel temporary storage FIFO; and performing dynamic bandwidth allocation and energy efficiency optimization on the whole data stream processing process to generate an efficient parallel processing result.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and device for processing multi-task parallel data streams based on FPGA. Background Art

[0002] The traditional FPGA-based video and audio processing solution has the following problems: Disadvantages of the storage method: Video and audio are cached in frames. When the DDR rate of multi-channel parallel video and audio streams is insufficient, data loss is likely to occur.

[0003] Uneven bandwidth usage: Some channels may occupy DDR bandwidth for a long time, resulting in less storage capacity on other data channels and affecting overall processing efficiency.

[0004] Limited channel support: The original solution can only support 5 channels of video stream data and 5 channels of audio stream data, which is difficult to meet the needs of multi-channel parallel processing.

[0005] Excessive bandwidth usage: Taking 1920*1080 image data as an example, the traditional method is to burst 512 bits of data at a time, which requires about 98,743 bursts. This will occupy the DDR bandwidth for a certain period of time, which is not conducive to the storage of data in other channels. Summary of the Invention

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: A processing method based on FPGA multi-task parallel data flow, comprising: constructing a multi-format original data receiving and decoding system, wherein the original data includes TMDS signal, I2S serial audio data, BT1120 internal synchronization data flow, SERDES SDI high-speed serial signal and self-defined SPI protocol data, and the decoding operation covers video synchronization signal separation, SDI rate and resolution automatic identification, SPI protocol command code analysis and audio 24-bit data flow generation; performing normalization processing on the decoded data to generate a unified format data flow and adding an adjustment factor; performing encoding and ID recognition code embedding processing on the normalized data to generate identified data to be stored; performing FIFO buffering and write priority judgment on the encoded data to generate priority queue data, triggering a preset priority application when the cache reaches a preset threshold; performing DDR writing and BANK scheduling processing on the priority queue data to generate stable storage parallel data, wherein the writing arbitrator is scheduled according to the priority, the BANK switch realizes frame-level read-write separation, and 4 pieces of DDR3 are parallelly realized to achieve 512-bit width burst writing; performing reading and output coordination processing on the DDR storage data to generate continuous output data flow, wherein the data is positioned by decoding the ID recognition code, the reading FIFO is read according to the depth, and the channel temporary storage FIFO monitors the data volume to trigger the reading application; and performing dynamic bandwidth allocation and energy efficiency optimization on the whole data flow processing process to generate efficient parallel processing results.

[0007] The application discloses a processing device based on FPGA multi-task parallel data flow, and the device comprises a construction module and a processing module; the construction module is used for constructing a multi-format original data receiving and decoding system, wherein the original data comprises TMDS signals, I2S serial audio data, BT1120 internal synchronization data flow, SERDES SDI high-speed serial signals and self-defined SPI protocol data, and the decoding operation covers video synchronization signal separation, SDI rate and resolution automatic identification, SPI protocol command code analysis and audio 24-bit data flow generation; the processing module is used for performing normalization processing on the decoded data, generating a unified format data flow, and adding an adjustment factor; performing encoding and ID identification code embedding processing on the normalized data to generate to-be-stored data with an identification; performing FIFO buffering and write priority judgment on the encoded data to generate priority queue data, triggering a preset priority application when the cache reaches a preset threshold; performing DDR writing and BANK scheduling processing on the priority queue data to generate stable storage parallel data, wherein a writing arbitrator is scheduled according to the priority, a BANK switch realizes frame-level read-write separation, and 4 pieces of DDR3 are parallelly used to realize 512-bit width burst writing; performing reading and output coordination processing on the DDR storage data to generate continuous output data flow, wherein the data is positioned by decoding an ID identification code, a reading FIFO is scheduled according to the depth to read, and a channel temporary storage FIFO monitors the data amount to trigger a reading application; and dynamic bandwidth allocation and energy efficiency optimization are performed on the whole data flow processing process to generate efficient parallel processing results.

[0008] The application provides a processing method based on FPGA multi-task parallel data flow, which avoids the situation that multi-channel parallel video and audio flow data is lost when the DDR rate is insufficient by performing normalization processing on external video / audio data flow in different data flow forms and adopting a priority arbitrator to perform channel gating and the like. The video is stored in the DDR in units of lines, and the audio is stored in the DDR in burst units of 92 blocks, which changes the situation that some channels occupy the DDR bandwidth for a long time, so that the storage amount of each data channel is more balanced. The prior art supports at most 5-channel video flow data and 5-channel audio flow data, and the application can realize 10-channel video flow data and 11-channel audio flow data under a non-full load condition, which greatly improves the channel support capability. For 1920*1080 image data, the traditional method needs about 98743 times of 512-bit data burst, while the application only needs 92 times of data burst, and the occupation time is about 1 / 1000 of the traditional method, so that more time segments are left for other channel data storage, and the DDR bandwidth utilization is improved.

[0009] Priority is determined based on the depth of the FIFO data. Data of different priorities occupy different proportions of bandwidth. Priority 1 data can occupy 80% to 90% of the bandwidth, and priority 2 data occupies approximately 50% to 60% of the bandwidth, ensuring that high-priority data is transmitted first. When the data queues of all channels are less than 1 line and the DDR read arbiter demand is less than 50%, the DDR will reserve 30% of its fragment time to reduce continuous read and write operations and lower chip power output. Frame-level read and write area separation is achieved through the bank switch. When the write operation is in a specific bank, the read operation is performed in other banks. This ensures that when the image is written in a distributed manner, the read can be performed in the entire bank area to avoid conflicts. Ensure data output continuity: The channel data temporary storage module combines the read and write separation mechanism of the bank switch. When the data volume is less than 1 line, the priority 1 read request is triggered. When the data volume is greater than 6 lines, the request is stopped, ensuring the continuity of the output data stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A flowchart of a method for processing multi-task parallel data streams based on FPGA provided by an embodiment of the present invention; Figure 2 A diagram of an SDI data decoding module within an FPGA provided by an embodiment of the present invention; Figure 3 A flowchart of normalization and DDR storage processing of a multi-channel TMDS video data stream in an FPGA provided by an embodiment of the present application; Figure 4 A flowchart of normalization and DDR storage processing of a BT1120 internal synchronization signal video data stream in an FPGA provided in an embodiment of the present application; Figure 5 A schematic diagram of a video line data frame structure with an ID identification code provided in an embodiment of the present application; Figure 6 A timing diagram of serial video data reception provided in an embodiment of the present application; Figure 7 A schematic diagram of a module of a FPGA-based multi-task parallel data flow processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0011] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention. Figure 1 To describe the FPGA-based multi-task parallel data flow processing method according to an exemplary embodiment of the present application.

[0012] In an embodiment of the present application, a method for processing multi-task parallel data streams based on FPGA is provided. Figure 1As shown: S101: Build a multi-format raw data receiving and decoding system.

[0013] In one embodiment, raw data in multiple formats is received, including TMDS signals for the HDMI interface, which transmit video and audio data; I2S serial audio data, which is used for serial transmission of audio data; BT1120 internal synchronous data stream, in which synchronization information is encoded in the data stream; and SERDES SDI high-speed serial signals, which are used to transmit high-definition video and other data. Custom SPI protocol data is a video data block transmitted based on a custom SPI bus protocol. Specifically, the system simultaneously receives TMDS signals from the HDMI interface, I2S serial audio data from an audio capture device, BT1120 internal synchronous data streams output by cameras, SERDES SDI high-speed serial signals from professional video equipment, and video data blocks transmitted via a custom SPI protocol.

[0014] The received video-related signals are processed to separate synchronization signals. For TMDS signals, the decoding extracts 16-bit YUV 4:2:0 external synchronization pattern data, which includes synchronization signals such as CLK, HS, VS, and DE. For the BT1120 internal synchronization data stream, synchronization signals such as HS, VS, and DE are extracted from the data stream through the protocol, eliminating the need for additional synchronization signal lines. Signals such as HS (horizontal synchronization) and VS (vertical synchronization) corresponding to 1920*1080 resolution video are separated from the TMDS signal. Synchronization signals are parsed from the BT1120 data stream to ensure correct video data timing.

[0015] It decodes SERDES SDI high-speed serial signals, separating the serial code into parallel code via software, and automatically identifies the signal rate, resolution, and format. Upon receiving the SDI signal, it automatically identifies the 3 Gbps rate and 1920 x 1080 resolution at 60p, marking this information for subsequent data processing. It parses the received SPI data based on a custom SPI protocol. When the CS signal is active, it reads the first 16 bits of data to determine the command, then performs subsequent data processing based on the command, such as specifying the DDR bank location and index number for data block storage using STA_IDX.

[0016] After receiving a valid CS signal, the 16-bit command code 0X3882 is read, indicating a switch to continuous data block write mode. STA_IDX[15:14] is 01, confirming that the data block is stored in BANK1. STA_IDX[5:0] is 000001, confirming it is a PGM static image data block. The 4-wire signal in the external I2S data format is used for digital decoding, converting the serial audio data into parallel data streams of 24 bits each for the left and right channels. Decoding the I2S-formatted audio signal yields 24-bit left and right channel audio data streams, preparing for subsequent audio processing.

[0017] In another embodiment, if Figure 2 As shown, the SDI IN interface receives a SERDES SDI high-speed serial signal (i.e., SDI signal input). The signal is first processed by Equalizer 1 to compensate for signal transmission losses. The transceiver, the core component of the decoding process, is associated with multiple clock signals, including the REFCLK (reference clock) provided by Reference Sources, as well as clock signals such as TXUSRCLK, TXOUTCLK, RXUSRCLK, and RXOUTCLK. These clock signals are processed by BUFG / BUFR (buffers) to provide timing support for the normal operation of the transceiver.

[0018] The incoming SDI signal enters the transceiver via RXN (receive negative), where it is processed and generates RXDATA (received data). The transceiver is also connected to CableDriver1 (cable driver). The processed signal is output to the SDIOUT connector via TXP (transmit positive) and TXN (transmit negative).

[0019] The FPGA's internal decoding module also includes a CONTROL module and a STATUS module, which control the entire decoding process and monitor decoding status. Furthermore, the decoding module includes HDin (high-definition input) and SDin (standard-definition input) interfaces, which are associated with RXVIDEO (received video), as well as modules such as SMPTESDICore, which handles decoding operations such as automatic SDI rate and resolution identification.

[0020] In another embodiment, if Figure 6As shown in the figure, the timing relationship when the FPGA receives and processes serial video data is demonstrated, specifically including the following key information: Clock signal: With rx_usrclk as the reference clock, the frequency is 148.5MHz, which is a common clock frequency in high-definition video processing (such as the pixel clock corresponding to the 1080i50 / 60 format). Clock cycle distribution: There is a periodic clock distribution pattern in the signal processing process, which cycles according to the rule of "5 clocks-6 clocks-5 clocks-6 clocks", with a total cycle of 22 clocks (5+6+5+6), which may correspond to the line synchronization or data block transmission cycle of the video data. Key signals: including rx_ce_sd (data valid signal), rx_ds1a (data signal, example values ​​are Y'(719), 3FF, 000, XYZ, CB'(361)), rx_trs (timing reference signal, used to mark the video line / field synchronization boundary), and rx_eav (valid video end signal, marking the end of a line of valid video data). These signals collectively reflect the timing characteristics of the video data stream at the FPGA receiver. rx_trs and rx_eav are key indicators for separating video synchronization signals, while the 148.5 MHz frequency of rx_usrclk matches the timing requirements of high-definition video resolutions (such as 1920 × 1080).

[0021] S102: normalize the decoded data to generate a data stream in a unified format.

[0022] In one implementation, chroma component interpolation for decoded video data in non-4:4:4 formats (such as 4:2:0 and 4:2:2) is a key step in video normalization. Its purpose is to convert video data of varying chroma sampling formats to a standard 4:4:4 format to meet the requirements for complete chroma information in subsequent processing. The specific process is as follows: For chroma component interpolation in 4:2:0 format, the chroma sampling density of 4:2:0 video data in both horizontal and vertical directions is half that of the luminance components, meaning that every four luminance pixels (a 2×2 matrix) correspond to one chroma component (one each for Cb and Cr). During interpolation, missing chroma information is estimated using an interpolation algorithm. Horizontal interpolation: For adjacent luminance pixels in the same row, linear interpolation is used to calculate the chroma value corresponding to the intermediate pixel based on the available chroma component values. For example, if the luminance pixels in a row have chroma values ​​Cb1 and Cb3, the intermediate pixel's Cb2 can be calculated as (Cb1 + Cb3) / 2. Vertical interpolation: For corresponding luminance pixels in two adjacent rows, interpolation is performed based on the existing chrominance components. For example, if the chrominance value at a location in row 1 is Cr1, and the corresponding location in row 3 is Cr3, the chrominance value at the corresponding location in row 2 can be calculated as Cr2 = (Cr1 + Cr3) / 2. After interpolation, each luminance pixel has independent Cb and Cr components, resulting in 4:4:4 format data with consistent horizontal and vertical resolution and brightness.

[0023] Chroma components are padded in the 4:2:2 format. The horizontal chroma sampling density of 4:2:2 video data is half that of the luma components (the vertical sampling density is the same as that of the luma components), meaning that every two luma pixels correspond to one chroma component (one Cb and one Cr). Padded primarily involves interpolation in the horizontal direction: For consecutive luma pixels in the same row (e.g., Y1, Y2, Y3, Y4), if the corresponding chroma components are Cb1 (for Y1 and Y2) and Cb2 (for Y3 and Y4), the Cb values ​​of the pixels between Y2 and Y3 are interpolated (e.g., Cb midpoint = (Cb1 + Cb2) / 2), ensuring that each luma pixel has its own dedicated chroma component. Vertically, since the chroma sampling is consistent with the luma, no additional padded is required; the original components can be retained.

[0024] After supplementing the chroma components through the interpolation algorithm described above, non-4:4:4 video data is converted to the standard 4:4:4 format. This eliminates data size differences caused by different chroma sampling formats and provides unified basic data for subsequent color space conversions (such as YUV to RGB). This restores chroma information at full horizontal and vertical resolution, ensuring accurate image color reproduction and avoiding color distortion caused by missing chroma information.

[0025] The unified data format facilitates subsequent encoding and storage with a fixed 24-bit pixel width (8 bits each for RGB), adapting to DDR's 512-bit burst transmission mode. For example, for a 1920×1080 resolution 4:2:0 video, after chroma padding, the original structure of every 2×2 luminance pixels sharing a set of chroma components is broken down. Each pixel in the 1920 columns × 1080 rows now has independent chroma information. The total data volume is converted from the original (1920×1080×8-bit luminance + 960×540×8-bit Tcb + 960×540×8-bit Tcr) to 1920×1080×24-bit (8 bits each for RGB), consistent with the 4:4:4 format, providing standardized data for subsequent processing.

[0026] The converted video data is subjected to color space conversion processing to generate RGB format data. The CSC algorithm is used to convert YUV and YCBCR formats into RGB4:4:4 format. Color component calculations are performed using a mathematical model, and the data bit width is unified to 24 bits and the color space range is 0-255. The CSC (Color Space Conversion) algorithm is a core technology for achieving video color space unification. In this application, this algorithm is used to convert input data in formats such as YUV and YCBCR into RGB4:4:4 format, resolving processing compatibility issues caused by different color space formats. Its core logic is based on the mathematical mapping relationship between color components, accurately converting the luminance (Y) and chrominance (U / Cb, V / Cr) components into the three primary color components of red (R), green (G), and blue (B), ensuring that color information is transmitted without distortion during the conversion process.

[0027] For YCBCR format data, this application uses the following mathematical formula for conversion: The red component (R') is calculated as R'=1.164*(Y'-16)+1.596*(Cr'-128), where Y' is the luminance component (range: 16-235) and Cr' is the red chrominance component (range: 16-240). The black level offset is eliminated by subtracting the offsets (16 and 128), and R' is calculated using the weighted coefficients.

[0028] The green component (G') is calculated as G'=1.164*(Y'-16)-0.813*(Cr'-128)-0.392*(Cb'-128). The green component is affected by luminance (Y'), red chrominance (Cr'), and blue chrominance (Cb'). Multi-coefficient weighting is used to offset cross-chrominance interference and ensure accurate green reproduction.

[0029] The blue component (B') is calculated as B'=1.164*(Y'-16)+2.017*(Cb'-128). The blue component is mainly determined by the brightness (Y') and the blue chrominance (Cb'). The chrominance effect is amplified by the coefficient to compensate for the information loss caused by chrominance compression in the original format.

[0030] For YUV format data, this application uses another set of mathematical formulas: R'=Y'+1.140*V'; G'=Y'-0.394*U'-0.581*V'; B'=Y'+2.032*U; where Y' is the luminance component, and U' and V' are the chrominance components. Fast conversion is achieved through direct mapping, making it suitable for scenarios with high real-time requirements.

[0031] The data bit width and color space range are unified as follows: The bit width is unified to 24 bits: The converted R, G, and B components each retain 8 bits of precision (range 0-255), and the three are combined to form 24-bit RGB data (R8G8B8), which adapts to the 512-bit burst transmission requirements of the subsequent storage encoding module (each 512 bits can carry 21 24-bit pixels). The color space range is adjusted to 0-255: In the original YCBCR format, the Y' range is 16-235, and the Cb' / Cr' range is 16-240. By using the offset correction in the formula (for example, Y'-16, Cr'-128), the output R', G', and B' components are normalized to the range of 0-255, eliminating color gradation differences between different devices.

[0032] After color space conversion and standardization using the CSC algorithm, video data has the following advantages: A unified format of RGB 4:4:4 24-bit ensures efficient processing in subsequent storage encoding, DDR read / write modules, and other modules based on this fixed format. Color components are accurately restored, avoiding color casts and color discontinuities caused by format differences. Standardized bit widths and ranges provide the foundation for parallel processing of multi-channel data, supporting the simultaneous storage and reading of 10 video streams. For example, for 1920×1080 resolution YCBCR format video, after the above conversion, each frame of data is stored in RGB 4:4:4 24-bit format. The color information for each row of 1920 pixels can be stored in DDR via 92 512-bit burst transfers, reducing bandwidth usage by approximately 1 / 1000 compared to traditional solutions.

[0033] Frequency identification is performed on the decoded audio data to generate sampling frequency parameters. The sampling frequency and quantization depth of the input audio are calculated using an FPGA counter combined with a 12.288MHz reference frequency. This calculation is based on the fixed relationship between the clock signal and the sampling frequency in the I2S audio protocol. The I2S protocol transmits audio data over three wires: SCK (serial clock), WS (channel select), and SD (serial data). The SCK frequency has a clear mathematical relationship with the sampling frequency and quantization depth: SCK frequency = sampling frequency × quantization depth × 2 (the multiplication by 2 is due to alternating transmission of left and right channels). The FPGA integrates a high-precision counter that synchronously counts the 12.288MHz reference frequency (which is 256 times the 48kHz sampling rate, facilitating integer frequency division) and the input audio SCK clock, enabling parameter identification through frequency comparison.

[0034] The sampling frequency calculation process is as follows: Clock counting and period measurement: The FPGA counter simultaneously counts the 12.288MHz base frequency and the audio SCK clock. A fixed measurement period (e.g., 1ms) is set, and the number of base frequency pulses (denoted as N_base) and SCK clock pulses (denoted as N_sck) within that period are counted. Because the 12.288MHz frequency is stable, N_base is a fixed value (12288 pulses in 1ms).

[0035] SCK frequency conversion: Calculate the SCK frequency based on the count result: SCK frequency = (N_sck / N_base) × 12.288 MHz. For example, if N_sck is 186624 within 1ms, then SCK frequency = (186624 / 12288) × 12.288 MHz = 15.36 MHz.

[0036] Sampling frequency derivation: Based on the relationship between SCK and sampling frequency in the I2S protocol, and given a known quantization depth (identifiable through the data frame structure), sampling frequency = SCK frequency / (quantization depth × 2). If the 15.36 MHz SCK corresponds to a quantization depth of 16 bits, then sampling frequency = 15.36 MHz / (16 × 2) = 48 kHz.

[0037] The quantization depth is determined by analyzing the I2S data frame structure. The data bit width for each channel in the I2S protocol is the quantization depth. Data is transmitted sequentially from MSB (most significant bit) to LSB (least significant bit). The WS signal transition marks a channel switch and the start of a data frame. The FPGA directly determines the quantization depth by monitoring the number of valid data bits on the SD line during the WS signal cycle (i.e., the number of pulses from the start to the end of the frame). For example, if the SD line transmits 24 valid data bits during the WS cycle, the quantization depth is 24 bits.

[0038] The recognized audio data is quantized and frequency converted to generate normalized audio data. The audio quantization depth is unified to 16 bits, and frequency alignment conversion is performed based on a FIFO method to normalize the sampling frequency to 48 kHz. Audio adjustment factors and volume adjustment parameters are also added. The CSC (Color Space Conversion) algorithm is the core technology for achieving video color space unification. In this application, this algorithm is used to convert input data in formats such as YUV, YCBCR, etc. into RGB4:4:4 format, resolving processing compatibility issues caused by different color space formats. Its core logic is based on the mathematical mapping relationship between color components, accurately converting the brightness (Y) and chrominance (U / Cb, V / Cr) components into the three primary color components of red (R), green (G), and blue (B), ensuring that color information is transmitted without distortion during the conversion process.

[0039] The data bit width and color space range are unified, with the bit width being standardized to 24 bits. The converted R, G, and B components each retain 8 bits of precision (ranging from 0 to 255), and the three are combined to form 24-bit RGB data (R8G8B8), adapting to the 512-bit burst transmission requirements of the subsequent storage encoding module (each 512 bits can carry 21 24-bit pixels). The color space range is adjusted to 0-255: In the original YCBCR format, the Y' range is 16-235, and the Cb' / Cr' range is 16-240. By using offset corrections in the formula (e.g., Y'-16, Cr'-128), the output R', G', and B' components are normalized to a range of 0-255, eliminating color gradation differences between different devices.

[0040] After color space conversion and standardization using the CSC algorithm, video data has the following advantages: A unified format of RGB 4:4:4 24-bit ensures efficient processing in subsequent storage encoding, DDR read / write modules, and other modules based on this fixed format. Color components are accurately restored, avoiding color casts and color discontinuities caused by format differences. Standardized bit widths and ranges provide the foundation for parallel processing of multi-channel data, supporting the simultaneous storage and reading of 10 video streams. For example, for 1920×1080 resolution YCBCR format video, after the above conversion, each frame of data is stored in RGB 4:4:4 24-bit format. The color information for each row of 1920 pixels can be stored in DDR via 92 512-bit burst transfers, reducing bandwidth usage by approximately 1 / 1000 compared to traditional solutions.

[0041] In another embodiment, if Figure 3 As shown in the figure, the input raw data consists of four channels of external video data (EXVCH1-EXVCH4), all in TMDS video data stream format with YUV16-bit external synchronization mode. After passing through the decoding module, each data stream enters the color space processing module. This module performs resolution format identification and color space conversion on each of the four channels of video data, converting the original 16-bit YUV format to 24-bit RGB format, completing the first stage of normalization.

[0042] After the first stage of normalization, the data from the four channels (EXVCH1-EXVCH4) enters the DDR write arbiter. The DDR write arbiter is responsible for controlling write priority, determining the order in which data from each channel is written to the DDR. The data then enters the second and third stages of normalization. It then enters the storage encoding sequence module and the data buffer module, which perform serial-to-parallel conversion on the stored data, converting the 24-bit data into 504 bits. During this processing, the channel number, DDR address code, and bank code are added using FIFO mode for subsequent data identification. A 512-bit data depth is also temporarily paused with a 512-bit data width. Data reading is performed by the DDR controller. The processed data is ultimately written to the corresponding DDR channels (DDRCH1-DDRCH4) for storage.

[0043] In another embodiment, if Figure 4 The figure shows the FPGA processing flow for the BT1120 internal synchronization signal video data stream (using BTACH9, BTAFCH9, and BTAFSCH9 as example channels). This flow specifically includes the following steps: The raw data includes external signals (such as video signals related to recording and playback, USB signal conversion data, network video, and audio streams) as well as the BT1120 internal synchronization signal video data stream. Each data stream decoding module decodes the input BT1120 internal synchronization signal video data stream. Normalization is performed in three stages. The first stage, the color space processing module performs resolution format identification and color space conversion, converting the original 16-bit YUV format to 24-bit RGB format. Subsequent stages further process the data to a unified format. After normalization, the data enters the storage encoding sequence module and data buffer module for serial conversion and processing (converting 24 bits to 504 bits). The channel number, DDR address code, and bank code are added in FIFO mode for identification. The data is then temporarily stored with a 512-bit width and 512-bit data depth. The DDR controller ultimately reads and writes the data to the corresponding DDR channel (e.g., DDRCH19).

[0044] S103, encoding and ID code embedding processing are performed on the normalized data to generate data to be stored with identification.

[0045] In one embodiment, the normalized video data is divided into coding units to generate video data blocks encoded by row, wherein an image row is used as a coding unit, and the number of bursts required for each row is calculated in combination with the 512-bit burst data width to ensure that each row of data can be independently stored and transmitted. The normalized video data is divided into coding units by row, and an image row is used as an independent coding unit. The number of bursts required for each row is calculated in combination with the 512-bit burst data width of DDR3 to ensure that each row of data can be independently stored and transmitted. The specific calculation method is: 512-bit burst data width can transmit 21 24-bit pixels in parallel (512 / 24 ≈ 21.33, rounded to 21), so the number of pixels in each row divided by 21 (rounded up) is the number of bursts required for each row.

[0046] For 1920×1080 resolution video data, each row contains 1920 pixels. The number of bursts required for each row is 1920 / 21≈91.42, rounded up to 92. A row of pixel data can be fully stored through 92 512-bit burst transfers, enabling independent storage and transmission of each row of data.

[0047] Normalized audio data is divided into coding units by block, with a custom burst unit of 92 blocks, adapted to the 512-bit bit width and burst mode of DDR3, enabling batch storage of audio data. This unit design aligns with the burst count per video line, facilitating multi-channel data scheduling and bandwidth allocation. Normalized audio data has a 48 kHz sampling rate and a 16-bit bit depth. Each block of data contains a specific number of audio frames, and 92 blocks constitute a burst unit. Transmitting 92 blocks of audio data in a burst, adapted to the 512-bit bit width of DDR3, matches the burst transmission rhythm of video data, and improves DDR bandwidth utilization.

[0048] Unique ID codes are embedded into the divided audio and video data blocks, creating identified data to be stored. The ID code consists of a fixed packet header identifier (F4F4F4, 1A1A1A), an 8-bit channel code (channels 0-255), an 8-bit resolution identifier (e.g., 01 corresponds to 1920x1080p60), an 8-bit packet length code, a 2-bit bank code (00-11 corresponds to banks 0-3), a 22-bit DDR base address code, and a fixed packet tail identifier (F4F41A). This ID code allows data block addressing and channel differentiation.

[0049] like Figure 5As shown in the figure, take the 1920*1080 resolution image data for channel 3, banker 01, and the planned storage location at DDR base address 0x300000 as an example. This packet structure contains multiple fields, as follows: Each row of packet header identification code: Contains two 24-bit identification codes, F4F4F4 and 1A1A1A. The ID code: 8-bit 03, representing channel 3. The resolution code: 8-bit 01, corresponding to 1920*1080 resolution. The data burst length: 8-bit 5C. The DDR bank code: 2 bits. The DDR base address code: 22 bits 300000, indicating the planned storage location at DDR base address 0x300000. The packet footer identification code: 24 bits F4F41A. Next, the pixel data: 24 bits, representing the actual pixel information. This ID code can be used to accurately locate the storage location and channel of the data block.

[0050] S104 , performing FIFO buffering and writing priority determination on the encoded data to generate priority queue data.

[0051] In one implementation, the encoded audio and video data is buffered in a FIFO to generate temporary cache data. Data for each channel is independently cached using a 512×512 FIFO, and the write depth is monitored in real time using the FIFO's data_count port to provide a basis for priority determination. Sending the encoded audio and video data to independent 512×512 FIFOs for temporary caching by channel is a key step in achieving multi-channel data parallel processing. Each channel occupies a dedicated FIFO, ensuring that data streams of different types (video / audio) and from different sources are isolated from each other during the caching phase to avoid cross-interference. The 512×512 FIFO specification indicates a data bit width of 512 bits and a depth of 512 groups, which is compatible with the 512-bit burst transmission bit width of DDR3 and can cache up to 512 groups of data (each group contains 21 24-bit pixels or corresponding audio blocks).

[0052] The FIFO's data_count port monitors the cache data depth in real time. This port provides real-time feedback on the number of cached data groups (ranging from 0 to 512), providing a quantitative basis for priority determination. For example, in the FIFO of channel 1 (video), the encoded 1920-pixel data per line is cached in the form of 92 512-bit burst transfers. The data_count increases gradually as the data is written. When the cache depth reaches 300, it indicates that 300 groups of data (21 pixels per group) are currently cached, corresponding to approximately 300 × 21 = 6300 pixels. This does not reach the priority trigger threshold (448) and is temporarily classified as low priority. In the FIFO of channel 2 (audio), the encoded audio data is written in bursts of 92 blocks. The data_count feedback depth is 450, indicating that 450 audio blocks have been cached, exceeding the priority threshold (448). Writing to the DDR must be prioritized to avoid FIFO overflow.

[0053] This isolated cache mechanism, combined with real-time deep monitoring, not only ensures the independent processing of multi-channel data, but also provides a precise quantitative basis for dynamic priority arbitration, ensuring that high-priority data (such as FIFOs that are about to overflow) can be processed first, thereby improving DDR bandwidth utilization and data transmission stability.

[0054] The depth of the FIFO cache data is determined and a priority identifier is generated. A data depth of 448 is marked as priority 1, 384 as priority 2, and 256 as priority 3. Priority 1 is triggered when the video data cache reaches 6 lines. Determining the data depth and generating a priority identifier based on the FIFO's data_count value is the core logic for implementing DDR dynamic bandwidth allocation. This ensures that high-priority data is written first by quantifying data cache pressure and preventing FIFO overflow. The specific rules and implementation logic are as follows: The general priority determination (applicable to audio and video data) is as follows. Based on the FIFO's data_count value (number of cached data groups, ranging from 0 to 512), priority levels are divided into three tiers: Priority 1: Triggered when data_count reaches 448. At this point, the FIFO has remaining buffer space (512-448 = 64 groups), corresponding to 64 × 21 = 1344 pixels (video) or 64 audio blocks. This is approaching the buffer limit and requires urgent write scheduling to prevent overflow. Priority 2: Triggered when data_count reaches 384. Remaining buffer space (512-384 = 128 groups) indicates moderate buffer pressure and requires timely scheduling after high-priority data is processed. Priority 3: Triggered when data_count reaches 256. Remaining buffer space (512-256 = 256 groups) indicates low buffer pressure and can be scheduled sequentially when there are no high-priority requests.

[0055] In addition to the general rules, a dedicated mandatory priority rule for video data is added: when the number of video lines cached in the FIFO reaches 6, the data is forcibly marked as priority 1, regardless of whether the current data_count value reaches 448. This rule is implemented because video data requires line continuity, and a 6-line cache is close to the real-time transmission threshold, so it must be written first to ensure frame integrity.

[0056] Example 1: The video channel triggers general priority 1: For a 1920×1080 resolution video channel, each line of data after encoding requires 92 512-bit burst transmissions (corresponding to 92 sets of data). When the FIFO data_count displays 450, it has exceeded the threshold of 448, triggering priority 1. At this point, the amount of data buffered in the FIFO is 450 × 21 = 9450 pixels, which is close to 1920 × 5 = 9600 pixels (approximately 5 lines). If data is not written in time, subsequent data may be lost due to FIFO fullness.

[0057] Example 2, audio channel trigger priority 2: An audio channel uses a burst unit of 92 blocks. When the FIFO data_count reaches 390, it exceeds the threshold of 384 and is marked as priority 2. At this point, the cached audio blocks can support 390 / 92, which is approximately 4.24 burst transmissions. The cache pressure is moderate, and writes can be scheduled after processing priority 1 data.

[0058] Example 3: The video channel triggers forced priority 1: A video channel with a high input frame rate (e.g., 60 fps) has a FIFO data_count of only 300 (corresponding to 300 × 21 = 6300 pixels, approximately 3.3 rows), but the actual number of cached rows has reached 6 (6 × 1920 = 11520 pixels), triggering forced priority 1. This rule ensures row continuity in high-frame-rate video and prevents frame fragmentation due to insufficient cache.

[0059] Priority-identified data is queued and sorted to generate priority queue data. Priority 1 data enters the queue array first. If there are three or more priority 1 data, the DDR read channel is closed and the write module is forced to write data in the queue order. Priority 2 and 3 data enter the queue in turn and await scheduling if there are no higher priority requests. Queuing and scheduling audio and video data with priority identifiers according to preset rules is a key step in achieving dynamic DDR bandwidth allocation. The core logic controls the data write order based on priority level and urgency, ensuring that high-priority data is stored first to avoid cache overflow or data gaps.

[0060] Priority 1 data (including data_count ≥ 448 or video cache up to 6 lines) enjoys the highest scheduling privilege and enters the queue array directly. When three or more priority 1 data items are present in the queue array, an emergency scheduling mechanism is triggered: a shutdown request is immediately sent to the DDR read channel. After the last burst read operation is completed, the DDR controller is forced to switch to the write module, processing the priority 1 data in queue order. This mechanism interrupts read operations to prioritize write operations, as the priority 1 data is approaching the FIFO overflow threshold (only 64 rows remaining) to prevent data loss.

[0061] Priority 2 (data_count ≥ 384) and priority 3 (data_count ≥ 256) data are not eligible for urgent scheduling and must wait in the queue. Only when there is no priority 1 data in the queue and no urgent read requests from the DDR read channel are they placed into the scheduling queue in the order of "priority 2 before priority 3" and are written to when the write module becomes idle.

[0062] There are three priority-one video channel data in the system simultaneously (from the HDMI, SDI, and BT1120 interfaces respectively), which triggers emergency scheduling: first, a shutdown signal is sent to the DDR read channel. Assuming that the current read channel is performing the fifth burst read, it is allowed to complete the read and then close the read channel. After the read channel is closed, the DDR controller is forced to switch to the write module and process the data of the three channels in sequence according to the queue order (such as HDMI → SDI → BT1120). Two data groups (each block contains 64 groups of data) are written to each channel, for a total of 3×2=6 data groups, quickly freeing up FIFO buffer space.

[0063] After the three priority 1 data are processed, the remaining priority 2 (1 audio channel, data_count=390) and priority 3 (1 video channel, data_count=280) data are queued in the array: the write module prioritizes the priority 2 audio data and writes one data group (64 groups); after the audio data is written, the priority 3 video data is scheduled and also written into one data group, achieving sequential processing when there are no high-priority requests.

[0064] Through the emergency scheduling mechanism of priority one, priority writing can be given when multiple channels are close to overflowing at the same time, avoiding data loss caused by FIFO overflow, which is especially suitable for high-bandwidth video data. During emergency scheduling, the read channel is allowed to complete the current burst read and then close, which not only ensures the continuity of the read data, but also minimizes the waiting time for write operations and reduces read and write conflicts. The waiting mechanism for priority two and three data avoids low-priority data occupying high-priority resources, and tilts the DDR bandwidth toward the channels that are really needed. It has been measured that the bandwidth occupancy of a single channel can be reduced to about 1 / 1000, supporting parallel processing of 10 video channels + 11 audio channels. In summary, this scheduling rule realizes the efficient and secure storage of multi-channel data in DDR through the logic of "emergency interrupt + sequential waiting", and is a core component of the dynamic bandwidth allocation mechanism.

[0065] S105 , performing DDR writing and BANK scheduling processing on the priority queue data to generate parallel data for stable storage.

[0066] In one embodiment, the priority queue data is written and scheduled to generate data to be written to the DDR, wherein the DDR write arbiter schedules data according to the priority queue order, and the data of priority one is written first after the read channel is forcibly closed, and two data groups are written. The data of priority two and three are written in sequence when there is no urgent read request, and one data group is written each time. The DDR write arbiter schedules data according to the priority queue order to generate data to be written to the DDR. The specific rules are as follows: the data of priority one has the highest priority and will be written first after the DDR read channel is forcibly closed (after reading the last burst), and two data groups are written each time (each block contains 64 groups of data) to quickly release FIFO cache space and avoid data overflow. The data of priority two and three are written in sequence according to the queue order when there is no urgent read request, and one data group is written each time to balance bandwidth usage.

[0067] Specifically, the priority queue contains three priority 1 video channel data, one priority 2 audio channel data, and one priority 3 video channel data. The DDR write arbiter first processes the priority 1 data, forcibly closes the read channel, and then sequentially writes two data blocks to each channel. After processing the priority 1 data, it then writes one data block to the priority 2 audio channel and one data block to the priority 3 video channel.

[0068] The bank switch allocates and switches write data to generate conflict-free storage instructions. The bank switch allocates write data to a designated bank while forcing read operations to occur in other banks, achieving frame-level read and write area separation and avoiding read-write conflicts within the same area. The bank switch allocates and switches write data to generate conflict-free storage instructions. Its core approach is to allocate write data to a designated bank while forcing read operations to occur in other banks, achieving frame-level read and write area separation, thereby avoiding read-write conflicts within the same area and ensuring stable data storage and access.

[0069] The bank switch allocates priority 1 video data to bank 1, and the read operation is forced to be performed in bank 2. When processing priority 2 audio data, it is allocated to bank 3, and the read operation is performed in bank 0. In this way, read and write operations are performed between different banks to avoid conflicts.

[0070] The allocated data stream is written to the DDR memory, generating stable parallel data storage. Four DDR3 chips, each with a 16-bit width and an 8-bit burst mode, operate in parallel, achieving 512-bit burst writes. This achieves a total bandwidth of 51.2 Gbps, ensuring efficient data storage. The data stream allocated by the bank is written to the DDR memory, generating stable parallel data storage. Four DDR3 chips, each with a 16-bit width and an 8-bit burst mode, operate in parallel, achieving 512-bit burst writes. This achieves a total bandwidth of 51.2 Gbps (16 bits × 4 × 800 MHz), ensuring efficient data storage and meeting the processing requirements of multi-channel parallel data streams.

[0071] Four DDR3 chips operate simultaneously, each handling 128 bits of data. Using eight bursts, 512 bits of data can be written at once. For a 1920×1080 resolution video data line, 92 such burst writes are needed to store a complete line of data in DDR. The total bandwidth supports the parallel storage of 10 channels of video data and 11 channels of audio data.

[0072] S106 , performing coordinated reading and output processing on the DDR storage data to generate a continuous output data stream.

[0073] In one embodiment, the DDR stored data is subjected to ID code decoding to generate data location information. A key ID code reader module parses the channel code, bank code, and DDR base address code in the data packet header to determine the channel to which the data block belongs, its storage location, and a read pointer. The key ID code reader module decodes the ID code of the data stored in the DDR, parses the 8-bit channel code, 2-bit bank code, and 22-bit DDR base address code in the data packet header, and thereby determines the channel to which the data block belongs, its storage bank location, and its specific read pointer, generating data location information.

[0074] Parsing the ID identification code of a data block, we get the 8-bit channel code 03 (corresponding to the 3rd channel), the 2-bit bank code 01 (corresponding to bank 1), and the 22-bit DDR basic address code 300000. This determines that the data block belongs to the 3rd channel and is stored at the address 0X300000 of bank 1. The read pointer points to this address.

[0075] The located data stream is buffered and prioritized in a read FIFO, generating an ordered read queue. Data is stored in a 512x512 read FIFO, and a read counter monitors the data depth. Data depth < 64 is marked as priority 1, < 256 as priority 2, and < 384 as priority 3. The DDR read arbiter schedules reads based on priority. The located data stream is buffered in a 512x512 read FIFO, and a read counter monitors the data depth in real time. Priority is assigned based on the depth: priority 1 for depth < 64, priority 2 for depth < 256, and priority 3 for depth < 384. The DDR read arbiter schedules reads based on these priorities, forming an ordered read queue. After the data stream from channel 2 is stored in the read FIFO, the read counter indicates a data depth of 50 (< 64), marking it as priority 1. After the data stream from channel 5 is stored, the depth is 200 (< 256), marking it as priority 2. The DDR read arbiter prioritizes the data from channel 2 for reading, and then schedules the data from channel 5.

[0076] Channels are temporarily stored and output coordinated for read data to generate a continuous output data stream. The channel data temporary storage module caches the 512-bit read data. When the data volume is less than one line, a priority 1 read request is triggered. When the data volume exceeds six lines, the request is stopped. Combined with the bank switcher's read-write separation mechanism, output continuity is ensured. The channel data temporary storage module temporarily caches the 512-bit read data. When the cached data volume is less than one line (such as 92 lines of data corresponding to 1920 pixels), a priority 1 read request is triggered. When the data volume exceeds six lines, the read request is stopped. Furthermore, combined with the bank switcher's read-write separation mechanism (writing in one bank, reading in another), the continuity of the output data stream is ensured.

[0077] The channel data buffer module has only 80 lines (less than 92) of 1920×1080 resolution video data remaining, triggering a priority 1 read request to continue reading data. When the data volume reaches 7 lines (more than 6), the request is terminated. At this point, the bank switch controls write operations to bank 0 and read operations to bank 1, avoiding read / write conflicts and ensuring a continuous and seamless video data stream.

[0078] S107, dynamically allocate bandwidth and optimize energy efficiency for the entire data stream processing process to generate efficient parallel processing results.

[0079] In one implementation, dynamic bandwidth allocation adjusts the DDR bandwidth usage ratio in real time based on data priority, cache depth, and channel type (video / audio), ensuring that high-priority data is transmitted first while balancing the bandwidth requirements of multi-channel data. The specific mechanism is as follows: Priority 1 data (FIFO depth ≥ 448 or video cache up to 6 lines) occupies 80%-90% of the bandwidth, Priority 2 data (depth ≥ 384) occupies 50%-60% of the bandwidth, and Priority 3 data (depth ≥ 256) occupies the remaining bandwidth only when there are no higher-priority requests. A priority queue mechanism prevents a single channel from occupying bandwidth for extended periods. When there are three or more Priority 1 data, the read channel is forcibly closed and bandwidth is concentrated for writes, ensuring timely storage of urgent data. A bank switch separates read and write areas (write operations in one bank, read operations in another bank), avoiding bandwidth waste due to conflict waiting. The system processes 10 video and 11 audio streams simultaneously. When three video channels trigger priority one, the dynamic bandwidth allocation mechanism allocates 80% of the bandwidth to these three videos, giving priority to writing two data groups; the remaining 20% ​​of the bandwidth is allocated to the audio channel of priority two when there is no urgent request, ensuring that multi-channel data can be transmitted efficiently.

[0080] Energy efficiency optimization reduces inactive power consumption by dynamically adjusting the DDR's operating state and frequency. Specific measures include: Idle period frequency reduction: When the data cache depth of all channels is less than 256 (no priority requests), the DDR allocates 30% of its time, reducing its operating frequency and minimizing power consumption associated with continuous read and write operations. Read / write merging scheduling: When low-priority data (second and third) is queued, adjacent read and write requests are merged, reducing the number of DDR controller state transitions and lowering power consumption. Dynamically shutting down idle channels: For channels with no data transmission for extended periods, the corresponding FIFO and arbiter ports are shut down, halting the clock supply and reducing static power consumption. At night, when system load is low and the data cache depth of all channels is less than 256, the DDR automatically allocates 30% of its time and reduces its clock frequency to 400MHz. Simultaneously, the ports for three idle video channels and two idle audio channels are shut down, reducing overall power consumption by approximately 40%.

[0081] By combining dynamic bandwidth allocation with energy efficiency optimization, the following benefits are achieved: Multi-channel parallel support: Under partial load, it stably supports parallel processing of 10 video channels (1920×1080P60) and 11 audio channels (48kHz, 16-bit) without packet loss or latency. Maximized bandwidth utilization: Utilization of the total DDR bandwidth (51.2Gbps) is dynamically adjusted based on load, reaching 90% during priority 1 data transmission and dropping to 20% during idle time, balancing efficiency and power consumption. Low-power operation: After energy efficiency optimization, average system power consumption is reduced by 30% to 40%, especially during low-load periods.

[0082] like Figure 7As shown, a processing device based on FPGA multi-task parallel data stream includes: a construction module 701 for constructing a multi-format raw data receiving and decoding system, wherein the raw data includes TMDS signals, I2S serial audio data, BT1120 internal synchronization data stream, SERDES SDI high-speed serial signal and custom SPI protocol data, and the decoding operation covers video synchronization signal separation, SDI rate and resolution automatic identification, SPI protocol command code parsing and audio 24-bit data stream generation; a processing module 702 for normalizing the decoded data to generate a unified format data stream, wherein the video data is converted into RGB4:4:424BIT format and the color space range is unified to 0~255 through the CSC algorithm, and the audio data is converted into 48KHz sampling rate and 16-bit bit depth data based on 12.288MHz reference frequency alignment and an adjustment factor is added; the normalized data is encoded and the ID identification code is embedded to generate data to be stored with an identifier; the encoded data is buffered in FIFO and the write priority is determined to generate priority queue data, wherein, through 51 The data_count of the 2×512 FIFO determines the data depth, with 448 set as priority one, 384 as priority two, and 256 as priority three. Priority one application is triggered when the video cache reaches 6 lines; DDR write and BANK scheduling are performed on the priority queue data to generate stable stored parallel data. Among them, the write arbitrator is scheduled according to priority, the BANK switch implements frame-level read and write separation, and 4 DDR3 chips implement 512-bit bit-width burst write in parallel; the DDR storage data is read and output coordinated to generate a continuous output data stream, among which the data is located by decoding the ID identification code, the read FIFO is scheduled according to the depth, and the channel temporary storage FIFO monitors the data volume to trigger a read application; dynamic bandwidth allocation and energy efficiency optimization are performed on the entire data stream processing process to generate efficient parallel processing results.

[0083] A computing device comprises a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute any one of the processing methods based on FPGA multi-task parallel data flow.

[0084] The methods and / or embodiments in the embodiments of the present application can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. When the computer program is executed by a processing unit, the above-mentioned functions defined in the method of the present application are performed.

[0085] It should be noted that the computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0086] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0087] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the present application is defined by the appended claims rather than the foregoing description, and all variations that come within the meaning and range of equivalents of the claims are intended to be embraced herein.

Claims

1. A method for processing multi-task parallel data stream based on FPGA, characterized in that: include: Build a multi-format raw data reception and decoding system. The raw data includes TMDS signals, I2S serial audio data, BT1120 internal synchronous data stream, SERDES SDI high-speed serial signal, and custom SPI protocol data. The decoding operations include video synchronization signal separation, automatic SDI rate and resolution identification, SPI protocol command code parsing, and audio 24-bit data stream generation. Normalize the decoded data to generate a unified format data stream and add a regulation factor; Encode the normalized data and embed the ID identification code to generate the data to be stored with identification; Perform FIFO buffering and write priority determination on the encoded data to generate priority queue data. When the cache reaches the preset threshold, the preset priority application is triggered. DDR write and bank scheduling are performed on priority queue data to generate stable parallel data. The write arbiter schedules data based on priority, the bank switch implements frame-level read and write separation, and four DDR3 chips are used in parallel to implement 512-bit burst writes. Coordinate the reading and output of DDR storage data to generate a continuous output data stream. Data is located by decoding the ID identification code, the read FIFO is read according to the depth, and the channel temporary storage FIFO monitors the data volume to trigger the read request. Dynamic bandwidth allocation and energy efficiency optimization are performed on the entire data stream processing process to generate efficient parallel processing results.

2. The method for processing multi-task parallel data stream based on FPGA according to claim 1, characterized in that: Normalize the decoded data to generate a unified format data stream, including: Performing chroma sampling conversion on the decoded video data to generate 4:4:4 standard chroma format data, wherein the chroma components of non-4:4:4 formats including 4:2:0 and 4:2:2 are supplemented to restore the chroma sampling standard to full horizontal and vertical resolution; Performing color space conversion on the converted video data to generate RGB format data; Perform frequency recognition processing on the decoded audio data to generate sampling frequency parameters. The sampling frequency and quantization depth of the input audio are calculated by combining the FPGA counter with the 12.288MHz reference frequency. The recognized audio data is quantized and frequency converted to generate normalized audio data. The audio quantization depth is unified to 16 bits, and the frequency alignment conversion is completed based on the FIFO method to normalize the sampling frequency to 48 kHz. The audio adjustment factor and volume adjustment parameters are also added.

3. The method for processing multi-task parallel data stream based on FPGA according to claim 1, characterized in that: Encode the normalized data and embed the ID identification code to generate the identified data to be stored, including: The normalized video data is divided into coding units to generate video data blocks coded by row. The image row is used as the coding unit, and the number of bursts required for each row is calculated based on the 512-bit burst data width to ensure that each row of data is stored and transmitted independently. The normalized audio data is divided into coding units to generate block-coded audio data blocks. A custom burst unit of 92 blocks is used to adapt to the 512-bit bit width and burst mode of DDR3 to achieve batch storage of audio data. The divided audio and video data blocks are embedded with ID identification codes to generate identified data to be stored. The ID identification code includes a fixed packet header flag code, channel code, resolution identification code, packet data length code, BANK code, DDR basic address code and a fixed packet tail flag code. The address positioning and channel distinction of the data block are achieved through the ID code.

4. The method for processing multi-task parallel data stream based on FPGA according to claim 1, characterized in that: Perform FIFO buffering and write priority determination on the encoded data to generate priority queue data, including: The encoded audio and video data is buffered in FIFO format to generate temporary cache data. Each channel's data is cached independently through a 512×512 FIFO. The data_count port of the FIFO is used to monitor the write depth in real time to provide a basis for priority determination. The depth of the FIFO buffer data is judged and a priority flag is generated. When the data depth reaches 448, it is marked as priority 1; when it reaches 384, it is marked as priority 2; when it reaches 256, it is marked as priority 3. When the video data buffer reaches 6 lines, priority 1 is forcibly triggered; The data with priority identifiers are queued and sorted to generate priority queue data. Among them, data with priority one enters the queue array first. If there are three or more data with priority one, the DDR read channel is closed and forced to switch to the write module, and the data is written in the queue order. Data with priority two and three enter the queue in turn and wait for scheduling if there is no higher priority request.

5. The method for processing multi-task parallel data stream based on FPGA according to claim 4, characterized in that: Perform DDR write and BANK scheduling on the priority queue data to generate stable parallel data storage, including: Perform write scheduling on the priority queue data to generate data to be written into the DDR. The DDR write arbiter schedules data according to the priority queue order. Priority 1 data is written first after the read channel is forcibly closed, and two data blocks are written. Priority 2 and 3 data are written sequentially when there is no urgent read request, and one data block is written each. Perform bank allocation and switching processing on write data to generate conflict-free storage instructions. The bank switch allocates write data to a specified bank and forces read operations to be performed in other banks, achieving frame-level read and write area separation to avoid read and write conflicts in the same area. The allocated data stream is written into the DDR storage to generate parallel data for stable storage.

6. The method for processing multi-task parallel data stream based on FPGA according to claim 1, characterized in that: Coordinate the reading and output of DDR storage data to generate a continuous output data stream, including: Decode the ID identification code of the DDR storage data to generate data location information. The key ID identification code module is used to parse the channel code, bank code and DDR basic address code of the data packet header to determine the channel to which the data block belongs, the storage location and the read pointer. The located data stream is buffered and prioritized in a read FIFO to generate an ordered read queue. Data is stored in a 512×512 read FIFO, and the data depth is monitored by a read counter. When the depth is <64, it is marked as priority 1; when it is <256, it is marked as priority 2; and when it is <384, it is marked as priority 3. The DDR read arbiter schedules reads based on priority. The read data is temporarily stored in the channel and coordinated with the output to generate a continuous output data stream. The channel data temporary storage module caches 512-bit read data. When the data volume is less than 1 line, the priority 1 read request is triggered. When the data volume is greater than 6 lines, the request is stopped. Combined with the read-write separation mechanism of the bank switch, the output continuity is guaranteed.

7. A multi-task parallel data flow processing device based on FPGA, characterized in that: The device comprises: Building blocks for constructing a multi-format raw data reception and decoding system. The raw data includes TMDS signals, I2S serial audio data, BT1120 internal synchronization data streams, SERDES SDI high-speed serial signals, and custom SPI protocol data. Decoding operations include video synchronization signal separation, automatic SDI rate and resolution identification, SPI protocol command code parsing, and audio 24-bit data stream generation. The processing module is used to normalize the decoded data, generate a unified format data stream, and add an adjustment factor; encode the normalized data and embed the ID identification code to generate identified data to be stored; perform FIFO buffering and write priority determination on the encoded data to generate priority queue data. When the cache reaches the preset threshold, the preset priority application is triggered; DDR write and BANK scheduling are performed on the priority queue data to generate stable stored parallel data, wherein the write arbitrator is scheduled according to priority, the BANK switch realizes frame-level read and write separation, and 4 DDR3 chips realize 512-bit bit width burst writing in parallel; the DDR storage data is read and output coordinated to generate a continuous output data stream, wherein the data is located by decoding the ID identification code, the read FIFO is read according to depth scheduling, and the channel temporary storage FIFO monitors the data volume to trigger a read application; dynamic bandwidth allocation and energy efficiency optimization are performed on the entire data stream processing process to generate efficient parallel processing results.

8. An electronic device, characterized in that: include: a first processor; and a memory for storing executable instructions of the first processor; The first processor is configured to execute the FPGA multi-task parallel data stream processing method according to any one of claims 1 to 6 by executing the executable instructions.

9. A computing device comprising a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein: When the computer program instructions are executed by the processor, the device is triggered to execute the FPGA-based multi-task parallel data flow processing method according to any one of claims 1 to 6.

Images