A method and apparatus for processing multi-task parallel data streams based on FPGA

Abstract

This invention discloses a method and apparatus for processing multi-task parallel data streams based on FPGA, applied in the field of data processing technology. The invention constructs a multi-format raw data receiving and decoding system; normalizes the decoded data to generate a unified format data stream; encodes and embeds ID identification codes into the normalized data to generate tagged data to be stored; performs FIFO buffering and write priority determination on the encoded data to generate a priority queue data; performs DDR write and BANK scheduling on the priority queue data to generate stable parallel data; and coordinates the reading and output of the DDR stored data to generate a continuous output data stream. Specifically, data is located by decoding the ID identification code, reads from the FIFO by depth scheduling, and the channel temporary storage FIFO monitors the data volume to trigger read requests. Dynamic bandwidth allocation and energy efficiency optimization are performed throughout the data stream processing process to generate efficient parallel processing results.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and apparatus for processing multi-task parallel data streams based on FPGA. Background Technology

[0002] Traditional FPGA-based video and audio processing solutions suffer from the following problems:

[0003] Disadvantages of the storage method: Video and audio are cached on a per-frame basis. When the DDR speed is insufficient for multi-channel parallel video and audio streams, data loss is likely to occur.

[0004] Uneven bandwidth usage: Some channels may occupy DDR bandwidth for a long time, resulting in less storage on other data channels and affecting overall processing efficiency.

[0005] Limited channel support: The original solution can only support a maximum of 5 channels of video stream data and 5 channels of audio stream data, which is difficult to meet the needs of multi-channel parallel processing.

[0006] Excessive bandwidth usage: Taking 1920*1080 image data as an example, the traditional method requires bursting 512 bits of data at a time, which requires approximately 98,743 bursts. This will continuously occupy DDR bandwidth for a certain period of time, which is not conducive to the storage of data in other channels. Summary of the Invention

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] A method for processing multi-task parallel data streams based on FPGA includes: constructing a multi-format raw data receiving and decoding system, wherein the raw data includes TMDS signals, I2S serial audio data, BT1120 internal synchronization data stream, SERDESSDI high-speed serial signals, and custom SPI protocol data; the decoding operation covers video synchronization signal separation, automatic identification of SDI rate and resolution, SPI protocol command code parsing, and generation of 24-bit audio data stream; normalizing the decoded data to generate a unified format data stream and adding an adjustment factor; encoding and embedding ID identification codes into the normalized data to generate tagged data to be stored; and buffering and writing the encoded data using a FIFO. Priority determination is performed to generate a priority queue. When the cache reaches a preset threshold, a preset priority request is triggered. DDR write and BANK scheduling are performed on the priority queue data to generate stable parallel data. The write arbiter schedules according to priority, the BANK switch implements frame-level read-write separation, and four DDR3 chips achieve 512-bit burst write in parallel. Read and output coordination is performed on the DDR storage data to generate a continuous output data stream. Data is located by decoding the ID identification code, read FIFO is read according to depth scheduling, and channel temporary storage FIFO monitors the data volume to trigger read requests. Dynamic bandwidth allocation and energy efficiency optimization are performed throughout the data stream processing process to generate efficient parallel processing results.

[0009] A multi-task parallel data stream processing device based on FPGA, comprising: a building module for constructing a multi-format raw data receiving and decoding system, wherein the raw data includes TMDS signals, I2S serial audio data, BT1120 internal synchronization data stream, and SERDES. The decoding process includes SDI high-speed serial signals and custom SPI protocol data, encompassing video synchronization signal separation, automatic SDI rate and resolution identification, SPI protocol command code parsing, and 24-bit audio data stream generation. The processing module normalizes the decoded data, generating a unified format data stream and adding adjustment factors. It then encodes and embeds ID identification codes into the normalized data, generating tagged data to be stored. The encoded data undergoes FIFO buffering and write priority determination, generating a priority queue. When the buffer reaches a preset threshold, a preset priority request is triggered. The priority queue data undergoes DDR write and BANK scheduling, generating stable parallel data storage. The write arbiter schedules by priority, the BANK switch implements frame-level read / write separation, and four DDR3 chips achieve 512-bit burst writes in parallel. The DDR stored data undergoes read and output coordination, generating a continuous output data stream. Data is located by decoding the ID identification code, reads are scheduled by depth in the FIFO, and the channel temporary storage FIFO monitors the data volume to trigger read requests. The entire data stream processing process is dynamically bandwidth-allocated and energy-efficiently optimized, generating highly efficient parallel processing results.

[0010] Its beneficial effects are as follows: This invention provides a multi-task parallel data stream processing method based on FPGA. By normalizing video / audio data streams of different external data stream formats and using a priority arbiter for channel selection, it avoids data loss when the DDR speed is insufficient for multi-channel parallel video and audio streams. By storing video data in one line and audio data in custom 92-block bursts, it changes the situation where some channels occupy DDR bandwidth for extended periods, resulting in a more balanced storage capacity across data channels. Existing technologies support a maximum of 5 video streams and 5 audio streams; this solution can achieve 10 video streams and 11 audio streams under non-full load conditions, significantly improving channel support capabilities. For 1920*1080 image data, the traditional method requires approximately 98,743 bursts of 512-bit data, while this solution only requires 92 bursts, reducing the time occupied by approximately 1 / 1000th of the traditional method. This frees up more time for other channel data storage, improving DDR bandwidth utilization.

[0011] Priority is determined based on FIFO data depth, with different priority data occupying different proportions of bandwidth. Priority 1 data can occupy 80%~90% of the bandwidth, while priority 2 data occupies approximately 50%~60% of the bandwidth, ensuring that high-priority data is transmitted first. When the data queues of all channels are less than one line and the DDR read arbitrator demand is less than 50%, the DDR allocates 30% of its time slot to reduce continuous read / write operations and lower chip power consumption. Frame-level read / write area separation is achieved through a BANK switch. When a write operation occurs in a specified BANK, a read operation occurs in another BANK, ensuring that during distributed image writing, reading can be performed within a complete BANK area, avoiding conflicts. Data output continuity is guaranteed: the channel data buffer module, combined with the read / write separation mechanism of the BANK switch, triggers a priority 1 read request when the data volume is less than one line and stops the request when it exceeds six lines, ensuring the continuity of the output data stream. Attached Figure Description

[0012] Figure 1 A flowchart illustrating a method for processing multi-task parallel data streams based on FPGA, provided in an embodiment of the present invention;

[0013] Figure 2 This invention provides a diagram of an SDI data decoding module within an FPGA.

[0014] Figure 3 A flowchart illustrating the normalization and DDR storage processing of a multi-channel TMDS video data stream in an FPGA, provided as an embodiment of this application;

[0015] Figure 4 A flowchart illustrating the normalization and DDR storage processing of the BT1120 internal synchronization signal video data stream in an FPGA, provided in an embodiment of this application;

[0016] Figure 5 A schematic diagram of a video line data frame structure with an ID identification code is provided in an embodiment of this application;

[0017] Figure 6 A timing diagram for serial video data reception provided in an embodiment of this application;

[0018] Figure 7 This is a schematic diagram of a module of an FPGA-based multi-task parallel data stream processing device provided in an embodiment of the present invention. Detailed Implementation

[0019] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention. Figure 1This application describes a method for processing FPGA-based multi-task parallel data streams according to exemplary embodiments of the present application.

[0020] In this application embodiment, a method for processing multi-task parallel data streams based on FPGA is described, such as... Figure 1 As shown:

[0021] S101, constructing a multi-format raw data receiving and decoding system.

[0022] In one implementation, raw data in multiple formats is received, including TMDS signals for the HDMI interface, used to transmit video and audio data; I2S serial audio data for serial transmission of audio data; BT1120 internal synchronization data stream, with synchronization information encoded in the data stream; and SERDES SDI high-speed serial signals for transmitting high-definition video data. Custom SPI protocol data consists of video data blocks transmitted based on a custom SPI bus protocol. Specifically, TMDS signals from the HDMI interface, I2S serial audio data from the audio acquisition device, the BT1120 internal synchronization data stream output by the camera, the SERDES SDI high-speed serial signal from the professional video equipment, and video data blocks transmitted via the custom SPI protocol are received simultaneously.

[0023] The received video-related signals are processed to extract the synchronization signals. For TMDS signals, 16-bit YUV4:2:0 external synchronization mode data is decoded and extracted, which includes synchronization signals such as CLK, HS, VS, and DE. For BT1120 internal synchronization data streams, HS, VS, and DE synchronization signals are extracted from the data stream via a protocol, eliminating the need for additional synchronization signal lines. The HS (horizontal sync) and VS (vertical sync) signals corresponding to 1920*1080 resolution video are extracted from the TMDS signals; the synchronization signals are parsed from the BT1120 data stream to ensure the correct timing of the video data.

[0024] The SERDES SDI high-speed serial signal is decoded by separating the serial code into parallel code using software, while automatically identifying the signal's rate, resolution, and format. Upon receiving the SDI signal, its rate is automatically identified as 3Gbps and its resolution as 1920*1080P60, and this information is marked for subsequent data processing. Based on a custom SPI protocol, the received SPI data is parsed. When the CS signal is valid, the first 16 bits of data are read to determine the command, and then subsequent data processing is performed according to the command, such as specifying the DDR BANK location and index number of the data block storage via STA_IDX.

[0025] Upon receiving a valid CS signal, the 16-bit command code 0X3882 is read, indicating a switch to continuous data block writing mode. Then, STA_IDX[15:14] being 01 confirms the data block is stored in BANK1, and STA_IDX[5:0] being 000001 confirms it's a static graph data block for PGM. Data decoding is performed using the 4-wire signal in the external I2S data format, converting the serial audio data into 24-bit parallel data streams for each of the left and right channels. Decoding the I2S format audio signal yields 24-bit audio data streams for the left and right channels, preparing for subsequent audio processing.

[0026] In another implementation, such as Figure 2 As shown, based on the SDI IN interface, the SERDES SDI high-speed serial signal (i.e., SDI signal input) is received. The signal is first processed by Equalizer1 to compensate for signal loss during transmission. The transceiver is the core component of the decoder and is associated with multiple clock signals, including REFCLK (reference clock) provided by ReferenceSources, as well as clock signals such as TXUSRCLK, TXOUTCLK, RXUSRCLK, and RXOUTCLK. These clock signals are processed by BUFG / BUFR (buffers) to provide timing support for the normal operation of the transceiver.

[0027] The input SDI signal enters the transceiver through RXN (receive negative terminal), and after processing, RXDATA (receive data) is generated. The transceiver is also connected to CableDriver1, and the processed signal is output to the SDIOUT interface through TXP (transmit positive terminal) and TXN (transmit negative terminal).

[0028] The FPGA's internal decoding module also includes a CONTROL module and a STATUS module, used to control the entire decoding process and monitor the decoding status. In addition, the decoding module has HDin (high-definition input) and SDin (standard-definition input) interfaces, associated with RXVIDEO (received video), as well as modules such as SMPTESDICore, which are involved in decoding operations such as automatic SDI rate and resolution recognition.

[0029] In another implementation, such as Figure 6As shown, the timing relationship of the FPGA receiving and processing serial video data is illustrated, specifically including the following key information: Clock signal: The reference clock is rx_usrclk, with a frequency of 148.5MHz, which is a common clock frequency in high-definition video processing (such as the pixel clock corresponding to the 1080i50 / 60 format). Clock cycle allocation: There is a periodic clock allocation pattern in the signal processing process, which cycles according to the rule of "5 clocks-6 clocks-5 clocks-6 clocks", with a total cycle of 22 clocks (5+6+5+6), which may correspond to the line synchronization or data block transmission cycle of video data. Key signals: Includes rx_ce_sd (data valid signal), rx_ds1a (data signal, example values ​​are Y'(719), 3FF, 000, XYZ, CB'(361)), rx_trs (timing reference signal, used to mark the video line / field synchronization boundary), and rx_eav (valid video end signal, marking the end of a line of valid video data). These signals collectively reflect the timing characteristics of the video data stream at the FPGA receiver. Among them, rx_trs and rx_eav are key indicators for the separation of video synchronization signals, while the 148.5MHz frequency of rx_usrclk matches the timing requirements of high-definition video resolution (such as 1920×1080).

[0030] S102 performs normalization processing on the decoded data to generate a data stream with a unified format.

[0031] In one implementation, chroma component supplementation for decoded video data in non-4:4:4 formats (such as 4:2:0 and 4:2:2) is a crucial step in video normalization. The aim is to uniformly convert video data with different chroma sampling formats into the standard 4:4:4 format to meet the requirements of subsequent processing for complete chroma information. The specific process is as follows: Chroma component supplementation for 4:2:0 format: In 4:2:0 format video data, the chroma sampling density in both the horizontal and vertical directions is half that of the luminance components, meaning that every four luminance pixels (2×2 matrix) correspond to one chroma component (one for Cb and one for Cr). During supplementation, the missing chroma information is estimated using an interpolation algorithm. Horizontal interpolation: For adjacent luminance pixels in the same row, based on the existing chroma component values, the chroma value corresponding to the middle pixel is calculated using linear interpolation. For example, if the chroma values ​​of luminance pixels in a certain row are Cb1 and Cb3, the chroma value of the middle pixel can be calculated as Cb2 = (Cb1 + Cb3) / 2. Vertical interpolation: For the luminance pixels corresponding to positions in two adjacent rows, interpolation is also performed based on the existing chrominance components. For example, if the chrominance value of a certain position in the first row is Cr1 and the corresponding position in the third row is Cr3, then the Cr2 value of the corresponding position in the second row can be calculated as Cr2 = (Cr1 + Cr3) / 2. After interpolation, each luminance pixel can correspond to independent Cb and Cr components, forming 4:4:4 format data with the same horizontal and vertical resolution as the luminance.

[0032] Chroma component supplementation in 4:2:2 format video data has a chroma sampling density of half that of the luminance component in the horizontal direction (the vertical direction is the same as the luminance), meaning that every two luminance pixels correspond to one chroma component (one for Cb and one for Cr). Supplementation primarily involves interpolation in the horizontal direction: for consecutive luminance pixels in the same row (e.g., Y1, Y2, Y3, Y4), if the corresponding chroma components are Cb1 (corresponding to Y1, Y2) and Cb2 (corresponding to Y3, Y4), then the Cb value of the pixels between Y2 and Y3 is calculated through interpolation (e.g., Cb median = (Cb1 + Cb2) / 2), ensuring that each luminance pixel has its own dedicated chroma component. Since the chroma sampling in the vertical direction is the same as the luminance, no additional supplementation is needed; the original components are maintained.

[0033] After supplementing the chroma components using the above interpolation algorithm, non-4:4:4 format video data is converted to the standard 4:4:4 format. The core purpose is to eliminate the data volume differences caused by different chroma sampling formats, providing a unified base data for subsequent color space conversions (such as YUV to RGB). This restores chroma information at full horizontal and vertical resolution, ensuring accurate image color reproduction and avoiding color distortion caused by missing chroma information.

[0034] After standardizing the data format, it is easier to encode and store the data with a fixed 24-bit pixel width (8 bits for each RGB), which is compatible with the 512-bit burst transmission mode of DDR. For example, for a 4:2:0 format video with a resolution of 1920×1080, after chroma supplementation, the original structure of each 2×2 luma pixel sharing one set of chroma components is broken. Each pixel in the 1920 columns × 1080 rows has independent chroma information. The total data volume is converted from the original (1920×1080×8-bit luma + 960×540×8 BITCb + 960×540×8 BITCr) to 1920×1080×24-bit (8 bits for each RGB) consistent with the 4:4:4 format, providing standardized data for subsequent processing.

[0035] The converted video data undergoes color space conversion to generate RGB format data. Specifically, the CSC algorithm converts YUV and YCBCR formats to RGB4:4:4 format, and a mathematical model is used to calculate color components, unifying the data bit width to 24 bits and the color space range to 0-255. The CSC (Color Space Conversion) algorithm is a core technology for achieving unified video color space. In this application, this algorithm is used to convert input YUV, YCBCR, and other format data to RGB4:4:4 format, solving the processing compatibility issues caused by different color space formats. Its core logic is based on the mathematical mapping relationship of color components, accurately converting the luminance (Y) and chrominance (U / Cb, V / Cr) components into the three primary color components of red (R), green (G), and blue (B), ensuring that color information is transmitted without distortion during the conversion process.

[0036] For YCBCR format data, this application uses the following mathematical formula for conversion:

[0037] The red component (R') is calculated as R'=1.164*(Y'-16)+1.596*(Cr'-128), where Y' is the luminance component (value range 16~235) and Cr' is the red chrominance component (value range 16~240). The black level offset is eliminated by subtracting the offset (16 and 128), and then R' is obtained by weighting the coefficients.

[0038] The green component (G') is calculated as G' = 1.164*(Y'-16) - 0.813*(Cr'-128) - 0.392*(Cb'-128). The green component is affected by luminance (Y'), red chromaticity (Cr'), and blue chromaticity (Cb'). Multi-coefficient weighting is used to cancel out chromaticity cross-interference and ensure accurate green reproduction.

[0039] The blue component (B') is calculated as B'=1.164*(Y'-16)+2.017*(Cb'-128). The blue component is mainly determined by luminance (Y') and blue chroma (Cb'). The influence of chroma is amplified by the coefficient to compensate for the information loss caused by chroma compression in the original format.

[0040] For YUV format data, this application uses another set of mathematical formulas:

[0041] R'=Y'+1.140*V';G'=Y'-0.394*U'-0.581*V';B'=Y'+2.032*U;where Y' is the luminance component, and U' and V' are the chrominance components. The direct mapping relationship enables fast conversion, which is suitable for scenarios with high real-time requirements.

[0042] The unified processing of data bit width and color space range is as follows: Bit width is unified to 24 bits: The converted R, G, and B components each retain 8 bits of precision (values ​​from 0 to 255), and the three are combined to form 24-bit RGB data (R8G8B8), adapting to the 512-bit burst transmission requirements of the subsequent storage encoding module (each 512 bits can carry 21 24-bit pixels). Color space range is adjusted to 0-255: In the original YCBCR format, Y' ranges from 16 to 235, and Cb' / Cr' ranges from 16 to 240. Through offset correction in the formula (e.g., Y'-16, Cr'-128), the output R', G', and B' component ranges are standardized to 0-255, eliminating color gradation differences between different devices.

[0043] After color space conversion and standardization using the CSC algorithm, video data offers the following advantages: A unified RGB4:4:424-bit format ensures efficient processing of subsequent storage encoding, DDR read / write modules based on this fixed format. Accurate color component reproduction avoids color casts and color banding issues caused by format differences. Standardized bit width and range provide a foundation for parallel processing of multi-channel data, supporting simultaneous storage and retrieval of 10 video streams. For example, for a 1920×1080 resolution YCBCR format video, after the above conversion, each frame exists in RGB4:4:424-bit format, and the color information of each line of 1920 pixels can be stored in DDR through 92 512-bit burst transmissions, reducing bandwidth usage to approximately 1 / 1000th compared to traditional solutions.

[0044] The decoded audio data undergoes frequency identification processing to generate sampling frequency parameters. Specifically, an FPGA counter, combined with a 12.288MHz reference frequency, calculates the sampling frequency and quantization depth of the input audio. The core principle of using an FPGA counter in conjunction with the 12.288MHz reference frequency to calculate the audio sampling frequency and quantization depth is based on the fixed relationship between the clock signal and the sampling frequency in the I2S audio protocol. The I2S protocol transmits audio data via three wires: SCK (serial clock), WS (channel select), and SD (serial data). The SCK frequency has a clear mathematical relationship with the sampling frequency and quantization depth: SCK frequency = sampling frequency × quantization depth × 2 (multiplied by 2 due to the alternating transmission of the left and right channels). The FPGA integrates a high-precision counter that synchronously counts the 12.288MHz reference frequency (256 times the 48kHz sampling rate, facilitating integer multiple frequency division adaptation) and the SCK clock of the input audio, achieving parameter identification through frequency comparison.

[0045] The sampling frequency calculation process is as follows: Clock counting and period measurement: The FPGA counter simultaneously counts the 12.288MHz reference frequency and the audio SCK clock. A fixed measurement period is set (e.g., 1ms), and the number of pulses of the reference frequency (denoted as N_base) and the number of pulses of the SCK clock (denoted as N_sck) within this period are counted. Since the 12.288MHz frequency is stable, N_base is a fixed value (12288 pulses within 1ms).

[0046] SCK frequency conversion: Calculate the SCK frequency based on the counting results: SCK frequency = (N_sck / N_base) × 12.288MHz. For example, if N_sck is 186624 within 1ms, then the SCK frequency = (186624 / 12288) × 12.288MHz = 15.36MHz.

[0047] Sampling frequency derivation: Based on the relationship between SCK and sampling frequency in the I2S protocol, given the quantization depth (which can be identified through the data frame structure), the sampling frequency = SCK frequency / (quantization depth × 2). If the above 15.36MHz SCK corresponds to a quantization depth of 16 bits, then the sampling frequency = 15.36MHz / (16 × 2) = 48kHz.

[0048] Quantization depth is determined by analyzing the I2S data frame structure: the data bit width of each channel in the I2S protocol is the quantization depth. During data transmission, data is sent sequentially from the MSB (most significant bit) to the LSB (least significant bit). The transition of the WS signal indicates channel switching and the start of a data frame. The FPGA directly determines the quantization depth by monitoring the number of valid data bits on the SD line within the WS signal cycle (i.e., the number of pulses from the start to the end of the frame). For example, if 24 valid data bits are transmitted on the SD line within the WS cycle, the quantization depth is 24 bits.

[0049] The identified audio data undergoes quantization and frequency conversion to generate normalized audio data. Specifically, the audio quantization depth is unified to 16 bits, and frequency alignment conversion is performed using a FIFO method to normalize the sampling frequency to 48kHz. Audio adjustment factors and volume adjustment parameters are also added. The CSC (Color Space Conversion) algorithm is a core technology for achieving unified video color space. In this application, this algorithm is used to convert input YUV, YCBCR, and other format data into RGB4:4:4 format, solving the processing compatibility issues caused by different color space formats. Its core logic is based on the mathematical mapping relationship of color components, accurately converting the luminance (Y) and chrominance (U / Cb, V / Cr) components into the three primary color components of red (R), green (G), and blue (B), ensuring that color information is transmitted without distortion during the conversion process.

[0050] The data bit width and color space range are unified, with the bit width standardized to 24 bits: the converted R, G, and B components each retain 8 bits of precision (values ​​from 0 to 255), and the three are combined to form 24-bit RGB data (R8G8B8), adapting to the 512-bit burst transmission requirements of the subsequent storage encoding module (each 512 bits can carry 21 24-bit pixels). The color space range is adjusted to 0 to 255: in the original YCBCR format, Y' ranges from 16 to 235, and Cb' / Cr' ranges from 16 to 240. Through offset correction in the formula (such as Y'-16, Cr'-128), the output R', G', and B' component ranges are standardized to 0 to 255, eliminating color gradation differences between different devices.

[0051] After color space conversion and standardization using the CSC algorithm, video data offers the following advantages: A unified RGB4:4:424-bit format ensures efficient processing of subsequent storage encoding, DDR read / write modules based on this fixed format. Accurate color component reproduction avoids color casts and color banding issues caused by format differences. Standardized bit width and range provide a foundation for parallel processing of multi-channel data, supporting simultaneous storage and retrieval of 10 video streams. For example, for a 1920×1080 resolution YCBCR format video, after the above conversion, each frame exists in RGB4:4:424-bit format, and the color information of each line of 1920 pixels can be stored in DDR through 92 512-bit burst transmissions, reducing bandwidth usage to approximately 1 / 1000th compared to traditional solutions.

[0052] In another implementation, such as Figure 3 As shown, the input raw data consists of four channels of external video data (EXVCH1-EXVCH4), all in TMDS video data stream format and YUV16-bit external synchronization mode. After passing through the decoding module, each data stream enters the color space processing module. The color space processing module performs resolution format recognition and color space conversion on the four channels of video data, converting the original 16-bit YUV format to 24-bit RGB format, completing the first stage of normalization processing.

[0053] After the first stage of normalization, the data from the four channels (EXVCH1-EXVCH4) enters the DDR write arbitrator. The DDR write arbitrator controls the write priority to determine the order in which data from each channel is written to the DDR. The data then enters the second and third stages of normalization. Afterwards, it enters the storage encoding sequence module and the data buffer module, where the stored data undergoes serial-to-parallel conversion, transforming 24 bits into 504 bits. During processing, the channel number, DDR address code, and BANK code are added in FIFO mode for subsequent data identification. Simultaneously, a 512-bit data buffer operation is performed with a data width of 512 bits. Data reading is performed by the DDR controller. The processed data is finally written to the corresponding DDR channels (DDRCH1-DDRCH4) for storage.

[0054] In another implementation, such as Figure 4 The diagram illustrates the processing flow of the BT1120 internal synchronization signal video data stream (using BTACH9, BTAFCH9, and BTAFSCH9 as example channels) in an FPGA. Specifically, it includes the following steps: The raw data encompasses external signals (such as video signals involved in recording and playback, USB signal conversion data, network video, audio streams, etc.) and the BT1120 internal synchronization signal video data stream. Each data stream decoding module decodes the input BT1120 internal synchronization signal video data stream. The normalization process is divided into three stages. The first stage uses a color space processing module to identify the resolution format and convert the color space, transforming the original 16-bit YUV format into a 24-bit RGB format. Subsequent stages further process the data to unify the format. After normalization, the data enters the storage encoding sequence module and the data buffer module for serial-to-parallel conversion (24-bit to 504-bit). The channel number, DDR address code, and BANK code are added in FIFO mode for identification, and the data is temporarily stored with a 512-bit width and 512 data depths. Finally, the DDR controller reads and writes the data to the corresponding DDR channel (e.g., DDRCH19).

[0055] S103, encodes and embeds ID identification codes into the normalized data to generate tagged data to be stored.

[0056] In one implementation, the normalized video data is divided into encoding units to generate line-encoded video data blocks. Each image line is used as an encoding unit, and the number of bursts required for each line is calculated based on a 512-bit burst data width, ensuring that each line of data is stored and transmitted independently. Specifically, the normalized video data is divided into encoding units line by line, with each image line being an independent encoding unit. The number of bursts required for each line is calculated using the 512-bit burst data width of DDR3, ensuring that each line of data can be stored and transmitted independently. The specific calculation method is as follows: a 512-bit burst data width can transmit 21 24-bit pixels in parallel (512 / 24≈21.33, rounded up to 21). Therefore, the number of pixels per line divided by 21 (rounded up) is the number of bursts required for each line.

[0057] For video data with a resolution of 1920×1080, each line contains 1920 pixels, and the number of bursts required per line is approximately 1920 / 21 ≈ 91.42, which is rounded up to 92. Through 92 512-bit burst transmissions, a complete line of pixel data can be stored, enabling independent storage and transmission of each line of data.

[0058] Normalized audio data is divided into encoding units by blocks, with a custom burst unit of 92 blocks. This adapts to the 512-bit width and burst mode of DDR3, enabling batch storage of audio data. The unit design is consistent with the number of bursts per line of video, facilitating multi-channel data scheduling and bandwidth allocation. The normalized audio data has a 48kHz sampling rate and a 16-bit bit depth. Each data block contains a specific number of audio frames, and 92 blocks form a burst unit. By transmitting 92 blocks of audio data in a single burst, adapting to the 512-bit width of DDR3, and matching the burst transmission rhythm of video data, DDR bandwidth utilization is improved.

[0059] Each segmented audio and video data block is embedded with a unique ID identifier, forming tagged data to be stored. The ID identifier includes a fixed packet header identifier (F4F4F4, 1A1A1A), an 8-bit channel code (channels 0~255), an 8-bit resolution identifier (e.g., 01 corresponds to 1920*1080P60), an 8-bit packet data length code, a 2-bit BANK code (00~11 corresponds to BANK0~3), a 22-bit DDR base address code, and a fixed packet tail identifier (F4F41A). The ID code enables address location and channel differentiation of the data block.

[0060] like Figure 5The example shown uses 1920*1080 resolution image data from channel 3, with a BANKER of 01 and a planned DDR base address of 0x300000. The data packet structure contains multiple fields, as follows: Each packet header identification code contains two 24-bit identification codes: F4F4F4 and 1A1A1A. ID identification code: 8-bit 03, representing channel 3. Resolution identification code: 8-bit 01, corresponding to 1920*1080 resolution. Burst data length: 8-bit 5C. DDR BANK code: 2 bits. DDR base address code: 22-bit 300000, i.e., the planned DDR base address 0x300000. Each packet tail identification code: 24-bit F4F41A. Next, pixel data: 24 bits, representing the actual pixel information. This ID code accurately locates the storage location and channel of the data block.

[0061] S104 performs FIFO buffering and write priority determination on the encoded data to generate priority queue data.

[0062] In one implementation, the encoded audio and video data is buffered using a FIFO to generate temporary buffer data. Each channel's data is independently buffered using a 512×512 FIFO, and the write depth is monitored in real-time using the FIFO's data_count port to provide a basis for priority determination. Sending the encoded audio and video data to independent 512×512 FIFOs for temporary buffering by channel is a key step in achieving parallel processing of multi-channel data. Each channel has its own FIFO, ensuring that data streams of different types (video / audio) and from different sources are isolated during the buffering stage, avoiding cross-interference. The 512×512 specification of the FIFO indicates that its data bit width is 512 bits and its depth is 512 groups, which can adapt to the 512-bit burst transmission bit width of DDR3 and can buffer up to 512 groups of data (each group contains 21 24-bit pixels or corresponding audio blocks).

[0063] The data_count port of the FIFO monitors the cache data depth in real time. This port can provide real-time feedback on the number of data groups currently cached (range 0~512), providing a quantitative basis for priority determination. For example, in the FIFO of channel 1 (video), each line of 1920-pixel data after encoding enters the cache in the form of 92 512-bit burst transmissions. The data_count will gradually increase as data is written. When the cache depth reaches 300, it indicates that 300 data groups (21 pixels each) are currently cached, corresponding to approximately 300 × 21 = 6300 pixels, which has not reached the priority trigger threshold (448), and is temporarily classified as low priority. In the FIFO of channel 2 (audio), the encoded audio data is written in bursts of 92 blocks. The data_count feedback depth is 450, indicating that 450 audio blocks have been cached, which exceeds the priority one threshold (448). It needs to be prioritized for writing to DDR to avoid FIFO overflow.

[0064] This isolation caching mechanism, combined with real-time deep monitoring, not only ensures independent processing of multi-channel data but also provides accurate quantitative basis for dynamic priority arbitration, ensuring that high-priority data (such as FIFOs about to overflow) can be processed first, thereby improving the utilization of DDR bandwidth and the stability of data transmission.

[0065] The depth of the FIFO cached data is determined, and priority identifiers are generated. Data depths of 448 are marked as priority one, 384 as priority two, and 256 as priority three. Priority one is forcibly triggered when the video data cache reaches six lines. Determining data depth and generating priority identifiers based on the FIFO's `data_count` value is the core logic for implementing dynamic DDR bandwidth allocation. This aims to quantify data cache pressure, ensuring that high-priority data is written first, thus preventing FIFO overflow. The specific rules and implementation logic are as follows:

[0066] The general priority determination (applicable to audio and video data) is as follows: Based on the FIFO's data_count value (number of cached data sets, ranging from 0 to 512), priorities are divided into three levels: Priority 1: Triggered when data_count reaches 448. At this time, the remaining FIFO cache space is 512-448=64 sets, corresponding to caching 64×21=1344 pixels (video) or 64 audio blocks. This is close to the cache limit, requiring urgent scheduling for writing to avoid overflow. Priority 2: Triggered when data_count reaches 384. The remaining cache space is 512-384=128 sets, with moderate cache pressure. Timely scheduling is required after high-priority data processing is completed. Priority 3: Triggered when data_count reaches 256. The remaining cache space is 512-256=256 sets, with low cache pressure. Ordered scheduling can be performed sequentially when there are no high-priority requests.

[0067] A specific mandatory priority rule for video data is implemented. In addition to the general rules, a "buffered lines" trigger condition is set for video data: when the number of buffered video lines in the FIFO reaches 6, it is forcibly marked as priority one, regardless of whether the current data_count value has reached 448. This rule is necessary because video data has line continuity requirements; 6 buffered lines are close to the real-time transmission threshold and must be written first to ensure frame integrity.

[0068] Example 1, video channel triggering general priority one:

[0069] A 1920×1080 resolution video channel requires 92 bursts of 512-bit data per line after encoding (corresponding to 92 sets of data). When the FIFO's data_count reaches 450, it exceeds the threshold of 448, triggering priority one. At this point, the amount of data buffered in the FIFO is 450×21=9450 pixels, approximately 1920×5=9600 pixels (about 5 lines). If not written in time, subsequent data may be lost due to the FIFO being full.

[0070] Example 2, audio channel trigger priority two:

[0071] An audio channel uses 92 blocks as burst units. When the FIFO's data_count reaches 390, it exceeds the threshold of 384 and is marked as priority two. At this point, the cached audio blocks can support approximately 390 / 92 ≈ 4.24 burst transmissions, resulting in moderate cache pressure. Data can be scheduled for writing after priority one data processing.

[0072] Example 3, video channel triggers forced priority one:

[0073] A certain video channel, due to its high input frame rate (e.g., 60fps), although the FIFO's data_count is only 300 (corresponding to 300 × 21 = 6300 pixels, approximately 3.3 lines), actually has 6 buffered lines (6 × 1920 = 11520 pixels), triggering a forced priority one rule. This rule ensures the line continuity of high frame rate video, preventing frame breaks due to insufficient buffering.

[0074] Data with priority identifiers is queued and sorted to generate a priority queue. Priority 1 data enters the queue array first. If there are three or more priority 1 data, the DDR read channel is closed and a forced switch to the write module is initiated, writing data according to the queue order. Priority 2 and 3 data enter the queue sequentially for scheduling when there are no higher priority requests. Queuing and scheduling audio and video data with priority identifiers according to preset rules is a key step in realizing dynamic DDR bandwidth allocation. The core logic controls the data writing order based on priority level and urgency, ensuring that high-priority data is stored first, avoiding buffer overflow or data gaps.

[0075] Priority data (including cases where data_count ≥ 448 or video buffer reaches 6 lines) enjoys the highest scheduling rights and is directly prioritized into the queuing array. When there are three or more priority data items in the queuing array, an emergency scheduling mechanism is triggered: a shutdown request is immediately sent to the DDR read channel. After the last burst read operation is completed, the DDR controller is forcibly switched to the write module, and priority data is processed sequentially according to the queuing order. This mechanism requires interrupting read operations to prioritize writing and avoid data loss because priority data is close to the FIFO overflow threshold (only 64 groups of space remain).

[0076] Priority data (data_count ≥ 384) and priority data (data_count ≥ 256) do not have emergency scheduling rights and must wait in the queuing array. Only when there is no priority data in the queuing array and no urgent read request in the DDR read channel will the data enter the scheduling queue in the order of "priority two before priority three" and wait for the write module to become available before being written.

[0077] The system simultaneously contains data from three priority-1 video channels (from HDMI, SDI, and BT1120 interfaces respectively). At this time, emergency scheduling is triggered: First, a shutdown signal is sent to the DDR read channel. Assuming that the current read channel is performing its 5th burst read, it is allowed to complete the read and then shut down the read channel. After the read channel is shut down, the DDR controller forces a switch to the write module and processes the data from the three channels in the queuing order (e.g., HDMI→SDI→BT1120). Two data blocks are written to each channel (each block contains 64 data blocks), for a total of 3×2=6 data blocks are written, quickly releasing the FIFO buffer space.

[0078] After the three priority-1 data are processed, the remaining data in the queue array are priority-2 (1 audio channel, data_count=390) and priority-3 (1 video channel, data_count=280): The write module prioritizes the audio data of priority-2 and writes it into 1 data block (64 blocks); after the audio data is written, the video data of priority-3 is scheduled and written into 1 data block, thus realizing sequential processing when there are no high-priority requests.

[0079] The priority-one emergency scheduling mechanism prioritizes writing when multiple channels are nearing overflow, preventing data loss due to FIFO overflow, which is particularly suitable for high-bandwidth video data. Emergency scheduling allows read channels to complete their current burst read before closing, ensuring data continuity while minimizing write operation wait time and reducing read / write conflicts. The waiting mechanisms for priority-two and three data prevent low-priority data from occupying high-priority resources, directing DDR bandwidth towards channels that truly need it. Real-world testing shows this can reduce single-channel bandwidth usage to approximately 1 / 1000, supporting parallel processing of 10 video channels and 11 audio channels. In summary, this scheduling rule, through the logic of "emergency interrupt + sequential waiting," achieves efficient and secure storage of multi-channel data in DDR, and is a core component of the dynamic bandwidth allocation mechanism.

[0080] S105 performs DDR writing and BANK scheduling on priority queue data to generate parallel data for stable storage.

[0081] In one implementation, priority queue data is written and scheduled to generate data to be written to DDR. The DDR write arbitrator schedules data according to the priority queue order. Priority-1 data is written first after the read channel is forcibly closed, with two data blocks written. Priority-2 and priority-3 data are written sequentially when there are no urgent read requests, with each data block written. The DDR write arbitrator schedules data according to the priority queue order to generate data to be written to DDR. Specifically, priority-1 data has the highest priority and is written first after the DDR read channel is forcibly closed (after reading the last burst), with two data blocks written each time (each block containing 64 data blocks) to quickly release FIFO buffer space and avoid data overflow. Priority-2 and priority-3 data are written sequentially according to the queue order when there are no urgent read requests, with each data block written each time to balance bandwidth usage.

[0082] Specifically, the priority queue contains three video channel data sets with priority 1, one audio channel data set with priority 2, and one video channel data set with priority 3. The DDR write arbitrator first processes the priority 1 data, forcibly closes the read channel, and then writes two data blocks to each channel in sequence. After the priority 1 data is processed, it then writes one data block to the audio channel with priority 2 and one data block to the video channel with priority 3.

[0083] The system performs bank allocation and switching for written data, generating conflict-free storage instructions. Specifically, the bank switcher allocates write data to a designated bank while forcing read operations to occur on other banks, achieving frame-level read / write region separation and avoiding read / write conflicts within the same region. The core of this system is allocating write data to a designated bank while forcing read operations to occur on other banks, thus achieving frame-level read / write region separation and preventing read / write conflicts within the same region, ensuring the stability of data storage and retrieval.

[0084] The BANK switcher allocates priority-1 video data to BANK1, at which point read operations are forced to be performed in BANK2; when processing priority-2 audio data, it allocates it to BANK3, and read operations are performed in BANK0. In this way, read and write operations are ensured to be performed between different BANKs to avoid conflicts.

[0085] The allocated data stream is written to DDR storage to generate stable parallel data. This is achieved by using four DDR3 chips working in parallel, utilizing their 16-bit width and 8-burst mode to achieve a 512-bit data burst write, with a total bandwidth of 51.2Gbps, ensuring efficient data storage. The data stream allocated by the bank is also written to DDR storage to generate stable parallel data. Using four DDR3 chips working in parallel, each with a 16-bit width and an 8-burst mode, a 512-bit data burst write is achieved, with a total bandwidth of 51.2Gbps (16bit × 4 × 800MHz), ensuring efficient data storage and meeting the processing requirements of multi-channel parallel data streams.

[0086] Four DDR3 chips operate simultaneously, each handling 128 bits of data transmission. Through an 8-burst mode, 512 bits of data are written at once. For a 1920×1080 resolution video data line, after 92 such burst writes, a complete line of data can be stored in the DDR, and the total bandwidth can support the parallel storage of 10 channels of video data streams and 11 channels of audio data streams.

[0087] S106 performs coordinated reading and output processing of DDR stored data to generate a continuous output data stream.

[0088] In one implementation, ID identification code decoding processing is performed on DDR stored data to generate data location information. Specifically, by reading the key ID identification code module, the channel code, bank code, and DDR base address code in the data packet header are parsed to determine the channel to which the data block belongs, its storage location, and the read pointer. The key ID identification code module decodes the ID identification code of the data stored in the DDR, parsing out the 8-bit channel code, 2-bit bank code, and 22-bit DDR base address code in the data packet header, thereby determining the channel to which the data block belongs, the storage bank location, and the specific read pointer, thus generating data location information.

[0089] Parsing the ID identification code of a certain data block yields an 8-bit channel code of 03 (corresponding to channel 3), a 2-bit BANK code of 01 (corresponding to BANK1), and a 22-bit DDR base address code of 300000. This determines that the data block belongs to channel 3 and is stored at address 0X300000 in BANK1. The read pointer points to this address.

[0090] The located data stream is buffered in a read FIFO and priority is determined to generate an ordered read queue. Data is stored in a 512×512 read FIFO, and the data depth is monitored by a read counter. Data with a depth < 64 is marked as priority 1, < 256 as priority 2, and < 384 as priority 3. The DDR read arbitrator schedules reads according to these priorities. The located data stream is buffered in a 512×512 read FIFO, and the data depth of the FIFO is monitored in real time by a read counter. Priorities are marked according to the depth: depth < 64 is marked as priority 1; depth < 256 is marked as priority 2; and depth < 384 is marked as priority 3. The DDR read arbitrator schedules reads according to these priorities, forming an ordered read queue. After the data stream from channel 2 is stored in the read FIFO, the read counter shows a data depth of 50 (< 64), and it is marked as priority 1; after the data stream from channel 5 is stored, the depth is 200 (< 256), and it is marked as priority 2. The DDR read arbiter prioritizes reading data from channel 2, and then schedules data from channel 5.

[0091] The system performs channel buffering and output coordination on the read data to generate a continuous output data stream. The channel data buffering module caches 512 bits of read data. When the amount of data is less than one line, a priority-one read request is triggered; when the amount of data is greater than six lines, the request is stopped. This, combined with the read-write separation mechanism of the BANK switch, ensures output continuity. The channel data buffering module temporarily caches the 512 bits of read data. When the cached data amount is less than one line (e.g., 92 lines of data corresponding to 1920 pixels), a priority-one read request is triggered; when the amount of data is greater than six lines, the read request is stopped. Simultaneously, combined with the read-write separation mechanism of the BANK switch (write operations in one BANK, read operations in other BANKs), the continuity of the output data stream is guaranteed.

[0092] When only 80 lines of 1920×1080 resolution video data remain in the buffer of the channel data storage module (less than 92 lines), a priority-1 read request is triggered to continue reading data. When the data volume reaches 7 lines (more than 6 lines), the request stops. At this time, the BANK switch controls write operations to BANK0 and read operations to BANK1 to avoid read-write conflicts and ensure that the output video data stream is continuous and uninterrupted.

[0093] S107 dynamically allocates bandwidth and optimizes energy efficiency throughout the entire data stream processing process, generating efficient parallel processing results.

[0094] In one implementation, dynamic bandwidth allocation adjusts the DDR bandwidth usage ratio in real time based on data priority, cache depth, and channel type (video / audio) to ensure high-priority data is transmitted first, while balancing the bandwidth demands of multiple channels. Specifically, priority 1 data (FIFO depth ≥ 448 or video cache of 6 lines) occupies 80%~90% of the bandwidth, priority 2 (depth ≥ 384) occupies 50%~60% of the bandwidth, and priority 3 (depth ≥ 256) only occupies the remaining bandwidth when there are no higher-priority requests. A priority queue mechanism prevents a single channel from occupying bandwidth for extended periods. When three or more priority 1 data items exist, the read channel is forcibly closed and bandwidth is concentrated for writing, ensuring timely storage of urgent data. A bank switch separates read and write regions (write operations are performed on one bank, while read operations are performed on other banks), avoiding bandwidth waste due to conflict waiting. The system processes 10 video streams and 11 audio streams simultaneously. When 3 video channels trigger priority one, the dynamic bandwidth allocation mechanism allocates 80% of the bandwidth to these 3 video channels and prioritizes writing 2 data blocks. The remaining 20% ​​of the bandwidth is allocated to the audio channel with priority two when there are no urgent requests, ensuring that all multi-channel data can be transmitted efficiently.

[0095] Energy efficiency optimization reduces unnecessary power consumption by dynamically adjusting the DDR's operating state and frequency. Specific measures include: Frequency reduction during idle periods: When all channel data cache depths are below 256 (no priority requests), the DDR allocates 30% of its time to reduce the operating frequency, minimizing power consumption from continuous read / write operations. Read / write merging scheduling: When low-priority data (second and third priority) is queued, adjacent read / write requests are merged, reducing the number of DDR controller state transitions and power consumption. Dynamically shutting down idle channels: For channels with no data transmission for extended periods, their corresponding FIFO and arbitrator ports are shut down, and clock supply is stopped, reducing static power consumption. At night, when system load is low and all channel data cache depths are below 256, the DDR automatically allocates 30% of its time to reduce the clock frequency to 400MHz; simultaneously, it shuts down the ports of three idle video channels and two audio channels, resulting in an overall power consumption reduction of approximately 40%.

[0096] By combining dynamic bandwidth allocation and energy efficiency optimization, the following effects are achieved: Multi-channel parallel support: Under non-full load conditions, it stably supports parallel processing of 10 video channels (1920×1080P60) and 11 audio channels (48KHz, 16bit) without packet loss or latency. Maximized bandwidth utilization: The utilization of the total DDR bandwidth (51.2Gbps) is dynamically adjusted according to the load, reaching 90% during priority data transmission and dropping to 20% during idle periods, balancing efficiency and power consumption. Low-power operation: After energy efficiency optimization, the average system power consumption is reduced by 30%~40%, with particularly significant effects during low-load periods.

[0097] like Figure 7As shown, a multi-task parallel data stream processing device based on FPGA includes: a construction module 701, used to construct a multi-format raw data receiving and decoding system, wherein the raw data includes TMDS signals, I2S serial audio data, BT1120 internal synchronization data stream, SERDES SDI high-speed serial signals, and custom SPI protocol data; the decoding operation covers video synchronization signal separation, automatic identification of SDI rate and resolution, SPI protocol command code parsing, and generation of 24-bit audio data stream; a processing module 702, used to normalize the decoded data to generate a unified format data stream, wherein the video data is converted to RGB4:4:424BIT format using the CSC algorithm and the color space range is unified to 0~255; the audio data is aligned to a 12.288MHz reference frequency, converted to 48KHz sampling rate, 16-bit depth data, and an adjustment factor is added; the normalized data is encoded and ID identification code embedded to generate tagged data to be stored; the encoded data is buffered by FIFO and write priority is determined to generate priority queue data, wherein the 51 The 2×512 FIFO uses `data_count` to determine data depth, assigning 448 as priority 1, 384 as priority 2, and 256 as priority 3. A priority 1 request is triggered when the video buffer reaches 6 lines. DDR writing and bank scheduling are performed on the priority queue data to generate stable, parallel data storage. The write arbiter schedules according to priority, the bank switch implements frame-level read / write separation, and four DDR3 chips perform 512-bit burst writes in parallel. Reading and output coordination are performed on the DDR stored data to generate a continuous output data stream. Data is located by decoding the ID identification code, reads are scheduled according to depth in the FIFO, and read requests are triggered by monitoring the data volume in the channel temporary storage FIFO. Dynamic bandwidth allocation and energy efficiency optimization are performed throughout the data stream processing to generate efficient parallel processing results.

[0098] A computing device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute any of the FPGA-based multi-task parallel data stream processing methods.

[0099] The methods and / or embodiments in this application can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by a processing unit, it performs the functions defined in the methods of this application.

[0100] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0101] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0102] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application.

Claims

1. A method for processing multi-task parallel data streams based on FPGA, characterized in that, include: A multi-format raw data receiving and decoding system is constructed. The raw data includes TMDS signals, I2S serial audio data, BT1120 internal synchronization data stream, SERDES SDI high-speed serial signals and custom SPI protocol data. The decoding operation covers video synchronization signal separation, automatic identification of SDI rate and resolution, SPI protocol command code parsing and audio 24-bit data stream generation. The decoded data is normalized to generate a data stream with a uniform format, and an adjustment factor is added. Normalized data is encoded and ID identification codes are embedded to generate identifiable data to be stored. This includes: dividing normalized video data into encoding units to generate line-encoded video data blocks, where each image line is the encoding unit, and the number of bursts required for each line is calculated based on a 512-bit burst data width to ensure that each line of data is stored and transmitted independently; dividing normalized audio data into encoding units to generate block-encoded audio data blocks, where 92 custom blocks are used as burst units, adapting to the 512-bit width and burst mode of DDR3 to achieve batch storage of audio data; and embedding ID identification codes into the divided audio and video data blocks to generate identifiable data to be stored. The ID identification code includes a fixed packet header flag code, channel code, resolution identification code, packet data length code, BANK code, DDR basic address code, and fixed packet tail flag code. The ID code enables address location and channel differentiation of the data block. The encoded data is buffered in FIFO and the write priority is determined to generate a priority queue. When the cache reaches the preset threshold, the preset priority request is triggered. Priority queue data is processed by DDR writing and BANK scheduling to generate parallel data for stable storage. The write arbiter schedules according to priority, the BANK switch realizes frame-level read-write separation, and four DDR3 chips are used in parallel to realize 512-bit wide burst writing. The DDR storage data is read and output in a coordinated manner to generate a continuous output data stream. In this process, the data is located by decoding the ID identification code, the FIFO is read by depth scheduling, and the channel temporary storage FIFO monitors the amount of data to trigger read requests. Dynamic bandwidth allocation and energy efficiency optimization are performed throughout the entire data stream processing process to generate efficient parallel processing results.

2. The FPGA-based multi-task parallel data stream processing method according to claim 1, characterized in that, The decoded data is normalized to generate a data stream with a uniform format, including: The decoded video data is subjected to chroma sampling conversion to generate 4:4:4 standard chroma format data. Among them, the non-4:4:4 format chroma components containing 4:2:0 and 4:2:2 are supplemented to complete the chroma sampling standard of full horizontal and vertical resolution. The converted video data undergoes color space conversion to generate RGB format data. The decoded audio data undergoes frequency identification processing to generate sampling frequency parameters. Specifically, the sampling frequency and quantization depth of the input audio are calculated using an FPGA counter in conjunction with a 12.288MHz reference frequency. The identified audio data is quantized and frequency converted to generate normalized audio data. The audio quantization depth is unified to 16 bits, and the frequency alignment conversion is completed based on FIFO to normalize the sampling frequency to 48KHz. Audio adjustment factors and volume adjustment parameters are added.

3. The FPGA-based multi-task parallel data stream processing method according to claim 1, characterized in that, The encoded data is buffered in a FIFO manner and write priority is determined to generate a priority queue of data, including: The encoded audio and video data are buffered using a FIFO to generate temporary buffer data. Specifically, each channel's data is independently buffered using a 512×512 FIFO, and the write depth is monitored in real time using the FIFO's data_count port to provide a basis for priority determination. The depth of the FIFO cache data is determined, and a priority identifier is generated. When the data depth reaches 448, it is marked as priority one; when it reaches 384, it is marked as priority two; when it reaches 256, it is marked as priority three; priority one is forcibly triggered when the video data cache reaches 6 lines. Data with priority identifiers are sorted into a priority queue. Priority data is given priority to enter the queue array. If there are three or more priority data, the DDR read channel is closed and the module is forcibly switched to write. Data is written in the queue order. Priority data of priority two and three enter the queue in turn to wait for scheduling when there are no higher priority requests.

4. The FPGA-based multi-task parallel data stream processing method according to claim 3, characterized in that, Priority queue data is written to DDR and scheduled via BANK to generate parallel data for stable storage, including: The priority queue data is written and scheduled to generate data to be written to DDR. The DDR write arbiter schedules the data according to the priority queue order. Priority 1 data is written first after the read channel is forcibly closed, and two data blocks are written. Priority 2 and 3 data are written sequentially when there is no urgent read request, and one data block is written for each. The write data is allocated and switched to a BANK, generating conflict-free storage instructions. The BANK switcher allocates the write data to a specified BANK, while forcing read operations to be performed on other BANKs, thus achieving frame-level read and write area separation and avoiding read and write conflicts in the same area. The allocated data stream is written to DDR storage to generate parallel data for stable storage.

5. The FPGA-based multi-task parallel data stream processing method according to claim 1, characterized in that, The system performs coordinated reading and output processing of DDR memory data to generate a continuous output data stream, including: The DDR storage data is processed by ID identification code decoding to generate data location information. Specifically, the channel code, BANK code and DDR basic address code of the data packet header are parsed by reading the key ID identification code module to determine the channel, storage location and read pointer of the data block. The data stream after positioning is buffered by a read FIFO and priority is determined to generate an ordered read queue. The data is stored in a 512×512 read FIFO. The data depth is monitored by a read counter. When the depth is <64, it is marked as priority 1; when it is <256, it is marked as priority 2; and when it is <384, it is marked as priority 3. The DDR read arbitrator schedules the reading according to the priority. The read data is temporarily stored in the channel and coordinated with the output to generate a continuous output data stream. The channel data storage module buffers 512 bits of read data. When the amount of data is less than 1 line, a priority 1 read request is triggered. When the amount of data is greater than 6 lines, the request is stopped. Combined with the read-write separation mechanism of the BANK switch, the output continuity is guaranteed.

6. A processing device for multi-task parallel data streams based on FPGA, characterized in that, The apparatus for implementing the method of claim 1 includes: The building module is used to build a multi-format raw data receiving and decoding system. The raw data includes TMDS signals, I2S serial audio data, BT1120 internal synchronization data stream, SERDES SDI high-speed serial signals and custom SPI protocol data. The decoding operation covers video synchronization signal separation, automatic identification of SDI rate and resolution, SPI protocol command code parsing and audio 24-bit data stream generation. The processing module is used to normalize the decoded data, generate a unified format data stream, and add adjustment factors; encode and embed ID identification codes into the normalized data to generate tagged data to be stored; perform FIFO buffering and write priority determination on the encoded data to generate a priority queue data, and trigger a preset priority request when the cache reaches a preset threshold; perform DDR write and BANK scheduling on the priority queue data to generate stable parallel data storage, wherein the write arbitrator schedules according to priority, the BANK switch implements frame-level read-write separation, and four DDR3 chips achieve 512-bit burst write in parallel; perform read and output coordination processing on the DDR stored data to generate a continuous output data stream, wherein data is located by decoding the ID identification code, read FIFO is read according to depth scheduling, and channel temporary storage FIFO monitors the data volume to trigger read requests; and perform dynamic bandwidth allocation and energy efficiency optimization throughout the data stream processing process to generate efficient parallel processing results.

7. An electronic device, characterized in that, include: First processor; and memory for storing executable instructions of the first processor; The first processor is configured to execute the FPGA-based multi-task parallel data stream processing method according to any one of claims 1 to 5 by executing the executable instructions.

8. A computing device, the device comprising a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein, When the computer program instructions are executed by the processor, the device is triggered to execute the FPGA-based multi-task parallel data stream processing method as described in any one of claims 1 to 5.

Images