A multi-layer video data fusion method and system based on FPGA

Abstract

The present invention discloses a multi-layer video data fusion method and system based on FPGA, comprising: respectively obtaining a frame of first video data sent by each video layer in a plurality of video layers, and performing a pre-processing operation of cropping and reducing the first video data to obtain a plurality of frames of second video data; generating a DMA write request and a DMA read request for each frame of the second video data, and sequentially writing a plurality of frames of the second video data into a DDR and sequentially reading each frame of the second video data from the DDR through an arbitration mechanism preset in the DMA; fusing a plurality of frames of the second video data and outputting the video data obtained after fusion, so as to improve the efficiency and management of the cache.

Description

Technical Field

[0001] The present invention relates to the field of digital video caching technology of FPGA, and in particular to a multi-layer video data fusion method and system based on FPGA. Background Art

[0002] In video data processing and transmission systems, due to the huge amount of data, CPU is generally not used for video data processing and caching. Instead, high-parallel chips such as FPGA are used. Thanks to the parallel processing capabilities, low latency and customizability of FPGA, FPGA has become an indispensable hardware platform in modern video processing systems, especially in the fields of real-time video stream processing, high-definition video encoding and decoding, video special effects rendering, etc. FPGA is often used to achieve high-performance data processing and caching functions.

[0003] Video data usually has the characteristics of high bandwidth and high transmission rate. Especially in multi-layer video processing scenarios, a large amount of pixel data from different layers needs to be processed simultaneously. In order to ensure the real-time performance of video data, an efficient data caching mechanism is essential. Video data cache can temporarily store the video frame data to be processed or processed, alleviate the mismatch between the data source and the processing unit in speed and capacity, reduce data access delay, and improve the overall system performance.

[0004] At present, FPGA video data caching technology mainly includes local storage resource utilization technology, external DRAM interface technology and DMA (Direct Memory Access) technology. Among them, the local storage resource utilization technology uses on-chip BlockRAM (BRAM) as a temporary cache to store video frames or part of its data, but the internal BRAM resources of FPGA are limited, and cannot meet the data caching requirements for high-resolution or multi-layer videos. The external DRAM interface technology uses a high-speed DDR or LPDDR interface to connect to an external DRAM to provide a larger capacity cache space, but DRAM has high requirements for data transmission management and scheduling. Finally, DMA (Direct Memory Access) technology uses a DMA controller to realize automatic data movement, which can reduce the intervention of the CPU (or FPGA internal logic) in data transmission and improve data transmission efficiency. However, FPGA-based DMA controllers usually only support single-channel or simple multi-channel modes, and have limited concurrent cache management capabilities for multi-layer video data.

[0005] Although the existing FPGA video data caching technology has improved the data processing efficiency to a certain extent, when processing multi-layer video, facing multi-layer video streams, the existing caching scheme cannot effectively balance the caching requirements of each layer data, resulting in insufficient caching of some layer data, while the cache resources of other layers are idle. When multiple layers request cache resources at the same time, data access conflicts may occur, increasing access delays and affecting the real-time performance of video processing. Furthermore, the cache strategy design for multi-layer video data is complex, and it is necessary to consider factors such as data priority, dependency, and cache replacement. The automated management mechanism of the existing technology still has room for improvement, and the existing DMA controller cannot fully utilize the bus bandwidth, especially when processing multi-layer video data, there is bandwidth waste. Summary of the invention

[0006] In order to solve the above technical problems, the present invention discloses a multi-layer video data fusion method and system based on FPGA, which are used to realize efficient caching and management of multi-layer video data.

[0007] In order to achieve the above object, the present invention discloses a multi-layer video data fusion method based on FPGA, comprising:

[0008] Respectively obtaining a frame of first video data sent by each of the plurality of video layers, and performing a preprocessing operation of cropping and reducing the first video data to obtain a plurality of frames of second video data;

[0009] Generate a DMA write request for each frame of the second video data, and write several frames of the second video data into the FIFO in sequence according to the DMA write request and an arbitration mechanism preset in the DMA;

[0010] When the amount of data in the FIFO buffer reaches a threshold, the DDR is notified to read the data, several frames of the second video data are respectively saved in the DDR, and a DDR address is allocated to each frame of the second video data;

[0011] Generate a DMA read request for each frame of the second video data, and read each frame of the second video data from the DDR in sequence according to the DMA read request and the arbitration mechanism;

[0012] A plurality of frames of the second video data are fused and the fused video data is output.

[0013] The present invention discloses a multi-layer video data fusion method based on FPGA. First, a frame of first video data sent in each acquired video layer is pre-processed by cropping and reducing so as to meet the needs of DDR cache. Then, a DMA write request for each frame of second video data is generated. The write request and an arbitration mechanism preset in the DMA control memory are used to cache the several frames of second video data in sequence into the FIFO of the DMA. Then, the DDR cache is triggered according to the cache capacity of the FIFO, and the efficient storage of the video data of each video layer is realized through the arbitration mechanism.

[0014] After caching the multiple frames of second video data into the DDR, a DMA read request is generated for each frame of second video data, and then the data is read from the DDR according to the DMA read request and the arbitration mechanism to perform video superposition, thereby ensuring data superposition of multiple video layers, making full use of the storage resources of the DDR, and improving cache efficiency.

[0015] As a preferred example, the preprocessing operation of cropping and reducing the first video data to obtain a plurality of frames of second video data includes:

[0016] Reducing each frame of the first video data according to the size requirement of the video to be output, to obtain initial second video data corresponding to the first video data;

[0017] Each frame of the initial second video data is cropped according to the burst length of the DDR and the length of the initial second video data to obtain each frame of the second video data.

[0018] The present invention first reduces the size of the video sent by the video layer according to the size requirement of the output video, reduces the amount of video data processed by the cache while meeting the demand for video data, and improves the efficiency of the cache. Furthermore, the burst length of DDR and the length of the video data are used, where the length is the pixels of each row, to crop the video data to meet the cache requirement of DDR, thereby facilitating the use of DDR storage resources to store video data.

[0019] As a preferred example, the generating of a DMA write request for each frame of the second video data, and writing a plurality of frames of the second video data into the FIFO respectively according to the DMA write request and an arbitration mechanism preset in the DMA, comprises:

[0020] Acquire a first field synchronization signal of each frame of the second video data, and generate a DMA write request corresponding to each frame of the second video data according to a rising edge of the first field synchronization signal;

[0021] Obtaining a polling value of a DMA status register, and determining a read / write state of a DDR and a target video layer of the DMA polling according to the polling value; wherein the read / write state includes a DDR write state and a DDR read state;

[0022] When the DMA write request corresponding to the target video layer is received and the read / write state of the DDR is the DDR write state, the DMA write state machine is controlled to write each frame of the second video data into the FIFO through a valid display data selection signal.

[0023] The present invention uses the state register of the DMA to obtain the read and write state of the DDR in real time and controls the DMA polling to poll the several video layers respectively, so that the data sent by each video layer can be written into the DDR in time in turn, ensuring the efficiency of the cache.

[0024] As a preferred example, the step of storing several frames of the second video data in the DDR respectively and allocating a DDR address to each frame of the second video data includes:

[0025] When the amount of data in the FIFO buffer reaches a threshold, the DMA is controlled to send a DDR write signal to the DDR, so that the DDR is controlled according to the DDR signal to read and buffer each frame of the second video data, and a DDR buffer address is allocated for each frame of the second video data.

[0026] In the present invention, the data of each video layer will be stored in a fixed address of DDR, and only the corresponding data address will be read when reading. Each layer data is assigned a fixed position in DDR, ensuring that there will be no data access conflict and improving the efficiency of the cache.

[0027] As a preferred example, the generating of a DMA read request for each frame of the second video data, and sequentially reading each frame of the second video data from the DDR according to the DMA read request and the arbitration mechanism, includes:

[0028] Generate a second field synchronization signal corresponding to the video to be output, and generate a DMA read request corresponding to each frame of second video data according to the rising edge of the second field synchronization signal;

[0029] Obtaining a polling value of a DMA status register, and determining a read / write state of a DDR and a target video layer of the DMA polling according to the polling value; wherein the read / write state includes a DDR write state and a DDR read state;

[0030] When the DMA read request corresponding to the target video layer is received and the read / write state of the DDR is the DDR read state, the DMA read state machine is controlled to send a DDR read signal to read the DDR address of the target video layer, and read each frame of the second video data according to the DDR address.

[0031] The present invention takes into account that DDR cannot read video data of multiple layers at the same time, and sets an arbitration mechanism to poll each layer at all times to obtain data in all layers, thereby improving the accuracy of video superposition.

[0032] As a preferred example, the step of fusing several frames of the second video data and outputting the fused video data includes:

[0033] Obtaining a value of a priority register of each of the video layers, and sorting the plurality of video layers by priority according to the value;

[0034] A plurality of frames of the second video data are superimposed according to the order to obtain superimposed video data, and the superimposed video data is output.

[0035] The present invention superimposes the data of the plurality of video layers according to the setting of the priority register, so as to improve the effect of video superposition.

[0036] On the other hand, the present invention discloses a multi-layer video data fusion system based on FPGA, including a pre-processing module, a video writing module, a video caching module, a video reading module and a video fusion module;

[0037] The preprocessing module is used to respectively obtain a frame of first video data sent by each of the plurality of video layers, and perform a preprocessing operation of cropping and reducing the first video data to obtain a plurality of frames of second video data;

[0038] The video writing module is used to generate a DMA write request for each frame of the second video data, and write several frames of the second video data into the FIFO in sequence according to the DMA write request and an arbitration mechanism preset in the DMA;

[0039] The video buffer module is used for notifying the DDR to read data when the amount of data in the FIFO buffer reaches a threshold, saving several frames of the second video data into the DDR respectively, and allocating a DDR address for each frame of the second video data;

[0040] The video reading module is used to generate a DMA read request for each frame of the second video data, and read each frame of the second video data from the DDR in sequence according to the DMA read request and the arbitration mechanism;

[0041] The video fusion module is used to fuse several frames of the second video data and output the fused video data.

[0042] The present invention discloses a multi-layer video data fusion system based on FPGA. First, a frame of first video data sent in each acquired video layer is cropped and reduced, etc., so as to meet the needs of DDR cache. Then, a DMA write request for each frame of second video data is generated. The write request and an arbitration mechanism preset in the DMA control memory are used to cache the several frames of second video data in sequence into the FIFO of the DMA. Then, the DDR cache is triggered according to the cache capacity of the FIFO, and the efficient storage of the video data of each video layer is realized through the arbitration mechanism.

[0043] After caching the multiple frames of second video data into the DDR, a DMA read request is generated for each frame of second video data, and then the data is read from the DDR according to the DMA read request and the arbitration mechanism to perform video superposition, thereby ensuring data superposition of multiple video layers, making full use of the storage resources of the DDR, and improving cache efficiency.

[0044] As a preferred example, the preprocessing module includes a cropping unit and a reduction unit;

[0045] The cropping unit is used to reduce each frame of the first video data according to the size requirement of the video to be output, so as to obtain initial second video data corresponding to the first video data;

[0046] The reduction unit is used to perform cropping processing on each frame of the initial second video data according to the burst length of the DDR and the length of the initial second video data to obtain each frame of the second video data.

[0047] The present invention first reduces the size of the video sent by the video layer according to the size requirement of the output video, reduces the amount of video data processed by the cache while meeting the demand for video data, and improves the efficiency of the cache. Furthermore, the burst length of DDR and the length of the video data are used, where the length is the pixels of each row, to crop the video data to meet the cache requirement of DDR, thereby facilitating the use of DDR storage resources to store video data.

[0048] As a preferred example, the video writing module includes a request unit, a polling unit and a data writing unit;

[0049] The request unit is used to obtain the first field synchronization signal of each frame of the second video data, and generate a DMA write request corresponding to each frame of the second video data according to the rising edge of the first field synchronization signal;

[0050] The polling unit is used to obtain the polling value of the DMA status register, and determine the read and write status of the DDR and the target video layer of the DMA polling according to the polling value; wherein the read and write status includes the DDR write status and the DDR read status;

[0051] The data write unit is used to control the DMA write state machine to write each frame of the second video data into the FIFO through a valid display data selection signal when the DMA write request corresponding to the target video layer is received and the read / write state of the DDR is the DDR write state.

[0052] The present invention uses the state register of the DMA to obtain the read and write state of the DDR in real time and controls the DMA polling to poll the several video layers respectively, so that the data sent by each video layer can be written into the DDR in time in turn, ensuring the efficiency of the cache.

[0053] As a preferred example, the video fusion module includes a priority unit and a superposition unit;

[0054] The priority unit is used to obtain the value of the priority register of each of the video layers, and sort the priorities of the plurality of video layers according to the value;

[0055] The superimposing unit is used for superimposing a plurality of frames of the second video data according to the order to obtain superimposed video data, and outputting the superimposed video data.

[0056] The present invention superimposes the data of the plurality of video layers according to the setting of the priority register, so as to improve the effect of video superposition. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 : A flowchart of a multi-layer video data fusion method based on FPGA disclosed in an embodiment of the present invention;

[0058] Figure 2 : is a structural diagram of a multi-layer video data fusion system based on FPGA disclosed in an embodiment of the present invention;

[0059] Figure 3 : A flowchart of a multi-layer video data fusion method based on FPGA disclosed in another embodiment of the present invention;

[0060] Figure 4 : A structural diagram of an FPGA for executing a video data fusion method disclosed in another embodiment of the present invention;

[0061] Figure 5: A schematic diagram of a DDR read-write cache process disclosed in another embodiment of the present invention;

[0062] Figure 6 : A schematic diagram of DDR address allocation disclosed in another embodiment of the present invention;

[0063] Figure 7 : A schematic diagram of a video reading structure disclosed in another embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] Embodiment 1

[0066] This embodiment discloses a multi-layer video data fusion method based on FPGA. Figure 1 , mainly including steps 101 to 105, the steps are:

[0067] Step 101: obtaining a frame of first video data sent by each of a plurality of video layers respectively, and performing a preprocessing operation of cropping and reducing the first video data to obtain a plurality of frames of second video data.

[0068] In this embodiment, this step mainly includes: reducing each frame of the first video data according to the size requirement of the video to be output, and obtaining the initial second video data corresponding to the first video data; cropping each frame of the initial second video data according to the burst length of the DDR and the length of the initial second video data, and obtaining each frame of the second video data.

[0069] In this embodiment, the video sent by the video layer is first scaled down according to the size requirement of the output video, and the amount of video data processed by the cache is reduced while meeting the demand for video data, thereby improving the efficiency of the cache. Furthermore, the burst length of DDR and the length of the video data are utilized, where the length is the pixels of each row, and the video data is cropped to meet the cache requirement of DDR, thereby facilitating the use of DDR storage resources to store video data.

[0070] Step 102: Generate a DMA write request for each frame of the second video data, and write several frames of the second video data into the FIFO in sequence according to the DMA write request and an arbitration mechanism preset in the DMA.

[0071] In this embodiment, this step mainly includes: obtaining the first field synchronization signal of each frame of the second video data, and generating a DMA write request corresponding to each frame of the second video data according to the rising edge of the first field synchronization signal; obtaining the polling value of the DMA status register, and determining the read and write status of the DDR and the target video layer of the DMA polling according to the polling value; wherein the read and write status includes the DDR write status and the DDR read status; when the DMA write request corresponding to the target video layer is received and the read and write status of the DDR is the DDR write status, controlling the DMA write state machine to write each frame of the second video data into the FIFO through a valid display data selection signal.

[0072] In this embodiment, this step uses the status register of the DMA to obtain the read and write status of the DDR in real time and controls the DMA polling to poll the several video layers respectively, so that the data sent by each video layer can be written into the DDR in time in turn to ensure the efficiency of the cache.

[0073] Step 103: when the amount of data in the FIFO buffer reaches a threshold, the DDR is notified to read data, several frames of the second video data are respectively saved in the DDR, and a DDR address is allocated to each frame of the second video data.

[0074] In this embodiment, this step mainly includes: when the amount of data in the FIFO cache reaches a threshold, controlling the DMA to send a DDR write signal to the DDR, so that the DDR is controlled according to the DDR signal to read and cache each frame of the second video data, and a DDR cache address is allocated to each frame of the second video data.

[0075] In this embodiment, the data of each video layer in this step will be stored in a fixed address of DDR, and only the corresponding data address will be read during reading. Each layer data is assigned a fixed position in DDR, ensuring that there will be no data access conflict and improving cache efficiency.

[0076] Step 104: Generate a DMA read request for each frame of the second video data, and read each frame of the second video data from the DDR in sequence according to the DMA read request and the arbitration mechanism.

[0077] In this embodiment, this step mainly includes: generating a second field synchronization signal corresponding to the video to be output, and generating a DMA read request corresponding to each frame of the second video data according to the rising edge of the second field synchronization signal; obtaining the polling value of the DMA status register, and determining the read and write status of the DDR and the target video layer of the DMA polling according to the polling value; wherein the read and write status includes a DDR write status and a DDR read status; when the DMA read request corresponding to the target video layer is received and the read and write status of the DDR is a DDR read status, controlling the DMA read state machine to send a DDR read signal to read the DDR address of the target video layer, and reading each frame of the second video data according to the DDR address.

[0078] In this embodiment, this step takes into account that DDR cannot read video data of multiple layers at the same time, and sets an arbitration mechanism to poll each layer at all times to obtain data in all layers, thereby improving the accuracy of video overlay.

[0079] Step 105: fuse several frames of the second video data and output the fused video data.

[0080] In this embodiment, this step mainly includes: obtaining the value of the priority register of each of the video layers, and sorting the priorities of the several video layers according to the value; superimposing several frames of the second video data according to the sorting to obtain superimposed video data, and outputting the superimposed video data.

[0081] In this embodiment, this step overlays the data of the multiple video layers according to the setting of the priority register, so as to improve the effect of video overlay.

[0082] On the other hand, this embodiment also provides a multi-layer video data fusion system based on FPGA. For the specific structure of the fusion system, please refer to Figure 2 , including a pre-processing module 201, a video writing module 202, a video caching module 203, a video reading module 204 and a video fusion module 205.

[0083] The preprocessing module 201 is used to respectively obtain a frame of first video data sent by each of the plurality of video layers, and perform a preprocessing operation of cropping and reducing the first video data to obtain a plurality of frames of second video data.

[0084] The video writing module 202 is used to generate a DMA write request for each frame of the second video data, and write several frames of the second video data into the FIFO in sequence according to the DMA write request and an arbitration mechanism preset in the DMA.

[0085] The video buffer module 203 is used to notify the DDR to read data when the amount of data in the FIFO buffer reaches a threshold, save several frames of the second video data into the DDR respectively, and allocate a DDR address for each frame of the second video data.

[0086] The video reading module 204 is used to generate a DMA read request for each frame of the second video data, and read each frame of the second video data from the DDR in sequence according to the DMA read request and the arbitration mechanism.

[0087] The video fusion module 205 is used to fuse several frames of the second video data and output the fused video data.

[0088] As a preferred example, the pre-processing module 201 includes a cropping unit and a reduction unit.

[0089] The cropping unit is used to reduce each frame of the first video data according to the size requirement of the video to be output, so as to obtain initial second video data corresponding to the first video data.

[0090] The reduction unit is used to perform cropping processing on each frame of the initial second video data according to the burst length of the DDR and the length of the initial second video data to obtain each frame of the second video data.

[0091] Preferably, the video writing module 202 includes a request unit, a polling unit and a data writing unit.

[0092] The request unit is used to obtain the first field synchronization signal of each frame of the second video data, and generate a DMA write request corresponding to each frame of the second video data according to the rising edge of the first field synchronization signal.

[0093] The polling unit is used to obtain the polling value of the DMA status register, and determine the read and write status of the DDR and the target video layer of the DMA polling according to the polling value; wherein the read and write status includes the DDR write status and the DDR read status.

[0094] The data write unit is used to control the DMA write state machine to write each frame of the second video data into the FIFO through a valid display data selection signal when the DMA write request corresponding to the target video layer is received and the read / write state of the DDR is the DDR write state.

[0095] As a preferred example, the video fusion module 205 includes a priority unit and an overlay unit.

[0096] The priority unit is used to obtain the value of the priority register of each of the video layers, and sort the priorities of the plurality of video layers according to the value.

[0097] The superimposing unit is used for superimposing a plurality of frames of the second video data according to the order to obtain superimposed video data, and outputting the superimposed video data.

[0098] Embodiment 2

[0099] This embodiment provides a multi-layer video data fusion method based on FPGA, wherein the specific implementation process of the fusion method is shown in FIG. Figure 3 , mainly including steps 301 to 304, the steps mainly include:

[0100] Step 301: receiving a frame of first video data sent by each of a plurality of video layers, and performing cropping and reduction preprocessing on the first video data to obtain a plurality of frames of second video data.

[0101] In this embodiment, this step mainly includes: reducing each frame of the first video data according to the size requirement of the video to be output, and cropping each frame of the first video data according to the burst length of the DDR to obtain each frame of the second video data.

[0102] Specifically, in this embodiment, multiple video preprocessing modules are set to receive the first video data sent by each video layer in the multiple video layers. Figure 4 , multiple video data signal sources are input from the outside to the video data preprocessing module. The video source can support up to 4K30. Each video source is a video layer. The video data preprocessing module can choose whether to crop and reduce the video layer. Among them, one video data preprocessing module corresponds to one video layer. For example, if there are 8 channels of video layer data, there will be 8 video data preprocessing modules. The video data preprocessing module is mainly used to crop and reduce the first video data sent by the video layer.

[0103] Specifically, the judgment criterion for the video data preprocessing module to perform the reduction processing is: when the length and width of the output video, that is, the video data that merges multiple video layers, is smaller than the length and width of the first video data sent by the video source, the reduction processing is performed; otherwise, the reduction processing is not performed. In this embodiment, if the length and width of the output video need to be 800*600, but the length and width of the first video sent by the video source is 1920*1080, the video source needs to be reduced to 800*600, wherein the video data preprocessing module can use the FPGA bilinear scaling algorithm to perform the reduction processing.

[0104] The judgment criterion for the video data preprocessing module to perform cropping processing is to perform a cropping operation when the length of the first video data sent by the video source (pixels in each row) is greater than the burst length of the DDR. In this embodiment, if the burst length of the DDR is set to 2048, the length of the first video sent by the video source needs to be less than 2048. Optionally, when the length and width of the first video sent by a video source is 3840*2160, and no reduction operation is performed, at this time, layer 1, i.e., video source 1, and layer 2, i.e., video source 2, need to be the same input source, and the length 3840 of the video source of the video preprocessing modules of layers 1 and 2 will be divided by 2, that is, the preprocessing modules of layers 1 and 2 each store half of the data into the DDR.

[0105] Step 302: Initiate a DMA write request for each frame of the second video data, so that each frame of the second video data is sequentially written into the DDR according to the DMA write request and a preset DMA read-write arbitration mechanism.

[0106] In this embodiment, the step is: generate a DMA write request for each frame of the second video data, and write several frames of the second video data into the FIFO in sequence according to the DMA write request and the arbitration mechanism preset in the DMA, when the amount of data cached in the FIFO reaches a threshold, notify the DDR to read the data, save several frames of the second video data into the DDR respectively, and assign a DDR address to each frame of the second video data.

[0107] Specifically, in this embodiment, a DMA read / write arbitration module is set to formulate the arbitration mechanism, and a DDR read / write controller is set to be electrically connected to the DMA read / write arbitration mechanism so as to cache each frame of the second video. Figure 4 , the video data preprocessing module generates a DMA write request, wherein the DMA write request includes a DMA write request corresponding to a frame of second video data. Specifically, when the video data preprocessing module of layer 1 has the video data sent and processed by layer 1, the VS edge will be continuously pulled high for a frame time when each frame of the video arrives, thereby initiating a DMA write request corresponding to the frame of second video data according to the rising edge of VS.

[0108] The DMA read-write arbitration module receives a DMA write request for each frame of the second video data, and then the DMA read-write arbitration module obtains a polling value of a DMA status register, determines the DDR read-write status and the currently polled layer number according to the polling value, and then determines whether to save the second video data according to the DDR read-write status and the currently polled layer number.

[0109] Specifically, in this embodiment, refer to Figure 5, set the polling value of the DMA status register to i, after the reset is completed, i will continue to increase by 1, when i is an odd number, it indicates the DDR write state, when i is an even number, it indicates the DDR read state, the DMA will sequentially poll in the multiple layers connected to the video data preprocessing module, in one embodiment, when i=1, it means that the DMA polls the write state of layer 1, at this time, if the DMA write request corresponding to layer 1 is received, the DMA arbitration mechanism will wait for the write to be completed and then increase i by 1, the DMA write state machine starts working, and then sends a DDR write signal, and stores a frame of the second video data sent by the layer 1 into the DDR, at this time, after a frame of video data is stored, i+1=2, specifically, the

[0110] Furthermore, when a frame of the second video data sent by the layer 1 is stored in the DDR, the DMA read-write arbitration module writes a pixel data into the fifo according to each de (valid display data selection signal) signal. When the fifo stores a fixed threshold, that is, when the burst amount of the DDR is reached, the DDR cache is performed. Since the pixel is 16 bits, the DDR cache bit width is set to 256 bits, and each 256 bits of data corresponds to a DDR address. Until a frame of data is written, a DDR address corresponding to a frame of the second video sent by each layer is generated. The allocation of the DDR address can refer to Figure 6 Each layer has a data address, and the DMA read-write arbitration module saves the video data sent by each layer to the data address according to the data address corresponding to each layer.

[0111] Step 303: Initiate a DMA read request for each frame of the second video data, so that each frame of the second video data is sequentially read from the DDR according to the DMA read request and the DMA read-write arbitration mechanism.

[0112] In this embodiment, the steps are as follows: Figure 4 , setting a DMA read-write arbitration module to formulate the arbitration mechanism, and setting a DDR read-write controller to be electrically connected to the DMA read-write arbitration mechanism to read each frame of the second video, and setting a video data superposition module to be connected to the DMA read-write arbitration module to obtain each frame of the second video data read, optionally, referring to Figure 7 When the video data superposition module outputs video, a VS signal will be generated, and the VS will be pulled high. The rising edge of VS is the sign of initiating a DMA read request. Since DDR cannot read data of multiple layers at the same time, the data of a specific layer is read through the read arbitration in the DMA read-write arbitration module before data superposition. The video data superposition module reads one layer of data per frame to the video data superposition module. After several frames of time, the video data of several layers can be read.

[0113] Specifically, refer to Figure 5 When the DMA read-write arbitration module performs the read arbitration, after the write arbitration of layer 1 is performed with reference to step 302, the i+1=2, and the layer 1 enters the read state. If the read signal of layer 1 is received at this time, the DMA read state machine starts to work, and sends a DDR read signal to read the address data of layer 1 in the DDR. When reading, it will read according to the corresponding layer start address. After reading a frame of video data, i is re-added by 1 to enter the write state of layer 2, and the cycle continues to cycle to realize the writing and reading of video data of all layers in turn. Optionally, the polling clock used in the arbitration mechanism is set to 300m, which is consistent with the clock of video data preprocessing. Therefore, even when the DMA status register value is an odd number, but the received signal is a read signal, the time required to wait for the DMA status register to re-poll to the read signal of the layer is reduced, thereby improving the efficiency of the cache.

[0114] Step 304: fuse several frames of the second video data and output the fused video data.

[0115] In this embodiment, the step mainly includes: obtaining the value of the priority register of each of the video layers, and sorting the priorities of the several video layers according to the value; superimposing several frames of the second video data according to the sorting to obtain superimposed video data, and outputting the superimposed video data.

[0116] In this embodiment, refer to Figure 4 The video data superposition module superimposes multiple frames of video data of multiple layers according to the priority register. In one embodiment, there are 8 layers of data that need to be superimposed, among which the priority register of layer 1 is set to 3'd7, that of layer 2 is set to 3'd4, that of layer 3 is set to 4'd6, and so on. The larger the value of the priority register, the closer it is to the front. The value of layer 1 is 7, the largest, so it is at the front, followed by layer 3, and layer 2 is third, and so on.

[0117] Preferably, refer to Figure 4 and Figure 7 After the video data is read from the DDR by the video data superposition module, the video data post-processing module can also perform post-processing of the video, such as amplification, filtering, sharpening and other image processing operations. For details, refer to Figure 7 In the video data post-processing module, multiple sub-modules are set, and one sub-module performs post-processing on a corresponding read video data, and finally multiple layers of data can be superimposed and outputted. Further, refer to Figure 4 , a video data output display module is set to be connected to the video data superposition module, and the video data output display module can be used to set an HDMI output interface to display the superimposed image.

[0118] The embodiment discloses a multi-layer video data fusion method based on FPGA, which makes full use of the storage resources of DDR, stores the data of each layer in a fixed address of DDR, and only reads the corresponding data address when reading. Each layer data is in a fixed position in DDR, ensuring that there will be no data access conflict. In addition, there will be no data reading delay problem under the 300m processing clock. At the same time, in the arbitration mechanism of DMA, it is always polling whether the layer needs to be cached. After the layer data preprocessing is completed, the DDR cache can be performed, ensuring the efficient storage of the data stream.

[0119] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-layer video data fusion method based on FPGA, characterized in that: include: Respectively obtaining a frame of first video data sent by each of the plurality of video layers, and performing a preprocessing operation of cropping and reducing the first video data to obtain a plurality of frames of second video data; Generate a DMA write request for each frame of the second video data, and write several frames of the second video data into the FIFO in sequence according to the DMA write request and the arbitration mechanism preset in the DMA; wherein, obtain the first field synchronization signal of each frame of the second video data, and generate the DMA write request corresponding to each frame of the second video data according to the rising edge of the first field synchronization signal; obtain the polling value of the DMA status register, and determine the read and write status of the DDR and the target video layer of the DMA polling according to the polling value; wherein the read and write status includes the DDR write status and the DDR read status; when the DMA write request corresponding to the target video layer is received and the read and write status of the DDR is the DDR write status, control the DMA write state machine to write each frame of the second video data into the FIFO through the valid display data selection signal; When the amount of data in the FIFO buffer reaches a threshold, the DDR is notified to read the data, several frames of the second video data are respectively saved in the DDR, and a DDR address is allocated to each frame of the second video data; Generate a DMA read request for each frame of the second video data, and read each frame of the second video data from the DDR in sequence according to the DMA read request and the arbitration mechanism; wherein, generate a second field synchronization signal corresponding to the video to be output, and generate a DMA read request corresponding to each frame of the second video data according to the rising edge of the second field synchronization signal; obtain a polling value of a DMA status register, and determine the read / write status of the DDR and the target video layer of the DMA polling according to the polling value; wherein the read / write status includes a DDR write status and a DDR read status; when the DMA read request corresponding to the target video layer is received and the read / write status of the DDR is a DDR read status, control the DMA read state machine to send a DDR read signal to read the DDR address of the target video layer, and read each frame of the second video data according to the DDR address; A plurality of frames of the second video data are fused and the fused video data is output.

2. The multi-layer video data fusion method based on FPGA according to claim 1, characterized in that: The preprocessing operation of cropping and reducing the first video data to obtain a plurality of frames of second video data includes: Reducing each frame of the first video data according to the size requirement of the video to be output, to obtain initial second video data corresponding to the first video data; Each frame of the initial second video data is cropped according to the burst length of the DDR and the length of the initial second video data to obtain each frame of the second video data.

3. The multi-layer video data fusion method based on FPGA according to claim 1, characterized in that: The step of storing a plurality of frames of the second video data in the DDR respectively and allocating a DDR address to each frame of the second video data comprises: When the amount of data in the FIFO buffer reaches a threshold, the DMA is controlled to send a DDR write signal to the DDR, so that the DDR is controlled according to the DDR write signal to read and buffer each frame of the second video data, and a DDR buffer address is allocated to each frame of the second video data.

4. The multi-layer video data fusion method based on FPGA according to claim 1, characterized in that: The step of fusing a plurality of frames of the second video data and outputting the fused video data comprises: Obtaining a value of a priority register of each of the video layers, and sorting the plurality of video layers by priority according to the value; A plurality of frames of the second video data are superimposed according to the order to obtain superimposed video data, and the superimposed video data is output.

5. A multi-layer video data fusion system based on FPGA, characterized in that: It includes a pre-processing module, a video writing module, a video caching module, a video reading module and a video fusion module; The preprocessing module is used to respectively obtain a frame of first video data sent by each of the plurality of video layers, and perform a preprocessing operation of cropping and reducing the first video data to obtain a plurality of frames of second video data; The video write module is used to generate a DMA write request for each frame of the second video data, and write several frames of the second video data into the FIFO in sequence according to the DMA write request and the arbitration mechanism preset in the DMA; wherein the video write module includes a request unit, a polling unit and a data write unit; the request unit is used to obtain the first field synchronization signal of each frame of the second video data, and generate a DMA write request corresponding to each frame of the second video data according to the rising edge of the first field synchronization signal; the polling unit is used to obtain the polling value of the DMA status register, and determine the read and write status of the DDR and the target video layer of the DMA polling according to the polling value; wherein the read and write status includes the DDR write status and the DDR read status; the data write unit is used to control the DMA write state machine to write each frame of the second video data into the FIFO through a valid display data selection signal when the DMA write request corresponding to the target video layer is received and the read and write status of the DDR is the DDR write status; The video buffer module is used for notifying the DDR to read data when the amount of data in the FIFO buffer reaches a threshold, saving several frames of the second video data into the DDR respectively, and assigning a DDR address to each frame of the second video data; The video reading module is used to generate a DMA read request for each frame of the second video data, and read each frame of the second video data from the DDR in sequence according to the DMA read request and the arbitration mechanism; wherein, a second field synchronization signal corresponding to the video to be output is generated, and a DMA read request corresponding to each frame of the second video data is generated according to the rising edge of the second field synchronization signal; the polling value of the DMA status register is obtained, and the read and write status of the DDR and the target video layer of the DMA polling are determined according to the polling value; wherein the read and write status includes a DDR write status and a DDR read status; when the DMA read request corresponding to the target video layer is received and the read and write status of the DDR is a DDR read status, the DMA read state machine is controlled to send a DDR read signal to read the DDR address of the target video layer, and each frame of the second video data is read according to the DDR address; The video fusion module is used to fuse several frames of the second video data and output the fused video data.

6. The FPGA-based multi-layer video data fusion system according to claim 5, characterized in that: The pre-processing module includes a cropping unit and a reduction unit; The cropping unit is used to reduce each frame of the first video data according to the size requirement of the video to be output, so as to obtain initial second video data corresponding to the first video data; The reduction unit is used to perform cropping processing on each frame of the initial second video data according to the burst length of the DDR and the length of the initial second video data to obtain each frame of the second video data.

7. The FPGA-based multi-layer video data fusion system according to claim 5, characterized in that: The video fusion module includes a priority unit and a superposition unit; The priority unit is used to obtain the value of the priority register of each of the video layers, and sort the priorities of the plurality of video layers according to the value; The superimposing unit is used for superimposing a plurality of frames of the second video data according to the order to obtain superimposed video data, and outputting the superimposed video data.

Images