Real-time video processing system based on ampere field programmable gate array (FPGA)

Abstract

The invention discloses a real-time video processing system based on an ampere-path FPGA. The real-time video processing system comprises an SD card, an ampere-path FPGA board card and an HDMI displayer. The SD card is used for storing original video data in a Bayer format; the A-path FPGA board card comprises an FPGA end and an ARM end, the FPGA end comprises an SD card reading control module, an image cache FIFO module, a Bayer-to-RGB module, a seven-in-one image processing module, an SDRAM read-write control module and an HDMI display control module, the first four modules are connected in sequence, and the SDRAM read-write control module is connected with the Bayer-to-RGB module, the seven-in-one image processing module and the HDMI display control module; wherein the ARM end comprises a Cortex-M0 processor core, an AHBLiite-Interconnect data bus and all peripheral interfaces, and all the modules are connected in sequence; the ARM end comprises a Cortex-M0 processor core, an AHBLiite-Interconnect data bus and all the peripheral interfaces; the ampere-path FPGA board card is used for completing all video processing functions; and the HDMI display outputs the processed video. The method is good in video processing effect, low in resource consumption, high in flexibility, high in stability and low in hardware cost, and the hardware is 100% localized.

Description

Technical Field

[0001] The present invention belongs to the technical field of video processing, and in particular relates to a real-time video processing system based on Anlu FPGA. Background Art

[0002] Video processing systems, a key technology in the information age, are widely used in fields such as security surveillance, education and healthcare, and artificial intelligence. Technological advancements and evolving market demands require systems to not only process increasingly high-quality images but also implement a variety of complex functions, such as video encoding and decoding, image quality enhancement, and video content analysis. This also places higher demands on system power consumption, size, and cost. Video processing systems based on ARM or x86 CPUs often suffer from slow processing speeds and poor real-time performance, limiting their practical application. For example, in low-light video surveillance, video images must be enhanced in real time at a rate of at least 30 frames per second and the processed video files must be streamed to a server, placing high demands on the real-time performance of the video processing system. While video processing systems based on ARM or x86 CPUs can handle highly complex processing tasks, their low processor sampling rates and serialized instruction processing structures make them difficult to meet the requirements of high real-time and high-resolution video processing.

[0003] By deploying the ARM Cortex-M0 soft core on the Anlu FPGA chip and combining the ARM and FPGA architectures, more efficient video processing algorithm deployment can be achieved. By accelerating algorithm processing on the FPGA and having the ARM core execute algorithm flow control, video processing can be effectively implemented. However, existing FPGA accelerator design methods are limited by on-chip FPGA resources, especially RAM resources, which restricts computational load and storage capacity. Executing video processing algorithms urgently requires deeply customized, concise, and efficient computing units to optimize execution efficiency. Furthermore, FPGA-based video processing systems mostly use FPGA chips from foreign companies such as Xilinx and Altera, which are relatively expensive. Therefore, domestically produced alternatives to FPGA video processing systems are imperative.

[0004] In summary, the existing problems are:

[0005] 1. The computing performance of video processing systems based on ARM or x86 CPUs has bottlenecks, making it difficult to meet the requirements of high real-time and high-resolution video processing.

[0006] 2. FPGA’s computing and storage resources are limited, which to some extent restricts its application in the field of video processing.

[0007] 3. The existing FPGA video processing system has a low degree of localization and its application in sensitive fields is limited. Summary of the Invention

[0008] The purpose of this invention is to provide a real-time video processing system based on an ARM FPGA that implements multiple video processing algorithms, thereby achieving real-time video processing and display. By combining the respective advantages of ARM and FPGA, the system better meets the requirements of video processing systems, providing improved display quality and processing speed, and resolving at least one of the technical problems mentioned in the background art.

[0009] To achieve the above object, the technical solution adopted by the present invention is:

[0010] A real-time video processing system based on Anlu FPGA includes an SD card, an Anlu FPGA board and an HDMI display; the SD card is used to store raw video data in Bayer format for reading by the Anlu FPGA board; the Anlu FPGA board includes an FPGA end and an ARM end, wherein the FPGA end includes an SD card reading control module, an image cache FIFO module, a Bayer to RGB module, a 7-in-1 image processing module, an SDRAM read-write control module and an HDMI display control module, and the connection relationship between the modules is as follows: the SD card reading control module is connected to the image cache FIFO module, the image cache FIFO module is connected to the Bayer to RGB module, and the Bayer to RGB module is connected Module B is connected to the 7-in-1 image processing module, and the SDRAM read-write control module is connected to the Bayer to RGB module, the 7-in-1 image processing module, and the HDMI display control module respectively; the ARM side includes the Cortex-M0 processor core, the AHBLite-Interconnect data bus, and various peripheral interfaces. The connection relationship between the modules is: the Cortex-M0 processor core is connected to the AHBLite-Interconnect data bus, and the various peripheral interfaces are connected to the AHBLite-Interconnect data bus; the Anlu FPGA board completes all video processing functions; and the HDMI display outputs the processed video.

[0011] Further, the following steps are included:

[0012] Step 1: The SDRAM read / write control module in the FPGA sends instructions to initialize the SDRAM in preparation for caching the processed video. The ARM side executes the initialization code and configures the key peripheral address mapping through the AHBLite-Interconnect data bus to prepare for executing the user key interrupt.

[0013] Step 2: The FPGA communicates with the SD card via the four-wire SDIO protocol, completing the SD card configuration and initialization. The FPGA then reads the original video data in Bayer format from the external SD card. Using the four-wire SDIO protocol in this step increases the speed of reading the original video file and reduces data access latency.

[0014] Step 3: The raw video data is sent to the image buffer FIFO module for caching for further processing. In this step, since the convolution or pixel interpolation algorithm of the two-dimensional image requires pixel data in the surrounding 3×3 range, at least two rows of pixel data must first be cached for the algorithm to call. After determining the width and height of the entire image, the image buffer FIFO module inputs the current pixel and directly outputs other pixels in the 3×3 range. At the same time, it outputs the current row, column, and frame counts for the subsequent 7-in-1 image processing module to call;

[0015] Step 4: Read the pixel data of the surrounding 3×3 range cached in step 3, and then perform the Bayer to RGB interpolation algorithm. The specific steps of the algorithm are:

[0016] (1) When R and B pixels are distributed above, below, left, and right of the interpolation center pixel, R and B are respectively taken as the average value of the two neighboring pixels;

[0017] (2) When the R and B pixels are distributed diagonally around the interpolation center pixel, the average of the four B or R pixels in the neighborhood is taken as the B value of the middle pixel.

[0018] (3) Since the human eye is most sensitive to green light and less sensitive to purple and red light, in order to achieve better image quality, the green component G needs to be specially processed. The specific formula is as follows:

[0019]

[0020] Where G(R) represents the green component corresponding to the red center pixel R, G(B) represents the green component corresponding to the blue center pixel B, G1-G4 represent the four green pixel values ​​within a 3×3 region of the center pixel, R1-R4 represent the four red pixel values ​​2 grids above, below, and left and right of the red center pixel, and B1-B4 represent the four blue pixel values ​​2 grids above, below, and left and right of the blue center pixel. The linear interpolation compensation algorithm used in this step focuses on the speed-quality trade-off, which is conducive to hardware implementation.

[0021] Step 5: The ARM side detects the user's key through the key interrupt. If the key is pressed, the value of the variable func_sel is increased by 1 (if it exceeds 7, it returns to 0). The value of this variable switches cyclically from 0 to 7 and is used to select different video processing algorithms in step 6. The specific correspondence between the value of func_sel and the video processing algorithm executed by the module is as follows:

[0022] (1) When func_sel=3'd0, the original video data stored in the SD card is directly displayed without any other video processing after steps 1 to 4;

[0023] (2) When func__sel = 3'd1, the RGB2Gray algorithm is executed on the original video based on (1);

[0024] (3) When func_sel = 3'd2, the image binarization algorithm is performed on the original video based on (1);

[0025] (4) When func_sel=3'd3, perform gamma transform algorithm on the original video based on (1);

[0026] (5) When func_sel=3'd4, the Gaussian filtering algorithm is performed on the original video based on (1);

[0027] (6) When func_sel=3'd5, the Sobel operator edge extraction algorithm is performed on the original video based on (1);

[0028] (7) When func_sel = 3'd6, the Laplace sharpening algorithm is performed on the original video based on (1);

[0029] (8) When func_sel = 3'd7, the corrosion and expansion algorithm is performed on the original video based on (1);

[0030] Step 6: Based on the RGB image interpolated in step 4, the video processing algorithms are executed, including: RGB2Gray, binarization, gamma transform, Gaussian filtering, Sobel operator edge extraction, Laplace sharpening, erosion, and dilation. After the algorithm is executed, the FPGA sends the processed video data to the built-in buffer of the 7-in-1 image processing module for subsequent video caching and display. Only the original video can be displayed or one of the seven algorithms can be executed at the same time. Algorithm switching is achieved through the function selection port func_sel. The data output of multiple video processing modules is controlled by a hardware multiplexer to ensure that the calculation results do not conflict.

[0031] Step 7: The SDRAM read / write control module reads the target data after the video processing algorithm is executed from the buffer built into the 7-in-1 image processing module, generates the SDRAM drive timing using a finite state machine, and writes the data into the SDRAM according to the SDRAM timing requirements as a frame buffer for video display;

[0032] Step 8: The HDMI display control module reads the processed video data from the SDRAM. The TMDS encoding unit in this module encodes the original RGB video data into three channels of differential data for output via the HDMI interface. The specific method is determined by the TMDS protocol: 8b / 10b encoding is used to convert the original 8-bit signal data per channel into 10 bits. The video data is then converted to a serial signal by the parallel-to-serial conversion unit. Finally, the output timing control unit outputs the data. The output video format is 1920×1080 resolution, 40 frames per second. Finally, the HDMI display control module outputs the three R, G, and B channels of video data and the serial clock channel to the HDMI display, enabling real-time display of the processed video.

[0033] Furthermore, in step 1, the SDRAM read / write control module in the FPGA sends instructions to initialize the SDRAM in preparation for post-processing video caching. Initialization requires instantiating a PLL to generate the SDRAM clock signal at a frequency of 200MHz. Furthermore, the SDRAM read / write timing parameters must be configured, and SDRAM initialization instructions must be sent to complete the initialization process. Initialization performed on the ARM side includes executing the Cortex-M0 core startup code, defining the IRQ0 interrupt function, executing the reset handler function, and calling the main function. The main function primarily implements interrupt initialization and port configuration, and is used to configure the key peripheral address mapping via the AHBLite-Interconnect data bus in preparation for executing user key interrupts. This ensures that whenever a key is pressed, the corresponding interrupt handler function is immediately executed, thereby completing the switch between different image algorithms.

[0034] Furthermore, in step 2, the FPGA communicates with the SD card via the four-wire SDIO protocol, completing the SD card configuration and initialization. The FPGA then reads the raw video data in Bayer format from the external SD card. Initialization prepares the card for operation, ensuring proper communication and data transfer between it and the host device. This process must be performed after the SD card is powered on. During this initialization phase, the host device sends a series of commands to the SD card, configuring the card's voltage range, capacity, speed category, and other operating modes, ensuring the host device understands how to interact with it.

[0035] Furthermore, in step 3, the original video data is sent to the image cache FIFO module for caching for further processing. In this step, since the convolution or pixel interpolation algorithm of the two-dimensional image requires pixel data in the surrounding 3×3 range, at least two rows of pixel data must first be cached for the algorithm to call. The specific caching method is: the pixel cache of the same row can be simply implemented using three cascaded shift registers, while the pixel cache of different rows is implemented using FIFO. The FIFO can store the 3×3 matrix data of a frame of image of the current clock in real time, and as the clock jumps, the sliding window operation in the two-dimensional convolution is achieved. There are two FIFO units with a bit width of 16 bits and a depth of 2048 inside the image cache FIFO module, each of which can temporarily store one row of image data. After determining the width and height of the entire image, the module inputs the current pixel and can directly output other pixels within the 3×3 range. At the same time, it outputs the current row, column, and frame counts for the subsequent 7-in-1 image processing module to call;

[0036] Furthermore, in step 4, the pixel data of the surrounding 3×3 range buffered in step 3 is read, and then the Bayer to RGB interpolation algorithm is executed. The specific steps of the algorithm are:

[0037] (1) When R and B pixels are distributed above, below, left, and right of the interpolation center pixel, R and B are respectively taken as the average value of the two neighboring pixels;

[0038] (2) When the R and B pixels are distributed diagonally across from the interpolation center pixel, the average of the four B or R pixels in the area is taken as the B value of the middle pixel.

[0039] (3) Since the human eye is most sensitive to green light and less sensitive to purple and red light, in order to achieve better image quality, the green component G needs to be specially processed. The specific formula is as follows:

[0040]

[0041] Where G(R) represents the green component corresponding to the red center pixel R, G(B) represents the green component corresponding to the blue center pixel B, G1-G4 represent the four green pixel values ​​within a 3×3 region around the center pixel, R1-R4 represent the four red pixel values ​​2 grids above, below, and left of the red center pixel, and B1-B4 represent the four blue pixel values ​​2 grids above, below, and left of the blue center pixel. The linear interpolation compensation algorithm used in this step focuses on the speed-quality trade-off, which is conducive to hardware implementation.

[0042] Furthermore, in step 5, the ARM side detects the user's key through the key interrupt. If it is pressed, the value of the variable func_sel is +1 (if it exceeds 7, it returns to 0). The value of this variable switches cyclically from 0 to 7 and is used to select different video processing algorithms in step 6. Among them, the key is assigned to the IRQ0 interrupt of Cortex-M0. The level change of the key will trigger the corresponding interrupt processing of Cortex-M0. The key debounce operation is then performed in the interrupt service function, that is, the key status is judged again after a delay of 10ms. After the debounce is completed, it is determined whether it is a valid key operation. This can avoid misoperation and improve the reliability of key detection.

[0043] Furthermore, in step 6, based on the RGB image interpolated in step 4, one of the seven video processing algorithms, namely RGB2Gray, binarization, gamma transform, Gaussian filtering, Sobel operator edge extraction, Laplace sharpening, erosion and dilation, is executed according to the value of the variable func_sel. The processed video data is then sent to the built-in buffer of the 7-in-1 image processing module for subsequent video caching and display.

[0044] RGB2Gray is used to convert a color image to grayscale, converting the RGB data into a single-channel grayscale value. Its core is to implement the following transformation formula: Gray1 = 0.299*image_in_r + 0.587*image_in_g + 0.114*image_in_b. Here, Gray1 represents the converted grayscale value, and image_in_r, image_in_g, and image_in_b represent the red, green, and blue components of the input color image, respectively. The bit depth is 8 bits, and the value range is 0-255.

[0045] Binarization is used to convert color or grayscale images into black and white. The core of the algorithm is to select an appropriate threshold value and then determine whether each pixel should be set to black or white based on the threshold value. If the pixel value is greater than the threshold value, it is set to white (usually represented by 1); otherwise, it is set to black (usually represented by 0).

[0046] Among them, gamma transform is used to adjust the brightness of the image and improve the visual effect of the image. The core of its algorithm is to perform logarithmic operation on the original pixel value. Its general formula is: g = c * r γ, where r and g are the pixel values ​​before and after the logarithmic operation, respectively, c is a constant coefficient, and γ is the gamma transformation factor. Its value determines the grayscale mapping between the input and output images, that is, whether to enhance low grayscale (shadow areas) or high grayscale (highlight areas). When γ > 1, the contrast of high-grayscale areas of the image is enhanced; when γ < 1, the contrast of low-grayscale areas of the image is enhanced; when γ = 1, the grayscale transformation is linear, meaning that the original image is not altered.

[0047] Among them, Gaussian filtering is used to remove noise in the image, and its formula is: Where x and y represent the two-dimensional image, and ρ represents the variance. Gaussian filtering requires selecting a Gaussian kernel (determining an appropriate ρ) and then performing convolution. This involves applying a two-dimensional Gaussian distribution formula to the pixels within a 3×3 area. These nine values ​​are then summed to obtain the Gaussian filter value at the center.

[0048] Among them, the Sobel operator edge extraction is used to extract image edges. It is an edge and detection algorithm based on first-order differential operation. Its core is the Sobel operator, which includes two 3×3 matrices, one is horizontal (G x ), one is vertical (G y ), and use them to perform two-dimensional convolution operations with the 3×3 matrix of the image to obtain the approximate values ​​of the horizontal and vertical brightness differences. This step is expressed as: Where A is the 3×3 matrix of the image. After obtaining the vertical operator and the horizontal operator, the gradient vector G of the image is obtained by processing according to the results of the two directions. The calculation formula is Many system designers use the modulo method instead for the convenience of calculation. However, in order to pursue the accuracy of the edge detection algorithm results, this system still uses the original formula to calculate the horizontal operator and the vertical operator.

[0049] Among them, Laplace sharpening is used to enhance the edges and details of the image. It uses the Laplace second-order differential operator to obtain the degree of transition of image colors, such as the transition between white and black. When the grayscale of the center pixel of the neighborhood is lower than the average grayscale of other pixels in its area, the grayscale of this center pixel should be further reduced; when the grayscale of the center pixel of the neighborhood is higher than the average grayscale of other pixels in its neighborhood, the grayscale of this center pixel should be further increased to achieve image sharpening. The 3×3 Laplace sharpening operator used in this system is The center pixel has a weight of 5, while the surrounding four pixels have a weight of -1. This template can highlight edges and details in the image, enhancing image sharpness. This convolution kernel covers the eight pixels surrounding the image, including horizontal, vertical, and diagonal directions. By applying this convolution kernel, edges and texture features in the image are emphasized, achieving image enhancement and sharpening.

[0050] Erosion and dilation enlarge dark and bright areas, respectively, and can be used to separate foreground and background. The calculation formula for the erosion algorithm is typically: P = P1 & P2 & P3 & P4 & P5 & P6 & P7 & P8 & P9, while the calculation formula for the dilation algorithm is: P = P1 | P2 | P3 | P4 | P5 | P6 | P7 | P8 | P9. In the formula, P represents the pixel value after the operation, and P1 to P9 represent the values ​​of the nine pixels surrounding it in a 3×3 window. All variables are either 0 or 1, corresponding to black and white pixels, respectively. In the erosion algorithm, if there is even one pixel that is not 1 in the 3×3 window, it is eroded to 0 after the AND operation, which can be implemented using an AND operation. In the dilation algorithm, if there is even one 1, it is expanded to nine valid pixels, which is implemented using an OR operation. However, this calculation method can cause excessive erosion and dilation, which can lead to anomalies. A threshold parameter is introduced to ensure that the result is 0 if there is even one 0 in the window, and 1 only if all pixels are 1. This not only allows you to adjust the strength of the erosion and dilation algorithms, but also allows you to combine the two operations into one by comparing them against a single threshold. Therefore, we introduce a threshold for a number within the window and compare the accumulated result within the window with the threshold.

[0051] Furthermore, in step 7, the SDRAM read-write control module reads the target data after the video processing algorithm is executed from the buffer built into the 7-in-1 image processing module, and uses a finite state machine to generate the SDRAM drive timing, and writes the data into the SDRAM according to the SDRAM timing requirements. In this step, the operation of the SDRAM uses a dual-port FIFO to indirectly access the data in the SDRAM to prevent read-write conflicts, to a certain extent circumventing the complex read-write timing of the SDRAM, and achieving the purpose of simplifying the operation. After the SDRAM is initialized and enters the normal working state, its state transition diagram is shown in the attached Figure 4 The main operations are "refresh," "read," and "write." Each operation involves several steps and varying delays, and cannot be interrupted. Furthermore, SDRAM read and write control uses a "ping-pong" model, which divides the SDRAM address space into two equal-sized areas and alternates read and write operations between them to avoid conflicts when reading and writing the same address, thus reducing SDRAM access latency.

[0052] Furthermore, in step 8, the corresponding parameters of the HDMI display control module are set as follows: the pixel clock PXLCLK_I is 100MHz, and the serial clock PXLCLK_5X_I is five times the pixel clock, or 500MHz. In addition, the horizontal synchronization signal VGA_HS, the vertical synchronization signal VGA_VS, the data enable signal VGA_DE, and the data signal VGA_RGB are also required. The timing of these signals must strictly follow the VESA standard when transmitting HDMI signals to ensure that the display does not display artifacts or black screens.

[0053] In another aspect, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0054] On the other hand, the present invention further discloses a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0055] As can be seen from the above technical solution, the present invention aims to overcome the shortcomings of existing ARM or x86 CPU-based video processing systems, such as low efficiency and poor real-time performance, as well as the limited on-chip resources of FPGAs. This system, based on Anlu FPGAs, implements a variety of video processing algorithms through customized, concise and efficient hardware computing modules, thereby achieving real-time video processing and display. By combining the respective advantages of ARM and FPGAs, the system better meets the requirements of video processing systems, providing improved display quality and processing speed. Compared with existing technologies, the system has the following significant technical effects:

[0056] (1) It fully utilizes the advantages of FPGA such as reconfigurability, strong parallel performance, and fast data processing speed, and can realize real-time video processing. The built-in algorithm can be switched with one click, and it has good effects in various scenarios.

[0057] (2) FPGA+ARM heterogeneous design can complete algorithms and functions that are difficult to implement with pure hardware through C language programming.

[0058] (3) The entire system has low hardware cost, low power consumption, and is easy to build. Most of the functions are defined by the program, and it has strong scalability. Without changing the hardware circuit, the image processing algorithm can be added by writing the corresponding Verilog HDL program.

[0059] (4) The clever reuse of FIFO resources is achieved. A group of FIFOs is shared in multiple algorithms using 3×3 convolution kernels to obtain neighborhood pixel data and then perform different operations, effectively saving FPGA on-chip resources while achieving better register-level speed performance.

[0060] In the present invention, the SD card reader module is used to read raw video data stored on an SD card. This module includes an SD card initialization unit and an SD card read control unit. The SD card initialization unit prepares the card for operation, ensuring proper communication and data transmission between the card and the host device. The SD card read control unit communicates with the SD card using the four-wire SDIO protocol, enabling high-speed, low-latency reading of raw video data.

[0061] In this invention, the image buffer FIFO module is used to cache three rows of pixel data for interpolation processing by the Bayer to RGB conversion module. It contains two 16-bit wide and 2048-bit deep FIFO units, each capable of temporarily storing one row of image data. After determining the width and height of the entire image, this module inputs the current pixel and directly outputs the other pixels within a 3×3 range. It also outputs the current row, column, and frame counts for subsequent use by the 7-in-1 image processing module.

[0062] In this invention, the Bayer to RGB module performs Bayer interpolation on the original video and outputs video stream data in the color-restored RGB565 format. Because the original video image has a GRBG Bayer distribution, each pixel in a Bayer-format image only contains a portion of the spectrum. To obtain the RGB format of each pixel from the Bayer format, interpolation is required to fill in the missing two colors. This module uses a linear interpolation compensation algorithm that balances speed and quality, effectively restoring the original video color effect.

[0063] In the present invention, the 7-in-1 image processing module integrates 7 image processing algorithms: RGB2Gray, image binarization, gamma transform, Gaussian filtering, Sobel operator edge extraction, Laplace sharpening, and erosion expansion, to achieve different video image processing effects. Each submodule is independent. The 7-in-1 image processing module externally leads to a function selection port func_sel, which is used to call different submodules and switch different video image processing effects. The corresponding relationship between the value of func_sel and the video processing algorithm executed by the module is as follows:

[0064] (1) When func_sel = 3'd0, the original video data stored in the SD card is processed by the Bayer to RGB module without any other video processing and is displayed directly;

[0065] (2) When func_sel = 3'd1, the RGB2Gray algorithm is executed on the original video based on (1);

[0066] (3) When func_sel = 3'd2, the image binarization algorithm is performed on the original video based on (1);

[0067] (4) When func_sel=3'd3, perform gamma transform algorithm on the original video based on (1);

[0068] (5) When func_sel=3'd4, the Gaussian filtering algorithm is performed on the original video based on (1);

[0069] (6) When func_sel=3'd5, the Sobel operator edge extraction algorithm is performed on the original video based on (1);

[0070] (7) When func_sel = 3'd6, the Laplace sharpening algorithm is performed on the original video based on (1);

[0071] (8) When func_sel = 3'd7, the corrosion and expansion algorithm is performed on the original video based on (1);

[0072] In the present invention, the SDRAM read-write control module is used to realize data transfer across clock domains. The module includes an on-chip sealed SDRAM, an SDRAM control module and a dual-port FIFO. Among them, the SDRAM control module is connected to the on-chip sealed SDRAM, and the dual-port FIFO is divided into a "read port" and a "write port" and is connected to the SDRAM control module. As a hardware unit, the on-chip sealed SDRAM has the same pins and control signals as an independent SDRAM, and requires initialization and appropriate drive timing to read and write it; the SDRAM control module is used to generate the drive timing of the SDRAM, and uses a finite state machine to complete the initialization and reading and writing of the SDRAM; the dual-port FIFO is used to indirectly access the data in the SDRAM. This module can read and write SDRAM data at any time, circumventing the complex read and write timing of the SDRAM, and achieving the purpose of simplifying operations. The SDRAM read-write control module is used to receive the output video stream data of the Bayer to RGB module and the 7-in-1 image processing module and cache a complete 1-frame image, and then send it to the HDMI display control module for display, achieving a high-speed and real-time effect.

[0073] In the present invention, the HDMI display control module is used to receive video stream data and generate standard HDMI timing through TMDS encoding and serialization operations, thereby displaying the processed video on the display. The HDMI display control module includes a TMDS encoding unit, a parallel-to-serial conversion unit, and an output timing control unit.

[0074] The video data first passes through the TMDS encoding unit, which encodes the original RGB video data into three channels of differential data for output on the HDMI interface. According to the TMDS protocol, 8b / 10b encoding is used to convert the original 8-bit signal data per channel into 10 bits. The video data then passes through the parallel-to-serial conversion unit to convert it into a serial signal. Finally, the output timing control unit outputs the three R, G, and B channels of video data, along with a serial clock channel, for correct display on the HDMI display. The final video display has a resolution of 1920×1080 and a frame rate of 40 frames per second.

[0075] In the present invention, the ARM side is used to implement user interaction functions in the video processing system. The ARM side implements the interaction between the Cortex-M0 processor and the FPGA peripherals via the AHB-Lite bus. The Cortex-M0 processor reads the key status to call different interrupt functions in the Cortex-M0 core. Finally, the data after interrupt processing is transmitted to the FPGA peripherals to realize the control function of the FPGA, that is, the key data is sent to the FPGA side to complete the switching of different video processing algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0077] Figure 2 This is a schematic diagram of the operation timing of the SD card reading module of the present invention;

[0078] Figure 3 This is a schematic diagram of the design structure of the image cache FIFO module of the present invention;

[0079] Figure 4 It is the state transition diagram of the SDRAM read-write control module of the present invention;

[0080] Figure 5 This is a schematic diagram of the implementation effect of the Bayer to RGB algorithm of the present invention;

[0081] Figure 6 This is a flow chart for implementing the corrosion expansion algorithm of the present invention; DETAILED DESCRIPTION

[0082] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0083] The following is a detailed description of the real-time video processing system based on Anlu FPGA according to an embodiment of the present invention with reference to the accompanying drawings. Figure 1As shown, it includes an SD card, an Anlu FPGA board and an HDMI display. The SD card is used to store raw video data in Bayer format for reading by the Anlu FPGA board; the Anlu FPGA board includes an FPGA end and an ARM end, wherein the FPGA end includes an SD card reading control module, an image cache FIFO module, a Bayer to RGB module, a 7-in-1 image processing module, an SDRAM read-write control module, and an HDMI display control module. The connection relationship between each module is as follows: the SD card reading control module is connected to the image cache FIFO module, the image cache FIFO module is connected to the Bayer to RGB module, the Bayer to RGB module is connected to the 7-in-1 image processing module, and the SDRAM read-write control module is connected to the Bayer to RGB module, the 7-in-1 image processing module, and the HDMI display control module respectively; wherein the ARM end includes a Cortex-M0 processor core, an AHBLite-Interconnect data bus, and various peripheral interfaces. The connection relationship between each module is as follows: the Cortex-M0 processor core is connected to the AHBLite-Interconnect data bus, and various peripheral interfaces are connected to the AHBLite-Interconnect data bus; the Anlu FPGA board completes all video processing functions; and the HDMI display outputs the processed video. The following steps are included:

[0084] Step 1: Initialize the FPGA hardware modules and the ARM Cortex-M0 soft core. Add two PLL IP cores to the TD project, enable a total of five clock output channels, and set the output frequencies as follows: sys_clk is the system master clock, with a frequency of 50MHz; clk_10m is the SD card read clock, with a frequency of 10MHz; clk_pixel is the HDMI display pixel clock, with a frequency of 100MHz; clk_pixel_5x is the HDMI display serial clock, with a frequency of 500MHz; and clk_sdr_ref is the SDRAM read and write control clock, with a frequency of 150MHz. Next, add the FPGA-side module code and instantiate the "cortexm0ds_logic.v" netlist file to implement the ARM processor deployment. Then, write the AHB-Lite interface module to implement the interaction between the ARM processor and the FPGA hardware resources. The AHB-Lite interface module instantiates two module units, named AHB-Decoder and AHB-SlaveMUX. The Decoder module receives address information written by the CPU and determines which peripheral module on the FPGA to enable. The SlaveMUX module selects a slave and chooses the corresponding control signal and data path. Finally, we wrote the ARM processor startup code on the Keil platform to initialize the interrupt handler function IRQ0 and call the main function from the reset handler to implement interrupt initialization.

[0085] Step 2: Configure the SD card reader module. Add the SD IP core to the TD project, set it to 4-bit SDIO mode, enable the corresponding ports, and set pin constraints based on the actual development board pinout, including the clock line (SCK), command line (CMD), and four data lines (DAT[3:0]). The clock line (SCK) is used to synchronize data transmission, the command line (CMD) is used to transmit commands and responses, and the four data lines (DAT[3:0]) allow four bits of data to be transmitted simultaneously, improving data transmission efficiency. An SD card requires initialization after power-up. During this initialization phase, the host device sends a series of commands to the SD card. These commands configure the SD card's voltage range, capacity, speed class, and other operating modes, allowing the host device to interact with it. During initialization, the SD card sets its internal busy signal to 1, indicating that the card is in the setup process and cannot perform other read or write operations. Once all necessary configuration commands have been executed correctly and the SD card is ready to receive instructions, the SD card will set the busy signal to 0, indicating that the initialization process has been completed and the card is now in an idle state and can receive new commands. At this time, the host device begins to read the video data in the SD card normally. The operation timing is as shown in the attached figure. Figure 2As shown. When the busy signal of the SD card module is 0, that is, the SD card is in idle state and ready to receive new commands, if the read signal port ren detects a high-level pulse at this time, it will trigger the module to enter the read data mode. At this time, in order to indicate that the SD card module is processing the read operation, the busy signal is set to 1 to prevent interference from other commands during the data transmission process. Once entering the read data mode, the CMD17 command is first sent to the SD card. This is a standard single-block read command used to instruct the SD card to prepare to send a data block with a specific address. After sending the command, wait for the response signal from the SD card. After receiving the SD card's ready response, it is necessary to wait for a certain number of clock cycles for the SD card to prepare the data and start transmission. After the data transmission starts, the SCK clock signal generated by dividing the main clock signal sys_clk by two is used to synchronize the data reception.

[0086] Step 3: The original video data is sent to the image cache FIFO module for caching for further processing. In this step, since the convolution or pixel interpolation algorithm of the two-dimensional image requires pixel data in the surrounding 3×3 range, at least two rows of pixel data must be cached first. The pixel cache of the same row can be simply implemented using three cascaded shift registers, while the pixel cache of different rows is implemented using FIFO. FIFO can store the 3×3 matrix data of a frame of the current clock in real time. As the clock jumps, the sliding window operation in the two-dimensional convolution is achieved. The specific implementation structure is shown in the attached. Figure 3 This method does not require SDRAM memory when temporarily storing data, and can automatically shift the video stream to achieve high-speed real-time effects. An image cache module is designed based on this structure, with two FIFO IP cores instantiated internally, each of which can temporarily store one line of image data. After determining the width and height of the entire image, the module inputs the current pixel and directly outputs other pixels within a 3×3 range. It also outputs the current row, column, and frame counts for the subsequent 7-in-1 image processing module to call.

[0087] Step 4: Read the pixel data in the surrounding 3×3 range buffered in step 3, and then perform the Bayer to RGB interpolation algorithm. Since the original video is arranged in GRGB pixels, there are only four possible arrangements of adjacent pixels for each pixel. The specific hardware implementation can be calculated based on the parity of the last bit of the current row and column counts output by the image buffer module in step 3, and the corresponding interpolation is performed respectively. The specific method is: when the row count is even and the column count is odd, the current point is R, the center pixel of the 3×3 template is G, the upper and lower edges of the pixel are R, and the left and right edges are B; when the row count is odd and the column count is even, the current point is B, the center pixel of the 3×3 template is G, the upper and lower edges of the pixel are B, and the left and right edges are R; when both the row count and column count are odd, the current point is G, the center pixel of the 3×3 template is B, the four diagonally opposite corners of the pixel are R, and the upper, lower, left, and right edges are G; when both the row count and column count are even, the current point is G, the center pixel of the 3×3 template is R, the four diagonally opposite corners of the pixel are B, and the upper, lower, left, and right edges are G. Following this rule, the entire frame of the image is interpolated, and the correct RGB image is obtained after completion.

[0088] Step 5: The ARM side detects the user's key press through a key interrupt. If pressed, the value of the variable func_sel increases by 1 (if it exceeds 7, it returns to 0). The value of this variable switches cyclically from 0 to 7 and is used to select different video processing algorithms in step 6. Among them, the key is assigned to the IRQ0 interrupt of the Cortex-M0. The level change of the key will trigger the corresponding interrupt processing of the Cortex-M0. The key debounce operation is then performed in the interrupt service function. That is, the key status is determined again after a delay of 10ms. After the debounce is completed, it is determined whether the key operation is valid. This can avoid misoperation and improve the reliability of key detection.

[0089] Step 6: Based on the RGB image interpolated in Step 4, one of seven video processing algorithms is executed, depending on the value of the variable func_sel: RGB2Gray, binarization, gamma transform, Gaussian filtering, Sobel operator edge extraction, Laplacian sharpening, erosion, and dilation. When func_sel = 3'd0, the original video data stored on the SD card is directly displayed after steps 1 through 4 without any other video processing. When func_sel = 3'd1, the RGB2Gray algorithm is executed on the original video, converting the RGB channels into a single-channel grayscale value. The original conversion formula is: Gray1 = 0.299*image_in_r + 0.587*image_in_g + 0.114*image_in_b. Here, Gray1 represents the converted grayscale value, and image_in_r, image_in_g, and image_in_b represent the red, green, and blue components of the input color image, respectively. The bit depth is 8 bits and the value range is 0-255. When implementing FPGA hardware, fixed-point arithmetic is generally used due to performance, cost, and power consumption considerations. To implement fixed-point arithmetic, INT8 quantization is required. This involves multiplying both sides of the original transformation equation by 256, resulting in the following equation: Gray2 = 77*image_in_r + 150*image_in_g + 29*image_in_b. Gray2 represents the grayscale value after INT8 quantization. Finally, the result is right-shifted by 8 bits, reducing it by a factor of 256, restoring the original grayscale value and obtaining the final conversion result. When func_sel = 3'd2, the image binarization algorithm is applied to the original video. An appropriate threshold is selected, and each pixel is iterated over, determining whether it should be set to black or white based on the threshold. If the pixel value is greater than the threshold, it is set to white (usually represented as 1); otherwise, it is set to black (usually represented as 0). When func_sel = 3'd3, the gamma transform algorithm is executed on the original video, and the original pixel values ​​are subjected to logarithmic operation. The original formula is g = c*rγ. In the formula, r and g are the pixel values ​​before and after the logarithmic operation, respectively, c is the constant coefficient, and γ is the gamma transform factor. Since the gamma transform is a power function, it involves the operation of complex functions, which is difficult to calculate directly using hardware resources. Therefore, when implementing FPGA hardware, a lookup table-based approach is used to perform gamma transform. Add a ROM IP core, set the bit width to 8 bits, and the depth to 256. Then use MATLAB to calculate the gamma transformed value of the pixel data in advance, export it, and store it in ROM. When the algorithm is executed, the image gamma transform algorithm can be implemented by simply using the ROM lookup table. When func_sel = 3'd4, the Gaussian filter algorithm is executed on the original video. The formula is: Where x, y represent two-dimensional images, and ρ represents variance. Gaussian filtering requires selecting a Gaussian kernel (determining a suitable ρ) and then performing convolution. That is, the pixels within a 3×3 range are first processed using the two-dimensional Gaussian distribution formula, and then these nine values ​​are added together to obtain the Gaussian filter value at the center point. Integer quantization is performed during FPGA hardware implementation to speed up the calculation, simplifying the two-bit Gaussian distribution to Then perform a two-dimensional convolution operation with the pixels within the 3×3 range of the original video to obtain the Gaussian filter result of the center pixel. When func_sel=3'd5, the Sobel operator edge extraction algorithm is executed on the original video. Its core is the Sobel operator, which includes two 3×3 matrices, one is horizontal (G x ), one is vertical (G y ), and use them to perform two-dimensional convolution operations with the 3×3 matrix of the image to obtain the approximate values ​​of the horizontal and vertical brightness differences. This step is expressed as: Where A is the 3×3 matrix of the image. After obtaining the vertical operator and the horizontal operator, the gradient vector G of the image is obtained by processing according to the results of the two directions. The calculation formula is The multiplication by 2 operation is implemented by shifting left by one bit. Multiply the operator by itself and sum it to get the data for which the square root needs to be calculated. Here, a lookup table is used to implement it. Select a suitable threshold Threshold and compare the obtained gradient value with the threshold. If it is greater than the value, the edge is determined. If it is less than the value, the opposite is true. The image value is flipped, and the edge of the target image is finally turned white, and the non-edge is turned black. When func_sel = 3'd6, the Laplace sharpening algorithm is performed on the original video. When the grayscale of the center pixel of the neighborhood is lower than the average grayscale of other pixels in its area, the grayscale of this center pixel should be further reduced; when the grayscale of the center pixel of the neighborhood is higher than the average grayscale of other pixels in its neighborhood, the grayscale of this center pixel should be further increased. The 3×3 Laplace sharpening operator used in this system is A two-dimensional convolution operation is performed on pixels within a 3×3 window of the original video to obtain the Laplacian sharpening result for the center pixel. The weight of the center pixel is 5, and the weight of the surrounding four pixels is -1. This emphasizes edges and details in the image, enhancing sharpness. When func_sel = 3'd7, the erosion and dilation algorithm is applied to the original video. A threshold parameter is introduced to ensure that the result is 0 if there is even one zero in the 3×3 window, and 1 only if all pixels are 1. This not only adjusts the strength of the erosion and dilation algorithms, but also allows the two operations to be combined into one through a single threshold comparison. In the FPGA hardware implementation, after obtaining the current 3×3 sliding window data through the FIFO, the erosion or dilation operation can be started. First, the cumulative sum of the three pixels in each row of the 3×3 window is calculated. Next, the cumulative sum of all nine pixels in the 3×3 window is calculated. Finally, the result is compared with the threshold. Here, the threshold for the erosion algorithm is set to 6, and the threshold for the dilation algorithm is set to 3. If the result is greater than or equal to the threshold, the output is 1 as the erosion or dilation result; otherwise, the output is 0 as the erosion or dilation result. In addition, it is also necessary to determine whether the center pixel of the 3×3 window is located at the image boundary. If it is located at the image boundary, the output is 0 as the result of the corrosion or expansion algorithm. The specific process of the algorithm is as follows Figure 6 After executing the video processing algorithm, the processed video data is sent to the built-in buffer of the 7-in-1 image processing module for subsequent video caching and display.

[0090] Step 7: Add the SDRAM IP core, set the corresponding port mapping to the on-chip sealed SDRAM, set the read and write timing parameters of the SDRAM, set the burst length of the SDRAM read and write to a fixed 256 bytes, and then introduce two asynchronous FIFOs to control the read and write of the SDRAM. The read and write FIFOs are used to indirectly access the data in the SDRAM, which to a certain extent avoids the complex read and write timing of the SDRAM and simplifies the operation. In addition, enable the "ping-pong operation" mode in the dual-port FIFO, divide the SDRAM address space into two areas of the same size, and read and write the two areas in turn to avoid conflicts in reading and writing the same address and reduce the access delay of the SDRAM. After the SDRAM is initialized and enters the normal working state, its state transition diagram is shown in the attached figure. Figure 4 The main operations are "refresh," "read," and "write." Each operation has several steps and different delays, and cannot be interrupted. After the SDRAM read and write control module is configured, the target data after the video processing algorithm is executed is read from the buffer built into the 7-in-1 image processing module and written to the SDRAM according to the SDRAM timing requirements, serving as the frame buffer for video display.

[0091] Step 8: Instantiate the HDMI-tx netlist for HDMI video output. Set the pixel clock PXLCLK_I to 100MHz, the serial clock PXLCLK_5X_I to five times the pixel clock (500MHz), and the resolution to 1080P, consistent with the original video. Set the refresh rate to 40Hz to avoid screen flickering and display issues caused by the high serial clock frequency at 1080P@60Hz, while also preventing compatibility issues with 30Hz and lower monitors. Furthermore, strictly follow the VESA standard to set the timing parameters for the horizontal sync signal VGA_HS, vertical sync signal VGA_VS, data enable signal VGA_DE, and data signal VGA_RGB to prevent display artifacts such as distorted or black screens. Finally, configure the corresponding pin constraints as differential outputs with LVDS3V3 levels. Configure the netlist to output the P-side signals for the three video data channels and the clock channel. The N-side signals are automatically constrained during the pin constraint stage. After the HDMI display control module is configured, the processed video data is read from the SDRAM. The TMDS encoding unit in this module is used to encode the original RGB video data into three-way differential data that can be output by the HDMI interface. The specific method is determined by the TMDS protocol: first, 8b / 10b encoding is used to convert the original 8-bit raw signal data of each channel into 10 bits; then the video data is converted into a serial signal through the parallel-to-serial conversion unit; finally, the output timing control unit outputs the processed video to the HDMI display.

[0092] At this point, the design of the real-time video processing system based on Anlu FPGA has been completed from step 1 to step 8.

[0093] To test the system's performance and compare it with mainstream video processing platforms, the following test plan was designed: A uniform RAW format original image was used, with Bayer GRGB pixel arrangement, a resolution of 1920*1080, 16-bit color depth, and 4MB of storage space. The CPU and GPU were deployed using Python and the OpenCV library. CUDA acceleration was enabled for the GPU deployment, but not for the CPU. The same eight image processing algorithms were run on different platforms, and the execution time required to process a single frame was compared. The eight image processing algorithms were: Bayer to RGB conversion, RGB2Gray, binarization, gamma transform, Gaussian filtering, Sobel operator edge extraction, Laplacian sharpening, erosion, and dilation. The comparative test results across different platforms are shown in Table 1, and the FPGA resource consumption is shown in Table 2.

[0094] Table 1

[0095]

[0096] Table 2

[0097]

[0098] The real-time video processing system based on Anlu FPGA has an execution speed of 5.50FPS, power consumption of only 1.2W, and a performance-to-power ratio of 4.583, which is much higher than the CPU platform and GPU platform; the performance-to-price ratio is 29.25, which is also higher than the CPU platform and GPU platform. Its low power consumption and low cost are more conducive to deployment in edge scenarios, and compared with server platform deployment, it has higher long-term reliability and maintenance-free characteristics, and the hardware used is 100% domestically produced. In the resource report of the TD software given in Table 2, the lookup table uses 35.64% of the available resources, REG uses 19.41% of the resources, BRAM uses 32.81% of the available resources, and DSP uses 41.38% of the available resources. By analyzing the proportion of occupied resources, it can be seen that the hardware utilization efficiency of the present invention is high, and only a part of the chip resources are used, and there is still a lot of room for expansion under the premise of unchanged cost.

[0099] In another aspect, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0100] On the other hand, the present invention further discloses a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0101] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the ZYNQ-based face detection acceleration design methods in the above embodiments.

[0102] It is understandable that the system, device and storage medium provided in the embodiments of the present invention correspond to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts of the above methods.

[0103] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0104] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0105] Each embodiment in this specification is described in a related manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so the description is relatively simple. For related parts, refer to the description of the method embodiment.

[0106] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A real-time video processing system based on Anlu FPGA, characterized in that: Including SD card, Anlu FPGA board and HDMI display; SD card is used to store raw video data in Bayer format for reading by Anlu FPGA board; Anlu FPGA board is used to complete all video processing functions, including FPGA side and ARM side, where the FPGA side includes SD card reading control module, image cache FIFO module, Bayer to RGB module, 7-in-1 image processing module, SDRAM read and write control module and HDMI display control module, where the SD card reading control module is connected to the image cache FIFO module, the image cache FIFO module is connected to the Bayer to RGB module, the Bayer to RGB module is connected to the 7-in-1 image processing module, the SDRAM read and write control module is connected to the Bayer to RGB module, the 7-in-1 image processing module and the HDMI display control module respectively; SD card reading The 10-bit ... HDMI display control module, used to receive video stream data and generate standard HDMI timing through TMDS encoding and serialization operations, so as to display the processed video on the display; The ARM side is used to implement the user interaction function in the video processing system, that is, to detect whether a key is pressed through an external interrupt and send the key data to the FPGA side to complete the switching of different video processing algorithms; the HDMI display is used to output the processed video.

2. The real-time video processing system based on Anlu FPGA according to claim 1 is characterized in that: The SD card reader module includes an SD card initialization unit and an SD card reader control unit; the SD card initialization unit is used to prepare the card to enter an operational state and ensure that it can communicate and transmit data correctly with the host device: the SD card will first perform an initialization process after power-on. During the initialization phase, the host device will send a series of commands to the SD card. These commands involve the configuration of the SD card's voltage range, capacity, speed category, and other operating modes; the SD card reader control unit uses the four-wire SDIO protocol to communicate with the SD card.

3. The real-time video processing system based on Anlu FPGA according to claim 1 is characterized in that: The image cache FIFO module is used to cache three rows of pixel data. Since the convolution or pixel interpolation algorithm of a two-dimensional image requires pixel data in a surrounding 3×3 range, at least two rows of pixel data must first be cached for algorithm call. Therefore, an image cache FIFO module is designed. It has two FIFO units with a bit width of 16 bits and a depth of 2048, each of which can temporarily store one row of image data. After determining the width and height of the entire image, the module inputs the current pixel and directly outputs other pixels in the 3×3 range. It also outputs the current row, column, and frame counts for the subsequent 7-in-1 image processing module to call. When temporarily storing data, the module does not need to call SDRAM memory and can automatically shift the video stream to achieve high-speed real-time effects.

4. The real-time video processing system based on Anlu FPGA according to claim 1, characterized in that: The Bayer to RGB module is configured to perform the following operations: Since the original video image has a GRBG Bayer distribution, each pixel in the Bayer format image only contains a portion of the spectrum. To obtain the RGB format of each pixel from the Bayer format, interpolation is required to fill in the missing two colors. This module uses a linear interpolation compensation algorithm that balances speed and quality. The specific steps are as follows: (1) When R and B pixels are distributed above, below, left, and right of the interpolation center pixel, R and B are respectively taken as the average value of the two neighboring pixels; (2) When R and B pixels are distributed diagonally across from the interpolation center pixel, the average of the four B or R pixels in the neighborhood is taken as the B value of the middle pixel; (3) Since the human eye is most sensitive to green light and less sensitive to purple and red light, in order to achieve better image quality, the green component G needs to be specially processed. The specific formula is as follows: Among them, G(R) represents the green component corresponding to the red center pixel R, G(B) represents the green component corresponding to the blue center pixel B, G1-G4 represent the four green pixel values ​​within the 3×3 area of ​​the center pixel, R1-R4 represent the four red pixel values ​​with a spacing of 2 grids above, below, left, and right of the red center pixel, and B1-B4 represent the four blue pixel values ​​with a spacing of 2 grids above, below, left, and right of the blue center pixel.

5. The real-time video processing system based on Anlu FPGA according to claim 1 is characterized in that: The 7-in-1 image processing module includes seven image processing algorithm submodules: RGB2Gray, image binarization, gamma transform, Gaussian filtering, Sobel operator edge extraction, Laplace sharpening, and erosion and dilation. Each submodule is independent of the others. The 7-in-1 image processing module provides a function selection port func_sel for calling different submodules and switching different video image processing effects. The corresponding relationship between the value of func_sel and the video processing algorithm executed by the module is as follows: (1) When func_sel = 3'd0, the original video data stored in the SD card is processed by the Bayer to RGB module without any other video processing and is displayed directly; (2) When func_sel = 3'd1, the RGB2Gray algorithm is executed on the original video based on (1); (3) When func_sel = 3'd2, the image binarization algorithm is performed on the original video based on (1); (4) When func_sel=3'd3, perform gamma transform algorithm on the original video based on (1); (5) When func_sel=3'd4, the Gaussian filtering algorithm is performed on the original video based on (1); (6) When func_sel=3'd5, the Sobel operator edge extraction algorithm is performed on the original video based on (1); (7) When func_sel = 3'd6, the Laplace sharpening algorithm is performed on the original video based on (1); (8) When func_sel=3'd7, the corrosion and expansion algorithm is performed on the original video based on (1).

6. The real-time video processing system based on Anlu FPGA according to claim 1, characterized in that: The SDRAM read / write control module includes an on-chip packaged SDRAM, an SDRAM control module, and a dual-port FIFO. The SDRAM control module is connected to the on-chip packaged SDRAM, and the dual-port FIFO is divided into a "read port" and a "write port" and is connected to the SDRAM control module. The on-chip packaged SDRAM is a hardware unit with the same pins and control signals as an independent SDRAM, requiring initialization and appropriate drive timing to read and write; The SDRAM control module is used to generate the SDRAM drive timing and use a finite state machine to complete the initialization and reading and writing of the SDRAM; The dual-port FIFO is used to indirectly access data in SDRAM. This module can read and write SDRAM data at any time, avoiding the complex read and write timing of SDRAM and simplifying operations.

7. The real-time video processing system based on Anlu FPGA according to claim 1, characterized in that: The HDMI display control module includes a TMDS encoding unit, a parallel-serial conversion unit, and an output timing control unit: The video data passes through the TMDS encoding unit, which encodes the original RGB video data into three-way differential data that can be output from the HDMI interface. According to the TMDS protocol, the original 8-bit signal data of each channel is first converted into 10 bits through 8b / 10b encoding. The video data is then converted into a serial signal through the parallel-to-serial conversion unit and finally output by the output timing control unit. The video display effect is 1920×1080 resolution and 40 frames per second. The corresponding parameters are set as follows: the pixel clock PXLCLK_I is 100MHz, and the serial clock PXLCLK_5X_I is five times the pixel clock, that is, 500MHz. In addition, the horizontal synchronization signal VGA_HS, the field synchronization signal VGA_VS, the data enable signal VGA_DE, and the data signal VGA_RGB must also be input. The timing of these signals must strictly comply with the VESA standard when sending HDMI signals. Finally, the three channels of video data R, G, and B and the serial clock channel are output through the HDMI display control module.

8. The real-time video processing system based on Anlu FPGA according to claim 1 is characterized in that: The ARM side is used to implement user interaction functions in the video processing system: The ARM side implements the interaction between the Cortex-M0 processor and the FPGA peripherals through the AHB-Lite bus. The Cortex-M0 processor reads the button status to call different interrupt functions in the Cortex-M0 core, and finally transmits the data after interrupt processing to the FPGA peripherals to realize the control function of the FPGA and complete the switching of different video processing algorithms.

Images