Multi-channel video display device based on FPGA and SoC

Abstract

The invention belongs to the technical field of video comprehensive display, and particularly discloses a multi-channel video display device based on an FPGA (Field Programmable Gate Array) and an SoC (System on Chip). According to the invention, the FPGA and the SoC work cooperatively and exert unique advantages respectively, the display resolution and the multi-window combined display condition are acquired simultaneously according to the mainboard bus command, and the SoC is responsible for code stream video decoding and picture composition such as decoded video multi-window superposition or splicing. The FPGA is responsible for high-real-time radar data polar coordinate conversion, fusion graph drawing and multi-window picture display control, the FPGA cuts windows on cached complete resolution pictures according to window control signals of all videos, the videos are transmitted in the windows, and the display function of picture composition such as window overlapping or splicing is completed more efficiently. And on the premise of not occupying a display interface of the mainboard, the superposed video is output to the display, so that the display information is further expanded.

Description

Technical Field

[0001] The present application belongs to the field of video integrated display technology, and more specifically, relates to a multi-channel video display device based on FPGA and SoC. Background Art

[0002] At present, the main transmission media for video data are optical fiber and network. Optical fiber can well meet the requirements of high bandwidth, low latency and high reliability. More and more sensors choose optical fiber interface for data transmission. The network has a mature topological structure and stable and efficient transmission effect. It can transmit many encoded and compressed videos to various network nodes and terminals, and various videos are displayed on the display in their own styles.

[0003] Existing video display devices are mostly based on the HDMI video of the motherboard, and multiple windows are superimposed on the display output by the motherboard. Combined with the upper-level application, it shows the display effect that the user wants. When the window video occupies a large area, most of the HDMI is occupied by the window, which often sacrifices the screen display space of the motherboard itself. Now there is a need for an extended HDMI display interface that does not occupy the HDMI display interface of the motherboard itself and realizes multi-channel video display. Summary of the invention

[0004] In view of the defects of the prior art, the purpose of this application is to provide a multi-channel video display device based on FPGA and SoC, aiming to solve the problem that most of the HDMI in the prior art is occupied by windows, which often sacrifices the picture display space of the motherboard itself.

[0005] To achieve the above-mentioned purpose, the present application provides a multi-channel video display device based on FPGA and SoC, comprising: a mainboard, FPGA, SoC, a first memory, a second memory and a display;

[0006] The mainboard is configured to synchronously send each video window control command to the FPGA and the SoC via a command bus;

[0007] The SoC communicates with the mainboard via a command bus and is configured to: receive network coded video stream data from the outside and cache it in a second memory, generate HDMI video with a video window according to each video window control command issued by the mainboard, and send it to the FPGA;

[0008] The FPGA communicates with the mainboard through a command bus and DMA, and is configured to: receive radar video data from the outside, perform polar coordinate conversion, receive background images and video window control commands from the mainboard, cache the background image and the radar video image data after polar coordinate conversion to the first memory, receive HDMI video with a video window from the SoC, fuse the timing-synchronized background image and the radar video data after polar coordinate conversion based on the row and field synchronization timing of the HDMI to obtain a fused image, generate a window synchronization digging signal, dig out a blank window on the fused video image, superimpose the timing-synchronized fused image with a blank window and the HDMI video with a video window frame by frame, and generate an image output to a display.

[0009] Preferably, the command bus is a PCIE bus, the FPGA receives radar video data from the outside through a GT interface, receives background images from the mainboard through DMA, the SoC receives network coded video stream data from the outside through a network port, and the first memory and the second memory are peripheral DDR.

[0010] Preferably, the SoC is configured to: generate a corresponding resolution base map according to the display resolution information contained in the control command, form windows of corresponding sizes according to the conditional control information of the multi-window combination display contained in the control command, form a decoded video stream consistent with the window size according to each window size, distribute the decoded video stream on the corresponding resolution base map according to the position of each window, form an HDMI video with a video window, and output it to the FPGA.

[0011] Preferably, the SoC is configured to adjust the video resolution by a bilinear difference scaling algorithm when the original resolution of the transmitted video does not match the window size.

[0012] Preferably, the SoC is configured to: when the video windows have overlapping areas, determine the display priority relationship of the window videos according to the hierarchical relationship of each window contained in the control command.

[0013] Preferably, the FPGA is configured to: while receiving HDMI video with a video window, based on the display resolution information of HDMI and taking the clock frequency of the pixel points as the basic frequency, perform clock counting according to the effective area range of the video pixels, the pixel data count of each row indicates row synchronization, perform row counting according to the row range of a frame of image, the row count of each frame indicates field synchronization, and generate corresponding row and field synchronization signals to indicate the display timing of the HDMI video.

[0014] Preferably, the FPGA is configured to: generate a window synchronization digging signal based on the row and field synchronization timing of HDMI, the window synchronization digging signal is included in the row and field synchronization timing, and marks the row and column range where the window is located; according to the row and column range where the window is located, dig out the same row and column range of the fused video image to obtain an image with a blank window.

[0015] Preferably, the FPGA is configured such that: when superimposing by frame, at a blank window, the HDMI video image with window video has a higher priority; outside the blank window, the fused image has a higher priority.

[0016] Preferably, the FPGA is configured to: adopt a weighted fusion method to fuse the time-series synchronized background image and the radar video data after polar coordinate conversion

[0017] Preferably, the FPGA is configured to upload the display image to the mainboard memory via a DMA uplink data channel for upper-layer software to extract and call as needed.

[0018] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art:

[0019] The present application provides a multi-channel video display device based on FPGA and SoC, which utilizes FPGA and SoC to work together, each of which plays its own unique advantages, and simultaneously obtains the display resolution and the conditions for multi-window combination display according to the motherboard bus command. SoC is responsible for the decoding of the code stream video and the multi-window superposition or splicing of the decoded video, and the FPGA is responsible for the high-real-time polar coordinate conversion and fusion graphics drawing of radar data and the display control of multi-window images. FPGA cuts the window on the cached full-resolution screen according to the window control signal of each video, and transmits the video in the window, so as to more efficiently complete the display function of the window superposition or splicing and other screen compositions. Without occupying the motherboard display interface, the superimposed video is output to the display, which further expands the display information. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a schematic diagram of the structure of a multi-channel video display device based on FPGA and SoC provided by the present application.

[0021] Figure 2 This is a flow chart of a multi-channel video display method based on FPGA and SoC provided by the present application.

[0022] Figure 3 It is a schematic diagram of the fused video image effect provided in an embodiment of the present application.

[0023] Figure 4 It is a schematic diagram of generating a window extraction range provided in an embodiment of the present application.

[0024] Figure 5 It is a schematic diagram of the window-cut video overlay effect provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0026] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0027] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0028] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0029] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0030] Next, the technical solutions provided in the embodiments of the present application are introduced.

[0031] like Figure 1 As shown, the present application provides a multi-channel video display device based on FPGA and SoC, including: a mainboard, FPGA and SoC.

[0032] The main board is responsible for: (1) synchronously sending conditional control information commands including display resolution and multi-window combination display to FPGA and SoC via the command bus; (2) transmitting the background image used for fusion to the peripheral DDR memory of FPGA via DMA. The resolution of the background image is consistent with the resolution transmitted by the command bus.

[0033] The SoC is responsible for: (1) video decoding, adjusting the resolution of the decoded video to match the window size; (2) displaying the decoded video in each window according to the window position command sent by the bus, realizing functions such as overlay or splicing; (3) generating a corresponding resolution background map according to the resolution information sent by the bus, forming windows of corresponding sizes according to the conditional control information of the multi-window combination display, forming a decoded video stream consistent with the window size according to each window size, distributing the decoded video stream on the corresponding resolution background map according to the window position, forming an HDMI video with a video window, and transmitting it to the FPGA.

[0034] FPGA is responsible for: (1) receiving photoelectric radar data from the optical fiber, responsible for high-real-time polar coordinate conversion of radar data, converting the radar data in polar coordinates into image data in rectangular coordinates, and caching it in the peripheral DDR; (2) caching the background image received by DMA and the radar video image data after polar coordinate conversion in the peripheral DDR respectively; (3) using the HDMI video timing with video window input by SoC as the reading reference timing of the cached image, and controlling the synchronous reading of multiple cached images; (4) according to the extracted HDMI synchronization timing, synchronously reading the background image and radar video image from the corresponding DDR memory, synchronizing the timing of the background image and radar video image, and according to a specific fusion algorithm, fusing the two images pixel by pixel to complete the fusion graphics drawing. to obtain a fused image; (5) according to the conditional control information of the multi-window combination display received by the command bus, a window extraction signal is generated for extracting the window of the fused video image; (6) according to the window extraction signal, the row and column range of the window is determined, and the window is extracted within the same row and column range of the fused video image to obtain a fused image with a blank window; (7) the fused image with the extracted blank window and the HDMI image are superimposed frame by frame. Under the control of the synchronous timing, the row and field occupied by the blank area completely overlap with the row and field occupied by the video window, so that the window video superimposed on the HDMI by the SoC is visible in the blank window of the fused image. The window video and the fused video are superimposed to form a new image video. The fused video image with the blank area is on top of the HDMI video, and the blank area shows the window video in the HDMI video.

[0035] FPGA receives fiber radar data through GT port, background image data through DMA, and control command information through bus interface. SoC receives network coded video data through network port and control command through bus interface.

[0036] The optical fiber radar data is data that meets the Fibre Channel (FC) transmission protocol or the 10 Gigabit Ethernet transmission protocol. The GT interface can realize the reception and transmission of high-speed serial signals, and the rate can usually reach several Gbps or even higher. The radar data is used for radar video display with high real-time requirements, and the original data is usually transmitted in an uncompressed manner. Therefore, in this embodiment, the GT interface is selected to receive the optical fiber radar data.

[0037] The DMA is a direct memory access operation that meets the PCIE (PCI Express) bus protocol and is used to achieve inter-memory movement of larger data. This application uses DMA to move the fused background image from the mainboard's memory to the FPGA's external DDR memory for image fusion.

[0038] The bus interface is mainly used to transmit control commands, including information such as the size of each window, the position of each window, the hierarchical relationship of each window, and the display resolution. The bus interface can be UART, I2C, SPI, CAN, PCI, PCIE, etc. The motherboard selects a suitable interface to connect to the FPGA and SoC. In this application, the PCIE interface is used to transmit commands. It is compatible with DMA operations and does not occupy additional FPGA logic resources.

[0039] The SoC receives network coded video data through the network port. SoC (System on Chip) is a system-on-chip, which is an integrated circuit with a dedicated template. The SoC described in this application is mainly used for network decoding and HDMI output integrated circuits. The network coded video can be a video coded in accordance with the mainstream coding methods H.264 or H.265.

[0040] like Figure 2 As shown, the mainboard, FPGA and SoC implement multi-channel video display through the following multiple steps.

[0041] Step S101. SoC caches the received video into the peripheral DDR, and generates HDMI video with video window according to the video window control commands issued by the mainboard, and transmits it to FPGA.

[0042] In one embodiment, the SoC decodes the encoded video data received through the network port, which requires a clear encoding method, such as H.264 or H.265, which is selected according to actual conditions and is not specifically limited in this embodiment.

[0043] The video data decoded by the SoC is adjusted in resolution according to the bus command to match the corresponding window size. When the original resolution of the video transmitted from the network does not match the window size, the decoded video needs to be reduced or enlarged, and the video resolution is adjusted through the bilinear difference scaling algorithm. The corresponding algorithm function is embedded in the SoC, and the user can directly call it to implement the video scaling adjustment function.

[0044] In one embodiment, the SoC generates an HDMI base video according to the display resolution size. The base video is used to place the video window and display the windowed video. The SoC displays the decoded video in a window at the corresponding position of the base video according to the window position, forming an HDMI video with a video window and sending it to the FPGA.

[0045] Step S102. FPGA performs coordinate conversion and caches the fiber optic radar data received from the GT interface. When video window display is required, it uses the HDMI video timing as a reference to synchronously read the background image data and the cached radar video data after polar coordinate conversion from the external DDR memory, and performs image fusion processing through a fusion algorithm to obtain a fused image.

[0046] When receiving HDMI video with a video window, FPGA performs clock counting based on the display resolution information of HDMI and the clock frequency of the pixel points as the basic frequency according to the effective area range of the video pixels. The pixel data count of each row indicates row synchronization, and row counting is performed based on the row range of a frame of image. The row count of each frame indicates field synchronization, and a corresponding row-field synchronization signal is generated to indicate the display timing of the HDMI video, and the row-field synchronization signal is used as a timing synchronization signal for multiple video images.

[0047] FPGA receives radar data from the optical fiber and needs to convert polar coordinates to rectangular coordinates before displaying. The display range of the radar video is determined by the size of the radar display window. Through the relationship between the polar coordinate system and the rectangular coordinate system, the radar rectangular coordinate P(X,Y) corresponding to P(ρ,θ) can be obtained:

[0048]

[0049] The radar data in polar coordinates are mapped one by one to the corresponding rectangular coordinate system to form radar video frames, which are cached in DDR.

[0050] FPGA uses the timing synchronization signal of HDMI to simultaneously read the background image for fusion received by DMA and the polar coordinate converted radar video frame cached in DDR, and uses the fusion algorithm to perform pixel-by-pixel fusion calculation to obtain a fused video image.

[0051] Preferably, the fusion algorithm is a weighted fusion algorithm. Specifically, assuming that the input video A is a radar video signal and B is a background video consisting of a nautical chart (map), then for each video pixel, the mixed video C can be expressed as the weighted sum of the two input videos:

[0052] C=m*A+n*B

[0053] Wherein, 0≤m≤1, 0≤n≤1; m and n are weighting coefficients. In this embodiment, the fused video image is as follows: Figure 3 shown.

[0054] Step 103: The FPGA generates a window extraction signal based on the HDMI timing, and extracts a blank window on the fused video image.

[0055] FPGA separates the reference clock Pixel_clk signal, pixel value RGB signal, line synchronization signal H_blank, field synchronization signal V_blank, and data valid signal Pixel_DE from the HDMI video. Under the reference clock Pixel_clk, each H_blank counts Pixel_DE from 0. The counting range of Pixel_DE in an H_blank is the X-axis length of the resolution rectangular coordinate system. A count is performed on the rising edge of a Pixel_DE signal in V_blank. At the end of V_blank, the counting range is the Y-axis length of the resolution rectangular coordinate system.

[0056] like Figure 4 As shown in the figure, a coordinate system is established at a 4K resolution. The first pixel of the image is used as the (0,0) point to establish a rectangular coordinate system. The length of the X axis is 3840, and the length of the Y axis is 2160, which is the same as the 4K resolution. The bus command sets the window position coordinate (X 2N-2 , Y 2N-2 ), (X 2N-1 , Y 2N-2 ), (X 2N-2 , Y 2N-1 ), (X 2N-1 , Y 2N-1 ) is sent to FPGA, N represents the Nth window. FPGA determines the size of each window and its position in the rectangular coordinate system according to the received window coordinates, marks the Pixel_DE counting range in H_blank and the Pixel_DE rising edge counting range in V_blank covered by the window position coordinates, and uses them as window digging signals. The inside of the square enclosed by the four coordinate points of each window is set as a blank area to realize the window digging function.

[0057] Step 104: FPGA superimposes the fused image with the blank window and the HDMI video image with the video window sent by the SoC to generate an image output to the display, and uploads it to the mainboard memory through DMA for the upper-layer software to call on demand.

[0058] The FPGA places a blank window on the fused image according to the separated HDMI synchronization timing, such as Figure 5 As shown in (a), a fused image with a blank window is obtained.

[0059] Furthermore, the window control command received by the SoC is the same as the window control command received by the FPGA, and the window position and size are also the same in the same image resolution coordinate system. Figure 5 As shown in (b), two images with the same resolution and synchronized timing are directly superimposed. The superposition priority is determined as follows: in the blank window, the HDMI video image with the window video has a higher priority; outside the blank window, the fused image has a higher priority. The superposition effect is as follows: Figure 5 As shown in (c).

[0060] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0061] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0062] It is understandable that the processor in the embodiment of the present application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0063] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0064] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0065] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0066] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A multi-channel video display device based on FPGA and SoC, characterized in that: include: Mainboard, FPGA, SoC, first memory, second memory and display; The mainboard is configured to synchronously send each video window control command to the FPGA and the SoC via a command bus; The SoC communicates with the mainboard via a command bus and is configured to: receive network coded video stream data from the outside and cache it in a second memory, generate HDMI video with a video window according to each video window control command issued by the mainboard, and send it to the FPGA; The FPGA communicates with the mainboard through a command bus and DMA, and is configured to: receive radar video data from the outside, perform polar coordinate conversion, receive background images and video window control commands from the mainboard, cache the background image and the radar video image data after polar coordinate conversion to the first memory, receive HDMI video with a video window from the SoC, fuse the timing-synchronized background image and the radar video data after polar coordinate conversion based on the row and field synchronization timing of the HDMI to obtain a fused image, generate a window synchronization digging signal, dig out a blank window on the fused video image, superimpose the timing-synchronized fused image with a blank window and the HDMI video with a video window frame by frame, and generate an image output to a display.

2. The multi-channel video display device according to claim 1, characterized in that: The command bus is a PCIE bus, the FPGA receives external radar video data through a GT interface, receives background images from a mainboard through DMA, the SoC receives external network coded video stream data through a network port, and the first memory and the second memory are peripheral DDR.

3. The multi-channel video display device according to claim 1, characterized in that: The SoC is configured to: generate a corresponding resolution base map according to the display resolution information contained in the control command, form windows of corresponding sizes according to the conditional control information of the multi-window combination display contained in the control command, form a decoded video stream consistent with the window size according to each window size, distribute the decoded video stream on the corresponding resolution base map according to the position of each window, form an HDMI video with a video window, and output it to the FPGA.

4. The multi-channel video display device according to claim 1, characterized in that: The SoC is configured to adjust the video resolution by a bilinear difference scaling algorithm when the original resolution of the transmitted video does not match the window size.

5. The multi-channel video display device according to claim 1, characterized in that: The FPGA is configured to: while receiving HDMI video with a video window, based on the display resolution information of HDMI and taking the clock frequency of the pixel points as the basic frequency, perform clock counting according to the effective area range of the video pixels, the pixel data count of each row indicates row synchronization, perform row counting according to the row range of a frame of image, the row count of each frame indicates field synchronization, and generate corresponding row and field synchronization signals to indicate the display timing of the HDMI video.

6. The multi-channel video display device according to claim 1, characterized in that: The FPGA is configured to: generate a window synchronization digging signal based on the row and field synchronization timing of HDMI, the window synchronization digging signal is included in the row and field synchronization timing, and marks the row and column range where the window is located; according to the row and column range where the window is located, dig out the same row and column range of the fused video image to obtain an image with a blank window.

7. The multi-channel video display device according to claim 1, characterized in that: The FPGA is configured such that: when superimposing by frames, at a blank window, the HDMI video image with the window video has a higher priority; outside the blank window, the fused image has a higher priority.

8. The multi-channel video display device according to claim 1, characterized in that: The FPGA is configured to: when there are overlapping areas of the video windows, determine the display priority relationship of the window videos according to the hierarchical relationship of each window contained in the control command.

9. The multi-channel video display device according to claim 1, characterized in that: The FPGA is configured to fuse the time-synchronized background image and the polar coordinate-converted radar video data in a weighted fusion manner.

10. The multi-channel video display device according to any one of claims 1 to 9, characterized in that: Place The FPGA is configured to upload the display image to the mainboard memory via the DMA uplink data channel. For upper-level software to extract and call on demand.

Images