Video signal acquisition circuit based on FPGA and RK3588

Abstract

The application discloses a video signal collection circuit based on FPGA and RK3588, and has the characteristics that the video signal collection circuit comprises a plurality of SDI decoders, a serial-parallel converter, an FPGA chip and an RK3588 chip, each of the plurality of SDI decoders receives 3G-SDI signals of one camera, the output signals of the plurality of SDI decoders are converted by the serial-parallel converter and then output to the FPGA chip, the FPGA chip performs video format analysis, pixel format processing and low-delay processing, and then outputs data through a video MIPI interface, and the MIPI interface of the RK3588 chip is connected with the video MIPI interface of the FPGA chip; the application has the advantages that the problem that the RK3588 chip does not support an SDI video interface is solved, the interface type of the RK3588 chip is expanded, and the RK3588 chip does not have the limitation of the interface type in actual application.

Description

Technical Field

[0001] This invention relates to the field of video signal acquisition, and more specifically to a video signal acquisition circuit based on FPGA and RK3588. Background Technology

[0002] The RK3588 chip is a flagship 8K SoC chip based on the ARM architecture, primarily used in PCs, edge computing devices, personal mobile internet devices, and other digital multimedia applications. The RK3588 integrates four Cortex-A76 cores and four Cortex-A55 cores, along with a dedicated NEON coprocessor, supporting 8K video encoding and decoding. It boasts a rich set of functional interfaces to meet the customized needs of various industries. However, it lacks a design scheme supporting multiple SDI interfaces. This means that in practical applications, when receiving 3G-SDI signals from external cameras via an SDI transceiver, the RK3588 chip cannot process the video signals. Therefore, the RK3588's lack of SDI video interface support presents a certain limitation.

[0003] Chinese Patent Publication No. CN218676024U discloses a novel, highly scalable and adaptable core board, comprising an integrated board. A control module is located in the center of the front of the integrated board, and memory modules are symmetrically arranged on one side of the control module. A storage module and a power module are sequentially arranged on the front of the integrated board away from the memory modules. Transmission connectors are equidistantly arranged on the back of the integrated board. The control module is electrically connected to the storage module, memory module, power module, and transmission connectors respectively through the integrated board. The control module includes a control chip U1, which is an RK3588 control chip. This patent application, through its novel circuit structure, allows for the rational distribution of various modules on the integrated board, enabling the placement of multiple electronic devices. This improves the functionality of the core board, significantly enhancing the functionality of the product and meeting the needs of different users. It also improves the scalability and iterability of the core board, allowing simultaneous adaptation to two target products without redesign, making it more convenient to use. Although this patent application expands and re-layouts the chip based on the RK3588, it does not support an SDI video interface, which still presents certain limitations. Summary of the Invention

[0004] The technical problem to be solved by the present invention is that the existing RK3588 chip does not support the SDI video interface, which has certain limitations.

[0005] The present invention solves the above-mentioned technical problems through the following technical means: a video signal acquisition circuit based on FPGA and RK3588, including a multi-channel SDI decoder, a serial-to-parallel converter, an FPGA chip and an RK3588 chip. Each SDI decoder receives a 3G-SDI signal from a camera. The output signals of the multi-channel SDI decoders are converted from serial to parallel by the serial-to-parallel converter and then output to the FPGA chip. The FPGA chip performs video format parsing, pixel format processing and low-latency processing and then outputs the data through the video MIPI interface. The MIPI interface of the RK3588 chip is connected to the video MIPI interface of the FPGA chip.

[0006] Beneficial effects: The circuit structure design of the multi-channel SDI decoder, serial-to-parallel converter, FPGA chip, etc. in this invention, as well as the video format parsing, pixel format processing, and low-latency processing of the FPGA chip, ultimately outputs the 3G-SDI signal originally received by the SDI decoder through the video MIPI interface. Then, the MIPI interface of the RK3588 chip is connected to the video MIPI interface of the FPGA chip, which solves the problem that the RK3588 chip does not support the SDI video interface, expands the interface types of the RK3588 chip, and makes the practical application of the RK3588 chip not limited by the interface type.

[0007] Furthermore, the output signal of the multi-channel SDI decoder is converted from serial to parallel by a serial-to-parallel converter and then output to the FPGA chip. This includes: the serial-to-parallel converter converts the 3G-SDI signal input from the multi-channel SDI decoder into serial and parallel signals and then outputs a BT1120 format video stream to the FPGA chip.

[0008] Furthermore, the process of video format parsing by the FPGA chip includes:

[0009] The FPGA chip acquires the timing of the BT1120 format video stream and converts it into a CEA861 format video stream based on the timing relationship between the BT1120 format video stream and the CEA861 format video stream.

[0010] Furthermore, after the FPGA chip performs video format parsing, it also includes:

[0011] For video streams in CEA861 format, pulse signals with pulse widths and pulse intervals exceeding or falling below a preset length are filtered out.

[0012] Furthermore, the pixel format processing process of the FPGA chip includes:

[0013] Both the CEA861 format video stream and the preset video generation signal are input to the selector AUX. The preset video generation signal is used as a test signal. When the selector AUX selects the preset video generation signal as input, it is used to test the performance of the FPGA. The selector AUX selects the CEA861 format video stream and transmits it through the bus. Its color space format is YUV422. The FPGA chip converts the color space format YUV422 to RGB format.

[0014] Furthermore, the low-latency processing of the FPGA chip includes: the RGB format video stream is directly transmitted to the video MIPI interface via a bus.

[0015] Furthermore, the video MIPI interface of the FPGA chip is its inherent interface.

[0016] Furthermore, the video MIPI interface of the FPGA chip is extended through an interface conversion chip, and the extension scheme is as follows:

[0017] The HDMI interface of the FPGA chip is connected to the HDMI interface of the interface conversion chip. The MIPI interface of the interface conversion chip outputs the video stream data after the FPGA chip has undergone video format parsing, pixel format processing and low latency processing.

[0018] Furthermore, the interface conversion chip is model LT6911.

[0019] Furthermore, the serial-to-parallel converter is model GS2971A.

[0020] The advantages of this invention are:

[0021] (1) The circuit structure design of the multi-channel SDI decoder, serial-to-parallel converter, FPGA chip, etc. of this invention, as well as the video format parsing, pixel format processing and low latency processing of the FPGA chip, finally outputs the 3G-SDI signal originally received by the SDI decoder through the video MIPI interface. Then, the MIPI interface of the RK3588 chip is connected to the video MIPI interface of the FPGA chip, which solves the problem that the RK3588 chip does not support the SDI video interface, expands the interface type of the RK3588 chip, and makes the RK3588 chip not limited by the interface type in practical applications.

[0022] (2) The video stream in CEA861 format is easy to process. Based on the timing relationship between the video stream in BT1120 format and the video stream in CEA861 format, this invention converts the video stream in BT1120 format to the video stream in CEA861 format, converts the data format to a more easily processed format, reduces the difficulty of data processing, and facilitates subsequent data processing.

[0023] (3) For video streams in CEA861 format, when there are interference signals or other noise, the pulse width and pulse spacing are inconsistent, which leads to disordered timing recognition. This invention filters out pulse signals whose pulse width and pulse spacing exceed or fall below the preset length to avoid signal interference and improve the accuracy of data processing.

[0024] (4) In the prior art, the RGB format video stream obtained by the FPGA is basically input into the VDMA cache architecture and requires cache space configuration. However, the RGB format video stream of the present invention is directly transmitted to the video MIPI interface through the bus, which reduces the VDMA cache architecture, thereby reducing space configuration and storage process, greatly reducing the latency of video transmission, and also greatly reducing the cost of equipment.

[0025] (5) In addition to using the MIPI interface IP purchased from the official website by the FPGA, which is its inherent interface, the present invention can also expand to obtain a video MIPI interface through an interface conversion chip. By expanding, it is not necessary to purchase IP, reducing equipment costs, and is easy to maintain and mass-produce. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the overall structure of the video signal acquisition circuit based on FPGA and RK3588 disclosed in the embodiments of the present invention;

[0027] Figure 2 This is a schematic diagram of the internal logic of the video signal acquisition circuit based on FPGA and RK3588 disclosed in the embodiments of the present invention;

[0028] Figure 3 This is a timing diagram of a BT1120 format video stream in a video signal acquisition circuit based on FPGA and RK3588 disclosed in an embodiment of the present invention.

[0029] Figure 4 This is a schematic diagram of the frame timing relationship of a BT1120 format video stream in a video signal acquisition circuit based on FPGA and RK3588 disclosed in an embodiment of the present invention.

[0030] Figure 5 This is a timing diagram of a CEA861 format video stream in a video signal acquisition circuit based on FPGA and RK3588 disclosed in an embodiment of the present invention; wherein, Figure 5 (a) is a timing diagram of a video stream in CEA861 format. Figure 5 (b) is a magnified view of a portion of the timing diagram of a video stream in CEA861 format;

[0031] Figure 6 This is a schematic diagram of a video stream processing scheme using VDMA buffering in a video signal acquisition circuit based on FPGA and RK3588 disclosed in an embodiment of the present invention.

[0032] Figure 7 This is a schematic diagram of a VDMA-free video stream processing scheme in a video signal acquisition circuit based on FPGA and RK3588 disclosed in an embodiment of the present invention.

[0033] Figure 8 This is a schematic diagram of the LT6911 chip in the video signal acquisition circuit based on FPGA and RK3588 disclosed in the embodiments of the present invention.

[0034] Figure 9 This is a schematic diagram of the RK3588 chip in the video signal acquisition circuit based on FPGA and RK3588 disclosed in the embodiments of the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] like Figure 1As shown, this invention provides a video signal acquisition circuit based on FPGA and RK3588, including a multi-channel SDI decoder, a serial-to-parallel converter, an FPGA chip, and an RK3588 chip. Each SDI decoder receives a 3G-SDI signal from a camera. The output signals of the multi-channel SDI decoders are converted from serial to parallel by the serial-to-parallel converter and then output to the FPGA chip. The FPGA chip performs video format parsing, pixel format processing, and low-latency processing before outputting the data through the video MIPI interface. The MIPI interface of the RK3588 chip is connected to the video MIPI interface of the FPGA chip. The overall solution adopts a SOC+FPGA hardware architecture. The system mainly consists of an FPGA, an SDI transceiver (SDI decoder), an RK3588 chip, and corresponding peripheral circuits for clock and power supply. The SDI transceiver mainly receives 3G-SDI signals from external cameras, decodes and converts them from serial to parallel, and then sends them to the FPGA. After receiving the signal, the FPGA performs video format parsing and pixel format processing, and then generates a MIPI signal via an LT6911 before sending it to the RK3588. The RK3588 is Rockchip's new generation flagship chip, using an advanced 8nm Fin-FET process, and featuring four A76 cores and four A55 cores. The large cores have a clock speed of 2.4GHz, and the small cores have a clock speed of 1.8GHz. In this invention, the RK3588 chip mainly performs functions such as real-time video display, compression, transmission, and storage. The following details the content of each part.

[0037] like Figure 2 As shown, the serial-to-parallel converter converts the 3G-SDI signals input from the multi-channel SDI decoder into serial-to-parallel signals and outputs a BT1120 format video stream to the FPGA chip. The serial-to-parallel converter is model GS2971A. Figure 2 TC stands for Timing Control. VIDEO TO AXIS means the video stream is transmitted via a bus, and AXIS TO VIDEO means the bus signal is converted into a video stream.

[0038] Continue reading Figure 2 The FPGA chip acquires the timing information of the BT1120 format video stream and, based on the timing relationship between the BT1120 and CEA861 format video streams, converts the BT1120 format video stream to the CEA861 format video stream. The specific process is as follows:

[0039] The timing diagram of the BT1120 format video stream is as follows: Figure 3 As shown, the frame timing relationship of a BT1120 format video stream is as follows: Figure 4 As shown, Figure 4The (F / V / H) values ​​reflect the status of bits F, V, and H. A 3-bit word composed of F, V, and H is counted in base 21 (BCD). F corresponds to the MSB (most significant bit), and H corresponds to the LSB (least significant bit). For example, a bit with the value 3 is F=0, V=1, and H=1. The timing diagram of a CEA861 format video stream is shown below. Figure 5 As shown, Figure 5 (a) is a timing diagram of a video stream in CEA861 format. Figure 5 (b) is a magnified partial view of the timing diagram of a CEA861 format video stream. A careful analysis of the correspondence between these two timing sequences reveals that the CEA861 timing can be derived from the BT1120 signal (the HVDe signal is inferred from the HVF signal; the HVDe signal represents the timing signals HSYNC and VSYNC of the CEA861 format video stream). Analysis Figure 3 and Figure 4 The BT1120 begins displaying valid data after SAV(200). The field containing SAV(2AC) and EAV(2DB) is the field blanking region. According to the chip datasheet, the Data Enable setting for the CEA861 corresponding to this region is 0. At this point, analysis... Figure 5 Timing Diagram: Locate the corresponding position in the CEA861 timing diagram where VSYNC is from 1122 to 41. According to the CEA861 timing diagram, VSYNC is high between 1 and 5. Analysis Figure 3 Timing diagram: The field-active region between SAV(200) and EAV(274). According to the datasheet, the V of BT1120 is 1 at this time. The same logic applies to the analysis... Figure 5 The corresponding region was found: this corresponds to the portion of the CEA861 VSYNC timing from 42 to 1121. The above discussion established the timing relationship between the CEA861 VSYNC and the BT1120 V. Between the BT1120 SAV and EAV, whether it's the field cancellation hidden region or the field active region, it corresponds to the line hidden region. At this time, the parameter value of BT1120 H is 1. Similarly, this region corresponds to the portion of the CEA861 HSYNC from 0 to 280. However, in the CEA861 timing, the value of HSYNC is only 1 from 88 to 132. Based on the above analysis, the correspondence between the two timings was basically determined. Therefore, according to the timing relationship between the BT1120 format video stream and the CEA861 format video stream, the BT1120 format video stream was converted to the CEA861 format video stream.

[0040] After the FPGA chip described above performs video format parsing, for CEA861 format video streams, in the presence of interference signals or other noise, the pulse width and pulse spacing are inconsistent, leading to timing recognition disorder. Therefore, this invention filters out pulse signals whose pulse width and pulse spacing exceed or fall below the preset length.

[0041] Continue reading Figure 2 Both the CEA861 format video stream and the preset video generation signal are input to the selector AUX. The preset video generation signal serves as a test signal. When the selector AUX selects the preset video generation signal as input, it is used to test the FPGA's performance. The process of processing the preset video generation signal input to the FPGA chip is the same as the process of processing the CEA861 format video stream input; however, the preset video generation signal is mainly for testing purposes. The following explanation uses the data processing flow of the CEA861 format video stream within the FPGA chip as an example. The selector AUX selects the CEA861 format video stream, which is transmitted via a bus. Its color space format is YUV422. The FPGA chip converts the YUV422 color space format to RGB format. The specific conversion process follows existing technology, and the general flow is as follows:

[0042] YUV422 and RGB are two different pixel encoding formats. The following formula is used to convert a YUV422 signal into an RGB signal.

[0043] R=1.164(Y-16)+1.793(CR-128)=1.164Y+1.793CR-248.128;

[0044] G=1.164(Y-16)-0.534(CR-128)-0.213(CB-128)=1.164Y-0.213CB-0.534CR+76.992;

[0045] B=1.164(Y-16)+2.115(CB-128)=1.164Y+2.115CB-289.344;

[0046] The above processing can convert the YUV422 signal into an RGB signal. Meanwhile, to reduce the slow processing speed caused by floating-point and division operations, the above formula is transformed by multiplying both sides by 512 and then multiplying them by 512 to implement the algorithm. Since 1024 is 2 to the power of 9, only a shift register method is needed for processing.

[0047] R<<9=596Y+817CR-114131;

[0048] G<<9=596Y-200CB-416CR+69370;

[0049] B<<9=596Y+1033CB-141787;

[0050] Based on the above formula, the RGB signal is shifted left by 9 bits, and after conversion, it is shifted right by 9 bits to restore it. It should be noted that during the conversion process, if the number is negative, it is treated as 0, and if it is greater than 255, it is treated as 255.

[0051] Continue reading Figure 2 RGB format video streams are directly transmitted to the video MIPI interface via a bus. In existing FPGA technology, the RGB format video streams processed are generally input into a VDMA buffer architecture and require buffer space configuration. A schematic diagram of the existing VDMA buffer video stream processing scheme is shown below. Figure 6 As shown, the RGB format video stream of this invention is directly transmitted to the video MIPI interface via a bus, reducing the VDMA cache architecture, thereby reducing space configuration and storage processes, greatly reducing video transmission latency, and also significantly reducing equipment costs. A schematic diagram of the VDMA-free video stream processing scheme of this invention is shown below. Figure 7 As shown. It should be noted that, Figure 6 In the middle, "Video in to axi4-s" means converting the input audio signal into the FPGA's internal AXI bus; "Procesing pipe" means pipelined processing; "VDMA" is an image buffer module for image interaction between the FPGA and DDR; "AXI4-S to Video out" means converting the AXI bus output into a video signal output. Figure 7 In this context, "Viedeo in to axi4-stream" means converting the input audio signal to the FPGA's internal AXI bus; "AXI4-stream to Video out" means converting the AXI bus output to a video signal output. "Video timing controller" refers to the timing generation module, and "Video Processing" refers to video processing, which in this solution is YUV to RGB conversion.

[0052] The FPGA chip's video MIPI interface is its inherent interface, meaning the MIPI interface IP of the FPGA chip can be purchased from the Xinlinx website. However, it is expensive and unsuitable for mass production and maintenance. Therefore, the FPGA chip's video MIPI interface can also be extended using an interface conversion chip. The extension scheme involves connecting the FPGA chip's HDMI interface to the interface conversion chip's HDMI interface. The interface conversion chip's MIPI interface outputs the video stream data after video format parsing, pixel format processing, and low-latency processing by the FPGA chip. The interface conversion chip is model LT6911. The LT6911 is a high-performance HDMI 1.4 to MIPI / CSI chip, configurable in single-port, dual-port, or quad-port modes, with one high-speed clock channel and 1-4 channels with a maximum speed of 1.5Gb / s, supporting a total bandwidth of up to 24Gbps. It supports burst mode DSI video data transmission. In this invention, after the FPGA chip processes the video format conversion, it outputs the video in HDMI format. Then, the LT6911 chip converts the HDMI signal into a MIPI signal and transmits it to the RK3588 chip for processing. This chip also supports LVDS signal output and will be used again in real-time display. The RK3588 chip uses an advanced 8nm FinFET process and integrates a quad-core A76 and quad-core A55 CPU (a total of eight cores). The RK3588 chip performs real-time display, storage, compression, and transmission of four MIPI video signals. A schematic diagram of the LT6911 chip is shown below. Figure 8 As shown in the diagram, the schematic diagram of the RK3588 chip is as follows: Figure 9 As shown.

[0053] Through the above technical solutions, the circuit structure design of the multi-channel SDI decoder, serial-to-parallel converter, FPGA chip, etc., and the video format parsing, pixel format processing, and low-latency processing of the FPGA chip, the 3G-SDI signal originally received by the SDI decoder is finally output through the video MIPI interface. Then, the MIPI interface of the RK3588 chip is connected to the video MIPI interface of the FPGA chip, which solves the problem that the RK3588 chip does not support the SDI video interface, expands the interface types of the RK3588 chip, and makes the practical application of the RK3588 chip not limited by the interface type.

[0054] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video signal acquisition circuit based on FPGA and RK3588, characterized in that, It includes a multi-channel SDI decoder, a serial-to-parallel converter, an FPGA chip, and an RK3588 chip. Each SDI decoder receives a 3G-SDI signal from one camera. The output signals of the multi-channel SDI decoders are converted from serial to parallel by the serial-to-parallel converter and then output to the FPGA chip. The FPGA chip performs video format parsing, pixel format processing, and low-latency processing before outputting the data through the video MIPI interface. The MIPI interface of the RK3588 chip is connected to the video MIPI interface of the FPGA chip. The output signal of the multi-channel SDI decoder is converted from serial to parallel by a serial-to-parallel converter and then output to the FPGA chip. This includes: the serial-to-parallel converter converts the 3G-SDI signal input from the multi-channel SDI decoder into a BT1120 format video stream and outputs it to the FPGA chip; the FPGA chip performs video format parsing by: acquiring the timing of the BT1120 format video stream and converting it to a CEA861 format video stream based on the timing relationship between the BT1120 and CEA861 format video streams; the FPGA chip performs pixel format processing by: inputting both the CEA861 format video stream and a preset video generation signal into the selector AUX, where the preset video generation signal serves as a test signal. When the selector AUX selects the preset video generation signal as input, it is used to test the FPGA's performance; the selector AUX selects the CEA861 format video stream and transmits it via a bus, with a color space format of YUV422. The FPGA chip converts the YUV422 color space format to RGB format.

2. The video signal acquisition circuit based on FPGA and RK3588 according to claim 1, characterized in that, After the FPGA chip performs video format parsing, it also includes: For video streams in CEA861 format, pulse signals with pulse widths and pulse intervals exceeding or falling below a preset length are filtered out.

3. The video signal acquisition circuit based on FPGA and RK3588 according to claim 1, characterized in that, The low-latency processing of the FPGA chip includes: the RGB format video stream is directly transmitted to the video MIPI interface via a bus.

4. The video signal acquisition circuit based on FPGA and RK3588 according to claim 1, characterized in that, The FPGA chip's video MIPI interface is its inherent interface.

5. The video signal acquisition circuit based on FPGA and RK3588 according to claim 1, characterized in that, The video MIPI interface of the FPGA chip is extended through an interface conversion chip, and the extension scheme is as follows: The HDMI interface of the FPGA chip is connected to the HDMI interface of the interface conversion chip. The MIPI interface of the interface conversion chip outputs the video stream data after the FPGA chip has undergone video format parsing, pixel format processing and low latency processing.

6. The video signal acquisition circuit based on FPGA and RK3588 according to claim 5, characterized in that, The interface conversion chip is model LT6911.

7. The video signal acquisition circuit based on FPGA and RK3588 according to claim 1, characterized in that, The serial-to-parallel converter is model GS2971A.

Images