Video acquisition and transmission device based on FPGA and ARM

The video acquisition and transmission device enhances acquisition speed and reduces system complexity by using FPGA and ARM components, addressing the limitations of traditional devices, and ensuring real-time performance for video transmission.

CN222128147UInactive Publication Date: 2024-12-06XUZHOU NORMAL UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202420816979.9
Authority / Receiving Office
CN · China
Patent Type
Utility models(China)
Current Assignee / Owner
Filing Date
2024-04-19
Publication Date
2024-12-06
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Traditional video acquisition and transmission devices struggle with acquisition speed, system cost, and reliability, particularly when using FPGA for Ethernet transmission, which increases system complexity and wastes internal resources.

Method used

A video acquisition and transmission device utilizing FPGA and ARM, where the transmitting end includes a video acquisition module connected to a data processing module, write FIFO module, and DDR3 read/write module, while the receiving end includes an Ethernet receiving module and HDMI control module, all running on separate FPGA and ARM components, with the UDP protocol for data transmission.

Benefits of technology

Improves acquisition speed, reduces system complexity, and saves FPGA internal resources, ensuring high flexibility and real-time performance for video transmission, suitable for remote video collection and transmission scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN222128147U_ABST
    Figure CN222128147U_ABST
Patent Text Reader

Abstract

The utility model discloses a video acquisition and transmission device based on FPGA and ARM, which comprises a sending end and a receiving end, and the sending end comprises a register configuration module, a video acquisition module, a data processing module, a DDR3 read-write module (SGDMA), a DDR3 write control module and an Ethernet sending module. And the receiving end comprises an Ethernet receiving module, a DDR3 read control module, a DDR3 read-write module (SGDMA), an HDMI control module and an HDMI driving module. According to the utility model, a mode of combined work of the FPGA and the ARM is used, video data is acquired through the FPGA, and the acquired data is sent through the HPS, so that the quality of acquired graphs is improved, the transmission delay is reduced, and the real-time performance of image transmission is ensured; tedious steps of realizing an Ethernet transmission protocol in the FPGA are avoided, and the complexity of the system is reduced; according to the utility model, the video data can be sent to the receiving end HPS, the data can also be sent to the receiving end PC, or the data can be remotely sent through the Internet, the flexibility of the device is high, and the device is suitable for scenes of remote video acquisition and transmission.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The utility model belongs to the technical field of image processing, and specifically refers to a video acquisition and transmission device based on FPGA and ARM. Background Art

[0002] Video acquisition and transmission technology has been widely used in more and more industries and personal use scenarios. For example, video conferencing, video calls, network monitoring, etc. With the further development of society, people have put forward higher requirements for video quality, and at the same time, the requirements for the real-time performance of videos are getting higher and higher. As the quality of video improves, the amount of data output at the same time is also relatively large, and the requirements for the acquisition speed of the video acquisition system are relatively strict. Although the use of traditional video acquisition and transmission devices can achieve the purpose of video acquisition and transmission, it cannot meet people's needs in terms of acquisition speed, system cost, and reliability.

[0003] Compared with traditional controllers, FPGA has the advantages of fast running speed and high concurrency, which can meet the needs of real-time video data processing. However, when using FPGA to implement Ethernet transmission, the underlying logic of the transmission protocol must be implemented using hardware description language. If the transmission protocol is changed, the underlying logic of the protocol must be implemented again, which increases the complexity of the system and wastes FPGA internal resources. Utility Model Content

[0004] The utility model aims to provide a video acquisition and transmission device, which can improve the acquisition speed, reduce the system complexity and save the internal resources of FPGA.

[0005] To achieve the above-mentioned purpose, the utility model video acquisition transmission device includes a transmitting end and a receiving end, wherein the transmitting end includes a video acquisition module, the video acquisition module receives a video signal from an OV5640 image sensor, the video acquisition module is connected to a data processing module, the data processing module transmits the processed data to a write FIFO module, the output end of the write FIFO module is connected to a transmitting end DDR3 read / write module, i.e., a transmitting end SGDMA, the transmitting end SGDMA is connected to a transmitting end DDR3 controller (HPS), the transmitting end DDR3 controller is connected to a transmitting end DDR3 The output end of the DDR3 write control module is connected to the sending end SGDMA; the sending end DDR3 controller (HPS) outputs data and sends it through the Ethernet sending module; the receiving end includes an Ethernet receiving module, the DDR3 read control module is connected to the receiving end SGDMA, the receiving end SDGMA output end is connected to the read FIFO module, the output end of the read FIFO module is connected to the HDMI control module, the output end of the HDMI control module is connected to the HDMI driver module, the output end of the HDMI driver module is connected to ADV7513, the output end of the ADV7513 is connected to the HDMI display, the receiving end SGDMA is connected to the receiving end DDR3 controller (HPS); the output end of the Ethernet receiving module is connected to the receiving end DDR3 controller (HPS), and the receiving end DDR3 controller is connected to the receiving end DDR3 SDRAM.

[0006] As a further solution of the utility model: the receiving end also includes a register configuration module, and the register configuration module performs parameter configuration on the OV5640 image sensor.

[0007] As a further solution of the utility model: the video acquisition module, data processing module, write FIFO module, sender SGDMA, register configuration module run in the sender FPGA; DDR3 write control module, Ethernet send module run in the sender HPS.

[0008] As a further solution of the utility model: the receiving end SGDMA, read FIFO module, HDMI control module, and HDMI driver module run on the receiving end FPGA; the DDR3 read control module and Ethernet receiving module run on the receiving end HPS.

[0009] As a further solution of the utility model: the video resolution of the OV5640 image sensor is 1280×720@56fps, and the output color format is RGB565.

[0010] As a further solution of the utility model: the video transmission uses the UDP protocol to transmit data.

[0011] As a further solution of the utility model: the transmitting end FPGA and the receiving end FPGA both adopt CycloneV5CSEBA6U23I7 FPGA chip.

[0012] Compared with the prior art, the present invention uses FPGA and ARM to work together, collects video data through FPGA, and sends the collected data through HPS, which improves the quality of collected graphics while reducing transmission delay and ensuring the real-time performance of image transmission. The present invention avoids the cumbersome steps of implementing Ethernet transmission protocol in FPGA and reduces the complexity of the system. In addition to sending video data to the receiving end HPS, the present invention can also send data to the receiving end PC or send it remotely through the Internet. The device has high flexibility and is suitable for scenarios of remote video collection and transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 The utility model is a system structure block diagram of the video acquisition and transmission device.

[0014] Figure 2 This is the FPGA pin diagram of the transmitting end of the utility model.

[0015] Figure 3 This is the pin diagram of the OV5640 image sensor of the utility model.

[0016] Figure 4 This is a signal diagram of the FPGA connection at the sending end of the utility model.

[0017] Figure 5 This is a signal diagram of the HPS connection at the transmitting end of the utility model.

[0018] Figure 6 It is a signal diagram between the transmitting end SGDMA and the transmitting end DDR3 SDRAM of the utility model.

[0019] Figure 7 This is a schematic diagram of the Ethernet PHY chip of the utility model.

[0020] Figure 8 This is a receiving end FPGA connection signal diagram of the utility model.

[0021] Fig. 9 It is a schematic diagram of the HDMI display chip at the receiving end of the utility model. DETAILED DESCRIPTION

[0022] The utility model will be further described below in conjunction with the accompanying drawings.

[0023] In the embodiment of the utility model, both the transmitting end FPGA and the receiving end FPGA adopt CycloneV5CSEBA6U23I7FPGA chip, which integrates an ARM Cortex-A9 dual-core processor as a hard-core processing system (HPS). The HPS includes a DDR3 controller and a 1Gbps Ethernet port for communication and data exchange with the outside world.

[0024] like Figure 1 As shown, the video acquisition and transmission device includes a transmitting end and a receiving end. The transmitting end includes a video acquisition module. The video acquisition module receives a video signal from an OV5640 image sensor. The video acquisition module is connected to a data processing module. The video acquisition module transmits the acquired data to the data processing module for data merging. The data processing module transmits the processed data to a write FIFO module. The output end of the write FIFO module is connected to the transmitting end SGDMA; the transmitting end SGDMA is connected to a transmitting end DDR3 controller (HPS), and the transmitting end DDR3 controller is connected to the transmitting end DDR3 The output end of the DDR3 write control module is connected to the transmitting end SGDMA, and the transmitting end SGDMA is specified to write the data size to the DDR3 and start writing the address; the transmitting end DDR3 controller (HPS) outputs the data and sends it through the Ethernet transmitting module; the receiving end includes an Ethernet receiving module, and the DDR3 read control module is connected to the receiving end SGDMA, and the receiving end SGDMA is specified to read the data size and start reading the address; the output end of the receiving end SDGMA is connected to the read FIFO module, and the output end of the read FIFO module is connected to the HDMI control module, and the output end of the HDMI control module is connected to the HDMI driver module, and the output end of the HDMI driver module is connected to ADV7513, and the video data is read, and the HDMI display timing is generated. The output end of the ADV7513 is connected to the HDMI display, and the video data and the control signal are transmitted to the HDMI display chip, and the current pixel coordinates are fed back to the HDMI control module; the receiving end SGDMA is connected to the receiving end DDR3 controller (HPS); the output end of the Ethernet receiving module is connected to the receiving end DDR3 controller (HPS), and the Ethernet receiving module stores the data in the receiving end DDR3 controller (HPS) after receiving the data; the receiving end DDR3 controller is connected to the receiving end DDR3 SDRAM is connected.

[0025] The receiving end also includes a register configuration module, which performs parameter configuration on the OV5640 image sensor.

[0026] The video acquisition module, data processing module, write FIFO module, sender SGDMA, and register configuration module run in the sender FPGA; the DDR3 write control module and Ethernet send module run in the sender HPS.

[0027] The receiving end SGDMA, read FIFO module, HDMI control module, and HDMI driver module run on the receiving end FPGA; the DDR3 read control module and Ethernet receiving module run on the receiving end HPS.

[0028] The OV5640 image sensor video resolution is 1280×720@56fps, and the output color format is RGB565.

[0029] Video transmission uses the UDP protocol to transmit data. The UDP protocol is a connectionless protocol that sends data directly to the other party without establishing a connection with the other party. It has a high transmission rate and is suitable for real-time video transmission scenarios.

[0030] like Figure 2 As shown in the figure, taking the pin diagram of the transmitting end FPGA as an example, the pins of the OV5640 image sensor are connected to the transmitting end FPGA, which is responsible for collecting, processing and storing the video data output by the OV5640 image sensor. The HDMI output chip ADV7513 is connected to the receiving end FPGA, which is responsible for outputting the received video data to the ADV7513 according to the HDMI display format, and generating HDMI line synchronization and field synchronization signals according to the resolution to drive the ADV7513 chip. The FPGA periphery includes a clock generator and several voltage converters to generate the clock and circuit voltage required to drive the FPGA and HPS.

[0031] like Figure 3 As shown in the figure, the OV5640 image sensor contains a CMOS image sensor, two LED flashes and a clock generator. The flash is controlled by the FPGA on the sending end to configure the internal registers of the image sensor; the clock generator generates a 24MHz clock to provide a reference clock for the OV5640 image sensor. The FPGA on the sending end controls the SCL and SDA signals through the SCCB protocol timing to set the internal registers of the image sensor and determine the resolution, frame rate and clock multiplier parameters. The video data generated by the OV5640 image sensor is transmitted to the FPGA on the sending end through the data signal, and the camera_vsync and camera_href signals are generated to the FPGA on the sending end, so that the FPGA on the sending end completes the data acquisition according to the timing of the video data generation. Camera_pclk is the multiplied clock signal, which is output by the image sensor to the FPGA on the sending end, driving the video acquisition module on the FPGA on the sending end to acquire video data at the same rate.

[0032] like Figure 4As shown, in the transmitting end FPGA, the register configuration module configures the internal registers of OV5640 through the SCCB protocol when powered on to determine the resolution, frame rate, output pixel clock frequency, and output color format; after the configuration is completed, the video acquisition module collects the video data output by OV5640. In order to prevent the problem of unstable output video data after the configuration is completed, the module waits for OV5640 to output 5 frames of video data before collecting. The video acquisition module determines the timing of collecting video data based on the camera_vsync field synchronization and camera_href line synchronization signals. Figure 4 Among them, camera_power_down and camera_flash are the power saving mode and flash control signals of OV5640, which are controlled by the video acquisition module; the data processing module is responsible for receiving the video data collected by the video acquisition module, and merging the 8-bit pixel data into complete 16-bit pixel data according to the camera_vsync field synchronization and camera_href line synchronization signals output by the video acquisition module, and driving the write_data_valid and fifo_write signals at the same time to write the merged video data into the FIFO; the DDR3 read-write module, i.e., the sending end SGDMA, waits for the write FIFO to store enough video data for one frame, reads the video data in the FIFO, and writes the data to the sending end DDR3 controller.

[0033] like Figure 5As shown, the transmitting end DDR3 SDRAM, Ethernet PHY chip and SD card slot are connected to the transmitting end HPS. When the transmitting end HPS starts, the internal data of the SD card is read to start and configure the transmitting end FPGA, complete the system initialization, and finally run the Linux system on the transmitting end HPS to run the application. The transmitting end DDR3 write control module, Ethernet transmitting module and the receiving end Ethernet receiving module and DDR3 read control module are implemented in the form of application. When the transmitting end SGDMA is specified to write data to DDR3, the transmitting end DDR3 controller of the transmitting end HPS receives the data from the transmitting end SGDMA, drives the HPS_DDR3_BA and HPS_DDR3_ADDR signals, determines the address of the data write, and then writes the data from the address specified by the transmitting end SGDMA to DDR3 through HPS_DDR3_DQ. The data is stored in the form of 32 bits, the upper 16 bits are stored in the first DDR3, and the lower 16 bits are stored in the second DDR3. The read data is the same. During the startup phase of the transmitter HPS, the transmitter HPS drives the HPS_ENET_MDC and HPS_ENET_MDIO signals to set the state of the PHY chip. When data needs to be sent, the transmitter HPS drives the HPS_TX_DATA to send data and the HPS_TX_EN to send control signals to send data to the Ethernet PHY chip, and drives the HPS_ENET_GTX_CLK clock signal to provide a clock reference for the transmitter PHY chip. The receiver is the same as the transmitter.

[0034] like Figure 6 As shown, taking the transmitting end as an example, the DDR3 read / write module, i.e., the transmitting end SGDMA, exchanges data with the transmitting end DDR3 controller of the HPS through the transmitting end FPGAto HPSSDRAM bus. When writing data, the DDR3 read / write module determines the data start writing address by driving the hps_0_f2h_sdram_data_address signal. After specifying the write address, it drives hps_0_f2h_sdram0_data_readdata to send the video data to the transmitting end DDR3 controller, and finally pulls up the hps_0_f2h_sdram0_data_write signal to start transmission. The receiving end reads data in the DDR3 in the same way.

[0035] like Figure 7As shown in the figure, the internal registers of the Ethernet PHY chip can be written and read by the HPS by controlling HPS_ENET_MDC and HPS_ENET_MDIO. When the sending end HPS sends the video data to the Ethernet PHY chip, the PHY chip converts the data into 4 pairs of differential signals, drives the MDI_HPS_P and MDI_HPS_N signals to transmit the data to the Ethernet port, and then transmits the data through the network cable. At the same time, the LED2_DUAL_1 and LED2_DUAL_2 signals are driven according to the current connection rate. At 1000MBps, only LED2_DUAL_2 is driven, and at 100MBps, LED2_DUAL_1 and LED2_DUAL_2 signals are driven, so that the corresponding LED lights of the network port light up to indicate the current connection rate. The receiving end is the same as the sending end.

[0036] At the transmitting end, in order to control the DDR3 read-write module, i.e., the transmitting end SGDMA, to read and write data to DDR3, the transmitting end SGDMA internal control register address and the mapped DDR3 memory address are defined in the HPS. The mapped address can be operated to directly write data to the transmitting end SGDMA internal register or DDR3. When the FIFO has enough data for one frame, the transmitting end HPS specifies the starting address and the write data size of the transmitting end SGDMA to write to DDR3. After the specification is completed, the internal register starts writing 1, and the transmitting end SGDMA is started to write data to the transmitting end DDR3. When the transmitting end SGDMA has finished writing, the transmitting end HPS creates a socket, determines whether to use internal communication or network communication, and the type of transmission protocol. Then set the port number, the sending address, and the sending data size. After the specification is completed, the data can be sent to the receiving end. After the sending is completed, wait for the transmitting end SGDMA to write data to DDR3.

[0037] At the receiving end, the receiving HPS creates a socket, sets the port and the size of the received data, and waits for the sending end to send data. When the sending end sends data, the receiving HPS receives the data and stores the received data in DDR3 using address mapping. After storage, it continues to wait for the sending end to send data. When the data has been received and written into DDR3, the receiving HPS specifies the receiving SGDMA to read the starting address of DDR3 and the data size. After the specification is completed, it writes 1 to the start reading bit of the internal register, and the sending receiving SGDMA reads the data from DDR3 and writes the data into the FIFO.

[0038] like Figure 8As shown in the figure, when the system is running, the receiving end FPGA drives the hdmi_scl and hdmi_sda signals to configure the internal registers of the ADV7513 chip through the IIC protocol to determine the HDMI resolution and other parameters. The HDMI driver module uses the row counter and field counter to generate the HDMI timing, and transmits the current pixel position to the HDMI control module. When the row and field counter is within the HDMI effective display range, the HDMI control module reads the video data in the FIFO. After reading the video data, the HDMI driver module converts the image format and fills it in the current pixel position. At the same time, the HDMI driver module generates the color_R red component, color_G green component, color_B blue component, hdmi_vsync field synchronization, hdmi_href row synchronization, and hdmi_data_valid data valid signal according to the row and field counter to drive the HDMI display chip, and drives hdmi_tx_clk according to the resolution to provide a reference clock for ADV7513.

[0039] like Fig. 9 As shown in the figure, the ADV7513 chip is driven by the receiving FPGA, hdmi_tx_data16-23 corresponds to color_R, hdmi_tx_data8-15 corresponds to color_G, and hdmi_tx_data0-7 corresponds to color_B. After receiving the HDMI video output by the receiving FPGA, ADV7513 performs TMDS encoding on the video data to improve the stability of data transmission, and transmits the encoded data to the HDMI interface by driving the tmds_txc_p and tmds_txc_n differential clock signals, tmds_tx_p0-2 and tmds_tx_n0-2 differential data signals, and then outputs the video to the HDMI display through the HDMI interface.

[0040] The color format output by OV5640 is RGB565, that is, 5 bits for red component, 6 bits for green component, and 5 bits for blue component. The color format displayed by HDMI is RGB888, that is, 8 bits for red component, green component, and blue component respectively. RGB565 is converted to RGB888 by filling the low bits of the color components with 0.

Claims

1. A video acquisition and transmission device based on FPGA and ARM, including a transmitting end and a receiving end, characterized in that: The transmitting end includes a video acquisition module, which receives a video signal from an OV5640 image sensor. The video acquisition module is connected to a data processing module. The data processing module transmits the processed data to a write FIFO module. The output end of the write FIFO module is connected to a transmitting end DDR3 read / write module, i.e., a transmitting end SGDMA. The transmitting end SGDMA is connected to a transmitting end DDR3 controller (HPS). The transmitting end DDR3 controller is connected to a transmitting end DDR3 The output end of the DDR3 write control module is connected to the sending end SGDMA; the sending end DDR3 controller (HPS) outputs data and sends it through the Ethernet sending module; the receiving end includes an Ethernet receiving module, the DDR3 read control module is connected to the receiving end SGDMA, the receiving end SDGMA output end is connected to the read FIFO module, the output end of the read FIFO module is connected to the HDMI control module, the output end of the HDMI control module is connected to the HDMI driver module, the output end of the HDMI driver module is connected to ADV7513, the output end of the ADV7513 is connected to the HDMI display, the receiving end SGDMA is connected to the receiving end DDR3 controller (HPS); the output end of the Ethernet receiving module is connected to the receiving end DDR3 controller (HPS), and the receiving end DDR3 controller is connected to the receiving end DDR3 SDRAM.

2. The FPGA and ARM based video acquisition and transmission device according to claim 1, characterized in that: The receiving end also includes a register configuration module, which performs parameter configuration on the OV5640 image sensor.

3. The FPGA and ARM based video acquisition and transmission device according to claim 1 or 2, characterized in that: The video acquisition module, data processing module, write FIFO module, sender SGDMA, and register configuration module run in the sender FPGA; the DDR3 write control module and Ethernet send module run in the sender HPS.

4. The FPGA and ARM based video acquisition and transmission device according to claim 1, characterized in that: The receiving end SGDMA, read FIFO module, HDMI control module, and HDMI driver module run on the receiving end FPGA; the DDR3 read control module and Ethernet receiving module run on the receiving end HPS.

5. The FPGA and ARM based video acquisition and transmission device according to claim 1, characterized in that: The OV5640 image sensor has a video resolution of 1280×720@56fps and an output color format of RGB565.

6. The FPGA and ARM based video acquisition and transmission device according to claim 1, characterized in that: Video transmission uses UDP protocol to transmit data.

7. The FPGA and ARM based video acquisition and transmission device according to claim 1, characterized in that: The transmitting end FPGA and the receiving end FPGA both adopt Cyclone V5CSEBA6U23I7 FPGA chip.