FPGA-based Long-Distance Data Transmission Control and Control System and Method for Cameras
The FPGA-based camera long-distance data transmission and control system enables long-distance connection between the camera module and the controller, solving the problems of signal transmission obstruction and difficulty in fiber optic link synchronization recovery in existing technologies, and reducing installation complexity and cost.
Patent Information
- Application Number
- CN202611595703.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-15
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies cannot achieve long-distance connection between camera modules and controllers, resulting in increased installation size and weight. Furthermore, fiber optic transmission solutions suffer from problems such as obstructed control signal readback direction, difficulty in synchronous recovery after fiber optic link interruption, redundancy of fiber optic bundles for multiple cameras, and high costs.
A camera long-distance data transmission and control system based on FPGA is adopted. The FPGA realizes the optical fiber link transmission between the front-end processing unit and the back-end processing unit. Data frames and control response frames reuse the same optical fiber link. The control response frames have higher priority than data frames. Signal synchronization and conversion are realized through buffering mechanism and bidirectional closed-loop mechanism.
It achieves low-latency, uncompressed, long-distance, and interference-resistant co-fiber transmission of MIPI video data with IIC bus, SPI bus, and GPIO level signals, and supports multiple camera modules sharing a single fiber optic link, reducing installation complexity and cost.
Smart Images

Figure CN123123690A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of high-speed video optical fiber transmission, and particularly relates to a camera remote data transmission and control system and method based on FPGA (Field Programmable Gate Array). BACKGROUND
[0002] With the rapid development of machine vision, high-end security, industrial detection and vehicle-mounted vision fields, high-resolution image sensors have been widely applied. Taking a 4K (3840x2160) image sensor as an example, its MIPI CSI-2 (Mobile Industry Processor Interface Camera Serial Interface 2) interface usually adopts four-lane data transmission, and the single-lane data rate can reach 1.5Gbps, and the four-lane total is about 6.0Gbps. Such high-speed MIPI (Mobile Industry Processor Interface) bus signals are usually connected between boards by FPC soft wires with a length of 5-20cm, and cannot be transmitted remotely, resulting in that the camera and its controller must be installed in the same place, increasing the installation volume and weight, and being not conducive to the erection of the camera in remote supports, high rack positions and other places.
[0003] Meanwhile, the camera chip needs to complete register configuration through an IIC (Inter-Integrated Circuit) bus after power-on, and needs to be reset by a reset signal; these low-speed control signals are also subject to the constraints of close-range connection, and cannot be configured and operated remotely. In addition, the traditional copper cable and network cable transmission scheme (such as ordinary GigE and USB3) faces a bottleneck in transmission distance, cable weight and electromagnetic interference environment, and is difficult to meet the requirements of high-end equipment on image quality and reliability.
[0004] Optical fiber transmission has advantages such as high bandwidth, long distance, light weight, and resistance to electromagnetic interference, and is an effective way to solve the above problems. In the existing technology, there are schemes that convert video signals into optical signals for long-distance transmission. These schemes usually package video data into frames and send them through a high-speed serial interface to drive the optical module. The receiving end deframes the frames and restores the video. However, such schemes still have the following shortcomings: (1) For bidirectional signal lines such as IIC data lines, there is a lack of control means for the readback direction, and the readback of the contents and status of each chip in the camera module is blocked during the forwarding process; (2) There is a lack of synchronous recovery mechanism after the optical fiber link is interrupted and reconnected, which can easily lead to false transmission, misalignment, or abnormal picture; (3) For the multiple control line interfaces of complex camera modules, the camera controller control bundle is redundant; (4) When building a multi-channel camera, optical fibers are laid out one by one, resulting in long-distance optical fiber bundle redundancy and increased cost. Summary of the Invention
[0005] To address the aforementioned technical problems, this invention provides an FPGA-based long-distance data transmission and control system and method for cameras.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] In a first aspect, the present invention provides a camera long-distance data transmission and control system based on FPGA. The system includes a front-end processing unit, a back-end processing unit, and an optical fiber link connecting the front-end processing unit and the back-end processing unit. The front-end processing unit is connected to a camera module, and the back-end processing unit is connected to a camera controller. Both the front-end processing unit and the back-end processing unit are implemented by FPGA.
[0008] The transmitting end of the front-end processing unit is used to package the MIPI (Mobile Industry Processor Interface) video data from the camera module into data frames, and to package the data read back from the camera chip and the lens control chip into control response frames. The data frames and the control response frames are multiplexed and transmitted using the same optical fiber link, and the transmission priority of the control response frames is higher than that of the data frames. The IIC (Inter-Integrated Circuit) bus connects the camera chip and the front-end processing unit; the SPI (Serial Peripheral Interface) bus connects the lens control chip and the front-end processing unit; the front-end processing unit is connected to the camera module through the first MIPI interface.
[0009] The receiving end of the back-end processing unit is used to receive the data frame and restore it to a MIPI bus signal, and send the MIPI bus signal to the camera controller. It is also used to receive the control response frame and convert it into a serial port signal, and send the serial port signal to the camera controller. The back-end processing unit is connected to the camera controller through a second MIPI interface.
[0010] The back-end processing unit's transmitting end is used to receive control commands transmitted from the camera controller via the serial port, parse the control commands, and package the control commands into control command frames and transmit them via the optical fiber link;
[0011] The receiving end of the front-end processing unit is used to receive control command frames and directly convert them into corresponding IIC bus timing signals, SPI bus timing signals and GPIO (General Purpose Input / Output) level signals, which are then sent to the camera chip, lens control chip and GPIO level interface, respectively.
[0012] Secondly, the present invention provides a method for long-distance data transmission and control of a camera based on an FPGA, based on the above-mentioned long-distance data transmission and control system for a camera based on an FPGA.
[0013] The front-end processing unit's transmitting end packages the MIPI video data from the camera module into data frames and packages the data read back from the camera chip and lens control chip into control response frames. The data frames and control response frames are multiplexed and transmitted using the same optical fiber link, and the transmission priority of the control response frames is higher than that of the data frames. The IIC bus connects the camera chip and the front-end processing unit; the SPI bus connects the lens control chip and the front-end processing unit; the front-end processing unit is connected to the camera module through a first MIPI interface.
[0014] The receiving end of the back-end processing unit receives the data frame and restores it to a MIPI bus signal, and sends the MIPI bus signal to the camera controller. It is also used to receive the control response frame, convert it into a serial port signal, and send the serial port signal to the camera controller. The back-end processing unit is connected to the camera controller through a second MIPI interface.
[0015] The back-end processing unit receives the control command transmitted from the camera controller via the serial port, parses the control command, and packages the control command into a control command frame and sends it through the optical fiber link.
[0016] The front-end processing unit receives control command frames and directly converts them into corresponding IIC bus timing signals, SPI bus timing signals, and GPIO level signals, which are then sent to the camera chip, lens control chip, and GPIO level interface, respectively.
[0017] The beneficial effects of this invention are as follows:
[0018] (1) Priority arbitration and breakdown insertion of control response frames. The front-end processing unit is equipped with a first buffer and a second buffer for buffering data frames and control response frames respectively. The sending end first checks whether the second buffer is not empty in each frame. If it is not empty, the control response frame is sent first. If the second buffer is not empty during the data frame transmission, the control response frame is inserted and sent in the next frame until the second buffer is empty. Then the subsequent content of the data frame is resumed and sent. This makes the response delay of the control signal determined by the buffer status of the control response frame and is independent of the length of the data frame.
[0019] (2) Direct generation of bidirectional closed loop of control response frame and control command frame and bus timing. After the front-end processing unit executes the read command, it will form a control response frame from the read data and transmit it back to the back-end processing unit through the same optical fiber link. The back-end processing unit will convert it into a serial port signal for the camera controller to read, forming a closed loop of "IIC / SPI downlink command - front-end execution - read back uplink". After the front-end processing unit receives the control command frame, it will directly convert the command content carried in the frame into IIC bus timing, SPI bus timing and GPIO level signals. The front-end processing unit itself will act as the master controller of the IIC bus and SPI bus to generate bus timing and drive the camera chip, lens control chip and GPIO level interface.
[0020] (3) Single-frame composite of multiple control signals and single serial port access. Both the control command frame and the control response frame are composed of multiple words of a set word length. The address segment and data segment of GPIO, IIC and SPI control signals are distinguished by the sequence identifier byte in the word, so that GPIO reset, lens motor control and IIC and SPI read and write share the same control frame type; the back-end processing unit only presents one serial port interface to the camera controller. The FPGA completes the bidirectional conversion between serial port commands and IIC, SPI and GPIO timings. The camera controller does not need to change the original control flow.
[0021] (4) Resynchronization after fiber optic link interruption. The front-end processing unit and the back-end processing unit detect whether the fiber optic link is disconnected or lost. If the fiber optic link is disconnected, the front-end processing unit maintains the connection state with the camera module, and the back-end processing unit stops sending MIPI video data to the camera controller until it continuously receives the preset fiber fill signal. Then, it repositions the start position of the data frame or control frame by detecting the start of the frame symbol, and the two ends resume forwarding the correct frame format. When the fiber optic link is idle, it continuously sends the fiber fill signal to indicate the link connection status.
[0022] (5) Multiple front-end processing units can output optical signals of different wavelengths, which are combined into the same optical fiber link through optical fiber couplers and transmitted to the back-end processing unit. After being split, they are received by optical modules of different receiving wavelengths, realizing that multiple cameras can share a single optical fiber link through wavelength division multiplexing.
[0023] (6) The present invention realizes low-latency, uncompressed, long-distance, and anti-interference co-fiber transmission of MIPI video data with IIC bus timing signals, SPI bus timing signals and GPIO level signals. The camera controller is only configured with serial port commands to realize the control of IIC bus, SPI bus and GPIO level interfaces on the long-distance camera device, and supports multiple front-end processing units to share a single optical fiber link. Attached Figure Description
[0024] Figure 1 This is a schematic diagram of a camera long-distance data transmission and control system based on FPGA according to an embodiment of the present invention;
[0025] Figure 2 This is a schematic diagram of another FPGA-based long-distance camera data transmission and control system according to an embodiment of the present invention;
[0026] Figure 3 This is a data transmission flow diagram of the front-end processing unit according to an embodiment of the present invention;
[0027] Figure 4 This is a data transmission flow diagram of the back-end processing unit according to an embodiment of the present invention. Detailed Implementation
[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0029] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.
[0030] The terms “first”, “second”, etc., are used to distinguish similar objects, not to describe or indicate a specific order or sequence.
[0031] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent in such process, method, article, or apparatus / device.
[0032] Furthermore, to better illustrate the present invention, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that the present invention can be practiced without certain specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art have not been described in detail in order to highlight the spirit of the invention.
[0033] like Figure 1 As shown, the present invention provides a camera long-distance data transmission and control system based on FPGA. The system includes a front-end processing unit, a back-end processing unit, and an optical fiber link connecting the front-end processing unit and the back-end processing unit. The front-end processing unit is connected to a camera module, and the back-end processing unit is connected to a camera controller. Both the front-end processing unit and the back-end processing unit are implemented by FPGA.
[0034] The front-end processing unit's transmitting end is used to package the MIPI video data from the camera module into data frames, and to package the data read back from the camera chip and lens control chip into control response frames; the data frames and the control response frames are multiplexed and transmitted using the same optical fiber link, and the transmission priority of the control response frames is higher than that of the data frames; wherein, the IIC bus connects the camera chip and the front-end processing unit; the SPI bus connects the lens control chip and the front-end processing unit; the front-end processing unit is connected to the camera module through the first MIPI interface;
[0035] The receiving end of the back-end processing unit is used to receive the data frame and restore it to a MIPI bus signal, and send the MIPI bus signal to the camera controller. It is also used to receive the control response frame and convert it into a serial port signal, and send the serial port signal to the camera controller. The back-end processing unit is connected to the camera controller through a second MIPI interface.
[0036] The back-end processing unit's transmitting end is used to receive control commands transmitted from the camera controller via the serial port, parse the control commands, and package the control commands into control command frames and transmit them via the optical fiber link;
[0037] The receiving end of the front-end processing unit is used to receive control command frames and directly convert them into corresponding IIC bus timing signals, SPI bus timing signals and GPIO level signals, which are then sent to the camera chip, lens control chip and GPIO level interface respectively.
[0038] In some embodiments, multiple front-end processing units share a single optical fiber link; the multiple front-end processing units output optical channel data of different wavelengths through multiple transmitting optical modules, which are then combined by an optical fiber coupler and shared through a single optical fiber link, and transmitted to the receiving optical modules of multiple back-end processing units for reception, thereby achieving wavelength division multiplexing; the optical channel data includes data frames and control response frames.
[0039] For example, the system also includes an IIC / SPI / GPIO transceiver control module, which is located inside the front-end processing unit. Its input end receives control data from the receiving optical module after the serial optical signal is converted into parallel data. Its output end is connected to the second buffer and the IIC / SPI bus device, respectively. It is used to read the content of the control command frame to control the IIC / SPI bus device. When executing the read data command, it will form a control response frame from the read data and store it in the second buffer, which will be read and sent by the transmitting end. The MIPI receiving module and the MIPI transmitting module are located on the MIPI video data output path of the back-end processing unit. The input end of the MIPI receiving module is connected to the MIPI interface to receive the restored MIPI bus signal. Its output end is connected to the input end of the MIPI transmitting module. The output end of the MIPI transmitting module is connected to the FPC flexible cable to send the MIPI bus signal to the downstream camera controller. The two work together to complete the reception and forwarding of the MIPI bus signal.
[0040] For example, the system also includes a serial transceiver control module and a serial transceiver module. The serial transceiver control module is located inside the back-end processing unit. Its input end receives the control response frame separated from the serial optical signal after it has been converted into parallel data by the receiving optical module. Its output end is connected to a serial port to convert the control response frame into a corresponding serial port frame and feed it back to the camera controller. At the same time, the serial transceiver control module is also used to collect the read values of the IIC / SPI / GPIO of the remote camera module and encapsulate them into control command frames, thereby realizing bidirectional conversion between control response frames and serial port signals. The serial transceiver module is located on the optical fiber data transmission path between the front-end processing unit and the back-end processing unit. It includes a serial transceiver module transmitter located on the front-end side and a serial transceiver module receiver located on the back-end side. The front-end processing unit sends the framed data frame and control response frame into the optical fiber link through the serial transceiver module transmitter. The back-end processing unit receives the optical fiber data and deframes it through the serial transceiver module receiver, thus collaboratively completing the serialization and deserialization of data transmission.
[0041] like Figure 2As shown, this invention also provides a more specific structure for an FPGA-based long-distance camera data transmission and control system. This system includes a first camera module and a second camera module, a first front-end processing unit and a second front-end processing unit, an optical fiber coupler, a first back-end processing unit and a second back-end processing unit, and a first camera controller and a second camera controller. Both the first and second camera modules include a camera chip and a lens control chip. The first camera module is connected to the first front-end processing unit via MIPI, IIC interfaces, and SPI, GPIO level interfaces, respectively; similarly, the second camera module is connected to the second front-end processing unit via MIPI, IIC interfaces, and SPI, GPIO level interfaces. The outputs of both the first and second front-end processing units are connected to an optical fiber coupler. This optical fiber coupler enables long-distance transmission via an optical fiber link (e.g., within 10km for single-mode transmission; that is, when using a single-mode optical fiber link, the maximum stable transmission distance is approximately 10km). The output of the other optical fiber coupler is connected to both the first and second back-end processing units. The first back-end processing unit is connected to the first camera controller via a MIPI interface and a serial port; the second back-end processing unit is also connected to the second camera controller via a MIPI interface and a serial port. Both the first and second camera controllers internally include an image processing controller and a driver controller. The MIPI interface of the first back-end processing unit is connected to the image processing controller, and its serial port is connected to the driver controller; the MIPI interface of the second back-end processing unit is connected to the image processing controller, and its serial port is connected to the driver controller. In some embodiments, after the front-end processing unit receives the control command frame and converts it into the corresponding bus timing signal and GPIO level signal, if the control command frame is a read command, the front-end processing unit reads data from the camera chip or lens control chip according to the read command, assembles the read data into a control response frame, and transmits it back to the back-end processing unit via the same fiber optic link. The back-end processing unit converts the received control response frame into a serial port signal for the camera controller to read.
[0042] In some embodiments, the front-end processing unit sets up a first buffer for caching data frames and a second buffer for caching control response frames; the sending end of the front-end processing unit first determines whether the second buffer is not empty at each frame; if the second buffer is not empty, it prioritizes reading and sending the control response frames in the second buffer;
[0043] If the second buffer is not empty during the transmission of a data frame, the front-end processing unit will send a control response frame in the next frame until the second buffer is empty, and then resume the transmission of the data frame.
[0044] In some embodiments, both the control command frame and the control response frame consist of multiple words of a set word length. Each word includes a K-code byte, a sequence identifier byte, a data byte, and a polarity byte. The K-code byte is used to identify the control response frame type. The sequence identifier byte is used to identify the control response frame sequence. The value of the sequence identifier byte is used to divide the control response frame into a GPIO segment for transmitting reset signals, control signals, and GPIO signals, an IIC segment for transmitting IIC read / write addresses and IIC data, and an SPI segment for transmitting SPI read / write addresses and SPI data, so that GPIO signals, IIC signals, and SPI signals share the same control frame format. The data byte is used to carry data content. The polarity byte is used to identify positive and negative polarities. The control response frame and the corresponding control command frame use the same sequence identifier.
[0045] In some embodiments, the front-end processing unit's transmitting end packages the MIPI video data of the camera module into data frames, including: the MIPI hard core of the FPGA of the front-end processing unit converts the multi-channel serial data of the camera module into parallel data; when a start character appears in the byte bits of each channel, a MIPI video data packet is determined to have started, and subsequent data is written into the buffer; when an end character appears in the byte bits of each channel, the MIPI video data packet is determined to have ended, writing to the buffer stops, and the number of bytes of data in the current buffer is recorded as the length of this packet, while the total number of received MIPI video data packets is incremented by one as the packet count; when the packet count is not zero, it indicates that at least one complete MIPI video data packet has been received, the buffered data is read and a frame start character, frame header, redundancy check code, and frame end character are added sequentially to form the data frame, which is then scrambled by 8b / 10b encoding and sent out through the fiber optic interface; wherein, the frame header occupies multiple word spaces, where the first word is the frame cycle count, the frame cycle count starts from zero and increments by one for each frame of data sent, and the position of the frame header except for the first word is reserved;
[0046] The calculation range of the redundancy check code starts from the first word of the frame header and ends at the word before the frame end character. The data to be checked is divided into groups of 64 bits each, and each group is divided into a high 32-bit part and a low 32-bit part. The same 32-bit CRC check polynomial is used to calculate the two parts respectively to obtain two 32-bit check values. The check values of all groups are concatenated to form the redundancy check code.
[0047] In some embodiments, when the fiber optic link is idle, the front-end processing unit and the back-end processing unit continuously send fiber fill signals to indicate that the fiber optic link between them is in a connected state; when the front-end processing unit and the back-end processing unit detect that the fiber optic link is disconnected, the front-end processing unit maintains its connection with the camera module, and the back-end processing unit stops sending MIPI video data to the camera controller; until a preset fiber fill signal is received, the front-end processing unit and the back-end processing unit reposition the start position of the data frame or control frame by detecting the start-of-frame character, and resume forwarding the correct frame format; the control frame includes a control response frame and a control command frame;
[0048] In some embodiments, the back-end processing unit is connected to the camera controller via a serial port interface; the FPGA of the back-end processing unit is used to package the control commands sent by the camera controller via the serial port into control command frames, send them to the front-end processing unit via an optical fiber link, and convert the control response frames returned by the front-end processing unit via the optical fiber link into serial port signals and return them to the camera controller; the FPGA of the front-end processing unit is used to convert the received control command frames into corresponding IIC bus timing signals, SPI bus timing signals and GPIO level signals, and package the data read back from the camera chip and lens control chip into control response frames, send them to the back-end processing unit via an optical fiber link.
[0049] In some embodiments, the back-end processing unit packages the GPIO level control commands transmitted from the camera controller via the serial port into a control command frame and sends it out via the fiber optic link; after receiving the control command frame, the front-end processing unit outputs the GPIO level signals of each channel; when a readback is required, the front-end processing unit packages the status of each GPIO level signal into a control response frame and transmits it back via the same fiber optic link.
[0050] For example, such as Figure 3 The present invention also provides the data flow process of the front-end processing unit, taking the camera module using 4 LAN ports as an example:
[0051] (1) The FPGA of the front-end processing unit uses the MIPI hard core to convert serial and parallel data, converting the four-channel signal into 64-bit data content for buffering. The MIPI hard core can be a MIPI DPHY hard core.
[0052] (2) When the MIPI hard core acquires the start character (SoT) in all four channels, it starts storing data into the buffer. When the end character (EoT) field appears in all four byte bits, it is considered that a MIPI video data packet has been received and stops storing data into the buffer. The length of this packet is recorded and the packet count is incremented by 1.
[0053] (3) When the packet count is not 0, the packet data is packaged into a data frame (S_Frame), placed in the first buffer, and waited for the front-end processing unit to read and send it;
[0054] (4) The data received by the front-end processing unit enters the IIC / SPI / GPIO transceiver control module and reads the content of the control command frame to control the IIC / SPI bus device. When the IIC / SPI / GPIO transceiver control module receives the control command frame, it executes the corresponding read / write timing command. If it is a read data command, the read data will be used to form a control response frame and stored in the second buffer when the corresponding read data is executed. The front-end processing unit will then read and send the data.
[0055] (5) When the front-end processing unit determines that the second buffer is not empty, it reads and sends a control response frame (C_Frame) first, and then determines whether the first buffer is not empty before reading and sending it. The control response frame has the highest priority. That is, if the control response frame is not empty during the data frame transmission, the next 64b will send a control response frame until the second buffer is empty.
[0056] (6) The front-end processing unit sends the data frames or control response frames in the first and second buffers to the transmitting optical module, which converts the 64-bit parallel data into a serial optical signal and transmits it through the optical fiber; the receiving optical module of the back-end processing unit receives the optical signal and converts it into 64-bit parallel data for the back-end processing unit to parse.
[0057] (7) The back-end processing unit sends the control command frame to the receiving optical module, which converts the serial optical signal into 64-bit parallel data and transmits it to the front-end processing unit for the IIC / SPI / GPIO transceiver control module of the front-end processing unit to read.
[0058] For example, such as Figure 4 The present invention also provides the data flow process of the back-end processing unit, taking a 4xLAN MIPI channel as an example, specifically including:
[0059] (1) The back-end processing unit converts the serial data transmitted from the receiving optical module into parallel data. The received serial data is divided into control response frames and data frames. The data frames are stored in the data buffer, and the control response frames are entered into the serial port transceiver control module.
[0060] (2) When the data cache is detected to be non-empty, the read data is validated and the valid data is transferred to the corresponding cache;
[0061] (3) After detecting that the corresponding buffer is not empty, the MIPI transmitting module reads the valid data and sends it to the MIPI bus;
[0062] (4) After receiving the control response frame, the serial port transceiver control module reads the content of the control response frame and converts it into a frame of the corresponding serial port and feeds it back to the camera controller; the serial port transceiver control module is also used to collect the read values of the IIC / SPI / GPIO of the remote camera module and encapsulate them into control command frames.
[0063] (5) The control command frame is converted into a serial optical signal by the transmitting optical module and sent to the front-end processing unit via the optical fiber link for parsing and execution by the IIC / SPI / GPIO transceiver control module of the front-end processing unit.
[0064] Data frames are converted into 64-bit parallel data by the MIPI hard core inside the front-end processing unit via the MIPI 4xLAN channel interface. This 64-bit data is buffered upon receiving the start symbol until the end symbol is received, at which point the buffering stops, and the packet length is recorded, incrementing the packet count by 1. When the packet count is not zero, the buffered data is read as the data content based on the current forwarding packet length, and the data frame is packaged in the specified format (adding a start-of-frame code, frame header, redundancy check code, and end-of-frame code). It is then serially scrambled using 8b / 10b encoding and sent concurrently to the fiber optic link. The 32b ordered frame format is shown in Table 1.
[0065] Table 1
[0066]
[0067] In Table 1, IDLE (idle state) represents the idle padding state, used for padding when there is no valid data on the link; K28.5, D21.4, D21.5, D23.2, and D21.6 are 8b / 10b encoded characters (where K28.5 is a control character and Dxx are data characters), used for frame delimitation and idle padding; SOF is the start of frame character, indicating the start of a data frame; FH is the frame header field, containing the frame count (i.e., frame cycle count) and reserved fields; ST represents the start marker, corresponding to SoT (start of transport / stream marker), used to indicate the start of a line or data packet; PH is the packet header field, containing the data... ID (data identifier), WC0 / WC1 (word count low / high byte, used to indicate data length or channel information) and ECC (error correction code); PD is the data payload field, carrying the pixel data of each channel Data(ch0)~Data(ch3); PF is the partial check / packet end field, including P_CRC16 (partial CRC16 check value) and EoT (transmission / stream end marker); ET is the end marker field, corresponding to EoT; CRC (64b) is a 64-bit cyclic redundancy check code, covering the data from the frame header to the end of the frame; EOFn and EOFp are the frame end negative delimiter and frame end positive delimiter, respectively, used to identify the end boundary of the data frame; "Reserved" indicates that unused fields are reserved. The start-of-frame (FOB) code for the data frame is K28.5, D21.5, D23.2, D23.2; the frame header is the header portion of the data frame format, consisting of 6 words (the first word is the frame cycle counter, incrementing by 1 for each frame starting from 0), with the remaining bits reserved; the start character, header, data, trailer, and end character are the contents of the MIPI CSI-2 protocol long packet format (no data or trailer data segments when MIPI transmission is in short packet format); the redundancy check code is calculated by performing redundancy checks on the data frame from the first word of the header to the end character field, using two identical 32-bit polynomials:
[0068] ;
[0069] Here, X is a variable in the polynomial.
[0070] For example, the CRC checksums for the high 32 bits and low 32 bits are calculated separately; the end-of-frame (EOF) bit of the data frame is K28.5, D21.4, D21.6, D21.6 (negative polarity) or K28.5, D21.5, D21.6, D21.6 (positive polarity). The front-end processing unit packages the MIPI received data (without distinguishing between long and short MIPI packets) into data frames and sends them from the high-speed serial interface inside the front-end processing unit to the fiber optic link.
[0071] The back-end processing unit deframes the data frames received from the fiber optic link and verifies the redundancy check codes, then restores the valid content (i.e., MIPI video data) into a MIPI 4xLAN signal through the MIPI hard core.
[0072] For example, control command frames are used to transmit sensor reset, IIC, and SPI signals. The frame format is shown in Table 2: the frame consists of multiple 32-bit frame formats, each 32-bit can carry 8 bits of data content.
[0073] Table 2
[0074]
[0075] As shown in Table 2, byte 0 represents the K code of each control frame, byte 1 represents the sequence of control frames, byte 2 represents the data content (Data occupies 1 byte), and byte 3 represents the positive / negative polarity byte. ORDER represents control frames with different sequence numbers, and the meaning of each bit of Data for different sequence numbers is shown in Table 3.
[0076] Table 3
[0077]
[0078] In Table 3, ORDER0 is used to transmit GPIO signals such as RSTN reset, CTR_A, and CTR_B; RSTN is the camera reset signal (active low); CTR_A is the camera GPIO control signal A;
[0079] CTR_B is the camera GPIO control signal B;
[0080] GPIO4-11: GPIO type signals;
[0081] ORDER2-5: Used to transmit address and data information in the IIC read / write timing sequence. When the back-end processing unit receives IIC command data from the serial port, it assembles the packet and sends it to the front-end processing unit. (If the front-end processing unit receives the packet and it is a read IIC timing sequence, it will send the read byte group ORDER2-5 control response frame to the back-end processing unit.)
[0082] ORDER6-9: Used to transmit address and data information in the SPI write timing. When the back-end processing unit receives the SPI command data from the serial port, it assembles the data into a packet and sends it to the front-end processing unit. (If the front-end processing unit receives the data and it is a read SPI timing, it will assemble the read data into an ORDER6-9 control response frame and send it to the back-end processing unit.)
[0083] ×: Reserved bit;
[0084] The delays described in this section refer to the additional delays introduced by the fiber optic link and the logic of the front-end and back-end processing units, excluding the transmission time of the serial port commands themselves. During MIPI video data forwarding, the data sequentially passes through the MIPI receiving module, the MIPI transmitting module, the FPC flexible flat cable, the serial transceiver module receiver, the serial transceiver module transmitter, and the fiber optic link, as well as logical fiber optic data framing and deframing transmission. Actual measurements using the MIPI interface and fiber optic loopback mode show that the delay for the MIPI receiving module, MIPI transmitting module, and FPC flexible flat cable is 144.44 ns; the delay for the serial transceiver module receiver and transmitter is 240 ns; the logical framing delay for fiber optic data frames is 8102.22 ns; and the logical deframing delay for fiber optic data frames is 6364.44 ns. Furthermore, the fiber optic link delay increases by 5 μs / km with fiber length. In summary, the MIPI latency added by this solution is only 19.8511us (taking 1km as an example). When transmitting 4K@25Hz image data, the latency is no more than two lines of image data, which is less than one-thousandth of the display time of one frame of image. The latency of GPIO, IIC, and SPI control write commands, as well as frame assembly and deframe assembly, is only 80ns, the one-way latency is only 5.320us, and the round-trip latency of readback commands is only 10.56us, which can achieve real-time control of the front-end camera module.
[0085] In some other embodiments, an FPGA-based long-distance camera data transmission and control method is also proposed, based on the aforementioned FPGA-based long-distance camera data transmission and control system; specifically including:
[0086] The front-end processing unit's transmitting end packages the MIPI video data from the camera module into data frames and packages the data read back from the camera chip and lens control chip into control response frames. The data frames and control response frames are multiplexed and transmitted using the same optical fiber link, and the transmission priority of the control response frames is higher than that of the data frames. The IIC bus connects the camera chip and the front-end processing unit; the SPI bus connects the lens control chip and the front-end processing unit; the front-end processing unit is connected to the camera module through a first MIPI interface.
[0087] The receiving end of the back-end processing unit receives the data frame and restores it to a MIPI bus signal, and sends the MIPI bus signal to the camera controller. It is also used to receive the control response frame, convert it into a serial port signal, and send the serial port signal to the camera controller. The back-end processing unit is connected to the camera controller through a second MIPI interface.
[0088] The back-end processing unit receives the control command transmitted from the camera controller via the serial port, parses the control command, and packages the control command into a control command frame and sends it through the optical fiber link.
[0089] The front-end processing unit receives control command frames and directly converts them into corresponding IIC bus timing signals, SPI bus timing signals, and GPIO level signals, which are then sent to the camera chip, lens control chip, and GPIO level interface, respectively.
[0090] The contents not described in detail in this specification are existing technologies known to those skilled in the art.
[0091] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the present invention. The above description is exemplary and not exhaustive. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A camera long-distance data transmission and control system based on FPGA, characterized in that, The system includes a front-end processing unit, a back-end processing unit, and an optical fiber link connecting the front-end processing unit and the back-end processing unit; the front-end processing unit is connected to a camera module, and the back-end processing unit is connected to a camera controller; both the front-end processing unit and the back-end processing unit are implemented by FPGA. The front-end processing unit's transmitting end is used to package the MIPI video data from the camera module into data frames, and to package the data read back from the camera chip and lens control chip into control response frames; the data frames and the control response frames are multiplexed and transmitted using the same optical fiber link, and the transmission priority of the control response frames is higher than that of the data frames; wherein, the IIC bus connects the camera chip and the front-end processing unit; the SPI bus connects the lens control chip and the front-end processing unit; the front-end processing unit is connected to the camera module through the first MIPI interface; The receiving end of the back-end processing unit is used to receive the data frame and restore it to a MIPI bus signal, and send the MIPI bus signal to the camera controller. It is also used to receive the control response frame and convert it into a serial port signal, and send the serial port signal to the camera controller. The back-end processing unit is connected to the camera controller through a second MIPI interface. The back-end processing unit's transmitting end is used to receive control commands transmitted from the camera controller via the serial port, parse the control commands, and package the control commands into control command frames and transmit them via the optical fiber link; The receiving end of the front-end processing unit is used to receive control command frames and directly convert them into corresponding IIC bus timing signals, SPI bus timing signals and GPIO level signals, which are then sent to the camera chip, lens control chip and GPIO level interface respectively.
2. The FPGA-based long-distance camera data transmission and control system according to claim 1, characterized in that, After the front-end processing unit receives the control command frame and converts it into the corresponding bus timing signal and GPIO level signal, if the control command frame is a read command, the front-end processing unit reads data from the camera chip or lens control chip according to the read command, assembles the read data into a control response frame and transmits it back to the back-end processing unit through the same optical fiber link. The back-end processing unit converts the received control response frame into a serial port signal for the camera controller to read.
3. The FPGA-based long-distance camera data transmission and control system according to claim 2, characterized in that, The front-end processing unit is configured with a first buffer for caching data frames and a second buffer for caching control response frames. The transmitting end of the front-end processing unit first determines whether the second buffer is not empty at each frame. If the second buffer is not empty, the control response frames in the second buffer are read and sent first. If the second buffer is not empty during the transmission of a data frame, the front-end processing unit will send a control response frame in the next frame until the second buffer is empty, and then resume the transmission of the data frame.
4. The FPGA-based long-distance camera data transmission and control system according to claim 1, characterized in that, Both the control command frame and the control response frame are composed of multiple words of a set word length. Each word includes a K-code byte, a sequence identifier byte, a data byte, and a polarity byte. The K-code byte is used to identify the control response frame type; The sequence identifier byte is used to identify the control response frame sequence. The value of the sequence identifier byte is divided into a GPIO segment for transmitting reset signals, control signals and GPIO signals, an IIC segment for transmitting IIC read / write addresses and IIC data, and an SPI segment for transmitting SPI read / write addresses and SPI data, so that GPIO signals, IIC signals and SPI signals share the same control frame format. The data bytes are used to carry data content; The polarity byte is used to identify positive and negative polarity; The control response frame and the corresponding control command frame use the same sequence identifier.
5. The FPGA-based long-distance camera data transmission and control system according to claim 1, characterized in that, The front-end processing unit's transmitting end packages the MIPI video data from the camera module into data frames, including: The MIPI hard core of the FPGA in the front-end processing unit converts the multi-channel serial data of the camera module into parallel data; When the start character appears in the byte bits of each channel, a MIPI video data packet is determined to have started, and subsequent data is written to the buffer; when the end character appears in the byte bits of each channel, the MIPI video data packet is determined to have ended, writing to the buffer stops, and the number of bytes of data in the current buffer is recorded as the length of this packet. At the same time, the total number of MIPI video data packets received is incremented by one as the packet count. When the packet count is not zero, it indicates that at least one complete MIPI video data packet has been received. The buffered data is read and the start-of-frame character, frame header, redundancy check code and end-of-frame character are added in sequence to form the data frame. After being scrambled by 8b / 10b encoding, it is sent out through the fiber optic interface. The frame header occupies multiple word spaces, where the first word is the frame cycle counter. The frame cycle counter starts from zero and increments by one for each frame of data sent. The position of the frame header is reserved except for the position of the first word. The calculation range of the redundancy check code starts from the first character of the frame header and ends at the character before the end-of-frame character. The data to be verified is divided into groups of 64 bits each, and each group is further divided into a high 32-bit part and a low 32-bit part. The same 32-bit CRC check polynomial is used to calculate the two parts respectively to obtain two 32-bit check values. The check values of all groups are concatenated to form the redundancy check code.
6. The FPGA-based long-distance camera data transmission and control system according to claim 1, characterized in that, When the optical fiber link is idle, the front-end processing unit and the back-end processing unit continuously send optical fiber filling signals to indicate that the optical fiber link between them is in a connected state. When the front-end processing unit and the back-end processing unit detect that the optical fiber link is disconnected, the front-end processing unit maintains its connection with the camera module, and the back-end processing unit stops sending MIPI video data to the camera controller. Until a preset optical fiber fill signal is received, the front-end processing unit and the back-end processing unit reposition the start position of the data frame or control frame by detecting the start frame symbol, and resume forwarding the correct frame format. The control frame includes a control response frame and a control command frame.
7. The FPGA-based long-distance camera data transmission and control system according to claim 6, characterized in that, The back-end processing unit is connected to the camera controller via a serial port interface; The FPGA of the back-end processing unit is used to package the control commands sent by the camera controller via serial port into control command frames, send them to the front-end processing unit via fiber optic link, and convert the control response frames returned by the front-end processing unit via fiber optic link into serial port signals and return them to the camera controller. The FPGA of the front-end processing unit is used to convert the received control command frames into corresponding IIC bus timing signals, SPI bus timing signals and GPIO level signals, and to package the data read back from the camera chip and lens control chip into control response frames, which are then sent to the back-end processing unit via fiber optic link.
8. The FPGA-based long-distance camera data transmission and control system according to claim 7, characterized in that, Multiple front-end processing units share a single optical fiber link; multiple front-end processing units output optical channel data of different wavelengths through multiple transmitting optical modules, which are then combined by an optical fiber coupler and shared through a single optical fiber link, and transmitted to the receiving optical modules of multiple back-end processing units for reception, thereby achieving wavelength division multiplexing; the optical channel data includes data frames and control response frames.
9. The FPGA-based long-distance camera data transmission and control system according to claim 8, characterized in that, The back-end processing unit packages the GPIO level control commands transmitted from the camera controller via the serial port into control command frames and sends them out via the fiber optic link; after receiving the control command frame, the front-end processing unit outputs the GPIO level signals of each channel; when a readback is required, the front-end processing unit packages the status of each GPIO level signal into a control response frame and transmits it back via the same fiber optic link.
10. A method for long-distance data transmission and control of a camera based on FPGA, characterized in that, The FPGA-based camera long-distance data transmission and control system according to any one of claims 1-9; The front-end processing unit's transmitting end packages the MIPI video data from the camera module into data frames and packages the data read back from the camera chip and lens control chip into control response frames. The data frames and control response frames are multiplexed and transmitted using the same optical fiber link, and the transmission priority of the control response frames is higher than that of the data frames. The IIC bus connects the camera chip and the front-end processing unit; the SPI bus connects the lens control chip and the front-end processing unit; the front-end processing unit is connected to the camera module through a first MIPI interface. The receiving end of the back-end processing unit receives the data frame and restores it to a MIPI bus signal, and sends the MIPI bus signal to the camera controller. It is also used to receive the control response frame, convert it into a serial port signal, and send the serial port signal to the camera controller. The back-end processing unit is connected to the camera controller through a second MIPI interface. The back-end processing unit receives the control command transmitted from the camera controller via the serial port, parses the control command, and packages the control command into a control command frame and sends it through the optical fiber link. The front-end processing unit receives control command frames and directly converts them into corresponding IIC bus timing signals, SPI bus timing signals, and GPIO level signals, which are then sent to the camera chip, lens control chip, and GPIO level interface, respectively.
