A serial data deserialization and synchronization system and method for high-speed imaging

Abstract

The present invention relates to an LVDS interface data deserialization and synchronization system and method, and specifically to a serial data deserialization and synchronization system and method for high-speed imaging, which solves the technical problem that the existing LVDS IP core is difficult to realize LVDS data deserialization with a data bit width greater than 10 bits, independent byte alignment of each data channel, and synchronization between data of each data channel. The serial data deserialization and synchronization system for high-speed imaging includes a clock and reset management module, a deserialization module, a bit width conversion module, a big-end and small-end and data splicing module, a data byte alignment and shift module, a data parsing module, and a data reorganization module; it can realize the deserialization and data synchronization of multi-data channel LVDS output data input to the deserialization module from an image sensor with a maximum image pixel-level data bit width of 20 bits; the serial data deserialization and synchronization method for high-speed imaging provided by the present invention realizes the independent byte synchronization function for a single data channel.

Description

Technical Field

[0001] The invention relates to an LVDS interface data deserialization and synchronization system and method, and in particular to a serial data deserialization and synchronization system and method for high-speed imaging. Background Art

[0002] With the development of acquisition and sensing technology, high-speed transmission and real-time processing have become the inevitable trend of signal acquisition system development. LVDS (low voltage differential signal) interface technology adopts low voltage differential signal transmission, and is widely used in high-speed imaging systems for its fast speed, low bit error rate and low power consumption. In order to realize the transmission of LVDS data, an LVDS transmitter is required at the data transmitting end, and an LVDS receiver is required at the data receiving end, and the two need to be used in coordination. In the image acquisition system, if the transmission of the LVDS interface is to be realized, the general image sensor has an LVDS transmitter. When the receiving end wants to restore the original image data, it is necessary to perform deserialization and data synchronization operations. Most of the current image acquisition systems adopt the solution of CMOS image sensor + FPGA + DDR data storage. Among them, the CMOS image sensor completes the acquisition and transmission of image data, the FPGA receives, decodes and synchronizes the high-speed image data, and realizes the reorganization and caching of the image data, and finally transmits it to the host computer for display or processing through the high-speed image data interface (such as Gigabit Ethernet).

[0003] Since the transmitter and receiver of LVDS can work at different times. When the receiver receives the first bit of a byte, it is not necessarily the first bit of the byte sent by LVDS, so the data needs to be synchronized. Usually, the transmitter will continuously send a training code before sending valid data. The receiver determines whether the correct data is received based on the received training code. If the received training code is equal to the value agreed upon by both parties, it is considered that the byte synchronization is achieved. Otherwise, a shift operation is performed and then compared again until the two are equal. In addition, due to the influence of layout and wiring during high-speed transmission of image data, there may be a phase difference between the data channels. At this time, the data between the data channels also need to be synchronized.

[0004] There are two common solutions for LVDS interface data deserialization and synchronization. One is to use the soft core provided by Xilinx. The other is to use the LVDS IP core provided by Intel. The system block diagram of the LVDS IP core is shown in Figure 1. Figure 1As shown, the LVDS IP core implements the deserialization and shifting operation of LVDS high-speed data. The function of the byte alignment control module is to compare the received data with the training code. If the comparison result is that the two are not equal, the bit_align instruction is issued. After the LVDS IP core receives the instruction, it performs a shift operation on the serial data. The byte alignment control module receives the shifted data and re-compares it until the received data is equal to the training code, then the comparison is completed, and finally a training completion flag is issued, so that the deserialization and byte alignment operation of the LVDS data are realized. Then, the data passes through the data parsing and reorganization module to complete the extraction and data reorganization of the image valid data, and finally outputs the image pixel-level data. The disadvantage of this solution is that it can only realize the deserialization of data with a bit width of no more than 10, and cannot realize independent byte alignment of each data channel. In high-resolution image acquisition systems, the data bit width of the image sensor is often required to be greater than 10 bits, so it is necessary to improve this solution to meet the deserialization of data greater than 10 bits.

[0005] In addition, with the increase in the number of data channels, the increase in the rate of the LVDS interface, and the influence of layout and wiring, the data output by the image sensor has a phase difference between the data channels after passing through the LVDS interface. It is no longer possible to achieve correct data collection by relying solely on the byte alignment of one data channel, and it is necessary to perform byte alignment operations on each data channel separately. Therefore, it is necessary to remove the byte alignment module inside the LVDS I core from the IP and perform byte alignment operations on each data channel separately.

[0006] In order to achieve data deserialization and synchronization of the LVDS interface and meet the requirements of high-speed and high-resolution applications, the above-mentioned problems need to be solved. Summary of the invention

[0007] The purpose of the present invention is to provide a serial data deserialization and synchronization system and method for high-speed imaging to address the technical problems that the existing LVDS IP core is difficult to achieve LVDS data deserialization with a data width greater than 10 bits, independent byte alignment of each data channel, and synchronization between data channels.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0009] A serial data deserialization and synchronization system for high-speed imaging, which is special in that it includes a clock and reset management module, a deserialization module, a bit width conversion module, a big-end and small-end and data splicing module, a data byte alignment and shift module, a data parsing module and a data reorganization module;

[0010] A clock and reset management module, which is used to connect the system clock signal and the system reset signal, and its output end is respectively connected to the clock signal and the reset signal corresponding to the input end of the deserialization module, the bit width conversion module, the data byte alignment and shift module, the data analysis module and the data reorganization module;

[0011] The input end of the deserialization module is used to receive N serial data generated by the external image sensor, and perform deserialization and bit alignment processing on the data; the data bit width after deserialization is P*N, where P is the serialization factor, and its value range is [1:10]; the output end of the deserialization module is connected to the input end of the bit width conversion module;

[0012] The bit width conversion module is used to convert data with a data bit width of P*N into data with a data bit width of M*N, where M is the pixel-level data bit width, wherein the value range of M is [1:20], and the value range of N is [1:18]; the output end of the bit width conversion module is connected to the big and small ends and the input end of the data splicing module;

[0013] The big and small end and data splicing module is used to rearrange the data output by the bit width conversion module in chronological order, and its output end is connected to the input end of the data byte alignment and shift module;

[0014] The data byte alignment and shift module decouples the big and small ends from the output data of the data splicing module, and realizes the data shift and byte alignment of each data channel of the deserialization module; its output end is respectively connected to the data parsing module and the data reassembly module;

[0015] The data analysis module is used to extract the frame valid signal and the line valid signal of the image data and send them to the data reorganization module;

[0016] The data reorganization module is used to realize synchronization and line-sequential output of image data according to the frame valid signal and line valid signal of the image data, and synchronously output the valid enable signal of the image data.

[0017] Further, the data byte alignment and shift module includes N byte alignment and shift units;

[0018] The data reassembly module is implemented using N FIFOs.

[0019] Furthermore, the deserialization module adopts an LVDS IP core; and the bit width conversion module adopts FIFO.

[0020] In addition, the present invention also provides a serial data deserialization and synchronization method for high-speed imaging, which is special in that it includes the following steps:

[0021] Step 1: Determine whether the image pixel-level data bit width M is greater than 10 bits;

[0022] If the image pixel-level data bit width M is greater than 10 bits, the serialization factor P = M / 2, and the N-channel serial data of the image sensor is input into the LVDS IP core, and the first deserialized data with a bit width of P*N is obtained after deserialization, and step 2 is executed;

[0023] If the image pixel-level data bit width M is less than or equal to 10 bits, the serialization factor P=M, and the N-channel serial data is input into the LVDS IP core, and after deserialization, the deserialized data with a bit width of P*N is obtained, and step 4 is executed;

[0024] Step 2: input the first deserialized data with a bit width of P*N into the bit width conversion module, and after conversion by the bit width conversion module, obtain the second deserialized data with a bit width of M*N;

[0025] Step 3: Input the second deserialized data into the big-endian and small-endian and data splicing module to implement big-endian and small-endian conversion and splicing of the second deserialized data, and obtain the third deserialized data with a bit width of M*N;

[0026] Step 4: Send the deserialized data or the third deserialized data to the data byte alignment and shift module, perform byte synchronization and shift operations on the deserialized data or the third deserialized data, and obtain N channels of image data with a bit width of M;

[0027] Step 5: Send the N-channel image data to the data analysis module, and use the data analysis module to extract the frame valid signal and the line valid signal of the N-channel image data;

[0028] Step 6: Extract the valid signals of N channels of image data through the frame valid signal and the row valid signal, and use the data reorganization module to cache and process the valid signals of the image data to achieve synchronous and sequential output of the image data.

[0029] Furthermore, in step 2, the bit width conversion module uses FIFO, and the conversion is specifically as follows:

[0030] 2.1) When the FIFO is not full, the first deserialized data with a bit width of P*N is written into the FIFO;

[0031] 2.2) Determine whether there is written data in the FIFO. If so, perform a FIFO read operation with a read data width of M*N. Otherwise, perform a wait operation.

[0032] Furthermore, step three is specifically as follows:

[0033] 3.1) Perform big-endian conversion

[0034] Swap the upper M*N / 2 bits and the lower M*N / 2 bits of the second deserialized data to obtain swapped data data_out[M*N-1:0]={data[M*N / 2-1:0],data[M*N-1:M*N / 2]};

[0035] 3.2 Splicing

[0036] Arrange the swapped data in chronological order, with each M bits of swapped data as a group, to obtain N groups of third deserialized data.

[0037] Further, in step 4, the byte synchronization and shifting operations of the deserialized data or the third deserialized data are implemented by using a state machine;

[0038] Step 4 is as follows:

[0039] 4.1) In IDLE state, determine whether the state machine meets the state transition condition

[0040] After the image sensor is powered on and its registers are configured, the counter lcnt of the state machine starts counting; when the counting time reaches the preset time required by the image sensor, step 4.2) is executed;

[0041] 4.2) Training in COMPARE state:

[0042] Compare the deserialized data or the third deserialized data with the training code to see if they are equal. If they are not equal, add 1 to the counter cnt and execute step 4.3);

[0043] If they are equal, proceed to step 4.4);

[0044] 4.3) In the SHIFT state, shift according to the count value of counter cnt, the shift width is equal to the count value, and the count value range is [0, M-1]; after the shift is completed, return to step 4.2);

[0045] 4.4) In the BITSLIP_DONE state, use the counter cnt_bitslip to count:

[0046] If the count value of the counter cnt_bitslip is greater than the preset value, the byte alignment is completed, and the image data with a bit width of M is output, and the loop ends; the preset value is a positive integer greater than or equal to 1;

[0047] If the count value of the counter cnt_bitslip is less than or equal to the preset value, return to step 4.2).

[0048] Further, in step 4.3), the shifting according to the count value of the counter cnt is specifically:

[0049] 4.3.1. Delay the deserialized data or the third deserialized data by at least one clock signal cycle to obtain data1[M-1,0]; and delay data1[M-1,0] by one clock signal cycle to obtain data2[M-1:0];

[0050] 4.3.2. Left Shift

[0051] If you need to shift one bit, shift data2[M-1:0] left by one bit, and move the highest bit of data1[M-1:0] to the lowest bit of data2[M-1:0]; and so on, if you need to shift M-1 bits, shift data2[M-1:0] left by M-1 bits, and move the high M-1 bits of data1[M-1:0] to the low M-1 bits of data2[M-1:0];

[0052] Or, shift right

[0053] If one bit needs to be shifted, data1[M-1,0] is shifted right by 1 bit, and the lowest bit of data2[M-1:0] is moved to the highest bit of data1[M-1:0]; and so on. If M-1 bits need to be shifted, data1[M-1,0] is shifted right by M-1 bits, and the lower M-1 bits of data2[M-1:0] are moved to the higher M-1 bits of data1[M-1:0].

[0054] Furthermore, the preset time in step 4.1) is the shortest time between the completion of register configuration and the start of training;

[0055] The default value in step 4.4) is 4.

[0056] Furthermore, in step six, a state machine is used to realize synchronization and sequential output of image data;

[0057] Step 6 is as follows:

[0058] 6.1) Set the number of FIFOs to be the same as the number of LVDS interfaces output by the external image sensor;

[0059] 6.2) Reset the FIFO;

[0060] 6.3) When a frame valid signal is received, the image data is controlled to be written into the FIFOs according to the N-way row valid signals, and the N FIFOs are read out sequentially according to the amount of data written into the FIFOs;

[0061] The image data is output in row order. When a frame of data is read and written, the state machine enters the initial state and waits for the next frame valid signal. The read clock signal frequency of the N FIFOs is greater than or equal to N times the clock signal frequency of the write FIFO.

[0062] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0063] (1) The serial data deserialization and synchronization system for high-speed imaging of the present invention utilizes a bit width conversion module, a big-endian and small-endian data splicing module, and a data byte alignment and shift module to transplant the data synchronization function from the inside of Intel's LVDS IP core to the outside thereof, and is used in conjunction with the LVDS IP core, thereby expanding the application scope of the LVDS IP core.

[0064] (2) The serial data deserialization and synchronization method for high-speed imaging of the present invention can realize the deserialization and data synchronization of multi-data channel LVDS output data input to the deserialization module from an image sensor with a maximum image pixel-level data width of 20 bits.

[0065] (3) The present invention realizes the independent byte synchronization function for a single data channel through the bit width conversion module, the big and small end and data splicing module and the data byte alignment, and the bit width conversion module, the big and small end and data splicing module, the data byte alignment and shift module and the data reorganization module are used in conjunction to realize the pixel-level output of the image data.

[0066] (4) The present invention utilizes a data reorganization module to achieve data synchronization between data channels.

[0067] (5) The clock and reset management module of the present invention is used to generate a reset signal and a clock signal, wherein the clock signal is used to drive the data, and the reset signal is used to reset the data to meet the timing requirements.

[0068] (6) The serial data deserialization and synchronization system for high-speed imaging of the present invention has the characteristics of high cost performance and short development cycle, and can be widely used in image acquisition systems with Intel FPGA as the hardware platform, and is particularly suitable for high-resolution applications, and has good economic benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 The figure is a schematic diagram of an existing LVDS IP-based data deserialization and synchronization system.

[0070] Figure 2 Schematic diagram of the serial data deserialization and synchronization system for high-speed imaging according to the present invention.

[0071] Figure 3Schematic diagram of a state machine for implementing data shifting and byte alignment in an embodiment of the present invention.

[0072] Figure 4 This is a schematic diagram of the corresponding relationship between each data channel and the data byte alignment and shift module in an embodiment of the present invention;

[0073] Figure 5 Schematic diagram of the state transition of the read control part of the data reassembly module FIFO in an embodiment of the present invention. DETAILED DESCRIPTION

[0074] A serial data deserialization and synchronization system for high-speed imaging, comprising a clock and reset management module, a deserialization module, a bit width conversion module, a big-end and small-end and data splicing module, a data byte alignment and shift module, a data parsing module and a data reorganization module;

[0075] A clock and reset management module, which is used to connect the system clock signal and the system reset signal, and its output end is respectively connected to the clock signal and the reset signal corresponding to the input end of the deserialization module, the bit width conversion module, the data byte alignment and shift module, the data analysis module and the data reorganization module;

[0076] The input end of the deserialization module is used to receive N serial data generated by the external image sensor, and perform deserialization and bit alignment processing on the data; the bit width of the deserialized data is P*N, where P is the serialization factor, and its value range is [1:10]; the output end of the deserialization module is connected to the input end of the bit width conversion module;

[0077] The bit width conversion module is used to convert data with a data bit width of P*N into data with a data bit width of M*N, where M is the pixel-level data bit width, wherein the value range of M is [1:20], and the value range of N is [1:18]; the output end and the big and small ends of the bit width conversion module are connected to the input end of the data splicing module;

[0078] The big and small end and data splicing module is used to rearrange the data output by the bit width conversion module in chronological order, and its output end is connected to the input end of the data byte alignment and shift module;

[0079] The data byte alignment and shift module decouples the big and small ends from the output data of the data splicing module, and realizes the data shift and byte alignment of each data channel of the deserialization module; its output end is respectively connected to the data parsing module and the data reassembly module;

[0080] The data analysis module is used to extract the frame valid signal and the line valid signal of the image data and send them to the data reorganization module;

[0081] The data reorganization module is used to realize synchronization and line-sequential output of image data according to the frame valid signal and line valid signal of the image data, and synchronously output the valid enable signal of the image data.

[0082] The data byte alignment and shift module includes N byte alignment and shift units; the number of the byte alignment and shift units is the same as the number of data channels of the LVDS differential data at the output end of the image sensor;

[0083] The data reassembly module is implemented with N FIFOs. The deserialization module uses the LVDS IP core; the bit width conversion module uses FIFO.

[0084] In addition, the present invention also provides a serial data deserialization and synchronization method for high-speed imaging, comprising the following steps:

[0085] Step 1: Determine whether the image pixel-level data bit width M is greater than 10 bits:

[0086] If the image pixel-level data bit width M is greater than 10 bits, the serialization factor P = M / 2, and the N-channel serial data of the image sensor is input into the LVDS IP core, and the first deserialized data with a bit width of P*N is obtained after deserialization, and step 2 is executed;

[0087] If the image pixel-level data bit width M is less than or equal to 10 bits, the serialization factor P=M, and the N-channel serial data is input into the LVDS IP core, and after deserialization, the deserialized data with a bit width of P*N is obtained, and step 4 is executed;

[0088] Step 2: Input the first deserialized data with a bit width of P*N into the bit width conversion module, and after conversion by the bit width conversion module, obtain the second deserialized data with a bit width of M*N; the bit width conversion module adopts FIFO, and its conversion is specifically as follows:

[0089] 2.1) When the FIFO is not full, the first deserialized data with a bit width of P*N is written into the FIFO;

[0090] 2.2) Determine whether there is written data in the FIFO. If so, perform a FIFO read operation with a read data width of M*N. Otherwise, perform a wait operation.

[0091] Step 3: Input the second deserialized data into the big-endian and small-endian data splicing module to implement big-endian and small-endian conversion and splicing of the second deserialized data, and obtain the third deserialized data with a bit width of M*N;

[0092] 3.1) Perform big-endian conversion

[0093] Swap the upper M*N / 2 bits and the lower M*N / 2 bits of the second deserialized data to obtain swapped data data_out[M*N-1:0]={data[M*N / 2-1:0],data[M*N-1:M*N / 2]};

[0094] 3.2 Splicing

[0095] Arrange the swapped data in chronological order, with each M bits of swapped data as a group, to obtain N groups of third deserialized data.

[0096] In step 4, the byte synchronization and shifting operations of the deserialized data or the third deserialized data are implemented by using a state machine;

[0097] Step 4: Send the deserialized data or the third deserialized data to the data byte alignment and shift module, perform byte synchronization and shift operations on the deserialized data or the third deserialized data, and obtain N channels of image data with a bit width of M;

[0098] 4.1) Determine whether the state machine meets the state transition condition in the IDLE state

[0099] If the image sensor is powered on and its registers are configured, the counter lcnt of the state machine starts counting; when the counting time is less than the preset time required by the image sensor, it remains in the IDLE state; the preset time is the shortest time between the completion of register configuration and the start of training;

[0100] Otherwise, go to step 4.2);

[0101] 4.2) Training in COMPARE state

[0102] Compare the deserialized data or the third deserialized data with the training code to see if they are equal. If not, add 1 to the counter cnt and execute step 4.3);

[0103] If they are equal, proceed to step 4.4);

[0104] 4.3) In the SHIFT state, shift according to the count value of counter cnt, the shift width is equal to the count value, and the count value range is [0, M-1]; after the shift is completed, return to step 4.2);

[0105] The specific shifting according to the count value of counter cnt is:

[0106] 4.3.1. Delay the deserialized data or the third deserialized data by at least one clock signal cycle to obtain data1[M-1,0]; and delay data1[M-1,0] by one clock signal cycle to obtain data2[M-1:0];

[0107] 4.3.2. Left Shift

[0108] If you need to shift one bit, shift data2[M-1:0] left by one bit, and move the highest bit of data1[M-1:0] to the lowest bit of data2[M-1:0]; and so on, if you need to shift M-1 bits, shift data2[M-1:0] left by M-1 bits, and move the high M-1 bits of data1[M-1:0] to the low M-1 bits of data2[M-1:0];

[0109] Or, shift right

[0110] If one bit needs to be shifted, data1[M-1,0] is shifted right by 1 bit, and the lowest bit of data2[M-1:0] is moved to the highest bit of data1[M-1:0]; and so on. If M-1 bits need to be shifted, data1[M-1,0] is shifted right by M-1 bits, and the lower M-1 bits of data2[M-1:0] are moved to the higher M-1 bits of data1[M-1:0].

[0111] 4.4) In the BITSLIP_DONE state, use the counter cnt_bitslip to count

[0112] If the count value of the counter cnt_bitslip is greater than the preset value, which is a positive integer greater than or equal to 1, the byte alignment is completed, and the image data with a bit width of M is output, and the loop ends;

[0113] If the count value of the counter cnt_bitslip is less than or equal to the preset value, return to step 4.2).

[0114] Step 5: Send the N-channel image data to the data analysis module, use the data analysis module to extract the frame valid signal and the line valid signal of the N-channel image data, and extract the valid signal of the N-channel image data through the frame valid signal and the line valid signal;

[0115] Step 6: Use the data reorganization module to cache and process the valid signal to achieve synchronization and sequential output of image data. The data reorganization module uses a state machine;

[0116] 6.1) Set the number of FIFOs to be the same as the number of LVDS interfaces output by the external image sensor;

[0117] 6.2) Reset the FIFO;

[0118] 6.3) When a frame valid signal is received, the image data is controlled to be written into the FIFOs according to the N-way row valid signals, and the N FIFOs are read out sequentially according to the amount of data written into the FIFOs; the image data is output sequentially in row order; when a frame of data is read and written, the state machine enters the initial state and waits for the next frame valid signal, wherein the read clock signal frequency of the N FIFOs is greater than or equal to N times the clock signal frequency of the write FIFO.

[0119] The present invention is used for the serial data deserialization and synchronization system of high-speed imaging. Figure 2 As shown, the clock and reset management module in this embodiment provides clock signals and reset signals for the entire system, wherein the reset signal rst_n is used to reset the deserialization module, the bit width conversion module, the data byte alignment and shift module, the data parsing module and the data reassembly module used in the system. The clk_out1 signal is used as the clock signal of the bit width conversion module, the data byte alignment and shift module, the data parsing module and the data reassembly module, and the clk_out2 is used as another clock signal of the data reassembly module.

[0120] The deserialization module uses the LVDS IP core to realize the deserialization of the LVDS serial differential data in each data channel input by the external image sensor. The so-called deserialization is to convert the LVDS serial differential data into parallel data. When the LVDS serial differential data bit width is not greater than 10 bits, the serialization factor P is equal to the image pixel-level data bit width M, otherwise, the serialization factor P is equal to half of the image pixel-level data bit width M.

[0121] Bit width conversion module: This is achieved through the read and write control of FIFO; the control of FIFO read and write ports adopts the strategy of reading when not empty and writing when not full. The bit width conversion module adjusts the bit width of the deserialized data from P*N to M*N to parse the image data. The bit width of the output image data is equal to twice the bit width of the input image data, and the output clock signal frequency is 1 / 2 of the input clock signal frequency.

[0122] Endian conversion and data splicing module: After the image data passes through the FIFO and completes the bit width conversion, it needs to be swapped to maintain the high and low bit order of the image data. To restore the original image data, the image data needs to be spliced. The image data of the same data channel is arranged in the order of high and low bits.

[0123] Data byte alignment and shift module: used to realize the shift and byte alignment of image data. Contains N byte alignment and shift units. The image data of one data channel corresponds to one byte alignment and shift unit. The byte synchronization function originally realized by the LVDS IP core is transferred to be realized by an external module, thereby realizing the synchronization of the image data of each data channel, decoupling the shift of the image data of a single data channel from the shift of the entire deserialized image data, and increasing the flexibility of application. The serial data deserialization and synchronization system for high-speed imaging of the present invention does not rely on the hardware platform. When the image data is parallel image data of multiple data channels after deserialization by the LVDS IP core, the data byte alignment and shift unit can be used to realize byte synchronization.

[0124] Data analysis module: realizes the extraction of image frame valid signals and line valid signals. It only extracts frame valid signals and line valid signals, but does not process image data.

[0125] Data reorganization module: realizes the synchronization of image data between each data channel; converts the image data from the interface clock signal domain to the pixel clock signal domain, and uses the FIFO cache function to realize the synchronization of image data of each data channel. The number of FIFOs is the same as the number of LVDS differential data channels, and has nothing to do with the image size. The input bit width of the FIFO is the same as the output bit width, and is the same as the image pixel-level data bit width. The output clock signal frequency of the FIFO is the input clock signal frequency multiplied by the number of data channels.

[0126] In this embodiment, the main signals and connection relationships used by each module are as follows:

[0127] The input signals of the clock and reset management module include the system's external clock signal sys_clk and external reset signal sys_rst_n, and the output signals include the reset signal rst_n and two clock signals clk_out1 and clk_out2. The reset signal rst_n is connected to the reset pin of the bit width conversion module, the reset pin of the data byte alignment and shift module, the reset pin of the data analysis module, and the reset pin of the data reassembly module. The reset signal rst_n is inverted and connected to the reset pin pll_areset of the deserialization module.

[0128] clk_out1 is respectively connected to the bit width conversion module rdclk, the data byte alignment and shift module clk2, the data analysis module clk3 and the data reorganization module clk4, and clk_out2 is connected to the data reorganization module clk5.

[0129] The input signals used in the deserialization module include N-channel LVDS differential data rx_in[N-1:0], clock signal rx_inclock, and reset signal pll_areset. Among them, the N-channel LVDS differential data rx_in[N-1:0] and clock signal rx_inclock come from the external image sensor. The reset signal pll_areset comes from the clock and reset management module. The output signal of the LVDS IP core includes the first deserialized data rx_out[P*N-1:0] with a bit width of P*N and the deserialized clock signal rx_outclock.

[0130] The input signals of the bit width conversion module include data Data[P*N-1:0], write clock signal wrclk, read clock signal rdclk, write request signal wrreq, read request signal rdreq and reset signal rst_n. Data[P*N-1:0] and wrclk come from the corresponding output of the LVDS IP core, while rdclk and rst_n come from the clock and reset management module. The output signals of this module include the FIFO read empty signal rdempty, write full signal wrfull_sig and data output signal Q[M*N-1:0]. The rdempty signal is inverted as the read request signal rdreq, and the wrfull_sig signal is inverted as the write request signal wrreq.

[0131] The input signal D_in1[M*N-1:0] of the big and small ends and the data splicing module comes from the output signal Q[M*N-1:0] of the bit width conversion module. D_out1[M*N-1:0] is the output signal of the module.

[0132] The input signals of the data byte alignment and shift module include data D_in2[M*N-1:0], clock signal clk2 and reset signal rst_n, where D_in2[M*N-1:0] comes from the output signal D_out1[M*N-1:0] of the big and small end and data splicing module, and the clock signal clk2 and reset signal rst_n come from the clock and reset management module. The output signal of this module is N-channel image data D_out2[M-1:0].

[0133] The input signals of the data analysis module include N data signals D_in3[M-1:0], clock signal clk3 and reset signal rst_n. D_in3[M-1:0] comes from the output of the data byte alignment and shift module, clk3 and reset signal rst_n come from the clock and reset management module. The output signals of this module include: frame valid signal Frame_valid and N line valid signals line_valid.

[0134] The input signals of the data reorganization module include N data signals D_in4[M-1:0], frame valid signal Frame_valid1, N line valid signals line_valid1, clock signal clk4 and clock signal clk5 and reset signal rst_n; the output signals include image data signal D_o4[M-1:0] and image data valid signal data_en. N D_in4[M-1:0] comes from the output signal N D_out2[M-1:0] of the data byte and shift module, and the Frame_valid1 signal comes from the Frame_valid signal of the data analysis module. N line_valid1 signals come from the N line_valid signals of the data analysis module. clk4 is homologous to the clock signal clk3 of the data analysis module. clk5 and rst_n come from the clock and reset management module, and their frequencies are N times that of the clock signal clk4. It should be noted that N used in the above modules is the number of data channels, and M is the pixel-level data bit width. In this embodiment, the value of N is 4.

[0135] In the embodiment of the present invention, after the serial data deserialization and synchronization system for high-speed imaging is powered on, the clock and reset management module starts working, specifically:

[0136] (1) After the system is powered on, the clock and reset management module outputs clock signals clk_out1 and clk_out2 according to the external input clock signal frequency and frequency division coefficient;

[0137] (2) At the same time, the reset signal outputs a low level, and the reset counter starts counting. When the count value reaches the preset value, the reset signal jumps from a low level to a high level, the reset is released, and the system starts to work normally.

[0138] In this embodiment, after the image sensor completes power-on startup and register configuration, it transmits data to the deserialization module through the LVDS interface. The deserialization module uses its own ALTLVDS_RXLVDS IP core to deserialize the low-voltage differential data. Specifically, assuming that there are N channels of LVDS differential data, and the serialization factor of each channel of LVDS differential data is P, the data bit width of the N channels of LVDS differential data after deserialization is P*N, which can be divided into two cases:

[0139] (1) When the serialization factor is less than or equal to 10, after the N-channel LVDS differential data is output by the ALTLVDS_RXLVDS IP core, the deserialized data with a data width of P*N can be directly obtained, and the deserialized data can be directly input into the data byte alignment and shift module to realize the functions of data shifting and byte synchronization.

[0140] (2) When the serialization factor is greater than 10, in order to achieve deserialization and synchronization of N-channel LVDS differential data, it is necessary to add a bit width conversion module and a big-endian conversion and data splicing module. Specifically:

[0141] First, the N-channel LVDS differential data is deserialized with a serialization factor of P through the ALTLVDS_RXLVDS IP core, where P=M / 2, and the deserialized data is parallel data with a bit width of P*N; then, it is converted from P*N to M*N through the bit width conversion module. Then, the big-endian conversion and data splicing are performed; finally, the data is processed through the data byte alignment and shift module to achieve data deserialization and byte synchronization.

[0142] When designing the parameters of the ALTLVDS_RXLVDS IP core, for the first case, the serialization factor P is the same as the value of the pixel-level data bit width M of the actual image sensor. If the pixel-level data bit width of the image sensor is 10 bits, the serialization factor is 10. For the second case, the serialization factor is half of the pixel-level data bit width of the actual image sensor. If the data bit width of the actual image sensor is 12 bits, the serialization factor should be 6.

[0143] The bit width conversion module is implemented using FIFO. The control strategy for the FIFO read and write port adopts the strategy of reading when not empty and writing when not full. Under this strategy, there is no specific requirement for the depth of the FIFO. The main design parameters are as follows: the input data bit width of the FIFO is P*N, and the output bit width is M*N. The clock signal at the input end of the FIFO uses the output clock signal of the LVDS IP core, and the clock signal at the output end of the FIFO is half of the input clock signal to achieve data rate matching. If M=12, N=8, P=6, the data bit width after deserialization by the ALTLVDS_RXLVDS IP core is 48, and the clock signal frequency is 6.25MHz, then after passing through the bit width conversion module, the data bit width should be 96, and the clock signal frequency should be 3.125MHz.

[0144] The function of the big-endian and small-endian and data splicing modules is to realize the big-endian and small-endian conversion of data and data splicing. First, the big-endian conversion is performed. Assuming that the data bit width of the FIFO output is: data[M*N-1:0], the converted data: data_out[M*N-1:0]={data[M*N / 2-1:0],data[M*N-1:M*N / 2]}; then the data is spliced, and the data splicing rules are:

[0145] data_out_o[M*N-1:0]={data_out[M*N-1:M*NM / 2],data_out[M*N / 2-1:M*(N-1) / 2-1],…,data_out[M*(N+1) / 2-1:M*N / 2],data_out[M / 2-1:0]}

[0146] Assuming that the pixel-level data bit width M is 12 bits and the number of LVDS data channels N is 2, then data_out_o[M*N-1:0]={data_out[23:18], data_out[11:6], data_out[17:12], data_out[5:0]}.

[0147] like Figure 3 As shown, the data byte alignment and shift module is implemented by a state machine, which is divided into four states: IDLE state, COMPARE state, SHIFT state and BITSLIP_DONE state, and includes a counter lcnt, a counter cnt and a counter cnt_bitslip.

[0148] (1) IDLE state: In this state, the state machine waits for the state transition condition. If the image sensor is powered on and the registers of the image sensor are configured, the counter lcnt starts counting. When the counting time is greater than the preset time required by the image sensor from the completion of register configuration to the start of training, the state machine jumps to the COMPARE state. The preset time is a real number greater than 1. In this example, the preset time is 32'h800.

[0149] (2) COMPARE state: In this state, data training is performed, specifically, the deserialized data or the third deserialized data is compared with the training code. If the two are not equal, the counter cnt is incremented by 1 and the SHIFT state is entered. Otherwise, the BITSLIP_DONE state is entered.

[0150] (3) SHIFT state: In this state, shifting is performed according to the value of counter cnt, and the shifted bit width is equal to the value of counter cnt; if the value of counter cnt is 1, it is shifted left by one bit (it can also be shifted right by one bit), and the maximum value of counter cnt is the image pixel level bit width M-1.

[0151] For the convenience of explanation, the data delayed by one clock signal cycle is defined as data1[M-1,0], and the data delayed by two clock signal cycles is defined as data2[M-1:0];

[0152] In this embodiment, the shift strategy is left shift, which is specifically: if it is a left shift of one bit, the data data2[M-1:0] is shifted left by one bit, and the highest bit of the data data1[M-1:0] is moved to the lowest bit of the data data2[M-1:0]; if it is a left shift of two bits, the data data2[M-1:0] is shifted left by two bits, and the upper two bits of the data data1[M-1:0] are moved to the lower two bits of the data data2[M-1:0]. Similarly, if it is a left shift of M-1 bits, the data data2[M-1:0] is shifted left by M-1 bits, and the upper M-1 bits of the data data1[M-1:0] are moved to the lower M-1 bits of the data data2[M-1:0]. (In other embodiments, the shift strategy can also be right shift, which is specifically: if it is shifted right by one bit, the data data1[M-1:0] is shifted right by one bit, and the lowest bit of the data data2[M-1:0] is moved to the highest bit of the data data1[M-1:0]; if it is shifted right by two bits, data1[M-1:0] is shifted right by two bits, and the lowest two bits of the data data2[M-1:0] are moved to the upper two bits of the data data1[M-1:0], and so on. If it is shifted right by M-1 bits, the data data1[M-1:0] is shifted right by M-1 bits, and the lower M-1 bits of the data data2[M-1:0] are moved to the upper M-1 bits of the data data1[M-1:0]. After the shift, the COMPARE state is entered).

[0153] (4) BITSLIP_DONE state, in which the counter cnt_bitslip is used to count. When the count value is greater than the preset value, the byte alignment is considered to be completed and the byte-aligned data is output. The state machine enters the IDLE state, otherwise it enters the COMPARE state.

[0154] In this embodiment, the preset value of the counter cnt_bitslip is 4, but is not limited to 4, and the preset value range may also be a positive integer greater than or equal to 1.

[0155] The counter cnt_bitslip described above will increase by 1 every time it passes through the BITSLIP_DONE state. Setting the value of the counter cnt_bitslip can control the number of comparisons, and multiple comparisons can prevent misjudgment.

[0156] from Figure 4 It can be seen that the number of byte alignment and shift units corresponds to the number of data channels. If the number of data channels is 4, the number of byte alignment and shift units is 4, which means that the byte alignment and shift unit is used to achieve byte synchronization of a single data channel.

[0157] The function of the data analysis module is to extract the frame valid signal and the line valid signal. The working strategy of the data analysis module is closely related to the image sensor. Generally, the image sensor will provide the frame head flag signal and the frame end flag signal, as well as the line head flag signal and the line end flag signal. The main working mode is:

[0158] (1) Detect the input data. If a frame header signal is detected, the frame valid signal is set to a high level. If a frame tail signal is detected, the frame valid signal is set to a low level.

[0159] (2) Detect the input data. If a row head signal is detected, the row valid signal is set to a high level. If a row end signal is detected, the row valid signal is set to a low level.

[0160] The function of the data reorganization module is to cache and process the image data, and realize the synchronization and sequential output of the data of each data channel while meeting the timing requirements. First, multiple FIFOs are used to cache the image data, and then the read-out control of the FIFO is completed through the state machine.

[0161] like Figure 5 As shown in FIG. 1 , it is a schematic diagram of the state transition of the read and write control part of the FIFO. Figure 5 It can be seen that 4 FIFOs are used, and the number of FIFOs is the same as the number of data channels.

[0162] Taking the four-channel LVDS output as an example, the working process of the state machine is described. The state machine includes IDLE,

[0163] FIFO_RST, FIFO_WAIT, FIFO1_RD, FIFO2_RD, FIFO3_RD, FIFO4_RD, FIFO4_STOP and other states. The counters included are cnt_rst, cnt1, cnt2, cnt3, cnt4, cnt_stop, line_cnt, etc. If the image size is K*J, K is the length of the image and J is the width of the image.

[0164] In the IDLE state, when the frame_pos pulse arrives, it enters the FIFO_RST state, in which the FIFO is reset and the counter cnt_rst is used for counting;

[0165] When cnt_rst == 8'd2, enter FIFO_WAIT state and detect the value of wrusedw_sig1;

[0166] When wrusedw_sig1≥K / 4, it enters the FIFO1_RD state. In this state, FIFO1 is read and the counter cnt1 starts counting.

[0167] When the count value of counter cnt1 reaches K / 4, the state switches to FIFO2_RD state. In this state, FIFO2 is read and counter cnt2 starts counting.

[0168] When the count value of counter cnt2 is K / 4, the state switches to FIFO3_RD state. In this state, FIFO3 is read and counter cnt3 starts counting.

[0169] When the count value of counter cnt3 reaches K / 4, the state switches to FIFO4_RD state. In this state, FIFO4 is read and counter cnt4 starts counting.

[0170] When the count value of counter cnt4 reaches K / 4, the state switches to FIFO4_STOP state, and line_cnt is incremented by 1 each time this state is passed;

[0171] When the count value of the counter line_cnt is equal to the actual image width J, the state machine enters the IDLE state, otherwise it enters the FIFO_WAIT state.

[0172] The write control of FIFO is determined by the output of the previous image sensor. The input bit width of FIFO is the same as the output bit width, which is the pixel-level bit width M of the image data. The output clock signal is N times the input clock signal to meet the rate matching. Assuming that the clock signal frequency of the FIFO input is 75MHz and the number of LVDS data channels is 4, the output clock signal frequency of FIFO is 300MHz. It should be particularly noted that the read control strategy of FIFO in this method is independent of the input data rate. Since the data is written into the FIFO at the same time each time, and the readout adopts a sequential readout strategy, the amount of data read out by each FIFO is K / N, where K is the length of the image and N is the number of FIFOs. The depth of each FIFO is required to be greater than or equal to the length K of the image.

[0173] Figure 5 Several important parameters are described as follows:

[0174] (1) frame_pos: The rising edge pulse signal of the image frame valid signal, used to indicate the beginning of a frame of data.

[0175] (2) cnt_rst: reset status counter, used to start the reset of FIFO. By setting different counter thresholds, the reset time of FIFO can be controlled. The value of this parameter is determined by the timing relationship between the frame_pos signal and the line_valid signal. The design principle is to complete the reset of FIFO before the arrival of valid data. The function of FIFO reset is to clear the data remaining in FIFO before the rising edge of image data arrives.

[0176] (3)wrusedw_sig1: indicates the amount of data written in the first FIFO.

[0177] (4) line_cnt: represents the line counter. The threshold condition is related to the image size. If the image size is 2048*2048, the value is set to 2048. If the image size is 4096*4096, the value is set to 4096.

[0178] (5) cnt4_stop is the FIFO4_STOP state counter, which is used to achieve timing matching and can also be used to control the frame rate of the output image. The minimum value is a positive integer greater than or equal to 1.

[0179] The entire implementation method of the above LVDS interface data deserialization and synchronization method is applicable to an image acquisition system using an Intel FPGA hardware platform. It is a perfection and improvement of the ALTLVDS_RXLVDS IP core, which expands its application scope and has strong practicality.

Claims

1. A serial data deserialization and synchronization system for high-speed imaging, characterized in that: It includes clock and reset management module, deserialization module, bit width conversion module, big and small end and data splicing module, data byte alignment and shift module, data parsing module and data reorganization module; The clock and reset management module is used to connect the system clock signal and the system reset signal, and its output end is respectively connected to the clock signal and the reset signal corresponding to the input end of the deserialization module, the bit width conversion module, the data byte alignment and shift module, the data analysis module and the data reorganization module; The input end of the deserialization module is used to receive N serial data generated by the external image sensor, and perform deserialization and bit alignment processing on the data; the bit width of the deserialized data is P*N, where P is the serialization factor, and its value range is [1:10]; the output end of the deserialization module is connected to the input end of the bit width conversion module; The bit width conversion module is used to convert data with a data bit width of P*N into data with a data bit width of M*N, where M is the pixel-level data bit width, wherein the value range of M is [1:20], and the value range of N is [1:18]; the output end of the bit width conversion module is connected to the large and small ends and the input end of the data splicing module; The big and small end and data splicing module is used to rearrange the data output by the bit width conversion module in chronological order, and its output end is connected to the input end of the data byte alignment and shift module; The data byte alignment and shift module decouples the big and small ends from the output data of the data splicing module, and realizes the data shift and byte alignment of each data channel of the deserialization module; Its output end is connected to the data analysis module and the data reorganization module respectively; The data analysis module is used to extract the frame valid signal and the row valid signal of the image data, and send them to the data reorganization module; The data reorganization module is used to realize synchronization and line-sequential output of image data according to the frame valid signal and line valid signal of the image data, and synchronously output the valid enable signal of the image data.

2. A serial data deserialization and synchronization system for high-speed imaging according to claim 1, characterized in that: The data byte alignment and shift module comprises N byte alignment and shift units; The data reorganization module is implemented by using N FIFOs.

3. The serial data deserialization and synchronization system for high-speed imaging according to claim 2, characterized in that: The deserialization module adopts LVDS IP core; the bit width conversion module adopts FIFO.

4. A serial data deserialization and synchronization method for high-speed imaging, characterized in that: The following steps are involved: Step 1: Determine whether the image pixel-level data bit width M is greater than 10 bits; If the image pixel-level data bit width M is greater than 10 bits, the serialization factor P = M / 2, and the N-channel serial data of the image sensor is input into the LVDS IP core, and the first deserialized data with a bit width of P*N is obtained after deserialization, and step 2 is executed; If the image pixel-level data bit width M is less than or equal to 10 bits, the serialization factor P=M, and the N-channel serial data is input into the LVDS IP core, and after deserialization, the deserialized data with a bit width of P*N is obtained, and step 4 is executed; Step 2: input the first deserialized data with a bit width of P*N into the bit width conversion module, and after conversion by the bit width conversion module, obtain the second deserialized data with a bit width of M*N; Step 3: Input the second deserialized data into the big-endian and small-endian and data splicing module to implement big-endian and small-endian conversion and splicing of the second deserialized data, and obtain the third deserialized data with a bit width of M*N; Step 4: Send the deserialized data or the third deserialized data to the data byte alignment and shift module, perform byte synchronization and shift operations on the deserialized data or the third deserialized data, and obtain N channels of image data with a bit width of M; Step 5: Send the N-channel image data to the data analysis module, and use the data analysis module to extract the frame valid signal and the line valid signal of the N-channel image data; Step 6: Extract the valid signals of N channels of image data through the frame valid signal and the row valid signal, and use the data reorganization module to cache and process the valid signals of the image data to achieve synchronous and sequential output of the image data.

5. The serial data deserialization and synchronization method for high-speed imaging according to claim 4, characterized in that: In step 2, the bit width conversion module uses FIFO, and the conversion is specifically as follows: 2.1) When the FIFO is not full, write the first deserialized data with a bit width of P*N into the FIFO; 2.2) Determine whether there is written data in the FIFO. If so, perform a FIFO read operation with a read data width of M*N. Otherwise, perform a wait operation.

6. A serial data deserialization and synchronization method for high-speed imaging according to claim 5, characterized in that: Step three is as follows: 3.1) Perform big-endian conversion Swap the upper M*N / 2 bits and the lower M*N / 2 bits of the second deserialized data to obtain swapped data data_out[M*N-1:0]={data[M*N / 2-1:0],data[M*N-1:M*N / 2]}; 3.2 Splicing Arrange the swapped data in chronological order, with each M bits of swapped data as a group, to obtain N groups of third deserialized data.

7. A serial data deserialization and synchronization method for high-speed imaging according to claim 6, characterized in that: In step 4, the byte synchronization and shifting operations of the deserialized data or the third deserialized data are implemented by using a state machine; Step 4 is as follows: 4.1) In the IDLE state, determine whether the state machine meets the state transition condition: After the image sensor is powered on and its registers are configured, the counter lcnt of the state machine starts counting; when the counting time reaches the preset time required by the image sensor, step 4.2) is executed; 4.2) Training in COMPARE state: Compare the deserialized data or the third deserialized data with the training code to see if they are equal. If they are not equal, add 1 to the counter cnt and execute step 4.3); If they are equal, proceed to step 4.4); 4.3) In the SHIFT state, shift is performed according to the count value of the counter cnt, the shift width is equal to the count value, and the count value range is [0, M-1]; After the shift is completed, return to step 4.2); 4.4) In the BITSLIP_DONE state, use the counter cnt_bitslip to count: If the count value of the counter cnt_bitslip is greater than a preset value, the byte alignment is completed, and the image data with a bit width of M is output, and the loop ends; the preset value is a positive integer greater than or equal to 1; If the count value of the counter cnt_bitslip is less than or equal to the preset value, return to step 4.2).

8. The serial data deserialization and synchronization method for high-speed imaging according to claim 7, characterized in that: In step 4.3), the shifting according to the count value of the counter cnt is specifically as follows: 4.3.

1. Delay the deserialized data or the third deserialized data by at least one clock signal cycle to obtain data1[M-1,0]; and delay data1[M-1,0] by one clock signal cycle to obtain data2[M-1:0]; 4.3.

2. Left Shift If you need to shift one bit, shift data2[M-1:0] left by one bit, and move the highest bit of data1[M-1:0] to the lowest bit of data2[M-1:0]; and so on, if you need to shift M-1 bits, shift data2[M-1:0] left by M-1 bits, and move the high M-1 bits of data1[M-1:0] to the low M-1 bits of data2[M-1:0]; Or, shift right If one bit needs to be shifted, data1[M-1,0] is shifted right by 1 bit, and the lowest bit of data2[M-1:0] is moved to the highest bit of data1[M-1:0]; and so on. If M-1 bits need to be shifted, data1[M-1,0] is shifted right by M-1 bits, and the lower M-1 bits of data2[M-1:0] are moved to the higher M-1 bits of data1[M-1:0].

9. The serial data deserialization and synchronization method for high-speed imaging according to claim 8, characterized in that: The preset time in step 4.1) is the shortest time between the completion of register configuration and the start of training; The preset value in step 4.4) is 4.

10. The serial data deserialization and synchronization method for high-speed imaging according to claim 9, characterized in that: In step six, a state machine is used to realize synchronization and sequential output of image data; Step 6 is as follows: 6.1) Set the number of FIFOs to be the same as the number of LVDS interfaces output by the external image sensor; 6.2) Reset the FIFO; 6.3) When a frame valid signal is received, the image data is controlled to be written into the FIFOs according to the N-way row valid signals, and the N FIFOs are read out sequentially according to the amount of data written into the FIFOs; Realize the output of image data in row order; When a frame of data is read and written, the state machine enters the initial state and waits for the next frame valid signal, wherein the read clock signal frequency of the N FIFOs is greater than or equal to N times the clock signal frequency of the write FIFO.

Images