Multi-camera data transmission system based on optical port

Abstract

The utility model relates to the field of digital signal processing, in particular to a multi-camera data transmission system based on an optical port, which comprises a DDR3 (Double Data Rate 3), the optical port, a plurality of cameras and a plurality of cache modules in one-to-one correspondence with the cameras, and the cameras are connected with the cache modules and are used for acquiring image data and transmitting the image data to the cache modules; the plurality of cache modules are connected with the DDR3 and are used for transmitting cache data to the DDR3 for partition storage after the cache data reaches a specified quantity; and the optical port is connected with the DDR3 and is used for reading the image data stored in each area in the DDR3 and sending the image data to target equipment.

Description

Technical Field

[0001] The utility model relates to the field of digital signal processing, in particular to a multi-camera data transmission system based on an optical port. Background Art

[0002] A camera is a device that uses optics to create and record images. Cameras are widely used in industry. Images captured by a camera need to be transmitted to a target receiving module within a specified timeframe. Especially when multiple cameras are working together, how to quickly and stably transmit all images to the target receiving module becomes a critical challenge.

[0003] With the rapid development of digital image processing technology, multi-camera systems are becoming increasingly common in various applications, such as autonomous driving, industrial inspection, medical imaging, and security monitoring. Cameras are widely used in industry, and images captured by these cameras need to be transmitted to the target receiving module within a specified time. Especially when multiple cameras are working together, how to quickly and stably transmit all images to the target receiving module is a challenge that needs to be solved. Utility Model Content

[0004] In order to solve the problems in the prior art, an embodiment of the present invention aims to provide a multi-camera data transmission system based on an optical port.

[0005] In order to achieve the above-mentioned purpose, the embodiment of the present invention provides a multi-camera data transmission system based on an optical port, comprising DDR3, an optical port, multiple cameras, and multiple cache modules corresponding to the cameras one by one.

[0006] The camera is connected to the cache module and is used to acquire image data and transmit it to the cache module;

[0007] The plurality of cache modules are all connected to the DDR3 and are used to transfer the cached data to the DDR3 for partition storage when the cached data reaches a specified amount;

[0008] The optical port is connected to the DDR3 and is used to read the image data stored in each area of ​​the DDR3 and send it to the target device.

[0009] Preferably, the DDR3 includes an interface occupancy arbitration module, a DDR memory and a plurality of FIFO function simulation modules, and the FIFO function simulation modules correspond one to one to the cameras;

[0010] Each of the FIFO function simulation modules includes a read cache FIFO and a write cache FIFO. The image data from the cache module is first stored in the write cache FIFO and then transferred to the DDR memory via the write cache FIFO. The optical port reads the DDR3 image data and stores it in the read cache FIFO first to prevent data loss when reading or writing data.

[0011] The interface occupancy arbitration module is connected to the FIFO function simulation module and is used to determine whether the data in the write cache FIFO can be written into the DDR memory, and to evaluate the read and write requests sent by each of the FIFO function simulation modules and determine the priority.

[0012] Preferably, the system includes an image output summary module and an image control module corresponding one-to-one to the FIFO function simulation module. The image control module is respectively connected to the FIFO function simulation module and the image output summary module, and is used to send the received image data to the write cache FIFO, and / or receive the data in the read cache FIFO and send it to the image output summary module; the image output summary module is connected to the optical port, and is used to send the image data to the host computer through the optical port.

[0013] Preferably, the system also includes a camera logic control module, which includes a logic controller and an image sensor. The logic controller is connected to the image sensor and is used to configure the registers of the image sensor after power-on or after receiving an initialization command, and to realize the register reading and writing, triggering photo taking and image receiving functions of the image sensor.

[0014] Preferably, the logic controller includes a sensor register configuration module, a sensor image receiving module and a control module.

[0015] The sensor register configuration module communicates with the image sensor using an I2C interface, and is used to read, write and verify the image sensor register;

[0016] The sensor image receiving module communicates with the image sensor using an LVDS interface to obtain image data taken by the image sensor;

[0017] The control module is used to receive control instructions from the host computer and parse and convert them into trigger signals that can be recognized by the image sensor, and send the trigger signal to the image sensor; and / or determine whether the control instructions issued by the host computer are legal. If not, ignore them; if legal, send the instructions to the sensor register configuration module or the sensor image receiving module.

[0018] Preferably, the sensor register configuration module includes a parameter checking unit, an I2C communication unit, a control unit and an initialization sequence unit.

[0019] The control unit is connected to the control module and is used to receive commands from the control module and perform configuration, modification or verification operations on the image sensor through the I2C communication unit;

[0020] The initialization sequence unit is connected to the I2C communication unit and the control unit respectively, and is used to initialize and configure the image sensor through the I2C communication unit after power-on to ensure that the image sensor works normally and the image taken meets the requirements;

[0021] The parameter checking unit is connected to the I2C communication unit and the control unit respectively, and is used to determine whether the command received by the control unit is reasonable, and to verify whether the configuration parameters returned by the I2C communication unit are correct.

[0022] Preferably, the sensor receiving module communicates with the image sensor using a 4-channel LVDS interface. Specifically, when the LVDS data path is in standby mode, the four data channels alternately repeat 0x80 and 0x10 data, the sync channel of the LVDS data path repeats 0x1f data and LVDS data alignment is performed based on the 0x1f data, and data transmission is in a row cycle, with each row containing 4n pixels.

[0023] Preferably, the logic controller further includes an error code encoding module for generating an error code to indicate which error the logic controller has generated.

[0024] Preferably, the logic controller further includes a test image verification module, which is connected to the sensor image receiving module. The logic controller uses the test image verification module to verify each non-pixel data during data transmission by the sensor image receiving module to compare whether it is consistent with expectations. If the verification passes, the image data is uploaded.

[0025] The above technical solution allows multiple cameras to work simultaneously, distinguishing image data and ensuring stable upload of image data. In order to allow FPGA to allow multiple flying cameras to work simultaneously, distinguish image data and ensure stable upload of image data, this application adopts the method of first caching the image data of multiple cameras into the corresponding cache modules. When the cache modules reach the specified number, they are respectively transferred to the corresponding FIFO function simulation modules in DDR3, and read and write transmissions are performed separately through the optical port, thereby realizing multiple cameras working simultaneously, effectively distinguishing the data sources of each camera, and ensuring that the image data can be uploaded stably.

[0026] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following detailed description, they are used to explain the embodiments of the present invention, but do not constitute a limitation of the embodiments of the present invention. In the accompanying drawings:

[0028] Figure 1 This is a schematic diagram of the structure of the multi-camera data transmission system based on optical port of the utility model;

[0029] Figure 2 This is a schematic diagram of the DDR3 structure in an embodiment of the present utility model;

[0030] Figure 3 This is a schematic diagram of the structure of the camera logic control module according to an embodiment of the present utility model;

[0031] Figure 4 This is a schematic diagram of the LVDS data path of an embodiment of the present utility model;

[0032] Figure 5 This is a schematic diagram of the LVDS data format of an embodiment of the present utility model. DETAILED DESCRIPTION

[0033] The following describes the specific implementation of the embodiment of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the embodiment of the present invention and is not intended to limit the embodiment of the present invention.

[0034] The present invention provides a multi-camera data transmission system based on optical port. For example, the system is applied to a chip placement machine for image acquisition and recognition processing. The camera is a flying camera. The system is implemented based on FPGA. Figure 1 As shown, it includes DDR3, an optical port, multiple cameras and multiple cache modules corresponding to the cameras one by one. The cameras are connected to the cache modules to obtain image data and transmit them to the cache modules; the multiple cache modules are all connected to DDR3 to transmit the cached data to the DDR3 for partitioned storage when the cached data reaches a specified amount; the optical port is connected to DDR3 to read the image data stored in each area of ​​DDR3 and send it to the target device.

[0035] Further, if Figure 2As shown, the DDR3 includes an interface occupancy arbitration module, a DDR memory, and multiple FIFO function simulation modules. The FIFO function simulation modules correspond to the cameras one by one. Each FIFO function simulation module includes a read cache FIFO and a write cache FIFO. The image data from the cache module is first saved in the write cache FIFO and then transferred to the DDR memory via the write cache FIFO. After the optical port reads the image data from the DDR3, it is first stored in the read cache FIFO to prevent data loss when reading or writing data. The interface occupancy arbitration module is connected to the FIFO function simulation module to determine whether the data in the write cache FIFO can be written to the DDR memory, and to evaluate the read and write requests sent by each FIFO function simulation module and determine the priority.

[0036] like Figure 2 As shown, the system includes an image output summary module and an image control module corresponding to the FIFO function simulation module one by one. The image control module is respectively connected to the FIFO function simulation module and the image output summary module, and is used to send the received image data to the write cache FIFO, and / or receive the data in the read cache FIFO and send it to the image output summary module; the image output summary module is connected to the optical port, and is used to send the image data to the host computer through the optical port.

[0037] For example, consider the image transmission from multiple flying cameras on a chip placement machine. Each flying camera corresponds to a cache module. The cache module initially stores image data captured by the flying camera. When the cached data reaches a certain amount, the image receiving logic retrieves the cached data and stores it in DDR3. Multiple cache modules can operate concurrently, each storing data in a different DDR3 area. The image receiving logic divides the areas to ensure that the cache does not overflow and that multiple flying cameras can operate simultaneously. For ease of operation, each DDR3 area is encapsulated as a FIFO, or FIFO functional simulation module. The FIFO functional simulation module has built-in write and read buffers. Each flying camera corresponds to a FIFO functional simulation module. Image data is first stored in the write buffer and then written to the DDR3. The image receiving logic also has an interface occupancy arbitration module connected to each FIFO functional simulation module to determine whether the FIFO functional simulation module can write the write buffer data to the DDR3. The FPGA's optical port module corresponds to an image output aggregation module, which reads the image data from the DDR3 and sends it to the main control board (i.e., the host computer) via the optical port. Each FIFO function simulation module is connected to an image control module, which in turn connects to the image output aggregation module to ensure stable transmission of image data from each flight camera. The image output aggregation module controls all image control modules. When image data from a specific flight camera needs to be output, the image output aggregation module selects the image control module for that flight camera and then outputs the image data from that flight camera. The image control module receives image data from the flight camera and stores it in the write buffer (FIFO) of the FIFO function simulation module. It also receives control from the image output aggregation module and outputs the image data in conjunction with the read buffer (FIFO) of the FIFO function simulation module.

[0038] When the data volume is low or the read speed is higher than the write speed, the data path through the DDR3 is skipped. Subsequent data is not allowed to enter the DDR3 until the read buffer is full. When the remaining data in the DDR3 is read and the read buffer is not full, the DDR3 path is skipped again, and data is directly transferred from the read buffer to the write buffer. The interface occupancy arbitration module connects to the DDR3 through the DDR control core. It evaluates the read and write requests sent by each FIFO function simulation module to determine which one to respond to first. The priority is: write requests from the module whose write buffer FIFO is about to fill > read requests > other write requests.

[0039] Furthermore, the system also includes a camera logic control module for controlling a single camera, the camera logic control module including a logic controller and an image sensor, the logic controller being connected to the image sensor and configured to configure the registers of the image sensor after power-on or after receiving an initialization command, and to implement the image sensor register reading and writing, triggering photo taking and image receiving functions; the logic controller including a sensor register configuration module, a sensor image receiving module and a control module, the sensor register configuration module communicating with the image sensor using an I2C interface for reading, writing and verifying the image sensor registers; the sensor image receiving module communicating with the image sensor using an LVDS interface for acquiring image data taken by the image sensor; the control module being configured to receive control instructions from a host computer and parse and convert them into trigger signals recognizable by the image sensor, and to send the trigger signals to the image sensor; and / or determining whether the control instructions issued by the host computer are legal, ignoring them if illegal, and sending them to the sensor register configuration module or the sensor image receiving module if legal.

[0040] For example, Figure 3 As shown in the figure, the camera logic control module of the flying camera is used to configure the registers of the SC130GS image data sensor after power-on and receiving the initialization command, and to implement register reading and writing, triggering photo taking and image receiving functions.

[0041] The camera logic control module communicates with the sensor registers using an I2C interface at a baud rate of 100 Kbps. The sensor register configuration module is responsible for reading, writing, and verifying the image sensor registers. It consists of a parameter check unit, an I2C communication unit, a control unit, and an initialization sequence unit. The control unit receives commands from the control module external to the module and then configures, modifies, or verifies the image sensor. The initialization sequence unit initializes the SC130GS image sensor using the I2C communication unit after power-up, ensuring proper function and that images captured meet the required specifications. The parameter check unit verifies the correctness of the configured parameters. This involves two steps: first, determining whether the command received by the control unit is valid. If so, the image sensor is configured; otherwise, the command is ignored. Second, the configured parameters are read back to verify the data written to the image registers.

[0042] The error code encoding module is used to produce error codes, which can prompt which errors are generated by this module, making it easier to troubleshoot and solve problems. The error codes include: taking pictures before initialization is completed, taking pictures again before the last picture is finished, receiving the next communication task during I2C communication, receiving the communication task before the I2C initialization configuration is completed, performing I2C communication during the receiving image, the data written to the register configuration is inconsistent with the read check value, I2C communication is disconnected, suspected image data is received when no photo trigger signal is given, the data receiving module is not ready when the photo trigger signal is generated, the image row head data reception wait timeout, LVDS SYNC channel data abnormality, the received image height is greater than the last register setting parameter, the received image height is less than the last register setting parameter, the received image width is greater than the last register setting parameter, the received image The width is smaller than the last register setting parameter, the LVDS line end data alignment status is abnormal, the LVDS image ID data is abnormal, the LVDS data alignment timeout, the operation instruction is received before the register initialization is completed, the next initialization instruction is received before the register initialization is completed, the operation instruction is received during the register reading and writing process, the operation instruction is received during the photo taking process, the photo taking instruction and other instructions are executed at the same time, the register initialization timeout, the image receiving module initialization timeout, the initialization test photo transfer timeout, the photo transfer timeout, the register read timeout, the register write timeout, the photo taking instruction is received in sleep mode, the register data is not suitable for photo taking when the photo taking instruction is received, the photo taking instruction is received when the image receiving module status is abnormal, and the image receiving buffer space overflow.

[0043] The control module within the camera logic control module is responsible for receiving control commands and transmitting the contents of the image registers. Upon receiving a control command, the control module first parses the command, converts it into a trigger signal recognized by the image sensor, and finally transmits the trigger signal to the image sensor. The control module also determines whether the command issued is legal and ignores any illegal commands. However, if a command is received to retrieve the contents of an image sensor register, the control module immediately returns the register contents to the relevant module.

[0044] The sensor image receiving module uses the LVDS interface to connect to the image sensor to obtain the image data taken by the image sensor. When the LVDS data path is in standby mode, the four data channels will alternately repeat 0x80 and 0x10 data, and the sync channel will repeat 0x1f data, such as Figure 4 As shown, LVDS data alignment can be performed based on this data.

[0045] Data transmission is in rows. Each row contains 4n pixels (if it is not an integer multiple of 4, irrelevant data will be padded at the end of each row until it is an integer multiple of 4). Each pixel data is a 1-byte grayscale value. Suppose the data of each pixel in a row is D[0], D[1], D[2], ..., D[4n-2], D[4n-1] in sequence. The data format of a row is as follows: Figure 5 As shown, (where FS[7:5]=0x5, LS[7:5]=0x1, FE[7:5]=0x6, LE[7:5]=0x2).

[0046] The camera logic control module uses the test image verification section to verify the non-pixel data during data transmission, comparing it to expected values. Only if verification passes will the image data be uploaded. The distance between two lines in the same image (between one FS / LS and the next LS) is determined by the data in sensor registers 0x320c and 0x320d. If the difference is too large, an error is reported. The image data cross-clock domain module is used for cross-clock processing, ensuring that image data can be processed in multiple clock domains. The control signal cross-clock domain module performs asynchronous processing of control signals to prevent signal loss during cross-clock processing.

[0047] The control logic of the optical port retrieves image data from DDR3 and sends the image to the main control board through the optical port. This is concurrent with the flight camera taking a picture and storing the image data in DDR3, and they do not interfere with each other. The control logic of the optical port uses the optical port as a physical interface to implement a set of control logic suitable for the transmission of flight camera image data. For example, the control logic of the optical port uses Xilinx's built-in Transceivers Wizard IP to control access. The communication frequency between the FPGA and the main control board optical port of this embodiment system is 156.25 MHz, and the data bit width is 32 bits. To facilitate data operations, the optical port is encapsulated into an asynchronous interface. The meaning of the encapsulated interface is shown in Table 1. pic_clk comes from the main control board and cannot be lower than 40 MHz. Other clocks are arbitrary. The main control board operates on the other data in Table 1 based on pic_clk.

[0048] Table 1 Interface after optical port encapsulation

[0049]

[0050]

[0051] The hardware circuit and Xilinx's own IP core have already built the physical layer and part of the data link layer in the transmission link. The design of the optical port in the present invention includes the data link layer, the transport layer, and the session layer. The data link layer involves block segmentation and assembly. The data in the data link layer is divided into three levels: data packets, data blocks, and frames. Frames are further divided into data frames (block frames) and heartbeat frames. Packets are data units that interact with the application layer, for example: a photo with attached shooting information, a remote update bitstream file with attached remote update information, and a communication task frame. The size of a packet can be very large and interacts with the application layer through a streaming mode. After receiving the frame from the application layer, the sending end cuts the packet into individual blocks. The size of each block has an upper limit (e.g., no more than 1k bytes). The block and the frame data are added to the head and tail of the block to form a block frame, which is sent to the receiving end through the data link layer and the physical layer. In addition, the two communicating ends will periodically send heartbeat frames that do not contain blocks to each other to synchronize status information on both sides. After the frame is sent to the receiving end, the receiving end verifies it and removes the frame header and tail to obtain blocks. The blocks are then spliced ​​into packets and sent to the application layer.

[0052] In summary, the technical solution of the present invention achieves the following beneficial effects: in order for FPGA to allow multiple flying cameras to work simultaneously, while distinguishing image data and ensuring stable uploading of image data, this application adopts the method of first caching the image data of multiple cameras into the corresponding cache modules respectively. When the cache modules reach the specified number, they are respectively transmitted to the corresponding FIFO function simulation modules in DDR3, and read and write transmissions are performed separately through the optical port, thereby realizing multiple cameras working simultaneously, effectively distinguishing the data sources of each camera, and ensuring that the image data can be uploaded stably.

[0053] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements, but also other elements not expressly listed, or elements inherent to such process, method, commodity, or apparatus.

[0054] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A multi-camera data transmission system based on an optical port, characterized in that: It includes DDR3, optical port, multiple cameras and multiple cache modules corresponding to the cameras. The camera is connected to the cache module and is used to acquire image data and transmit it to the cache module; The plurality of cache modules are all connected to the DDR3 and are used to transfer the cached data to the DDR3 for partition storage when the cached data reaches a specified amount; The optical port is connected to the DDR3 and is used to read the image data stored in each area of ​​the DDR3 and send it to the target device.

2. The optical port-based multi-camera data transmission system according to claim 1, characterized in that: The DDR3 includes an interface occupancy arbitration module, a DDR memory and a plurality of FIFO function simulation modules, wherein the FIFO function simulation modules correspond one to one with the cameras; Each of the FIFO function simulation modules includes a read cache FIFO and a write cache FIFO. The image data from the cache module is first stored in the write cache FIFO and then transferred to the DDR memory via the write cache FIFO. After the optical port reads the image data from the DDR3, it is first stored in the read buffer FIFO to prevent data loss when reading or writing data; The interface occupancy arbitration module is connected to the FIFO function simulation module and is used to determine whether the data in the write cache FIFO can be written into the DDR memory, and to evaluate the read and write requests sent by each of the FIFO function simulation modules and determine the priority.

3. The optical port-based multi-camera data transmission system according to claim 2, characterized in that: The system includes an image output summary module and an image control module corresponding to the FIFO function simulation module one by one. The image control module is respectively connected to the FIFO function simulation module and the image output summary module, and is used to send the received image data to the write cache FIFO, and / or receive the data in the read cache FIFO and send it to the image output summary module; the image output summary module is connected to the optical port, and is used to send the image data to the host computer through the optical port.

4. The optical port-based multi-camera data transmission system according to claim 1, characterized in that: The system also includes a camera logic control module, which includes a logic controller and an image sensor. The logic controller is connected to the image sensor and is used to configure the registers of the image sensor after power-on or receiving an initialization command, and to implement register reading and writing, triggering photo taking and image receiving functions of the image sensor.

5. The optical port-based multi-camera data transmission system according to claim 4, characterized in that: The logic controller includes a sensor register configuration module, a sensor image receiving module and a control module. The sensor register configuration module communicates with the image sensor using an I2C interface, and is used to read, write and verify the image sensor register; The sensor image receiving module communicates with the image sensor using an LVDS interface to obtain image data taken by the image sensor; The control module is used to receive control instructions from the host computer and parse and convert them into trigger signals that can be recognized by the image sensor, and send the trigger signal to the image sensor; and / or determine whether the control instructions issued by the host computer are legal. If not, ignore them; if legal, send the instructions to the sensor register configuration module or the sensor image receiving module.

6. The optical port-based multi-camera data transmission system according to claim 5, characterized in that: The sensor register configuration module includes a parameter checking unit, an I2C communication unit, a control unit and an initialization sequence unit. The control unit is connected to the control module and is used to receive commands from the control module and perform configuration, modification or verification operations on the image sensor through the I2C communication unit; The initialization sequence unit is connected to the I2C communication unit and the control unit respectively, and is used to initialize and configure the image sensor through the I2C communication unit after power-on to ensure that the image sensor works normally and the image taken meets the requirements; The parameter checking unit is connected to the I2C communication unit and the control unit respectively, and is used to determine whether the command received by the control unit is reasonable, and to verify whether the configuration parameters returned by the I2C communication unit are correct.

7. The optical port-based multi-camera data transmission system according to claim 5, characterized in that: The sensor receiving module communicates with the image sensor using a 4-channel LVDS interface. Specifically, when the LVDS data path is in standby mode, the four data channels alternately repeat 0x80 and 0x10 data, the sync channel of the LVDS data path repeats 0x1f data and LVDS data alignment is performed based on the 0x1f data. Data transmission is in a row cycle, and each row contains 4n pixels.

8. The optical port-based multi-camera data transmission system according to any one of claims 4 to 7, characterized in that: The logic controller further comprises an error code encoding module for generating an error code to indicate which error the logic controller has generated.

9. The optical port-based multi-camera data transmission system according to claim 8, characterized in that: The logic controller also includes a test image verification module, which is connected to the sensor image receiving module. The logic controller uses the test image verification module to verify each non-pixel data during data transmission by the sensor image receiving module to check whether it is consistent with expectations. If the verification passes, the image data is uploaded.

Images