Multi-camera real-time concurrent data transmission system

Abstract

The present invention relates to the field of digital signal processing technology, and discloses a multi-camera real-time concurrent data transmission system, including a flying camera board, a main control board and an industrial computer. The flying camera board communicates with the main control board via an optical port, and is used to actively upload multiple flying camera data to the main control board in a multi-buffer concurrent manner; the main control board communicates with the industrial computer via PCIe, and is used to passively receive data from the flying camera board, and transmit the data to the industrial computer via a PCIe interface. This solution can realize an efficient, real-time concurrent data transmission system by combining the two interfaces of PCIe and optical port, thereby solving the bottleneck problem of multi-camera image transmission, not only improving the speed and efficiency of data transmission, but also ensuring the integrity and real-time nature of data, and meeting the needs of various high-performance image processing applications.

Description

Technical Field

[0001] The present invention relates to the field of digital signal processing, and in particular to a multi-camera real-time concurrent data transmission system. Background Art

[0002] A camera is a device that uses optical imaging to record images. In industry, cameras are widely used. The pictures taken by the camera need to be transmitted to the target receiving module within a specified time. Especially when multiple cameras work together, how to quickly and stably transmit all pictures to the target receiving module is a problem that needs to be solved.

[0003] With the rapid development of digital image processing technology, multi-camera systems are becoming more and more common in various applications, such as autonomous driving, industrial inspection, medical imaging, and security monitoring. These systems often need to process a large amount of high-resolution image data, and how to effectively transmit and process this data has become a key technology.

[0004] FPGA is an ideal choice for solving this problem due to its high concurrent processing capability and customizability. FPGA can realize real-time data processing and transmission, meeting the requirements of high performance and multi-tasking. However, as the number and resolution of cameras increase, the need for data transmission bandwidth also increases, which puts higher requirements on the interface design of the system. Therefore, how to realize real-time concurrent FPGA data transmission of multiple cameras needs to be solved. Summary of the invention

[0005] In order to solve the problems in the prior art, an object of an embodiment of the present invention is to provide a multi-camera real-time concurrent data transmission system.

[0006] In order to achieve the above object, the present invention provides a multi-camera real-time concurrent data transmission system, including a flight camera board, a main control board and an industrial computer.

[0007] The flight camera board communicates with the main control board via an optical port, and is used to actively upload multiple flight camera data to the main control board in a multi-buffer concurrent manner;

[0008] The main control board communicates with the industrial computer via PCIe, is used for passively receiving data from the flight camera board, and transmitting the data to the industrial computer via the PCIe interface.

[0009] Preferably, the flight camera board includes a plurality of flight cameras, a plurality of first cache modules corresponding to the flight cameras one by one, a first DDR module and an optical port,

[0010] The flying camera is used to receive the flying camera photo-taking instruction issued by the industrial computer and perform a photo-taking action after parsing the instruction;

[0011] The first cache module is used to store image data obtained from the flight camera after taking pictures, and when the cached data reaches a specified amount, it is transmitted to the first DDR module;

[0012] The first DDR module is used to receive data from the plurality of first cache modules and store the data in different areas respectively;

[0013] The optical port is used to read data from the first DDR module and send the data to the main control board.

[0014] Preferably, each area of ​​the first DDR module is encapsulated into a FIFO function simulation module, and each of the FIFO function simulation modules includes a read cache FIFO and a write cache FIFO. The image data from the first cache module is first saved in the write cache FIFO and then written into the memory of the first DDR module. After the optical port reads the image data of the first DDR module, it is first stored in the read cache FIFO to prevent data loss when reading or writing data.

[0015] Through the above technical solution, by combining the two interfaces of PCIe and optical port, an efficient, real-time concurrent data transmission system can be realized, thus solving the bottleneck problem of multi-camera image transmission. This system can not only improve the speed and efficiency of data transmission, but also ensure the integrity and real-time performance of data, meeting the needs of various high-performance image processing applications.

[0016] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0018] Figure 1 It is a schematic diagram of the structure of the multi-camera real-time concurrent data transmission system of the present invention;

[0019] Figure 2 It is a schematic diagram of the structure of the main control board in an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of the link structure of the second DDR module according to an embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of the image link control logic of PFGA in an embodiment of the present invention;

[0022] Figure 5 It is a schematic diagram of the image data transmission process of the industrial control board of the present invention;

[0023] Figure 6 is a control logic schematic diagram of a single flying camera of the present invention;

[0024] Figure 7 It is a schematic diagram of the image collection control logic of multiple flying cameras of the present invention;

[0025] Figure 8 is a schematic diagram of the data link layer architecture of an optical port according to an embodiment of the present invention;

[0026] Fig. 9 This is a schematic diagram of the conventional full-duplex communication mode between optical ports;

[0027] Fig.10 is a schematic diagram of data interaction between optical ports according to an embodiment of the present invention;

[0028] Fig.11 This is a schematic diagram of reconnecting a disconnected line of an optical port according to an embodiment of the present invention;

[0029] Fig.12 This is a schematic diagram of restarting and reconnecting between optical ports according to an embodiment of the present invention;

[0030] Fig.13 It is a schematic diagram of the structure of an industrial control computer according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0032] The embodiment of the present invention provides a multi-camera real-time concurrent data transmission system, such as Figure 1 As shown, it includes a flying camera board, a main control board and an industrial computer. The flying camera board communicates with the main control board via an optical port, and is used to actively upload multiple flying camera data to the main control board in a multi-buffer concurrent manner; the main control board communicates with the industrial computer via PCIe, and is used to passively receive data from the flying camera board, and transmit the data to the industrial computer via a PCIe interface.

[0033] By combining the PCIe and optical interfaces, an efficient, real-time concurrent data transmission system can be realized, thus solving the bottleneck problem of multi-camera image transmission. This system can not only improve the speed and efficiency of data transmission, but also ensure the integrity and real-time nature of the data, meeting the needs of various high-performance image processing applications.

[0034] In the embodiment of the present invention, Figure 2As shown, the main control board is provided with an optical port, a PCIe interface, a second DDR module and an FPGA, wherein the optical port is connected to the FPGA as a data input interface; the second DDR module is connected to the FPGA for receiving and storing data from the flight camera board; the PCIe interface is connected to the FPGA as a data output interface for outputting data in the second DDR module. The second DDR module can be DDR3, DDR4 or DDR5, and DDR3 is selected as an example in the embodiment of the present invention.

[0035] In order to forward the image data of the flight camera board to the industrial computer, it is necessary to design FPGA control logic to control the optical port, PCIe and DDR3 (i.e. the second DDR module). The image link control logic design in the main control board is as follows: Figure 4 shown.

[0036] PCIe uses XDMA to control access, based on the DMA principle, to achieve data access to the main control board and industrial computer. In the image link, XDMA uses the AXI4 bus, the clock frequency of AXI4 is 125MHz, the data bit width is 64 bits, the address width is 64 bits, the channel width of the PCIe interface is X4, and the reference clock frequency is 100MHz. When the main control board receives a picture data sent by the flight camera board, it needs to send an interrupt to the industrial computer, so an MSI interrupt is configured in XDMA to notify the industrial computer that it can request image data. At the same time, the image data requires a DMA read channel and a DMA write channel, and an additional c2h and h2c channels are required in XDMA.

[0037] The optical port data write DDR3 logic contains a synchronous FIFO with a data width of 128 bits and a depth of 2048. This synchronous FIFO is responsible for caching the pic_data in Table 1 first, and the caching timing is when pic_valid is 1. The optical port data write DDR3 logic converts the image data into the AXI4 protocol and then writes it to DDR3. It also interacts with the flight camera image data control logic.

[0038] The PCIe read DDR3 logic consists of two parts, one is to obtain the relevant control signal of the industrial computer, and the other is to read and write the DDR3 logic. For obtaining the control signal of the industrial computer, the image link consists of three parts: one is to store the first address and length information of the image data, the second is to request 0x00. Address, indicating that the industrial computer has received the interrupt signal, and the third is to request 0x10 address, indicating that the industrial computer has received the image data.

[0039] The main control board is responsible for receiving the image data transmitted from the flight camera board and transmitting the image data to the industrial computer through the PCIe interface. Since DDR3 can only mutually access DDR3 according to a specific clock and a specific data bit width at the same time, in order to ensure that the main control board can receive image data in time and synchronously access DDR3, the present invention provides a general multi-bit width and multi-clock domain concurrent reading and writing DDR3 solution. Based on this solution, the main control board combines the optical port control logic, PCIe control logic and internal image link control logic together to realize an image link solution based on the optical port and PCIe, which solves the problem of cross-clock data conversion between the optical port and PCIe.

[0040] Furthermore, in order to implement a general multi-bit width multi-clock domain concurrent reading and writing method for the second DDR module, the link structure of the second DDR module (ie, DDR3) is as follows: Figure 3 As shown, it includes a configuration arbitration module, a DDR control logic module and a DDR storage module. The DDR control logic module is used to receive data from the configuration arbitration module and return relevant data; the configuration arbitration module is used to configure the access logic and arbitrate the devices that are allowed to access the second DDR module at the same time according to the priority of the access logic; the DDR storage module is used to cache the accessed data or instructions.

[0041] Furthermore, the configuration arbitration module includes a plurality of access logic units, and arbitration logic units respectively connected to the plurality of access logic units, and the arbitration logic unit is connected to the DDR control logic module via an AXI4 bus. The plurality of access logic units are respectively connected to the FPGA via an AXI4 bus. A second cache module is also provided on the main control board. When the main control board is in an idle state and image data arrives at the optical port, the image data is first stored in the second cache module, and the image data is stored in the second DDR module according to whether the second cache module has data. When an image is stored, the main control board initiates an interrupt to notify the industrial computer that the image data can be read.

[0042] In order to facilitate data interaction between module logic, the entire solution is designed based on the AXI4 protocol. The access to DDR3 requires the use of the AXI4 protocol, and the access to the configuration arbitration module also requires the AXI4 protocol. The DDR control logic module implements access control to hardware DDR3. Relevant commands such as read and write control are all in this module. The DDR control logic module receives data on the AXI4 bus of the configuration arbitration module and returns relevant data to the configuration arbitration module. The DDR control logic module is implemented using MIG IP. The system clock is 200MHz, the clock provided to users is 200MHz, and the clock for controlling DDR3 is 800MHz. The interface provided by MIG IP for users to access is the AXI4 interface, with a frequency of 200MHz, a data bit width of 128 bits, and an address bit number of 64 bits. The access of other modules to DDR3 ultimately needs to be converted into an access with a frequency of 200MHz, a data bit width of 128 bits, and an address bit number of 64 bits.

[0043] The configuration arbitration module contains two units, one is the arbitration logic unit and the other is the access logic unit. The arbitration logic unit arbitrates the modules that are allowed to access DDR3 at the same time according to the priority of the access logic. The access logic can configure the priority, data width and clock frequency. The data width and clock frequency are set to be configurable for the purpose of accessing the logic of DDR3. It can be a variety of data widths and a variety of clock frequencies. According to the AXI4 protocol, you can access DDR3 by attaching it to the configuration arbitration module, without considering the data width and clock frequency of DDR3. The data cache caches the data accessing DDR3, and the instruction cache caches the instructions accessing DDR3. The configuration arbitration module is implemented using AXI Interconnect RTL. All accesses to DDR3 need to be attached to the AXI Interconnect RTL IP through the AXI4 protocol, and the data of DDR3 is accessed through the AXIInterconnect RTL IP.

[0044] Based on the DDR3 solution implemented above, the main control board combines the optical port control logic, PCIe control logic and FPGA internal image link control logic to implement an image link solution based on optical port and PCIe. The purpose of this solution is to stably and accurately transmit the image data of the flight camera board to the industrial computer. The flight camera board and the main control board transmit the image data in units of one image. The transmission process can be interrupted. The image data is first stored in the cache and then stored in the DDR3 through the cache. The image data transmission process of the industrial control board is as follows: Figure 5As shown, after the flying camera board transmits the image data, the optical port has image data; the image data is stored in the cache (i.e., the second cache module) through the optical port, and when it is detected that there is data in the buffer, the data in the cache is taken out and then stored in DDR3 (i.e., the second DDR module); after the transmission is completed in units of one image, the transmission process is interrupted, and the main control board notifies the industrial computer software to read the address through the PCIe interface to obtain the address and length of the image data storage; the industrial computer software reads the image data from DDR3 through the PCIe interface.

[0045] Therefore, the above-mentioned general multi-bit width and multi-clock domain concurrent read and write DDR3 solution has changed the traditional logic of accessing DDR3, that is, only one read or write operation can be performed at the same time. Under this solution, multiple logics can access DDR3 concurrently, the access instructions will not be lost, and the DDR3 data can be read and written stably; it has changed the mode in which DDR3 can only be accessed with a specific bit width and a specific clock frequency under FPGA. The module accessing DDR3 can access DDR3 with multiple data bit widths and multiple clock frequencies, and the access priority can be set at the same time.

[0046] In an embodiment of the present invention, the flight camera board includes a plurality of flight cameras (such as Figure 1 As shown, 5 flying cameras are selected as an example), a plurality of first cache modules, a first DDR module and an optical port corresponding to the flying cameras one by one, the flying camera is used to receive the flying camera shooting instruction issued by the industrial computer, and perform the shooting action after parsing the instruction; the first cache module is used to store the image data obtained from the flying camera after shooting, and when the cache data reaches the specified number, it is transmitted to the first DDR module; the first DDR module is used to receive the data of the plurality of first cache modules and store them in different areas respectively; the optical port is used to read the data of the first DDR module and send it to the main control board. When the amount of image data in the first cache module is less than the preset value or the reading speed of the first DDR module is greater than the writing speed, the image data in the first cache module is directly transmitted to the optical port without passing through the first DDR module; when the image data fills the read cache FIFO, the subsequent image data is allowed to enter the first DDR module; when the residual data in the memory of the first DDR module is read and the read cache FIFO is not full, the memory transmission path of the first DDR module is skipped, and the image data is directly transferred from the write cache FIFO to the read cache FIFO.

[0047] The function of the flying camera board is to take pictures and upload the image data taken at the same time. In order to allow the flying camera board to allow multiple flying cameras to work simultaneously, distinguish image data and ensure stable uploading of image data, a multi-buffer concurrent multi-flight camera data uploading solution is designed based on the flying camera board of the present invention. This solution includes four aspects: 1. The control logic of the flying camera; 2. The image receiving control logic of multiple flying cameras; 3. The control logic of the optical port; 4. The protocol frame of the image data.

[0048] Regarding the control logic of the flying camera in aspect 1, multiple cameras are allowed to work simultaneously. The schematic diagram of the control logic module of the flying camera is as follows: Figure 6 As shown in the figure, each flying camera corresponds to such a control logic. It is worth noting that the control logic module of the flying camera is built into the flying camera board, and the FPGA in the flying camera board implements the following Figure 6 The flying camera image data control logic is shown.

[0049] The control logic is based on (SC130GS) image sensor, I2C and LVDS. After power-on, FPGA is used as a carrier to control multiple flying cameras to take pictures. The control command of the flying camera comes from the industrial computer. The software in the industrial computer sends the flying camera shooting command, which is sent to the flying camera board through the PCIe and RS485 interfaces. The control logic of the flying camera parses the command and then takes the picture. The register read and write communication between the FPGA of this flying camera board and the sensor uses the I2C interface, and the I2C interface uses a baud rate of 100kbps for communication. The transmission of image data between the FPGA of this flying camera board and the sensor uses 4-channel LVDS communication, which will monitor these related registers. If the configuration status does not meet the designed receiving mode, taking pictures will be prohibited. Data transmission is in line cycles. Each line contains 4n pixels (if it is not an integer multiple of 4, irrelevant data will be filled at the end of each line until it is an integer multiple of 4). Each pixel data is a 1-byte grayscale value. Suppose the data of each pixel in a line is D[0], D[1], D[2], ..., D[4n-2], D[4n-1] in sequence, then the data of a line is in the following format (FS[7:5]=0x5, LS[7:5]=0x1, FE[7:5]=0x6, LE[7:5]=0x2).

[0050] Regarding the image receiving control logic of multiple flight cameras in the second aspect, the corresponding modules of the image receiving control logic of multiple flight cameras are built into the flight camera board, with the first DDR module (DDR3 is selected as an example) and the first cache module of the flight camera board as carriers, such as Figure 7As shown, each flying camera corresponds to a first cache module, and the first cache module first stores the image data taken by the flying camera. When the cached data reaches a certain amount of data, the image receiving logic will take out the cached data and then store it in the DDR memory of the first DDR module. Multiple first cache modules can work directly concurrently. Each first cache module stores different areas in the first DDR module. The image receiving logic divides the area, so that the cache will not overflow, and multiple flying cameras can always work simultaneously. In order to facilitate operation, each area of ​​the first DDR module is encapsulated into a FIFO form, that is, a FIFO function simulation module. The FIFO function simulation module has a built-in write cache FIFO and a read cache FIFO. Each flying camera corresponds to a FIFO function simulation module. The image data is first saved in the write cache and then written to the first DDR module. At the same time, the image receiving logic has an interface occupation arbitration module connected to each FIFO function simulation module to determine whether the FIFO function simulation module can write the write cache data into the first DDR module. The optical port module of the flying camera board corresponds to an image output summary module, and the image output summary module will read the image data in the first DDR module and then send it to the main control board through the optical port. Each FIFO function simulation module is connected to an image control module, which is connected to the image output summary module through the image control module to ensure that the image data of each flight camera can be transmitted stably. The image output summary module is used to control all image control modules. When the image data of a certain flight camera needs to be output, the image output summary module selects the image control module of this flight camera and then outputs the image data of this flight camera. The image control module is used to receive the image data of the flight camera, and then store the image data in the write cache FIFO of the FIFO function simulation module, and at the same time receive the control of the image output summary module, combined with the read cache FIFO of the FIFO function simulation module, to output the image data.

[0051] When the amount of data is small or the read speed is greater than the write speed, the path that the data passes through the first DDR module will be skipped. When the data fills the read cache, subsequent data will be allowed to enter the first DDR module. When the remaining data in the first DDR module is read and the read cache is not full, the first DDR module path will be skipped again to allow data to go directly from the read cache to the write cache. The interface occupancy arbitration module is connected to the first DDR module through the DDR control core. The interface occupancy arbitration module will evaluate the read and write requests for DDR sent by each FIFO function simulation module to decide which one to respond to first. The priority is: write request of the module whose write cache FIFO is about to be full> read request> other write requests.

[0052] For the control logic of the optical port in the third aspect, the control logic of the optical port takes out the image data from the first DDR module and sends the image to the main control board from the optical port, and the flying camera takes pictures and stores the image data in the first DDR module. The two are concurrent and do not interfere with each other. The control logic of the optical port uses the optical port as a physical interface to implement a set of control logic suitable for the transmission of flying camera image data. Exemplarily, the control logic of the optical port uses the Transceivers Wizard IP that comes with xilinx to control access. The communication frequency between the flying camera board and the optical port of the main control board is 156.25MHz, and the data bit width is 32 bits. In order to facilitate data operations, the optical port is encapsulated into an asynchronous interface. The meaning of the encapsulated interface is shown in Table 1. pic_clk comes from the main control board and cannot be lower than 40MHz. Other clocks are arbitrary. The main control board operates on other data in Table 1 based on pic_clk.

[0053] Table 1 Interfaces after optical port encapsulation

[0054]

[0055] The hardware circuit and the IP core of Xilinx have built the physical layer and part of the data link layer in the transmission link. The design of the optical port in the present invention includes the data link layer, the transmission layer and the session layer. The data link layer involves the block and combination parts, such as Figure 8 As shown in the figure, the data of the data link layer is divided into three levels: packet, block, and frame. The frame is further divided into two types: block frame and heart beat frame. Packet is a data unit that interacts with the application layer, for example: a photo with shooting information attached, a remote update bit stream file with remote update additional information attached, and a communication task frame. The size of the packet can be very large, and it interacts with the application layer through the stream mode. After the sender receives the frame from the application layer, it cuts the packet into blocks, and the size of each block has an upper limit (for example, no more than 1k bytes). The block is added to the head and tail of the frame data to obtain a blockframe, which is sent to the receiver through the data link layer and the physical layer. In addition, the two ends of the communication will periodically send heart beat frames that do not contain blocks to each other to synchronize the status information of both sides. After the frame is sent to the receiver, the receiver verifies it and removes the frame head and tail to obtain a block, and then splices the blocks into a packet and sends it to the application layer.

[0056] In order to fully utilize the bandwidth of the channel and increase the data transmission speed, the communication between optical ports adopts full-duplex communication, such as Fig. 9As shown in FIG. 1 , in principle, both parties in communication are in the state of sending frames almost all the time. There is no need to wait for the other party's return frame before sending the next frame. Fig.10 As shown in the figure, when sending data, you don't have to wait for the return frame of the previous data frame to have data before sending the next data frame directly. Each data frame has a frame sequence number, which is accumulated cyclically. If the frame sequence number received by the receiver jumps, it means that there is a missing frame phenomenon, and this data frame is invalid and not accepted. The receiver periodically sends a heartbeat packet to indicate the sequence number of the last valid data frame it received. The sender periodically checks the sequence number of the valid frame returned by the receiver. If this sequence number does not change for a long time, it means that there is a missing frame phenomenon, and it is necessary to resend the frame starting from the returned sequence number + 1.

[0057] If a machine does not receive a frame from the other party within a specified time, it determines that the communication link is temporarily disconnected, stops sending data, and instead polls for heartbeat frames with a connection attempt flag, such as Fig.11 As shown. After the connection is restored, the existence of the other party is confirmed through the following three handshakes: When the machine receives a heartbeat frame with a connection attempt mark, it will return a heartbeat frame with a reconnection mark. After the machine receives a heartbeat frame with a reconnection mark, it returns a normal heartbeat frame. At this time, data can be received but data cannot be sent. After the machine receives a normal heartbeat frame, it means that the communication is successfully established and enters normal mode. After the three reconnection handshakes described above, the communication is restored after the handshake is passed, specifically: the first handshake: receiving an attempt to reconnect frame, replying with a response to reconnect frame; the second time, receiving a response to reconnect frame, replying with a normal heartbeat frame, allowing data to be received; the third time, receiving a normal heartbeat frame, entering normal mode, and allowing data to be sent.

[0058] After the machine on one side is powered on again, the two connected parties will perform a 4-way handshake to establish the connection. Fig.12 As shown in the figure, the restarting party will send a heartbeat frame with a special power-on mark. After receiving it, the detection party will send a heartbeat frame with a special power-on mark. After receiving it, the restarting party will send a normal heartbeat frame. After receiving it, the detection party will also send a normal heartbeat frame. The 4-way handshake is completed. If both parties are restarted, then similarly, both parties are both the restarting party and the detection party.

[0059] The protocol frame of the image data in aspect 4 includes the protocols of Table 1, Table 2 and Table 3. The optical port determines whether a new image data can be sent according to the ready signal given by the main control board. The image transmission between the optical port and the main control board is based on the protocol frames of Table 2 and Table 3.

[0060] Table 2 The image data format stored in the first DDR module of the flight camera board

[0061]

[0062] The communication between the flying camera board and the main control board is carried out according to the prescribed protocol. The flying camera board is the active party and the main control board is the passive party. The flying camera board sends communication frames regularly, and the main control board returns communication frames after receiving them. The frame format sent by the flying camera board is shown in Table 3, and the frame format returned by the main control board is shown in Table 4.

[0063] Table 3 Frame format sent by the flight camera board

[0064]

[0065] Table 4 Main control board return frame

[0066]

[0067] Based on the protocol frame design of image data, the control logic of the optical port can be realized. The protocol frame of image data and the control logic of the optical port are designed in a coordinated manner, so that the transmission of multi-camera image data can be quickly and stably transmitted to the target receiving mode. At the same time, it can distinguish which camera took the picture and when it was taken, which is convenient for the target receiving module to process the picture data.

[0068] Based on the design of the four aspects of the control logic of a single flying camera, the image receiving control logic of multiple flying cameras, the control logic of the optical port and the protocol frame of the image data, a multi-buffer concurrent multi-flight camera data upload solution is implemented, which allows multiple flying cameras to work at the same time, distinguishes image data and ensures stable upload of image data, ensuring that image data can be stably uploaded to the industrial computer.

[0069] In the embodiment of the present invention, the industrial computer has built-in software, PCIe driver and PCIe interface, such as Fig.13 As shown, the software is used to parse the image data and issue image-related commands at the same time, and display the image. For example, the software MCTools can be used. The PCIe driver connects the software and the PCIe interface, provides relevant function functions for the software to use, and identifies the PCIe interface. The PCIe interface is used to connect the main control board and is the channel for data exchange between the industrial computer and the main control board.

[0070] In summary, the technical solution of the present invention achieves the following beneficial effects: 1. The image data received from the optical port can be written into the second DDR module of the main control board with a selectable bit width, and the industrial computer obtains the image data through the PCIe interface control. This image link transmission solution based on the optical port and PCIe solves the problem of cross-clock data conversion between the optical port and PCIe, and realizes the conversion of image data between the optical port and the PCIe interface under the condition of compatibility with the optical port transmission rate and the PCIe rate; 2. Through the concurrent reading and writing of the second DDR module with a general multi-bit width and multi-clock domain, the traditional logic of accessing DDR3 (that is, only one read or write operation can be performed at the same time) is changed. Under this solution, multiple logics can concurrently access the second DDR module, the access instruction will not be lost, and the data of the second DDR module can be stably read and written. At the same time, the mode of accessing the second DDR module under FPGA with only a specific bit width and a specific clock frequency is changed. Taking DDR3 as an example, the module accessing DDR3 can be accessed with multiple data bit widths and multiple clock frequencies, and the access priority can be set. Based on the general multi-bit width and multi-clock domain concurrent reading and writing second DDR module solution, the main control board combines the optical port control logic, PCIe control logic and internal image link control logic together, thereby realizing the image link solution based on optical port and PCIe; 3. Based on the design of the solution in four aspects, namely the control logic of a single flying camera, the image collection control logic of multiple flying cameras, the control logic of the optical port and the protocol frame of the image data, a multi-cache concurrent multi-flight camera data upload solution is realized, which allows multiple flying cameras to work at the same time, distinguishes the image data and ensures the stable upload of the image data, thus ensuring that the image data can be stably uploaded to the industrial computer.

[0071] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity, or apparatus.

[0072] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A multi-camera real-time concurrent data transmission system, characterized in that: Including flight camera board, main control board and industrial computer, The flight camera board communicates with the main control board through an optical port, and is used to actively upload multiple flight camera data to the main control board in a multi-buffer concurrent manner; the flight camera board includes multiple first cache modules and first DDR modules corresponding to the flight cameras one by one, each area of ​​the first DDR module is encapsulated into a FIFO function simulation module, and each of the FIFO function simulation modules includes a read cache FIFO and a write cache FIFO. When the amount of image data in the first cache module is less than a preset value or the read speed of the first DDR module is greater than the write speed, the image data in the first cache module is transmitted directly to the optical port without passing through the first DDR module; when the image data fills the read cache FIFO, the subsequent image data is allowed to enter the first DDR module; when the residual data in the memory of the first DDR module is read and the read cache FIFO is not full, the memory transmission path of the first DDR module is skipped, and the image data is directly transferred from the write cache FIFO to the read cache FIFO; The main control board communicates with the industrial computer via PCIe, is used for passively receiving data from the flight camera board, and transmitting the data to the industrial computer via the PCIe interface.

2. The system according to claim 1, characterized in that The flight camera board includes a plurality of flight cameras and an optical port. The flying camera is used to receive the flying camera photo-taking instruction issued by the industrial computer and perform a photo-taking action after parsing the instruction; The first cache module is used to store image data acquired after taking pictures by the flight camera, and when the cached data reaches a specified amount, it is transmitted to the first DDR module; The first DDR module is used to receive data from a plurality of the first cache modules and store the data in different areas respectively; The optical port is used to read data from the first DDR module and send the data to the main control board.

3. The system according to claim 2, characterized in that The image data from the first cache module is first saved in the write cache FIFO and then written into the memory of the first DDR module. The optical port reads the image data from the first DDR module and stores it in the read cache FIFO first to prevent data loss when reading or writing data.

4. The system according to claim 3, characterized in that The flight camera board also includes an interface occupancy arbitration module, which is used to determine whether the data in the write cache FIFO can be written into the memory of the first DDR module, and to evaluate the read and write requests sent by each of the FIFO function simulation modules to the first DDR module to decide which one to respond to first, and the priority is: write request of the module whose write cache FIFO is about to be full>read request>other write requests.

5. The system according to claim 2, characterized in that The process of the optical port reading the data of the first DDR module and sending it to the main control board is concurrent with the process of the flying camera storing the image data into the first DDR module and does not interfere with each other.

6. The system according to claim 1, characterized in that The optical port between the flight camera board and the main control board adopts full-duplex communication, which specifically includes the following process: when sending data, there is no need to wait for the return frame of the previous data frame to have data, and the next data frame can be sent directly, wherein each data frame has a frame sequence number, and the frame sequence number is accumulated cyclically. If the frame sequence number received by the receiver jumps, it means that there is a frame leakage phenomenon, and the corresponding data frame is invalid and not accepted; The receiver periodically sends a heartbeat packet to indicate the sequence number of the last valid data frame received. The sender periodically checks the sequence number of the valid frame returned by the receiver. If the sequence number does not change within a preset time, it indicates that there is a frame leakage phenomenon. In this case, the frames starting from the returned sequence number + 1 are retransmitted.

7. The system according to claim 6, characterized in that The optical port communication process between the flight camera board and the main control board also includes: If one party does not receive a frame from the other party within a preset time, it determines that the communication link is temporarily disconnected, stops sending data, and instead polls for heartbeat frames with a connection attempt flag. After the connection is restored, the following three handshakes are performed to confirm the existence of the other party: When the machine receives a heartbeat frame with a connection attempt flag, it returns a heartbeat frame with a reconnect flag. After the machine receives the heartbeat frame with the reconnection flag, it returns to the normal heartbeat frame. At this time, it can receive data but cannot send data; After the machine receives the normal heartbeat frame, it means that the communication is successfully established and enters the normal mode.

8. The system according to claim 7, characterized in that After the machine on one side is powered on again, the two connected parties will perform a 4-way handshake to establish a connection. The specific process is as follows: The restarting party will send a heartbeat frame with a special power-on mark; After receiving it, the detection party will send a heartbeat frame with a special power-on mark; After receiving the heartbeat frame, the restarting party will send a normal heartbeat frame instead; After receiving the message, the detecting party also sends a normal heartbeat frame to complete the 4-way handshake.

9. The system according to any one of claims 1 to 8, characterized in that: The main control board is provided with a second DDR module for receiving and storing data from the flight camera board; the second DDR module includes a configuration arbitration module, a DDR control logic module and a DDR storage module, The DDR control logic module is used to receive data from the configuration arbitration module and return relevant data; The configuration arbitration module is used to configure the access logic and arbitrate the devices that are allowed to access the second DDR module at the same time according to the priority of the access logic; The DDR storage module is used to cache accessed data or instructions.

10. The system according to claim 9, characterized in that The configuration arbitration module includes a plurality of access logic units and arbitration logic units respectively connected to the plurality of access logic units. The arbitration logic unit is connected to the DDR control logic module via an AXI4 bus.

11. The system according to claim 10, characterized in that The access logic unit includes access priority, data bit width and clock frequency, and the data bit width and clock frequency are set to be configurable.

12. The system according to claim 9, characterized in that The main control board is also provided with a second cache module. When the main control board is in an idle state and image data arrives at the optical port, the image data is first stored in the second cache module. Whether there is data in the second cache module, if so, the image data is stored in the second DDR module. When an image is stored, the main control board initiates an interrupt to notify the industrial computer that the image data can be read.

Images