An image acquisition method based on FPGA and NVIDIA SOC mobile processor

Abstract

The application discloses an image acquisition method based on FPGA and NVIDIA SOC mobile processors, characterized in that FPGA transmits a video stream to the DDR4 of GPU through a PCIE bus, and immediately sends a frame interrupt to the GPU after transmitting a frame of image, so as to identify the transmission completion of the frame of image; the GPU periodically reads out the image and performs image processing under the prompt of the frame interrupt, avoids the time sequence disorder of acquired images caused by too fast or too slow processing speed of the image processing algorithm, and improves the universality of software.

Description

Technical Field

[0001] This invention belongs to the field of foreign object detection and identification technology on airport pavement, specifically relating to an image acquisition method using FPGA and NVIDIA SOC mobile processor. Background Technology

[0002] In 2014, NVIDIA released its first Tegra® K1 SOC mobile processing platform, which integrated an ARM core, a GPU core, and shared physical memory. In recent years, NVIDIA has successively released upgraded products such as the Tegra® X1 SOC, Tegra® X2 SOC, and AGX Xavier mobile processors. These mobile processor platforms support Linux systems and the PCIe bus, and the ARM and GPU cores share a single physical memory, utilizing zero-copy technology to avoid unnecessary data transfer overhead. These mobile processors also support NVIDIA CUDA programming models, allowing developers to leverage CUDA acceleration to develop cutting-edge applications, including image processing fields such as computer vision and deep learning.

[0003] Traditional image processing hardware architectures typically utilize DSP or FPGA+DSP architectures for low frame rate image processing. While FPGA+DSP architectures can meet the requirements for high frame rate image processing, FPGA+GPU is the optimal choice when accelerating algorithms using artificial intelligence algorithms such as deep learning. In particular, combining FPGAs with low-power, miniaturized embedded GPUs can provide more efficient computing power at the same power level.

[0004] V4L2 is a video framework specifically designed for Linux devices. Developers can use the standard workflows provided by V4L2 to operate video devices and acquire image data. It primarily uses a series of callback functions to implement these functions, allowing setting parameters such as the video device's frame rate, video compression format, image size, image buffer, and image acquisition method.

[0005] Currently, when using deep learning algorithms for foreign object detection and identification on airport pavements, the real-time performance of image processing is poor because the image processing algorithm's processing speed is too fast or too slow, causing temporal disorder in image acquisition. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides an image acquisition method based on FPGA and NVIDIA SOC mobile processor.

[0007] The specific plan is as follows:

[0008] An image acquisition method based on FPGA and NVIDIA SOC mobile processor is disclosed. The FPGA transmits the video stream to the DDR4 of the GPU via the PCIe bus. After each frame of the image is transmitted, the FPGA immediately sends a frame interrupt to the GPU to indicate that the transmission of a frame of the image is complete. The GPU periodically reads out the image and performs image processing in response to the frame interrupt.

[0009] The FPGA transmits the video stream to the GPU through the following steps:

[0010] S1): After passing through the format conversion logic module, the video stream enters the DMA transfer engine, which packages the video stream into a TLP layer packet in Mwr32 format.

[0011] S2): The DMA transfer engine calls the TLP TX Engine to continuously transfer the organized MWR32 packets to the PCIe hard core IP;

[0012] S3): The PCIe controller on the GPU side transfers the received MWR32 packets to the GPU's DDR main memory.

[0013] After each frame of video transmission is completed, the FPGA sends a frame interrupt to the GPU side via the DMA transfer engine.

[0014] Before periodically reading out images upon being prompted by a frame interrupt, the GPU includes an initialization step, which is as follows:

[0015] M1): The GPU driver allocates three buffers, namely the first buffer, the second buffer, and the third buffer. The starting address of the first buffer is buf0, the starting address of the second buffer is buf1, and the starting address of the third buffer is Third.

[0016] In the FPGA's DMA transfer engine, registers Q0 and Q1 are configured to store the address of the image buffer;

[0017] During initialization, the starting address buf0 of the first buffer is placed into register Q0, the starting address buf1 of the second buffer is placed into register Q1, and the starting address Third of the third buffer is temporarily stored in the device structure of the GPU driver and is in an idle state.

[0018] M2): The GPU driver starts the DMA engine on the FPGA side to begin image acquisition on the GPU.

[0019] The process by which the GPU periodically reads out images in response to a frame interrupt is as follows:

[0020] STEP1: When the first frame is interrupted, i.e. ISR:0, the buffer corresponding to register Q0 is loaded with valid video. The user takes out the starting address buf0 of the first buffer loaded in register Q0 at this time and submits it to the V4L2 user space. At the same time, the starting address Third of the third buffer is put into register Q0.

[0021] STEP2: When the second frame is interrupted, i.e. ISR:1, the buffer corresponding to register Q1 is loaded with valid video; the user retrieves the starting address buf1 of the second buffer loaded in register Q1 at this time and submits it to the V4L2 user space; at the same time, the starting address buf0 of the first buffer is put into register Q1.

[0022] STEP3: When the 3rd frame is interrupted, i.e. ISR:2 arrives: the buffer corresponding to register Q0 is loaded with valid video; the user retrieves the starting address Third of the third buffer loaded in register Q0 at this time and submits it to the V4L2 user space; at the same time, the starting address buf1 of the second buffer is put into register Q0;

[0023] STEP4: When the 4th frame is interrupted, i.e. ISR: 3, the buffer corresponding to register Q1 is loaded with valid video; the user retrieves the starting address buf0 of the first buffer loaded in register Q1 at this time and submits it to the V4L2 user space; at the same time, the starting address Third of the third buffer is put into register Q1.

[0024] STEP5: When the 5th frame is interrupted, i.e. ISR:4, the buffer corresponding to register Q0 is loaded with valid video; the user retrieves the starting address buf1 of the second buffer loaded in register Q0 at this time and submits it to the V4L2 user space; at the same time, the starting address buf0 of the first buffer is put into register Q0.

[0025] STEP6: When the 6th frame is interrupted, i.e. ISR: 5, the buffer corresponding to register Q1 is loaded with valid video; the user retrieves the starting address Third of the third buffer loaded in register Q1 at this time and submits it to the V4L2 user space; at the same time, the starting address buf1 of the second buffer is put into register Q1.

[0026] At this point, the starting addresses of the buffers loaded in registers Q0 and Q1 are buf0 and buf1, respectively. This is exactly the same as the situation when the first frame interrupt occurs in step STEP1. When subsequent frame interrupts occur, the driver will repeat steps STEP1 to STEP6 in a loop.

[0027] The image processing method is as follows: when the rate at which the buffer fills the image is greater than the rate at which the V4L2 user space algorithm processes the image, that is, when the rate on the output side is less than the rate on the input side, the circular queue formed by the remaining buffer has been filled multiple times by the video stream coming from the input side.

[0028] If the rate at which the image is filled in the buffer is less than the rate at which the V4L2 user space algorithm processes it, meaning the rate at the output side is greater than the rate at the input side, then the V4L2 user space algorithm will repeatedly output the video from the same buffer block in order to maintain the continuity of the output timing.

[0029] This invention discloses an image acquisition method based on FPGA and NVIDIA SOC mobile processor. Under the Linux system, the underlying hardware and logic modules in the FPGA are registered as V4L2 video nodes. Then, the image is moved to the mobile processor's memory through three buffers and provided to the GPU for algorithm processing. This effectively solves the problem of high overall latency when using deep learning algorithms for foreign object detection and recognition on airport runways, providing effective support for improving the real-time performance of image processing. It avoids timing disorder of image acquisition caused by image processing algorithms being too fast or too slow, improving the software's versatility. At the same time, after the GPU driver registers the FPGA as a V4L2 standard video node, users can acquire images according to a standardized method, completely blocking the dependence of the top-level user code on specific hardware design details, greatly reducing the cost of software porting and upgrading, enhancing reliability and maintainability, and facilitating engineering implementation. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the path for the GPU to acquire images via PCIe.

[0031] Figure 2 This is a schematic diagram of a multi-buffered periodic image acquisition process. Detailed Implementation

[0032] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the implementation of the present invention, not all of it. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0033] An image acquisition method based on FPGA and NVIDIA SOC mobile processor employs FPGA+embedded GPU technology. The FPGA transmits the video stream to the GPU's DDR4 memory via the PCIe bus, and immediately sends an interrupt to the GPU after each frame is transmitted to indicate the completion of that frame's transmission. Upon receiving the frame interrupt (FrameIRQ) notification, the GPU can periodically read out the images and perform image processing. Figure 1 , Figure 1 The dashed box in the image represents the internal structure of the FPGA. Path-2 is the path for the GPU to acquire images. The video stream first passes through the format conversion logic module and then enters the DMA transfer engine. The DMA transfer engine packages this video stream into MWR32 format TLP layer packets. Then, the DMA transfer engine calls the TLP TX Engine to continuously transmit the organized MWR32 packets to the PCIe hard core IP, and the GPU-side PCIe controller transfers them to the GPU's DDR main memory. When a frame of video has been completely transmitted according to the MWR32 packets carrying fixed bytes, the DMA transfer engine sends an interrupt to the GPU to notify the user algorithm that the most recent complete image frame available for algorithm analysis is in the GPU's main memory. The user algorithm then immediately begins processing the acquired image.

[0034] like Figure 2 As shown, Q0 and Q1 represent the two registers in the FPGA's DMA transfer engine that store the image buffer address, and user refers to the user space of V4L2.

[0035] For ease of explanation, let's assume that "*" is the starting address of the image buffer allocated in the driver, which is also the DMA buffer. For example, "*" can represent the starting address of any one of the buffers buf0, buf1, and Third.

[0036] use:"* "Q0" means that the starting address of the buffer represented by "*" is placed into register Q0. "Q1" means that the starting address of the buffer represented by "*" is placed into register Q1. "user" refers to submitting the starting address of the buffer represented by "*" to the user layer of V4L2 for algorithm processing.

[0037] For image buffers submitted to the V4L2 user layer, users can obtain them through the standard V4L2 I / O operation IOCTL_DQBUF. After use, IOCTL_QBUF needs to be executed again to return the image buffer to the driver.

[0038] At the same time, the FPGA marked the valid video with a five-pointed star next to the register, and placed it into the corresponding register, such as... Figure 2 As shown, when the first frame interrupt arrives, i.e., when ISR: 0, a five-pointed star is marked next to register Q0, indicating that the FPGA has put the valid video into register Q1. When the second frame interrupt arrives, i.e., when ISR: 1, a five-pointed star is marked next to register Q1, indicating that the FPGA has put the valid video into register Q2.

[0039] During initialization, the GPU driver allocates three buffers: the starting addresses of the three buffers are buf0, buf1, and Third, respectively. Figure 2 As shown, the driver places buf0 and buf1 into registers Q0 and Q1 respectively; while Third is in an idle state and temporarily stored in the driver's device structure. Then the driver enables the DMA engine on the FPGA side, thus starting the GPU's image acquisition process.

[0040] STEP1: When the first frame is interrupted, i.e. ISR:0, the buffer corresponding to register Q0 is loaded with valid video. The user takes out the starting address buf0 of the first buffer loaded in register Q0 at this time and submits it to the V4L2 user space. At the same time, the starting address Third of the third buffer is put into register Q0.

[0041] STEP2: When the second frame is interrupted, i.e. ISR:1, the buffer corresponding to register Q1 is loaded with valid video; the user retrieves the starting address buf1 of the second buffer loaded in register Q1 at this time and submits it to the V4L2 user space; at the same time, the starting address buf0 of the first buffer is put into register Q1.

[0042] STEP3: When the 3rd frame is interrupted, i.e. ISR:2 arrives: the buffer corresponding to register Q0 is loaded with valid video; the user retrieves the starting address Third of the third buffer loaded in register Q0 at this time and submits it to the V4L2 user space; at the same time, the starting address buf1 of the second buffer is put into register Q0;

[0043] STEP4: When the 4th frame is interrupted, i.e. ISR: 3, the buffer corresponding to register Q1 is loaded with valid video; the user retrieves the starting address buf0 of the first buffer loaded in register Q1 at this time and submits it to the V4L2 user space; at the same time, the starting address Third of the third buffer is put into register Q1.

[0044] STEP5: When the 5th frame is interrupted, i.e. ISR:4, the buffer corresponding to register Q0 is loaded with valid video; the user retrieves the starting address buf1 of the second buffer loaded in register Q0 at this time and submits it to the V4L2 user space; at the same time, the starting address buf0 of the first buffer is put into register Q0.

[0045] STEP6: When the 6th frame is interrupted, i.e. ISR: 5, the buffer corresponding to register Q1 is loaded with valid video; the user retrieves the starting address Third of the third buffer loaded in register Q1 at this time and submits it to the V4L2 user space; at the same time, the starting address buf1 of the second buffer is put into register Q1.

[0046] At this point, the starting addresses of the buffers loaded in registers Q0 and Q1 are buf0 and buf1, respectively. This is exactly the same as the situation when the first frame interrupt occurs in step STEP1. When subsequent frame interrupts occur, the driver will repeat steps STEP1 to STEP6 in a loop.

[0047] Because a frame cycle has already passed before the rising edge of the first frame interrupt (ISR:0) arrives, as shown in the attached diagram. Figure 2 The left curly brace of the code is 1st frame INTR. Therefore, in terms of the operation state of the image buffer in the frame interrupt service function, its cycle is 6, that is, every 6 frame interrupts, the layout of the buffer, namely buf0, buf1, and Third in Q0, Q1, and user, will return to the same initial state: buf0 is stored in Q0, buf1 is stored in Q1, and Third is submitted to user.

[0048] During steps STEP1 to STEP6, if the rate at which the buffer fills the image is greater than the rate at which the V4L2 user space algorithm processes the image, that is, the rate on the output side is less than the rate on the input side, this means that before the output buffer block has finished outputting the video it contains, the circular queue formed by the remaining buffers has been filled multiple times by the video stream coming from the input side, at least more than once. This can be understood as "adaptive" frame discarding of the video stream on the input side in order to maintain the continuity of the timing on the output side.

[0049] If the rate at which the buffer fills the image is less than the rate at which the V4L2 user space algorithm processes it, that is, the rate on the output side is greater than the rate on the input side.

[0050] In this embodiment, we assume two adjacent buffer blocks, Buf[N] and Buf[N+1]; where Buf[N] is the current output buffer block, and Buf[N+1] is the input buffer block currently being filled by the input side, and the remaining Buf[*] are in an unfilled state; due to the high output rate, the video of Buf[N] is quickly output, while the input side has not yet finished filling Buf[N+1], and the remaining Buf[*] are in an unfilled state and therefore unusable. So the V4L2 user space algorithm can only continue to stay on Buf[N] and output its video again until the input side finishes filling Buf[N+1]. Only then is Buf[N] returned to the circular input buffer queue to participate in the circular filling of the input side, and at the same time, Buf[N+1] is used as a new output buffer block to participate in the V4L2 user space algorithm. This can be understood as the V4L2 user space algorithm repeatedly outputting the same buffer block of video in order to maintain the continuity of the output timing, which is to say, it performs "adaptive" repeated frame interpolation.

[0051] In this invention, three buffers are used to move images and map them to the V4L2 user space, improving image transmission efficiency. The use of frame discarding or repeated frame interpolation avoids timing disruptions in image acquisition caused by excessively fast or slow image processing algorithms, improving software versatility. Simultaneously, after the GPU driver registers the FPGA as a V4L2 node, users can acquire images using standardized methods, completely eliminating the dependence of top-level user code on specific hardware design details. This significantly reduces software portability and upgrade costs, enhances reliability and maintainability, and facilitates engineering implementation.

[0052] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention.

Claims

1. An image acquisition method based on FPGA and NVIDIA SOC mobile processor, characterized in that: The FPGA transmits the video stream to the GPU's DDR4 via the PCIe bus. After each frame is transmitted, the FPGA immediately sends a frame interrupt to the GPU to indicate that the transmission of a frame is complete. The GPU periodically reads out the image and performs image processing in response to the frame interrupt. Before periodically reading out images upon being prompted by a frame interrupt, the GPU includes an initialization step, which is as follows: M1): The GPU driver allocates three buffers, namely the first buffer, the second buffer, and the third buffer. The starting address of the first buffer is buf0, the starting address of the second buffer is buf1, and the starting address of the third buffer is Third. In the FPGA's DMA transfer engine, registers Q0 and Q1 are configured to store the address of the image buffer; During initialization, the starting address buf0 of the first buffer is placed into register Q0, the starting address buf1 of the second buffer is placed into register Q1, and the starting address Third of the third buffer is temporarily stored in the device structure of the GPU driver and is in an idle state. M2): The GPU driver starts the DMA engine on the FPGA side to begin image acquisition on the GPU; The process by which the GPU periodically reads out images in response to a frame interrupt is as follows: STEP1: When the first frame is interrupted, i.e. ISR:0, the buffer corresponding to register Q0 is loaded with valid video. The user takes out the starting address buf0 of the first buffer loaded in register Q0 at this time and submits it to the V4L2 user space. At the same time, the starting address Third of the third buffer is put into register Q0. STEP2: When the second frame is interrupted, i.e. ISR:1, the buffer corresponding to register Q1 is loaded with valid video; the user retrieves the starting address buf1 of the second buffer loaded in register Q1 at this time and submits it to the V4L2 user space; at the same time, the starting address buf0 of the first buffer is put into register Q1. STEP3: When the 3rd frame is interrupted, i.e. ISR:2 arrives: the buffer corresponding to register Q0 is loaded with valid video; the user retrieves the starting address Third of the third buffer loaded in register Q0 at this time and submits it to the V4L2 user space; at the same time, the starting address buf1 of the second buffer is put into register Q0; STEP4: When the 4th frame is interrupted, i.e. ISR: 3, the buffer corresponding to register Q1 is loaded with valid video; the user retrieves the starting address buf0 of the first buffer loaded in register Q1 at this time and submits it to the V4L2 user space; at the same time, the starting address Third of the third buffer is put into register Q1. STEP5: When the 5th frame is interrupted, i.e. ISR:4, the buffer corresponding to register Q0 is loaded with valid video; the user retrieves the starting address buf1 of the second buffer loaded in register Q0 at this time and submits it to the V4L2 user space; at the same time, the starting address buf0 of the first buffer is put into register Q0. STEP6: When the 6th frame is interrupted, i.e. ISR: 5, the buffer corresponding to register Q1 is loaded with valid video; the user retrieves the starting address Third of the third buffer loaded in register Q1 at this time and submits it to the V4L2 user space; at the same time, the starting address buf1 of the second buffer is put into register Q1. At this point, the starting addresses of the buffers loaded in registers Q0 and Q1 are buf0 and buf1, respectively. This is exactly the same as the situation when the first frame interrupt occurs in step STEP1. When subsequent frame interrupts occur, the driver will repeat steps STEP1 to STEP6 in a loop. The image processing method is as follows: when the rate at which the buffer fills the image is greater than the rate at which the V4L2 user space algorithm processes the image, that is, when the rate on the output side is less than the rate on the input side, the circular queue formed by the remaining buffer has been filled multiple times by the video stream coming from the input side. If the rate at which the image is filled in the buffer is less than the rate at which the V4L2 user space algorithm processes it, meaning the rate at the output side is greater than the rate at the input side, then the V4L2 user space algorithm will repeatedly output the video from the same buffer block in order to maintain the continuity of the output timing.

2. The image acquisition method based on FPGA and NVIDIA SOC mobile processor according to claim 1, characterized in that: The FPGA transmits the video stream to the GPU through the following steps: S1): After passing through the format conversion logic module, the video stream enters the DMA transfer engine, which packages the video stream into a TLP layer packet in Mwr32 format. S2): The DMA transfer engine calls the TLP TX Engine to continuously transfer the organized MWR32 packets to the PCIe hard core IP; S3): The PCIe controller on the GPU side transfers the received MWR32 packets to the GPU's DDR main memory.

3. The image acquisition method based on FPGA and NVIDIA SOC mobile processor according to claim 1, characterized in that: After each frame of video transmission is completed, the FPGA sends a frame interrupt to the GPU side via the DMA transfer engine.

Images