An embedded image processing board card based on multi-vision

Abstract

The application discloses a kind of embedded image processing board based on multi-view vision, belong to embedded image processing technical field.The board includes multi-view camera interface module (support 4-8 ways), multi-view synchronous acquisition and timing alignment module based on FPGA, FPGA+NPU heterogenous processing core, multi-path image pre-processing special hardware acceleration module and frame-level load balancing scheduling module.The application is by FPGA hardware level synchronization, and sub-microsecond multi-path camera synchronization precision is realized, and by FPGA+NPU heterogenous architecture, the efficient cooperation of pre-processing and AI inference is realized, by special hardware acceleration module, low-delay image pre-processing is realized, by frame-level load balancing scheduling, the optimal resource utilization of multi-path parallel inference is realized.The application is suitable for industrial detection, intelligent monitoring, automatic driving assistance, robot vision and other multi-view vision application scenarios.

Description

Technical Field

[0001] This invention relates to the field of embedded image processing technology, and in particular to an embedded image processing board based on multi-view vision, which is especially suitable for application scenarios that require simultaneous acquisition and parallel processing of multiple cameras, such as industrial inspection, intelligent monitoring, autonomous driving assistance, and robot vision. Background Technology

[0002] With the rapid development of machine vision technology, multi-camera collaborative operation has been widely applied in fields such as industrial automation, intelligent transportation, and security monitoring. Traditional monocular or binocular vision systems can no longer meet the growing application demands for large field of view, high precision, and real-time performance. Multi-camera vision systems, by deploying multiple cameras to simultaneously acquire images from different angles, can obtain richer spatial information and a wider field of view, thereby improving the overall perception capability of the system.

[0003] However, multi-view vision systems face numerous technical challenges in practical applications. First, synchronized acquisition and temporal alignment of multiple cameras are fundamental prerequisites for achieving high-quality multi-view visual perception; existing technologies lack efficient hardware-level multi-view synchronization mechanisms. Second, parallel processing of multiple high-resolution images places extremely high performance demands on computing platforms, and traditional single-processor architectures struggle to meet real-time processing requirements. Third, preprocessing of multiple images (such as distortion correction, stitching, and fusion) requires dedicated hardware acceleration; otherwise, it will become a bottleneck in system performance.

[0004] In the existing technology, several related technical solutions have been proposed: Patent CN109413392A(D1) discloses an embedded multi-channel video image acquisition and parallel processing system and method. This solution supports the acquisition and parallel processing of multi-channel video, but it adopts a general multi-channel acquisition architecture and lacks a dedicated synchronization mechanism and hardware acceleration unit for multi-view vision (≥3 channels) scenarios, making it difficult to meet the requirements of multi-view vision systems for frame-level accurate synchronization.

[0005] Patent CN102903074A(D2) discloses an FPGA-based image processing device that utilizes the parallel processing capabilities of the FPGA to achieve image processing functions. However, this solution is based on a single FPGA processing architecture and lacks AI inference acceleration capabilities, failing to meet the growing deep learning inference requirements of current multi-view vision systems.

[0006] Patent CN104835163A(D3) discloses an embedded real-time high-speed binocular vision system image acquisition and processing board for moving target detection, which supports the collaborative operation of dual cameras. However, this solution is limited to binocular vision systems, does not support expansion to multiple cameras (4-8 channels), and lacks dedicated hardware modules for preprocessing such as multi-channel image stitching and fusion.

[0007] In summary, the existing technologies have the following shortcomings: (1) lack of hardware-level synchronous acquisition mechanism to support multiple (≥3) cameras; (2) single processor architecture is difficult to meet the performance requirements of multi-channel image preprocessing and AI inference at the same time; (3) lack of dedicated hardware acceleration module for multi-channel image preprocessing; (4) lack of frame-level load balancing scheduling method for multi-channel parallel inference. Summary of the Invention

[0008] The technical problem to be solved by this invention is: how to provide an embedded image processing board that supports simultaneous acquisition and parallel processing of multiple (≥3) cameras, so as to achieve efficient image acquisition, preprocessing and AI inference acceleration in multi-view vision scenarios.

[0009] To address the aforementioned technical problems, this invention provides an embedded image processing board based on multi-view vision, comprising: (1) Multi-view camera interface module, supporting 4-8 cameras to be connected in parallel via MIPI CSI-2 or LVDS interface, equipped with a multiplexer to realize channel management; (2) Multi-camera synchronous acquisition and timing alignment module, based on FPGA to realize hardware-level frame synchronization, controls all cameras to start exposure on the same clock edge through global trigger signal, and realizes sub-microsecond timing alignment inside FPGA; (3) FPGA+NPU heterogeneous processing architecture, in which FPGA is responsible for image acquisition, preprocessing and scheduling, and NPU is responsible for deep learning inference acceleration. The two interact with each other at high speed through AXI bus and shared DDR4 memory area; (4) A dedicated hardware acceleration module for multi-channel image preprocessing, integrating preprocessing functions such as distortion correction (based on lookup table LUT), image stitching, and multi-channel fusion, all implemented by FPGA hardware logic; (5) Frame-level load balancing scheduling module: Based on the complexity of each image frame and the real-time load status of each NPU core, dynamically allocate inference tasks to achieve optimal scheduling of multi-path parallel inference.

[0010] Beneficial effects Compared with the prior art, the use of this invention patent has the following beneficial effects: (1) This invention achieves hardware-level multi-view synchronous acquisition through FPGA, with synchronization accuracy reaching the sub-microsecond level, far exceeding the software synchronization scheme (millisecond level), ensuring the spatiotemporal consistency of multiple images and providing high-quality input data for subsequent multi-view vision algorithms.

[0011] (2) The present invention adopts an FPGA+NPU heterogeneous processing architecture, giving full play to the respective advantages of FPGA in parallel preprocessing and NPU in AI inference. Compared with a single FPGA solution (such as D2), the AI ​​inference performance is improved by more than 10 times; compared with a general CPU solution, the overall processing latency is reduced by more than 80%.

[0012] (3) The present invention integrates a dedicated hardware preprocessing acceleration module. Distortion correction, splicing, fusion and other operations are executed in parallel in a pipeline manner in the FPGA. The preprocessing delay is less than 1ms / frame, which has a significant advantage over the software implementation scheme (usually 5-20ms / frame).

[0013] (4) The present invention uses a frame-level load balancing scheduling method to dynamically allocate inference tasks based on real-time load and frame complexity, so that the utilization rate of multi-channel NPU cores is kept above 85%, which improves the overall throughput by 30%-50% compared with the static allocation scheme.

[0014] (5) This invention supports flexible expansion of 4-8 cameras. Compared with binocular solutions (such as D3), the field of view coverage is increased by 2-4 times, which can adapt to a wider range of application scenarios. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall hardware structure of the board of the present invention; Figure 2 This is a diagram showing the multi-view camera interface and FPGA connection architecture of the present invention; Figure 3 This is a flowchart of the multi-channel image parallel processing of the present invention; Figure 4 This is a block diagram of the FPGA+NPU heterogeneous processing architecture of the present invention; Figure 5 This is a flowchart of the frame-level load balancing scheduling process of the present invention; Figure 6 This is a block diagram of the multi-channel image preprocessing module of the present invention. Detailed Implementation

[0016] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0017] Example 1: Board Hardware Design like Figure 1As shown, the embedded image processing board based on multi-view vision of the present invention mainly includes the following functional modules: multi-camera interface module (4-8 channels), image acquisition and preprocessing FPGA, FPGA+NPU heterogeneous processing core, DDR4 cache, NPU inference engine, multi-channel synchronous timing controller, communication control module (MCU), system management bus, gigabit Ethernet interface, power management module, board-level clock and reset circuit, and external interface.

[0018] In this embodiment, the multi-camera interface module supports parallel access from eight MIPI CSI-2 interface cameras, each supporting up to 4K@30fps image acquisition. The interface module manages channels through a MIPI CSI-2 / LVDS multiplexer, such as... Figure 2 As shown, the FPGA's internal channel deserialization module deserializes and decodes each data stream, the synchronization alignment module achieves multi-frame-level synchronization, the distortion correction module performs real-time distortion correction based on a pre-generated LUT table, and the cache management module is responsible for data buffering and flow control.

[0019] FPGA+NPU heterogeneous processing architecture, such as Figure 4 As shown. The FPGA is a Xilinx Zynq UltraScale+ series, providing abundant programmable logic resources and an ARM hard-core processor. The NPU uses a domestically produced AI inference chip (such as Cambricon or Horizon series), supporting INT8 / INT4 mixed-precision inference, with a single-chip computing power of over 8 TOPS. The FPGA and NPU are interconnected via an AXI4 bus, sharing a 4GB DDR4 memory area. The FPGA writes pre-processed image frames to the shared memory area through a DMA engine, while the NPU obtains input data through direct memory access.

[0020] Example 2: Multi-view synchronous acquisition and timing alignment mechanism The multi-view synchronous acquisition and timing alignment mechanism of this invention is key to achieving high-quality multi-view visual perception. Traditional software synchronization schemes achieve synchronization by sending trigger commands to each camera separately. However, due to the uncertainty of command transmission and response delays, the synchronization accuracy is usually at the millisecond level, which cannot meet the application requirements of precision measurement and moving target detection.

[0021] This invention employs a hardware global triggering scheme: the FPGA's GPIO pins are connected to the hardware trigger inputs of all cameras via equal-length PCB traces. When the FPGA issues a global trigger signal, all cameras begin exposure on the same clock edge. The FPGA's internal synchronization timing controller accurately records the frame timestamps of each camera and ensures timing alignment of each image frame through sub-microsecond calibration.

[0022] In actual testing, the hardware synchronization scheme of this invention achieved a synchronization accuracy of better than 500ns for 8 cameras, which is far superior to the typical accuracy of software synchronization schemes (1-10ms), thus meeting the requirements of multi-view vision systems for frame-level precise synchronization.

[0023] Example 3: Hardware Acceleration for Multi-channel Image Preprocessing like Figure 6 As shown, the multi-channel image preprocessing module of the present invention is integrated in an FPGA and includes the following sub-modules: (1) White balance / color correction module: performs real-time white balance and color space conversion correction based on the color response characteristics of each camera; (2) Lens Distortion Correction Module (LUT): Real-time distortion correction is achieved based on a lookup table. The LUT is pre-generated during the system initialization phase according to the calibration parameters (intrinsic parameter matrix and distortion coefficients) of each camera. For a 1920×1080 resolution image, the distortion correction time is no more than 0.5ms; (3) Image stitching module: Supports automatic stitching of multiple images, and achieves seamless stitching through feature point matching and transformation matrix calculation; (4) Image fusion module: Supports multi-exposure fusion and multi-focus fusion, and achieves high-quality fusion through weighted average or Laplacian pyramid method.

[0024] All the above preprocessing operations are executed in parallel using an FPGA hardware pipeline. The complete preprocessing pipeline for 8 channels of 1080p images has a latency of less than 1ms / frame, which is 5-20 times faster than CPU software implementations (typically 5-20ms / frame).

[0025] Example 4: Frame-level load balancing scheduling method like Figure 5 As shown, the frame-level load balancing scheduling method of the present invention includes the following steps: Step S1: Receive image frames from each channel and parse the frame metadata, including timestamp, resolution, number of targets, and other information; Step S2: Assess the current load status of each NPU core, including queue depth, progress of ongoing inference tasks, etc. Step S3: Evaluate the complexity of each frame. Complexity metrics include image resolution, target density, scene complexity, etc. Step S4: Based on frame complexity and NPU load status, a weighted scoring strategy is used to allocate frames to the optimal NPU cores; Step S5: Each NPU core executes inference tasks in parallel; Step S6: Aggregate, post-process, and output the inference results.

[0026] Performance test results show that, using the frame-level load balancing scheduling method of this invention, the average utilization of the four NPU cores is increased from about 60% in the static allocation scheme to over 85%, and the overall inference throughput is increased by 30%-50%. Under a typical workload of 8 channels of 1080p@30fps, the end-to-end processing latency (from acquisition to result output) does not exceed 35ms, meeting the requirements for real-time processing.

[0027] Example 5: Performance Data Summary The performance data of this invention under typical working scenarios are as follows: | Performance Indicators | Test Conditions | Test Results | | Camera synchronization accuracy | 8-channel MIPI CSI-2 | <500ns | | Preprocessing latency | 8×1080p | <1ms / frame | AI Inference Latency | YOLOv5s, INT8 | <15ms / frame | End-to-end latency | 8-way end-to-end | <35ms | NPU core utilization | 4-core load balancing | >85% | Total Power Consumption | 8-channel Full Load | <15W | | Board Size | Standard Embedded | 100mm × 80mm | The above data shows that the board of the present invention has excellent real-time processing capabilities and energy efficiency in multi-view vision scenarios.

Claims

1. An embedded image processing board based on multi-view vision, characterized in that, include: A multi-camera interface module is used to support the parallel access of no less than 3 cameras. The interface module is connected to each camera through a MIPICSI-2 or LVDS interface. The multi-camera synchronous acquisition and timing alignment module, implemented on FPGA, is used to control the synchronous exposure of all cameras through a global trigger signal and to complete sub-microsecond timing alignment within the FPGA. The FPGA+NPU heterogeneous processing core is used for image acquisition, preprocessing and task scheduling, while the NPU is used for deep learning inference acceleration. The FPGA and NPU interact with each other through an AXI bus and shared memory. A multi-channel image preprocessing hardware acceleration module, integrated in the FPGA, is used to perform at least one preprocessing operation among distortion correction, image stitching, and multi-channel fusion. The frame-level load balancing scheduling module is used to dynamically allocate inference tasks based on the complexity of each image frame and the real-time load status of each NPU core.

2. The board according to claim 1, characterized in that, The multi-camera interface module supports parallel access of 4-8 cameras, with each camera independently configured with acquisition parameters, including resolution, frame rate, and exposure time.

3. The board according to claim 1, characterized in that, The multi-camera synchronous acquisition and timing alignment module uses a global hardware trigger signal to simultaneously drive the trigger input terminals of all cameras through the GPIO pins of the FPGA, achieving a synchronization accuracy better than 1 microsecond.

4. The board according to claim 1, characterized in that, In the FPGA+NPU heterogeneous processing core, the FPGA and NPU exchange data through a shared DDR4 memory area. The FPGA writes the pre-processed image frames into the shared memory area through the DMA engine, and the NPU obtains input data by directly reading the shared memory area.

5. The circuit board according to claim 1, characterized in that, The distortion correction in the multi-channel image preprocessing hardware acceleration module is implemented using a lookup table (LUT) method. The LUT is pre-generated and stored in the on-chip memory of the FPGA based on the calibration parameters of each camera during the board initialization stage.

6. The board according to claim 1, characterized in that, The scheduling strategy adopted by the frame-level load balancing scheduling module includes: receiving each image frame and parsing frame metadata; evaluating the current load status and queue depth of each NPU core; and allocating frames to the optimal NPU core according to the frame complexity index and load status.

7. The board according to claim 1, characterized in that, It also includes a communication control module, which communicates with an external host via a gigabit Ethernet, PCIe, or USB 3.0 interface to transmit inference results and receive configuration commands.

8. The board according to claim 1, characterized in that, It also includes board-level clock and reset circuitry, providing independent programmable clock sources for FPGAs and NPUs, and supporting multi-clock domain collaborative operation.

9. A multi-view vision image processing method, applied to the board as described in any one of claims 1-8, comprising the following steps: Multiple images are simultaneously acquired via a multi-view camera interface; the acquired images are then time-aligned and preprocessed in the FPGA. The preprocessed image frames are allocated to the NPU core through frame-level load balancing scheduling; the NPU core performs parallel inference on the allocated image frames; and outputs the inference results.

10. An electronic device comprising a memory and a processor, the memory storing computer-executable instructions which, when executed by the processor, implement the method of claim 9.

Images