Real-time high-speed visual inspection system based on deep learning

Abstract

The invention discloses a real-time high-speed visual inspection system based on deep learning, and relates to the technical field of crossing of computer vision and industrial automation. Comprising multiple paths of GigE cameras, and the frame rate is greater than or equal to 500fps; the industrial controller carrying a GPU co-processing card realizes high-speed data exchange through a PCIe bus; a real-time task scheduling module based on Linux RT, wherein the priority is greater than 99%; a reasoning tool of a TensorRT engine is integrated, and dynamic input size adjustment is supported; and the image preprocessing module optimized by the CUDA realizes zero-copy video memory mapping. Through collaborative optimization of software and hardware, an NI PXIe-8881 controller and an NVIDIA T4 GPU co-processing card are adopted, and Linux RT real-time task scheduling and self-developed AI Vision Toolkit are combined, so that high-speed synchronous acquisition and low-delay reasoning (the total delay is 3.5 ms) of four cameras at 526.5 fps are realized, and the detection flux reaches 13000 + piece / min. The system supports dynamic input size adjustment, a wide-temperature environment and electromagnetic interference resistance, the deployment period is shortened to 3 days, and the contradiction among real-time performance, precision and stability in an industrial scene is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intersection of computer vision and industrial automation, and specifically relates to a real-time, high-speed visual inspection system based on a deep learning algorithm, which is suitable for parts inspection and sorting scenarios in high-speed industrial production lines. Background Art

[0002] With the advancement of industrial automation, the requirements for inspection systems continue to increase. Traditional visual inspection methods are no longer able to meet the speed and accuracy requirements of modern production lines. This is especially true in high-speed production environments. Implementing real-time, efficient deep learning inspection has become a pressing challenge. Existing hardware platforms and software environments often cannot simultaneously meet the dual requirements of deep learning inference for both computational performance and real-time performance.

[0003] Current mainstream real-time visual inspection technologies are based on two main approaches: First, traditional image processing algorithms rely on manually designed feature extraction (such as SIFT and HOG) and classifiers (such as Support Vector Machines), using a combination of template matching, edge detection, and morphological operations. Second, deep learning detection algorithms are based on convolutional neural network (CNN) detection frameworks (such as the YOLO series, SSD, and EfficientDet), using a hybrid architecture of two-stage detectors (such as Faster R-CNN) and single-stage detectors.

[0004] As industrial automation develops towards high speed and high precision, visual inspection systems face severe challenges in scenarios such as electronics manufacturing and automotive parts sorting. Traditional solutions mainly rely on two types of technologies:

[0005] 1. Detection systems based on traditional image processing: These use artificially designed features (such as SIFT and HOG) and classifiers (such as SVM) combined with template matching or morphological operations. This type of solution can meet basic requirements in static scenes, but has significant drawbacks in high-speed dynamic environments:

[0006] Moving targets (speed > 5m / s) have feature distortion due to motion blur, resulting in a detection success rate of less than 70%;

[0007] Sudden changes in illumination (such as welding flash) can cause the false detection rate to rise to over 20%;

[0008] In scenarios with dense targets (>40 per frame), the traditional non-maximum suppression (NMS) algorithm has a missed detection rate exceeding 15%.

[0009] 2. Deep learning-based detection systems: These use the CNN architecture (such as the YOLO series and SSD). Although this system has improved accuracy, it is limited by real-time performance and hardware bottlenecks.

[0010] Computing power and latency conflict: High-precision models (such as YOLOv5x) at 1080p resolution, even with high-end GPUs (such as Tesla V100), still experience single-frame inference latency exceeding 30ms, making it difficult to meet the requirements of 500fps production lines.

[0011] Poor adaptability to dynamic scenes: Lightweight models (such as MobileNet-SSD) sacrifice accuracy to ensure real-time performance. In scenes with large object scale changes or severe occlusion, mAP drops by over 12% (verified by the COCO dataset).

[0012] Hardware resource limitations: Embedded platforms (such as Jetson Xavier NX) typically consume >30W of power, and INT8 quantization results in >8% loss in detection accuracy for small targets (<32×32 pixels).

[0013] High deployment cost: New scenarios require labeling over 10,000 samples and fine-tuning the model. Post-migration domain adaptation accuracy is generally less than 75%.

[0014] Summary of existing technical bottlenecks:

[0015] The current solution is unable to take into account the four core requirements of real-time performance (delay of less than 5ms), high accuracy (mAP>80%), environmental robustness (anti-motion blur / illumination mutation) and low deployment cost, which restricts its application in industrial high-speed detection scenarios.

[0016] In summary, the present invention proposes a real-time high-speed visual inspection system based on deep learning to solve the above problems. Summary of the Invention

[0017] The purpose of the present invention is to provide a real-time high-speed visual inspection system based on deep learning to solve the existing problems.

[0018] To solve the above technical problems, the present invention is achieved through the following technical solutions:

[0019] The present invention is a real-time high-speed visual inspection system based on deep learning, which includes the following core modules:

[0020] 1. Hardware Architecture

[0021] 4-channel GigE cameras: resolution 720×540, frame rate ≥526.5fps, support for synchronous acquisition;

[0022] PXIe-8881 controller: Equipped with an Intel Xeon 8-core processor, connected to an NVIDIA T4 GPU co-processor card via a PCIe bus, sharing a 64GB DDR4 memory pool;

[0023] PXIe-8245 Ethernet module × 2: realizes synchronous transmission of 4-channel camera data;

[0024] PXIe-8431 communication module: connects to PLC sorting equipment via RS485 bus, with a communication rate of ≥10Mbps.

[0025] 2. Software Architecture

[0026] Linux RT real-time operating system: Critical tasks (image acquisition, model inference) priority > 99%, time jitter rate < 1%;

[0027] Self-developed AI Vision Toolkit: Integrates the TensorRT engine and CUDA optimization operators, supporting automatic compilation of the YOLOv8 model;

[0028] Dynamic Input Adjustment Module: Automatically switches input size (640×640 to 1280×1280) based on load, balancing speed and accuracy.

[0029] 3. Working Principle

[0030] Image acquisition: Camera data is transmitted to the controller via the PXIe-8245 module. NI Vision software maps image pointers directly to GPU memory, reducing memory copies.

[0031] Model inference: LabVIEW RT calls CUDA kernel functions for normalization and scaling. The YOLOv8 model uses the TensorRT engine to achieve single-frame inference time of ≤1.4ms (Batch = 16).

[0032] Sorting control: The detection results are transmitted to the PLC module through shared memory, triggering the servo motor to sort, with a delay of ≤2ms.

[0033] The present invention has the following beneficial effects:

[0034] This invention achieves breakthrough optimization through hardware and software collaborative design. It uses the NIPXIe-8881 controller and GPU co-processing card to increase computing power utilization by 40%, and combines with the PXIe-8245 Ethernet module to achieve high-speed synchronous acquisition of 4 cameras @526.5fps; based on the real-time task scheduling mechanism of Linux RT (priority>99%) and the self-developed AI Vision Toolkit, the inference speed is 30% faster than the traditional solution, the total delay of four parallel channels is only 3.5ms, and the detection throughput reaches 13,000+ pieces / minute. In terms of stability, the system has a failure rate of <0.01% in 72 hours of continuous operation, supports a wide temperature environment of -20℃ to 60℃ and resists 4kV electromagnetic interference. The deployment efficiency is significantly improved. Through the LabVIEW RT modular interface and automatic compilation of the YOLOv8 model into the TensorRT engine, the development cycle is shortened to 3 days, providing an efficient and reliable deep learning solution.

[0035] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 This is a block diagram of the hardware architecture of the system of the present invention;

[0038] Figure 2 This is a flow chart of triggering conditions for dynamic input size adjustment in the present invention. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0040] In the description of the present invention, it should be understood that the terms "upper", "middle", "outer", "inner" and the like indicating directions or positional relationships are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the components or elements referred to must have a specific direction, be constructed and operate in a specific direction, and therefore should not be understood as limiting the present invention.

[0041] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "provided with," "connected," etc., should be understood in a broad sense. For example, "connected" may refer to a fixed connection, a detachable connection, or an integral connection; it may refer to a mechanical connection or an electrical connection; it may refer to a direct connection or an indirect connection through an intermediate medium; it may refer to internal communication between two components. Those skilled in the art will be able to understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0042] See also Figure 1-Figure 2 As shown, the present invention is a real-time high-speed visual inspection system based on deep learning, which includes the following core modules:

[0043] 1. High-speed camera

[0044] 4-channel GigE cameras (resolution 720×540, frame rate ≥526.5fps) are used to capture real-time images of production line parts.

[0045] 2. PXI image acquisition device

[0046] PXIe-8881 controller: Equipped with an Intel Xeon 8-core processor, responsible for overall system scheduling and data processing;

[0047] PXIe-8245 Ethernet module × 2: Connect to 4 cameras and support multi-channel image synchronous acquisition;

[0048] GPU co-processing cards (such as NVIDIA T4): used to accelerate deep learning model inference.

[0049] 3.PLC control system

[0050] Connect to the Inovance servo system via the PXIe-8431 communication module (RS485 / RS422) to achieve high-speed sorting control.

[0051] 4. Software Architecture

[0052] LabVIEW RT: Developed based on the Linux RT operating system, ensuring real-time task scheduling (priority > 99%).

[0053] AI Vision Toolkit: A proprietary inference tool that integrates CUDA and TensorRT acceleration and supports YOLOv8 model optimization.

[0054] The structures and connection relationships in the present invention are as follows:

[0055] 1. Hardware Connection

[0056] Camera and PXIe-8245 module: 4-channel GigE camera directly connected to PXIe-

[0057] 8245 module, realizes synchronous transmission of image data.

[0058] PXIe-8881 and GPU co-processing card: connected via the PCIe bus, share a memory pool (DDR4 64GB), and support high-speed data exchange.

[0059] PLC communication link: The PXIe-8431 module is connected to the PLC sorting equipment via the RS485 bus, with a communication rate of ≥10Mbps.

[0060] 2. Physical layout

[0061] Chassis layout: The PXIe-8881 controller and GPU coprocessor card are installed in the front half of the PXI chassis, while the PXIe-8245 and PXIe-8431 modules are placed in the back half to reduce signal interference.

[0062] Heat dissipation design: Forced air cooling system (dual fan redundancy) ensures stable operation of the hardware in an environment of -20℃~60℃.

[0063] Working principle of the present invention:

[0064] 1. Image acquisition and transmission

[0065] The camera acquires images at 800 fps via the PXIe-8245 module, and the data is transmitted to the PXIe-8881 controller via Ethernet.

[0066] NI Vision software maps image data pointers directly to GPU memory, reducing memory copy overhead.

[0067] 2. Model Inference Process

[0068] Preprocessing: LabVIEW RT calls the CUDA kernel function to normalize and resize the image (Resize to 640×640).

[0069] Parallel inference: The YOLOv8 model is loaded onto the GPU via the TensorRT engine and processed in multiple batches (Batch = 16). Single-frame inference takes 1.4ms.

[0070] Dynamic Optimization: The AI ​​Vision Toolkit automatically adjusts the model input size (640×640 to 1280×1280) based on the load, balancing speed and accuracy.

[0071] The following table compares the YOLOv8 speed on different graphics cards using the TensorRT toolkit:

[0072]

[0073] 3. Sorting control and feedback

[0074] The inference results (detection box coordinates and categories) are passed to the PLC control module through shared memory.

[0075] The PXIe-8431 module triggers the servo motor based on the detection results, with a sorting delay of ≤2ms, enabling the sorting of more than 13,000 parts per minute.

[0076] 4. Real-time guarantee

[0077] Linux RT scheduling: Critical tasks (image acquisition, model inference) are assigned real-time priority, with a time jitter rate of <1%.

[0078] Memory reuse mechanism: GPU memory and host memory use zero-copy technology, reducing bandwidth usage by 40%.

[0079] This embodiment is a model training and optimization of a real-time high-speed visual inspection system based on deep learning:

[0080] 1. Dataset: The COCO industrial augmented dataset contains 100,000 annotated images, covering typical industrial scenes such as welding flash and motion blur.

[0081] 2. Training parameters: YOLOv8 model initial learning rate 0.01, cosine annealing strategy, training cycle 300 epochs;

[0082] 3. Quantized deployment: The FP32 model is quantized to INT8 through TensorRT, and the calibration set contains 2000 representative samples.

[0083] This embodiment is a dynamic optimization algorithm for a real-time high-speed visual inspection system based on deep learning:

[0084] Input size switching threshold: When GPU utilization is greater than 85% or frame rate fluctuation is greater than 5%, the input size is automatically reduced to 640×640;

[0085] Exception handling mechanism: Enable degradation mode (skipping non-critical frames) when data is blocked to ensure continuous system operation.

[0086] It needs further explanation:

[0087] 1. Key performance indicator experimental data

[0088] 1. Test environment description

[0089] Hardware configuration:

[0090] Controller: NIPXIe-8881 (Intel Xeon 8-core processor, 64GB DDR4 memory)

[0091] GPU: NVIDIA T416GB, TensorRT support, CUDA 11.x

[0092] Camera: 4-channel GigE industrial camera, resolution 720×540, frame rate ≥526.5fps

[0093] Communication module: PXIe-8245 (Ethernet module) + PXIe-8431 (RS485)

[0094] Software environment:

[0095] Operating system: Linux RT (real-time kernel), LabVIEW RT environment

[0096] Inference framework: YOLOv8+TensorRT, INT8 quantization model

[0097] Data flow optimization: zero-copy GPU memory mapping + Batch = 16 batches of parallel inference

[0098] 2. Scenario and Dataset Description

[0099] Dataset: Based on the COCO industrial enhancement dataset, a total of 100,000 images, including typical industrial scenes such as welding flash, motion blur, and high-density occlusion

[0100] Deployment environment: high-speed assembly line (object speed up to 8m / s), temperature range -20℃ to 60℃, with 4kV electromagnetic interference simulation capability.

[0101] 3. Key performance indicator verification

[0102]

[0103]

[0104] 2. Triggering Conditions and Algorithm Flow for Dynamic Input Size Adjustment

[0105] 1. Trigger conditions

[0106] The system dynamically adjusts the input image size according to the current operating status. The trigger logic is as follows:

[0107] GPU utilization > 85% (average 3-second sliding window) or

[0108] Frame rate fluctuation > 5% (5 consecutive frames below the mean by 5%).

[0109] 2. Mathematical formula explanation

[0110] Frame rate fluctuation detection:

[0111] Volatility = |Fˉ-Ft|Fˉ×100%\text{Volatility} = \frac{|\bar{F}-F_t|}{\bar{F}}\times 100\%

[0112] Among them, Fˉ\bar{F}: average frame rate of sliding window, FtF_t: current frame rate,

[0113] GPU utilization determination:

[0114] Use the NVIDIA SMI tool to periodically sample (1 Hz) and determine whether the average value over the past three seconds exceeds 85%.

[0115] 3. Hardware parameters

[0116] NVIDIA T4 GPU:

[0117] CUDA core count: 2,560

[0118] Video memory capacity: 16GB GDDR6

[0119] Tensor Cores: 320 (supports FP16 / INT8 / INT4)

[0120] TDP power consumption: 70W

[0121] Interface: PCIe Gen3 x16

[0122] Deep learning acceleration features: support TensorRT, CUDA 11.x, mixed precision computing (FP32 / FP16 / INT8)

[0123] PXIe-8881 Controller:

[0124] CPU model: Intel Xeon W-2278E (8 cores and 16 threads, base frequency 3.4GHz, turbo frequency 5.0GHz)

[0125] Supported memory: 64GB DDR4 ECC

[0126] PCIe version: Gen3×16

[0127] Storage interface: M.2NVMe (read and write speed > 3.5GB / s)

[0128] Bus bandwidth: 24GB / s (high-speed interconnection with GPU)

[0129] Real-time control: Compatible with the NI Linux RT kernel and supports hardware interrupt response priority setting. 4. Software implementation - self-developed AI Vision Toolkit

[0130] The core implementation modules are as follows:

[0131] CUDA optimized operators:

[0132] Image preprocessing uses custom CUDA kernel functions, including:

[0133] cuResizeLinear(): implements bilinear interpolation based on texture memory, supporting dynamic size input;

[0134] cuNormalizeFast(): Parallel normalization within the same batch (Mean / Std Dev normalization);

[0135] cuPack2NCHW(): Converts the image format to the NCHW format required by TensorRT, enabling pipeline parallelism.

[0136] Integration logic with TensorRT:

[0137] Export the YOLOv8 model using ONNX;

[0138] Use TensorRT's Builder to build an Engine and perform layer fusion and tensor elimination.

[0139] Register custom Plugin (for non-standard YOLOv8 Decode);

[0140] Use the above CUDA operator as the plugin input pre-processing module;

[0141] The inference process is controlled by the LabVIEW RT main thread and asynchronously triggers GPU batch execution (Batch=16).

[0142] 5. Accuracy compensation strategy in dynamic optimization mechanism

[0143] The following precision preservation mechanisms have been implemented (enabled when the input is downscaled to 640×640):

[0144] Strategy 1: Local feature enhancement

[0145] Image pyramid compensation is performed on the image after the input resolution is reduced, and a specific region of interest (ROI) interpolation and enlargement is introduced in cuResizeLinear() to ensure that small targets (area < 32×32) do not lose edge information due to downsampling.

[0146] Strategy 2: Dual-resolution detection fusion (lightweight implementation)

[0147] Every N frames (e.g., every 30 frames), a "precision refresh frame" inference is performed using a 1280×1280 size. During post-processing, historical detection frames are integrated to maintain detection continuity and avoid missed detections in a short period of time.

[0148] 3. Model Training and Quantization Deployment Details

[0149] 1. Data enhancement strategy in model training

[0150] A variety of enhancements for industrial scenarios have been adopted, including:

[0151] Motion blur simulation:

[0152] Use OpenCV custom convolution kernel to simulate linear / radial blur, with a maximum kernel length of 15px and random rotation angle;

[0153] Each training image is blurred with different intensities with a probability of 50% to enhance the robustness to high-speed objects.

[0154] Photocatalytic synthesis:

[0155] Gamma transformation (γ∈[0.3,1.5]) is used to simulate local overexposure;

[0156] Synthetic arc / welding flash: Add randomly distributed highlight patches (local saturated pixel values) and perform Gaussian blur edge transition;

[0157] All synthetic augmentations preserve the target mask, ensuring labeling accuracy.

[0158] 2. Specific steps for INT8 quantization deployment

[0159] The calibration process is as follows:

[0160] Cal Set Selection Criteria:

[0161] 2,000 representative images were selected from the COCO Industrial Augmentation dataset, covering all typical defect types and lighting conditions;

[0162] The samples were chosen to have a balanced distribution, and the target sizes ranged from 16×16 to 300×300.

[0163] Calibration execution method:

[0164] Use TensorRT's built-in Entropy Calibrator v2 and enable INT8 precision by setting builder.int8_mode = True after loading the ONNX model.

[0165] At the same time, register the calibrator, traverse the first 1000 images of the calibration set, count the layer tensor distribution and build the quantization table (Scale / Zero Point);

[0166] The second stage of calibration is performed on the remaining images for error analysis and accuracy verification (compared to FP32 results).

[0167] Error compensation method:

[0168] Use KL divergence (Kullback-Leibler divergence) to reconstruct the hierarchical distribution;

[0169] Per-channel quantization is used in the detection layer (YOLO Detect Head) to improve the accuracy of small target detection;

[0170] After calibration, the INT8 inference model is deployed with an mAP loss of <1%, and the inference speed is increased by 42%.

[0171] Alternatives

[0172] Hardware compatibility: The GPU co-processor card can be replaced with A30 or Jetson AGX Orin;

[0173] Communication module: RS485 bus can be replaced with Profinet or EtherCAT protocol.

[0174] Throughout this specification, references to terms such as "one embodiment," "example," and "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0175] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A real-time high-speed visual inspection system based on deep learning, characterized in that: include: Multi-channel GigE cameras, frame rate ≥ 500fps; Industrial controllers equipped with GPU co-processing cards achieve high-speed data exchange through the PCIe bus; Real-time task scheduling module based on Linux RT, priority > 99%; Inference tools integrated with the TensorRT engine, supporting dynamic input size adjustment; CUDA-optimized image preprocessing module to achieve zero-copy memory mapping.

2. A real-time high-speed visual inspection system based on deep learning according to claim 1, characterized in that: The GPU co-processing card is NVIDIA T4, which supports multi-batch reasoning (Batch=16).

3. A real-time high-speed visual inspection system based on deep learning according to claim 1, characterized in that: The triggering condition for the dynamic input size adjustment is that the GPU utilization rate is greater than 85% or the frame rate fluctuation is greater than 5%.

4. A real-time high-speed visual inspection system based on deep learning according to claim 1, characterized in that: The industrial controller is NIPXIe-8881, which is connected to the PXIe-8245 Ethernet module to achieve synchronous acquisition of 4 cameras.

5. The real-time high-speed visual inspection system based on deep learning according to claim 1, characterized in that: The inference tool supports automatic compilation of the YOLOv8 model into the TensorRT engine.

Images