Industrial AI Vision Inspection System and Method Based on Task Deconstruction and Hardware-Software Collaboration

Abstract

This invention provides an industrial AI vision inspection system and method based on task deconstruction and hardware-software collaboration, comprising: acquiring high-resolution raw images of the surface of industrial products; performing real-time preprocessing on the high-resolution raw images; scanning the entire image using a feature saliency algorithm to obtain several regions of interest; using a lightweight deep learning model to infer the regions of interest to obtain the defect classification and localization results of the industrial product surface; generating a defect rejection control signal based on the defect results and localization results; sending the defect rejection control signal to the production line actuator to control the production line actuator to screen the corresponding defective industrial products; by deconstructing the detection task at the hardware level and utilizing heterogeneous hardware collaborative processing, on-demand allocation of computing resources is achieved, thereby realizing high-speed, high-precision, and highly robust defect detection at the edge.

Description

Technical Field

[0001] This invention relates to the fields of industrial automation and artificial intelligence technology, and in particular to an industrial AI visual inspection system and method based on task deconstruction and hardware-software collaboration. Background Technology

[0002] Current industrial machine vision inspection is primarily undergoing a transformation from traditional algorithms to deep learning. In high-speed production line environments, existing technical architectures typically consist of local vision processing based on industrial control computers, cloud-based deep learning analysis, and preliminary edge AI inference. While these solutions are effective in specific scenarios, they generally face problems of uneven computing power distribution and response latency when handling inspection tasks with high-resolution images, complex background interference, and extremely high real-time requirements.

[0003] Traditional machine vision solutions based on general-purpose industrial PCs acquire images using industrial cameras and employ traditional image processing algorithms such as template matching, edge detection, and threshold segmentation through vision software like OpenCV or Halcon on the industrial PC. This approach has low hardware requirements, fast processing speed, and is suitable for inspecting standardized products with simple backgrounds and fixed defect features. However, it has extremely poor robustness, is highly sensitive to changes in lighting, material reflections, and product positional shifts, and is prone to false alarms. Furthermore, it cannot handle complex defects, struggles to identify irregularly shaped and finely textured defects, and has a low level of intelligence.

[0004] In cloud-based centralized deep learning detection solutions, the production line terminal is only responsible for image acquisition. Raw images are uploaded to a cloud server or a local central server via a high-speed network, and a high-precision deep neural network is run using a powerful GPU cluster on the server side for full-scale detection. This solution suffers from high transmission latency, limited by network bandwidth and jitter, and cannot meet the millisecond-level real-time feedback requirements of high-speed production lines. Furthermore, bandwidth costs are high; frequent transmission of high-definition raw images consumes significant network bandwidth resources and poses a risk of data privacy breaches.

[0005] The full-scale AI inference solution based on a general-purpose edge computing box deploys edge computing devices with integrated GPUs or NPUs next to the production line. Pre-trained deep learning models are deployed at the edge, and the system performs full-size, full-region deep neural network inference on each frame of high-definition image to achieve localized recognition. However, this solution suffers from significant computational bottlenecks. Edge devices have limited computing power, and performing full-scale deep convolution calculations on large images leads to a substantial drop in frame rate, making it unsuitable for high-speed production lines. Furthermore, it results in severe waste of computational resources, as most areas in industrial images are typically defect-free backgrounds, and performing the same intensity of deep inference on the entire image causes significant ineffective consumption of computational resources.

[0006] Therefore, this invention provides an industrial AI visual inspection system and method based on task deconstruction and hardware-software collaboration. Summary of the Invention

[0007] This invention relates to an industrial AI vision inspection system and method based on task deconstruction and hardware-software collaboration. By deconstructing the inspection task at the hardware level and utilizing heterogeneous hardware collaborative processing, it achieves on-demand allocation of computing resources, thereby enabling high-speed, high-precision, and highly robust defect detection at the edge.

[0008] This invention provides an industrial AI vision inspection system based on task deconstruction and hardware-software collaboration, comprising: The image acquisition module is used to acquire high-resolution raw images of the surface of industrial products, perform real-time preprocessing on the high-resolution raw images, and use a feature saliency algorithm to scan the entire image to obtain several regions of interest. The edge AI computing module is used to perform inference on the region of interest using a lightweight deep learning model to obtain the defect classification and location results of the industrial product surface; The control output module is used to generate a defect rejection control signal based on the defect results and location results, and send the defect rejection control signal to the production line actuator to control the production line actuator to screen the corresponding defective industrial products.

[0009] In one implementable manner, the image acquisition module includes: The image acquisition unit is used to acquire high-resolution raw images of the surface of industrial products through an industrial camera and transmit the raw images to the image preprocessing unit; The image preprocessing unit is used to perform high-frequency feature enhancement processing and histogram equalization processing on the high-resolution original image to obtain an enhanced grayscale image that retains the gradient features of minor defects. The high-frequency feature enhancement processing includes at least one of Gaussian difference filtering or Laplacian edge enhancement. The full-image saliency scanning unit is used to perform full-image scanning based on the high-frequency gradient features of the enhanced grayscale image using multi-scale sliding window technology, calculate the variance of high-frequency features and the proportion of gradient magnitude within each window as feature saliency indicators, and determine and mark suspected defect areas. The region of interest (ROI) cropping unit is used to crop out several corresponding ROIs from the high-resolution original image based on the corresponding coordinates of the marked suspected defective regions in the high-resolution original image, and then adjust the ROIs to a uniform size before transmitting them to the edge AI computing module.

[0010] In one implementable manner, the edge AI computing module includes: The model loading unit is used to load the optimized lightweight deep learning model into the dedicated accelerated memory of the edge computing device to complete the model initialization; The batch inference unit is used to concatenate the regions of interest into batch tensors and input them into a lightweight deep learning model, and output the defect category probability vector and bounding box coordinates corresponding to each region of interest. The result post-processing unit is used to filter low-confidence detection results according to a preset confidence threshold and remove duplicate bounding boxes using a non-maximum suppression algorithm. The defect result generation unit is used to fuse the detection results corresponding to all regions of interest to generate defect classification and location results for the surface of industrial products.

[0011] In one implementable embodiment, the control output module includes: The control signal generation unit is used to generate corresponding defect rejection control signals based on the defect classification and location results. An industrial communication unit is used to send the defect rejection control signal to the production line actuator via a GPIO interface; The local display unit is used to display the defect classification and location results in real time on the local human-computer interaction terminal, and to store all detection results and corresponding region of interest images; The defect screening unit is used to determine the location of the defective industrial products based on the defect rejection control signal, and to control the production line actuator to screen the corresponding defective industrial products.

[0012] In one implementable manner, the industrial communication unit is further configured to: The defect rejection control signal is sent to the production line actuator via industrial Ethernet.

[0013] In one implementable manner, the full-image saliency scanning unit is further configured to: The enhanced grayscale image is divided into several overlapping sliding windows, with window sizes ranging from 64×64 pixels to 512×512 pixels, and the sliding step size is 50% of the window size.

[0014] This invention provides an industrial AI visual inspection method based on task deconstruction and hardware-software collaboration, including: Step 1: Acquire a high-resolution raw image of the surface of the industrial product, perform real-time preprocessing on the high-resolution raw image, and use a feature saliency algorithm to scan the entire image to obtain several regions of interest; Step 2: Use a lightweight deep learning model to infer the region of interest to obtain the defect classification and location results on the surface of the industrial product; Step 3: Generate a defect rejection control signal based on the defect results and location results, and send the defect rejection control signal to the production line actuator to control the production line actuator to screen the corresponding defective industrial products.

[0015] In one feasible embodiment, step 1 includes: Step 11: Acquire high-resolution raw images of the industrial product surface using an industrial camera and transmit the raw images to the image preprocessing unit; Step 12: Perform high-frequency feature enhancement processing and histogram equalization processing on the high-resolution original image to obtain an enhanced grayscale image that retains the gradient features of minor defects. The high-frequency feature enhancement processing includes at least one of Gaussian difference filtering or Laplacian edge enhancement. Step 13: Based on the high-frequency gradient features of the enhanced grayscale image, a full-image scan is performed using multi-scale sliding window technology. The variance of high-frequency features and the proportion of gradient magnitude within each window are calculated as feature significance indicators to determine and mark suspected defect areas. Step 14: Based on the corresponding coordinates of the marked suspected defective regions in the high-resolution original image, crop out several corresponding regions of interest from the high-resolution original image, adjust the regions of interest to a uniform size, and then transmit them to the edge AI computing module.

[0016] The beneficial effects of this invention are as follows: To fundamentally solve the contradiction between the waste of computing power and insufficient real-time performance in traditional solutions, low-latency, high-precision, and highly robust industrial defect detection is achieved at the resource-constrained edge, while significantly reducing hardware costs and data security risks. First, the image acquisition module acquires high-resolution images based on FPGA, preprocesses them to eliminate light reflection interference, and removes more than 90% of defect-free backgrounds through feature saliency scanning, extracting regions of interest (ROIs) with suspected defects. The processing latency is ≤5ms, significantly saving subsequent computing power and improving system robustness. Then, the edge AI computing module loads an optimized lightweight model based on the edge SoC, performs batch inference on the ROIs and post-processes them to generate defect classification and location results. The single RoI inference latency is ≤8ms, and mAP@0.5≥92.5%, balancing detection speed and accuracy. Finally, the control output module generates rejection control signals based on the detection results, which are sent to the actuator via the industrial bus to screen defective products. At the same time, the data is displayed and stored locally, realizing real-time closed-loop control of the production line, ensuring product quality, and eliminating the risk of data leakage.

[0017] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0018] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the composition of an industrial AI vision inspection system based on task deconstruction and hardware-software collaboration in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the workflow of the industrial AI visual inspection method based on task deconstruction and hardware-software collaboration in an embodiment of the present invention. Detailed Implementation

[0020] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0021] Example 1: This example provides an industrial AI vision inspection system based on task deconstruction and hardware-software collaboration, such as... Figure 1 As shown, it includes: The image acquisition module is used to acquire high-resolution raw images of the surface of industrial products, perform real-time preprocessing on the high-resolution raw images, and use a feature saliency algorithm to scan the entire image to obtain several regions of interest. The edge AI computing module is used to perform inference on the region of interest using a lightweight deep learning model to obtain the defect classification and location results of the industrial product surface; The control output module is used to generate a defect rejection control signal based on the defect results and location results, and send the defect rejection control signal to the production line actuator to control the production line actuator to screen the corresponding defective industrial products.

[0022] In this example, the high-resolution raw image represents a digital image of an industrial product surface without any processing, captured by an industrial camera, such as a 2048×2048 pixel image of an aluminum foil surface in high-speed aluminum foil inspection, or a 4096×3000 pixel image of a circuit board in PCB solder joint inspection.

[0023] In this example, real-time preprocessing refers to the bilinear interpolation scaling and histogram equalization operations performed on the original image in a pipeline manner on the FPGA hardware to eliminate the effects of uneven lighting and material reflection, such as the process of improving the image contrast from 0.35 to 0.82 in the inspection of pharmaceutical glass bottles.

[0024] In this example, the feature saliency algorithm represents a calculation method for distinguishing between the background and suspected defect areas in an image. In this invention, the gradient variance calculation method based on a sliding window is mainly adopted, which can be replaced by the frequency domain residual method in complex backgrounds.

[0025] In this example, the region of interest represents the local image region suspected of having defects that is retained after being filtered by the feature saliency algorithm, such as the 19 suspected scratch regions selected from 3969 windows in aluminum foil detection.

[0026] In this example, the lightweight deep learning model represents an improved MobileNet-V3 Large model optimized by knowledge distillation, structured pruning, and INT8 quantization, and introduces the SE attention mechanism to enhance the ability to extract small defect features.

[0027] In this example, defect classification refers to determining the type of defect within the region of interest, such as classifying defects into categories like scratches, stains, cracks, poor solder joints, and bridging.

[0028] In this example, the localization result represents the specific location information of the defect in the original image, output in the form of bounding box coordinates [x,y,w,h], such as the coordinates of a scratch in aluminum foil detection (450,1200).

[0029] In this example, the control signal refers to the electrical signal generated based on the defect detection results to control the actions of the production line actuators, such as a defect rejection trigger signal that can be recognized by the PLC.

[0030] In this example, the production line actuator refers to the automated equipment on the production line used to remove defective products, such as jetting devices, robotic arms, and pusher devices.

[0031] In this example, defective industrial products refer to industrial products with surface defects that affect quality, such as scratched aluminum foil, PCB boards with poor soldering, and medical glass bottles with cracks at the bottom.

[0032] The working principle and beneficial effects of the above technical solution are as follows: To fundamentally solve the contradiction between the waste of computing power and insufficient real-time performance in traditional solutions, low-latency, high-precision, and highly robust industrial defect detection is achieved at the resource-constrained edge, while significantly reducing hardware costs and data security risks. First, the image acquisition module acquires high-resolution images based on FPGA, and after preprocessing to eliminate light reflection interference, more than 90% of defect-free backgrounds are eliminated through feature saliency scanning, and regions of interest (ROIs) suspected of being defects are extracted. The processing latency is ≤5ms, which greatly saves subsequent computing power and improves system robustness. Then, the edge AI computing module loads an optimized lightweight model based on the edge SoC, performs batch inference on the ROIs, and performs post-processing to generate defect classification and location results. The single RoI inference latency is ≤8ms, and mAP@0.5≥92.5%, balancing detection speed and accuracy. Finally, the control output module generates rejection control signals based on the detection results, which are sent to the actuator through the industrial bus to screen defective products. At the same time, the data is displayed and stored locally, realizing real-time closed-loop control of the production line, ensuring product quality, and eliminating the risk of data leakage.

[0033] Example 2: Based on Example 1, the industrial AI vision inspection system based on task deconstruction and hardware-software collaboration, the image acquisition module includes: The image acquisition unit is used to acquire high-resolution raw images of the surface of industrial products through an industrial camera and transmit the raw images to the image preprocessing unit; The image preprocessing unit is used to perform high-frequency feature enhancement processing and histogram equalization processing on the high-resolution original image to obtain an enhanced grayscale image that retains the gradient features of minor defects. The high-frequency feature enhancement processing includes at least one of Gaussian difference filtering or Laplacian edge enhancement. The full-image saliency scanning unit is used to perform full-image scanning based on the high-frequency gradient features of the enhanced grayscale image using multi-scale sliding window technology, calculate the variance of high-frequency features and the proportion of gradient magnitude within each window as feature saliency indicators, and determine and mark suspected defect areas. The region of interest (ROI) cropping unit is used to crop out several corresponding ROIs from the high-resolution original image based on the corresponding coordinates of the marked suspected defective regions in the high-resolution original image, and then adjust the ROIs to a uniform size before transmitting them to the edge AI computing module.

[0034] In this example, histogram equalization refers to an algorithm that enhances image contrast by adjusting the gray-level histogram distribution of an image, such as the process of increasing image contrast from 0.35 to 0.82 in the detection of pharmaceutical glass bottles.

[0035] In this example, the enhanced grayscale image represents a single-channel grayscale image whose pixel value range has been normalized to 0-255 levels after preprocessing, and serves as the input data for subsequent feature saliency scanning.

[0036] In this example, the sliding window technique refers to a method of dividing an image into multiple fixed-size and overlapping local windows and calculating features for each window, such as the 64×64 pixel window and 32 pixel step scanning method used in aluminum foil detection.

[0037] In this example, the suspected defect area refers to the local area of ​​the image that may have a defect after being determined by the feature saliency algorithm and whose gradient variance is higher than a preset threshold, such as the 120 suspected solder joint defect areas screened in PCB solder joint inspection.

[0038] In this example, the coordinates of the suspected defect area represent the location information of the suspected defect area in the original high-resolution image, and are stored in the form of the top-left corner coordinates (x, y) and the area size (w, h).

[0039] In this example, uniform size refers to the fixed input image size required by edge AI deep learning models, such as the default 224×224 pixels or 128×128 pixels specifically for PCB solder joint inspection.

[0040] The working principle and beneficial effects of the above technical solution are as follows: To fundamentally reduce the computational load of subsequent edge AI inference and significantly improve the contrast between defects and background through preprocessing steps, thereby enhancing the system's robustness in complex industrial environments, the system first acquires high-resolution raw images of industrial product surfaces according to the production line's rhythm using an industrial camera. These images are then transmitted in real-time to the FPGA's on-chip BRAM cache via a GigE Vision or LVDS high-speed interface, ensuring the real-time and completeness of image acquisition and providing high-quality raw data for subsequent inspection. A hard synchronization mechanism ensures strict alignment between image acquisition and production line operation, preventing missed or incorrect acquisitions. Next, bilinear interpolation scaling and histogram equalization are sequentially performed on the cached raw images to uniformly adjust raw images of different resolutions to a preset processing size and remap the image's grayscale distribution. This effectively eliminates common industrial issues such as uneven lighting, material reflection, and shadow interference, significantly improving the contrast between defect and background areas while reducing the computational data required for subsequent salient scanning. Furthermore, the enhanced grayscale image is divided into multiple overlapping sliding windows. The algorithm utilizes multiple DSP48E1 computing units on an FPGA to compute the gradient variance feature significance index in each window in parallel. The computation results are compared with a preset threshold, and suspected defect areas with gradient variance higher than the threshold are marked. This allows for full-image background filtering with extremely low latency, ensuring that no potential defect areas are missed. Finally, based on the coordinates of the suspected defect areas marked by the full-image significance scanning unit, the corresponding regions of interest are accurately cropped from the original high-resolution image. All regions of interest are adjusted to the uniform input size required by the edge AI model and transferred in batches to the edge AI computing module through DMA zero-copy technology. This ensures that the image size input to the deep learning model is consistent, facilitating batch inference, while keeping the data transmission latency within 1ms.

[0041] Example 3: Based on Example 1, the industrial AI vision inspection system based on task deconstruction and hardware-software collaboration, the edge AI computing module includes: The model loading unit is used to load the optimized lightweight deep learning model into the dedicated accelerated memory of the edge computing device to complete the model initialization; The batch inference unit is used to concatenate the regions of interest into batch tensors and input them into a lightweight deep learning model, and output the defect category probability vector and bounding box coordinates corresponding to each region of interest. The result post-processing unit is used to filter low-confidence detection results according to a preset confidence threshold and remove duplicate bounding boxes using a non-maximum suppression algorithm. The defect result generation unit is used to fuse the detection results corresponding to all regions of interest to generate defect classification and location results for the surface of industrial products.

[0042] In this example, edge computing devices refer to embedded computing devices deployed alongside the production line that integrate dedicated NPU or GPU acceleration units, such as the NVIDIA Jetson Orin Nano and the Xilinx Zynq UltraScale+ MPSoC.

[0043] In this example, the defect category probability vector represents a one-dimensional array output by the deep learning model that represents the probability that the region of interest belongs to various types of defects. For example, [0.02, 0.01, 0.95, 0.02] correspond to the probabilities of scratches, stains, cracks, and normal, respectively.

[0044] In this example, the bounding box coordinates represent the position and size information of the rectangle surrounding the defect area in the original image, expressed in the form of [x,y,w,h], where x and y are the coordinates of the top left corner, and w and h are the width and height.

[0045] In this example, the preset confidence threshold represents the critical probability value used to determine whether the detection result is valid. It is usually set to 0.5-0.7. Only when the probability of the defect category is higher than this threshold is the result considered reliable.

[0046] In this example, a low-confidence detection result means that the probability of the defect category is lower than the preset confidence threshold. Such results are usually judged as false detections and discarded directly.

[0047] In this example, the nonmaximum suppression algorithm represents a post-processing algorithm for removing duplicate bounding boxes. It compares the confidence scores of overlapping bounding boxes, retains the bounding box with the highest confidence score, and suppresses other bounding boxes with high overlap.

[0048] In this example, the bounding box represents a rectangular box used to mark the location and extent of a defect in the image, and is a visual representation of the defect localization result.

[0049] The working principle and beneficial effects of the above technical solution are as follows: In order to solve the core contradiction between speed and accuracy in traditional edge full inference solutions, the improved MobileNet-V3, which has undergone knowledge distillation, structured pruning, and INT8 quantization optimization, is first applied at system startup. The large model is loaded into the NPU's dedicated accelerated memory on the edge computing device to initialize model parameters and configure hardware acceleration. This fully utilizes the NPU's dedicated computing resources, reduces memory access latency, and lays the foundation for subsequent high-speed inference. Then, it receives multiple regions of interest (ROIs) transmitted from the image acquisition module, concatenates them into batch tensors of uniform size, and inputs them into the lightweight deep learning model. Parallel forward propagation calculations are performed, outputting the defect category probability vector and bounding box coordinates for each ROI. This batch processing significantly improves the NPU's computational utilization and greatly reduces the total inference time for multiple ROIs. Furthermore, low-confidence detection results with probabilities below a preset confidence threshold are filtered out. Then, a non-maximum suppression algorithm is used to remove multiple duplicate bounding boxes corresponding to the same defect, retaining only the bounding box with the highest confidence. This effectively reduces false detections and duplicate detections, improving the accuracy and reliability of the detection results. Finally, the post-processed detection results of all ROIs are fused together, integrating the defect category, location, size, and confidence information to generate a complete defect classification and localization result for the industrial product surface in the current frame. This provides accurate and comprehensive decision-making basis for the subsequent control output module.

[0050] Example 4: Based on Example 1, the industrial AI vision inspection system based on task deconstruction and hardware-software collaboration, wherein the control output module includes: The control signal generation unit is used to generate corresponding defect rejection control signals based on the defect classification and location results. An industrial communication unit is used to send the defect rejection control signal to the production line actuator via a GPIO interface; The local display unit is used to display the defect classification and location results in real time on the local human-computer interaction terminal, and to store all detection results and corresponding region of interest images; The defect screening unit is used to determine the location of the defective industrial products based on the defect rejection control signal, and to control the production line actuator to screen the corresponding defective industrial products.

[0051] In this example, the GPIO interface refers to a general-purpose input / output interface, which is the most commonly used simple digital signal interface in industrial control. It can output high and low level signals to trigger the action of switching devices. In this system, the GPIO interface is used to send defect rejection trigger signals to the PLC or actuators such as jet devices and robotic arms, achieving millisecond-level hard real-time control. The working principle and beneficial effects of the above technical solution are as follows: To fundamentally eliminate the risk of production data leakage and provide data support for quality traceability and continuous system optimization, the system first receives the defect classification and location results output by the edge AI computing module. Based on the defect type, severity, and location information, it generates a digital defect rejection control signal that conforms to industrial control standards. This signal can output differentiated control instructions for different defect types, providing accurate action triggering basis for production line actuators and avoiding false or missed triggers. Then, through a GPIO interface or industrial Ethernet bus, the generated defect rejection control signal is transmitted to the PLC controller of the production line or directly connected to the actuator in hard real-time mode. The transmission latency is as low as 1ms, ensuring that the control signal is synchronized with the production line's operating rhythm. Strict synchronization ensures real-time control of high-speed production lines. Furthermore, defect classification, location, and confidence levels are displayed in real-time on the local human-machine interface terminal, allowing operators to monitor production status. Simultaneously, all detection results and corresponding regions of interest images are encrypted and stored on the local hard drive, eliminating the need to upload original high-definition images. This achieves visualized management of the production process and provides a complete data foundation for subsequent quality traceability, defect analysis, and model iteration. Finally, based on the defect rejection control signal and the production line's operating speed, the system accurately calculates the time it takes for defective products to reach the actuator, synchronously triggering the production line actuator to complete the screening and rejection of defective products. The system can dynamically adjust the triggering timing according to the production line speed, ensuring accurate separation of defective products and preventing the accidental rejection of qualified products.

[0052] Example 5: Based on Example 4, in the industrial AI vision inspection system based on task deconstruction and hardware-software collaboration, the industrial communication unit is further used for: The defect rejection control signal is sent to the production line actuator via industrial Ethernet.

[0053] The working principle and beneficial effects of the above technical solution are as follows: When transmitting defect rejection control signals via industrial Ethernet, the control information, including defect type, location, severity, and triggering timing, is first encapsulated into standard industrial Ethernet protocol data packets such as Modbus TCP and Profinet. These packets are then transmitted to the production line PLC controller or intelligent actuator via twisted pair or optical fiber. After parsing, the receiving end executes the corresponding rejection action and can return execution status feedback. This method overcomes the limitations of GPIO interface, such as short transmission distance and the ability to transmit only simple high and low levels. It supports long-distance transmission of over 100 meters and has stronger anti-electromagnetic interference capabilities. It can adapt to complex and harsh industrial environments and can transmit richer control information to support differentiated defect processing. At the same time, it supports multi-device networking communication, facilitating centralized integration, unified scheduling, and digital management of multiple detection devices and production line control systems.

[0054] Example 6: Based on Example 2, the industrial AI vision inspection system based on task deconstruction and hardware-software collaboration, wherein the full-image saliency scanning unit is further used for: The enhanced grayscale image is divided into several overlapping sliding windows, with window sizes ranging from 64×64 pixels to 512×512 pixels, and the sliding step size is 50% of the window size.

[0055] The working principle and beneficial effects of the above technical solution are as follows: The enhanced grayscale image is divided into several overlapping sliding windows. The window size can be flexibly adjusted within the range of 64×64 pixels to 512×512 pixels according to the size of the detection target, and the sliding step size is fixed at 50% of the window size, so that adjacent windows maintain 0% regional overlap. Then, the feature saliency index is independently calculated for each window to determine the suspected defect area. This method can not only completely avoid the problem of missed detection caused by defects being truncated by the window boundary through the overlapping design, but also accurately match detection targets of different sizes through the adjustable window size. At the same time, this regular window division method is perfectly adapted to the parallel computing architecture of FPGA, which can make full use of multiple DSP computing units to process different windows at the same time, greatly improve the speed of full-image saliency scanning, and ensure the real-time requirements of the system.

[0056] Example 7: This example provides an industrial AI visual inspection method based on task deconstruction and hardware-software collaboration, such as... Figure 2 As shown, it includes: Step 1: Acquire a high-resolution raw image of the surface of the industrial product, perform real-time preprocessing on the high-resolution raw image, and use a feature saliency algorithm to scan the entire image to obtain several regions of interest; Step 2: Use a lightweight deep learning model to infer the region of interest to obtain the defect classification and location results on the surface of the industrial product; Step 3: Generate a defect rejection control signal based on the defect results and location results, and send the defect rejection control signal to the production line actuator to control the production line actuator to screen the corresponding defective industrial products.

[0057] In this example, the high-resolution raw image represents a digital image of an industrial product surface without any processing, captured by an industrial camera, such as a 2048×2048 pixel image of an aluminum foil surface in high-speed aluminum foil inspection, or a 4096×3000 pixel image of a circuit board in PCB solder joint inspection.

[0058] In this example, real-time preprocessing refers to the bilinear interpolation scaling and histogram equalization operations performed on the original image in a pipeline manner on the FPGA hardware to eliminate the effects of uneven lighting and material reflection, such as the process of improving the image contrast from 0.35 to 0.82 in the inspection of pharmaceutical glass bottles.

[0059] In this example, the feature saliency algorithm represents a calculation method for distinguishing between the background and suspected defect areas in an image. In this invention, the gradient variance calculation method based on a sliding window is mainly adopted, which can be replaced by the frequency domain residual method in complex backgrounds.

[0060] In this example, the region of interest represents the local image region suspected of having defects that is retained after being filtered by the feature saliency algorithm, such as the 19 suspected scratch regions selected from 3969 windows in aluminum foil detection.

[0061] In this example, the lightweight deep learning model represents an improved MobileNet-V3 Large model optimized by knowledge distillation, structured pruning, and INT8 quantization, and introduces the SE attention mechanism to enhance the ability to extract small defect features.

[0062] In this example, defect classification refers to determining the type of defect within the region of interest, such as classifying defects into categories like scratches, stains, cracks, poor solder joints, and bridging.

[0063] In this example, the localization result represents the specific location information of the defect in the original image, output in the form of bounding box coordinates [x,y,w,h], such as the coordinates of a scratch in aluminum foil detection (450,1200).

[0064] In this example, the control signal refers to the electrical signal generated based on the defect detection results to control the actions of the production line actuators, such as a defect rejection trigger signal that can be recognized by the PLC.

[0065] In this example, the production line actuator refers to the automated equipment on the production line used to remove defective products, such as jetting devices, robotic arms, and pusher devices.

[0066] In this example, defective industrial products refer to industrial products with surface defects that affect quality, such as scratched aluminum foil, PCB boards with poor soldering, and medical glass bottles with cracks at the bottom.

[0067] The working principle and beneficial effects of the above technical solution are as follows: To fundamentally solve the contradiction between the waste of computing power and insufficient real-time performance in traditional solutions, low-latency, high-precision, and highly robust industrial defect detection is achieved at the resource-constrained edge, while significantly reducing hardware costs and data security risks. First, the image acquisition module acquires high-resolution images based on FPGA, and after preprocessing to eliminate light reflection interference, more than 90% of defect-free backgrounds are eliminated through feature saliency scanning, and regions of interest (ROIs) suspected of being defects are extracted. The processing latency is ≤5ms, which greatly saves subsequent computing power and improves system robustness. Then, the edge AI computing module loads an optimized lightweight model based on the edge SoC, performs batch inference on the ROIs, and performs post-processing to generate defect classification and location results. The single RoI inference latency is ≤8ms, and mAP@0.5≥92.5%, balancing detection speed and accuracy. Finally, the control output module generates rejection control signals based on the detection results, which are sent to the actuator through the industrial bus to screen defective products. At the same time, the data is displayed and stored locally, realizing real-time closed-loop control of the production line, ensuring product quality, and eliminating the risk of data leakage.

[0068] Example 8: Based on Example 7, the industrial AI visual inspection method based on task deconstruction and hardware-software collaboration, step 1 includes: Step 11: Acquire high-resolution raw images of the industrial product surface using an industrial camera and transmit the raw images to the image preprocessing unit; Step 12: Perform high-frequency feature enhancement processing and histogram equalization processing on the high-resolution original image to obtain an enhanced grayscale image that retains the gradient features of minor defects. The high-frequency feature enhancement processing includes at least one of Gaussian difference filtering or Laplacian edge enhancement. Step 13: Based on the high-frequency gradient features of the enhanced grayscale image, a full-image scan is performed using multi-scale sliding window technology. The variance of high-frequency features and the proportion of gradient magnitude within each window are calculated as feature significance indicators to determine and mark suspected defect areas. Step 14: Based on the corresponding coordinates of the marked suspected defective regions in the high-resolution original image, crop out several corresponding regions of interest from the high-resolution original image, adjust the regions of interest to a uniform size, and then transmit them to the edge AI computing module.

[0069] In this example, bilinear interpolation scaling refers to a method of adjusting the image size by calculating the weighted average of the four neighboring pixels around the target pixel, such as the process of scaling industrial images of different resolutions to a uniform 2048×2048 pixels.

[0070] In this example, histogram equalization refers to an algorithm that enhances image contrast by adjusting the gray-level histogram distribution of an image, such as the process of increasing image contrast from 0.35 to 0.82 in the detection of pharmaceutical glass bottles.

[0071] In this example, the enhanced grayscale image represents a single-channel grayscale image whose pixel value range has been normalized to 0-255 levels after preprocessing, and serves as the input data for subsequent feature saliency scanning.

[0072] In this example, the sliding window technique refers to a method of dividing an image into multiple fixed-size and overlapping local windows and calculating features for each window, such as the 64×64 pixel window and 32 pixel step scanning method used in aluminum foil detection.

[0073] In this example, the suspected defect area refers to the local area of ​​the image that may have a defect after being determined by the feature saliency algorithm and whose gradient variance is higher than a preset threshold, such as the 120 suspected solder joint defect areas screened in PCB solder joint inspection.

[0074] In this example, the coordinates of the suspected defect area represent the location information of the suspected defect area in the original high-resolution image, and are stored in the form of the top-left corner coordinates (x, y) and the area size (w, h).

[0075] In this example, uniform size refers to the fixed input image size required by edge AI deep learning models, such as the default 224×224 pixels or 128×128 pixels specifically for PCB solder joint inspection.

[0076] The working principle and beneficial effects of the above technical solution are as follows: To fundamentally reduce the computational load of subsequent edge AI inference and significantly improve the contrast between defects and background through preprocessing steps, thereby enhancing the system's robustness in complex industrial environments, the system first acquires high-resolution raw images of industrial product surfaces according to the production line's rhythm using an industrial camera. These images are then transmitted in real-time to the FPGA's on-chip BRAM cache via a GigE Vision or LVDS high-speed interface, ensuring the real-time and completeness of image acquisition and providing high-quality raw data for subsequent inspection. A hard synchronization mechanism ensures strict alignment between image acquisition and production line operation, preventing missed or incorrect acquisitions. Next, bilinear interpolation scaling and histogram equalization are sequentially performed on the cached raw images to uniformly adjust raw images of different resolutions to a preset processing size and remap the image's grayscale distribution. This effectively eliminates common industrial phenomena such as uneven lighting, material reflection, and shadow interference, significantly improving the contrast between defect and background areas while reducing the computational data required for subsequent salient scanning. Furthermore, the enhanced grayscale image is divided into multiple overlapping sliding windows, utilizing the FPGA... GA's multiple DSP48E1 computing units compute the gradient variance feature significance index in parallel within each window, compare the calculation results with a preset threshold, and mark suspected defect areas where the gradient variance is higher than the threshold. It can complete the background filtering of the entire image with extremely low latency, and the background filtering accuracy can reach more than 98%, ensuring that no potential defect areas are missed. Finally, based on the coordinates of the suspected defect areas marked by the full-image significance scanning unit, the corresponding regions of interest are accurately cropped from the original high-resolution image, and all regions of interest are adjusted to the uniform input size required by the edge AI model. They are then transferred in batches to the edge AI computing module through DMA zero-copy technology, which can ensure that the image size input to the deep learning model is consistent, which is convenient for batch inference, while keeping the data transmission latency within 1ms.

[0077] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An industrial AI vision detection system based on task deconstruction and soft-hard collaboration, characterized in that, include: The image acquisition module is used to acquire high-resolution raw images of the surface of industrial products, perform real-time preprocessing on the high-resolution raw images, and use a feature saliency algorithm to scan the entire image to obtain several regions of interest. The edge AI computing module is used to perform inference on the region of interest using a lightweight deep learning model to obtain the defect classification and location results of the industrial product surface; The control output module is used to generate a defect rejection control signal based on the defect results and location results, and send the defect rejection control signal to the production line actuator to control the production line actuator to screen the corresponding defective industrial products.

2. The industrial AI visual inspection system based on task deconstruction and hardware-software collaboration as described in claim 1, characterized in that, The image acquisition module includes: The image acquisition unit is used to acquire high-resolution raw images of the surface of industrial products through an industrial camera and transmit the raw images to the image preprocessing unit; The image preprocessing unit is used to perform high-frequency feature enhancement processing and histogram equalization processing on the high-resolution original image to obtain an enhanced grayscale image that retains the gradient features of minor defects. The high-frequency feature enhancement processing includes at least one of Gaussian difference filtering or Laplacian edge enhancement. The full-image saliency scanning unit is used to perform full-image scanning based on the high-frequency gradient features of the enhanced grayscale image using multi-scale sliding window technology, calculate the variance of high-frequency features and the proportion of gradient magnitude within each window as feature saliency indicators, and determine and mark suspected defect areas. The region of interest (ROI) cropping unit is used to crop out several corresponding ROIs from the high-resolution original image based on the corresponding coordinates of the marked suspected defective regions in the high-resolution original image, and then adjust the ROIs to a uniform size before transmitting them to the edge AI computing module.

3. The industrial AI visual inspection system based on task deconstruction and hardware-software collaboration as described in claim 1, characterized in that, The edge AI computing module includes: The model loading unit is used to load the optimized lightweight deep learning model into the dedicated accelerated memory of the edge computing device to complete the model initialization; The batch inference unit is used to concatenate the regions of interest into batch tensors and input them into a lightweight deep learning model, and output the defect category probability vector and bounding box coordinates corresponding to each region of interest. The result post-processing unit is used to filter low-confidence detection results according to a preset confidence threshold and remove duplicate bounding boxes using a non-maximum suppression algorithm. The defect result generation unit is used to fuse the detection results corresponding to all regions of interest to generate defect classification and location results for the surface of industrial products.

4. The industrial AI visual inspection system based on task deconstruction and hardware-software collaboration as described in claim 1, characterized in that, The control output module includes: The control signal generation unit is used to generate corresponding defect rejection control signals based on the defect classification and location results. An industrial communication unit is used to send the defect rejection control signal to the production line actuator via a GPIO interface; The local display unit is used to display the defect classification and location results in real time on the local human-computer interaction terminal, and to store all detection results and corresponding region of interest images; The defect screening unit is used to determine the location of the defective industrial products based on the defect rejection control signal, and to control the production line actuator to screen the corresponding defective industrial products.

5. The industrial AI visual inspection system based on task deconstruction and hardware-software collaboration as described in claim 4, characterized in that, The industrial communication unit is also used for: The defect rejection control signal is sent to the production line actuator via industrial Ethernet.

6. The industrial AI visual inspection system based on task deconstruction and hardware-software collaboration as described in claim 2, characterized in that, The full-image saliency scanning unit is also used for: The enhanced grayscale image is divided into several overlapping sliding windows, with window sizes ranging from 64×64 pixels to 512×512 pixels, and the sliding step size is 50% of the window size.

7. An industrial AI visual inspection method based on task deconstruction and hardware-software collaboration, characterized in that, include: Step 1: Acquire a high-resolution raw image of the surface of the industrial product, perform real-time preprocessing on the high-resolution raw image, and use a feature saliency algorithm to scan the entire image to obtain several regions of interest; Step 2: Use a lightweight deep learning model to infer the region of interest to obtain the defect classification and location results on the surface of the industrial product; Step 3: Generate a defect rejection control signal based on the defect results and location results, and send the defect rejection control signal to the production line actuator to control the production line actuator to screen the corresponding defective industrial products.

8. The industrial AI visual inspection method based on task deconstruction and hardware-software collaboration as described in claim 7, characterized in that, Step 1 includes: Step 11: Acquire high-resolution raw images of the industrial product surface using an industrial camera and transmit the raw images to the image preprocessing unit; Step 12: Perform high-frequency feature enhancement processing and histogram equalization processing on the high-resolution original image to obtain an enhanced grayscale image that retains the gradient features of minor defects. The high-frequency feature enhancement processing includes at least one of Gaussian difference filtering or Laplacian edge enhancement. Step 13: Based on the high-frequency gradient features of the enhanced grayscale image, a full-image scan is performed using multi-scale sliding window technology. The variance of high-frequency features and the proportion of gradient magnitude within each window are calculated as feature significance indicators to determine and mark suspected defect areas. Step 14: Based on the corresponding coordinates of the marked suspected defective areas in the high-resolution original image, crop out several corresponding regions of interest from the high-resolution original image, adjust the regions of interest to a uniform size, and then transmit them to the edge AI computing module.

Images