Multi-mode AI collaborative industrial visual defect detection system

Abstract

The invention discloses a multi-mode AI collaborative industrial visual defect detection system, and relates to the technical field of industrial visual defect detection and machine learning, and the system comprises a multi-source image collection module which is composed of a plurality of cameras and sensors and supports multi-mode image parallel collection; the image preprocessing module optimizes the image quality through enhancement, denoising and semantic segmentation; the multi-mode AI model collaboration module integrates three types of core models to realize feature collaboration fusion; the defect hierarchical classification module is used for constructing a secondary classification system association knowledge base; the defect positioning and quantifying module is used for accurately positioning and quantifying defect parameters; the detection result intelligent output module supports multi-form output and alarm; and the closed-loop optimization module is used for updating the model and the knowledge base based on feedback iteration. According to the invention, through multi-mode data fusion and multi-mode AI cooperation, the defect detection precision and scene adaptability are improved; the method has cross-scene rapid adaptation, real-time response and closed-loop optimization capabilities, and adapts to multi-industry detection requirements.

Description

Technical Field

[0001] This invention relates to the fields of industrial visual defect detection and machine learning technology, and in particular to a multi-modal AI collaborative industrial visual defect detection system. Background Technology

[0002] In industrial production, product defect detection is a crucial step in ensuring quality and improving efficiency. As manufacturing transitions towards intelligent manufacturing, traditional detection methods are struggling to meet the demands for high precision, speed, and adaptability. Early manual inspection relied on operator experience, resulting in high subjectivity, low efficiency, and high rates of missed and false detections, and it was unsuitable for high-speed production lines and scenarios requiring the detection of minute defects. With the development of machine vision technology, defect detection systems based on single AI models have gradually been applied. For example, convolutional neural networks are used to extract local image features, or traditional machine learning algorithms are used for defect classification, achieving some success in specific scenarios, but still possessing many limitations.

[0003] Existing AI-based industrial visual defect detection technologies face multiple challenges. Single-model capabilities have inherent limitations. While convolutional neural networks excel at extracting local texture and edge features, they are insufficient at capturing global feature associations and spatial layout information. Transformer models have advantages in global feature learning but lack sensitivity to the details of minute defects. Recurrent neural networks, while capable of processing temporal data, struggle to handle multi-scale feature extraction from static images. Multimodal data utilization is inadequate; most systems rely solely on visible light images, ignoring crucial information such as temperature differences and three-dimensional morphology contained in infrared and depth modal data, failing to comprehensively characterize complex defect features. The diversity and dynamism of industrial scenarios exacerbate detection difficulties. Defect manifestations vary significantly across different materials, and factors such as lighting changes, environmental pollution, and production line vibrations can easily degrade image quality. Existing systems lack effective environmental adaptation mechanisms, resulting in significant fluctuations in detection performance. Small sample defect detection presents a significant challenge. The low incidence of some defects and the scarcity of samples lead to insufficient model training, inadequate generalization ability, and a susceptibility to overfitting.

[0004] Furthermore, existing systems lack end-to-end closed-loop optimization capabilities. Once the model is trained, its parameters are fixed, making dynamic iteration based on production feedback and new defect samples impossible. This leads to a decline in detection accuracy over long-term operation. Defect detection is disconnected from the production process, only enabling defect identification and labeling, failing to trace the root cause of defects, provide targeted process optimization suggestions, and form a closed-loop management system of "detection-analysis-improvement." Deployment flexibility and data security are insufficient. Most systems rely on cloud computing power, resulting in high latency in image data transmission, which cannot meet the real-time detection requirements of high-speed production lines. Moreover, industrial production data contains core process information, and existing systems lack robust access control and encryption mechanisms, posing a risk of data leakage. Simultaneously, maintenance is challenging, lacking intuitive status monitoring and remote maintenance functions, which is detrimental to the long-term stable operation of the system. These problems collectively restrict the large-scale application and intelligent upgrading of industrial vision defect detection technology. Summary of the Invention

[0005] This invention proposes a multi-modal AI collaborative industrial visual defect detection system to solve the problems mentioned in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a multi-modal AI collaborative industrial visual defect detection system, comprising the following modules: Multi-source image acquisition module: Composed of multiple cameras and depth sensors, it is adapted to different industrial scenarios. It achieves multi-device acquisition synchronization through a synchronous clock, dynamically adjusts the acquisition frequency, supports parallel acquisition of multimodal images, and outputs standardized format and timestamp information. Image preprocessing module: Built-in enhancement, denoising and semantic segmentation functions, improve image contrast, remove environmental noise, segment background and target regions, retain effective defect information, and output enhanced single-modal image and multimodal fusion base image; Multi-modal AI model collaboration module: Integrates three types of core models to extract local features, capture global correlations, and analyze temporal evolution patterns. It also integrates inference results through a dynamic weight allocation mechanism to construct a multi-scale, multi-dimensional defect detection model system. Defect Hierarchy Classification Module: Based on industrial defect classification standards, a two-level classification system is constructed to divide defect types and severity levels, link to the defect knowledge base, and complete the automatic matching and labeling of defect types and severity. Defect location and quantification module: It uses mask generation and contour extraction algorithms to locate defect positions, quantifies key dimension parameters through calibration algorithms, outputs defect coordinate information, and forms structured quantitative inspection data; Intelligent output module for test results: Supports multiple output formats, visualized reports containing complete test information, data for system integration, and PLC signal linkage to production line alarm device; Closed-loop optimization module: Based on detection results and production feedback, it uses transfer learning and incremental learning to iteratively update AI model parameters and regularly generate performance evaluation reports.

[0007] Furthermore, it also includes an intelligent data augmentation module, which is divided into two units: offline augmentation and online augmentation. The offline augmentation unit expands the training dataset in a certain way and supports the generation of samples that simulate real defects based on generative adversarial networks. The online augmentation unit performs random cropping, color jittering, brightness adjustment, and blurring processing on the input image in real time during the model inference process. Combined with the diversity and randomness of defects in industrial scenarios, it generates augmented samples that fit the actual detection environment.

[0008] Furthermore, it also includes a defect confidence calibration module, which dynamically calibrates the original confidence level of the multi-modal AI model output by introducing scene adaptation factors and historical data feedback. The calculation expression is as follows: in The confidence level for the defect after calibration. Output the raw confidence score for the model. The scene complexity coefficient. This represents the historical detection accuracy rate in the current scenario. To ensure the reliability of the matching between defect features and knowledge base samples, , , It is the sum of the three weighting coefficients.

[0009] Furthermore, it also includes a cross-scene transfer adaptation module, with a built-in scene feature extractor and model adaptation unit. The scene feature extractor extracts environmental features, product material features, and defect morphology features of different industrial scenarios through deep metric learning, and builds a scene feature vector library. The model adaptation unit is based on domain adaptation technology in transfer learning, adjusts the network layer parameters of the multi-modal AI model, freezes the general feature extraction layer, fine-tunes the scene adaptation layer and output layer, and introduces a domain adversarial training mechanism, so that it can quickly adapt to new detection scenarios without retraining the entire model.

[0010] Furthermore, it also includes a multimodal data fusion module. This module receives multimodal image data output from the image preprocessing module, and simultaneously accesses data from temperature, vibration, and pressure sensors at the production site. It integrates multi-source information through feature-level fusion, and calculates the expression as follows: in This is the fused global feature vector. The number of modes includes visible light, infrared, depth, temperature, vibration, and pressure modes. For the first Weighting coefficients for each mode For the first High-dimensional feature vectors of various modalities For the first The cosine similarity between a modal feature and the average feature of all modalities. For the first The reliability coefficient of modal data is determined by the sensor accuracy and data integrity.

[0011] Furthermore, it also includes a model self-diagnosis and fault tolerance module, with a built-in model performance monitoring unit, fault diagnosis unit and backup model library. The performance monitoring unit collects model indicators in real time, the fault diagnosis unit identifies model problems through the isolated forest algorithm, and the backup model library stores redundant models trained in different scenarios. When a main model failure is detected or the performance drops below a threshold, it automatically switches to the adapted model in the backup model library, while recording fault information and issuing maintenance alarms.

[0012] Furthermore, it also includes a defect tracing and analysis module, which links process flow data, equipment operating parameters, raw material batch information, and environmental monitoring data from the production execution system. By constructing a causal reasoning model through Bayesian networks, it analyzes the key factors that cause defects, generates tracing reports for high-frequency defect types, identifies the process, equipment parameter deviations, or raw material quality issues that cause defects, and provides targeted suggestions for production process optimization.

[0013] Furthermore, it also includes an edge computing deployment module, which uses model lightweighting technology to prune, quantize, and distill multi-modal AI models. It removes redundant network layers and invalid connections through channel pruning, builds a teacher-student model architecture based on knowledge distillation technology, retains the core detection capabilities of the main model, adapts to the hardware resources of edge computing nodes, and supports local deployment on edge servers or embedded devices in the production site.

[0014] Furthermore, it also includes a permission management and data security module, establishing a three-level permission management system to assign different data access permissions and operation permissions to system administrators, testing operators, and maintenance personnel. System administrators have full-function operation permissions, testing operators can only view testing results and export reports, and maintenance personnel can only perform equipment maintenance and parameter adjustments. An operation log auditing function is set up to record all model adjustments, parameter modifications, result exports, and permission changes. The AES-256 encryption algorithm is used to encrypt the storage and transmission of collected image data, testing results, and model parameters.

[0015] Furthermore, it also includes a visual operation and maintenance module, which has a built-in system status monitoring panel, model performance analysis tools, and defect statistical report generation unit. It can intuitively display key equipment indicators through the dashboard, support filtering and querying historical test data by time, product type, and defect type, provide trend analysis and visual chart display, automatically generate daily, weekly, and monthly defect statistical reports, provide model parameter adjustment interface and remote maintenance function, and support operation and maintenance personnel to monitor the system operation status in real time and perform remote fault diagnosis through mobile or PC terminals.

[0016] Compared with existing technologies, the beneficial effects of this invention are: The present invention provides a multi-modal AI collaborative industrial visual defect detection system. Through multi-module collaborative design and technological innovation, it comprehensively solves the pain points of existing technologies and achieves significant improvements in detection accuracy, adaptability, and practicality, thus possessing outstanding technical advantages and application value.

[0017] At the data processing level, the multi-source image acquisition module supports parallel acquisition of visible light, infrared, and depth multimodal images, adapting to defects of different sizes and industrial scenarios. The synchronization clock module ensures data time consistency, providing a comprehensive and high-quality data foundation for subsequent detection. The image preprocessing module effectively improves image quality, preserves core defect information, and reduces the impact of environmental interference on detection results through adaptive enhancement, intelligent denoising, and semantic segmentation techniques. Compared to traditional single preprocessing methods, data validity and reliability are significantly improved. The intelligent data augmentation module expands the training dataset through a combination of offline and online methods, especially supporting the generation of adversarial networks to simulate real defect samples, effectively alleviating the overfitting problem in small-sample defect detection and improving the model's generalization ability.

[0018] At the core detection level, the multi-modal AI collaboration module integrates three core models, achieving the synergistic fusion of local, global, and temporal features through a dynamic weight allocation mechanism. This constructs a multi-scale, multi-dimensional detection system, compensating for the limitations of single models and significantly improving the accuracy of defect identification, effectively detecting both minute and complex defects. The defect confidence calibration module dynamically calibrates detection confidence through scene adaptation factors and historical data feedback, reducing false alarm and false negative rates in complex environments. The multi-modal data fusion module integrates image data and sensor data from the production site, compensating for the information limitations of single data through feature-level fusion, further improving detection reliability, especially suitable for defect detection under complex working conditions.

[0019] In terms of adaptation and optimization, the cross-scenario migration adaptation module, based on transfer learning and domain adversarial training, can quickly adapt to new industrial scenarios without retraining, supports multi-industry deployment, and significantly reduces system promotion costs. The closed-loop optimization module uses a combination of transfer learning and incremental learning to dynamically update model parameters and expand the defect knowledge base based on detection results and production feedback, achieving continuous improvement in detection accuracy. The defect source tracing and analysis module constructs a causal reasoning model through Bayesian networks, links data from the entire production process, clarifies the root causes of defects, and provides process optimization suggestions, achieving closed-loop management from defect detection to process improvement, thus contributing to increased production efficiency.

[0020] At the deployment and security level, the edge computing deployment module reduces computational complexity and storage requirements through model lightweighting technology, supports local deployment, reduces data transmission latency, meets the real-time detection needs of high-speed pipelines, and reduces reliance on cloud computing power. The access control and data security module ensures the security and compliance of industrial production data through a three-level access control system, operation auditing, and high-strength encryption technology, preventing data leakage, loss, or tampering. The visual operation and maintenance module provides intuitive status monitoring, performance analysis, and remote maintenance functions, reducing operation and maintenance difficulty and costs, and ensuring long-term stable system operation.

[0021] Overall, the system realizes intelligent, multimodal, and closed-loop industrial visual defect detection, effectively improving detection accuracy, adaptability, and practicality. It is suitable for diverse industrial scenarios and high-speed production needs, providing comprehensive technical support for quality control in the manufacturing industry, and has broad application value. Attached Figure Description

[0022] Figure 1 This is a schematic block diagram of a multi-modal AI collaborative industrial visual defect detection system proposed in this invention; Figure 2 Grouped bar charts comparing defect detection accuracy in different industrial scenarios; Figure 3 Line graph showing the comparison of generalization ability for small sample defect detection; Figure 4 A horizontal bar chart comparing the response times of different typical defect detection methods; Figure 5 A line graph showing the comparison of detection accuracy retention rate within the model iteration cycle; Figure 6 A bar chart showing the time taken to adapt to different product models. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0025] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified. Furthermore, the terms "installed," "connected," and "linked" should be interpreted broadly; for example, they may refer to a fixed connection, a detachable connection, or an integral connection; they may refer to a mechanical connection or an electrical connection; they may refer to a direct connection or an indirect connection through an intermediate medium; and they may refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. The invention will now be described in further detail with reference to the accompanying drawings.

[0026] Reference Figures 1 to 6 A multi-modal AI collaborative industrial visual defect detection system includes the following modules: The multi-source image acquisition module consists of an area array camera, a line array camera, an infrared thermal imaging camera, and a depth sensor. It is adaptable to different industrial scenarios such as metal components, electronic components, and plastic products. It achieves data synchronization of multiple devices through a synchronization clock module. The acquisition frequency is dynamically adjusted according to the production line speed. It supports parallel acquisition of visible light, infrared, and depth multimodal images and outputs standardized image formats and timestamp information. It is suitable for the detection needs of defects of different sizes from 0.1mm to 10mm. The image preprocessing module incorporates an adaptive enhancement algorithm, an intelligent denoising unit, and a semantic segmentation module. It uses adaptive histogram equalization and the Retinex algorithm to improve image contrast, removes salt-and-pepper noise and Gaussian noise in industrial environments by combining median filtering and wavelet denoising, and uses the U-Net semantic segmentation model to segment the background and detection target regions, retaining effective information related to defects, and outputs enhanced single-modal images and multimodal fusion base images. The multi-modal AI model collaboration module integrates three core models: convolutional neural networks, Transformer models, and recurrent neural networks. The convolutional neural network uses the ResNet50 architecture to extract local defect textures and edge features, the Transformer model captures global feature associations and spatial layout information through a self-attention mechanism, and the recurrent neural network analyzes the defect evolution law in pipeline time-series images based on the LSTM structure. Through a dynamic weight allocation mechanism, the inference results of the three models are synergistically fused to construct a multi-scale, multi-dimensional defect detection model system. The defect classification module is based on the ISO2859 industrial defect classification standard and constructs a two-level classification system. The first-level classification includes 12 types of defects such as surface scratches, cracks, holes, deformation, impurity embedding, and color difference. The second-level classification is divided into four severity levels: minor, general, serious, and fatal, according to the defect size, depth, and scope of impact. It is linked to the industrial product defect knowledge base to realize automatic matching and labeling of defect type and severity. The defect localization and quantification module uses pixel-level mask generation technology and contour extraction algorithm to accurately locate the defect position. It quantifies the key size parameters of the defect, such as length, width, area, and depth, through camera calibration parameters and pixel calibration algorithm, and outputs the coordinate information of the defect in the image coordinate system and the world coordinate system to form structured quantitative detection data. The intelligent output module for detection results supports three output formats: visual reports, JSON data, and PLC signals. The visual reports include information such as defect images, location coordinates, type labels, severity, quantitative parameters, and detection time. The JSON data is used for integration with the production management system. The PLC signals directly link with the production line alarm device, triggering a shutdown alarm when a serious or fatal defect is detected. The closed-loop optimization module iteratively updates the parameters of the multi-modal AI model based on the detection results and production feedback data, using a combination of transfer learning and incremental learning. It automatically expands the defect knowledge base, optimizes the model feature extraction layer and weight allocation strategy for high-frequency defect types, and regularly generates model performance evaluation reports to continuously improve detection accuracy and generalization ability.

[0027] This invention also includes an intelligent data augmentation module, which is divided into offline augmentation and online augmentation units. The offline augmentation unit expands the training dataset through methods such as rotation, flipping, scaling, noise injection, and defect synthesis, and supports the generation of samples that simulate real defects based on generative adversarial networks. The online augmentation unit performs random cropping, color jittering, brightness adjustment, and blurring processing on the input image in real time during the model inference process. Combined with the diversity and randomness of defects in industrial scenarios, it generates augmented samples that fit the actual detection environment, improves the adaptability of the multi-modal AI model to defects under different lighting, angles, and pollution conditions, and reduces the overfitting phenomenon in small sample defect detection.

[0028] This invention also includes a defect confidence calibration module, which dynamically calibrates the original confidence level of the multi-modal AI model output by introducing scene adaptation factors and historical data feedback. The calculation expression is as follows: in The confidence level for the defect after calibration. Output the raw confidence score for the model. The scene complexity coefficient ranges from 0 to 1; the larger the industrial environment interference, the smaller the coefficient value. The historical detection accuracy value for the current scenario ranges from 0 to 1. The reliability of matching defect features with knowledge base samples ranges from 0 to 1. , , The sum of the three weighting coefficients is 1. By dynamically adjusting the weighting coefficients, the system can adapt to different industrial scenarios and reduce the false alarm rate and false negative rate in complex environments.

[0029] This invention also includes a cross-scene transfer adaptation module, which has a built-in scene feature extractor and a model adaptation unit. The scene feature extractor extracts environmental features, product material features and defect morphology features of different industrial scenarios through deep metric learning, and builds a scene feature vector library. The model adaptation unit adjusts the network layer parameters of the multi-modal AI model based on the domain adaptation technology in transfer learning, freezes the general feature extraction layer, fine-tunes the scene adaptation layer and the output layer, and introduces a domain adversarial training mechanism to reduce the distribution shift caused by scene differences. It can quickly adapt to new detection scenarios without retraining the entire model, and supports rapid deployment in multiple industries such as automotive parts manufacturing, semiconductor packaging, and 3C product assembly.

[0030] This invention also includes a multimodal data fusion module. This module receives multimodal image data output from the image preprocessing module, and simultaneously accesses data from temperature sensors, vibration sensors, and pressure sensors at the production site. It integrates multi-source information through feature-level fusion, and the calculated expression is as follows: in This is the fused global feature vector. The number of modes includes visible light, infrared, depth, temperature, vibration, and pressure modes. For the first Weighting coefficients for each mode For the first High-dimensional feature vectors of various modalities For the first The cosine similarity between a modal feature and the average feature of all modalities. For the first The reliability coefficient of multimodal data is determined by the sensor accuracy and data integrity. Multimodal data fusion can compensate for the information limitations of single image data and improve the detection accuracy of both minute and complex defects.

[0031] This invention also includes a model self-diagnosis and fault tolerance module, which incorporates a model performance monitoring unit, a fault diagnosis unit, and a backup model library. The performance monitoring unit collects real-time data on model inference time, confidence distribution, detection accuracy, false negative rate, and false positive rate. The fault diagnosis unit uses the isolated forest algorithm to identify problems such as model overfitting, parameter drift, and hardware failures. The backup model library stores redundant models trained under different scenarios. When a main model failure is detected or its performance drops below a threshold, the system automatically switches to the appropriate model in the backup model library to maintain continuous operation. At the same time, it records fault information and issues maintenance alarms. After the fault is resolved, the main model resumes operation and the parameters are updated synchronously.

[0032] This invention also includes a defect tracing and analysis module, which associates process flow data, equipment operating parameters, raw material batch information, and environmental monitoring data from the production execution system. It constructs a causal reasoning model through a Bayesian network, analyzes the key factors that cause defects, builds a defect-factor correlation graph, generates tracing reports for high-frequency defect types, clarifies the process, equipment parameter deviations, or raw material quality problems that cause defects, provides targeted production process optimization suggestions, and supports writing the tracing results back to the production management system to achieve closed-loop management from defect detection to process improvement.

[0033] This invention also includes an edge computing deployment module, which employs model lightweighting technology to prune, quantize, and distill multi-modal AI models. Channel pruning removes redundant network layers and invalid connections, quantizes model weights to 8-bit integer precision, and constructs a teacher-student model architecture based on knowledge distillation technology. This retains the core detection capabilities of the main model, reduces model computational complexity and storage requirements, adapts to the hardware resources of edge computing nodes, and supports local deployment on edge servers or embedded devices in production environments. This reduces image data transmission latency, controls detection response time to the millisecond level, meets the real-time detection requirements of high-speed pipelines, and reduces reliance on cloud computing power and network bandwidth consumption for data transmission.

[0034] This invention also includes a permission management and data security module, establishing a three-level permission management system to assign different data access permissions and operation permissions to system administrators, testing operators, and maintenance personnel. System administrators have full-function operation permissions, testing operators can only view testing results and export reports, and maintenance personnel can only perform equipment maintenance and parameter adjustments. An operation log auditing function is set up to record all model adjustments, parameter modifications, result exports, and permission changes. The AES-256 encryption algorithm is used to encrypt the storage and transmission of collected image data, testing results, and model parameters. Data backup and integrity verification are performed regularly to prevent data leakage, loss, or tampering, ensuring the security and compliance of industrial production data.

[0035] This invention also includes a visual operation and maintenance module, which has a built-in system status monitoring panel, model performance analysis tools, and defect statistics report generation unit. The dashboard intuitively displays key indicators such as equipment operating status, model detection accuracy, defect distribution, and production line pass rate. It supports filtering and querying historical detection data by time, product type, and defect type, provides trend analysis and visual charts, and automatically generates daily, weekly, and monthly defect statistics reports, including information such as defect type percentage, severity distribution, production line defect occurrence rate, and process improvement effects. It provides a model parameter adjustment interface and remote maintenance function, allowing operation and maintenance personnel to monitor the system operating status in real time and perform remote fault diagnosis through mobile or PC terminals, thereby reducing operation and maintenance costs.

[0036] The following two examples further illustrate the specific implementation of this system: Example 1: Application of Defect Detection in Metal Stamped Parts of Automobiles This embodiment is applied to the defect detection scenario of metal stamping parts for automotive components. The objects to be detected include stamping parts such as door inner panels and engine brackets, which are made of high-strength steel and aluminum alloy. The types of defects to be detected include surface scratches, cracks, holes, deformation, embedded impurities, edge burrs, etc. The defect size ranges from 0.1mm to 8mm. The production line speed is 3 meters / minute. It faces complex environmental interference such as changes in workshop lighting, metal surface reflection, and oil pollution. High-precision, real-time defect detection needs to be achieved through multi-mode AI collaboration.

[0037] The multi-source image acquisition module consists of two area scan cameras, one line scan camera, one infrared thermal imaging camera, and one depth sensor. The area scan cameras are deployed on the upper and lower surfaces of the stamped part to acquire visible light images, the line scan cameras acquire panoramic images along the production line, the infrared thermal imaging camera captures temperature differences in the stamped part caused by defects, and the depth sensor acquires three-dimensional morphological data. A synchronization clock module enables data synchronization across multiple devices, with a timestamp error of less than 1ms. The acquisition frequency is dynamically adjusted to 10 frames per second according to the production line speed, outputting a standardized image format with a resolution of 1920×1080, along with information such as acquisition time and device number.

[0038] The image preprocessing module uses a combination of adaptive histogram equalization and Retinex algorithm to improve the image contrast of reflective areas on the metal surface. Median filtering is used to remove salt-and-pepper noise caused by oil stains, and wavelet denoising is used to eliminate Gaussian noise caused by workshop lighting interference. The U-Net semantic segmentation model is used to segment the stamping part background and the detection area, and irrelevant backgrounds such as fixtures and conveyor belts are removed. Effective information related to defects is retained, and the enhanced single-modal image and the multimodal fusion base image are output.

[0039] The offline enhancement unit of the intelligent data augmentation module expands the training dataset by rotating it from 0° to 360°, flipping it horizontally and vertically, scaling it from 0.8 to 1.2 times, adding noise of different intensities, and synthesizing crack samples of different depths. Based on the generative adversarial network, it generates 5,000 sets of defect samples simulating real scratches and holes. During the model inference process, the online enhancement unit performs random cropping, color jitter ±10%, brightness adjustment ±15%, and blur change processing on the input image in real time to generate enhanced samples that fit the actual detection environment.

[0040] The multi-modal AI model collaboration module integrates ResNet50 convolutional neural network, Transformer model and LSTM recurrent neural network. ResNet50 extracts the texture and edge features of local defects such as scratches and burrs. Transformer captures global feature association and spatial layout information such as deformation and large-area impurity embedding through self-attention mechanism. LSTM analyzes the defect evolution law in pipeline time series images. Through dynamic weight allocation mechanism, the inference weights of the three models are automatically adjusted according to the defect type to achieve collaborative fusion.

[0041] The defect classification module is based on the ISO2859 industrial defect classification standard. The first-level classification includes seven types of defects: surface scratches, cracks, holes, deformation, embedded impurities, edge burrs, and color difference. The second-level classification is divided into four levels: minor, moderate, severe, and fatal, according to the defect size, depth, and scope of influence. For example, scratches less than 1 mm in length are minor defects, and cracks greater than 0.5 mm in depth are severe defects. It is linked to the automotive parts defect knowledge base to automatically match the defect type and severity and label them.

[0042] The defect localization and quantization module employs pixel-level mask generation technology and contour extraction algorithms to accurately locate defects in the image. Through camera calibration parameters and pixel calibration algorithms, it quantifies key dimensional parameters such as the length, width, area, and depth of the defect, outputting the defect's coordinates in both the image coordinate system and the world coordinate system. This forms structured quantified detection data; for example, a 0.3mm diameter hole will have its center coordinates and area of ​​0.07mm recorded. 2 Information such as...

[0043] The defect confidence calibration module introduces scene adaptation factors, historical detection accuracy, and feature matching reliability to dynamically calibrate the original confidence of the model output. It adjusts the scene complexity coefficient based on the workshop light intensity and oil pollution level, and optimizes the weight allocation by combining historical detection data from the past three months, thereby reducing the false alarm rate and false negative rate in complex environments.

[0044] The multimodal data fusion module receives pre-processed visible light, infrared, and depth multimodal image data, and simultaneously accesses data from stamping pressure sensors and mold temperature sensors at the production site. It integrates multi-source information through feature-level fusion to compensate for the information limitations of single image data, especially improving the detection accuracy of micro-cracks and internal impurities.

[0045] The edge computing deployment module adopts model lightweighting technology, which removes redundant network layers of ResNet50 and Transformer models through channel pruning, quantizes model weights to 8-bit integer precision, and builds a teacher-student model architecture based on knowledge distillation technology. After lightweighting, the model size is compressed to 30% of the original size, and the detection response time is controlled within 50ms. It is deployed on the edge server in the production site to achieve local real-time detection.

[0046] Based on detection results and production feedback data, the closed-loop optimization module iteratively updates the parameters of the multi-modal AI model by combining transfer learning and incremental learning for every 10,000 sets of valid detection data. It automatically expands the defect knowledge base, optimizes the edge feature extraction layer and weight allocation strategy of the model for frequently occurring edge burr defects, and generates a model performance evaluation report every month to continuously improve detection accuracy and generalization ability.

[0047] The defect tracing and analysis module links the stamping process parameters, mold operation data, raw material batch information, and workshop environmental monitoring data of the production execution system. It constructs a causal reasoning model through Bayesian network to analyze the key factors that cause defects. For example, for frequently occurring crack defects, it traces the problem back to excessive stamping pressure and low mold temperature, generates a tracing report and provides process optimization suggestions, and writes the results back to the production management system.

[0048] The access control and data security module establishes a three-tier access control system. System administrators have full access to functions such as model parameter adjustment and permission allocation. Inspection operators can only view inspection results and export visual reports, while maintenance personnel can only perform equipment maintenance and parameter viewing. The AES-256 encryption algorithm is used to encrypt the collected image data, inspection results, and model parameters during storage and transmission. Data backup and integrity verification are automatically performed every morning.

[0049] The visual operation and maintenance module intuitively displays key indicators such as equipment operating status, model detection accuracy, defect distribution, and production line pass rate through the dashboard. It supports filtering and querying historical inspection data by time, stamping part type, and defect type, and automatically generates daily, weekly, and monthly defect statistical reports, including information such as defect type percentage, severity distribution, and production line defect occurrence rate. Operation and maintenance personnel can remotely monitor the system's operating status and troubleshoot faults through the PC terminal.

[0050] Example 2: Application of semiconductor packaged chip defect detection This embodiment is applied to the defect detection scenario of semiconductor packaged chips. The detection objects include QFP and BGA type packaged chips. The types of defects to be detected include gold wire breakage, bonding misalignment, package bubbles, pin deformation, chip misalignment, pad contamination, etc. The defect size ranges from 0.1mm to 3mm. The production line speed is 1 meter / minute. It faces challenges such as differences in the light transmittance of packaging materials, difficulty in identifying small defects, and high requirements for adaptability to multiple chip models. High-precision and highly adaptable defect detection needs to be achieved through multi-mode AI collaboration.

[0051] The multi-source image acquisition module consists of two high-resolution area scan cameras, one infrared thermal imaging camera, and one depth sensor. The high-resolution area scan cameras acquire visible light images of the top and bottom of the chip, respectively. The infrared thermal imaging camera captures the thermal conduction differences caused by encapsulation bubbles. The depth sensor acquires data on the three-dimensional shape of the pins and bonding height. A synchronization clock module enables synchronized data acquisition from multiple devices, with a timestamp error of less than 0.5ms. The acquisition frequency is dynamically adjusted to 5 frames per second according to the pipeline speed, outputting a standardized image format with a resolution of 2560×2048, along with information such as chip model and production batch.

[0052] The image preprocessing module employs adaptive histogram equalization and the Retinex algorithm to enhance image contrast in areas with differences in the light transmittance of the packaging material. It removes salt-and-pepper noise caused by pad contamination through median filtering, eliminates Gaussian noise generated by packaging material particles through wavelet denoising, and segments the chip region and background using the U-Net semantic segmentation model. Irrelevant backgrounds such as packaging trays and conveyor belts are removed, while retaining effective information related to defects. The module outputs an enhanced single-modal image and a multimodal fusion base image.

[0053] The offline augmentation unit of the intelligent data augmentation module expands the training dataset by rotating it from 0° to 90°, flipping it horizontally and vertically, scaling it from 0.9 to 1.1 times, adding noise of different intensities, and synthesizing encapsulation bubble samples of different sizes. Based on the generative adversarial network, it generates 3,000 sets of defect samples simulating gold wire breakage and bonding misalignment. During the model inference process, the online augmentation unit performs random cropping, color jitter ±5%, brightness adjustment ±10%, and blur change processing on the input image in real time to adapt to the image features of different encapsulation materials.

[0054] The multi-modal AI model collaboration module integrates ResNet50 convolutional neural network, Transformer model and LSTM recurrent neural network. ResNet50 extracts texture features of local minor defects such as broken gold wires and solder pad contamination. Transformer captures global feature associations such as chip offset and overall pin deformation. LSTM analyzes the packaging process stability in pipeline timing images. Through a dynamic weight allocation mechanism, the inference weights of the three models are automatically adjusted according to the chip model and defect type to achieve collaborative fusion.

[0055] The defect classification module is based on the ISO2859 industrial defect classification standard. The first-level classification includes seven types of defects: gold wire breakage, bond misalignment, package bubble, pin deformation, chip misalignment, pad contamination, and package crack. The second-level classification is divided into four levels according to the scope of the defect's impact: minor, moderate, severe, and fatal. For example, bond misalignment less than 0.2mm is a minor defect, while gold wire breakage is a fatal defect. The module is linked to the semiconductor packaging defect knowledge base to automatically match the defect type and severity and label them.

[0056] The defect localization and quantification module employs pixel-level mask generation technology and contour extraction algorithms to accurately locate defects on the chip. Through camera calibration parameters and pixel calibration algorithms, it quantifies key dimensional parameters such as the defect's length, width, area, and offset, outputting the defect's coordinates in both the image coordinate system and the world coordinate system. This forms structured quantified inspection data; for example, a 0.5mm diameter bubble will have its center coordinates and an area of ​​0.2mm recorded. 2 Information such as...

[0057] The cross-scenario transfer adaptation module incorporates a scene feature extractor and a model adaptation unit. The scene feature extractor uses deep metric learning to extract packaging structure features, pin distribution features, and defect morphology features of different chip models, constructing a scene feature vector library containing 50 chip models. The model adaptation unit is based on domain adaptation techniques in transfer learning, freezing the general feature extraction layers of ResNet50 and Transformer, fine-tuning the scene adaptation layer and output layer, and introducing a domain adversarial training mechanism to reduce the distribution shift caused by different chip models. This eliminates the need to retrain the entire model, and adaptation to new chip models can be completed in just 2 hours.

[0058] The multimodal data fusion module receives pre-processed visible light, infrared, and depth multimodal image data, and simultaneously accesses data from bonding pressure sensors, packaging temperature sensors, and curing time sensors at the production site. By integrating multi-source information through feature-level fusion, it improves the detection accuracy of minute and internal defects, especially for defects that are difficult to detect with a single image, such as packaging bubbles and hidden breaks in gold wires.

[0059] The edge computing deployment module adopts model lightweighting technology, which removes redundant network layers and invalid connections through channel pruning, quantizes model weights to 8-bit integer precision, and builds a teacher-student model architecture based on knowledge distillation technology. After lightweighting, the model size is compressed to 25% of the original size, and the detection response time is controlled within 30ms. It is deployed on embedded devices in the production site to achieve local real-time detection.

[0060] Based on detection results and production feedback data, the closed-loop optimization module iteratively updates the parameters of the multi-modal AI model every 5,000 sets of valid detection data by combining transfer learning and incremental learning, automatically expands the defect knowledge base, optimizes the feature extraction layer and weight allocation strategy of the model for frequently occurring bonding offset defects, and generates a model performance evaluation report every half month to continuously improve detection accuracy and generalization ability.

[0061] The defect tracing and analysis module links the bonding process parameters, packaging temperature curves, raw material batch information, and equipment operation data of the production execution system. It constructs a causal reasoning model through Bayesian networks to analyze the key factors that cause defects. For example, for the frequently occurring packaging bubble defect, it traces the problem to excessive humidity of the packaging material and uneven curing temperature, generates a tracing report and provides process optimization suggestions, and writes the results back to the production management system.

[0062] The access control and data security module establishes a three-level access control system. System administrators have full access to all functions, testing operators can only view testing results and export reports, and maintenance personnel can only perform equipment maintenance and parameter adjustments. The module also includes an operation log audit function to record all operations. Data is stored and transmitted using the AES-256 encryption algorithm and is backed up and its integrity is verified regularly.

[0063] The visual operation and maintenance module intuitively displays key indicators such as equipment operating status, model detection accuracy, defect distribution, and production line pass rate through the dashboard. It supports filtering and querying historical detection data by time, chip model, and defect type, and automatically generates daily, weekly, and monthly defect statistical reports, including information such as defect type percentage, severity distribution, and production line defect occurrence rate. It also supports operation and maintenance personnel to remotely monitor and troubleshoot faults through mobile devices.

[0064] Reference Figure 2 This diagram visually demonstrates the core inspection advantages of this invention across various industrial scenarios, directly addressing the pain points of traditional solutions, such as poor scenario adaptability and low detection accuracy. Traditional solutions employ a single image acquisition and a single AI model architecture, lacking targeted adaptation to different product materials, defect morphologies, and environmental interference in various scenarios. For high-difficulty inspection scenarios like aerospace precision components, the accuracy rate is only 68.9%, and even for conventional 3C product casings, the accuracy rate is less than 82%, failing to meet the quality control requirements of high-end manufacturing. This invention, through core modules such as multi-source multimodal image acquisition, multi-modal AI model collaboration, and cross-scenario transfer adaptation, optimizes model weights and extraction strategies based on product characteristics and defect patterns in different scenarios. The overall detection accuracy remains stable above 95% across all scenarios, reaching as high as 97.5% for 3C product casings. Multi-module collaborative adaptation improves detection accuracy and scenario generalization capabilities from the source, providing solid support for industrial deployment across multiple industries and scenarios.

[0065] Reference Figure 3This figure clearly illustrates the generalization advantages of this invention in small-sample defect detection scenarios, solving the problems of high sample dependence and severe overfitting in small-sample scenarios of traditional solutions. Traditional solutions lack effective data augmentation and feature transfer capabilities. When the number of samples is insufficient, model training is inadequate, making it difficult to capture the core features of defects. In a small-sample scenario of 100 sets, the generalization accuracy is only 56.2%, and even with the sample size increased to 5000 sets, the generalization accuracy is only 83.5%, which cannot meet the detection needs of low-occurrence defects. This invention, through offline synthesis and online enhancement of the intelligent data augmentation module, combined with generative adversarial networks to generate simulated real defect samples, and relying on the general feature extraction capabilities of multi-modal AI models, achieves efficient generalization in small-sample scenarios. The generalization accuracy is still as high as 92.1% in a 100-sample scenario, and approaches 97.1% after the sample size is increased to 5000 sets. The improved small-sample generalization capability significantly reduces the sample collection cost for model training, providing a guarantee for the accurate detection of low-occurrence, high-hazard defects.

[0066] Reference Figure 4 This figure highlights the real-time response advantages of this invention in various defect detection scenarios, overcoming the pain points of traditional solutions, such as long inference times and inability to meet the needs of high-speed production lines. Traditional solutions employ heavy model architectures lacking lightweight optimization and rely on cloud data transmission and inference. Defect detection requires significant computing resources to process image data, with response times generally exceeding 18 milliseconds, and reaching as high as 25.9 milliseconds for complex defects such as encapsulated bubbles, making them unsuitable for high-speed production lines in semiconductors, automotive parts, and other industries. This invention significantly reduces model computational complexity and size through model pruning, quantization, and distillation using edge computing deployment modules. Simultaneously, relying on a highly efficient inference mechanism based on multi-model AI collaboration, it achieves local real-time detection, controlling the response time of various defects to within 4.5 milliseconds, and even requiring only 3.8 milliseconds for minute defects such as broken gold wires. This rapid detection response capability meets the real-time management requirements of high-speed production lines while reducing reliance on cloud computing power and data transmission latency, providing technical support for industrial real-time quality inspection.

[0067] Reference Figure 5This figure clearly reveals the superior accuracy stability of this invention in long-term operation scenarios, overcoming the pain points of traditional solutions where model parameters are fixed and accuracy declines rapidly over time. Traditional solutions lack closed-loop optimization and incremental learning capabilities. After model training, they cannot iteratively update based on production feedback and new defect samples. As equipment ages, the environment changes, and defect morphology evolves, detection accuracy continuously declines, with an accuracy retention rate of only 41.8% after 12 months of operation, requiring frequent model retraining and incurring high maintenance costs. This invention, through a closed-loop optimization module, iteratively updates model parameters based on detection results and production feedback data, using a combination of transfer learning and incremental learning. It automatically expands the defect knowledge base and optimizes the model feature extraction layer for high-frequency defects. After 12 months of operation, the accuracy retention rate remains as high as 94.7%, consistently above 94%. This improved long-term accuracy stability significantly reduces the cost of model retraining and manual maintenance, extends the effective operating cycle of the system, and possesses outstanding engineering application value.

[0068] Reference Figure 6 This diagram visually demonstrates the high efficiency of this invention in multi-product model adaptation scenarios, solving the problem of traditional solutions requiring retraining and significant time consumption when adapting to new models. Traditional solutions lack cross-scenario transfer and adaptation capabilities. When faced with new product models, a large number of samples need to be collected again and the entire model retrained, with adaptation time generally exceeding 10 hours, and reaching as high as 16.8 hours for engine bracket stamping parts, significantly impacting production line changeover and new product launch efficiency. This invention, through a cross-scenario transfer and adaptation module, extracts common features from different product models and freezes the corresponding network layers. Only minor adjustments to the scenario adaptation layer and output layer are needed, eliminating the need to retrain the entire model. Adaptation time for various product models is controlled within 2.5 hours, and for mobile phone casing parts, only 1.5 hours. This efficient product model adaptation capability significantly shortens the production line changeover cycle, reduces the deployment cost of new product testing systems, and provides strong support for modern production models with multiple varieties and small batches.

[0069] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multi-modal AI collaborative industrial visual defect detection system, characterized in that, Includes the following modules: Multi-source image acquisition module: Composed of multiple cameras and depth sensors, it is adapted to different industrial scenarios. It achieves multi-device acquisition synchronization through a synchronous clock, dynamically adjusts the acquisition frequency, supports parallel acquisition of multimodal images, and outputs standardized format and timestamp information. Image preprocessing module: Built-in enhancement, denoising and semantic segmentation functions, improve image contrast, remove environmental noise, segment background and target regions, retain effective defect information, and output enhanced single-modal image and multimodal fusion base image; Multi-modal AI model collaboration module: Integrates three types of core models to extract local features, capture global correlations, and analyze temporal evolution patterns. It also integrates inference results through a dynamic weight allocation mechanism to construct a multi-scale, multi-dimensional defect detection model system. Defect Hierarchy Classification Module: Based on industrial defect classification standards, a two-level classification system is constructed to divide defect types and severity levels, link to the defect knowledge base, and complete the automatic matching and labeling of defect types and severity. Defect location and quantification module: It uses mask generation and contour extraction algorithms to locate defect positions, quantifies key dimension parameters through calibration algorithms, outputs defect coordinate information, and forms structured quantitative inspection data; Intelligent output module for test results: Supports multiple output formats, visualized reports containing complete test information, data for system integration, and PLC signal linkage to production line alarm device; Closed-loop optimization module: Based on detection results and production feedback, it uses transfer learning and incremental learning to iteratively update AI model parameters and regularly generate performance evaluation reports.

2. The multi-modal AI collaborative industrial visual defect detection system according to claim 1, characterized in that, It also includes an intelligent data augmentation module, which is divided into two units: offline augmentation and online augmentation. The offline augmentation unit expands the training dataset in a certain way and supports the generation of samples that simulate real defects based on generative adversarial networks. The online augmentation unit performs random cropping, color jittering, brightness adjustment, and blurring processing on the input image in real time during the model inference process. Combined with the diversity and randomness of defects in industrial scenarios, it generates augmented samples that fit the actual detection environment.

3. The multi-modal AI collaborative industrial visual defect detection system according to claim 1, characterized in that, It also includes a defect confidence calibration module, which dynamically calibrates the original confidence level of the multi-modal AI model output by introducing scene adaptation factors and historical data feedback. The calculation expression is as follows: in The confidence level for the defect after calibration. Output the raw confidence score for the model. The scene complexity coefficient. This represents the historical detection accuracy rate in the current scenario. To ensure the reliability of the matching between defect features and knowledge base samples, , , It is the sum of the three weighting coefficients.

4. The multi-modal AI collaborative industrial visual defect detection system according to claim 1, characterized in that, It also includes a cross-scene transfer adaptation module, with a built-in scene feature extractor and model adaptation unit. The scene feature extractor extracts environmental features, product material features and defect morphology features of different industrial scenarios through deep metric learning, and builds a scene feature vector library. The model adaptation unit is based on domain adaptation technology in transfer learning, adjusts the network layer parameters of the multi-modal AI model, freezes the general feature extraction layer, fine-tunes the scene adaptation layer and output layer, and introduces a domain adversarial training mechanism, so that it can quickly adapt to new detection scenarios without retraining the entire model.

5. The multi-modal AI collaborative industrial visual defect detection system according to claim 1, characterized in that, It also includes a multimodal data fusion module, which receives multimodal image data output from the image preprocessing module, and simultaneously accesses data from temperature, vibration, and pressure sensors at the production site. It integrates multi-source information through feature-level fusion, and the calculated expression is: in This is the fused global feature vector. The number of modes includes visible light, infrared, depth, temperature, vibration, and pressure modes. For the first Weighting coefficients for each mode For the first High-dimensional feature vectors of various modalities For the first The cosine similarity between a modal feature and the average feature of all modalities. For the first The reliability coefficient of modal data is determined by the sensor accuracy and data integrity.

6. The multi-modal AI collaborative industrial visual defect detection system according to claim 1, characterized in that, It also includes a model self-diagnosis and fault tolerance module, with a built-in model performance monitoring unit, fault diagnosis unit and backup model library. The performance monitoring unit collects model indicators in real time, the fault diagnosis unit identifies model problems through the isolated forest algorithm, and the backup model library stores redundant models trained in different scenarios. When a main model failure is detected or the performance drops below a threshold, it automatically switches to the adapted model in the backup model library, while recording fault information and issuing maintenance alarms.

7. The multi-modal AI collaborative industrial visual defect detection system according to claim 1, characterized in that, It also includes a defect source analysis module, which links process flow data, equipment operating parameters, raw material batch information, and environmental monitoring data from the production execution system. By constructing a causal reasoning model through Bayesian networks, it analyzes the key factors that cause defects, generates source reports for high-frequency defect types, identifies the process, equipment parameter deviations, or raw material quality issues that cause defects, and provides targeted production process optimization suggestions.

8. The multi-modal AI collaborative industrial visual defect detection system according to claim 1, characterized in that, It also includes an edge computing deployment module, which uses model lightweighting technology to prune, quantize and distill multi-mode AI models. It removes redundant network layers and invalid connections through channel pruning, builds a teacher-student model architecture based on knowledge distillation technology, retains the core detection capabilities of the main model, adapts to the hardware resources of edge computing nodes, and supports local deployment on edge servers or embedded devices in the production site.

9. The multi-modal AI collaborative industrial visual defect detection system according to claim 1, characterized in that, It also includes a permission management and data security module, establishing a three-level permission management system to assign different data access permissions and operation permissions to system administrators, testing operators, and maintenance personnel. System administrators have full-function operation permissions, testing operators can only view testing results and export reports, and maintenance personnel can only perform equipment maintenance and parameter adjustments. An operation log auditing function is set up to record all model adjustments, parameter modifications, result exports, and permission changes. The AES-256 encryption algorithm is used to encrypt the storage and transmission of collected image data, testing results, and model parameters.

10. The multi-modal AI collaborative industrial visual defect detection system according to claim 1, characterized in that, It also includes a visual operation and maintenance module, which has a built-in system status monitoring panel, model performance analysis tools and defect statistical report generation unit. It can intuitively display key equipment indicators through the dashboard, support filtering and querying historical test data by time, product type and defect type, provide trend analysis and visual chart display, automatically generate daily, weekly and monthly defect statistical reports, provide model parameter adjustment interface and remote maintenance function, and support operation and maintenance personnel to monitor the system operation status in real time and perform remote fault diagnosis through mobile terminal or PC terminal.

Images