Real-time defect detection system and method based on lightweight multi-branch model

Abstract

The invention discloses a real-time defect detection system and method based on a lightweight multi-branch model, and belongs to the technical field of intelligent manufacturing and industrial visual inspection, the system comprises an image acquisition module, an image preprocessing module, an NPU reasoning module, the lightweight multi-branch model and a decision output module; according to the method, the neural network processing unit (NPU) is embedded in the detection equipment, and the lightweight multi-branch model and hardware-algorithm collaborative optimization are combined, so that intelligent identification and classification of defect images are completed at the edge side with delay less than or equal to 1ms, the capability of distinguishing'false defects and real defects' can be improved, the false detection rate can be reduced, and the detection efficiency can be improved. And the comprehensive requirements of industrial online detection on high speed, accuracy and low power consumption are met. The method can be applied to the following scenes: wafer defect detection, PCB welding spot defect detection, microstructure detection in medical images, precision metal part scratch and pit detection, film coating uniformity judgment and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent manufacturing and industrial visual inspection, and particularly relates to a real-time defect detection system and method based on a lightweight multi-branch model. BACKGROUND

[0002] With the development of intelligent manufacturing industry, industrial product quality detection gradually develops from traditional manual inspection to automatic detection based on image recognition and machine vision.

[0003] In high-precision and high-throughput manufacturing scenarios such as semiconductors, consumer electronics, and automobile manufacturing, it is necessary to detect and classify micro-defects such as wafer surface defects, PCB solder joint flaws, and thin film scratches, with extremely high requirements for real-time performance and accuracy. Traditional image processing algorithms have problems such as difficulty in adapting to complex scene changes, high false detection rate, and long delay, and are gradually being replaced by intelligent detection models based on deep learning.

[0004] However, deploying deep neural networks in edge terminals such as factory workshops and other environments with strict limitations on power consumption, computing power, and physical space still has the following challenges:

[0005] 1) There is a model delay problem: traditional CNN (Convolutional Neural Network) or Transformer architecture usually requires a long inference time, even after ONNX (Open Neural Network Exchange) optimization and TensorRT acceleration, the common model still has tens of milliseconds of delay, which is far from meeting the ultra-low delay requirement of ≤1ms.

[0006] 2) Power consumption and hardware resources are limited: current industrial detection equipment is sensitive to power consumption, and deploying GPU at the edge is not realistic, and the CPU computing performance is insufficient, requiring low-power, high-parallel computing units such as NPU.

[0007] 3) There is a false defect misjudgment problem: traditional methods are difficult to distinguish between real defects (such as cracks, pinholes, scratches) and false defects (such as dust, stains, and reflections), resulting in high false detection rate.

[0008] 4) Poor model deployability: the training and testing of many models (such as ResNet, YOLO, ViT, etc.) need to be based on high-performance servers, which are difficult to migrate to resource-limited embedded edge devices.

[0009] In summary, current researches are mostly focused on algorithm accuracy optimization and model structure improvement, less consideration is given to the adaptability to hardware platforms, especially the lack of NPU-level instruction optimization and resource scheduling strategy, which leads to the fact that the model performs well on GPU, but the performance will decrease significantly once it is migrated to the edge. Typical defects include:

[0010] A. Large number of model parameters and high computational complexity, difficult to deploy on the edge;

[0011] B. Unable to complete inference within 1ms;

[0012] C. Unable to balance detection speed, accuracy and device power consumption at the same time;

[0013] D. Insufficient accuracy for false defect identification, false detection rate is as high as more than 20%;

[0014] E. No dynamic model scheduling mechanism is constructed for hardware awareness, and the model structure design is not matched with the NPU bottom layer execution characteristics.

[0015] Therefore, there is an urgent need for a detection system that can meet the requirements of high-precision, low-latency and edge computing integrated defect detection.

[0016] Based on this, the real-time defect detection system and method based on lightweight multi-branch model are designed to solve the above problems. SUMMARY

[0017] In view of the above-mentioned defects existing in the prior art, the real-time defect detection system and method based on lightweight multi-branch model are provided. By embedding a neural network processing unit (Neural Processing Unit, NPU) in the detection device, combining lightweight multi-branch model and hardware-algorithm collaborative optimization, intelligent recognition and classification of defect images are realized on the edge side with a delay of ≤1ms, which can improve the ability to distinguish between "false defects and real defects", reduce the false detection rate, and meet the comprehensive requirements of high speed, accuracy and low power consumption for industrial online detection.

[0018] To achieve the above purpose, the technical scheme is as follows:

[0019] The real-time defect detection system based on lightweight multi-branch model comprises an image acquisition module, an image preprocessing module, an NPU inference module, a lightweight multi-branch model and a decision output module.

[0020] The image acquisition module is used to acquire the surface image of the object to be detected.

[0021] The image preprocessing module is used to standardize, enhance and denoise the image.

[0022] The NPU inference module comprises an NPU processing unit, and the NPU processing unit comprises a convolution fusion module, a data pre-cache management module, an instruction scheduling controller, a quantization execution unit and a heterogeneous interface, and is used for performing inference operation of the lightweight multi-branch model.

[0023] The lightweight multi-branch model is deployed in the NPU processing unit, comprises a backbone network, a defect semantic embedding layer, a multi-branch distillation mechanism and a Softmax output layer, is used for completing defect classification and outputting an identification result; specifically, intermediate features are output through the backbone network, and the intermediate features are input into the defect semantic embedding layer for defect discrimination, and the lightweight multi-branch model adopts the multi-branch distillation mechanism.

[0024] The defect semantic embedding layer formula is:

[0025] z=GELU(W f ·f+b f )

[0026] wherein f is the intermediate feature, b f is a learnable bias vector, GELU is a Gaussian error linear activation function, and z is an embedding vector;

[0027] The distillation loss is:

[0028]

[0029] wherein L total is a total loss, λ1 and λ2 are weight coefficients, L CE is a cross-entropy loss, and L KD is a knowledge distillation loss;

[0030] The cross-entropy loss is:

[0031]

[0032] wherein y is a real label value, p s is a probability predicted by a student model, and the value range is [0, 1];

[0033] The knowledge distillation loss is:

[0034]

[0035] wherein p t and p s are probabilities predicted by a teacher model and a student model respectively, T is a temperature coefficient, τ is a scaling factor, KL is a KL divergence, p t i is a probability that the teacher model predicts that a sample x belongs to the i-th class, and p s is a probability that the student model predicts that the sample x belongs to the i-th class.i predict the probability that a student model sample x belongs to the i-th class;

[0036] a decision output module for converting the recognition result into a control signal and feeding back to an external PLC device control system.

[0037] Further, the image acquisition module uses a high-speed industrial camera, and the resolution of the acquired image is 2048x2048, the inter-frame delay is <0.2ms, and the image cache stores image data alternately through two groups of caches;

[0038] The image acquisition module is synchronized with the production line beat, and the time window length is calculated using the following formula:

[0039]

[0040] where T capture is the time window length, f line is the production line frequency, and ∈ is the image exposure + cache overhead time, usually ∈≤0.05ms.

[0041] Further, the image preprocessing module converts the acquired image into a standard tensor format, and performs grayscale normalization, contrast enhancement, distortion correction, background suppression, and FFT-based high-frequency noise filtering preprocessing on the image. The preprocessed image is encoded into a tensor input NPU inference module.

[0042] Further, the NPU processing unit includes the following processes:

[0043] All convolution layer operations use NPU native operators;

[0044] The activation function operation uses vectorized parallel execution;

[0045] The channel fusion operation is rewritten as:

[0046]

[0047] where W is the weight, b is the bias, γ is the scaling factor, and β is the offset factor;

[0048] The NPU inference delay is:

[0049] T NPU =T load +T compute +T store ≤0.85ms

[0050] where T NPU is the NPU inference delay, T load is the time for loading data into the NPU memory, and T computeTime for NPU to perform inference computation, T store Time for storing inference result from NPU memory back to external storage.

[0051] Further, the computational complexity in inference process Through model pruning and quantization, the model size is reduced Control within the peak capability of edge NPU;

[0052]

[0053] Wherein, L is the number of network layers, H l , W l are the height and width of the feature map respectively, K l is the size of the convolution kernel, and C , O

[0054] are the input and output channel numbers respectively.

[0054] Further, the backbone network adopts ShuffleNetV2, MobileNetV3 or Fused-MBConv, and integrates SE or CBAM attention mechanism.

[0055] Further, the intermediate feature calculation formula is:

[0056]

[0057] Wherein, f(x) is the intermediate feature, f i (x) is the output of the i-th sub-network path, and a i is a dynamic scheduling weight determined dynamically by hardware performance and input complexity.

[0058] Further, the Softmax output layer outputs the model recognition result, and the recognition result includes a probability vector Wherein, C is the number of defect categories;

[0059] The maximum probability criterion is used to determine the predicted category of the model:

[0060]

[0061] Wherein, is the category label predicted by the model, and p i is the probability that the sample belongs to the i-th category predicted by the model.

[0062] At the same time, it is compared with the set system judgment threshold value θ, when , an uncertain flag is output.

[0063] In order to better achieve the purpose of the present application, the present application further provides a real-time defect detection method based on a lightweight multi-branch model, comprising the following steps:

[0064] Step one, the image acquisition module acquires the surface image of the object to be detected;

[0065] Step two, the image preprocessing module standardizes, enhances and denoises the image, and then inputs the NPU inference module;

[0066] Step three, the NPU inference module performs inference operation of the lightweight multi-branch model, including the following steps:

[0067] All convolution layer operations use NPU native operators;

[0068] The activation function operation uses vectorized parallel execution;

[0069] The channel fusion operation is rewritten as:

[0070]

[0071] Where W is the weight, b is the bias, γ is the scaling factor, and β is the offset factor;

[0072] Step four, the lightweight multi-branch model outputs intermediate features through the backbone network And input the intermediate features into the defect semantic embedding layer for defect discrimination;

[0073] The formula of the defect semantic embedding layer is:

[0074] z=GELU(W f ·f+b f )

[0075] Where f is the intermediate feature, b f is a learnable bias vector, GELU is the Gaussian error linear activation function, and z is the embedding vector;

[0076] Step five, adopt multi-branch distillation mechanism, and the distillation loss is:

[0077]

[0078] Where L total is the total loss, λ1 and λ2 are weight coefficients, L CE is the cross-entropy loss, and L KD is the knowledge distillation loss;

[0079] The cross-entropy loss is:

[0080]

[0081] Where y is the true label value, and p s is the probability predicted by the student model;

[0082] The knowledge distillation loss is:

[0083]

[0084] Wherein, p t And p s Respectively, the probability of the teacher model and the student model prediction, T is the temperature coefficient, tau is the scaling factor, KL is the KL divergence, p t i The probability of the teacher model predicting that the sample x belongs to the i-th class, p s i The probability of the student model predicting that the sample x belongs to the i-th class.

[0085] Step six, the Softmax output layer outputs the model recognition result.

[0086] Step seven, the decision output module converts the recognition result into a control signal and feeds back to the external PLC device control system.

[0087] Further, the parameter setting: lambda 1=0.7, lambda 2=0.3.

[0088] The application has the following advantages in industrial practical scenarios:

[0089] 1) High-precision defect classification capability: the application is tested on multiple industrial detection data sets (such as NEU-DET, DeepPCB, Dagm2007), and the classification accuracy is more than 98.5%. In the distinction between false defects and real defects, the F1 value is improved by about 11%.

[0090] 2) Low delay response performance: the system is tested on mainstream NPU chips, and the average single-frame inference delay is less than or equal to 0.89ms, meeting the high-throughput requirement of 1000fps level.

[0091] 3) Low resource occupation: the model parameter quantity is controlled to be less than 1.2 million, and the peak power consumption is less than 1.5W, which is suitable for various industrial embedded terminals.

[0092] 4) Good universality and scalability: the model can be quickly reconstructed according to different hardware platforms (such as ASIC, FPGA), and supports ONNX, TFLite, OpenVINO multi-format export. BRIEF DESCRIPTION OF DRAWINGS

[0093] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the accompanying drawings in the following description only represent some of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0094] Fig. 1 The structural diagram of the real-time defect detection system based on the lightweight multi-branch model of the present application.

[0095] Fig. 2 The structural diagram of the lightweight multi-branch model of the present application. DETAILED DESCRIPTION

[0096] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the accompanying drawings in the following description only represent some of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0097] Embodiment one: please refer to the accompanying drawings of the specification Fig. 1-2 , the real-time defect detection system based on the lightweight multi-branch model, comprising an image acquisition module, an image preprocessing module, an NPU inference module, a lightweight multi-branch model and a decision output module;

[0098] 1.1, the image acquisition module is used for acquiring the surface image of the object to be detected, adopts a high-speed industrial camera (frame rate ≥ 200 fps), is connected with the detection terminal through a GigE or USB3.1 interface, the resolution of the acquired image is 2048×2048, the interframe delay is <0.2 ms, and double-buffering control (image data is alternately stored through two groups of buffers (such as Buffer A and Buffer B)) is adopted for image caching, so as to ensure the continuity of data.

[0099] The image acquisition module is synchronized with the production line beat, and image sampling is carried out according to an external trigger, and the time window length is calculated by the following formula:

[0100]

[0101] Wherein, T capture is the time window length, f line is the production line frequency, and ∈ is the image exposure + cache overhead time, and generally ∈≤0.05 ms.

[0102] 1.2, image preprocessing module, for standardizing, enhancing, denoising the image; specifically, the collected image is converted to a standard tensor format, and the image is preprocessed by grayscale normalization, contrast enhancement, distortion correction and background suppression, FFT-based high-frequency noise filtering, etc.; the preprocessed image is encoded into a tensor Input NPU inference module.

[0103] Grayscale normalization is performed using the following formula:

[0104]

[0105] Where x is the original image pixel, x' is the normalized pixel value, μ and σ are the global mean and standard deviation of the image.

[0106] The contrast enhancement processing uses the CLAHE (Contrast Limited Adaptive Histogram Equalization) method.

[0107] 1.3, NPU inference module, including NPU processing unit (such as Cambrian MLU220, Hisense Ascend310B or Kneron KL730, etc.), NPU processing unit including convolution fusion module, data pre-cache management module, instruction scheduling controller, quantization execution unit and heterogeneous interface, for executing inference operation of lightweight multi-branch model; using ONNX-Tensor conversion intermediate format, the trained lightweight multi-branch model is imported into NPU running environment; the model is compiled to generate intermediate representation (IR, Intermediate Representation), which contains optimized network structure and operator information, which can be directly loaded and parsed by NPU built-in inference engine. To speed up the inference process, the NPU processing unit includes the following processes:

[0108] All convolution layer operations use NPU native operators (such as conv2d_int8);

[0109] Activation function operations (such as ReLU, Sigmoid) use vectorized parallel execution;

[0110] Channel fusion operation is rewritten as:

[0111]

[0112] Where W is the weight, b is the bias, γ is the scaling factor, and β is the offset factor.

[0113] The convolutional fusion module accelerates inference by fusing BN (batch normalization) layers and activation functions using native operators. A data pre-caching management module reduces latency in handling image tensors and intermediate features; an instruction scheduling controller pipelines the execution order of operators. A quantization execution unit enables model execution with INT8 precision, improving computational concurrency. A heterogeneous interface supports collaboration with external CPUs / MCUs.

[0114] By incorporating the BN parameters into the convolution kernel, separate operator calls are avoided, thus improving throughput.

[0115] NPU inference latency is:

[0116] T NPU =T load +T compute +T store ≤0.85ms

[0117] Among them, T NPU For NPU inference latency, T load The time T is the time to load data into NPU memory. compute T is the time T takes for the NPU to perform inference computation. store The time required to store the inference results from NPU memory back to external storage.

[0118] This invention employs an NPU-embedded neural network collaborative inference system, which can integrate an NPU processing unit into the detection terminal to build an embedded AI system with the following capabilities: realizing a closed-loop process of image acquisition, processing, and recognition on the edge; reconstructing the computation graph based on the NPU native operator library to avoid scheduling delays of heterogeneous chips; and achieving an inference time of ≤1ms per image, meeting the requirements of high-speed production lines.

[0119] To maximize NPU performance, computational complexity is reduced throughout the inference process. Through model pruning and quantization, It is kept within the peak capacity of the edge NPU.

[0120]

[0121] Where L is the number of network layers, H l W l These represent the height and width of the feature map, respectively, and K. l The kernel size is the convolution kernel size. These represent the number of input and output channels, respectively.

[0122] 1.4 A lightweight multi-branch model is deployed in the NPU processing unit, which includes a backbone network, a defect semantic embedding layer, a multi-branch distillation mechanism, and a Softmax output layer to complete defect classification and output recognition results.

[0123] The backbone network adopts ShuffleNetV2 (or MobileNetV3, Fused-MBConv), and integrates SE or CBAM attention mechanism, and outputs intermediate features through the backbone network And the intermediate features are input into the defect semantic embedding layer for defect discrimination.

[0124] The intermediate feature calculation formula is:

[0125]

[0126] Wherein, f(x) is the intermediate feature, f i (x) is the output of the i-th sub-network path, a i is a dynamic scheduling weight, which is dynamically determined by hardware performance and input complexity.

[0127] The model parameters are as follows:

[0128] The input tensor size is:

[0129] The model layer number is: 13 layers of convolution + 2 layers of attention gate.

[0130] The total number of parameters is: about 1.1x10 6 ;

[0131] The total number of multiply-add operations is: about 150xMFLOPs.

[0132] In view of the defect feature distribution dense and difficult to classify, in order to improve the robustness of the lightweight multi-branch model to false defects such as dust, light spot and reflection, a defect semantic embedding mechanism is proposed, and a defect semantic embedding layer is added before the classification layer:

[0133] z=GELU(W f ·f+b f )

[0134] Wherein, f is the intermediate feature, b f is a learnable bias vector, GELU is a Gaussian error linear activation function, and z is an embedding vector, which is used as the final classification basis.

[0135] The defect semantic embedding layer is used to enhance the model's ability to distinguish false defects (reflection, dust, stains, etc.) before output; the output dimension (32 dimensions) of the defect semantic embedding layer is less than the feature dimension (64 dimensions) of the backbone network.

[0136] The model adopts a multi-branch distillation mechanism, and the distillation training target is to make the lightweight student model output approximate the teacher model. The teacher model is a large-scale pre-training classification network (such as ResNet-101), which is trained on a GPU. A reflective class embedding layer is introduced in the student model, reflective samples are added in the training set, and data enhancement (mirror, Gaussian spot) is performed.

[0137] The distillation loss is:

[0138]

[0139] Parameter setting: λ1=0.7, λ2=0.3.

[0140] wherein, L total is the total loss, λ1, λ2 are weight coefficients, and the value range is [0, 1], L CE is the cross-entropy loss, L KD is the knowledge distillation (Knowledge Distillation) loss;

[0141] The cross-entropy loss is:

[0142]

[0143] wherein, y is the true label value, p s is the probability predicted by the student model, and the value range is [0, 1].

[0144] The knowledge distillation loss is:

[0145]

[0146] wherein, p t and p s are the probabilities predicted by the teacher model and the student model respectively, T is the temperature coefficient, set T=3, τ is the scaling factor, KL is the KL divergence, p t i is the probability predicted by the teacher model that the sample x belongs to the i-th class, p s i is the probability predicted by the student model that the sample x belongs to the i-th class.

[0147] The Softmax output layer outputs the model recognition result, and the recognition result contains a probability vector wherein C is the number of defect categories (such as crack, scratch, reflection, pollution, etc.). The maximum probability criterion is used to determine the predicted class of the model:

[0148]

[0149] wherein, p is the class label predicted by the model i pi is the probability that the sample belongs to the i-th class predicted by the model.

[0150] At the same time, compare with the set system judgment threshold θ, when , then output the "uncertain" flag.

[0151] 1.5, a decision output module for converting the recognition result into a control signal and feeding back to the external PLC device control system.

[0152] The recognition result is fed back to the PLC device control system through a signal output interface such as Modbus RTU, OPC-UA, RS-485 or TTL; when the predicted class is not empty, the PLC device control system controls the rejection mechanism to trigger and reject the defective workpiece, and controls the alarm unit to execute the alarm signal; when the "uncertain" flag is output, the PLC device control system feeds back the recheck information to the human, reminding to recheck.

[0153] The application adopts a lightweight multi-branch model, greatly reduces the model parameters and calculation amount on the basis of maintaining the feature expression ability; combines the multi-branch distillation mechanism to train the model, retains the discrimination ability of the main model in the lightweight model; adds a defect semantic embedding layer before the classification layer, improves the distinguishing ability of the system to "false defects"; accelerates the inference process through the NPU processing unit, ensures ultra-low delay while realizing controllable energy consumption.

[0154] Example two: as shown in the figure, the real-time defect detection method based on the lightweight multi-branch model includes the following steps: Fig. 1-2

[0155] Step one, the image acquisition module acquires the surface image of the object to be detected.

[0156] Step two, the image preprocessing module standardizes, enhances and denoises the image, and then inputs the NPU inference module.

[0157] Step three, the NPU inference module performs inference operation of the lightweight multi-branch model, including the following steps:

[0158] All convolution layer operations use NPU native operators (such as conv2d_int8);

[0159] The activation function operation (such as ReLU, Sigmoid) uses vectorized parallel execution;

[0160] The channel fusion operation is rewritten as:

[0161]

[0162] ​Wherein, W is weight, b is bias, gamma is scaling factor, beta is offset factor.

[0163] The convolution fusion module fuses the BN (batch normalization) layer and the activation function through a native operator to accelerate inference. The data pre-cache management module is used to reduce the delay of image tensor and intermediate feature transfer; the instruction scheduling controller is used to arrange the execution sequence of each operator in a pipelined manner. The quantization execution unit realizes model execution with INT8 precision, improving calculation concurrency. The heterogeneous interface supports cooperation with external CPU / MCU.

[0164] The BN parameters are combined into the convolution kernel, avoiding separate operator calling and improving throughput.

[0165] The NPU inference delay is:

[0166] T NPU = T load + T compute + T store ≤ 0.85ms

[0167] Wherein, T NPU is the NPU inference delay, T load is the time for loading data into the NPU memory, T compute is the time for NPU to perform inference calculation, and T store is the time for storing the inference result from the NPU memory back to the external storage.

[0168] The neural network collaborative inference system with NPU embedded deployment adopted by the application can integrate the NPU processing unit in the detection terminal, construct an embedded AI system, and has the following capabilities: realizing the whole-process closed loop of terminal-side image acquisition, processing and identification; reconstructing the calculation graph based on the NPU native operator library to avoid heterogeneous chip scheduling delay; the inference time of each image is ≤1ms, meeting the demand of high-speed production line.

[0169] In order to maximize the performance of NPU, the calculation complexity Through model pruning and quantization, the control is within the peak capability of edge NPU.

[0170]

[0171] Wherein, L is the number of network layers, H l and W l are the height and width of the feature map respectively, K l is the convolution kernel size, and are the input and output channel numbers respectively.

[0172] Step four, the lightweight multi-branch model outputs the intermediate feature through the backbone network And input the intermediate feature to the defect semantic embedding layer for defect discrimination;

[0173] The intermediate feature calculation formula is:

[0174]

[0175] Wherein, f(x) is the intermediate feature, f i (x) is the output of the i-th sub-network path, a i is a dynamic scheduling weight, which is dynamically determined by hardware performance and input complexity.

[0176] The defect semantic embedding layer formula is:

[0177] z=GELU(W f ·f+b f )

[0178] Wherein, f is the intermediate feature, b f is a learnable bias vector, GELU is a Gaussian error linear activation function, and z is an embedding vector, which is used as the final classification basis.

[0179] The defect semantic embedding layer is used to enhance the model's ability to distinguish false defects (reflection, dust, stains, etc.) before output; the output dimension (32 dimensions) of the defect semantic embedding layer is less than the feature dimension (64 dimensions) of the backbone network.

[0180] Step five, adopt a multi-branch distillation mechanism, train by minimizing the KL divergence between the teacher model and the student model, the teacher model is a large-scale pre-training classification network (such as ResNet-101), and the training is carried out on GPU. Introduce a reflection class embedding layer in the student model, increase the reflection samples in the training set, and do data enhancement (mirror, Gaussian spot).

[0181] The distillation loss is:

[0182]

[0183] Parameter setting: λ1=0.7, λ2=0.3.

[0184] Wherein, L total is the total loss, λ1 and λ2 are weight coefficients, and the value range is [0, 1], L CE is the cross-entropy loss, and L KD is the knowledge distillation (Knowledge Distillation) loss;

[0185] The cross-entropy loss is:

[0186]

[0187] Where y is the true label value, p s It is the probability predicted by the student model, and its value ranges from [0, 1].

[0188] The knowledge distillation loss is:

[0189]

[0190] Where, p t and p s Here, T represents the probabilities predicted by the teacher model and the student model, respectively; T is the temperature coefficient, set to T=3; τ is the scaling factor; KL is the KL divergence; and p... t i For the teacher model, predict the probability that sample x belongs to the i-th class, p. s i Predict the probability that sample x belongs to class i for the student model.

[0191] Step 6: The Softmax output layer outputs the model's recognition results, which include probability vectors. Where C represents the number of defect categories (such as cracks, scratches, reflections, contamination, etc.).

[0192] Use the maximum probability criterion to determine the model's predicted category:

[0193]

[0194] in, p represents the class label predicted by the model. i Predict the probability that a sample belongs to class i for the model.

[0195] Simultaneously, it is compared with the set system judgment threshold θ, when... If the condition is met, then the "uncertain" flag will be output.

[0196] Step 7: The decision output module converts the recognition results into control signals and feeds them back to the external PLC equipment control system.

[0197] The identification results are fed back to the PLC equipment control system via signal output interfaces such as Modbus RTU, OPC-UA, RS-485, or TTL. When the predicted category is not empty, the PLC equipment control system triggers the rejection mechanism to reject the defective workpiece and controls the alarm unit to issue an alarm signal. When the "uncertain" flag is output, the PLC equipment control system feeds back the re-inspection information to the operator to remind them to perform a re-inspection.

[0198] The application adopts a lightweight multi-branch model, greatly reduces the model parameters and calculation amount on the basis of maintaining the feature expression ability; combines the multi-branch distillation mechanism to train the model, retains the discrimination ability of the main model in the lightweight model; adds a defect semantic embedding layer before the classification layer, improves the distinguishing ability of the system to "false defects"; accelerates the inference process through the NPU processing unit, ensures ultra-low delay while realizing controllable energy consumption.

[0199] Experimental example one: real-time evaluation of system modules

[0200] Module Processing time (ms) Proportion Image acquisition module 0.20 20％ Image preprocessing module 0.05 5％ NPU inference module 0.85 85％ Decision output module 0.05 5％ Total delay ≤1.00 100％

[0201] The application can be applied to the following scenarios: wafer defect detection, PCB solder joint defect, microstructure detection in medical images, scratch detection of precision metal parts, pit detection, thin film coating uniformity judgment, etc.

[0202] Experimental example two: the system of the application is applied to online detection in a precision electronics factory, and needs to identify solder joint defects, missed welding, false welding, stains, ghosting and other problems, the production line speed is 400 pieces per minute, and the inference time requirement for each product image is not more than 1ms. The detection effect is as follows: the inference accuracy is 98.7%; the false detection rate is reduced from about 15% of the traditional method to 4.2%; the average delay of the whole process is 0.97ms (meets the requirements).

[0203] Experimental example three: application of traditional threshold edge method and the method of the application to identify reflective false defects: in mirror stainless steel part detection, reflective area is generated due to strong light, which leads to false detection of scratches or cracks by traditional detection, and the false detection rate is as high as 28%.

[0204] Method Accuracy False defect misjudgment rate Traditional threshold edge method 88.2％ 28.3％ The method of the present application 98.3％ 5.9％

[0205] The above embodiments are only used to illustrate the technical solutions of the application, but not limit it; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements will not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application.

Claims

1. A real-time defect detection system based on a lightweight multi-branch model, characterized in that, It includes an image acquisition module, an image preprocessing module, an NPU inference module, a lightweight multi-branch model, and a decision output module; The image acquisition module is used to acquire surface images of the object to be detected. The image preprocessing module is used to standardize, enhance, and denoise images. The NPU inference module includes an NPU processing unit, which includes a convolutional fusion module, a data pre-caching management module, an instruction scheduling controller, a quantization execution unit, and a heterogeneous interface, for performing inference operations on a lightweight multi-branch model. The lightweight multi-branch model, deployed in the NPU processing unit, includes a backbone network, a defect semantic embedding layer, a multi-branch distillation mechanism, and a Softmax output layer, which is used to complete defect classification and output recognition results. Specifically, the backbone network outputs intermediate features, which are then input into the defect semantic embedding layer for defect discrimination. The lightweight multi-branch model adopts a multi-branch distillation mechanism. The formula for the defect semantic embedding layer is: z=GELU(W f ·f+b f ) Where f is an intermediate feature, b f For learnable bias vectors, GELU is a Gaussian error linear activation function, and z is the embedding vector; Distillation loss is: Among them, L total The total loss is λ1 and λ2, which are weighting coefficients, and L is the weighting coefficient. CE For cross-entropy loss, L KD For knowledge distillation loss; The cross-entropy loss is: Where y is the true label value, p s It is the probability predicted by the student model, and its value ranges from [0, 1]. The knowledge distillation loss is: Where, p t and p s Here, T represents the probabilities predicted by the teacher model and the student model, respectively; T is the temperature coefficient; τ is the scaling factor; KL is the KL divergence; and p... t i For the teacher model, predict the probability that sample x belongs to the i-th class, p s i Predict the probability that sample x belongs to the i-th class for the student model; The decision output module is used to convert the recognition results into control signals and feed them back to the external PLC equipment control system.

2. The real-time defect detection system based on a lightweight multi-branch model according to claim 1, characterized in that, The image acquisition module uses a high-speed industrial camera, which acquires images with a resolution of 2048×2048 and an inter-frame delay of <0.2ms. The image buffer stores image data alternately through two sets of buffers. The image acquisition module is synchronized with the production line cycle time, and the time window length is calculated using the following formula: Among them, T capture f is the length of the time window. line Where is the production line frequency, and ∈ is the image exposure + buffering overhead time, which is usually ∈ ≤ 0.05ms.

3. The real-time defect detection system based on a lightweight multi-branch model according to claim 1, characterized in that, The image preprocessing module converts the acquired image into a standard tensor format and performs grayscale normalization, contrast enhancement, distortion correction and background suppression, and FFT-based high-frequency noise filtering preprocessing. The preprocessed image is then encoded into a tensor. Input the NPU inference module.

4. The real-time defect detection system based on a lightweight multi-branch model according to claim 1, characterized in that, The NPU processing unit includes the following processes: All convolutional layer operations use native NPU operators; Activation function operations are executed in parallel using vectorization; The channel fusion operation is rewritten using an optimized scheduling graph as follows: Where W is the weight, b is the bias, γ is the scaling factor, and β is the offset factor; NPU inference latency is: T NPU =T load +T compute +T store ≤0.85ms Among them, T NPU For NPU inference latency, T load The time T is the time to load data into NPU memory. compute T is the time T takes for the NPU to perform inference computation. store The time required to store the inference results from NPU memory back to external storage.

5. The real-time defect detection system based on a lightweight multi-branch model according to claim 4, characterized in that, Calculate complexity during the reasoning process Through model pruning and quantization, Keep it within the peak capacity of the edge NPU; Where L is the number of network layers, H l W l These represent the height and width of the feature map, respectively, and K. l The kernel size is the convolution kernel size. These represent the number of input and output channels, respectively.

6. The real-time defect detection system based on a lightweight multi-branch model according to claim 1, characterized in that, The backbone network adopts ShuffleNetV2, MobileNetV3 or Fused-MBConv, and incorporates SE or CBAM attention mechanisms.

7. The real-time defect detection system based on a lightweight multi-branch model according to claim 1, characterized in that, The formula for calculating intermediate features is: Where f(x) is an intermediate feature, f i (x) represents the output of the i-th sub-network path, α i The weights are dynamically allocated based on hardware performance and input complexity.

8. The real-time defect detection system based on a lightweight multi-branch model according to claim 1, characterized in that, The softmax output layer outputs the model's recognition results, which include probability vectors. Where C is the number of defect categories; Use the maximum probability criterion to determine the model's prediction category: in, p represents the class label predicted by the model. i Predict the probability that a sample belongs to the i-th class for the model; Simultaneously, it is compared with the set system judgment threshold θ, when... If the condition is met, an uncertain flag will be output.

9. A real-time defect detection method based on a lightweight multi-branch model, characterized in that, Includes the following steps: Step 1: The image acquisition module acquires an image of the surface of the object to be detected; Step 2: The image preprocessing module performs standardization, enhancement, and noise reduction on the image, and then inputs it into the NPU inference module; Step 3: The NPU inference module performs inference operations on the lightweight multi-branch model, including the following steps: All convolutional layer operations use native NPU operators; Activation function operations are executed in parallel using vectorization; The channel fusion operation is rewritten using an optimized scheduling graph as follows: Where W is the weight, b is the bias, γ is the scaling factor, and β is the offset factor; Step 4: The lightweight multi-branch model outputs intermediate features through the backbone network. The intermediate features are then input into the defect semantic embedding layer for defect identification. The formula for the defect semantic embedding layer is: z=GELU(W f ·f+b f ) Where f is an intermediate feature, b f For learnable bias vectors, GELU is a Gaussian error linear activation function, and z is the embedding vector; Step 5: Employ a multi-branch distillation mechanism. The distillation loss is: Among them, L total The total loss is λ1 and λ2, which are weighting coefficients, and L is the weighting coefficient. CE For cross-entropy loss, L KD For knowledge distillation loss; The cross-entropy loss is: Where y is the true label value, p s It is the probability predicted by the student model; The knowledge distillation loss is: Where, p t and p s Here, T represents the probabilities predicted by the teacher model and the student model, respectively; T is the temperature coefficient; τ is the scaling factor; KL is the KL divergence; and p... t i For the teacher model, predict the probability that sample x belongs to the i-th class, p s i Predict the probability that sample x belongs to the i-th class for the student model; Step 6: The Softmax output layer outputs the model recognition results; Step 7: The decision output module converts the recognition results into control signals and feeds them back to the external PLC equipment control system.

10. The real-time defect detection method based on a lightweight multi-branch model according to claim 9, characterized in that, Parameter settings: λ1 = 0.7, λ2 = 0.3.

Images