Industrial defect detection-oriented adaptive lightweight visual Transform network structure optimization method and system

Abstract

The invention discloses a self-adaptive lightweight visual Transform network structure optimization method and system for industrial defect detection, and the method comprises the following steps: obtaining the surface image data of an industrial product, and carrying out the preprocessing and defect labeling of an image; the method comprises the following steps of: constructing a self-adaptive lightweight visual Transform defect detection model; performing training optimization on the defect detection model, including optimizing network structure hyper-parameters by using an evolutionary algorithm and balancing detection precision and model complexity by using a multi-objective loss function; and deploying the trained and optimized model in an industrial detection system, performing defect detection on the image of the to-be-detected product, and outputting the type and position of the defect. Compared with the prior art, the method has the advantages that the parameter quantity and the calculation quantity of the model can be greatly reduced while the defect detection accuracy is ensured, adaptive structure optimization and efficient deployment of the model are realized, and the method is particularly suitable for real-time online industrial defect detection application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial visual inspection, and in particular to an adaptive lightweight visual Transformer network structure optimization method and system for industrial defect detection. BACKGROUND

[0002] Surface defect detection of industrial products is a key link in manufacturing quality control, which traditionally relies on manual visual inspection, with low efficiency and accuracy affected by human factors. With the development of computer vision and deep learning, automatic defect detection methods based on convolutional neural networks (CNN) have been gradually applied to industrial scenarios. However, traditional CNN models have the shortcomings of insufficient local feature extraction and difficulty in modeling long-range dependencies, while directly using Transformer models faces the problems of high computational complexity and large number of parameters, affecting real-time performance and deployment efficiency. Especially in some complex background and subtle defect scenarios, the model needs to have both global context awareness and efficient local detail feature extraction.

[0003] Some existing researches attempt to introduce Transformer into industrial defect detection to improve global feature modeling capability. For example, some literature proposes a lightweight deep learning model based on Swin Transformer for TFT-LCD panel defect classification, which reduces the dimension of Token fusion for each layer feature map, introduces a depth separable convolution module to increase the convolution inductive bias, and uses knowledge distillation to compensate for the precision decline caused by lightweight. This method balances the accuracy and speed requirements of defect detection to some extent. However, current industrial defect detection models still have many shortcomings: on the one hand, many models are designed for specific defect types or datasets, with limited generalization; on the other hand, most methods use fixed network structures, which are difficult to adaptively adjust according to the characteristics of different products or defects, and cannot balance real-time performance and high accuracy in environments with limited computing resources.

[0004] Therefore, there is an urgent need for a new technical solution that combines the global feature extraction advantages of Transformer networks and lightweight structure optimization methods, significantly reduces model complexity while ensuring accuracy, and adaptively adjusts the model structure according to actual application requirements to meet the real-time and accurate defect detection requirements of industrial sites. SUMMARY

[0005] Technical purpose: In view of the deficiencies of the prior art, the present application discloses an adaptive lightweight visual Transformer network structure optimization method and system for industrial defect detection, which realizes automatic optimization and on-demand adjustment of model structure under the visual Transformer framework, thereby significantly reducing the computational overhead of the model while ensuring the accuracy of defect detection.

[0006] Technical scheme: To achieve the above technical purpose, the application adopts the following technical scheme:

[0007] An adaptive lightweight visual Transformer network structure optimization method for industrial defect detection, comprising the following steps:

[0008] Obtain image data of industrial products, and pre-process and defect label the image data to establish a defect sample data set for training;

[0009] Construct an adaptive lightweight visual Transformer defect detection model, which includes a feature extraction network based on visual Transformer and an output network for defect judgment, wherein the feature extraction network adopts a multi-head self-attention mechanism to obtain global features, and a convolution module is introduced to enhance local feature extraction, the structure of the feature extraction network is designed according to multi-scale hierarchical design, and has adjustable network depth and width;

[0010] Train and optimize the defect detection model using the defect sample data set, and use knowledge distillation to improve the accuracy of the lightweight defect detection model during the training process, and automatically adjust part of the layer, node or parameter configuration of the defect detection model through a structure optimization algorithm to reduce the model complexity;

[0011] Deploy the trained and optimized defect detection model to an industrial defect detection system to detect defects in the image to be detected of the industrial product to be detected, including inputting the image to be detected into the defect detection model, extracting image features through the feature extraction network and generating defect detection results through the output network, the defect detection results including the existence or non-existence of defects, defect types and the location of defects in the image;

[0012] According to the defect detection result, output alarm or classification information.

[0013] Preferably, constructing an adaptive lightweight visual Transformer defect detection model comprises: block encoding the input image to obtain an image patch sequence, inputting the image patch sequence into the multi-head self-attention module of the Transformer for feature calculation, and performing Token fusion dimension reduction on the feature map after each layer of self-attention calculation to reduce the length of the feature sequence layer by layer, thereby reducing the calculation amount.

[0014] Preferably, the convolution module introduced in the feature extraction network is a depth separable convolution module, which is used to provide the local perception ability of the convolutional neural network to make up for the deficiency of the pure Transformer architecture in small defect feature extraction.

[0015] Preferably, the structure optimization algorithm used in the training optimization process is an evolutionary algorithm, which balances the accuracy and complexity of the defect detection model through a preset multi-objective loss function, and iteratively searches for a model structure that meets the complexity constraint while maintaining the accuracy of the defect detection.

[0016] Preferably, the structure optimization algorithm used in the training optimization process is a reinforcement learning algorithm, which balances the accuracy and complexity of the defect detection model through a preset multi-objective loss function, and adjusts the defect detection model structure parameters based on the feedback of the reinforcement learning agent to obtain the optimal model structure that meets the complexity requirement.

[0017] Preferably, the training optimization process uses knowledge distillation technology, including using a pre-trained high-performance teacher model to guide the training of the defect detection model to minimize the difference between the output of the student model and the teacher model, thereby improving the detection accuracy of the lightweight defect detection model.

[0018] An adaptive lightweight visual Transformer network structure optimization system for industrial defect detection, characterized by being used to implement an adaptive lightweight visual Transformer network structure optimization method for industrial defect detection as described above, comprising:

[0019] An image acquisition device for acquiring surface images of industrial products;

[0020] A data processing device with a memory and a processor, the processor running a trained and optimized adaptive lightweight visual Transformer defect detection model, for defect detection processing on images acquired by the image acquisition device and outputting defect detection results;

[0021] A display and alarm device for receiving and outputting defect detection results provided by the data processing device.

[0022] Preferably, the data processing device includes an image preprocessing module, a feature extraction module, a defect discrimination module, and a result output module, wherein:

[0023] The image preprocessing module is used for grayscale, normalization, and filter denoising preprocessing of the acquired images;

[0024] The feature extraction module is implemented by a visual Transformer network and is used for extracting multi-scale image feature representations;

[0025] The defect discrimination module is used for defect classification and positioning judgment based on the extracted image features to obtain defect detection results;

[0026] The result output module is configured to send the defect detection result to a display and alarm device for output and prompting.

[0027] Preferably, the data processing device is an edge computing device containing an artificial intelligence acceleration unit to accelerate the inference calculation process of the visual Transformer defect detection model.

[0028] Beneficial effects: The adaptive lightweight visual Transformer network structure optimization method and system for industrial defect detection provided by the present application has the following beneficial effects:

[0029] 1. The present application integrates a depth separable convolution module after the multi-head self-attention module of the visual Transformer, and introduces a Token fusion dimension reduction mechanism in each layer, so that the model can efficiently capture global context information and finely extract local defect features. This collaborative design significantly enhances the recognition ability of small defects and complex backgrounds, and through layer-by-layer compression of feature sequences, the computational complexity and memory occupation of the model are greatly reduced, thereby meeting the dual needs of speed and resources for real-time detection in industrial sites.

[0030] 2. The present application uses knowledge distillation technology to transfer the discrimination ability of a high-performance teacher model to a lightweight student model, and combines structure optimization based on evolutionary algorithms (or reinforcement learning) to automatically search for the optimal network depth, width and attention head configuration. This strategy not only makes the lightweight model close to or reach the level of large models in terms of accuracy, but also effectively controls the model size. More importantly, relying on the input-dependent structure generated by Token fusion dimension reduction, the model can skip redundant calculations for simple scenes and perform in-depth execution for complex scenes during inference, thereby ensuring high detection accuracy while realizing on-demand allocation of computing resources, providing a flexible and reliable deployment solution for edge devices or resource-constrained industrial PCs, and significantly expanding the application range and practical value of industrial defect detection technology. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description.

[0032] Figure 1 The figure is a system structure diagram of the present application;

[0033] Figure 2 The figure is a method flow diagram of the present application;

[0034] Figure 3 The figure is a visual Transformer defect detection model structure diagram. DETAILED DESCRIPTION

[0035] The application will be described in more detail below by way of a preferred embodiment and with reference to the drawings, but the application is not limited to the embodiment described.

[0036] As shown in the drawings, the application discloses an adaptive lightweight visual Transformer network structure optimization method for industrial defect detection, comprising the following steps: Figure 2

[0037] Image data of industrial products are acquired, and the image data are preprocessed and defect-labeled to establish a defect sample data set for training.

[0038] An adaptive lightweight visual Transformer defect detection model is constructed. The defect detection model includes a feature extraction network based on visual Transformer and an output network for defect judgment. The feature extraction network adopts a multi-head self-attention mechanism to obtain global features and introduces a convolution module to enhance local feature extraction. The structure of the feature extraction network is designed in a multi-scale hierarchical manner and has adjustable network depth and width.

[0039] The defect detection model is trained and optimized using the defect sample data set. Knowledge distillation is used in the training process to improve the accuracy of the lightweight defect detection model. The structure optimization algorithm is used to automatically adjust the number of layers, nodes or parameter configurations of the defect detection model to reduce the complexity of the model.

[0040] The trained and optimized defect detection model is deployed in an industrial defect detection system to detect defects in the images of industrial products to be detected. The defect detection includes inputting the images to be detected into the defect detection model, extracting image features by the feature extraction network, and generating defect detection results by the output network. The defect detection results include the presence or absence of defects, defect types, and the location of defects in the images.

[0041] According to the defect detection results, alarm or classification information is output.

[0042] Specifically, digital images containing the surfaces of industrial products are acquired, and the images are labeled to form a training data set for defect detection tasks. In the preprocessing stage, the images can be subjected to grayscale, scale normalization, noise filtering and other operations to improve the defect feature signal-to-noise ratio.

[0043] ​An adaptive lightweight visual Transformer defect detection model is constructed. The model adopts a hierarchical encoder-decoder structure, including a visual Transformer-based feature extraction network and an output network for outputting defect judgment results. The feature extraction network performs block embedding at the input end, divides the input image into several image blocks and maps them into a low-dimensional feature vector sequence; then it extracts multi-scale feature representations by stacking multiple self-attention modules. Lightweight strategies are introduced in this process: for example, a multi-head self-attention mechanism is used to capture global context information, and a Token fusion layer is added after each self-attention module to fuse and reduce the dimension of adjacent redundant feature representations, gradually reducing the length of the feature sequence to reduce the subsequent computational load. In addition, convolution operators are integrated into the multi-layer perceptron sublayer of the Transformer network, and local operation modules such as depth separable convolution are introduced to increase the sensitivity of the model to local defect patterns and improve the detection ability of small size defects.

[0044] As shown in Figure 1 The industrial defect detection system of the embodiment of the present application includes an image acquisition device, a data processing device, and a display and alarm device. The image acquisition device can be a high-resolution industrial camera arranged on the production line site to obtain product surface images. The data processing device includes a processor and a memory, and internally deploys the above-mentioned visual Transformer defect detection model to perform defect detection algorithms on the obtained image data. The display and alarm device is used to prompt the detection results to the operator in the form of human-machine interface or sound-light alarm.

[0045] The process of the industrial defect detection method is further described below in combination with the system. First, the surface image of the product to be tested is obtained by the image acquisition device and transmitted to the data processing device; the data processing device performs image preprocessing to adjust the image to the required format and scale of the model input; then, the preprocessed image is converted into a patch sequence by the embedding module and input into the visual Transformer feature extraction network for feature calculation. In the feature extraction process, each Transformer encoding layer contains a self-attention unit and a lightweight module, as shown in Figure 3As shown, it includes an image embedding module, a self-attention calculation unit, a convolution module, a Token fusion dimension reduction module, and an optional dynamic decision module for adaptively adjusting the calculation of the subsequent layer according to the current feature distribution. Through the above structure, the defect detection model can extract rich feature representation layer by layer and effectively control the calculation cost. The feature extraction network finally outputs multi-scale fused defect discrimination features. Then, the defect discrimination features are sent to the output network, which includes several fully connected layers or convolution detection heads, for generating defect detection results, such as classification scores and location information of defect regions (bounding box or pixel mask). Finally, according to the defect detection results, the corresponding alarm signal or classification information is output through the display and alarm device.

[0046] The defect detection model constructed is trained using the above training data set, and a structure optimization mechanism is introduced during the training process. On the one hand, the knowledge distillation technology is used to improve the performance of the lightweight model: a pre-trained teacher model with higher performance is selected, and the predicted results of the defect classification or detection task are used as soft targets to guide the learning of the model. By minimizing the difference between the output of the student model and the output of the teacher model (for example, using Kullback-Leibler divergence as the distillation loss), the lightweight model can also achieve accuracy close to the teacher model on small sample data. On the other hand, an evolutionary algorithm is used to automatically optimize the model structure. On the basis of the initial model, a structure code containing network depth, width, and attention head number is defined, and each structure is regarded as an individual. A multi-objective fitness function F suitable for the defect detection task is designed, for example:

[0047]

[0048] where Acc represents the defect detection accuracy of the defect detection model on the validation data set, Param and FLOPs represent the parameter quantity and computational quantity of the defect detection model, Param0 and FLOPs0 are the parameters and computational quantity of the baseline model, and w1, w2, w3 are weight coefficients for balancing accuracy and complexity. The fitness function is used to evaluate the pros and cons of different structure individuals, and the selection, crossover, and mutation operations of the genetic algorithm are used to iteratively generate new model structure candidates. After multiple generations of evolution, the model structure with the highest detection accuracy under the premise of meeting the computational complexity constraint is gradually selected. Experiments show that this evolutionary optimization-based strategy can effectively discover non-obvious lightweight network architectures, significantly reduce model parameters and computational overhead, while maintaining a high defect detection rate.

[0049] The best model structure after training and optimization is deployed to the actual industrial defect detection system. The processor of the data processing device loads the model parameters and is connected with the image acquisition device and the display and alarm device in the industrial field to realize online defect detection. For new product images to be tested, the system runs according to the above-mentioned industrial defect detection method process: real-time image acquisition and preprocessing are completed, and then efficient defect feature extraction and discrimination are performed through the optimized visual Transformer defect detection model. In the inference process, the adaptive computing module (for example, the dynamic decision module in Figure 3 ) included in the defect detection model automatically adjusts the calculation graph according to the complexity of the image content: when the image is simple or has no obvious defects, part of the calculation is skipped to speed up the processing; when the image contains suspected defects, the deep network is fully enabled to extract details to ensure detection accuracy. This on-demand allocation of computing resources further improves the overall efficiency of the system.

[0050] Embodiments

[0051] In this embodiment, it is necessary to detect the tiny defects (such as tin beads, virtual welding, cracks) on the surface of the electronic component welding points. Referring to Figure 1 , first, the digital image of the product welding point area is acquired by using an industrial camera (image acquisition device). The image is input to the data processing device installed with the defect detection system. The processor of the data processing device first runs the image preprocessing module to perform denoising, contrast enhancement and other operations on the original image to obtain a clear grayscale image as the model input.

[0052] Then, the visual Transformer feature extraction module starts to work. The image embedding module crops the preprocessed welding point image into several 16x16 pixel blocks and maps them into feature vectors with a length of d (where d is the embedding dimension). These vector sequences are input into the multi-head self-attention calculation unit 302 after position encoding. Assuming that the input feature representation of a certain layer is , where N is the current feature sequence length, and d is the feature dimension. The self-attention unit is calculated as follows:

[0053]

[0054] where Q=XA Q , K=XA K , and V=XA VQuery, Key, Value matrices obtained by affine transformation of input X, M is a sparse mask matrix generated by the dynamic decision module based on the current feature distribution, used to suppress the attention weight of irrelevant tokens. Through this improved attention mechanism, the model can effectively filter redundant information while maintaining the ability of global modeling. Subsequently, the convolution module extracts local features from the attention output, and in this embodiment, a 3x3 depth separable convolution is used to efficiently extract the detail patterns within the neighborhood. Then, the Token fusion dimension reduction module performs weighted fusion on adjacent feature vectors, for example, adding and averaging adjacent two vectors element by element to generate a new compressed feature representation, thereby reducing the sequence length by half. These operations constitute a layer of the Transformer encoder. The model stacks L layers of the above structure to form a pyramid encoder that gradually shrinks the sequence length. The encoder output is restored to the spatial resolution through decoder shortcut layer-by-layer upsampling, and is fused with the multi-scale features extracted in the encoding process, and finally fed into the defect discrimination output network.

[0055] In the training phase, the ResNet50 pre-trained on ImageNet is selected as the teacher model, and the corresponding student model is the backbone of the above-mentioned Transformer defect detection model. The class probability distribution obtained by reasoning the training set image by the teacher model is used as the soft label. During the training process, the cross-entropy loss and the distillation loss are combined to optimize the student model parameters, wherein the distillation loss adopts the formula , wherein is the distillation loss, which is used to measure the difference between the output distribution of the student model and the output distribution of the teacher model, c is the class index, and represent the probabilities output by the teacher and student models for class c, respectively. By minimizing this loss, the student model effectively absorbs the knowledge of the teacher model, and improves the classification accuracy of the welding spot defects.

[0056] In addition, the model structure parameters are further optimized by the evolutionary algorithm. Specifically, the individual code is defined to contain the parameters of the number of hidden units, the number of attention heads, the size of the convolution kernel, etc. of each layer of the Transformer. In the initial population, several models with different structures are randomly generated, and their accuracy on the validation set and other indicators such as parameter quantity and reasoning time are evaluated, and the better individuals are selected according to the aforementioned fitness function F. Then, the selected individuals are crossed and exchanged part of the structure parameters, and the part of the parameters are randomly fine-tuned to generate a new generation of model architecture. After multiple iterations, a group of Pareto optimal model structure candidates are finally obtained. After considering the accuracy and efficiency, the finally adopted model structure is determined and the final model is obtained after complete training.

[0057] When the training is completed, the model is deployed in an industrial detection device for real-time defect detection. In actual operation, the system only needs millisecond-level processing time for each detection of a solder joint image, which can meet the production line beat requirements. Once a defect is detected, a display and alarm device will immediately issue a warning signal and highlight the location of the defect on the screen (for example, mark the location of the tin bead defect with a red border). The application of the embodiments of the present application in electronic component defect detection shows that while ensuring a defect recognition accuracy of nearly 99%, the model parameter quantity is reduced by about 60% compared to the original Transformer model, and the average single image detection time is shortened to 50% of the original, realizing the unification of high accuracy and high speed.

[0058] The above only describes the preferred embodiments of the present application, and it should be noted that for ordinary skilled persons in the art, without departing from the principles of the present application, several improvements and refinements can be made, and these improvements and refinements should also be considered within the scope of protection of the present application.

Claims

1. An adaptive lightweight visual Transformer network structure optimization method for industrial defect detection, characterized in that, The method comprises the following steps: acquiring image data of an industrial product, and pre-processing and defect labeling the image data to establish a defect sample data set for training; constructing an adaptive lightweight visual Transformer defect detection model, the defect detection model comprising a feature extraction network based on a visual Transformer and an output network for defect judgment, wherein the feature extraction network adopts a multi-head self-attention mechanism to obtain global features, and a convolution module is introduced to enhance local feature extraction, the structure of the feature extraction network is designed in a multi-scale hierarchical manner, and has adjustable network depth and width; training and optimizing the defect detection model using the defect sample data set, and using knowledge distillation to improve the accuracy of the lightweight defect detection model during the training process, and automatically adjusting part of the layers, nodes or parameter configurations of the defect detection model through a structure optimization algorithm to reduce the complexity of the model; deploying the trained and optimized defect detection model to an industrial defect detection system to detect defects in a to-be-detected image of a to-be-detected industrial product, comprising inputting the to-be-detected image into the defect detection model, extracting image features through the feature extraction network, and generating a defect detection result through the output network, the defect detection result comprising the presence or absence of a defect, the type of the defect, and the location of the defect in the image; outputting alarm or classification information according to the defect detection result.

2. The adaptive lightweight visual Transformer network structure optimization method for industrial defect detection according to claim 1, characterized in that, The construction of the adaptive lightweight visual Transformer defect detection model comprises: block encoding an input image to obtain an image patch sequence, inputting the image patch sequence into a multi-head self-attention module of the Transformer to perform feature calculation, and performing Token fusion dimension reduction on the feature map after each layer of self-attention calculation to reduce the length of the feature sequence layer by layer, thereby reducing the computational load.

3. The adaptive lightweight visual Transformer network structure optimization method for industrial defect detection according to claim 1, characterized in that, The convolution module introduced in the feature extraction network is a depth separable convolution module, which is used to provide the local perception ability of the convolutional neural network to make up for the deficiency of the pure Transformer architecture in small defect feature extraction.

4. The adaptive lightweight visual Transformer network structure optimization method for industrial defect detection according to claim 1, characterized in that, The structure optimization algorithm used in the training and optimization process is an evolutionary algorithm, which balances the accuracy and complexity of the defect detection model through a pre-set multi-objective loss function, and iteratively searches for a model structure that meets the complexity constraint while maintaining the accuracy of the defect detection.

5. The adaptive lightweight visual Transformer network structure optimization method for industrial defect detection according to claim 1, characterized in that, The structure optimization algorithm used in the training and optimization process is a reinforcement learning algorithm, which balances the accuracy and complexity of the defect detection model through a pre-set multi-objective loss function, and adjusts the structure parameters of the defect detection model based on the feedback of a reinforcement learning agent to obtain an optimal model structure that meets the complexity requirement.

6. The adaptive lightweight visual Transformer network structure optimization method for industrial defect detection according to claim 1, characterized in that, The knowledge distillation technology is used in the training and optimization process, which comprises using a pre-trained high-performance teacher model to guide the training of the defect detection model to minimize the difference between the outputs of the student model and the teacher model, thereby improving the detection accuracy of the lightweight defect detection model.

7. An adaptive lightweight visual Transformer network structure optimization system for industrial defect detection, characterized in that, A method for optimizing an adaptive lightweight visual Transformer network structure for industrial defect detection, as claimed in any one of claims 1-6, comprises: an image acquisition device for acquiring surface images of industrial products; The data processing device comprises a memory and a processor, the processor runs a trained and optimized adaptive lightweight visual Transformer defect detection model, and is used for defect detection processing on images acquired by an image acquisition device and output of a defect detection result. The display and alarm device is used for receiving and outputting the defect detection result provided by the data processing device.

8. The adaptive lightweight visual Transformer network structure optimization system for industrial defect detection according to claim 7, characterized in that, The data processing device comprises an image preprocessing module, a feature extraction module, a defect discrimination module and a result output module, wherein: The image preprocessing module is used for grayscale, normalization and filter denoising preprocessing of the acquired images; The feature extraction module is realized by a visual Transformer network and is used for extraction of multi-scale image feature representation; The defect discrimination module is used for defect classification and positioning judgment according to the extracted image features to obtain a defect detection result; The result output module is used for sending the defect detection result to the display and alarm device for output prompt.

9. The adaptive lightweight visual Transformer network structure optimization system for industrial defect detection according to claim 7, wherein, The data processing device is an edge computing device comprising an artificial intelligence acceleration unit and is used for accelerating the inference calculation process of the visual Transformer defect detection model.

Images