An adaptive lightweight visual transformer network structure optimization method and system for industrial defect detection

Abstract

The application discloses an adaptive lightweight visual Transformer network structure optimization method and system for industrial defect detection, comprising the following steps: acquiring surface image data of industrial products, pre-processing the images and defect labeling; constructing an adaptive lightweight visual Transformer defect detection model; training and optimizing the defect detection model, including optimizing network structure hyperparameters by using an evolutionary algorithm, and balancing detection accuracy and model complexity by using a multi-objective loss function; deploying the trained and optimized model in an industrial detection system to detect defects in images of products to be measured, and outputting the types and positions of the defects. Compared with the prior art, the application can greatly reduce the model parameter quantity and calculation amount while ensuring the defect detection accuracy, realizes adaptive structure optimization and efficient deployment of the model, and is especially suitable for real-time online industrial defect detection applications.

Description

Technical Field

[0001] This invention relates to the field of industrial vision inspection technology, specifically to an adaptive lightweight vision Transformer network structure optimization method and system for industrial defect detection. Background Technology

[0002] Surface defect detection in industrial products is a crucial step in manufacturing quality control. Traditionally, it relies on manual visual inspection, which is inefficient and its accuracy is affected by human factors. With the development of computer vision and deep learning, automatic defect detection methods based on convolutional neural networks (CNNs) are increasingly being applied in industrial scenarios. However, traditional CNN models suffer from insufficient local feature extraction and difficulty in modeling long-distance dependencies, while directly using Transformer models faces high computational complexity and a large number of parameters, affecting real-time performance and deployment efficiency. Especially in scenarios with complex backgrounds and subtle defects, the model needs both global context awareness and efficient extraction of local detailed features.

[0003] Some existing research attempts to introduce Transformers into industrial defect detection to improve global feature modeling capabilities. For example, some literature proposes a lightweight deep learning model based on the Swin Transformer for TFT-LCD panel defect classification. This model achieves this by using token fusion for dimensionality reduction of each feature map layer, introducing depthwise separable convolutional modules to increase convolutional inductive bias, and employing knowledge distillation to compensate for the accuracy reduction caused by lightweighting. Such methods, to some extent, balance the accuracy and speed requirements of defect detection. However, current industrial defect detection models still have many shortcomings: on the one hand, many models are designed for specific defect types or datasets, resulting in limited generalization; on the other hand, most methods use fixed network structures, making it difficult to adaptively adjust to the characteristics of different products or defects, and thus failing to balance real-time performance and high accuracy in environments with limited computational resources.

[0004] Therefore, there is an urgent need for a new technical solution that can combine the global feature extraction advantages of Transformer networks with lightweight structure optimization methods, significantly reduce model complexity while ensuring accuracy, and adaptively adjust the model structure according to actual application needs to meet the requirements of real-time and accurate defect detection in industrial settings. Summary of the Invention

[0005] Technical Objective: To address the shortcomings of existing technologies, this invention discloses an adaptive lightweight visual Transformer network structure optimization method and system for industrial defect detection. Under the visual Transformer framework, it realizes automatic optimization and on-demand adjustment of the model structure, thereby significantly reducing the computational cost of the model while ensuring the accuracy of defect detection.

[0006] Technical solution: To achieve the above technical objectives, the present invention adopts the following technical solution:

[0007] An adaptive lightweight visual Transformer network structure optimization method for industrial defect detection includes the following steps:

[0008] Acquire image data of industrial products, preprocess and annotate the image data to establish a defect sample dataset for training;

[0009] An adaptive lightweight visual Transformer defect detection model is constructed. The defect detection model includes a feature extraction network based on visual Transformer and an output network for defect judgment. The feature extraction network adopts a multi-head self-attention mechanism to obtain global features and introduces a convolution module to enhance local feature extraction. The structure of the feature extraction network is designed according to a multi-scale layered structure and has adjustable network depth and width.

[0010] The defect detection model is trained and optimized using a defect sample dataset. During the training process, knowledge distillation is used to improve the accuracy of the lightweight defect detection model. The model complexity is reduced by automatically adjusting some layers, nodes, or parameter configurations of the defect detection model through a structure optimization algorithm.

[0011] The trained and optimized defect detection model is deployed to the industrial defect detection system to perform defect detection on the image of the industrial product to be inspected. This includes inputting the image to be inspected into the defect detection model, extracting image features through a feature extraction network, and generating defect detection results by an output network. The defect detection results include the presence or absence of defects, the type of defects, and the location of defects in the image.

[0012] The system outputs alarms or classification information based on the defect detection results.

[0013] Preferably, constructing an adaptive lightweight visual Transformer defect detection model includes: dividing the input image into blocks to obtain an image patch sequence, inputting the image patch sequence into the multi-head self-attention module of the Transformer for feature calculation, and performing token fusion dimensionality reduction on the feature map after each layer of self-attention calculation to reduce the length of the feature sequence layer by layer, thereby reducing the computational load.

[0014] Preferably, the convolutional module introduced in the feature extraction network is a depthwise separable convolutional module, which is used to provide the local perception capability of the convolutional neural network to make up for the shortcomings of the pure Transformer architecture in small defect feature extraction.

[0015] Preferably, the structure optimization algorithm used in the training and optimization process is an evolutionary algorithm. By pre-setting a multi-objective loss function, the accuracy and complexity of the defect detection model are weighed, and the model structure that meets the complexity constraints is obtained through iterative search while keeping the defect detection accuracy basically unchanged.

[0016] Preferably, the structural optimization algorithm used in the training and optimization process is a reinforcement learning algorithm. By pre-setting a multi-objective loss function, the accuracy and complexity of the defect detection model are weighed, and the structural parameters of the defect detection model are adjusted based on the feedback of the reinforcement learning agent to obtain the optimal model structure that meets the complexity requirements.

[0017] Preferably, the optimization process employs knowledge distillation techniques, including using a pre-trained high-performance teacher model to guide the training of the defect detection model, thereby minimizing the difference between the output of the student model and the teacher model, and thus improving the detection accuracy of the lightweight defect detection model.

[0018] An adaptive lightweight visual Transformer network structure optimization system for industrial defect detection, characterized in that it includes a method for implementing the aforementioned adaptive lightweight visual Transformer network structure optimization for industrial defect detection, comprising:

[0019] Image acquisition device used to acquire surface images of industrial products;

[0020] The data processing device has a built-in memory and processor. The processor runs a trained and optimized adaptive lightweight visual Transformer defect detection model, which is used to perform defect detection processing on images acquired by the image acquisition device and output defect detection results.

[0021] Display and alarm devices are used to receive and output defect detection results provided by data processing devices.

[0022] Preferably, the data processing device includes an image preprocessing module, a feature extraction module, a defect discrimination module, and a result output module, wherein:

[0023] The image preprocessing module is used to perform grayscale conversion, normalization, and filtering and noise reduction preprocessing on the acquired image;

[0024] The feature extraction module is implemented by a visual Transformer network and is used to extract multi-scale image feature representations;

[0025] The defect discrimination module is used to classify and locate defects based on the extracted image features, and obtain defect detection results;

[0026] The result output module is used to send the defect detection results to the display and alarm devices for output prompts.

[0027] Preferably, the data processing device is an edge computing device containing an artificial intelligence acceleration unit, used to accelerate the inference calculation process of the visual Transformer defect detection model.

[0028] Beneficial Effects: The adaptive lightweight visual Transformer network structure optimization method and system for industrial defect detection provided by this invention have the following beneficial effects:

[0029] 1. This invention integrates a depthwise separable convolutional module after the multi-head self-attention module of the visual Transformer, and introduces a token fusion dimensionality reduction mechanism in each layer. This enables the model to efficiently capture global contextual information while precisely extracting local defect features. This collaborative design significantly enhances the ability to identify minute defects and complex backgrounds. At the same time, by compressing the feature sequence layer by layer, it greatly reduces the computational load and memory consumption of the model, thereby meeting the dual requirements of speed and resources for real-time detection in industrial settings.

[0030] 2. This invention employs knowledge distillation technology to transfer the discriminative power of a high-performance teacher model to a lightweight student model. Combined with structure optimization based on evolutionary algorithms (or reinforcement learning), it automatically searches for the optimal network depth, width, and attention head configuration. This strategy not only enables the lightweight model to approach or reach the accuracy level of large models but also effectively controls model size. More importantly, relying on the input dependency structure generated by token fusion dimensionality reduction, the model can skip redundant calculations for simple scenarios and perform in-depth execution for complex scenarios during inference. This ensures high detection accuracy while enabling on-demand allocation of computing resources, providing a flexible and reliable deployment solution for edge devices or resource-constrained industrial PCs, significantly expanding the application scope and practical value of industrial defect detection technology. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0032] Figure 1 This is a schematic diagram of the system structure of the present invention;

[0033] Figure 2 This is a schematic diagram of the method flow of the present invention;

[0034] Figure 3 This is a schematic diagram of the adaptive lightweight visual Transformer defect detection model. Detailed Implementation

[0035] The present invention will now be described more clearly and completely by way of a preferred embodiment in conjunction with the accompanying drawings, but this does not limit the invention to the scope of the described embodiment.

[0036] like Figure 2 As shown, this invention discloses an adaptive lightweight visual Transformer network structure optimization method for industrial defect detection, comprising the following steps:

[0037] Acquire image data of industrial products, preprocess and annotate the image data to create a defect sample dataset for training.

[0038] An adaptive, lightweight visual Transformer defect detection model is constructed. The defect detection model includes a feature extraction network based on visual Transformer and an output network for defect judgment. The feature extraction network adopts a multi-head self-attention mechanism to obtain global features and introduces a convolutional module to enhance local feature extraction. The structure of the feature extraction network is designed according to a multi-scale hierarchical structure and has adjustable network depth and width.

[0039] The defect detection model is trained and optimized using a defect sample dataset. During the training process, knowledge distillation is used to improve the accuracy of the lightweight defect detection model. Furthermore, a structural optimization algorithm is used to automatically adjust the number of layers, nodes, or parameter configurations of the defect detection model to reduce model complexity.

[0040] The trained and optimized defect detection model is deployed to an industrial defect detection system to perform defect detection on images of industrial products to be inspected. Defect detection involves inputting the image to be inspected into the defect detection model, extracting image features through a feature extraction network, and generating defect detection results from the output network. The defect detection results include the presence or absence of defects, the type of defects, and the location of defects in the image.

[0041] The system outputs alarms or classification information based on the defect detection results.

[0042] Specifically, digital images containing the surface of industrial products are acquired, and these images are labeled to form a training dataset for the defect detection task. In the preprocessing stage, operations such as grayscale conversion, scale normalization, and noise filtering can be performed on the images to improve the signal-to-noise ratio of defect features.

[0043] An adaptive, lightweight visual Transformer defect detection model is constructed. This model employs a hierarchical encoder-decoder structure, including a feature extraction network based on the visual Transformer and an output network for determining defect results. The feature extraction network performs block embedding on the input image, dividing it into several image blocks and mapping them to a sequence of low-dimensional feature vectors. Then, it extracts multi-scale feature representations by stacking multiple self-attention modules. Lightweight strategies are introduced in this process: for example, a multi-head self-attention mechanism is used to capture global image context information, and a token fusion layer is added after each self-attention module to fuse and reduce the dimensionality of adjacent redundant feature representations, gradually reducing the length of the feature sequence to decrease subsequent computational cost. Furthermore, convolution operators are fused into the multi-layer perceptron sublayers of the Transformer network, and local operation modules such as depthwise separable convolution are introduced to increase the model's sensitivity to local defect patterns and improve its ability to detect small-sized defects.

[0044] like Figure 1 As shown, the industrial defect detection system of this invention includes an image acquisition device, a data processing device, and a display and alarm device. The image acquisition device can be a high-resolution industrial camera, deployed on the production line to acquire images of the product surface. The data processing device includes a processor and a memory, and internally deploys the aforementioned visual Transformer defect detection model for executing defect detection algorithms on the acquired image data. The display and alarm device is used to prompt the operator with the detection results through a human-machine interface or an audible and visual alarm.

[0045] The process of the industrial defect detection method is further explained below with reference to the system. First, an image acquisition device acquires a surface image of the product to be tested and transmits it to a data processing device. The data processing device performs image preprocessing, adjusting the image to the format and scale required for the model input. Then, the preprocessed image is converted into a patch sequence via an embedding module and input into a visual Transformer feature extraction network for feature calculation. During feature extraction, each Transformer encoding layer contains a self-attention unit and a lightweight module, such as... Figure 3As shown, the model includes an image embedding module, a self-attention computation unit, a convolutional module, a token fusion and dimensionality reduction module, and an optional dynamic decision module, used to adaptively adjust the computation of subsequent layers based on the current feature distribution. Through this structure, the defect detection model can extract rich feature representations layer by layer while effectively controlling computational costs. The feature extraction network ultimately outputs multi-scale fused defect discrimination features. Next, the defect discrimination features are fed into the output network, which includes several fully connected layers or convolutional detection heads to generate defect detection results, such as classification scores and the location information of the defect region (bounding box or pixel mask). Finally, based on the defect detection results, corresponding alarm signals or classification information are output through a display and alarm device.

[0046] The constructed defect detection model is trained using the aforementioned training dataset, and a structural optimization mechanism is introduced during the training process. On one hand, knowledge distillation technology is used to improve the performance of the lightweight model: a pre-trained, high-performance teacher model is selected, and the prediction results of defect classification or detection tasks are used as a soft objective to guide the learning of the model. By minimizing the difference between the student model output and the teacher model output (e.g., using Kullback-Leibler divergence as distillation loss), the lightweight model can achieve accuracy close to that of the teacher model even on small sample data. On the other hand, an evolutionary algorithm is used to automatically optimize the model structure. Based on the initial model, a structural encoding including parameters such as network depth, width, and number of attention heads is defined, treating each structure as an individual. A multi-objective fitness function F suitable for defect detection tasks is designed, for example:

[0047]

[0048] Here, Acc represents the defect detection accuracy of the defect detection model on the validation dataset, Param and FLOPs represent the number of parameters and computational cost of the defect detection model, respectively, Param0 and FLOPs0 are the parameters and computational cost of the baseline model, and w1, w2, and w3 are weight coefficients used to balance accuracy and complexity. This fitness function is used to evaluate the performance of different model structures, and a genetic algorithm is employed for selection, crossover, and mutation operations to iteratively generate new candidate model structures. After multiple generations of evolution, the model structure with the highest detection accuracy while satisfying computational complexity constraints is gradually selected. Experiments show that this evolutionary optimization-based strategy can effectively discover non-obvious lightweight network architectures, significantly reduce model parameters and computational overhead, while maintaining a high defect detection rate.

[0049] The optimized model structure, trained and optimized, is deployed to a real-world industrial defect detection system. The processor of the data processing unit loads the model parameters and connects it to the image acquisition, display, and alarm devices in the industrial field to achieve online defect detection. For a new image of a product to be tested, the system operates according to the industrial defect detection method described above: real-time image acquisition and preprocessing, followed by efficient defect feature extraction and discrimination using the optimized visual Transformer defect detection model. During inference, the defect detection model includes adaptive computation modules (e.g., ...). Figure 3 The dynamic decision-making module automatically adjusts the computation layers based on the complexity of the image content: when the image is simple or has no obvious defects, some computations are skipped to speed up processing; when the image contains suspected defects, deep networks are fully utilized to extract details to ensure detection accuracy. This mechanism of allocating computational resources on demand further improves the overall operating efficiency of the system.

[0050] Example

[0051] In this embodiment, it is necessary to detect minute defects (such as solder balls, cold solder joints, and cracks) on the surface of electronic component solder joints. (See reference...) Figure 1 First, an industrial camera (image acquisition device) is used to acquire digital images of the weld area of ​​the product. The images are then input into a data processing unit equipped with a defect detection system. The processor of the data processing unit first runs the image preprocessing module, performing operations such as noise reduction and contrast enhancement on the original image to obtain a clear grayscale image as model input.

[0052] Then, the visual Transformer feature extraction module begins its work. The image embedding module crops the preprocessed solder joint image into several 16×16 pixel blocks and maps them into feature vectors of length d (where d is the embedding dimension). These vector sequences are then positionally encoded and input into the multi-head self-attention computation unit 302. Assume the input features of a certain layer are represented as... Where N is the length of the current feature sequence and d is the feature dimension. The self-attention unit is calculated as follows:

[0053]

[0054] Where Q=XA Q K=XA K V=XA VHere, X represents the query, key, and value matrices obtained through affine transformation of the input X, respectively. M is a sparse mask matrix generated by the dynamic decision module based on the current feature distribution, used to suppress the attention weights of irrelevant tokens. Through this improved attention mechanism, the model can effectively filter redundant information while maintaining global modeling capabilities. Subsequently, the convolution module refines local features from the attention output. In this embodiment, a 3×3 depthwise separable convolution is used to efficiently extract detailed patterns within the neighborhood. Next, the token fusion and dimensionality reduction module performs weighted fusion on adjacent feature vectors, for example, by adding adjacent vectors element-wise and averaging them to generate a compressed new feature representation, thereby halving the sequence length. These operations constitute one layer of the Transformer encoder. The model stacks L layers of the above structure to form a pyramid encoder that gradually shrinks the sequence length. The encoder output is short-circuited by the decoder and upsampled layer by layer to restore the spatial resolution, fused with the multi-scale features extracted during the encoding process, and finally fed into the defect discrimination output network.

[0055] During the training phase, a ResNet50 pre-trained on ImageNet was selected as the teacher model, and the corresponding student model served as the backbone of the aforementioned Transformer defect detection model. The teacher model was used to infer the class probability distribution from the training set images, which served as soft labels. During training, the student model parameters were optimized by combining cross-entropy loss and distillation loss, where the distillation loss was calculated using the formula... ,in Distillation loss measures the difference between the student model's output distribution and the teacher model's output distribution, where c is the class index. and Let represent the probabilities of the teacher and student models outputting category c, respectively. By minimizing this loss, the student model effectively absorbs the knowledge from the teacher model, improving the accuracy of classifying solder joint defects.

[0056] Furthermore, evolutionary algorithms are used to further optimize the model's hyperparameters. Specifically, individual encodings are defined to include parameters such as the number of hidden units per layer of the Transformer, the number of attention heads, and the kernel size. Several models with different structures are randomly generated from the initial population, and their accuracy on the validation set, as well as metrics such as parameter count and inference time, are evaluated. The best individuals are selected based on the aforementioned fitness function F. Then, the selected individuals undergo crossover to exchange some structural parameters and random fine-tuning of some parameters to generate a new generation of model architectures. After multiple iterations, a set of Pareto-optimal model architecture candidates is finally obtained. Considering both accuracy and efficiency, the final model architecture is determined, and the model is fully trained to obtain the final model.

[0057] Once training is complete, the model is deployed in industrial inspection equipment for real-time defect detection. In actual operation, the system requires only milliseconds of processing time to detect each solder joint image, which meets the production line cycle time requirements. Once a defect is detected, the display and alarm devices immediately issue a warning signal and highlight the location of the defect on the screen (e.g., marking the location of solder ball defects with a red border). The application of this invention in electronic component defect detection shows that while maintaining a defect recognition accuracy of nearly 99%, the number of model parameters is reduced by approximately 60% compared to the original Transformer model, and the average detection time per image is shortened to 50%, achieving a balance between high accuracy and high speed.

[0058] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An adaptive lightweight visual Transformer network structure optimization method for industrial defect detection, characterized in that, Includes the following steps: Acquire image data of industrial products, preprocess and annotate the image data to establish a defect sample dataset for training; An adaptive lightweight visual Transformer defect detection model is constructed. The defect detection model includes a feature extraction network based on visual Transformer and an output network for defect judgment. The feature extraction network adopts a multi-head self-attention mechanism to obtain global features and introduces a convolution module to enhance local feature extraction. The structure of the feature extraction network is designed according to a multi-scale layered structure and has adjustable network depth and width. The construction of the adaptive lightweight visual Transformer defect detection model includes: dividing the input image into blocks to obtain an image patch sequence, inputting the image patch sequence into the multi-head self-attention module of the Transformer for feature calculation, and performing token fusion dimensionality reduction on the feature map after each layer of self-attention calculation to reduce the length of the feature sequence layer by layer, thereby reducing the amount of computation. The defect detection model is trained and optimized using a defect sample dataset. During the training process, knowledge distillation is used to improve the accuracy of the lightweight defect detection model. The model complexity is reduced by automatically adjusting some layers, nodes, or parameter configurations of the defect detection model through a structure optimization algorithm. The trained and optimized defect detection model is deployed to the industrial defect detection system to perform defect detection on the image of the industrial product to be inspected. This includes inputting the image to be inspected into the defect detection model, extracting image features through a feature extraction network, and generating defect detection results by an output network. The defect detection results include the presence or absence of defects, the type of defects, and the location of defects in the image. The system outputs alarms or classification information based on the defect detection results.

2. The adaptive lightweight visual Transformer network structure optimization method for industrial defect detection according to claim 1, characterized in that, The convolutional module introduced in the feature extraction network is a depthwise separable convolutional module, which is used to provide the local perception capability of the convolutional neural network to make up for the shortcomings of the pure Transformer architecture in small defect feature extraction.

3. The adaptive lightweight visual Transformer network structure optimization method for industrial defect detection according to claim 1, characterized in that, The structure optimization algorithm used in the training and optimization process is an evolutionary algorithm. By pre-setting a multi-objective loss function, the accuracy and complexity of the defect detection model are weighed. Under the premise of keeping the defect detection accuracy basically unchanged, the model structure that meets the complexity constraints is obtained through iterative search.

4. The adaptive lightweight visual Transformer network structure optimization method for industrial defect detection according to claim 1, characterized in that, The structure optimization algorithm used in the training and optimization process is a reinforcement learning algorithm. By pre-setting a multi-objective loss function, the accuracy and complexity of the defect detection model are weighed. Based on the feedback of the reinforcement learning agent, the structural parameters of the defect detection model are adjusted to obtain the optimal model structure that meets the complexity requirements.

5. The adaptive lightweight visual Transformer network structure optimization method for industrial defect detection according to claim 1, characterized in that, The optimization process employs knowledge distillation techniques, including using a pre-trained high-performance teacher model to guide the training of the defect detection model, in order to minimize the difference between the output of the student model and the teacher model, thereby improving the detection accuracy of the lightweight defect detection model.

6. An adaptive lightweight visual Transformer network structure optimization system for industrial defect detection, characterized in that, An adaptive lightweight visual Transformer network structure optimization method for industrial defect detection as described in any one of claims 1-5 includes: Image acquisition device used to acquire surface images of industrial products; The data processing device has a built-in memory and processor. The processor runs a trained and optimized adaptive lightweight visual Transformer defect detection model, which is used to perform defect detection processing on images acquired by the image acquisition device and output defect detection results. Display and alarm devices are used to receive and output defect detection results provided by data processing devices.

7. The adaptive lightweight visual Transformer network structure optimization system for industrial defect detection according to claim 6, characterized in that, The data processing device includes an image preprocessing module, a feature extraction module, a defect discrimination module, and a result output module, wherein: The image preprocessing module is used to perform grayscale conversion, normalization, and filtering and noise reduction preprocessing on the acquired image; The feature extraction module is implemented by a visual Transformer network and is used to extract multi-scale image feature representations; The defect discrimination module is used to classify and locate defects based on the extracted image features, and obtain defect detection results; The result output module is used to send the defect detection results to the display and alarm devices for output prompts.

8. The adaptive lightweight visual Transformer network structure optimization system for industrial defect detection according to claim 6, characterized in that, The data processing device is an edge computing device containing an artificial intelligence acceleration unit, used to accelerate the inference calculation process of the visual Transformer defect detection model.

Images