Construction method and application of defect target detection model in industrial defect detection scenarios

Abstract

The present invention provides a method for constructing a defect target detection model for an industrial defect detection scenario and its application. The defect target detection model includes an image block processing module, a sequence vector processing module, a backbone network and a prediction head, wherein the image block processing module performs image block processing on an input image to obtain image blocks, the sequence vector processing module processes the image blocks to obtain an input vector sequence, the input vector sequence is input into the backbone network and sequentially undergoes processing by an efficient cascade module, a downsampling layer, a dynamic information interaction module, a downsampling layer and an inverted residual module with a self-attention mechanism to obtain an output vector, the output vector is input into the prediction head to obtain a predicted target, the defect target detection model can perform well in defect target detection of images of different sizes, and has the advantage of efficient calculation, and is particularly suitable for the detection of defect targets in industrial defect detection scenarios.

Description

Technical Field

[0001] The present application relates to the field of visual target detection, and in particular to a method for constructing and applying a defect target detection model for industrial defect detection scenarios. Background Art

[0002] Defect detection is one of the most common functions of machine vision inspection equipment, primarily examining various surface characteristics of products. In modern automated industrial manufacturing, each process in continuous batch production has a corresponding defect rate. While these individual percentages may not be significant, when multiplied, they become a significant challenge for companies in improving yield rates. Furthermore, removing defective products after a complete production process is significantly more expensive. Therefore, timely inspection and elimination of defective products are crucial for quality and cost control, and are a fundamental foundation for further advancements in the manufacturing industry.

[0003] There is also research currently applying deep learning machine vision technology to industrial defect detection scenarios, such as using the Visual Transformer (ViT) model to detect defective objects. The difference between the Visual Transformer (ViT) and the CNN model lies in the way they process input images: ViT uses patching to cut images into non-overlapping patches and then performs subsequent operations on tokens obtained by linear mapping these patches; CNN, on the other hand, processes input images through dense, overlapping convolutions to convert them into feature maps. It can be found that ViT's completely different image processing method from CNN unlocks new capabilities. For example, unimportant tokens can be randomly discarded to improve model efficiency, additional tokens can be added to adapt to new tasks, and tokens of different modalities can be mixed.

[0004] However, the application of the Visual Transformer (ViT) model still has limitations:

[0005] (1) Most lightweight and efficient Vision transformers models are constructed by reducing model parameters or FLOPs. However, model parameters or FLOPs are indirect indicators of speed and cannot reflect the actual inference throughput of the model.

[0006] (2) The standard Vision transformers model can only achieve good performance at the patch size it is trained on; when the patch size changes, the model generally needs to be retrained. Summary of the Invention

[0007] The embodiments of the present application provide a method for constructing a defect target detection model for an industrial defect detection scenario and its application, which can perform well in defect target detection for images of different sizes and has the advantage of efficient computing.

[0008] In a first aspect, an embodiment of the present application provides a method for constructing a defect target detection model for an industrial defect detection scenario, comprising the following steps:

[0009] Obtain an image to be detected marked with defective objects as a training image;

[0010] The training images are processed in parallel by image block processing and sequence vector processing, and then respectively input into the two branches of the defect target detection model and the teacher model for training iterative training until the training conditions are met, wherein the training images are processed by image block processing to obtain image blocks, and the image blocks are processed by sequence vector processing to obtain input vector sequences, and the input vector sequences of the two branches corresponding to the defect target detection model and the teacher model are vector reshaped using the same block embedding weights; the defect target detection model and the teacher model both include a backbone network and a prediction head, and the input vector sequence is input into the backbone network for self-attention mechanism processing and then input into the prediction head for prediction to obtain a predicted target, and the weights of the backbone networks of the teacher model and the defect target detection model are shared;

[0011] The backbone network of the teacher model adopts the ViT-B / 8 model that is pre-trained by marking the image to be detected with defective targets. The backbone network of the defective target detection model includes an efficient cascade module, a downsampling layer, a dynamic information interaction module, a downsampling layer and an inverted residual module with a self-attention mechanism in sequence. The efficient cascade module is composed of a plurality of cascaded multi-head self-attention mechanism head modules, and each multi-head self-attention mechanism head module includes a local information feedforward processing module and a cascaded self-attention module; the dynamic information interaction module includes a feature inversion module, a local partition feature module connected in parallel to the feature inversion module, and a global partition feature module. The local partition result of the local partition feature module and the global partition result of the global partition feature module are element-wise added to obtain the output result of the dynamic information interaction module; the inverted residual module with a self-attention mechanism introduces a multi-head self-attention mechanism on the residual architecture.

[0012] In a second aspect, an embodiment of the present application provides a method for applying a defect target detection model in an industrial defect detection scenario, comprising the following steps:

[0013] Acquire an image to be tested, input the image to be tested into the defect target detection model of the industrial defect detection scenario constructed by the method for constructing the defect target detection model of the industrial defect detection scenario described above, and output the defect target.

[0014] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute a method for constructing a defect target detection model for an industrial defect detection scenario or a method for applying a defect target detection model for an industrial defect detection scenario.

[0015] In a fourth aspect, an embodiment of the present application provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute a process, and the process includes a method for constructing a defect target detection model for an industrial defect detection scenario or a method for applying a defect target detection model for an industrial defect detection scenario.

[0016] The main contributions and innovations of the present invention are as follows:

[0017] An embodiment of the present application provides a method for constructing and applying a defect target detection model for an industrial defect detection scenario. The model can dynamically adjust the size of image blocks during training so that the model can perform well for input images of different sizes during the inference process, thereby improving the generalization ability of the model. The introduction of efficient cascade modules and cascade self-attention modules reduces the use of attention, alleviates the problem of memory access time consumption caused by attention calculation, and introduces inductive bias.

[0018] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a diagram of the overall architecture of a defect target detection model according to an embodiment of the present application;

[0021] Figure 2 is an architecture diagram of a backbone network of a defect target detection model according to an embodiment of the present application;

[0022] Figure 3 1 is an architectural diagram of an efficient cascade module of a defect target detection model according to an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of a cascaded self-attention module of a defect target detection model according to an embodiment of the present application;

[0024] Figure 5 is a schematic diagram of a dynamic information interaction module of a defect target detection model according to an embodiment of the present application;

[0025] Figure 6 2 is a schematic diagram of an inverted residual module that introduces a self-attention mechanism into a defect target detection model according to an embodiment of the present application;

[0026] Figure 7 Schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments are not intended to represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims.

[0028] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0029] Example 1

[0030] The present invention provides a method for constructing a defect target detection model for an industrial defect detection scenario, including the following steps:

[0031] Obtain an image to be detected marked with defective objects as a training image;

[0032] The training images are processed in parallel by image block processing and sequence vector processing, and then respectively input into the two branches of the defect target detection model and the teacher model for training iterative training until the training conditions are met, wherein the training images are processed by image block processing to obtain image blocks, and the image blocks are processed by sequence vector processing to obtain input vector sequences, and the input vector sequences of the two branches corresponding to the defect target detection model and the teacher model are vector reshaped using the same block embedding weights; the defect target detection model and the teacher model both include a backbone network and a prediction head, and the input vector sequence is input into the backbone network for self-attention mechanism processing and then input into the prediction head for prediction to obtain a predicted target, and the weights of the backbone networks of the teacher model and the defect target detection model are shared;

[0033] The backbone network of the teacher model adopts the ViT-B / 8 model that is pre-trained by marking the image to be detected with defective targets. The backbone network of the defective target detection model includes an efficient cascade module, a downsampling layer, a dynamic information interaction module, a downsampling layer and an inverted residual module with a self-attention mechanism in sequence. The efficient cascade module is composed of a plurality of cascaded multi-head self-attention mechanism head modules, and each multi-head self-attention mechanism head module includes a local information feedforward processing module and a cascaded self-attention module; the dynamic information interaction module includes a feature inversion module, a local partition feature module connected in parallel to the feature inversion module, and a global partition feature module. The local partition result of the local partition feature module and the global partition result of the global partition feature module are element-wise added to obtain the output result of the dynamic information interaction module; the inverted residual module with a self-attention mechanism introduces a multi-head self-attention mechanism on the residual architecture.

[0034] In other words, this solution uses a teacher-student model and knowledge distillation to train the defective object detection model. This solution also optimizes the backbone network design of the defective object detection model itself. Knowledge distillation is described as follows: Knowledge distillation uses a teacher-student model. Once the teacher model is trained, the information learned by the teacher model is compressed into the student network, thereby reducing the size of the defective object detection model, which serves as the student model. The teacher is the "knowledge" outputter, and the student is the "knowledge" receiver.

[0035] The teacher model is more complex than the defect target detection model. This solution does not impose any restrictions on the backbone network of the teacher model in terms of model architecture, parameter number, or integration. The only requirement is that the classification probability can be output for the input features. This solution focuses on structural optimization of the defect target detection model to reduce its parameter number and simplify its structure. The teacher model is used to perform knowledge distillation on the defect target detection model, and the Kullback-Leibler divergence between the softmax of the teacher model and the softmax of the defect target detection model is minimized during training. The large ViT-B / 8 model is used as the teacher model for experiments, and optimization is performed by minimizing the KL distance between the predictions of the teacher and defect target detection models and the random block embeddings.

[0036] Specifically, in the step of "the training image is image-processed to obtain image blocks, the image blocks are sequence-vectored to obtain input vector sequences, and the input vector sequences of the two branches corresponding to the defect target detection model and the teacher model are vector-reshaped using the same block embedding weights", the training image enters the two branches corresponding to the defect target detection model and the teacher model respectively, and in each branch, the input training image is divided into multiple image blocks and each image block is converted into an embedding vector using embedded coding, and the learned position coding is added to the corresponding embedding vector to obtain the input vector sequence.

[0037] It should be noted that the image blocks in the defect target detection model branch undergo vector reshaping through the block embedding weights when the embedding code is converted to an embedding vector. The image blocks in the teacher model branch undergo vector reshaping through the block embedding weights when the embedding code is converted to an embedding vector, and they share the same block embedding weights. The block embedding in the block embedding weights is also called patch embedding. In order to input these image blocks into the backbone network for processing, each image block needs to be converted into a vector representation. This process is called patch embedding. The size of the image block controls the length of the input sequence. The block embedding weights refer to the weight parameters used when performing block embedding. These weight parameters define how to convert the image block into a vector representation. Convolutional Neural Networks (CNNs) are usually used to implement block embedding. The weights of the convolution kernels of the CNN can be regarded as block embedding weights. The input image block is convolved with the block embedding weights through a convolution operation to obtain a vector representation of each image block (that is, the vector sequence in this article). The choice of block embedding weights has a significant impact on model performance and result quality. Appropriate block embedding weights can extract key information from the image and preserve the image's semantic information. Typically, block embedding weights are learned by training on large-scale image datasets.

[0038] In short, the block embedding weight is a parameter used in the image block embedding process to convert the image block into a vector representation so that it can be input into the neural network for subsequent processing. The purpose is to adjust the length of the vector sequence, that is, the size of the image block. Smaller image blocks correspond to longer sequences that are more expressive, but slower, and vice versa.

[0039] Specifically, for a given input image x∈R h×w×c , where h, w, and c are the height, width, and number of channels of the input image respectively. The image block processing module divides the input image into s image blocks of size p*p x i ∈R p×p×c, i∈{1,...,s}, it should be noted that s=h / p·w / p is the number of image blocks or tokens after image segmentation, which controls the computational complexity of the subsequent model; the sequence vector processing module uses embedded coding Convert each image block into an embedding vector, representing the vector sequence of image blocks, calculated as in A sequence of vectors representing a single image block, w k ∈R p×p×c represents the block embedding weights that convert image blocks into vector sequences, <·,·> is the calculation of dot product, vec is the process of vectorizing multidimensional tensors, and the learned position encoding π i ∈R d Add to the embedded coding to get the input vector sequence t i =e i +π i ,π i It represents the position of the vector sequence of each image block, t i Represents a vector sequence containing position information, corresponding to the input vector sequence.

[0040] It should be noted that this scheme dynamically adjusts the image size of the image block when inputting the training image into the defect target detection model for training. The image size can be {12, 16, 24, 40, 48}, and the input vector sequence is sampled from the distribution of these image block sizes in each iteration.

[0041] like Figure 2 As shown in the figure, the architecture of the backbone network of the defect target detection model is presented. The backbone network consists of a sequentially connected efficient cascade module, a downsampling layer, a dynamic information interaction module, a downsampling layer, and an inverted residual module with a self-attention mechanism. In the embodiment of this scheme, it contains 1 efficient cascade module, 3 dynamic information interaction modules, and 4 inverted residual modules with a self-attention mechanism. The scale of the features input to the efficient cascade module is h / p*w / p*C1, the scale of the features input to the dynamic information interaction module is h / 2p*w / 2p*C2, and the scale of the features input to the inverted residual module with a self-attention mechanism is h / 4p*w / 4p*C3, where p is the image block, c1 has 192 channels, c2 has 288 channels, c3 has 384 channels, and h is the height of the input feature.

[0042] like Figure 3As shown, the architecture of an efficient cascade module is shown. The efficient cascade module includes a plurality of cascaded multi-head self-attention mechanism head modules, each of which includes a local information feedforward processing module and a cascaded self-attention module. In some embodiments, the efficient cascade module includes three multi-head self-attention mechanism head modules.

[0043] Specifically, each head module of the multi-head self-attention mechanism contains N local information feedforward processing modules and a cascaded self-attention module. The local information feedforward processing module includes a cascaded local information interaction module and a feedforward neural network. The local information interaction module uses a 7*7 depth-separable convolution. The feedforward neural network first normalizes the features, followed by a 1*1 convolution to convolve the features and reduce the channel dimension, followed by a 3*3 convolution with the channel dimension unchanged, followed by a 1*1 convolution and increasing the channel dimension to obtain the convolution feature. The initial features input into each local information feedforward processing module are first convolved with the local information interaction module and then element-wise added with the initial input features before being input into the feedforward neural network. The convolution features after the feedforward neural network processing are element-wise added with the initial input features to obtain the features input into the cascaded self-attention module.

[0044] It should be noted that this solution applies more feedforward neural network layers to allow communication between different channels, which is more memory-efficient. The application of feedforward neural network layers reduces the use of attention and alleviates the problem of memory access time consumption caused by attention calculation. At the same time, this solution adds a layer of depth-wise separable convolution before the feedforward neural network layer as information interaction between local tokens and helps introduce inductive bias.

[0045] like Figure 4 Figure 2 shows the architecture of the cascaded self-attention module in the efficient cascade module. Each cascaded self-attention module includes multiple cascaded self-attention heads. The input features of the cascaded self-attention module are split into multiple separate features according to the number of self-attention heads. Each separate feature is input to each self-attention head. The self-attention features output by the previous self-attention head are element-wise added to the separate features of the next self-attention head and then used as the input features of the next self-attention head. The self-attention features of all self-attention heads are stacked and projected to obtain the output features of the current cascaded self-attention module.

[0046] Specifically, each self-attention head projects the input features into a vector sequence V, a vector sequence K, and a vector sequence Q. Vector sequence Q is then input into the local information interaction module for convolution. After convolution, vector sequence V and vector sequence K are processed through the self-attention mechanism to obtain the self-attention features of the current self-attention head. In this embodiment, the local information interaction module uses a 7x7 depthwise separable convolution, which serves as information exchange between local vector sequences and introduces an inductive bias.

[0047] This scheme splits the features input to the cascaded self-attention module to obtain separation features to provide separate separation features to each self-attention head. The benefit of this is that it reduces computational redundancy. In order to explicitly encourage the self-attention heads to learn different patterns, this scheme applies an intuitive solution, that is, only providing a part of the separation features of the complete features to each self-attention head. Using different channel segmentation features in different self-attention heads can effectively reduce attention computational redundancy.

[0048] like Figure 5 As shown, the architecture of the dynamic information interaction module of the defect target detection model is demonstrated. The dynamic information interaction module includes a feature inversion module, a local partitioning feature module connected in parallel to the feature inversion module, and a global partitioning feature module. The local partitioning result of the local partitioning feature module and the global partitioning result of the global partitioning feature module are added element-wise to obtain the output result of the dynamic information interaction module. The dynamic information interaction module can help the defect target detection model to more accurately understand the target content of different sizes in the image, so that the spatial information between the encodings can obtain semantically stronger and more positionally accurate features, dynamically learn local and global information and maintain linear complexity. The dynamic information interaction module of this scheme emphasizes the flexible switching and integration of local information and global information during the learning process, aiming to improve the generalization of the defect target detection model.

[0049] The feature inversion module of this scheme is used to convert the one-dimensional vector sequence input into the dynamic information interaction module into a two-dimensional matrix, so as to input the two-dimensional matrix into the local partition feature module and the global partition feature module respectively to convert the local partition result and the global partition result into a one-dimensional vector sequence. The local partition result and the global partition result are added element-wise to obtain the output result of the dynamic information interaction module, wherein the local partition block in the local partition feature module is a spatially continuous image block of size P, and the global partition block in the global partition feature module is an overlapping image block of size 2P.

[0050] Specifically, the feature inversion module includes two branch bypasses, wherein the first branch bypass is a bypass connection of the identity mapping, and the feature input to the feature inversion module is processed by the first branch bypass to obtain the first feature, and the second branch bypass includes a 1*1 point-by-point convolution for extracting features and increasing the dimension of the feature channel, a batch normalization operation, a first activation function for increasing the nonlinearity of the feature, a depth-separable convolution for enriching the semantic feature with a small increase in the number of parameters, a second batch normalization, a second activation function, a spatial attention residual block, a 1*1 convolution for reducing the dimension of the feature channel, and batch normalization. The feature input to the feature inversion module is processed by the second branch bypass to obtain the second feature, and the first feature and the second feature are element-wise added to obtain the output feature of the feature inversion module.

[0051] The spatial attention residual block of this scheme includes a first bypass for feature identity mapping, and a second bypass consisting of global average pooling, a 1*1 convolution for reducing the dimension of the feature channel, and a 1*1 convolution for widening the feature channel. The features input into the spatial attention residual block after feature identity mapping by the first bypass are element-wise multiplied with the features after processing by the second bypass to obtain the spatial attention features of the spatial attention residual block.

[0052] The output features obtained by the feature inversion module of this scheme are two-dimensional matrices, which are then input into the local partition feature module and the global partition feature module respectively. The two-dimensional matrix features are converted into spatially continuous image blocks with a block size of P through the local partition feature module, and the two-dimensional matrix features are converted into image blocks with a block size of 2P through the global partition feature module. In other words, the local partitions of the local partition feature module are attention matrices composed of non-overlapping continuous image blocks. The global partition feature module uses a displacement window offset, so there will be local overlap between image blocks, and the image blocks of the global partition are twice the size of the image blocks of the local partition. The difference between the global partition feature module and the local partition feature module is that the global partition feature module adds an offset window (feature local intersection) to realize information interaction between different windows (image blocks).

[0053] Specifically, the local feature partitioning module includes a local partitioning module and a local residual structure connected in sequence, wherein the local partitioning module locally partitions the output features of the feature inversion module in the form of a two-dimensional matrix to obtain a spatially continuous image block of a block size of P, and the locally partitioned image block is output to the local residual structure to obtain a local partitioning result. The local residual structure is composed of two parallel local residual branches, the first local residual branch is an identity mapping branch, and the second local residual branch is composed of sequence feature extraction, channel dimension information averaging, feature flattening for flattening into a one-dimensional sequence, a feedforward network for performing a fully connected operation, and a softmax function connected in sequence. The features of the locally partitioned image block processed by the first local residual branch are element-wise multiplied with the features of the locally partitioned image block processed by the second local residual branch to obtain a local partitioning result.

[0054] The global feature partitioning module includes a global partitioning module and a global residual structure connected in sequence, wherein the global partitioning module globally partitions the output features of the feature inversion module in the form of a two-dimensional matrix to obtain image blocks with a block size of 2P, and the image blocks of the global partitions overlap with each other. In addition, the image blocks of the global partitions are input into the global residual structure to obtain a global partitioning result. The global partitioning residual structure is composed of three parallel global residual branches, the first global residual branch and the second global residual branch are both identity mapping branches, the third global residual branch is composed of a multi-head self-attention mechanism and a multi-layer perceptron operation, the third global residual feature obtained by the global partition image block through the third global residual branch is added to the second global residual feature obtained by the global partition image block through the second global residual branch at the element level and then a softmax operation is performed, and the feature after the softmax operation is then multiplied element-wise with the first global residual feature obtained by the global partition image block through the first global residual branch to obtain a global partitioning result.

[0055] Regarding the third global residual branch, the globally partitioned image blocks input into the third global residual branch are layer-normalized to obtain vector sequence Q / vector sequence K and vector sequence V. Vector sequence Q / vector sequence K and vector sequence V are linearly projected respectively. The features of vector sequence K after linear projection are element-wise multiplied with the features of vector sequence Q after linear projection, and then a Softmax operation is performed. The features after the Softmax operation are element-wise multiplied with the features of vector sequence V after linear projection. The multiplied features are then linearly projected, layer-normalized, and multi-layer perceptron-based to obtain the third global residual features.

[0056] like Figure 6As shown in the figure, the architecture of the defect target detection model with the introduction of the self-attention inverted residual attention module is demonstrated. The self-attention inverted residual attention module combines the inductive bias effect of convolution and the global modeling capability of self-attention in Transformer encoding, so that the self-attention inverted residual attention module has both the local modeling of CNN and relies on the global modeling capability of Transformer.

[0057] Specifically, the self-attention inverted residual attention module includes three parallel residual attention branches, wherein the first residual attention branch is the identity mapping branch, the second residual attention branch is composed of the first 1*1 point-by-point convolution, the embedded residual structure and the second 1*1 point-by-point convolution connected in sequence, wherein the embedded residual structure is composed of a parallel identity mapping branch and a depth-separable convolution branch, and the third residual attention branch is a multi-head attention mechanism, which inputs the vector sequence Q and the vector sequence K into the features of the self-attention inverted residual attention module into the third residual attention branch to obtain the third residual branch features, and inputs the features of the self-attention inverted residual attention module into the second residual attention branch. After the first 1*1 point-by-point convolution in the branch, a vector sequence V is obtained. The vector sequence V and the third residual branch features after the third residual attention branch are element-wise multiplied and input into the embedded residual structure. The features of the input embedded residual structure are respectively subjected to the identity mapping branch and the depth-wise separable convolution branch and then element-wise addition to obtain the embedded residual features. The embedded residual features are then subjected to the second 1*1 point-by-point convolution to obtain the second residual branch features. The features input into the self-attention inverted residual attention module are subjected to the identity mapping of the first residual attention branch to obtain the first residual branch features. The second residual branch features and the first residual branch features are element-wise added to obtain the output features of the self-attention inverted residual attention module.

[0058] Here, the vector sequence V is multiplied with the third residual branch feature formed by the vector sequence Q and the vector sequence K, which is equivalent to a multi-head self-attention operation; the vector sequence V has as many heads as the vector sequence Q and the vector sequence K, and is multiplied with the third residual branch feature.

[0059] Specifically, the vector sequence Q and vector sequence K input into the features of the self-attention inverted residual attention module satisfy Q = K ∈ R h×w×c , and the features input to the second residual attention branch are convolved by the first 1*1 point-by-point convolution to obtain the vector sequence V,V∈R h×w×λcThe dimension parameter is λ>1, which means that the dimension is increased along the way to enrich the features. The depthwise separable convolution branch embedded in the residual structure performs feature inversion on the features and then performs depthwise separable convolution to obtain local inductive bias semantic information. The features obtained by the depthwise separable convolution are then feature-inverted again and then element-wise added with the identity mapping features from the identity mapping branch to obtain the embedded residual features.

[0060] The corresponding defect target detection model of this solution and the prediction head of the teacher model include a bounding box regression branch of a classification branch, where the bounding box regression branch is used to output the predicted target box, and the classification branch outputs the confidence of the predicted target box.

[0061] It should be noted that in "the training images are processed in parallel by image segmentation and sequence vector processing and then input into the two branches of the defect target detection model and the teacher model for iterative training until the training conditions are met", the temperature parameter is applied to the output of the teacher model to obtain the smoothed probability distribution, where the smoothed probability distribution is used as the soft target of the defect target detection model, and the loss function of the defect target detection model is constructed according to the labeling of the soft target and the training image, and the loss function is controlled to be minimum to complete the training of the defect target detection model.

[0062] The temperature parameter is a hyperparameter used to control the flatness of the probability distribution. By adjusting the temperature parameter, the output of the teacher model can be made smoother. The loss function of the defect object detection model consists of a classification loss and a soft target loss. The classification loss guides the defect object detection model to correctly classify based on the labeling of the training images, while the soft target loss is used to keep the defect object detection model close to the soft target.

[0063] About the training of the defect target detection model:

[0064] Randomly initialize the connection weights w and bias b of each layer, give the learning rate η and the minimum batch Batch, select the activation function RELU, select the loss function Loss and the maximum number of iterations (algebra) under the current data; use the k-means algorithm to generate 9 candidate boxes, and assign three sizes of candidate boxes to each layer according to the large, medium and small targets, that is, divide the cells on the final output feature map of each layer, and use the feature pyramid where the output feature map size is W*H. According to the current data clustering result, if the clustering result under the current data shows that there are many large target data, assign more than 3 candidate boxes on the output feature map to make them appear in the output feature map. The feature map is traversed cell by cell in the figure. Secondly, according to the clustering results, less than 3 candidate boxes are allocated on the medium target feature map. Similarly, less than 3 candidate boxes are allocated on the small target feature map, so that the number of candidate boxes allocated on the three prediction heads is 9; after determining the above parameters, the labeled images are input into the architecture neural network of the network model in batches, and a candidate box corresponding to the target is predicted by the prediction head, and the final convolution output result is performed; the input data is iterated repeatedly until the error of the neural network output layer reaches the preset accuracy requirement or the number of training iterations reaches the maximum number of iterations, the training is ended, the network structure and parameters are saved, and a trained defect target detection model is obtained.

[0065] Of course, it should be noted that the defect target detection model provided by this solution can be continuously iterated during use. Regarding model iteration: collect a batch of new data, let the defect target detection model detect this batch of data, and divide the detection results into two categories: framed images and frameless images. Framed images are divided into true target images and false alarm target images. Frameless images can be divided into images with undetected targets and images without targets. False alarm target images are used as negative samples, and images containing industrial defect targets but not detected are used as training samples. These undetected target images are then labeled and enhanced. A new model is then trained based on the original model to test the model effect and check whether the accuracy meets the standards. If the new model does not meet the standards, new data is added and the network parameters are adjusted for training. If the model accuracy has met the requirements and is optimal under the current training data, training is stopped. This step is repeated to achieve a model that is suitable for the complexity of samples in the actual environment.

[0066] In the step of "obtaining images to be detected with defective objects marked as training images", this solution

[0067] Example 2

[0068] Based on the same content as Example 1, this solution provides a defect target detection model for an industrial defect detection scenario constructed according to the same content as Example 1. The defect target detection model is used to detect targets with defects, cracks and foreign objects.

[0069] Regarding the construction of the defect target detection model, the method of obtaining training images in this solution, that is, the method of preparing training and test data is as follows:

[0070] Surveillance video image acquisition: In the surveillance video, find L segments of videos containing surface defects, cracks, and foreign objects at different locations, times, and angles. Vi represents the i-th segment of video. There are Ni video images in Vi. From Ni video images, Mi video images are selected as training and test images. Then, from L segments of video, we can obtain Video images are used as training and test images. Data augmentation for the method of selecting M video images from N video images in a video as training and test images is as follows: Geometric transformation data augmentation: P target images of each category are collected and augmented by translation, rotation (45°, 90°, 180°, 270°), image reduction (1 / 3, 1 / 2), mosaic data augmentation, and shearing transformation. 8.25×P images are collected, for a total of 4×8.25×P target images of three categories. 0.9×4×8.25×P images are used as training data, and 0.1×4×8.25×P images are used as test data. The training and test data do not overlap.

[0071] The defect target detection model constructed according to Example 1 using the above training images can output the size of the industrial defect target bounding box and the confidence of the target when used, where the industrial defect target is a defect target with cracks, defects and foreign matter on the surface.

[0072] Example 3

[0073] Based on the same content as in Example 1, this solution provides an application method of a defect target detection model for an industrial defect detection scenario constructed based on the same content as in Example 1, including the following steps:

[0074] The image to be tested is acquired, and the image to be tested is input into the defect target detection model of the industrial defect detection scenario constructed according to the same content as in the first embodiment to output the defect target.

[0075] It should be noted that the methods provided in the field application of the present invention can be further expanded to other suitable application environments, not limited to specific application environments, including the following: Manufacturing: In the fields of automobile manufacturing, electronic equipment manufacturing, textile production, etc., the use of deep learning for industrial defect detection can effectively detect defects on products, such as surface defects, cracks, foreign matter, etc. Food processing: During the food processing process, deep learning can be used to detect problems such as appearance defects, deterioration, and foreign matter in food, helping to improve product quality and safety. Packaging and printing: Deep learning can be applied in the packaging industry to perform quality inspections on packaging materials such as paper and plastic film, as well as to detect defects and color differences in printed products during the printing process. Power and energy industry: Deep learning can be used for surface defect detection of power equipment, solar panels, pipelines, etc., as well as fault detection and prevention of power transmission lines. Steel and metal industry: In steel smelting and metal processing, deep learning technology can be used to automatically detect surface defects, cracks, etc. on products, improving product quality and safety. Medical device production: During the medical device manufacturing process, deep learning can be applied to detect problems such as appearance defects and dimensional deviations in the devices, ensuring product quality and compliance with standard requirements. In fact, deep learning has broad application prospects in the field of industrial defect detection and can be applied to various industries and fields that require automated inspection and quality control of products.

[0076] Example 4

[0077] This embodiment also provides an electronic device, referring to Figure 4 , including a memory 304 and a processor 302, wherein the memory 304 stores a computer program, and the processor 302 is configured to run the computer program to execute the steps in any of the above-mentioned embodiments of the method for constructing or applying a defect target detection model for an industrial defect detection scenario.

[0078] Specifically, the processor 302 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0079] Among them, the memory 304 may include a large-capacity memory 304 for data or instructions. By way of example and not limitation, the memory 304 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 304 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 304 may be inside or outside the data processing device. In a specific embodiment, the memory 304 is a non-volatile memory. In a specific embodiment, the memory 304 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, abbreviated as PROM), an erasable PROM (Erasable Programmable Read-Only Memory, abbreviated as EPROM), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, abbreviated as EEPROM), an electrically alterable ROM (Electrically Alterable Read-Only Memory, abbreviated as EAROM) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (Static Random-Access Memory, abbreviated as SRAM) or a dynamic random access memory (Dynamic Random Access Memory, abbreviated as DRAM), where the DRAM can be a fast page mode dynamic random access memory 304 (Fast Page Mode Dynamic Random Access Memory, abbreviated as FPMDRAM), an extended data output dynamic random access memory (Extended Date Out Dynamic Random Access Memory, abbreviated as EDODRAM), a synchronous dynamic random access memory (Synchronous Dynamic Random-Access Memory, abbreviated as SDRAM), etc.

[0080] The memory 304 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 302 .

[0081] The processor 302 reads and executes computer program instructions stored in the memory 304 to implement the construction or application method of the defect target detection model for any industrial defect detection scenario in the above embodiments.

[0082] Optionally, the electronic device may further include a transmission device 306 and an input / output device 308 , wherein the transmission device 306 is connected to the processor 302 , and the input / output device 308 is connected to the processor 302 .

[0083] Transmission device 306 can be used to receive or send data via a network. Specific examples of the aforementioned network may include a wired or wireless network provided by a telecommunications provider of the electronic device. In one embodiment, the transmission device includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In one embodiment, the transmission device 306 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0084] The input / output device 308 is used to input or output information. In this embodiment, the input information may be an image to be tested, and the output information may be a defect target.

[0085] Optionally, in this embodiment, the processor 302 may be configured to execute the following steps through a computer program:

[0086] Obtain an image to be detected marked with defective objects as a training image;

[0087] The training images are processed in parallel by image block processing and sequence vector processing, and then respectively input into the two branches of the defect target detection model and the teacher model for training iterative training until the training conditions are met, wherein the training images are processed by image block processing to obtain image blocks, and the image blocks are processed by sequence vector processing to obtain input vector sequences, and the input vector sequences of the two branches corresponding to the defect target detection model and the teacher model are vector reshaped using the same block embedding weights; the defect target detection model and the teacher model both include a backbone network and a prediction head, and the input vector sequence is input into the backbone network for self-attention mechanism processing and then input into the prediction head for prediction to obtain a predicted target, and the weights of the backbone networks of the teacher model and the defect target detection model are shared;

[0088] The backbone network of the teacher model adopts the ViT-B / 8 model that is pre-trained by marking the image to be detected with defective targets. The backbone network of the defective target detection model includes an efficient cascade module, a downsampling layer, a dynamic information interaction module, a downsampling layer and an inverted residual module with a self-attention mechanism in sequence. The efficient cascade module is composed of a plurality of cascaded multi-head self-attention mechanism head modules, and each multi-head self-attention mechanism head module includes a local information feedforward processing module and a cascaded self-attention module; the dynamic information interaction module includes a feature inversion module, a local partition feature module connected in parallel to the feature inversion module, and a global partition feature module. The local partition result of the local partition feature module and the global partition result of the global partition feature module are element-wise added to obtain the output result of the dynamic information interaction module; the inverted residual module with a self-attention mechanism introduces a multi-head self-attention mechanism on the residual architecture.

[0089] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be repeated here.

[0090] In general, various embodiments may be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or a controller or other computing device, or some combination thereof.

[0091] Embodiments of the present invention can be implemented by computer software, which is executable by the data processor of the mobile device, such as in the processor entity, or is implemented by hardware, or is implemented by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product can include one or more computer executable components configured to perform the embodiment when the program is running. One or more computer executable components can be at least one software code or a part thereof. In addition, at this point, it should be noted that any box of the logic flow in the figure can represent a program step, or interconnected logical circuits, boxes and functions, or a combination of program steps and logical circuits, boxes and functions. The software can be stored in physical media such as memory chips or storage blocks implemented in the processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs. Physical media is non-transient media.

[0092] Those skilled in the art should understand that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The above embodiments merely illustrate several embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for constructing a defect target detection model for an industrial defect detection scenario, characterized in that: The following steps are involved: Obtain an image to be detected marked with defective objects as a training image; The training images are processed in parallel by image block processing and sequence vector processing, and then respectively input into the two branches of the defect target detection model and the teacher model for training iterative training until the training conditions are met, wherein the training images are processed by image block processing to obtain image blocks, and the image blocks are processed by sequence vector processing to obtain input vector sequences, and the input vector sequences of the two branches corresponding to the defect target detection model and the teacher model are vector reshaped using the same block embedding weights; the defect target detection model and the teacher model both include a backbone network and a prediction head, and the input vector sequence is input into the backbone network for self-attention mechanism processing and then input into the prediction head for prediction to obtain a predicted target, and the weights of the backbone networks of the teacher model and the defect target detection model are shared; The backbone network of the teacher model adopts the ViT-B / 8 model that is pre-trained by marking the image to be detected with defective targets. The backbone network of the defective target detection model includes an efficient cascade module, a downsampling layer, a dynamic information interaction module, a downsampling layer and an inverted residual module with a self-attention mechanism in sequence. The efficient cascade module is composed of a plurality of cascaded multi-head self-attention mechanism head modules, and each multi-head self-attention mechanism head module includes a local information feedforward processing module and a cascaded self-attention module; the dynamic information interaction module includes a feature inversion module, a local partition feature module connected in parallel to the feature inversion module, and a global partition feature module. The local partition result of the local partition feature module and the global partition result of the global partition feature module are element-wise added to obtain the output result of the dynamic information interaction module; the inverted residual module with a self-attention mechanism introduces a multi-head self-attention mechanism on the residual architecture.

2. The method for constructing a defect target detection model for an industrial defect detection scenario according to claim 1, characterized in that: The local information feedforward processing module includes a cascaded local information interaction module and a feedforward neural network. The local information interaction module adopts a 7*7 depth-separable convolution. The feedforward neural network first normalizes the features, followed by a 1*1 convolution to convolve the features and reduce the channel dimension, then a 3*3 convolution with the channel dimension unchanged, followed by a 1*1 convolution and increasing the channel dimension to obtain the convolution feature. The initial features input into each local information feedforward processing module are first convolved by the local information interaction module and then element-wise added with the initial features of the input and then input into the feedforward neural network. The convolution features after processing by the feedforward neural network are element-wise added with the initial features of the input to obtain the features input into the cascaded self-attention module.

3. The method for constructing a defect target detection model for an industrial defect detection scenario according to claim 1, characterized in that: Each cascaded self-attention module includes multiple cascaded self-attention heads. The features input to the cascaded self-attention module are split according to the number of self-attention heads to obtain multiple separation features. Each separation feature is input to each self-attention head, and the self-attention features output by the previous self-attention head are element-wise added to the separation features of the next self-attention head as the input features of the next self-attention head. The self-attention features of all self-attention heads are feature stacked and feature projected to obtain the output features of the current cascaded self-attention module.

4. The method for constructing a defect target detection model for an industrial defect detection scenario according to claim 1, characterized in that: The feature inversion module is used to convert the one-dimensional vector sequence input into the dynamic information interaction module into a two-dimensional matrix, so as to input the two-dimensional matrix into the local partition feature module and the global partition feature module respectively to convert the local partition result and the global partition result into a one-dimensional vector sequence. The local partition result and the global partition result are added element-wise to obtain the output result of the dynamic information interaction module, wherein the local partition block in the local partition feature module is a spatially continuous image block of size P, and the global partition block in the global partition feature module is an overlapping image block of size 2P.

5. The method for constructing a defect target detection model for an industrial defect detection scenario according to claim 4, characterized in that: The local feature partitioning module includes local partitioning modules and local residual structures connected in sequence, wherein the local partitioning module locally partitions the output features of the feature inversion module in the form of a two-dimensional matrix to obtain spatially continuous image blocks with a block size of P, and the locally partitioned image blocks are output to the local residual structure to obtain local partitioning results. The local residual structure is composed of two parallel local residual branches, the first local residual branch is an identity mapping branch, and the second local residual branch is composed of sequence feature extraction, channel dimension information averaging, feature flattening for flattening into a one-dimensional sequence, a feedforward network for performing a fully connected operation, and a softmax function connected in sequence. The features of the locally partitioned image blocks processed by the first local residual branch are element-wise multiplied with the features of the locally partitioned image blocks processed by the second local residual branch to obtain local partitioning results.

6. The method for constructing a defect target detection model for an industrial defect detection scenario according to claim 4, characterized in that: The global feature partitioning module includes a global partitioning module and a global residual structure connected in sequence, wherein the global partitioning module globally partitions the output features of the feature inversion module in the form of a two-dimensional matrix to obtain image blocks with a block size of 2P. The image blocks of the global partitions overlap with each other. The global partitioning residual structure includes three parallel global residual branches, the first global residual branch and the second global residual branch are both identity mapping branches, and the third global residual branch is composed of a multi-head self-attention mechanism and a multi-layer perceptron operation. The third global residual feature obtained by the global partitioned image block through the third global residual branch is added element-wise to the second global residual feature obtained by the global partitioned image block through the second global residual branch, and then a softmax operation is performed. The feature after the softmax operation is then multiplied element-wise with the first global residual feature obtained by the global partitioned image block through the first global residual branch to obtain the global partitioning result.

7. The method for constructing a defect target detection model for an industrial defect detection scenario according to claim 1, characterized in that: The self-attention inverted residual attention module includes three parallel residual attention branches, wherein the first residual attention branch is the identity mapping branch, the second residual attention branch is composed of the first 1*1 point-by-point convolution, the embedded residual structure and the second 1*1 point-by-point convolution connected in sequence, wherein the embedded residual structure is composed of a parallel identity mapping branch and a depth-separable convolution branch, and the third residual attention branch is a multi-head attention mechanism. The vector sequence Q and the vector sequence K input into the features of the self-attention inverted residual attention module are input into the third residual attention branch to obtain the third residual branch features, and the features input into the self-attention inverted residual attention module are input into the second residual attention branch. After the first 1*1 point-by-point convolution, a vector sequence V is obtained. The vector sequence V and the third residual branch features after the third residual attention branch are element-wise multiplied and input into the embedded residual structure. The features of the input embedded residual structure are respectively subjected to the identity mapping branch and the depth-separable convolution branch and then element-wise addition to obtain the embedded residual features. The embedded residual features are then subjected to the second 1*1 point-by-point convolution to obtain the second residual branch features. The features input into the self-attention inverted residual attention module are subjected to the identity mapping of the first residual attention branch to obtain the first residual branch features. The second residual branch features and the first residual branch features are element-wise added to obtain the output features of the self-attention inverted residual attention module.

8. An application method of a defect target detection model in an industrial defect detection scenario, characterized in that: The following steps are involved: Acquire an image to be tested, input the image to be tested into the defect target detection model of the industrial defect detection scenario constructed according to the method for constructing a defect target detection model of the industrial defect detection scenario according to any one of claims 1 to 7, and output the defect target.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method for constructing a defect target detection model for an industrial defect detection scenario as described in any one of claims 1 to 7 or the method for applying a defect target detection model for an industrial defect detection scenario as described in claim 8.

10. A readable storage medium, characterized in that: A computer program is stored in the readable storage medium, and the computer program includes a program code for controlling a process to execute a process, and the process includes a method for constructing a defect target detection model for an industrial defect detection scenario as described in any one of claims 1 to 7 or a method for applying a defect target detection model for an industrial defect detection scenario as described in claim 8.

Images