Image anomaly detection method and system based on lightweight target detection model

Abstract

The invention discloses an image anomaly detection method and system based on a lightweight target detection model, and the method comprises the steps: constructing an improved lightweight target detection model based on a YOLOv8 model, which comprises a backbone network, a feature fusion module and a lightweight shared group convolution detection head; selecting an image containing to-be-detected anomalies, marking the positions of the to-be-detected anomalies in the image, and recording coordinate information of the anomalies to form an anomaly detection data set; and training the constructed target detection model by using the image anomaly detection data set, and outputting the trained target detection model for realizing image anomaly detection. The target detection model constructed by the method has high flexibility and adaptability, and can be widely applied to various industrial scenes including fabric defect detection, electrical equipment defect detection, drug appearance anomaly detection and other actual requirements.

Description

Technical Field

[0001] The present invention belongs to the field of target detection and relates to a lightweight target detection network model, and specifically to image anomaly detection, which can be applied to lightweight neural network image anomaly detection in various scenarios. Background Art

[0002] Object detection algorithms play a vital role in industrial production and quality management, and can effectively improve production efficiency and ensure product quality. However, traditional manual detection methods have significant shortcomings, such as slow detection speed and high missed detection rate, which are difficult to meet the fast-paced and high-precision manufacturing needs of modern industry. With the advancement of science and technology, especially the development of deep learning methods, object detection algorithms based on convolutional neural networks (CNNs) have become the mainstream solution. At present, mainstream object detection methods are mainly divided into two categories: region proposal-based methods, such as R-CNN, Faster R-CNN, etc. These methods use region proposal networks (RPNs) to generate candidate regions, and then classify and regress each candidate region. Although these methods perform well in detection accuracy, they have high computational costs and slow detection speeds, making it difficult to meet real-time requirements. Regression-based methods, such as YOLO (You Only Look Once) and SSD (Single Shot MultiBox Detector), transform the object detection problem into a regression problem, and directly predict the location and category of the target through a single forward propagation. These methods are favored for their faster detection speed and are more suitable for real-time application scenarios. Although the above methods have made significant progress, they still face many challenges in practical applications. Target images in industrial environments often contain complex local and global features, and their anomalies or defects may be embedded in normal structures in an extremely hidden way, increasing the difficulty of detection. In addition, in order to achieve large-scale deployment, reducing the computational cost and storage requirements of deep learning models remains an important issue to be solved.

[0003] Therefore, developing a lightweight target detection method with both high precision and low computing resource consumption can not only meet the detection needs in complex scenes, but also meet the deployment requirements of real-time and resource-constrained devices. This is of great significance for promoting the widespread application of target detection technology in industrial production and other fields. Summary of the invention

[0004] The present invention aims to propose solutions to two core problems, namely, further optimizing the network structure based on the existing YOLOv8 model to significantly reduce the consumption of computing resources, while ensuring that the lightweight design of the model does not sacrifice detection accuracy. Through this improvement, the present invention can effectively meet the high-precision and real-time requirements for target detection in industrial production, quality inspection and other practical application scenarios, and provide a reliable solution for efficient deployment in resource-constrained environments.

[0005] In order to solve the above problems, the present invention provides an image anomaly detection method based on a lightweight target detection model, which combines innovative network architecture and optimization technology to ensure that while maintaining high detection accuracy, the number of model parameters and computational complexity are significantly reduced, thereby improving the real-time performance and applicability of the method. The specific technical solution includes the following:

[0006] Step 1: Build a lightweight target detection model based on the improved YOLOv8 model, including three parts: backbone network, feature fusion module and lightweight shared group convolution detection head;

[0007] Combining ghost convolution GhostConv and deep convolution DWConv, and integrating with the lightweight backbone network HGNetV2, a new backbone network GH-HGV2Net is constructed to extract features from the input image. The entire new backbone network generates two output feature layers in the middle stage.

[0008] The processing process of the feature fusion module is as follows: the final output features of the new backbone network are passed through the SPPF module to generate the third output feature layer; the phantom dynamic convolution layer C2fGhostDynaMic is constructed by combining the dynamic convolution DynaMicConv, C2f module and ghost convolution GhostConv; the three output feature layers are multi-level fused through the ghost dynamic convolution layer, upsampling layer, ghost convolution and splicing operations, and finally three feature maps of different sizes are output;

[0009] Combining group convolution and convolution operations with shared parameters to build a lightweight shared group convolution detection head LSCD for target prediction;

[0010] Step 2: Select an image containing anomalies to be detected, mark the positions of the anomalies to be detected in the image, and record the coordinate information of the anomalies to form an anomaly detection data set;

[0011] Step 3: Use the anomaly detection dataset to train the constructed target detection model, and output the trained target detection model to realize image anomaly detection.

[0012] Furthermore, the new backbone network GH-HGV2Net includes a preliminary extraction module HGStem and 4 stages;

[0013] The four stages are stage1 to stage4, where stage1 contains a GH-HGBlock, stage2 contains a deep convolution DWConv and a GH-HGBlock, stage3 contains a DWConv and three consecutive GH-HGBlocks, and stage4 is the same as stage2.

[0014] Furthermore, the preliminary extraction module HGStem includes multiple ordinary convolution and pooling layers.

[0015] Furthermore, the processing flow of GH-HGBlock is as follows: first, the input features are subjected to n ghost convolution GhostConv operations to generate a series of feature maps; then, these feature maps are spliced ​​through the Concatenate operation to connect the feature maps after each GhostConv operation; next, the spliced ​​feature maps are subjected to the SqueezeConv and ExcitationConv operations to extract the channel information and feature response of the feature maps; finally, according to the condition that the number of channels of the original feature map and the number of channels of the feature map after the intermediate operation are the same, the original input and the feature map generated by the GhostConv and Concatenate operations are added to generate the final feature map.

[0016] Furthermore, the ghost dynamic convolution layer C2fGhostDynaMic includes a dynamic convolution DynamicConv and a ghost bottleneck layer GhostBottleNeck; the processing process of C2fGhostDynaMic is as follows: the input feature map first passes through the dynamic convolution DynamicConv, and then the output result is divided into two parts, the first part of the result passes through two identical GhostBottleNecks, and then the second part of the feature map after the segmentation is spliced ​​with the feature map after the first GhostBottleNeck and the second GhostBottleNeck operation, and finally passes through a DynamicConv;

[0017] The processing process of GhostBottleNeck is as follows: the input first passes through the ghost convolution GhostConv, then passes through a batch normalization layer and an activation layer, then passes through GhostConv, and finally adds it to the original feature map to output the final feature map.

[0018] Furthermore, the specific processing process of the feature fusion module is as follows:

[0019] The features extracted by the new backbone network are first processed by the SPPF module, followed by an upsample operation, and the upsampled feature map is spliced ​​with the feature map provided by the intermediate stage stage3; then, it passes through a C2fGhostDynaMic module marked as 13 layers, and then upsamples again, and splices the upsampled feature map with the feature map provided by the intermediate stage stage2; at this time, it passes through another C2fGhostDynaMic module marked as 16 layers, and the generated feature map is used to provide to the detection head for detecting small targets; the 16-layer feature map then passes through a GhostConv upsampling operation, and then the upsampled feature map is spliced ​​again with the first spliced ​​feature map, that is, the 13-layer feature map, and then passes through a C2fGhostDynaMic module marked as 19 layers, and provides it to the detection head of the target under detection; in addition, the 19-layer feature map passes through another GhostConv operation, and then spliced ​​with the feature map after SPPF; finally, the spliced ​​feature map passes through a C2fGhostDynaMic module marked as 22 layers and is provided to the detection head for detecting large targets.

[0020] Furthermore, the processing process of the lightweight shared group convolution detection head LSCD is as follows:

[0021] First, feature maps of different sizes are processed by group convolution Conv_GN1x1 to reduce the amount of calculation; then, the features processed by Conv_GN1x1 are input into the group convolution of two Conv_GN3x3 shared parameters to further extract feature information. Next, after the group convolution with shared parameters, the features are divided into three branches for detecting targets of different sizes, including small targets, medium targets and large targets; in each branch, the regression convolution operation Conv_Reg and the classification convolution operation Conv_Cls with shared parameters are used for target regression and classification tasks, respectively; at the same time, in each Conv_Reg operation, a trainable scalar parameter scale is introduced to adjust the scale of target regression to adapt to abnormal areas of industrial products of different sizes; the parameter scale is initialized to a positive value of 1.0 and then gradually optimized by gradient descent.

[0022] Furthermore, step 1 also includes using LAMP pruning technology to further compress and speed up the lightweight object detection model;

[0023] In LAMP, the optimization goal of the current layer is to optimize the mask matrix under the constraints. , to minimize the impact of pruning operations on model performance:

[0024] (1)

[0025] in, For the current The binary mask matrix of the layer. Each element of this matrix is ​​0 or 1, which is used to indicate whether the weight at that position is retained or pruned. It will automatically adjust its learning; is the zero norm of the mask matrix, representing the number of non-zero weights allowed in the current layer, for The pruning sparsity threshold of the layer is used to constrain the upper limit of non-zero elements in the mask. represents the normalized representation of the input data, is the 2-norm of the input data, which is used to constrain the scale of the input data. To satisfy Find vectors under conditions The maximum value of Presentation Layer The weight matrix of is element-wise multiplication;

[0026] The optimization of the entire lightweight target detection model is expressed as:

[0027] (2)

[0028] in, is a constant that limits the number of non-zero weights in the model, Lightweight target detection model Using the weight parameter Time input The output, is the weight after sparse For input The output, Indicates the first layer to the The set of all weight matrices of the layer, is the total number of layers of the lightweight object detection model;

[0029] Under the greedy strategy, the connection with the lowest importance score is removed in each iteration, and the above optimization is relaxed by equation (3):

[0030] (3)

[0031] in, is the sparse weight tensor, is the F norm, and the weight tensor Flatten to a one-dimensional vector, sort the one-dimensional vector so that for any constant When, meet , Respectively represent the sorted and vectors; due to the product term is a constant value, so the weight tensor Middle The importance score of an index is defined as the weight tensor Middle The LAMP score for an index is defined as follows:

[0032] (4)

[0033] Finally, the connections with the smallest LAMP scores are pruned globally until the required global sparsity constraint is met.

[0034] Furthermore, step 1 also includes enhancing the lightweight target detection model by using knowledge distillation; dividing the lightweight target detection model into three scales of n, s, and m according to the number of network layers, the amount of parameters, and the computational complexity, wherein the n scale has the smallest number of network layers, the amount of parameters, and the computational complexity, the m scale has the largest scale, and the s scale is medium, the s scale is used as the teacher network, and the n scale is used as the student network;

[0035] To perform knowledge distillation on a specific channel in the feature fusion module, we first convert the activation map of the channel into a probability distribution and use the probability distance metric to measure the difference. The teacher network and the student network are represented as and , and The activation maps are expressed as and , the channel distillation loss is generally expressed as:

[0036] (5)

[0037] in, is the KL divergence, which is used to calculate the probability distance metric between two inputs. is the softmax function, which is used to convert the activation map into a probability distribution; and are the activation maps of the teacher network and the student network, respectively. and They are the activation maps of specific channels in the teacher network and the student network, respectively;

[0038] use To convert the activation value into a probability distribution, as shown in formula (6):

[0039] (6)

[0040] in , Represents the index channel; represents the spatial position of the index channel, and is the width and height of the channel feature map, is a hyperparameter.

[0041] The present invention also provides an image anomaly detection system based on a lightweight target detection model, comprising:

[0042] A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the image anomaly detection method based on the lightweight target detection model as described in the above technical solution.

[0043] The present invention proposes a backbone network named GH-HGV2Net for target detection tasks. The hierarchical structure can well handle the local and global structural characteristics of the target. Such a design can effectively extract features at different levels, so as to better capture the local subtle structure and global overall structure in the image. The network adopts the HG Stage composed of GH-HGBlock, and the convolution operation in each GH-HGBlock adopts the GhostConv technology. GhostConv convolution uses a small number of ordinary convolution kernels to extract feature information from the input feature map, and then generates the final feature map through a cheaper linear transformation operation. This special convolution operation can reduce the network calculation amount and parameter amount while maintaining the original feature map size and number of channels, thereby improving the network's calculation efficiency and performance. In addition, the backbone network also includes an HGStem module, which is composed of ordinary convolution and pooling layers, and is used to reduce the amount of calculation and extract preliminary feature representations. The function of the HGStem module is to convert the input data image from high resolution to low resolution suitable for network processing, and extract preliminary feature representations. The entire backbone network consists of an HGStem module and four HG Stages, which aims to improve the accuracy and robustness of target detection through hierarchical feature representation and multi-scale feature extraction. Finally, the output features of the backbone network will be sent to an SPPF (Spatial Pyramid Pooling Fusion) module to extract and fuse multi-scale feature information. Through hierarchical feature representation, accurate capture of local and global structures in the image and sensitive detection of the next target are achieved, so as to further enhance the detection capability and robustness of the network.

[0044] In the target detection task, the detection of subtle targets is often very challenging, especially when the feature size of the target is very small and easily confused with the background texture. This scenario requires that the lightweight network pays special attention to the extraction and expression capabilities of detail features when designing. To this end, the present invention proposes a feature fusion module C2fGHDY-PAFPN (Neck) based on dynamic convolution to improve the performance of lightweight networks in subtle target detection. Dynamic convolution is introduced as part of feature extraction, and its powerful feature expression ability can better distinguish the slight differences between target features and background textures, thereby improving detection accuracy. Although dynamic convolution slightly increases the amount of calculation compared to traditional convolution, its significant advantages in feature extraction accuracy can make up for this, greatly improving the detection performance. The designed new module C2fGHDY-PAFPN (Neck) is obtained by replacing all the ordinary convolution blocks in Bottelneck in the C2f module with GhostConv and dynamic convolution. The structure uses cross-stage feature fusion strategy and truncated gradient flow technology to enhance the variability of learning features between different network layers, thereby reducing the influence of redundant gradient information and enhancing the learning ability of the network. Due to the introduction of GhostConv and dynamic convolution, the C2fGHDY-PAFPN module reduces a large number of 3×3 ordinary convolutions in the original structure, greatly compresses the model size of the network, and reduces the number of parameters and calculations.

[0045] In order to solve the problems of high model calculation amount and diversified target size and shape in target detection tasks, the present invention designs a lightweight group shared weight detection head (LSCD). The detection head can effectively reduce the number of parameters and improve computational efficiency, and is particularly suitable for target detection tasks in resource-constrained environments. That is, the multi-scale feature map Feature Map obtained from the feature fusion part uses the same group of convolutions to predict the detection frame, and then each layer uses a learnable Scale value as a coefficient to scale the predicted frame. The group normalization layer GroupNorm has been proven to improve the performance of detection head positioning and classification, and by using shared convolution, the number of parameters can be greatly reduced, which makes the model lighter, especially on resource-constrained devices. While using shared convolution, the normalization layer in the convolution is replaced with GroupNorm. The reason why the parameters of the classification loss can be shared is that no matter whether it is a small target, a medium target or a large target, the information of the same target is learned in the detection layer without affecting it. However, it is known that the premise that the Reg loss function can be shared must consider the scale problem, so a Scale layer is added. In order to deal with the problem of inconsistent scale of the target detected by each detection head while dealing with the shared convolution layer, the Scale layer is used to scale the features. In order to further compress the size of the target detection model and improve the inference speed, the present invention introduces the LAMP (Layer-wise Automatic Magnitude Pruning) pruning technology. This technology is designed for lightweight target detection models. Through efficient pruning strategies, it can significantly reduce the demand for computing resources while retaining the model performance to the maximum extent. It is suitable for lightweight optimization scenarios of various target detection tasks. The core advantage of the LAMP pruning technology is that it does not rely on hyperparameters and does not need to rely on the knowledge of a specific model, making it highly versatile. The principle is to regard each layer of the neural network as an independent operator, and determine the priority of the sparsity of each layer by analyzing the impact of pruning on the "model-level" distortion. Specifically, LAMP calculates the LAMP score, readjusts the network weights to approximately measure the performance loss caused by pruning, and automatically determines the sparsity of each layer based on the score, thereby realizing a globally optimized pruning process. This method retains all the advantages of layer-wise pruning (MP) during the pruning process. By globally removing low-priority weights, the LAMP pruning technology can significantly reduce the number of model parameters and computational complexity while maintaining the detection accuracy and robustness of the model.

[0046] The network model will lose some accuracy after pruning. In order to improve the detection accuracy, channel-based knowledge distillation technology can be used. The s version of the improved model is used as the teacher model, and the n version is used as the student model. The three feature layers (16, 19, 22) generated by the feature fusion module are used for knowledge distillation. Channel-level distillation is used to improve the student model's learning ability and detection accuracy.

[0047] In summary, through in-depth research, the inventor team proposed a lightweight target detection model GH_HGV2-LSCD-YOLOX based on YOLOX. This model is highly flexible and adaptable and can be widely used in various industrial scenarios, including fabric defect detection, power equipment defect detection, drug appearance abnormality detection and other practical needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart provided by an embodiment of the present invention.

[0049] Figure 2 It is a model framework diagram of the improved lightweight target detection model provided by an embodiment of the present invention.

[0050] Figure 3 It is a model diagram of GhostConv provided by an embodiment of the present invention.

[0051] Figure 4 It is a model diagram of GH-HGBlock in the backbone network provided by an embodiment of the present invention.

[0052] Figure 5 It is a structural diagram based on channel knowledge distillation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] This paper optimizes and improves the YOLOv8 model and proposes a lightweight target detection model GH_HGV2-LSCD-YOLOX. Figure 1 As shown in the figure, the model achieves lightweight and efficient detection through the following key steps:

[0054] First, GhostConv and DWConv were combined and integrated with the lightweight backbone network HGNetV2 to build a new backbone network GH-HGV2Net, which significantly reduced the number of model parameters. In the feature fusion stage, DynaMicConv and C2f modules with strong feature expression capabilities were combined with GhostConv to design a two-stream PANet structure to achieve feature fusion. Finally, the lightweight shared group convolution detection head (LSCD) was used to further reduce the number of parameters while improving the accuracy and inference speed of the model.

[0055] GH_HGV2-LSCD-YOLOX consists of three core modules: backbone network, feature fusion module and lightweight YOLO detection head. Among them, the backbone network is the improved GH-HGV2Net, and the feature fusion module is C2fGHDY-PAFPN. The overall network structure is as follows: First, during the model training and testing process, the backbone network first extracts features from the input image, and after passing through the HGStem (Hierarchical Group Stem) module, 4 stages (HGStage) and SPPF modules, three output feature layers are generated: (80, 80, 512), (40, 40, 1024) and (20, 20, 1024). Secondly, these feature layers are then sent to the C2fGHDY-PAFPN feature fusion network. The feature fusion network uses a bidirectional fusion mechanism, two downsampling and two upsampling operations to generate three feature maps (corresponding to small targets, medium targets, and large targets), which are 16, 19, and 22 layers respectively, and are finally sent to the LSCD detection head for target prediction.

[0056] In order to further optimize the lightweight performance of the model, this paper introduces the LAMP (Layer-wise Automatic Magnitude Pruning) pruning technology. Compared with traditional pruning methods, LAMP pruning technology does not require hyperparameter adjustment, does not rely on specific model knowledge, and can globally evaluate the importance of weights, thereby achieving automated and efficient pruning operations, significantly compressing the model size and improving the inference speed.

[0057] In addition, in order to enhance the detection accuracy of the pruned model, the present invention also adopts channel knowledge distillation technology. Specifically, the s version of the YOLOv8 model is used as the teacher model, the n version is used as the student model, and the three feature layers (16, 19, 22) generated by the feature fusion module are used for knowledge distillation. The channel-level distillation is used to improve the learning ability and detection accuracy of the student model.

[0058] The GH_HGV2-LSCD-YOLOX model proposed in this paper shows excellent performance in an n-scale network structure. It can significantly reduce the consumption of computing resources while maintaining a high level of detection accuracy, providing an efficient and reliable solution for industrial production and other real-time target detection scenarios.

[0059] To more clearly explain the specific implementation method of the present invention, the following will explain in detail three key aspects: network model structure, pruning technology, and knowledge distillation technology. Through the organic combination of these three core modules, the present invention achieves lightweight, efficient, and high-precision target detection models, providing an efficient and reliable solution for the industrial field.

[0060] 1. Model Structure

[0061] GH_HGV2-LSCD-YOLOX consists of three core modules: backbone network, feature fusion module and lightweight YOLO detection head. The backbone network is the improved GH-HGV2Net, and the feature fusion module is C2fGHDY-PAFPN. The overall network structure is as follows Figure 2 As shown, during the model training and testing process, the backbone network first extracts features from the input image, and then goes through the initial extraction module HGStem module, as shown in Figure 3 As shown in Figure 1, the HGStem module consists of multiple ordinary convolutions, maximum pooling layers, and connection operations to reduce the amount of computation and extract preliminary feature representations. The function of the HGStem module is to convert the input industrial product image from high resolution to a low resolution suitable for network processing and extract preliminary feature representations.

[0062] Then it goes through 4 stages (HGStage), stage 1 consists of a GH-HGBlock, stage 2 consists of a depthwise convolution DWConv (Depthwise Convolution) and a GH-HGBlock, stage 3 consists of a DWConv and three consecutive GH-HGBlocks, and stage 4 is the same as stage 2. It aims to improve the accuracy and robustness of image anomaly detection through hierarchical feature representation and multi-scale feature extraction.

[0063] Finally, the output features of the backbone network are sent to the SPPF module to extract multi-scale feature information and perform fusion. The entire backbone network generates three output feature layers after stage2, stage3 and SPPF modules, namely: (80,80,512), (40,40,1024) and (20,20,1024). Secondly, these feature layers are then sent to the C2fGHDY-PAFPN feature fusion network. The feature fusion network uses a bidirectional fusion mechanism, two downsampling and two upsampling operations to generate three feature maps (corresponding to small targets, medium targets, and large targets), and finally sent to the LSCD detection head for target prediction.

[0064] Among them Figure 3 As shown in Figure 1, the working process of GhostConv is divided into two stages. In the first stage, a small number of convolutions are performed (for example, 64 convolution kernels are used instead of 128 kernels normally, thus reducing the amount of computation by half). In the second stage, CheapOperation, Figure 3The Φ in the figure indicates that Φ is a convolution such as 3*3 or 5*5, and convolution is performed on each feature map (Depth-wise Convolutional, depth convolution) (the convolution operation is a full combination of convolution-batch normalization BN-nonlinear activation, and the so-called linear transformation or cheap operation refers to ordinary convolution, excluding batch normalization and nonlinear activation.). The final feature map is the feature map output by the first stage and the feature map output by the second stage, and the feature map with the same number of feature maps as ordinary convolution is obtained (the advantage is that the amount of calculation is reduced).

[0065] like Figure 4 As shown in the figure, GH-HGBlock is the basic building block of the backbone network, and its internal process is as follows: First, the input features are subjected to n GhostConv convolution operations to generate a series of feature maps. Then, these feature maps are spliced ​​through the Concatenate operation to connect the feature maps after each GhostConv operation. Next, the spliced ​​feature map is subjected to the SqueezeConv and ExcitationConv operations to extract the channel information and feature response of the feature map. Finally, according to the condition: the number of channels of the original feature map is the same as the number of channels of the feature map after the intermediate operation, the original input and the feature map generated by the GhostConv and Concat operations are added to generate the final feature map. This design can make full use of the advantages of the GhostConv convolution operation, reduce the amount of network calculation and parameters, and at the same time, through the Concatenate and sum operations, the feature information of different stages is fused together, thereby improving the feature expression ability and detection performance of the network.

[0066] like Figure 2As shown in the figure, the phantom dynamic convolution layer C2fGhostDynaMic is a new model designed, which replaces the ordinary convolution in the original model BottleNeck with GhostConv and dynamic convolution DynamicConv. The C2fGhostDynaMic module is mainly composed of DynamicConv and GhostBottleNeck. Among them, GhostBottleNeck first passes through GhostConv, then passes through a batch normalization layer and an activation layer, and then passes through GhostConv, and finally adds it to the original feature map to output the final feature map. The input feature map of the C2fGhostDynaMic module first passes through the dynamic convolution DynamicConv, and then the output result is divided into two parts. The first part results in two identical GhostBottleNeck modules, and then the second part of the feature map after the segmentation is spliced ​​with the feature map after the first GhostBottleNeck module and the second GhostBottleNeck module, and finally passes through a DynamicConv module. This structure uses the cross-stage feature fusion strategy and truncated gradient flow technology to enhance the variability of learning features between different network layers, thereby reducing the impact of redundant gradient information and enhancing the learning ability of the network. By introducing GhostConv and dynamic convolution, the C2fGhostDynaMic module reduces a large number of 3×3 ordinary convolutions in the original structure, greatly compresses the model size of the network, reduces the number of parameters and the amount of calculation, so that the model can be deployed on the mobile terminal, making it easier to realize edge computing detection in abnormal areas.

[0067] according to Figure 2As shown in the figure, the features of size 20x20x1024 extracted by the backbone network are first processed by the SPPF (Spatial Pyramid Pooling Fusion) module. Then an upsampling operation is performed, and the upsampled feature map is concatenated with the 40x40x1024 feature map provided by the third stage. Next, it passes through a C2fGhostDynaMic module (marked as 13 layers), and then an upsampling operation is performed again, and the upsampled feature map is concatenated with the 80x80x512 feature map provided by the second stage. At this point, it passes through another C2fGhostDynaMic module (marked as 16 layers), and the generated feature map is used to provide to the detection head for detecting small targets. The 16-layer feature map then goes through a GhostConv upsampling operation, and then the upsampled feature map is spliced ​​again with the first spliced ​​feature map (13-layer feature map), and then passed through a C2fGhostDynaMic module (marked as 19 layers) to provide it to the detection head of the target under detection. In addition, the 19-layer feature map passes through another GhostConv operation and is then spliced ​​with the 20x20x1024 feature map that has passed through SPPF. Finally, the spliced ​​feature map passes through a C2fGhostDynaMic module (marked as 22 layers) and is provided to the detection head of the large target. In this way, the process of feature fusion network feature map circulation is completed.

[0068] according to Figure 2As shown in the figure, in order to reduce the amount of computation of the model and the problem of the diversity of the size and shape of the abnormal area, this paper designs a detection head LSCD with group shared weights, which operates on the features extracted from the 16th, 19th, and 22nd layers of the network. Specifically, these features are first processed by Conv_GN1x1 (group convolution) to reduce the amount of computation. Then, the features processed by Conv_GN1x1 are input into the group convolution of two Conv_GN3x3 shared parameters to further extract feature information. Then, after the group convolution with shared parameters, we divide the features into three branches for detecting objects of different sizes, including small objects, medium objects, and large objects. In each branch, we again use the convolution operations Conv_Reg and Conv_Cls with shared parameters for the regression and classification tasks of the object, respectively. It is worth noting that in each Conv_Reg operation, we introduced a trainable scalar parameter scale to adjust the scale of target regression to adapt to abnormal areas of different sizes. The scale parameter is initialized to a positive value of 1.0, and then gradually optimized by gradient descent to minimize the loss function and improve model performance (the loss function uses the existing loss function of the YOLOv8 model). The scale parameter that has been trained in the inference stage is directly applied to the regression value adjustment of the prediction box. The introduction of the scale parameter enables our model to more flexibly adapt to the size and shape of various industrial product anomalies. By continuously optimizing the scale parameter during the training process, we can achieve effective detection and positioning of targets of different sizes, improving the accuracy and robustness of detection.

[0069] 2. Pruning Technology

[0070] This paper uses the LAMP (Layer-wise Automatic Magnitude Pruning) pruning technique to further compress and speed up the improved lightweight network. Compared with other pruning methods, LAMP pruning does not require hyperparameters and does not rely on any model-specific knowledge. The principle is to regard each neural network layer as an operator, and determine the priority of the hierarchical sparsity of the weight-based pruning method MP (Magnitude Pruning) by examining the "model-level" distortion caused by the pruned layer, that is, to readjust the weight size through the LAMP score, approximate the model-level distortion caused by pruning, and automatically determine the hierarchical sparsity, thereby achieving further compression and speeding up of the lightweight network. Global pruning with LAMP score is equivalent to MP with automatically determined hierarchical sparsity, and the use of LAMP pruning can fully retain the advantages of MP.

[0071] In LAMP, the optimization goal of the current layer is to optimize the mask matrix under the constraints. , to minimize the impact of pruning operations on model performance:

[0072] (1)

[0073] in, For the current The binary mask matrix of the layer. Each element of this matrix is ​​0 or 1, indicating whether the weight at that position is retained (1) or pruned (0). Will automatically adjust learning. is the zero norm of the mask matrix, representing the number of non-zero weights allowed in the current layer. for The pruning sparsity threshold of the layer, which is used to constrain the upper limit of non-zero elements in the mask. represents the normalized representation of the input data, is the 2-norm of the input data, used to constrain the scale of the input data. To satisfy Find vectors under conditions The maximum value of . Presentation Layer The weight matrix of is element-wise multiplication.

[0074] The optimization of the entire network can be expressed as:

[0075] (2)

[0076] in, is a constant used to limit the number of non-zero weights in the model. For the network (GH_HGV2-LSCD-YOLOX model) Using the weight parameter Time input The output, is the weight after sparse For input The output, Indicates the first layer to the The collection of all weight matrices of the layers.

[0077] Under the greedy strategy, the connection with the lowest importance score is removed at each iteration. That is, the maximum singular value and Frobenius norm of matrix A That is, the square root of the sum of the squares of the matrix elements. According to the inequality relationship The above optimization can be relaxed by using the following upper bound on the model output distortion:

[0078] (3)

[0079] in, is the sparse weight tensor, is the F norm; the weight tensor Flatten into a one-dimensional vector, sort the one-dimensional vector so that for any When, meet , Respectively represent the sorted and vectors. Since the product term is a constant value, so the weight tensor Middle The importance score of an index is defined as the weight tensor Middle The LAMP score for an index is defined as follows:

[0080] (4)

[0081] In simple terms, the LAMP score (Equation 4) measures the relative importance of the target connection among all surviving connections belonging to the same layer, where connections with smaller weight magnitudes (in the same layer) have been pruned. Therefore, two connections with the same weight magnitude have different LAMP scores used. Once the LAMP score is calculated, the connection with the smallest LAMP score is pruned globally until the desired global sparsity constraint is met; this process is equivalent to performing MP with automatically selected layer-wise sparsity.

[0082] 3. Knowledge Distillation

[0083] In order to enhance the pruned model. Knowledge distillation is an effective way to improve the detection accuracy of the algorithm. Existing knowledge distillation methods usually use point-by-point alignment, or alignment of structured information between spatial positions, but channels contain a lot of knowledge that is ignored in the distillation process. Channel-based knowledge distillation CWD can better utilize the knowledge in each channel, and the activations of corresponding channels between the teacher and student networks should be adjusted in small steps. To this end, the activations of the channels (corresponding to the outputs of layers 16, 19, and 22 in this embodiment) are first converted into a probability distribution so that we can use probabilistic distance metrics, such as KL scattering, to measure the difference. The teacher and student networks are represented as and , and The activation maps are expressed as and , the channel distillation loss is generally expressed as:

[0084] (5)

[0085] in, is the KL divergence, which is used to calculate the probabilistic distance metric between two inputs. is the softmax function, which is used to convert the activation map into a probability distribution. and are the activation maps of the teacher network and the student network, respectively. and are the activation maps of specific channels in the teacher network and the student network, respectively.

[0086] use To convert the activation value into a probability distribution, as shown in formula (6):

[0087] (6)

[0088] in , Represents the index channel; Indicates the spatial position of the index channel. and is the width and height of the channel feature map. is a hyperparameter (temperature). If we use a larger , the probability will become softer, which means that we will focus on a wider spatial region of each channel. By applying softmax normalization, we eliminate the scale effect between large networks and compact networks (if the number of channels between the teacher and the student does not match, a 1×1 convolutional layer is used to upsample the number of channels in the student network). The lightweight network framework GH_HGV2-LSCD-YOLOx proposed in the present invention has three network scales of n, s, and m according to the number of network layers, the number of parameters, and the computational complexity. The n scale is the smallest, the m scale is the largest, and the s scale is medium. The embodiment of the present invention uses the s model as the teacher model and the n model as the student model, wherein the above three scale division methods are well known in the art. Specifically, the three feature layers (small, medium, and large) (16, 19, 22) extracted from the feature fusion module are subjected to knowledge distillation, such as Figure 5 shown.

[0089] 4. Model training and testing

[0090] This example uses a public industrial product anomaly data set of 2,800 images, including 4 types of anomalies (hole, knot, line, stain), and randomly selects them for classification according to the ratio of training set: test set = 4:1. The input size of the image is 640x640, and the experiment is carried out in the Ubuntu 20.04.4 operating system. The CPU model is Intel i9-9900k@3.60GHz, the GPU model is NVIDIA GeForce GTX3060Ti graphics card, and the video memory is 8G. The deep learning framework selected is python-3.8.0, cuda-11.7, and pytorch-2.0.1.

[0091] As shown in Table 1, the lightweight model is compared with the original model in terms of model size, inference speed, number of parameters, and mAP value. It can be seen that the lightweight model improves Map while its inference speed decreases, but its number of parameters and model size are greatly reduced, which reduces the computational cost.

[0092] Table 1 Comparison results between the lightweight model and the original model

[0093] Another invention, an embodiment of the present invention also provides an image anomaly detection system based on a lightweight target detection model, comprising:

[0094] A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the image anomaly detection method based on the lightweight target detection model as described in the above technical solution.

[0095] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. An image anomaly detection method based on a lightweight target detection model, characterized in that: The steps include: Step 1: Build a lightweight target detection model based on the improved YOLOv8 model, including three parts: backbone network, feature fusion module and lightweight shared group convolution detection head; Combining ghost convolution GhostConv and deep convolution DWConv, and integrating with the lightweight backbone network HGNetV2, a new backbone network GH-HGV2Net is constructed to extract features from the input image. The entire new backbone network generates two output feature layers in the middle stage. The processing process of the feature fusion module is as follows: the final output features of the new backbone network are passed through the SPPF module to generate the third output feature layer; the phantom dynamic convolution layer C2fGhostDynaMic is constructed by combining the dynamic convolution DynaMicConv, C2f module and ghost convolution GhostConv; the three output feature layers are multi-level fused through the ghost dynamic convolution layer, upsampling layer, ghost convolution and splicing operations, and finally three feature maps of different sizes are output; Combining group convolution and convolution operations with shared parameters to build a lightweight shared group convolution detection head LSCD for target prediction; Step 2: Select an image containing an anomaly to be detected, mark the position of the anomaly to be detected in the image, and record the coordinate information of the anomaly to form an anomaly detection data set; Step 3: Use the anomaly detection dataset to train the constructed target detection model, and output the trained target detection model to realize image anomaly detection.

2. The image anomaly detection method based on a lightweight target detection model according to claim 1, characterized in that: The new backbone network GH-HGV2Net includes a preliminary extraction module HGStem and 4 stages; The four stages are stage1 to stage4, where stage1 contains a GH-HGBlock, stage2 contains a deep convolution DWConv and a GH-HGBlock, stage3 contains a DWConv and three consecutive GH-HGBlocks, and stage4 is the same as stage2.

3. The image anomaly detection method based on a lightweight target detection model according to claim 2, characterized in that: The preliminary extraction module HGStem includes multiple ordinary convolution and pooling layers.

4. The image anomaly detection method based on a lightweight target detection model according to claim 2, characterized in that: The processing flow of GH-HGBlock is as follows: first, the input features are subjected to n ghost convolution GhostConv operations to generate a series of feature maps; then, these feature maps are spliced ​​through the Concatenate operation to connect the feature maps after each GhostConv operation; next, the spliced ​​feature maps are subjected to the SqueezeConv and ExcitationConv operations to extract the channel information and feature response of the feature maps; finally, according to the condition that the number of channels of the original feature map is the same as the number of channels of the feature map after the intermediate operation, the original input and the feature map generated by the GhostConv and Concatenate operations are added to generate the final feature map.

5. The image anomaly detection method based on a lightweight target detection model according to claim 1, characterized in that: The ghost dynamic convolution layer C2fGhostDynaMic includes dynamic convolution DynamicConv and ghost bottleneck layer GhostBottleNeck; the processing process of C2fGhostDynaMic is as follows: the input feature map first passes through dynamic convolution DynamicConv, and then the output result is divided into two parts. The first part of the result passes through two identical GhostBottleNecks, and then the second part of the feature map after the segmentation is spliced ​​with the feature map after the first GhostBottleNeck and the second GhostBottleNeck operation, and finally passes through a DynamicConv; The processing process of GhostBottleNeck is as follows: the input first passes through the ghost convolution GhostConv, then passes through a batch normalization layer and an activation layer, then passes through GhostConv, and finally adds it to the original feature map to output the final feature map.

6. The image anomaly detection method based on a lightweight target detection model according to claim 1, characterized in that: The specific processing process of the feature fusion module is as follows: The features extracted by the new backbone network are first processed by the SPPF module, followed by an upsample operation, and the upsampled feature map is spliced ​​with the feature map provided by the intermediate stage stage3; then, it passes through a C2fGhostDynaMic module marked as 13 layers, and then upsamples again, and splices the upsampled feature map with the feature map provided by the intermediate stage stage2; at this time, it passes through another C2fGhostDynaMic module marked as 16 layers, and the generated feature map is used to provide to the detection head for detecting small targets; the 16-layer feature map then passes through a GhostConv upsampling operation, and then the upsampled feature map is spliced ​​again with the first spliced ​​feature map, that is, the 13-layer feature map, and then passes through a C2fGhostDynaMic module marked as 19 layers, and provides it to the detection head of the target under detection; in addition, the 19-layer feature map passes through another GhostConv operation, and then spliced ​​with the feature map after SPPF; finally, the spliced ​​feature map passes through a C2fGhostDynaMic module marked as 22 layers and is provided to the detection head for detecting large targets.

7. The image anomaly detection method based on a lightweight target detection model according to claim 1, characterized in that: The processing process of the lightweight shared group convolution detection head LSCD is as follows: First, feature maps of different sizes are processed by group convolution Conv_GN1x1 to reduce the amount of calculation; then, the features processed by Conv_GN1x1 are input into the group convolution of two Conv_GN3x3 with shared parameters to further extract feature information. Then, after the group convolution with shared parameters, the features are divided into three branches for detecting targets of different sizes, including small targets, medium targets and large targets; in each branch, the regression convolution operation Conv_Reg and the classification convolution operation Conv_Cls with shared parameters are used for target regression and classification tasks respectively; at the same time, in each Conv_Reg operation, a trainable scalar parameter scale is introduced to adjust the scale of target regression to adapt to defects of different sizes; The parameter scale is initialized to a positive value of 1.0 and then gradually optimized by gradient descent.

8. The image anomaly detection method based on a lightweight target detection model according to claim 1, characterized in that: Step 1 also includes using LAMP pruning technology to further compress and speed up the lightweight object detection model; In LAMP, the optimization goal of the current layer is to optimize the mask matrix under the constraints. , to minimize the impact of pruning operations on model performance: (1); in, For the current The binary mask matrix of the layer. Each element of this matrix is ​​0 or 1, which is used to indicate whether the weight at that position is retained or pruned. Will automatically adjust learning; is the zero norm of the mask matrix, representing the number of non-zero weights allowed in the current layer, for The pruning sparsity threshold of the layer is used to constrain the upper limit of non-zero elements in the mask. represents the normalized representation of the input data, is the 2-norm of the input data, used to constrain the scale of the input data, To satisfy Find vectors under conditions The maximum value of Presentation Layer The weight matrix of is element-wise multiplication; The optimization of the entire lightweight target detection model is expressed as: (2); in, is a constant that limits the number of non-zero weights in the model, Lightweight target detection model Using the weight parameter Time input The output, is the weight after sparse For input The output, Indicates the first layer to the The set of all weight matrices of the layer, is the total number of layers of the lightweight object detection model; Under the greedy strategy, the connection with the lowest importance score is removed in each iteration, and the above optimization is relaxed by equation (3): (3); in, is the sparse weight tensor, is the F norm, and the weight tensor Flatten to a one-dimensional vector, sort the one-dimensional vector so that for any constant When, meet , Respectively represent the sorted and vectors; due to the product term is a constant value, so the weight tensor Middle The importance score of an index is defined as the weight tensor Middle The LAMP score for an index is defined as follows: (4); Finally, the connections with the smallest LAMP scores are pruned globally until the required global sparsity constraint is met.

9. The image anomaly detection method based on a lightweight target detection model according to claim 1, characterized in that: Step 1 also includes using knowledge distillation to enhance the lightweight target detection model; dividing the lightweight target detection model into three scales of n, s, and m according to the number of network layers, parameter amount, and computational complexity. The n scale has the smallest number of network layers, parameter amount, and computational complexity, the m scale has the largest scale, and the s scale is medium. The s scale is used as the teacher network and the n scale is used as the student network. To perform knowledge distillation on a specific channel in the feature fusion module, we first convert the activation map of the channel into a probability distribution and use the probability distance metric to measure the difference. The teacher network and the student network are represented as and , and The activation maps are expressed as and , the channel distillation loss is expressed as: (5); in, is the KL divergence, which is used to calculate the probability distance metric between two inputs. is the softmax function, which is used to convert the activation map into a probability distribution; and are the activation maps of the teacher network and the student network, respectively. and They are the activation maps of specific channels in the teacher network and the student network, respectively; use To convert the activation value into a probability distribution, as shown in formula (6): (6); in , Represents the index channel; represents the spatial position of the index channel, and is the width and height of the channel feature map, is a hyperparameter.

10. An image anomaly detection system based on a lightweight target detection model, characterized in that: include: A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the image anomaly detection method based on a lightweight target detection model as described in any one of claims 1 to 9.

Images