Method for lightening metal surface defect detection network model
Through knowledge distillation and semi-structured sparse pruning technology, a lightweight student model is built, which solves the problem of large scale and calculation of metal surface defect detection models, achieves efficient and accurate detection results and resource savings, and supports the metal manufacturing industry development.
Patent Information
- Application Number
- CN202410929900.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-07-11
AI Technical Summary
The existing metal surface defect detection model is huge in scale and has a large amount of calculation, resulting in high hardware resource consumption, limiting its application in actual production environments.
Knowledge distillation and semi-structured sparse pruning methods are used to construct a lightweight student model, and the metal surface defect characteristics in the teacher model are transferred to the student model, and the student model weight is optimized in combination with key distillation and global distillation to reduce computing resource consumption.
It realizes efficient and accurate identification of metal surface defect detection, while reducing model size and calculation needs, maintaining sufficient detection accuracy, and supporting the high-quality development of metal manufacturing industry.
Smart Images

Figure CN119048420A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of machine vision detection, and in particular to a lightweight method for a metal surface defect detection network model. Background Art
[0002] With the rapid development of the era of intelligent manufacturing, metal materials play an indispensable role in many key fields such as construction, aerospace, and automobile manufacturing due to their unique physical and chemical properties. However, tiny defects on the metal surface, such as cracks, scratches, holes, etc., even if they are extremely small, may have a serious impact on the performance and service life of the final product, and even cause safety accidents. Therefore, efficient and accurate metal surface defect detection technology has become an important part of ensuring product quality and improving production efficiency.
[0003] At present, the network model based on deep learning has shown excellent performance in the field of metal surface defect detection. However, the models currently used in metal surface defect detection applications are often large in scale, computationally intensive, and require high hardware resources. Their resource consumption and computational efficiency limit their application in actual production environments. Summary of the invention
[0004] The present invention provides a lightweight method for a metal surface defect detection network model to solve the technical problem that the models currently used in metal surface defect detection applications are often large in scale, large in computational complexity, and have high requirements for hardware resources, and their resource consumption and computational efficiency limit their application in actual production environments.
[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:
[0006] In one aspect, the present invention provides a lightweight method for a metal surface defect detection network model, comprising:
[0007] Constructing a teacher model for the knowledge distillation process;
[0008] Compress the teacher model to obtain the student model of the knowledge distillation process;
[0009] Taking the metal surface defect image as the input image, based on the knowledge distillation method, the knowledge learned by the teacher model is used to guide the training of the student model, so that the metal surface defect features learned in the teacher model are transferred to the student model, so as to realize metal surface defect detection using the trained student model.
[0010] Furthermore, the teacher model for constructing the knowledge distillation process includes:
[0011] An efficient multi-scale attention module EMA is introduced into the Head network of the YOLOv7 network model; and the loss function of the YOLOv7 network model is replaced with the WIoU v3 loss function to obtain the teacher model.
[0012] Furthermore, the calculation formula of WIoU v3 loss function is as follows:
[0013]
[0014] Among them, L WIoU represents the WIoU v3 loss function; β represents the outlier degree, which is used to define the quality of the anchor box; α and δ are both hyperparameters; x and y are the coordinates of the predicted box; x gt ,y gt is the target box coordinate; IoU stands for intersection over union, which is used to measure the overlap between the prediction box and the target box; W g and H g Represented as the width and height of the minimum enclosing box of the prediction box and the target box respectively; Indicates non-empty IoU; represents a sliding average with momentum m, t represents the batch, n represents the number of training times; (.) * W represents the prediction box and the target box that do not intersect g and H g Separate from the computation graph to prevent L WIoU Produces gradients that hinder convergence.
[0015] Furthermore, compressing the teacher model to obtain the student model of the knowledge distillation process includes:
[0016] A semi-structured sparse pruning method is used to compress the teacher model to obtain a lightweight student model.
[0017] Furthermore, the semi-structured sparse pruning method is used to compress the teacher model to obtain a lightweight student model, including:
[0018] Remove the structural blocks except 3×3 convolution and 1×1 convolution from the teacher model, and keep the structural blocks that need to be pruned. If the structural block contains only one set of convolutions, then this set of convolutions is defined as the parent layer. If the structural block contains multiple sets of convolutions, the frontmost convolution is defined as the parent layer, and the remaining convolutions are defined as sublayers of the parent layer.
[0019] Using the 3×3 kernel weight of the parent layer as the initialization variable, traverse all kernel patterns and calculate the L2 norm to obtain the optimal kernel pattern, iterate all kernels of the parent layer in turn, and obtain the optimal kernel pattern group to prune the parent layer and the corresponding sublayers;
[0020] Using the 1×1 kernel weight of the parent layer as the initialization variable, every 9 weights are combined into a temporary 3×3 weight matrix, and those less than 9 are treated as zero weight pruning;
[0021] Use the weights of the temporary 3×3 weight matrix as the initialization variables, traverse all kernel modes and calculate the L2 norm to get the optimal kernel mode, iterate all kernels of the parent layer in turn, get the optimal kernel mode group, prune the parent layer and the corresponding sublayers, then transform the 3×3 kernel back to a 1×1 kernel and attach it back to the original 1×1 kernel weight.
[0022] Furthermore, the kernel pattern is to generate all possible pattern masks through standard combinations, and the calculation formula is:
[0023]
[0024] Wherein, N represents the size of the matrix; K represents the size of the mask; K∈[1,8], for all possible pattern masks, the patterns without adjacent non-zero weights need to be discarded to maintain the semi-structured nature of the kernel pattern. Furthermore, based on the knowledge distillation method, the knowledge learned by the teacher model is used to guide the training of the student model, so that the metal surface defect features learned in the teacher model are transferred to the student model, including:
[0025] The student model weights are optimized based on focused distillation and global distillation respectively; wherein, the focused distillation separates the foreground and background of the input image by using an outlier mask, pays more attention to the key pixels and channels of the teacher model, and uses the spatial and channel attention of the teacher model as weights to jointly guide the student model to learn; the global distillation uses the GcBlock module to extract the global information of the student model output and the teacher model output respectively, extracts the relationship between different pixels, and enhances the relationship between the foreground and background.
[0026] Furthermore, based on the knowledge distillation method, the loss function L when using the knowledge learned by the teacher model to guide the training of the student model is expressed as:
[0027] L=L focal +L global
[0028] Among them, L focal represents the focus distillation loss function; L global represents the global distillation loss function;
[0029] L focal The calculation process is as follows:
[0030] The outlier mask is used to distinguish the foreground and background of the input image to form a binary image;
[0031] The spatial attention module and channel attention module are used to distinguish key pixels and channels. First, the absolute average values of different pixels and different channels are calculated respectively. The formula is:
[0032]
[0033] Among them, H, W, and C represent the height, width, and number of channels of the feature respectively; F c and F i,j Represents the features of the cth channel and the pixel with coordinate values i and j respectively; G S (F) and G C (F) represents the spatial attention map and channel attention map respectively; then the attention mask is calculated, the formula is:
[0034] A S (F) = HW softmax(G S (F) / T)
[0035] A C (F) = C softmax(G C (F) / T)
[0036] Where T represents the temperature hyperparameter used to adjust the distribution;
[0037] Calculate the focus distillation loss function L focal , the formula is:
[0038]
[0039] L focal =L f +L att
[0040] Among them, ω and σ are hyperparameters used to balance the loss between foreground and background; B i,j represents the pixel data with coordinates i, j in the binary image; f(.) represents an adaptive function that reshapes the features obtained by the student model into the same dimension as the features obtained by the teacher model; Represents the spatial attention mask at the position with coordinates i, j; represents the channel attention mask at the kth channel; Represents the feature value at the position with coordinates i,j under the kth channel in the features obtained by the teacher model; Represents the feature value at the position with coordinates i, j under the kth channel in the features obtained by the student model; represents the spatial attention mask used by the teacher model; represents the spatial attention mask used by the student model; represents the channel attention mask used by the teacher model; represents the channel attention mask used by the student model; γ represents the hyperparameter that balances the spatial attention mask and the channel attention mask;
[0041] L global The calculation process is as follows:
[0042]
[0043] Among them, ε represents the hyperparameter used to balance the loss; W k , W v1 , W v2 represents convolution; LN represents normalization; N p1 Indicates F T The number of features in; N p2 Indicates F S The number of features in F T represents the image features obtained by the teacher model; F S represents the image features obtained by the student model; F j represents the jth eigenvalue; F m represents the mth eigenvalue.
[0044] Furthermore, the process of obtaining the outlier mask is as follows:
[0045] Starting from the first pixel of the input image, traverse the pixels in the input image according to the set step size, and perform the following operations for the current pixel traversed:
[0046] Select an image block of a preset size with the current pixel as the center point, and then calculate the average value p(c i ,c j ,w,h), the calculation formula is as follows:
[0047]
[0048] Among them, c i ,c j represents the center pixel position coordinates of the image block; w represents the width of the image block; h represents the height of the image block; f src (i, j) represents the grayscale value of the pixel with coordinates i, j; [*] represents rounding operation;
[0049] Take the average value p(c i ,c j ,w,h) is used as the local correction factor of the central pixel of the image block;
[0050] The local correction factor, the reference grayscale of the correction image and the original grayscale value of the central pixel of the image block are multiplied to obtain the processed grayscale pixel value of the central pixel of the image block;
[0051] After traversing all the pixels, the rectified image is obtained, and the rectified image is divided into multiple sub-images. Each sub-image overlaps with the adjacent sub-image, and then the grayscale mean and variance of each sub-image are calculated; the threshold φ is set, and the binary image is obtained by threshold segmentation, where the calculation formula for the pixels in the binary image is:
[0052]
[0053] Where f(i,j) is the grayscale of the pixel with coordinates i,j in the rectified image. is the grayscale variance of the i-th sub-image; δ g is the global variance of the image; ψ is the global compensation amount, and μ is the local threshold segmentation weight.
[0054] On the other hand, the present invention further provides an electronic device, comprising a processor and a memory; wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the above method.
[0055] In yet another aspect, the present invention further provides a computer-readable storage medium, wherein at least one instruction is stored in the storage medium, and the instruction is loaded and executed by a processor to implement the above method.
[0056] The beneficial effects brought about by the technical solution provided by the present invention include at least:
[0057] The present invention first improves the YOLOv7 model and uses the improved model as the teacher model to improve the detection accuracy of the teacher model. Then, the semi-structured sparse pruning method is used to compress the teacher model to obtain a lightweight student model. Finally, the improved knowledge distillation is used to transfer the defect features learned in the teacher model to the student model, so that the student model can efficiently and accurately identify metal surface defects while reducing computing resource consumption and improving detection speed. This can not only significantly reduce the model size and computing requirements, but also maintain sufficient detection accuracy, thereby providing strong technical support for the high-quality development of the metal manufacturing industry. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0059] Figure 1 It is a flow chart of a lightweight method for metal surface defect detection network model provided by an embodiment of the present invention;
[0060] Figure 2 It is a schematic diagram of a lightweight method for metal surface defect detection network model provided by an embodiment of the present invention;
[0061] Figure 3 It is an improved YOLOv7 network model diagram provided by an embodiment of the present invention;
[0062] Figure 4 is an example diagram of the application of the WIoU v3 loss function provided in an embodiment of the present invention;
[0063] Figure 5 is a flow chart of a semi-structured sparse pruning method provided by an embodiment of the present invention;
[0064] Figure 6 It is an example diagram of the kernel mode of the three modes provided in the embodiment of the present invention;
[0065] Figure 7 is a data processing flow chart of an outlier mask module provided by an embodiment of the present invention;
[0066] Figure 8 is a structural diagram of a GcBlock module in global distillation provided by an embodiment of the present invention;
[0067] Fig. 9 It is a system block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0068] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0069] First of all, it should be noted that in the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplarily" is intended to present the concept in a concrete way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.
[0070] First embodiment
[0071] This embodiment provides a lightweight method for metal surface defect detection network model, which uses sparse pruning and knowledge distillation to obtain a lightweight network to detect metal surface defects; the method can be implemented by an electronic device, which can be a terminal or a server. The execution process of the method is as follows Figure 1 As shown, the following steps are included:
[0072] S1, the teacher model for constructing the knowledge distillation process;
[0073] It should be noted that, for the teacher model, in theory, any existing target detection network model can be used, and this embodiment does not specifically limit this. In a feasible implementation, Figure 2 As shown in Figure 1, the improved YOLOv7 network model can be used as the teacher model for the knowledge distillation process. The improvements to the YOLOv7 network model include: 1) adding an efficient multi-scale attention module (EMA) at the P3-P5 feature layer position where the original YOLOv7 model backbone network and the Neck network are connected to capture cross-dimensional interactions and establish inter-dimensional dependencies, such as Figure 3 As shown in Figure 2; 2) Using WIoU v3 loss function as the loss function, reduce the impact of large gradients or harmful gradients in extreme samples, and improve the network's ability to locate the target. The application of WIoU v3 is shown in Figure 2. Figure 4 As shown, the calculation formula is as follows:
[0074]
[0075] Among them, L WIoU represents the WIoU v3 loss function; R WIoU Used to amplify the loss value of the anchor box with lower quality; β represents the outlier degree, which is used to define the quality of the anchor box; α and δ are both hyperparameters; x and y are the coordinates of the predicted box; x gt ,y gt is the target box coordinate; IoU stands for intersection over union, which is used to measure the overlap between the prediction box and the target box; W g and H g Represented as the width and height of the minimum enclosing box of the prediction box and the target box respectively; Indicates non-empty IoU; represents a sliding average with momentum m, t represents the batch, n represents the number of training times. In this embodiment, t=890, n=34; (.) * W represents the prediction box and the target box that do not intersect g and H g Separate from the computation graph to prevent L WIoU Produces gradients that hinder convergence.
[0076] S2, compresses the teacher model to obtain the student model of the knowledge distillation process;
[0077] It should be noted that the teacher model is compressed to obtain the student model of the knowledge distillation process by using a semi-structured sparse pruning method to compress the teacher model to obtain a lightweight student model; wherein the semi-structured sparse pruning method can use any existing semi-structured sparse pruning method, which is not specifically limited in this embodiment; in a feasible implementation manner, such as Figure 5 As shown, the steps of pruning the teacher model using the semi-structured sparse pruning method may include the following process:
[0078] S21, remove the structural blocks except 3×3 convolution and 1×1 convolution from the teacher model, and keep the structural blocks that need to be pruned. If the structural block contains only one set of convolutions, then this set of convolutions is defined as the parent layer. If the structural block contains multiple sets of convolutions, the frontmost convolution is defined as the parent layer, and the remaining convolutions are defined as sublayers of the parent layer.
[0079] S22, using the 3×3 kernel weight of the parent layer as the initialization variable, traverse all kernel patterns and calculate the L2 norm to obtain the optimal kernel pattern, iterate all kernels of the parent layer in turn, obtain the optimal kernel pattern group to prune the parent layer and the corresponding sublayer;
[0080] S23, using the 1×1 kernel weight of the parent layer as the initialization variable, every 9 weights are combined into a temporary 3×3 weight matrix, and those less than 9 are treated as zero weight pruning;
[0081] S24, the temporary weight matrix is used to generate the optimal kernel pattern group in the manner of S22 to prune the parent layer and the corresponding child layer, and then the 3×3 kernel is transformed back to a 1×1 kernel and attached back to the original 1×1 kernel weight; wherein the kernel pattern is to generate all possible pattern masks through standard combinations, and the calculation formula is:
[0082]
[0083] Where N is the matrix size; K is the mask size; K∈[1,8], for all possible pattern masks, the patterns without adjacent non-zero weights need to be discarded to maintain the semi-structured nature of the kernel pattern.
[0084] like Figure 6 As shown, a kernel pattern with a mask size of 3 is obtained during the pruning process.
[0085] S3, using the metal surface defect image as the input image, based on the knowledge distillation method, uses the knowledge learned by the teacher model to guide the student model training, so that the metal surface defect features learned in the teacher model are transferred to the student model, so as to realize metal surface defect detection using the trained student model;
[0086] Among them, it should be noted that in the method based on knowledge distillation, when using the knowledge learned by the teacher model to guide the training of the student model, any existing knowledge distillation algorithm can be used for implementation, and this embodiment does not make specific limitations on this. In a feasible implementation, improved knowledge distillation can be used to use the knowledge learned by the teacher model to guide the training of the student model, specifically: optimize the student model weights based on focus distillation and global distillation respectively, and obtain the optimal weights of the student model for metal surface defect detection. Among them, the focus distillation includes an outlier mask module, a spatial and channel attention module, and a final loss function calculation. By separating the foreground and background of the image, more attention is paid to the key pixels and channels of the teacher model, and the teacher's spatial and channel attention is used as weights to jointly guide the student model to learn. The specific steps are:
[0087] S31, using an outlier mask to distinguish the foreground and background of the input image to form a binary image;
[0088] S32 uses the spatial and channel attention modules to distinguish key pixels and channels. First, the absolute averages of different pixels and different channels are calculated respectively. The formula is:
[0089]
[0090] Among them, H, W, and C represent the height, width, and number of channels of the feature respectively; F c and F i,j Represent the features of the cth channel and the pixel with coordinate value (i, j); G S (F) and G C (F) represents the spatial attention map and channel attention map respectively; then the formula for calculating the attention mask is:
[0091] A S (F) = HW softmax(G S (F) / T)
[0092] A C (F) = C softmax(G C (F) / T)
[0093] Where T represents the temperature hyperparameter used to adjust the distribution;
[0094] S33, calculate the final focus distillation loss function L focal , first calculate the feature loss function L f , and then calculate the attention loss L att The student imitates the teacher's spatial and channel attention mask, and finally adds the feature loss function and the attention loss function to obtain the final focus distillation loss function L focal , the formula is:
[0095]
[0096] L focal =L f +L att
[0097] Among them, ω and σ are hyperparameters used to balance the loss between foreground and background. In this embodiment, ω is set to 1×10 -3 , σ is set to 5×10 -4 ; B i,j represents the pixel data with coordinates (i, j) in the binary image; t and s represent the teacher and the student respectively; F T and F S represent the features from teachers and students respectively, and f is the sum of F S Reshaped to match F T Adaptive layers of the same dimension; Represents the spatial attention mask at the position with coordinates i, j; represents the channel attention mask at the kth channel; Represents the feature value at the position with coordinates i,j under the kth channel in the features obtained by the teacher model; Represents the feature value at the position with coordinates i, j under the kth channel in the features obtained by the student model; represents the spatial attention mask used by the teacher model; represents the spatial attention mask used by the student model; represents the channel attention mask used by the teacher model; represents the channel attention mask used by the student model; γ represents the hyperparameter for balancing the spatial attention mask and the channel attention mask; in this example, γ is set to 1×10 -3 .
[0098] Among them, the steps for obtaining the outlier mask in S31 are as follows: Figure 7 As shown, including:
[0099] S311, starting from the first pixel of the input image, select a small image of size w*h, and then calculate the average value of all pixels on the small image, the average value p(c i ,c j ,w,h) is as follows:
[0100]
[0101] Among them, (c i ,c j ) represents the center pixel position coordinates of a small image of size w*h; w represents the width of the small image; h represents the height of the small image; fsrc (i,j) represents the grayscale value of position (i,j); [.] represents the rounding operation;
[0102] S312, take the average value p(c i ,c j ,w,h) as the inverse of the pixel (c i ,c j ) local correction factor, local correction factor r(c i ,c j ) is as follows:
[0103] r(c i ,c j )=1 / p(c i ,c j ,w,h)
[0104] S313, by combining the local correction factor, the calibration image reference grayscale and the position (c i ,c j ) is multiplied by the original grayscale value to obtain the processed grayscale pixel value, and the calculation formula is:
[0105] f dst (c i ,c j ) = f src (c i ,c j )·gray base ·r(c i ,c j )
[0106] Among them, f src (c i ,c j ) indicates the position (c i ,c j )’s original grayscale value, gray base is the reference grayscale of the calibration image;
[0107] S314, set the step size to λ c , perform the above three steps on the image until all pixels are traversed;
[0108] S315, the rectified image is divided into n small images, which can be expressed as {m1,m2,...,m n}, each small image overlaps with the adjacent small image, and then the grayscale mean and variance of each small image are calculated; a threshold φ is set, where a global compensation is added when setting the threshold φ in order to maintain the local and global correlation of the image, and a binary image is obtained by threshold segmentation. The calculation formula for pixels in the binary image is:
[0109]
[0110] Where f(i,j) is the grayscale of the enhanced image pixel (i,j); is the grayscale variance of the i-th small image; δ g is the global variance of the image; ψ is the global compensation amount, and μ is the local threshold segmentation weight. In this embodiment, ψ∈[0.5,1], and μ is set to 1.5.
[0111] Furthermore, global distillation is performed using Figure 8 The GcBlock modules shown extract the global information of the student and the teacher respectively, extract the relationship between different pixels, and enhance the relationship between the foreground and the background. The calculation formula of the global distillation loss function is:
[0112]
[0113] Where ε represents the hyperparameter used to balance the loss. In this embodiment, ε = 5 × 10 -6 ; W k , W v1 , W v2 represents the convolutional layer; LN represents the normalization layer; N p Indicates the number of features; F T and F S Represent the characteristics from teachers and students respectively; F j represents the jth eigenvalue; F m represents the mth eigenvalue.
[0114] Based on the above, the final loss function L of focused distillation and global distillation is expressed as:
[0115] L=L focal +L global
[0116] Among them, L focal represents the focus distillation loss function; L global represents the global distillation loss function.
[0117] In summary, this embodiment provides a lightweight method for metal surface defect detection network model. The method first improves the detection accuracy of the teacher model by improving the YOLOv7 model and using the improved model as the teacher model. Then, the semi-structured sparse pruning method is used to compress the teacher model to obtain a lightweight student model. Finally, the improved knowledge distillation is used to transfer the defect features learned in the teacher model to the student model, so that the student model can efficiently and accurately identify metal surface defects while reducing computing resource consumption and improving detection speed. This can not only significantly reduce the model size and computing requirements, but also maintain sufficient detection accuracy, thereby providing strong technical support for the high-quality development of the metal manufacturing industry.
[0118] Second embodiment
[0119] This embodiment provides an electronic device, such as Fig. 9 As shown, the electronic device includes: a processor and a memory; wherein the processor and the memory can be connected via a communication bus; the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method of the first embodiment. In addition, the electronic device may also include a transceiver, the processor and the transceiver can be connected via a communication bus, and the transceiver is used to communicate with other devices.
[0120] Next, combine Fig. 9 The following is a detailed introduction to the various components of the electronic device:
[0121] Among them, the processor is the control center of the electronic device, and the electronic device may include multiple processors, each of which may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may be a processor or a general term for multiple processing elements. For example, the processor is one or more central processing units (CPUs), or other general-purpose processors, application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement an embodiment of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor may execute various functions of the electronic device by running or executing software programs stored in the memory and calling data stored in the memory.
[0122] In a specific implementation, as an embodiment, the processor may include one or more CPUs, such as Fig. 9 The CPU0 and CPU1 shown in the figure are, of course, only exemplary.
[0123] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment and will not be repeated here.
[0124] Optionally, the memory may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be integrated with the processor or exist independently and accessed through the interface circuit ( Fig. 9 (not shown) is coupled to the processor, which is not specifically limited in this embodiment of the present invention.
[0125] The transceiver may include a receiver and a transmitter ( Fig. 9 The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function. The transceiver can be integrated with the processor or exist independently and communicate with the electronic device through the interface circuit ( Fig. 9 (not shown) is coupled to the processor, which is not specifically limited in this embodiment of the present invention.
[0126] In addition, it should be noted that Fig. 9 The structure of the electronic device shown in the figure does not constitute a limitation on the device, and the actual device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In addition, the technical effects achieved by the electronic device when executing the method of the first embodiment above can refer to the technical effects described in the first embodiment above, so they are not repeated here.
[0127] Third embodiment
[0128] This embodiment provides a computer-readable storage medium, which stores at least one instruction, and the instruction is loaded and executed by a processor to implement the method of the first embodiment. The computer-readable storage medium may be a ROM, a random access memory, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc. The instructions stored therein may be loaded by a processor in a terminal to execute the method.
[0129] In addition, it should be noted that the present invention can be provided as a method, an apparatus or a computer program product. Therefore, the embodiment of the present invention can be in the form of a full or partial hardware embodiment, a full or partial software embodiment or an embodiment combining software and hardware. Moreover, when implemented using software, the embodiment of the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program codes. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center containing one or more available media sets. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium. The semiconductor medium may be a solid state hard disk.
[0130] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0131] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0132] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements. In addition, the term "and / or" is only an association relationship describing the associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone, wherein A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding. "At least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can be represented by: a, b, c, ab, ac, bc or abc, where a, b, c can be single or plural.
[0133] In addition, it can be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0134] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0135] In several embodiments provided by the present invention, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of functional modules / units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.
[0136] If the method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0137] Finally, it should be noted that the above is only a preferred embodiment of the present invention. It should be pointed out that although the preferred embodiment of the present invention has been described, for ordinary technicians in this technical field, once the basic creative concept of the present invention is known, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. Therefore, the attached claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the embodiments of the present invention.
Claims
1. A lightweight method for metal surface defect detection network model, characterized in that: include: Constructing a teacher model for the knowledge distillation process; Compress the teacher model to obtain the student model of the knowledge distillation process; Taking the metal surface defect image as the input image, based on the knowledge distillation method, the knowledge learned by the teacher model is used to guide the training of the student model, so that the metal surface defect features learned in the teacher model are transferred to the student model, so as to realize metal surface defect detection using the trained student model.
2. The lightweight method for metal surface defect detection network model according to claim 1, characterized in that: The teacher model for constructing the knowledge distillation process includes: An efficient multi-scale attention module EMA is introduced into the Head network of the YOLOv7 network model; and the loss function of the YOLOv7 network model is replaced with the WIoU v3 loss function to obtain the teacher model.
3. The lightweight method for metal surface defect detection network model according to claim 2, characterized in that: The calculation formula of WIoU v3 loss function is as follows: Among them, L WIoU represents the WIoU v3 loss function; β represents the outlier degree, which is used to define the quality of the anchor box; α and δ are both hyperparameters; x and y are the coordinates of the predicted box; x gt ,y gt is the target box coordinate; IoU stands for intersection over union, which is used to measure the overlap between the prediction box and the target box; W g and H g Represented as the width and height of the minimum enclosing box of the prediction box and the target box respectively; Indicates non-empty IoU; represents a sliding average with momentum m, t represents the batch, n represents the number of training times; (.) * W represents the prediction box and the target box that do not intersect g and H g Separate from the computation graph to prevent L WIoU Produces gradients that hinder convergence.
4. The lightweight method for metal surface defect detection network model according to claim 1, characterized in that: The teacher model is compressed to obtain the student model of the knowledge distillation process, including: A semi-structured sparse pruning method is used to compress the teacher model to obtain a lightweight student model.
5. The lightweight method for metal surface defect detection network model according to claim 4, characterized in that: The semi-structured sparse pruning method is used to compress the teacher model to obtain a lightweight student model, including: Remove the structural blocks except 3×3 convolution and 1×1 convolution from the teacher model, and keep the structural blocks that need to be pruned. If the structural block contains only one set of convolutions, then this set of convolutions is defined as the parent layer. If the structural block contains multiple sets of convolutions, the frontmost convolution is defined as the parent layer, and the remaining convolutions are defined as sublayers of the parent layer. Using the 3×3 kernel weight of the parent layer as the initialization variable, traverse all kernel patterns and calculate the L2 norm to obtain the optimal kernel pattern, iterate all kernels of the parent layer in turn, and obtain the optimal kernel pattern group to prune the parent layer and the corresponding sublayers; Using the 1×1 kernel weight of the parent layer as the initialization variable, every 9 weights are combined into a temporary 3×3 weight matrix, and those less than 9 are treated as zero weight pruning; Use the weights of the temporary 3×3 weight matrix as the initialization variables, traverse all kernel modes and calculate the L2 norm to get the optimal kernel mode, iterate all kernels of the parent layer in turn, get the optimal kernel mode group, prune the parent layer and the corresponding sublayers, then transform the 3×3 kernel back to a 1×1 kernel and attach it back to the original 1×1 kernel weight.
6. The lightweight method for metal surface defect detection network model according to claim 5, characterized in that: The kernel mode generates all possible mode masks through standard combinations, and the calculation formula is: Where N represents the size of the matrix; K represents the size of the mask; K∈[1,8], for all possible pattern masks, the patterns without adjacent non-zero weights need to be discarded to maintain the semi-structured nature of the kernel pattern.
7. The lightweight method for metal surface defect detection network model according to claim 1, characterized in that: Based on the knowledge distillation method, the knowledge learned by the teacher model is used to guide the training of the student model, so that the metal surface defect features learned in the teacher model are transferred to the student model, including: The student model weights are optimized based on focused distillation and global distillation respectively; wherein, the focused distillation separates the foreground and background of the input image by using an outlier mask, pays more attention to the key pixels and channels of the teacher model, and uses the spatial and channel attention of the teacher model as weights to jointly guide the student model to learn; the global distillation uses the GcBlock module to extract the global information of the student model output and the teacher model output respectively, extracts the relationship between different pixels, and enhances the relationship between the foreground and background.
8. The lightweight method for metal surface defect detection network model according to claim 7, characterized in that: Based on the knowledge distillation method, the loss function L when using the knowledge learned by the teacher model to guide the training of the student model is expressed as: L=L focal +L global Among them, L focal represents the focus distillation loss function; L global represents the global distillation loss function; L focal The calculation process is as follows: The outlier mask is used to distinguish the foreground and background of the input image to form a binary image; The spatial attention module and channel attention module are used to distinguish key pixels and channels. First, the absolute average values of different pixels and different channels are calculated respectively. The formula is: Among them, H, W, and C represent the height, width, and number of channels of the feature respectively; F c and F i,j Represents the features of the cth channel and the pixel with coordinate values i and j respectively; G S (F) and G C (F) represents the spatial attention map and channel attention map respectively; then the attention mask is calculated, the formula is: A S (F)=HW·softmax(G S (F) / T) A C (F)=C·softmax(G C (F) / T) Where T represents the temperature hyperparameter used to adjust the distribution; Calculate the focus distillation loss function L focal , the formula is: L focal =L f +L att Among them, ω and σ are hyperparameters used to balance the loss between foreground and background; B i,j represents the pixel data with coordinates i, j in the binary image; f(.) represents an adaptive function that reshapes the features obtained by the student model into the same dimension as the features obtained by the teacher model; Represents the spatial attention mask at the position with coordinates i, j; represents the channel attention mask at the kth channel; Represents the feature value at the position with coordinates i,j under the kth channel in the features obtained by the teacher model; Represents the feature value at the position with coordinates i, j under the kth channel in the features obtained by the student model; represents the spatial attention mask used by the teacher model; represents the spatial attention mask used by the student model; represents the channel attention mask used by the teacher model; represents the channel attention mask used by the student model; γ represents the hyperparameter that balances the spatial attention mask and the channel attention mask; L global The calculation process is as follows: Among them, ε represents the hyperparameter used to balance the loss; W k , W v1 , W v2 represents convolution; LN represents normalization; N p1 Indicates F T The number of features in; N p2 Indicates F S The number of features in F T represents the image features obtained by the teacher model; F S represents the image features obtained by the student model; F j represents the jth eigenvalue; F m represents the mth eigenvalue.
9. The lightweight method for metal surface defect detection network model according to claim 8, characterized in that: The process of obtaining the outlier mask is as follows: Starting from the first pixel of the input image, traverse the pixels in the input image according to the set step size, and perform the following operations for the current pixel traversed: Select an image block of a preset size with the current pixel as the center point, and then calculate the average value p(c i ,c j ,w,h), the calculation formula is as follows: Among them, c i ,c j represents the center pixel position coordinates of the image block; w represents the width of the image block; h represents the height of the image block; f src (i, j) represents the grayscale value of the pixel with coordinates i, j; [*] represents rounding operation; Take the average value p(c i ,c j ,w,h) is used as the local correction factor of the central pixel of the image block; The local correction factor, the reference grayscale of the correction image and the original grayscale value of the central pixel of the image block are multiplied to obtain the processed grayscale pixel value of the central pixel of the image block; After traversing all the pixels, the rectified image is obtained, and the rectified image is divided into multiple sub-images. Each sub-image overlaps with the adjacent sub-image, and then the grayscale mean and variance of each sub-image are calculated; the threshold φ is set, and the binary image is obtained by threshold segmentation, where the calculation formula for the pixels in the binary image is: Where f(i,j) is the grayscale of the pixel with coordinates i,j in the rectified image. is the grayscale variance of the i-th sub-image; δ g is the global variance of the image; ψ is the global compensation amount, and μ is the local threshold segmentation weight.
Citation Information
Patent Citations
Aircraft skin surface defect detection method based on lightweight neural network
CN117474914A
Steel plate surface defect detection method and system based on knowledge distillation
CN118096768A
Cited By
Photovoltaic panel defect identification method based on knowledge distillation and pruning technology
CN119444751A
