Patch fuse surface defect detection method and device based on reparameterization and attention mechanism

Abstract

The application discloses a patch fuse surface defect detection method and device based on reparameterization and an attention mechanism, and the method comprises the following steps: acquiring a surface image of a labeled patch fuse and forming a total training set; constructing and training a defect detection model based on the total training set; reparameterizing the trained defect detection model to convert it into an inference model; optimizing the trained defect detection model by using an NVIDIA TensorRT library, performing format conversion and saving; and detecting and classifying surface images of patch fuses to be detected. The application proposes a novel deep learning model based on reparameterization and an attention mechanism, which has small parameter quantity, high training speed, high inference speed, high accuracy and the like, and applies the deep learning model to a patch fuse defect detection task, and proposes a complete fuse defect detection process.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of industrial surface defect detection, and in particular to a surface defect detection method and device for a patch fuse based on reparameterization and attention mechanism. BACKGROUND

[0002] Electronic component surface defect detection is an indispensable important technology in the industrial field, and a successful electronic component surface defect detection method can help factories to discover and repair defects on the surface of electronic components in the production process, thereby improving product quality and production efficiency. The patch fuse is a common component in electronic components. The patch fuse defect may cause the electronic device to malfunction. If the patch fuse has a defect, it may cause poor contact, which may eventually cause the entire circuit to short circuit or overload. The defects of the patch fuse generally include missing cap, missing cap, tube cracking and cracked cap. Among them, the missing cap means that the two ends of the patch fuse are not covered with a protective layer, exposing the internal structure. The exposed cap means that one end of the patch fuse is not covered with a protective layer, exposing the internal structure. The tube cracking means that the shell of the patch fuse is cracked, exposing the internal structure. The cracked cap means that the two ends of the patch fuse are not covered with a protective layer, exposing the internal structure. These defects may cause the electronic device to malfunction, for example, may cause the circuit to malfunction, the circuit to be unstable, or poor contact.

[0003] Currently, in actual industrial production, patch fuse surface defect detection still relies heavily on manual inspection, but its low efficiency and poor generalization cannot meet the needs of current large-scale applications. With the rise of artificial intelligence, image-based deep learning detection methods have gradually emerged in the field of industrial surface defect detection. The mainstream surface defect detection network based on target detection mainly includes Faster-RCNN and YOLO series network. However, such networks have defects such as large number of parameters and difficult training, which are not suitable for patch fuse defect detection.

[0004] In addition, in the patch fuse surface defect, the patch fuse defect data is easy to obtain and the defect is generally easy to identify, and the patch fuse detection task requires high speed, so the patch fuse defect detection is suitable for using a lightweight model. The current mainstream lightweight model includes the efficent net series, mobile net series, and MobileViT, swin transformer, repvgg network, but its inference speed, parameter quantity and accuracy cannot all reach a high standard at the same time. At the same time, the implementation means of lightweight network generally includes pruning, knowledge distillation and reparameterization. However, in this task, pruning relies on a specific algorithm library and is difficult to implement, and knowledge distillation is unstable and difficult to converge.

[0005] Therefore, in view of the above problems, it is a difficult problem to be solved in the field to provide a patch fuse defect detection method which has high reasoning speed, parameter quantity and accuracy, and also considers stability and convergence. SUMMARY

[0006] The main purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and provide a patch fuse surface defect detection method and device based on reparameterization and attention mechanism. The present application proposes a new deep learning model based on reparameterization and attention mechanism, which has small parameter quantity, fast training speed and reasoning speed, and high accuracy. For the first time, reparameterization technology and attention mechanism are combined in the fuse defect detection task, a new attention mechanism is proposed, and the reparameterization technology is used to decouple the training and reasoning structure, and combined with the attention mechanism, the attention mechanism is used to select and amplify the features that are more important for fuse defect detection task in different scales.

[0007] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0008] In a first aspect, the present application provides a patch fuse surface defect detection method based on reparameterization and attention mechanism, comprising the following steps:

[0009] Obtain the surface image of the labeled patch fuse and pre-process it to form a total training set; each image in the total training set corresponds to a label; the label includes cap missing, cap missing, tube cracking and cap cracking, and good product;

[0010] Construct a defect detection model, and train the defect detection model using the total training set to obtain a trained defect detection model; the defect detection model includes a first module, a second module, a third module, a fourth module, an attention module and a linear classification layer; the first module, the second module, the third module and the fourth module are used to expand the receptive field of the patch fuse image features and extract local features; the attention module is used to strengthen the attention of the neural network to the surface image feature space dimension of the patch fuse; the linear classification layer is used to calculate the classification probability result and perform back propagation;

[0011] Reparameterize the trained defect detection model to convert it into a reasoning model; specifically, the convolutional layers and residual branches in the first module, the second module, the third module and the fourth module of the defect detection model are converted into new convolutional layers by filling the convolutional kernels, and the convolutional layers parallel to them are merged to obtain the reasoning model;

[0012] Save the reasoning model in ONNX format, call the NVIDIA TensorRT library for model optimization, and then convert the ONNX format reasoning model into a trt format reasoning model and save it;

[0013] The surface image of the to-be-tested patch fuse is detected by using the inference model in the trt format, a probability of the to-be-tested patch fuse belonging to a certain category is obtained, and the label corresponding to the maximum probability is taken as the fuse defect category.

[0014] As a preferred technical solution, the first module, the second module, the third module, the fourth module, the attention module and the linear classification layer are specifically:

[0015] The first module, the second module, the third module and the fourth module include a plurality of sub-modules, each of which includes a plurality of parallel convolution layers and residual branches; the convolution layer is followed by a Batch Normalization layer; the convolution layer absorbs the Batch Normalization layer; the outputs of the plurality of parallel convolution layers and residual branches are added and then input into the next module through a ReLU layer;

[0016] The attention module includes a plurality of spatial attention modules with different N values; in the attention module, the input is processed through a plurality of spatial attention modules, the calculation results are obtained and spliced, information exchange is performed through shuffle, and finally the output is input into the linear classification layer; the spatial attention module is spliced after Max pooling and Average pooling, and then the output is input into a convolution layer with an N*N size convolution kernel;

[0017] The linear classification layer calculates and obtains a classification probability result, and performs back propagation.

[0018] As a preferred technical solution, the input total training set is used to train the model, and a trained defect detection model is obtained, and the specific process is as follows:

[0019] The surface images of the patch fuses labeled in the total training set are converted into tensors, and are divided into a training set and a validation set according to a proportion;

[0020] The training samples in any training set are represented as:

[0021] X=={(x j ,y j ),1≤j≤M,y x ∈[1,...,m]}

[0022] Wherein x m represents a tensor representing the image, y j corresponds to the label corresponding to the image, M is the number of training samples, and m is the number of categories;

[0023] The output of the defect detection model is

[0024] O j= [O1(X), O2(X),..., O m (x)]

[0025] wherein O j (x) represents logits output corresponding to the j-th image;

[0026] updating the parameters of the continual learning model by optimizing the loss function; the optimization loss function is CrossEntropyLoss of the predicted value and the real label;

[0027] training the defect prediction model using the AdamW optimizer, and training all parameters jointly until convergence;

[0028] testing the prediction effect of the defect detection model according to a certain evaluation standard using the validation set; the evaluation standard is accuracy, misjudgment rate and defective product recall rate; the misjudgment rate is the probability of misjudging a good product as a defective product; the defective product recall rate is the probability of correctly judging a defective product as a defective product.

[0029] As a preferred technical solution, the first module, the second module, the third module and the fourth module include a plurality of sub-modules, each sub-module includes a plurality of parallel convolution layers and a residual branch, and specifically further includes:

[0030] Each sub-module in the first module includes two parallel convolution layers and a residual branch; the two parallel convolution layers include a convolution layer with a convolution kernel size of 5x5 and a convolution layer with a convolution kernel size of 3x3.

[0031] Each sub-module in the second module, the third module and the fourth module includes two parallel convolution layers and a residual branch; the two parallel convolution layers include a convolution layer with a convolution kernel size of 3x3 and a convolution layer with a convolution kernel size of 1x1.

[0032] As a preferred technical solution, the re-parameterization of the trained defect detection model further includes the following steps:

[0033] The convolution layer with a convolution kernel size of 3x3 and the residual branch in each sub-module of the first module are converted into a convolution layer with a convolution kernel size of 5x5 by filling 0 in the convolution kernel.

[0034] Merge the parallel convolution layers with a convolution kernel size of 5x5.

[0035] The convolution layer with a convolution kernel size of 1x1 and the residual branch in each sub-module of the second module, the third module and the fourth module are converted into a convolution layer with a convolution kernel size of 3x3 by filling 0 in the convolution kernel.

[0036] The convolutional layer with a parallel kernel size of 3*3 is merged.

[0037] As a preferred technical solution, the first module includes 1 sub-module, the second module includes 2 sub-modules, the third module includes 4 sub-modules, and the fourth module includes 14 sub-modules.

[0038] As a preferred technical solution, the pre-processing method specifically includes:

[0039] After scaling to the same size, the training samples are expanded by randomly adjusting the brightness, contrast, saturation and hue of the images, and randomly flipping and rotating the images.

[0040] In a second aspect, the present application also provides a patch fuse surface defect detection system based on re-parameterization and attention mechanism, which is applied to the patch fuse surface defect detection method based on re-parameterization and attention mechanism, and includes a collection task module, a first processing module, a second processing module, a format conversion module and a detection and classification module.

[0041] The collection task module acquires the surface images of the labeled patch fuses, scales them to the same size, expands the training samples by randomly adjusting the image parameters, and forms a total training set; each image in the total training set corresponds to a label; the label is used to reflect the detection result corresponding to the surface image of the patch fuse.

[0042] The first processing module is used to construct a defect detection model, train the defect detection model by using the total training set, and obtain a trained defect detection model; the defect detection model includes a first module, a second module, a third module, a fourth module, an attention module and a linear classification layer; the first module, the second module, the third module and the fourth module are used to expand the receptive field of the patch fuse image features and extract the local features thereof; the attention module is used to strengthen the attention of the neural network to the surface image feature space dimension of the patch fuse; and the linear classification layer is used to calculate and obtain the classification probability result and perform back propagation.

[0043] The second processing module is used to re-parameterize the trained defect detection model and convert it into an inference model; specifically, the convolutional layers and residual branches in each sub-module of the first module, the second module, the third module and the fourth module of the defect detection model are converted into new convolutional layers by convolution kernel padding, and the convolutional layers parallel thereto are merged to obtain the inference model; the convolutional layers are followed by a Batch Normalization layer.

[0044] The format conversion module is used to save the inference model in an ONNX format, call an NVIDIA TensorRT library to optimize the model, and then convert the ONNX format inference model into a trt format.

[0045] The detection classification module is configured to detect the surface image of the to-be-detected patch fuse by using the inference model in the TRT format, obtain the probability of the to-be-detected patch fuse belonging to a certain category, and take the label corresponding to the maximum probability as the fuse defect category.

[0046] In a third aspect, the present application provides an electronic device, which comprises:

[0047] at least one processor; and

[0048] a memory connected in communication with the at least one processor; wherein

[0049] The memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to execute the patch fuse surface defect detection method based on reparameterization and attention mechanism.

[0050] In a fourth aspect, the present application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the patch fuse surface defect detection method based on reparameterization and attention mechanism.

[0051] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0052] (1) Compared with the method based on the mainstream image classification model such as EfficientNetV1, EfficientNetV2, MobilenetV2, MobileViT-Vision Transformer, Swin Transformer and RepVGG, the parameter quantity of the present application is significantly reduced, and the training speed, accuracy and inference speed are improved to a certain extent, which greatly exceeds the defect detection accuracy standard and speed requirement required by the patch fuse industrial production;

[0053] (2) In the present application, the convolution layers and residual branches in the first module, the second module, the third module and the fourth module of the defect detection model are converted into new convolution layers by filling the convolution kernels, and the convolution layers parallel thereto are merged to reparameterize the lightweight network, thereby reducing the dependence on a specific algorithm library and improving the stability and convergence of the patch fuse defect detection model.

[0054] (3) The application first combines the reparameterization technique and the attention mechanism in the fuse defect detection task, uses the reparameterization technique to decouple the training and inference structures, and uses the attention mechanism to select and amplify features of different scales that are more important to the fuse defect detection task, so that the inference speed and the model parameter amount are considered.

[0055] (4) The application proposes a new attention mechanism for image data, combines different scale convolution layers and pooling layers in different branches to sufficiently capture features of different scales in the feature map, uses average pooling and max pooling to retain background and texture information of the feature map, and uses the shuffle technology to perform feature fusion, so that the calculation cost is greatly reduced. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0057] Figure 1 The flow chart of the patch fuse surface defect detection method based on reparameterization and attention mechanism in the embodiment of the present application;

[0058] Figure 2 The first schematic diagram of the defect detection model in the embodiment of the present application;

[0059] Figure 3 The second schematic diagram of the defect detection model in the embodiment of the present application;

[0060] Figure 4 The sub-module schematic diagram of the first module of the defect detection model in the embodiment of the present application;

[0061] Figure 5 The schematic diagram of the sub-module inference model of the first module in the embodiment of the present application; Figure 6 The sub-module schematic diagram of the second module, the third module and the fourth module of the defect detection model in the embodiment of the present application;

[0062] Figure 7 The schematic diagram of the sub-module inference model of the second module, the third module and the fourth module in the embodiment of the present application;

[0063] Figure 8 The schematic diagram of the attention stage structure in the embodiment of the present application;

[0064] Figure 9A structure schematic diagram of a patch fuse surface defect detection system based on reparameterization and attention mechanism according to an embodiment of the present application;

[0065] Figure 10 A structure diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to enable persons skilled in the art to better understand the scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work are within the scope of protection of the present application.

[0067] In the present application, the phrase "embodiment" means that the specific features, structures or characteristics described in combination with the embodiment can be contained in at least one embodiment of the present application. The appearance of this phrase at various places in the specification does not necessarily mean the same embodiment, nor is it an independent or alternative embodiment to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present application can be combined with other embodiments.

[0068] Please refer to Figure 1 The patch fuse surface defect detection method based on reparameterization and attention mechanism according to the present embodiment comprises the following steps:

[0069] S1. Obtain the surface images of labeled patch fuses, scale and unify them to the same size, randomly adjust the image parameters to expand the training samples, and form a total training set;

[0070] Optionally, each image in the total training set corresponds to a label;

[0071] Optionally, the label is used to reflect the detection result corresponding to the surface image of the patch fuse;

[0072] Optionally, the surface images of the labeled patch fuses come from the patch fuse image dataset collected in the industrial field, and the total sample number is 3963 after artificial cleaning, pretreatment and labeling;

[0073] Further, the average processing speed of one picture is 1.0 ms, the average inference time of one picture after the model is converted into the trt format is 0.40 ms, the model parameter amount is about 5.73 m, and the memory occupation during the training process is about 2600 M;

[0074] Preferably, the detection result corresponding to the surface image of the patch fuse includes cap missing, cap missing, tube cracking and cap cracking, and good product;

[0075] Preferably, the pre-processing method, specifically, the training samples are expanded by scaling to the same size, randomly adjusting brightness, contrast, saturation and hue, and randomly flipping and rotating the images.

[0076] S2. Constructing a defect detection model and inputting the total training set to train the model to obtain a trained defect detection model;

[0077] Optionally, as shown in Figure 2 , 3 , the defect detection model comprises a first module, a second module, a third module, a fourth module, an attention module and a linear classification layer.

[0078] Preferably, the first module, the second module, the third module, the fourth module, the attention module and the linear classification layer are specifically,

[0079] The first module, the second module, the third module and the fourth module each comprise a plurality of sub-modules, each sub-module comprising a plurality of parallel convolution layers and a residual branch; the convolution layer is followed by a Batch Normalization layer; the outputs of the plurality of parallel convolution layers and the residual branch are added and then input to the next module through a ReLU layer.

[0080] Further, the first module, the second module, the third module and the fourth module each comprise a plurality of sub-modules, each sub-module comprising parallel convolution layers and a residual branch, as shown in Figures 4-8 , specifically further comprising,

[0081] Each sub-module in the first module comprises two parallel convolution layers and a residual branch; the two parallel convolution layers comprise a convolution layer with a convolution kernel size of 5x5 and a convolution layer with a convolution kernel size of 3x3, as shown in Figure 4 , 5 ;

[0082] Each sub-module in the second module, the third module and the fourth module comprises two parallel convolution layers and a residual branch; the two parallel convolution layers comprise a convolution layer with a convolution kernel size of 3x3 and a convolution layer with a convolution kernel size of 1x1, as shown in Figure 6 , 7 ;

[0083] Further, the number of sub-modules of the first module, the second module, the third module and the fourth module can be 1, 2, 4 and 14, respectively.

[0084] The attention module comprises a plurality of spatial attention modules, as shown in Figure 8As shown; in the attention module, the input passes through multiple spatial attention modules, the calculation results are obtained and spliced, information exchange is performed through shuffle, and finally the output is output as the input of the linear classification layer; the spatial attention module is spliced after Max pooling and Average pooling, and then output through the convolution layer of the N×N size convolution kernel;

[0085] Further, the N value of the N×N size convolution kernel of the spatial attention module is 3, 5;

[0086] The linear classification layer calculates and obtains the classification probability result, and performs back propagation;

[0087] Optionally, the input total training set is used to train the model to obtain a trained defect detection model, specifically

[0088] S21. Convert the surface image of the labeled patch fuse in the total training set into a tensor, and divide it into a training set and a validation set according to a proportion;

[0089] S22. The training sample in the training set is represented as

[0090] X=={(x j ,y j ),1≤j≤M,y j ∈[1,...,m]}

[0091] Where x j represents a tensor representing the image, y j is the corresponding label corresponding to the image, M is the number of training samples, and m is the number of categories;

[0092] S23. The output of the defect detection model is

[0093] O x =[O1(X),O2(X),...,O m (x)]

[0094] Where O j (x) represents the logits output corresponding to the jth image;

[0095] S24. Update the parameters of the continuous learning model by optimizing the loss function; the optimization loss function is the CrossEntropyLoss of the predicted value and the true label;

[0096] S25. Use the AdamW optimizer to train the model, and train all parameters jointly until convergence;

[0097] Optionally, the learning rate of the AdamW optimizer training model is 1e-4, and the weight_decay is 2e-5.

[0098] S26. Test the prediction effect of the defect detection model according to certain evaluation criteria using the validation set;

[0099] Preferably, the total training set is divided into a training set and a validation set at a ratio of 4:1.

[0100] Preferably, the evaluation criteria are accuracy, misjudgment rate, and defective product recall rate.

[0101] Further, the misjudgment rate is the probability of actual label being good product but misjudged as defective product; and the defective product recall rate is the probability of actual defective product being correctly judged as defective product.

[0102] Further, the five-classification accuracy on the patch fuse data set is 95.08%, the misjudgment rate is 1.84%, and the defective product recall rate is 99.42%; the two-classification (defective product and good product) accuracy on the patch fuse data set is 95.96%, the misjudgment rate is 4.74%, and the defective product recall rate is 96.33%. The accuracy of the model on the patch fuse data set reaches 99.12% after loading the imagenet pre-training weight.

[0103] S3. Re-parameterize the trained defect detection model to convert it into an inference model;

[0104] Optionally, the first module, the second module, the third module, and the fourth module of the defect detection model are converted into new convolution layers by filling the convolution kernels, and the convolution layers parallel thereto are merged to obtain the inference model, as shown in Figure 8 .

[0105] Preferably, the convolution layer is followed by a Batch Normalization layer.

[0106] Further, the re-parameterization specifically includes the following steps,

[0107] S31. The convolution layer with a convolution kernel size of 3x3 and the residual branch in each sub-module of the first module are converted into a convolution layer with a convolution kernel size of 5x5 by filling the convolution kernel with 0.

[0108] S32. Merge the convolution layers with a convolution kernel size of 5x5 in parallel.

[0109] S33. The convolution layer with a convolution kernel size of 1x1 and the residual branch in each sub-module of the second module, the third module, and the fourth module are converted into a convolution layer with a convolution kernel size of 3x3 by filling the convolution kernel with 0.

[0110] S34. Merge and convolution kernel size of 3x3 convolution layer in parallel.

[0111] S4. Save the inference model in ONNX format, call the NVIDIA TensorRT library for model optimization, and then convert the inference model in ONNX format to trt format and save;

[0112] Optionally, the NVIDIA TensorRT precision is INT32.

[0113] S5. Use the inference model in trt format to detect the surface image of the test patch fuse, obtain the probability of the test patch fuse belonging to a certain category, and take the label corresponding to the maximum probability as the fuse defect type;

[0114] Optionally, the present application is completed on a linux (ubuntu) system, the GPU uses 2080ti, the cuda version is 11.6, the cudnn version is 8.3.2, the Python version is 3.9.13, the tensorrt version is 8.5.3.1, the pytorch version is 1.13.1, and the opencv version is 4.7.0.

[0115] Based on the same idea as the patch fuse surface defect detection method based on reparameterization and attention mechanism in the above embodiment, the present application also provides a patch fuse surface defect detection system based on reparameterization and attention mechanism, which can be used to execute the patch fuse surface defect detection method based on reparameterization and attention mechanism. For the convenience of description, in the structure diagram of the embodiment of the patch fuse surface defect detection system based on reparameterization and attention mechanism, only the part related to the embodiment of the present application is shown, and those skilled in the art can understand that the structure shown in the diagram does not constitute a limitation on the device, and more or fewer components than those shown in the diagram can be included, or some components can be combined, or different components can be arranged.

[0116] Please refer to Figure 9 In another embodiment of the present application, a patch fuse surface defect detection system 10 based on reparameterization and attention mechanism is provided, which includes a collection task module 11, a first processing module 12, a second processing module 13, a format conversion module 14 and a detection and classification module 15.

[0117] The collection task module 11 is used to obtain the surface images of labeled patch fuses, scale and unify them to the same size, randomly adjust the image parameters to expand the training samples, and form a total training set; each image in the total training set corresponds to a label; the label is used to reflect the detection result corresponding to the surface image of the patch fuse.

[0118] The first processing module 12 is configured to construct a defect detection model, train the defect detection model by using the total training set, and obtain a trained defect detection model; the defect detection model comprises a first module, a second module, a third module, a fourth module, an attention module, and a linear classification layer; the first module, the second module, the third module, and the fourth module are configured to expand the receptive field of the patch fuse image features and extract local features thereof; the attention module is configured to strengthen the attention of the neural network to the surface image feature space dimension of the patch fuse; and the linear classification layer is configured to calculate and obtain a classification probability result and perform back propagation;

[0119] The second processing module 13 is configured to re-parameterize the trained defect detection model and convert it into an inference model; specifically, the convolution layers and residual branches in each sub-module of the first module, the second module, the third module, and the fourth module of the defect detection model are converted into new convolution layers by filling the convolution kernels, and the convolution layers parallel thereto are merged to obtain the inference model; and the convolution layers are followed by a Batch Normalization layer.

[0120] The format conversion module 14 is configured to save the inference model in an ONNX format, call an NVIDIA TensorRT library to optimize the model, and then convert the inference model in the ONNX format into a trt format and save it.

[0121] The detection and classification module 15 is configured to detect the surface image of the to-be-tested patch fuse by using the inference model in the trt format, obtain the probability of the to-be-tested patch fuse belonging to a certain category, and take the label corresponding to the maximum probability as the fuse defect type.

[0122] It should be noted that the patch fuse surface defect detection system based on re-parameterization and attention mechanism of the present application corresponds to the patch fuse surface defect detection method based on re-parameterization and attention mechanism of the present application, and the technical features and advantages described in the embodiment of the patch fuse surface defect detection method based on re-parameterization and attention mechanism are applicable to the embodiment of the patch fuse surface defect detection based on re-parameterization and attention mechanism. For specific content, please refer to the description in the method embodiment of the present application. Here, no further description is given, and it is hereby declared.

[0123] Further, the logical division of the program modules in the implementation of the patch fuse surface defect detection system based on reparameterization and attention mechanism in the above embodiments is only illustrative. In actual applications, the above functions can be completed by different program modules according to needs, for example, for the configuration requirements of corresponding hardware or the convenience of software implementation. That is, the internal structure of the patch fuse surface defect detection system based on reparameterization and attention mechanism is divided into different program modules to complete all or part of the functions described above.

[0124] Please refer to Figure 10 In one embodiment, an electronic device implementing a patch fuse surface defect detection method based on reparameterization and attention mechanism is provided. The electronic device 20 can include a first processor 21, a first memory 22 and a bus, and can further include a computer program stored in the first memory 22 and executable on the first processor 21, such as a patch fuse surface defect detection program based on reparameterization and attention mechanism 23.

[0125] The first memory 22 includes at least one type of readable storage medium, including a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. The first memory 22 can be an internal storage unit of the electronic device 20 in some embodiments, such as a mobile hard disk of the electronic device 20. The first memory 22 can also be an external storage device of the electronic device 20 in other embodiments, such as a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the first memory 22 can include both an internal storage unit and an external storage device of the electronic device 20. The first memory 22 can be used not only to store application software and various data installed in the electronic device 20, such as the code of the patch fuse surface defect detection program based on reparameterization and attention mechanism 23, but also to temporarily store data that has been output or will be output.

[0126] The first processor 21 may, in some embodiments, be composed of integrated circuits, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple packaged integrated circuits of the same function or different functions, including one or more combinations of central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 21 is the control core of the electronic device, connects various components of the entire electronic device through various interfaces and lines, and executes various functions and processes data of the electronic device 20 by running or executing programs or modules stored in the first memory 22 and calling data stored in the first memory 22.

[0127] Figure 10 Only the electronic device with components is shown, and those skilled in the art can understand that, Figure 10 The structure shown does not constitute a limitation on the electronic device 20, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0128] The first memory 22 in the electronic device 20 stores a patch fuse surface defect detection program 23 based on reparameterization and attention mechanism, which is a combination of multiple instructions and can realize:

[0129] Obtain the surface image of the labeled patch fuse and pre-process it to form a total training set; each image in the total training set corresponds to a label; the label includes cap missing, cap missing, tube cracking, and cap cracking, and good product;

[0130] Construct a defect detection model and train the defect detection model using the total training set to obtain a trained defect detection model; the defect detection model includes a first module, a second module, a third module, a fourth module, an attention module, and a linear classification layer; the first module, the second module, the third module, and the fourth module are used to expand the receptive field of the patch fuse image features and extract local features; the attention module is used to strengthen the attention of the neural network to the surface image feature space dimension of the patch fuse; and the linear classification layer is used to calculate the classification probability result and perform back propagation;

[0131] Reparameterize the trained defect detection model to convert it into an inference model; specifically, the convolution layers and residual branches in the first module, the second module, the third module, and the fourth module of the defect detection model are converted into new convolution layers by filling the convolution kernels, and the convolution layers parallel thereto are merged to obtain the inference model;

[0132] The inference model is saved in an ONNX format, a NVIDIA TensorRT library is called for model optimization, and then the inference model in the ONNX format is converted into an inference model in a trt format and saved;

[0133] The inference model in the trt format is used to detect the surface image of the to-be-tested patch fuse, to obtain the probability of the to-be-tested patch fuse belonging to a certain category, and the label corresponding to the maximum probability is taken as the fuse defect category.

[0134] Further, the modules / units of the electronic device 20, if realized in the form of software function units and sold or used as independent products, can be stored in a nonvolatile computer readable storage medium. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM).

[0135] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The program can be stored in a non-volatile computer readable storage medium, and when the program is executed, the processes of the above-mentioned embodiments can be included. In the embodiments provided in the present application, any reference to memory, storage, database or other medium can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0136] Each technical feature of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of each technical feature in the above embodiments are not described, however, as long as the combination of these technical features does not exist contradictory, it should be considered as the scope of the present application.

[0137] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications, etc. made without departing from the spirit and principles of the present application should be equivalent replacement manners and should be included in the protection scope of the present application.

Claims

1. A method for surface defect detection of a patch fuse based on reparameterization and attention mechanism, characterized in that, The method comprises the following steps: Obtaining the surface image of the labeled patch fuse and pre-processing to form a total training set; each image in the total training set corresponds to a label; the label includes cap missing, cap missing, tube cracking and cap cracking, and good product; A defect detection model is constructed, and the total training set is used to train the defect detection model to obtain a trained defect detection model; the defect detection model comprises a first module, a second module, a third module, a fourth module, an attention module and a linear classification layer, and the first module, the second module, the third module and the fourth module are used to expand the receptive field of the patch fuse image features and extract local features thereof; The attention module is used to strengthen the attention of the neural network to the surface image feature space dimension of the patch fuse; and the linear classification layer is used to calculate and obtain a classification probability result and perform back propagation; The trained defect detection model is re-parameterized to convert into an inference model; specifically, the convolution layers and residual branches in the first module, the second module, the third module and the fourth module of the defect detection model are converted into new convolution layers through convolution kernel padding, and the convolution layers parallel thereto are merged to obtain the inference model; The inference model is saved in ONNX format, the NVIDIA TensorRT library is called for model optimization, and then the inference model in ONNX format is converted into an inference model in trt format and saved; The inference model in trt format is used to detect the surface image of the patch fuse to be tested to obtain the probability of the patch fuse to be tested belonging to a certain category, and the label corresponding to the maximum probability is taken as the defect type of the fuse; The first module, the second module, the third module, the fourth module, the attention module and the linear classification layer are specifically: The first module, the second module, the third module and the fourth module each comprise 1, 2, 4 and 14 sub-modules respectively, each sub-module comprises two parallel convolution layers and a residual branch; the two parallel convolution layers are followed by a BatchNormalization layer; the outputs of the two parallel convolution layers and the residual branch are added and then input into the next module through a ReLU layer; The two parallel convolutional layers in the first module include a convolutional layer with a convolution kernel size of and a convolutional layer with a convolution kernel size of ​ The two parallel convolutional layers in each of the second module, the third module, and the fourth module include a convolutional layer with a convolution kernel size of and a convolutional layer with a convolution kernel size of ​ The attention module includes two spatial attention modules, in which the input is subjected to the two spatial attention modules, the calculation results are obtained and spliced, information exchange is performed through shuffle, and finally the output is output as the input of the linear classification layer; the two spatial attention modules are respectively subjected to Max pooling and Average pooling and then spliced, and the output is subjected to a convolution layer with a convolution kernel of a size of N*1*N, and N respectively takes a value of 3 and 5. a convolution layer with a convolution kernel of a size of N*1*N, and N respectively takes a value of 3 and 5. The linear classification layer is used to calculate and obtain a classification probability result and perform back propagation.

2. The method according to claim 1, wherein, The total training set is input into the model to obtain a trained defect detection model, specifically: The surface image of the labeled patch fuse in the total training set is converted into a tensor, and is divided into a training set and a validation set according to a proportion; The training sample in the training set is represented as: ； wherein a tensor representing the image, a corresponding label corresponding to the image, M is the number of training samples, and m is the number of categories. The output of the defect detection model is: O X =[ O 1( X ), O 2( X ),…， O M ( X )]; wherein O j ( X ) represents logits output corresponding to the jth image pair; The parameters of the continuous learning model are updated by optimizing the loss function; the optimization loss function is the CrossEntropyLoss of the predicted value and the true label; The defect prediction model is trained using the AdamW optimizer, and all parameters are trained jointly until convergence; The prediction effect of the defect detection model is tested according to a certain evaluation standard using the validation set; the evaluation standard is specifically the accuracy, the misjudgment rate and the defective product recall rate; the misjudgment rate is the probability that the actual label is a good product but is misjudged as a defective product; and the defective product recall rate is the probability that the actual defective product is correctly judged as a defective product.

3. The method of claim 1, wherein, The re-parameterization of the trained defect detection model further comprises the following steps: The size of the convolution kernel in the sub-module of the first module is The convolution layer with the convolution kernel size of is converted into a convolution layer with the convolution kernel size of by filling 0 with the convolution kernel. The size of the merged parallel convolution kernel is convolution layers The size of the convolution kernel in each submodule of the second module, the third module, and the fourth module is The convolution layer with the convolution kernel size of is converted into a convolution layer with the convolution kernel size of The size of the merged parallel convolution kernel is convolution layers.

4. The method of claim 1, wherein, The pre-processing method specifically comprises: The images are scaled to the same size, the brightness, contrast, saturation and hue are randomly adjusted, and the images are randomly flipped and rotated to expand the training samples.

5. A patch fuse surface defect detection system based on reparameterization and attention mechanism, characterized in that, The patch fuse surface defect detection method based on re-parameterization and attention mechanism according to any one of claims 1-4 comprises a collection task module, a first processing module, a second processing module, a format conversion module and a detection and classification module. The collection task module acquires surface images of the labeled patch fuse, scales them to the same size, randomly adjusts the image parameters to expand the training samples, and forms a total training set. Each image in the total training set corresponds to a label; the label is used to reflect the detection result corresponding to the surface image of the patch fuse. The first processing module is used to construct a defect detection model, and train the defect detection model by using the total training set to obtain a trained defect detection model; the defect detection model comprises a first module, a second module, a third module, a fourth module, an attention module and a linear classification layer; the first module, the second module, the third module and the fourth module are used to expand the receptive field of the patch fuse image features and extract local features thereof; The attention module is used to strengthen the attention of the neural network to the surface image feature space dimension of the patch fuse; and the linear classification layer is used to calculate and obtain a classification probability result and perform back propagation; The second processing module is used to re-parameterize the trained defect detection model into an inference model; specifically, the convolution layers and residual branches in each sub-module of the first module, the second module, the third module and the fourth module of the defect detection model are converted into new convolution layers by filling the convolution kernels, and the convolution layers in parallel therewith are merged to obtain the inference model; the convolution layers in each sub-module are followed by a Batch Normalization layer; The format conversion module is used to save the inference model in an ONNX format, call a NVIDIA TensorRT library to optimize the model, and then convert the ONNX format inference model into a trt format; The detection and classification module is used to detect the surface image of the to-be-tested patch fuse by using the inference model in the trt format, obtain the probability of the to-be-tested patch fuse belonging to a certain category, and take the label corresponding to the maximum probability as the fuse defect type.

6. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to execute the patch fuse surface defect detection method based on re-parameterization and attention mechanism according to any one of claims 1-4.

7. A computer-readable storage medium storing a program, the program comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 6. The program is executed by the processor to implement the patch fuse surface defect detection method based on re-parameterization and attention mechanism according to any one of claims 1-4.

Images