A Multi-Scale Feature Fusion 3D Printing Defect Detection Method Based on Improved YOLOv8

Abstract

The present invention discloses a multi-scale feature fusion 3D printing defect detection method based on improved YOLOv8, which relates to the technical field of 3D printing defect detection. Through an adaptive multi-scale feature fusion network, the way of feature fusion is automatically adjusted according to the feature scale of the defect, and an adaptive feature pyramid, a cross-scale connection block and a spatial attention guidance module are integrated to improve the detection ability of 3D printing defects of different scales and types.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D printing defect detection, and specifically to a multi-scale feature fusion 3D printing defect detection method based on improved YOLOv8. Background Technique

[0002] 3D printing defect detection refers to using various technologies and methods to identify and evaluate possible defects in the 3D printing process. These defects may include inaccurate geometric shapes, surface defects (such as cracks, holes, layer misalignment), internal defects (such as inclusions, voids), etc. These defects may affect the structural integrity, functionality, and aesthetics of the printed parts, so they need to be detected and repaired in a timely manner during the production process.

[0003] Existing defect detection systems often only focus on defect detection without considering defect classification or degree estimation, greatly limiting the versatility of the model. At the same time, when collecting data singly, it often leads to insufficient ability of the model to distinguish different defect types. Therefore, we propose a multi-scale feature fusion 3D printing defect detection method based on improved YOLOv8 to solve the problems raised above. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-scale feature fusion 3D printing defect detection method based on improved YOLOv8 to solve the problems in the current market proposed in the above background technique.

[0005] To achieve the above purpose, the present invention provides the following technical solutions:

[0006] A multi-scale feature fusion 3D printing defect detection method based on improved YOLOv8, comprising the following steps:

[0007] Step 1: Collect 3D printing image data, and perform data augmentation on the collected data using DE-GAN;

[0008] Step 2: Construct AMF-Net, integrating AFP, CSC, and SAGM modules;

[0009] Step 3: Adopt the MTCL strategy for model training, and at the same time perform defect detection, classification, and degree estimation;

[0010] Step 4: Optimize the model performance through hyperparameter tuning, model pruning, and quantization;

[0011] Step 5: Develop IFOSystem to perform real-time detection of defects and optimize printing parameters;

[0012] Step 6: Use the CPI evaluation framework to comprehensively evaluate the model.

[0013] As a further optimization solution of the present invention, the specific steps for data augmentation using DE-GAN for data are as follows:

[0014] Step 1: Use CycleGAN as the GAN architecture for generating 3D printing defect images, and pre-train the GAN model using partially labeled images;

[0015] Step 2: Use the labeled defect images for training the GAN. Provide the real defect images as inputs to the generator of the GAN to generate realistic defect images. The discriminator learns to distinguish between real images and generated images, and alternately update the parameters of the generator and the discriminator until the GAN can generate high-quality defect images;

[0016] Step 3: Use the trained GAN to generate new defect images, manually evaluate the quality of the generated images to ensure that the images conform to the actual defect characteristics, and fuse the generated images with the original dataset to increase the diversity and scale of the dataset.

[0017] As a further optimization solution of the present invention, the specific steps for constructing AMF-Net and integrating AFP, CSC, and SAGM modules are as follows:

[0018] Step 1: Extract feature maps of different scales in the backbone network of YOLOv8, perform upsampling and downsampling operations on the feature maps to construct a pyramid structure, and use a learnable weight matrix to dynamically fuse feature maps of different scales;

[0019] Step 2: Add skip connections between adjacent layers of the pyramid to allow information to flow directly, and use Element-wise Addition to merge features from different layers;

[0020] Step 3: Compress the channels of the feature maps to calculate the spatial attention map, and use the calculated attention map to re-weight the feature maps to highlight important features;

[0021] Step 4: Define a new network class in PyTorch, inherit from the network class of YOLOv8, implement the AFP, CSC, and SAGM modules in the class, and integrate them according to the designed architecture. Write a forward propagation function to enable the correct flow of data in the network;

[0022] Step 5: Load the pre-trained weights of YOLOv8 as part of the network initialization, and set the hyperparameters of the network.

[0023] As a further optimization solution of the present invention, the specific steps for model training using the MTCL strategy and simultaneously performing defect detection, classification, and degree estimation are as follows:

[0024] Step 1: Keep the detection head of YOLOv8, which is used to generate bounding boxes and defect confidence. Add a new classification head for outputting defect types and another fully connected layer for outputting classification labels of defect degrees.

[0025] Step 2: Define the detection loss using the localization loss of YOLOv8, define the classification loss using cross-entropy loss, and define the degree evaluation loss using mean squared error.

[0026] Step 3: Extract two patches with similar defect features but different positions from the same image, extract two patches with different defect features from different images. Use InfoNCE Loss to measure the similarity between positive sample pairs and the difference between negative sample pairs. Extract the features for contrast from the network and calculate the contrast loss based on the extracted features.

[0027] Step 4: Pass the input data through the network, calculate the outputs of detection, classification, and degree estimation simultaneously, calculate the total loss, perform backpropagation based on the calculated loss, update the weights of the network, and use Adam to update the weights of the network.

[0028] Step 5: Record the loss value after each training batch to monitor the training progress. Regularly evaluate the model performance on the validation set, adjust the hyperparameters to prevent overfitting. According to the performance on the validation set, use a learning rate scheduler to adjust the learning rate and reduce overfitting by applying weight decay.

[0029] Step 6: After training is completed, evaluate the final performance of the model on the test set, calculate the mAP of detection, classification accuracy, and error metrics of degree estimation.

[0030] As a further optimization scheme of the present invention, the specific steps to optimize the model performance through hyperparameter tuning, model pruning, and quantization are as follows:

[0031] Step 1: Select the hyperparameters to be tuned, systematically traverse multiple combinations of hyperparameters, randomly select hyperparameter combinations for testing, find the best combination, use a probability model to predict the performance of hyperparameter combinations to guide the search process, record the hyperparameters and performance of each experiment, and select the hyperparameter combination with the best performance.

[0032] Step 2: Reduce the number of model parameters by removing unimportant weights, reduce the computational amount, and speed up the forward propagation of the model. Remove individual weights and entire neurons or convolutional kernels, train the complete model, remove weights according to the pruning criteria, and fine-tune the pruned model to restore performance. Start with a small pruning ratio and gradually increase it until satisfactory performance and model size are achieved.

[0033] Step 3: Reduce the model size by reducing the precision of the weights, quantize the weights after training, and fix the quantization values. Collect the maximum values of the weights and activations required for quantization, convert the floating-point weights to quantization values according to the statistics, fine-tune the quantized model to restore performance, evaluate the inference speed of the quantized model, evaluate the accuracy of the model on the test set, compare the sizes of the models before and after optimization, and measure the average inference time for the model to process a single sample.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0035] Through the adaptive multi-scale feature fusion network, the way of feature fusion is automatically adjusted according to the feature scale of the defect, and the adaptive feature pyramid, cross-scale connection block and spatial attention guidance module are integrated to improve the detection ability of 3D printing defects of different scales and types.

[0036] Adopt a multi-task contrastive learning strategy to simultaneously perform defect detection, classification and degree estimation to improve the comprehensive performance of the model. Use InfoNCE Loss to measure the similarity between positive sample pairs and the difference between negative sample pairs, and enhance the model's understanding of the differences between different defect types.

[0037] Use the defect-enhanced generative adversarial network to generate new defect images and fuse them with the original dataset to increase the diversity and scale of the dataset. And develop an IFOSystem to achieve real-time defect detection and printing parameter optimization, improving printing quality and efficiency. Through cross-domain adaptation methods, the model can maintain high performance in different 3D printing environments and materials.

[0038] The above summary is only for the purpose of the specification and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the drawings and the following detailed description. Brief Description of the Drawings

[0039] Figure 1 It is a flow block diagram of a multi-scale feature fusion 3D printing defect detection method based on improved YOLOv8 of the present invention;

[0040] Figure 2 It is a flow block diagram of model training using the multi-task contrastive learning strategy in the present invention. Detailed Embodiments

[0041] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0042] Embodiment 1

[0043] Please refer to Figure 1 、 Figure 2 , a multi-scale feature fusion 3D printing defect detection method based on improved YOLOv8, comprising the following steps:

[0044] Step 1: Collect 3D printing image data, and use DE-GAN to enhance the collected data.

[0045] Specifically, use CycleGAN as the GAN architecture for generating 3D printing defect images, and pre-train the GAN model with partially labeled images.

[0046] Use the labeled defect images to train the GAN. Provide the real defect images as inputs to the generator of the GAN to generate realistic defect images. The discriminator learns to distinguish between real images and generated images, and alternately updates the parameters of the generator and the discriminator until the GAN can generate high-quality defect images.

[0047] Use the trained GAN to generate new defect images, manually evaluate the quality of the generated images to ensure that the images conform to the actual defect characteristics, and fuse the generated images with the original dataset to increase the diversity and scale of the dataset.

[0048] Step 2: Build AMF-Net, integrating AFP, CSC, and SAGM modules.

[0049] Among them, AFP is an adaptive feature pyramid, which is used to dynamically fuse features of different scales and extract feature information of different scales.

[0050] Extract feature maps of different scales in the backbone network of YOLOv8, perform upsampling and downsampling operations on the feature maps to construct a pyramid structure, use a learnable weight matrix to dynamically fuse features of different scales, and automatically adjust the fusion method according to the feature scale of the defect.

[0051] CSC is a cross-scale connection block, which is used to allow information to flow directly between different layers and can effectively retain feature information of different levels.

[0052] Add skip connections between adjacent layers of the pyramid, use Element-wise Addition to merge features from different layers, and combine the high-resolution information of low-level features with the semantic information of high-level features.

[0053] SAGM is a spatial attention guidance module used to highlight important features and improve the robustness of the model.

[0054] Compress the channels of the feature map to calculate the spatial attention map, and use the calculated attention map to re-weight the feature map to highlight important features and suppress unimportant features.

[0055] Specifically, extract feature maps of different scales in the backbone network of YOLOv8, perform upsampling and downsampling operations on the feature maps, construct a pyramid structure, and use a learnable weight matrix to dynamically fuse features of different scales;

[0056] Add skip connections between adjacent layers of the pyramid to allow information to flow directly, and use Element-wise Addition to merge features from different layers;

[0057] Compress the channels of the feature map to calculate the spatial attention map, and use the calculated attention map to re-weight the feature map to highlight important features;

[0058] Define a new network class in PyTorch, inherit from the network class of YOLOv8, implement the AFP, CSC, and SAGM modules in the class, and integrate them according to the designed architecture, and write a forward propagation function to enable data to flow correctly in the network;

[0059] Load the pre-trained weights of YOLOv8 as part of the network initialization and set the hyperparameters of the network.

[0060] Furthermore, the formula used in the convolutional layer is:

[0061]

[0062] Among them, is the input feature map, and are the convolutional kernel and bias, represents the convolutional operation, and ReLU is the activation function.

[0063] In the cross-scale connection block of the feature pyramid network, by concatenating feature maps from different layers, the mathematical formula is:

[0064]

[0065] Among them, Represents the splicing operation of the feature map.

[0066] The spatial attention mechanism is used in the spatial attention guidance module, and its mathematical formula is:

[0067]

[0068] Among them, Squeeze is used to compress the dimension of the feature map, Excitation is used to stimulate the features, and the product is used to calculate the attention map.

[0069] Step 3: Adopt the MTCL strategy for model training, and at the same time perform defect detection, classification, and degree estimation;

[0070] Specifically, keep the detection head of YOLOv8, which is used to generate bounding boxes and defect confidence, add a new classification head for outputting defect types, and add another fully connected layer for outputting classification labels of defect degrees;

[0071] Use the localization loss of YOLOv8 to define the detection loss, use the cross-entropy loss to define the classification loss, and use the mean squared error to define the degree evaluation loss;

[0072] Extract two patches with similar defect features but different positions from the same image, extract two patches with different defect features from different images, use InfoNCE Loss to measure the similarity between positive sample pairs and the difference between negative sample pairs, extract the features for comparison from the network, and calculate the contrast loss according to the extracted features;

[0073] Pass the input data through the network, calculate the outputs of detection, classification, and degree estimation at the same time, calculate the total loss, perform backpropagation according to the calculated loss, update the weights of the network, and use Adam to update the weights of the network;

[0074] Record the loss value after each training batch for monitoring the training progress, regularly evaluate the model performance on the validation set, adjust the hyperparameters to prevent overfitting, according to the performance of the validation set, use the learning rate scheduler to adjust the learning rate, and reduce overfitting by applying weight decay;

[0075] After the training is completed, evaluate the final performance of the model on the test set, and calculate the mAP of detection, classification accuracy, and error metrics of degree estimation.

[0076] Furthermore, use IoU Loss to optimize the position of the bounding box, use binary cross-entropy loss to optimize the confidence of the bounding box, and use cross-entropy loss to optimize the classification of defects.

[0077] In the classification loss, the cross-entropy loss is used to calculate the difference between the probability distribution predicted by the model and the true label, and its formula is:

[0078]

[0079] Among them, is the predicted probability distribution of the model, is the true label.

[0080] In the degree estimation loss, the mean square error is used to evaluate the difference between the defect degree predicted by the model and the true value, and its formula is:

[0081]

[0082] Among them, is the predicted defect degree of the model, is the true defect degree.

[0083] In the contrast loss, InfoNCE Loss is used to measure the similarity between positive sample pairs and the difference between negative sample pairs, and its formula is:

[0084]

[0085] Among them, and are the features of the positive sample pair, is the feature of the negative sample, is the temperature parameter, is the feature representation of the positive sample pair.

[0086] Step 4: Optimize the model performance through hyperparameter tuning, model pruning, and quantization;

[0087] Specifically, select the hyperparameters to be tuned, systematically traverse multiple combinations of hyperparameters, randomly select hyperparameter combinations for testing, find the best combination, use a probability model to predict the performance of the hyperparameter combination, guide the search process, record the hyperparameters and performance of each experiment, and select the hyperparameter combination with the best performance;

[0088] Reduce the number of model parameters by removing unimportant weights, reduce the computational amount, speed up the forward propagation of the model, remove single weights and entire neurons or convolutional kernels, train the complete model, remove weights according to the pruning criteria, and fine-tune the pruned model to restore performance. Start from a small pruning ratio and gradually increase it until satisfactory performance and model size are achieved;

[0089] Reduce the model size by lowering the precision of the weights, quantize the weights after training and fix the quantization values, collect the maximum values of the weights and activations required for quantization, convert the floating-point weights to quantization values based on the statistics, fine-tune the quantized model to recover performance, evaluate the inference speed of the quantized model, evaluate the accuracy of the model on the test set, compare the sizes of the models before and after optimization, and measure the average inference time for the model to process a single sample.

[0090] Step Five: Develop an IFOSystem to perform real-time detection of defects and optimize the printing parameters;

[0091] Step Six: Use the CPI evaluation framework to comprehensively evaluate the model.

[0092] Any process or method description represented in the flowchart or otherwise described herein can be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a specific logical function or process. The scope of the preferred embodiments of the present invention includes additional implementations, where the functions may be executed in a substantially simultaneous manner or in a reverse order according to the functions involved, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0093] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in conjunction with these instruction execution systems, apparatuses, or devices.

[0094] It should be understood that the various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0095] Those of ordinary skill in the art of the present technology can understand that all or part of the steps carried by the method of implementing the above embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium, and when executed, includes one or a combination of the steps of the method embodiments.

[0096] In addition, each functional unit in various embodiments of the present invention may be integrated into a processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0097] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A multi-scale feature fusion 3D printing defect detection method based on improved YOLOv8, characterized by: The following steps are involved: Step 1: Collect 3D printing image data and use deep enhancement generative adversarial network to enhance the collected data; Specifically, CycleGAN is used as the GAN architecture for generating 3D printing defect images, and the GAN model is pre-trained using partially annotated images; The labeled defect images are used to train GAN, and the real defect images are provided as input to the GAN generator to generate realistic defect images. The discriminator learns to distinguish between real images and generated images, and the parameters of the generator and discriminator are updated alternately until GAN ​​can generate high-quality defect images. Generate new defect images using the trained GAN, manually evaluate the quality of the generated images to ensure that the images match the actual defect characteristics, and fuse the generated images with the original dataset to increase the diversity and scale of the dataset; Step 2: Build an adaptive multi-feature fusion network, integrating adaptive feature pyramid, context-aware segmentation and self-attention guidance modules; Specifically, feature maps of different scales are extracted from the backbone network of YOLOv8, up-sampled and down-sampled the feature maps, a pyramid structure is constructed, and a learnable weight matrix is ​​used to dynamically fuse features of different scales. Add skip connections between adjacent layers of the pyramid to allow information to flow directly, and use Element-wiseAddition to merge features from different layers; Compress the channels of the feature map to calculate the spatial attention map, and use the calculated attention map to reweight the feature map to highlight important features; Define a new network class in PyTorch, inherit the network class of YOLOv8, implement AFP, CSC and SAGM modules in the class, integrate them according to the designed architecture, and write the forward propagation function so that the data flows correctly in the network; Load the pre-trained weights of YOLOv8 as part of the network initialization and set the network's hyperparameters; Step 3: Use a multi-task collaborative learning strategy to train the model and perform defect detection, classification, and degree estimation simultaneously; The specific steps are: keep the YOLOv8 detection head and use it to generate bounding boxes and defect confidence, add a new classification head to output defect types, and add another fully connected layer to output classification labels for defect severity; Use YOLOv8's positioning loss to define the detection loss, use the cross entropy loss to define the classification loss, and use the mean square error to define the degree evaluation loss; Extract two patches with similar defect features but different positions from the same image, extract two patches with different defect features from different images, use InfoNCE Loss to measure the similarity between positive sample pairs and the difference between negative sample pairs, extract features for comparison from the network, and calculate the comparison loss based on the extracted features; Pass the input data through the network, calculate the output of detection, classification and degree estimation at the same time, calculate the total loss, backpropagate based on the calculated loss, update the weights of the network, and use Adam to update the weights of the network; Record the loss value after each training batch to monitor the training progress, evaluate the model performance on the validation set regularly, adjust the hyperparameters to prevent overfitting, adjust the learning rate using the learning rate scheduler based on the performance of the validation set, and reduce overfitting by applying weight decay; After training is completed, the final performance of the model is evaluated on the test set, and the mAP of the detection, the classification accuracy, and the error index of the degree estimation are calculated; Step 4: Optimize model performance through hyperparameter tuning, model pruning, and quantization; Specifically, select the hyperparameters to be tuned, systematically traverse multiple combinations of hyperparameters, randomly select hyperparameter combinations for testing, find the best combination, use probability models to predict the performance of hyperparameter combinations to guide the search process, record the hyperparameters and performance of each experiment, and select the hyperparameter combination with the best performance; By removing unimportant weights, reducing the number of model parameters, reducing the amount of computation, speeding up the forward propagation of the model, removing individual weights and entire neurons or convolution kernels, training the complete model, removing weights according to the pruning criteria, and fine-tuning the pruned model to restore performance, starting with a smaller pruning ratio and gradually increasing it until satisfactory performance and model size are achieved; Reduce the model size by reducing the precision of weights, quantize the weights after training and fix the quantization values, collect the maximum values ​​of weights and activations required for quantization, convert floating-point weights to quantized values ​​based on statistical data, perform fine-tuning on the quantized model to restore performance, evaluate the inference speed of the quantized model, evaluate the accuracy of the model on the test set, compare the size of the model before and after optimization, and measure the average inference time of the model processing a single sample; Develop intelligent optimization systems to detect defects and optimize printing parameters in real time; The models are thoroughly evaluated using a comprehensive performance evaluation framework.

Images