Defect real-time monitoring method for 3D printing intestinal fistula patch based on machine vision

Abstract

The invention discloses a real-time defect monitoring method for a 3D printing intestinal fistula patch based on machine vision. The method comprises the following steps: establishing a 3D printing intestinal fistula patch defect data set comprising a plurality of defect types; based on the improved YOLOv8 neural network, establishing a neural network structure model; training the neural network structure model through the established 3D printing intestinal fistula patch defect data set, and taking the trained neural network structure model as a defect identification model; and the defects in the intestinal fistula patch monitoring image are classified and positioned through the defect recognition model, and real-time defect monitoring of the intestinal fistula patch in the 3D printing process is achieved. According to the invention, the real-time monitoring and correction printing process of the 3D printing of the intestinal fistula patch is realized, and the 3D printing effect of the intestinal fistula patch is ensured.

Description

Technical Field

[0001] The present invention belongs to the field of intestinal fistula patch manufacturing, relates to a five-axis 3D printed intestinal fistula patch, and specifically relates to a real-time defect monitoring method of a 3D printed intestinal fistula patch based on machine vision. Background Art

[0002] Five-axis 3D printing is based on traditional 3D printing technology, with two rotation axes added for adaptive adjustment of the printing platform posture, which can realize surface orientation and swing deposition, surface unequal thickness slice and body deposition, etc. Intestinal stent is an implantable curved tube structure device for intestinal fistula treatment made of bio-additive manufacturing technology and biocompatible flexible polymer materials. With the development of 3D printing technology, personalized customized intestinal fistula stents have become possible, which can better adapt to the patient's anatomical structure and improve the treatment effect. However, various problems may occur in the 3D printing process of the stent, such as material blockage, printing deviation, poor interlayer bonding, etc., which will affect the quality and function of the stent. Since intestinal fistula stents are directly used in patients, their quality and safety requirements are extremely high. Any minor defects may lead to serious consequences, so it is crucial to implement strict quality control and real-time monitoring during the manufacturing process. An effective monitoring algorithm is needed to monitor and correct the printing process in real time to ensure the quality of the final product. Summary of the invention

[0003] Purpose of the invention: In order to overcome the deficiencies in the prior art, a real-time defect monitoring method for 3D printed intestinal fistula patches based on machine vision is provided, which realizes real-time monitoring and correction of the 3D printing of intestinal fistula patches and ensures the 3D printing effect of intestinal fistula patches.

[0004] Technical solution: To achieve the above purpose, the present invention provides a real-time defect monitoring method for a 3D printed intestinal fistula patch based on machine vision, comprising the following steps:

[0005] S1: Establish a 3D printed intestinal fistula patch defect dataset including various defect types;

[0006] S2: Based on the improved YOLOv8 neural network, a neural network structure model is established;

[0007] S3: The neural network structure model is trained by establishing a 3D printed intestinal fistula patch defect dataset, and the trained neural network structure model is used as a defect recognition model;

[0008] S4: The defects in the intestinal fistula patch monitoring image are classified and located through the defect recognition model to achieve real-time defect monitoring of the intestinal fistula patch 3D printing process.

[0009] Furthermore, the method for establishing the 3D printed intestinal fistula patch defect data set in step S1 includes:

[0010] A1: Set up the printing models of defective and non-defective 3D printed fistula patches respectively, collect video of the entire printing process, and collect video images of the 3D printed non-defective fistula patches and defective fistula patches respectively;

[0011] A2: Image feature preprocessing and image data enhancement of 3D printed intestinal fistula patch defect images;

[0012] A3: Use the labelimg data labeling tool to label the defective parts of the collected inspection images.

[0013] Furthermore, the method of image feature preprocessing in step A2 is: converting the collected color image into a grayscale image, adding salt and pepper noise, and then performing corrosion and expansion operations on the image;

[0014] The method of image data enhancement is to perform geometric transformations such as random angle rotation, flipping and scaling on the image, then apply noise, occlusion and other operations, and finally use mosaic data enhancement.

[0015] Furthermore, the defect types of the defective intestinal fistula patch video image in step A1 include interlayer peeling, unformed printing, and nozzle clogging.

[0016] Furthermore, the target detection network architecture of the neural network structure model in step S2 includes a backbone network and a feature fusion network. The backbone network adopts a target detection algorithm based on a one-stage network, and the feature fusion network is composed of three optimization modules: a content-aware feature reconstruction framework, a heavy-parameter generalized feature pyramid network, and a strong intersection-over-union loss function.

[0017] Furthermore, the processing flow of the target detection algorithm based on the one-stage network in the backbone network is as follows: preprocess the input image, configure it as different scaling and filling modes according to the parameters, and then perform convolution, maximum pooling and other operations in the backbone network. Use two 3*3 convolutions in succession to directly reduce the resolution by 4 times. When the receptive field is small, down-sample after using one layer of convolution twice in succession. Through more cross-layer connections with different scores, the gradient flow of the model is enriched, and different numbers of channels are adjusted for models of different scales. At the same time, two convolution connection layers are removed, and the mainstream decoupling head structure is used at the output end.

[0018] Furthermore, the content-aware feature reconstruction framework in the feature fusion network uses a lightweight network to upsample the CARAFE operator to design a one-stage network model. The calculation process of the CARAFE operator includes: according to the feature reconstruction of the content of each target position, the low-resolution feature map is upsampled to generate a feature map of the same size as the high-resolution feature map for interaction to obtain the context information required for reconstruction; then the predicted features are reconstructed, and the interactive feature map is subjected to corresponding convolution operations to obtain the final upsampling result.

[0019] Furthermore, the heavy-parameter generalized feature pyramid network in the feature fusion network is improved on the basis of GFPN, and the fusion module is replaced on the basis of utilizing skip layers and cross-scale connections; the skip layer connection reduces the vanishing gradient during the back propagation process and utilizes two feature connection methods; cross-scale connections are utilized to overcome multi-scale changes, and bilinear interpolation and maximum pooling are respectively used as upsampling and downsampling functions.

[0020] Furthermore, the robust intersection-over-union loss function in the feature fusion network combines an object size adaptive penalty factor and a gradient adjustment function based on the quality of the anchor box;

[0021] The strong intersection-over-union loss function is:

[0022]

[0023] L GIoU Loss =1-L GIoU

[0024] Gradient adjustment function based on anchor box quality:

[0025]

[0026] α i =exp(-β·IoU(A i ,B i ))

[0027] SmoothL1 is a regression loss function. The complete loss function expression is:

[0028] L=L GIoU +L reg .

[0029] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0030] 1. The present invention proposes a real-time defect monitoring method for 3D printed intestinal fistula patches based on machine vision, and improves the model based on the YOLOv8 framework. In terms of model design, the upsampling operator is improved to improve the effect of feature extraction, and the multi-scale information of the target object is captured by combining the heavy parameter generalized feature pyramid fusion network. The regional extraction idea is used to introduce a smaller local receptive field, pay more attention to the target object area, provide a more effective information transmission method, expand to a deeper network, and introduce a strong intersection-over-union loss function to enhance the focusing ability of medium-quality anchor frames, thereby achieving an increase in average accuracy and performance improvement.

[0031] 2. The present invention collects high-resolution images and other sensor data in real time, selects industrial cameras, lenses, and designs adapter brackets to build a machine vision system, effectively collects images during the printing process, and provides a basis for subsequent data set construction and real-time detection research.

[0032] 3. The present invention realizes real-time monitoring of 3D printed intestinal fistula patches, which can timely discover defects of 3D printed intestinal fistula patches, help improve product quality, reduce production costs, and improve production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the method of the present invention;

[0034] Figure 2 It is a schematic diagram of the operation process of the neural network structure model;

[0035] Figure 3 Schematic diagram of the operation of the content-aware feature reorganization framework;

[0036] Figure 4 This is a schematic diagram of the operation of the heavy parameter generalized feature pyramid network;

[0037] Figure 5 Printing process images for five-axis additive manufacturing;

[0038] Figure 6 Data enhancement methods for five-axis additive manufacturing.

[0039] Figure 7 This is a display diagram of the five-axis additive manufacturing dataset images and annotation files;

[0040] Figure 8 Time and accuracy graph for five-axis additive manufacturing printing defect type monitoring;

[0041] Fig. 9 Line graph of ablation comparison experiment results for neural network performance improvement;

[0042] Fig.10 is the detection image and the prediction result image. DETAILED DESCRIPTION

[0043] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0044] like Figure 1 As shown, the present invention provides a real-time defect monitoring method for a 3D printed intestinal fistula patch based on machine vision, comprising the following steps:

[0045] S1: Use the CCD camera image acquisition system to collect 3D printed intestinal fistula patch process images, obtain 3D printed intestinal fistula patch monitoring images, and establish a 3D printed intestinal fistula patch defect dataset including various defect types:

[0046] S2: Based on the improved YOLOv8 neural network, a neural network structure model is established;

[0047] S3: The neural network structure model is trained by establishing a 3D printed intestinal fistula patch defect dataset, and the trained neural network structure model is used as a defect recognition model;

[0048] S4: The defects in the intestinal fistula patch monitoring image are classified and located through the defect recognition model to achieve real-time defect monitoring of the intestinal fistula patch 3D printing process.

[0049] The method for establishing the 3D printed intestinal fistula patch defect data set in step S1 includes:

[0050] A1: Set up the defective and non-defective printing models of 3D printed fistula patches respectively, use the display screen and camera to study the relevant configuration method content and camera call, and collect video of the entire printing process, and collect video images of 3D printed fistula patches without defects and defective fistula patches respectively;

[0051] The defect types of defective intestinal fistula patch video images include interlayer peeling, printing failure, and nozzle clogging. Based on different printing temperatures, printing layer heights, printing speeds, and extrusion coefficients, TPU87A and TPU95A materials were subjected to single-material and composite material basic experimental research and printing performance tests; interlayer peeling is caused by layer adhesion problems due to system parameters such as too fast printing speed, too high layer height, and too low printing temperature; printing failure is caused by under-extrusion and over-extrusion problems due to system parameters such as too small extrusion coefficient and too high printing temperature; nozzle clogging is caused by under-extrusion and over-extrusion problems due to system parameters such as too low printing temperature, too fast printing speed, and too high extrusion coefficient;

[0052] A2: Image feature preprocessing and image data enhancement of 3D printed intestinal fistula patch defect images:

[0053] The method of image feature preprocessing is: converting the collected color image into a grayscale image, adding salt and pepper noise, and then performing erosion and dilation operations on the image;

[0054] like Figure 6 As shown in the figure, the image data enhancement methods include original image, rotation and flipping, adding noise, and mosaic data enhancement. Specifically, the image is subjected to geometric transformations such as random angle rotation, flipping, and scaling, and then noise and occlusion operations are applied. Finally, mosaic data enhancement is used to improve the anti-interference ability of the network.

[0055] A3: Use Labelimg to mark the print defect areas. The marking results are in yolo label format and saved as txt files. After the data set is marked, randomly select images to set as training sets and data sets, and collect images and marking files. Figure 7 shown.

[0056] In this embodiment, reference Figure 5 , Aiming at the five-axis additive manufacturing printing process images, including four situations: normal printing process, interlayer peeling defects, printing non-forming defects, and nozzle clogging failure, a five-axis additive manufacturing defect dataset including multiple defect types is established.

[0057] The target detection network architecture of the neural network structure model in step S2 includes a backbone network and a feature fusion network. The backbone network adopts a target detection algorithm based on a one-stage network, and the feature fusion network consists of three optimization modules: a content-aware feature recombination framework, a heavy-parameter generalized feature pyramid network, and a strong intersection-over-union loss function.

[0058] Reference Figure 2 The processing flow of the target detection algorithm based on the one-stage network in the backbone network is as follows: preprocess the input image of 640*640 size, configure it as different scaling and filling modes according to the parameters, and then perform convolution, maximum pooling and other operations in the backbone network. Use two 3*3 convolutions in succession to directly reduce the resolution by 4 times. When the receptive field is small, use one layer of convolution twice in succession and then downsample. Through more cross-layer connections (directly transfer features between different layers to improve the performance of the model), enrich the gradient flow of the model, and adjust the number of channels for different scale models. At the same time, remove two convolution connection layers, and use the mainstream decoupling head structure at the output end.

[0059] Feature maps of different scales use different numbers of channels to meet different feature representation requirements. In the YOLOv8 network, the number of channels can be adjusted by changing the number of filters in the convolution layer. The two convolution connection layers removed are removed in the process of "using two 3*3 convolutions to directly reduce the resolution by 4 times" and "using convolution when the receptive field is small".

[0060] Reference Figure 3 ,The content-aware feature reconstruction framework in the feature fusion network uses a lightweight network to upsample the CARAFE operator to design a one-stage network model. ,The calculation process of the CARAFE operator includes : ,feature reconstruction based on the content of each target position, upsampling the low-resolution feature map to generate a feature map of the same size as the high-resolution feature map for interaction, and obtaining the context information required for reconstruction; ,then reconstructing the predicted features, and performing the corresponding convolution ,operation on the interacted feature map to obtain the final upsampling ,result.

[0061] Reference Figure 4 The heavy parameter generalized feature pyramid network in the feature fusion network is improved on the basis of GFPN, replacing the fusion module based on the use of skip layers and cross-scale connections; the skip layer connection reduces the vanishing gradient during the back propagation process and utilizes two feature connection methods; cross-scale connections are used to overcome multi-scale changes, and bilinear interpolation and maximum pooling are used as upsampling and downsampling functions respectively.

[0062] The robust intersection-over-union loss function in the feature fusion network combines an object size adaptive penalty factor and a gradient adjustment function based on the quality of the anchor box.

[0063] Strong Intersection-over-Union Loss (GIoU Loss)

[0064]

[0065] L GloU Loss 1-L GIoU

[0066] Gradient adjustment function based on anchor box quality:

[0067] L reg =∑ i α i ·SmoothL1(t i -t i * )

[0068] α i =exp(-β·IoU(A i ,B i ))

[0069] SmoothL1 is a commonly used regression loss function. The complete loss function expression is:

[0070] L=L GIoU +L reg

[0071] In step S3, the neural network structure model is trained by using the established 3D printed intestinal fistula patch defect data set, and the trained neural network structure model is used as a defect recognition model;

[0072] In step S4, the defects in the intestinal fistula patch monitoring image are classified and located through the defect recognition model, so as to realize real-time monitoring of defects in the intestinal fistula patch 3D printing process.

[0073] Among them, the backbone network is used to extract the features of the input image; YOLOv8 uses a convolutional neural network (such as CSPDarknet53, EfficientNet, etc.) as its backbone network. The main responsibility of the backbone network is to extract low-level and high-level features in the image through a series of convolutional layers, pooling layers, and activation functions.

[0074] The feature fusion network is used to combine multi-scale features from the backbone network. YOLOv8 uses Path Aggregation Network (PANet) or FPN (Feature Pyramid Network) to achieve this function. Feature fusion is to fuse feature maps from different layers through upsampling and downsampling operations to generate multi-scale feature maps; cross-layer connection: It is to enhance feature fusion through horizontal and vertical connections, so that the network can better capture the details and context information of the target.

[0075] Finally, the final target detection and classification are obtained through the prediction head. The classification and positioning in this embodiment: each pixel on each feature map predicts multiple anchor boxes, and each anchor box outputs a category probability and four bounding box coordinates (center point coordinates, width and height). Anchor box mechanism adopted: YOLOv8 uses an anchor box mechanism to capture targets of different scales and aspect ratios through a predefined set of anchor boxes.

[0076] Based on the above scheme, in order to verify the effectiveness and practical effect of the method of the present invention, the five-axis additive manufacturing defect image detection test results of 3D printed intestinal fistula patches were compared in this embodiment. Figure 8 As shown in the figure, we mainly focus on two parameter indicators: accuracy and recognition speed. Accuracy is measured by mAP value, and recognition speed is reflected by processing frame rate. AP value is the area enclosed by the recall rate and accuracy curve and the coordinate axis under different confidence values, and mAP value is the average AP value;

[0077] At the same time, in order to ensure the accuracy of the comparative test, the parameters of the network training performed in this embodiment are kept consistent, the basic learning rate is set to 0.0001, the number of iterations is set to 300 times, and the batch size of the pictures sent to the neural network is set to 8. The input image is unified in RGB format of 640*640 pixels;

[0078] The three optimization modules were integrated with each other to form a new network structure model to train the collected and processed data set. The basic network framework model was compared through ablation experiments, and the performance effects under three different defect conditions were compared. A total of 9 experiments were conducted in Table 1. Except for Experiment 2, the mAP values ​​of two of the three defects were lower than those of the original model, and the other 8 experiments achieved a certain improvement. At the same time, it can be clearly seen that the data results obtained by combining the two modules have been improved to a certain extent compared with those using only one improved module. In Experiment 5, CARAFE and PIoU were integrated, and the detection effect for interlaminar stripping reached 91.5%, which was the best effect among the 9 experiments;

[0079] Table 1

[0080]

[0081] Finally, the three improved modules were integrated together, and the detection results for blockage and unformed defects achieved the best results in the ablation test. The detection results for interlaminar stripping also reached 91.3%, which was only 0.2% lower than the best result. Fig. 9 As shown;

[0082] For a test set of 1000 printing process defects and fault images, the defect recognition model proposed in this invention is used for recognition and detection. The results are as follows: Fig.10 As shown, the first figure on the right indicates that the probability of unformed is 0.89; the second figure on the right indicates that the probability of interlayer delamination is 0.89; and the last figure on the right indicates that the probability of nozzle clogging is 0.85.

Claims

1. A real-time defect monitoring method for 3D printed intestinal fistula patches based on machine vision, characterized in that: The steps include: S1: Establish a 3D printed intestinal fistula patch defect dataset including various defect types; S2: Based on the improved YOLOv8 neural network, a neural network structure model is established; S3: The neural network structure model is trained by using the established 3D printed intestinal fistula patch defect dataset, and the trained neural network structure model is used as a defect recognition model; S4: The defects in the intestinal fistula patch monitoring image are classified and located through the defect recognition model, so as to realize the real-time monitoring of defects in the intestinal fistula patch 3D printing process.

2. The method for real-time defect monitoring of a 3D printed intestinal fistula patch based on machine vision according to claim 1, characterized in that: The method for establishing the 3D printed intestinal fistula patch defect data set in step S1 includes: A1: Set up the printing models of defective and non-defective 3D printed fistula patches respectively, collect video of the entire printing process, and collect video images of the 3D printed non-defective fistula patches and defective fistula patches respectively; A2: Image feature preprocessing and image data enhancement of 3D printed intestinal fistula patch defect images; A3: Use the labelimg data labeling tool to label the defective parts of the collected inspection images.

3. The method for real-time defect monitoring of a 3D printed intestinal fistula patch based on machine vision according to claim 2, characterized in that: The method for preprocessing the image features in step A2 is: converting the collected color image into a grayscale image, adding salt and pepper noise, and then performing corrosion and expansion operations on the image; The method of image data enhancement is to rotate, flip and scale the image at random angles, then apply noise and occlusion operations, and finally use mosaic data enhancement.

4. The method for real-time defect monitoring of a 3D printed intestinal fistula patch based on machine vision according to claim 2, characterized in that: The defect types of the defective intestinal fistula patch video image in step A1 include interlayer peeling, unformed printing, and nozzle clogging.

5. The method for real-time defect monitoring of a 3D printed intestinal fistula patch based on machine vision according to claim 1, characterized in that: The target detection network architecture of the neural network structure model in step S2 includes a backbone network and a feature fusion network. The backbone network adopts a target detection algorithm based on a one-stage network, and the feature fusion network is composed of three optimization modules: a content-aware feature reorganization framework, a heavy-parameter generalized feature pyramid network, and a strong intersection-over-union loss function.

6. The method for real-time defect monitoring of a 3D printed intestinal fistula patch based on machine vision according to claim 5, characterized in that: The processing flow of the target detection algorithm based on the one-stage network in the backbone network is as follows: preprocess the input image, configure it to different scaling and filling modes according to the parameters, and then perform convolution and maximum pooling operations in the backbone network. Two 3*3 convolutions are used continuously to directly reduce the resolution by 4 times. When the receptive field is small, downsampling is performed after using one layer of convolution twice in a row. The gradient flow of the model is enriched through more cross-layer connections with more scores, and different numbers of channels are adjusted for models of different scales. At the same time, two convolution connection layers are removed, and the mainstream decoupling head structure is used at the output end.

7. The method for real-time defect monitoring of a 3D printed intestinal fistula patch based on machine vision according to claim 5, characterized in that: The content-aware feature reconstruction framework in the feature fusion network uses a lightweight network to perform upsampling CARAFE operators to design a one-stage network model. The calculation process of the CARAFE operator includes: according to the feature reconstruction of the content of each target position, the low-resolution feature map is upsampled to generate a feature map of the same size as the high-resolution feature map for interaction to obtain the context information required for reconstruction; then the predicted features are used for reconstruction, and the interactive feature map is subjected to corresponding convolution operations to obtain the final upsampling result.

8. The method for real-time defect monitoring of 3D printed intestinal fistula patch based on machine vision according to claim 5, characterized in that: The heavy parameter generalized feature pyramid network in the feature fusion network is improved on the basis of GFPN, and replaces the fusion module on the basis of using skip layer and cross-scale connection; the skip layer connection reduces the disappearance of gradient in the back propagation process and uses two feature connection methods; Cross-scale connections are used to overcome multi-scale changes, and bilinear interpolation and maximum pooling are used as upsampling and downsampling functions, respectively.

9. The method for real-time defect monitoring of a 3D printed intestinal fistula patch based on machine vision according to claim 5, characterized in that: The robust IoU loss function in the feature fusion network combines an object size adaptive penalty factor and a gradient adjustment function based on the quality of the anchor box.

10. The method for real-time defect monitoring of 3D printed intestinal fistula patch based on machine vision according to claim 9, characterized in that: The strong intersection-over-union loss function is: L GIoU Loss =1-L GIoU Gradient adjustment function based on anchor box quality: L reg =∑ i α i ·SmoothL1(t i -t i * ) a i =exp(-β·IoU(A i ,B i )) SmoothL1 is a regression loss function. The complete loss function expression is: L=L GIoU +L reg .

Images