Full-automatic three-dimensional composite material CT image segmentation and reconstruction method

Abstract

The invention provides a full-automatic three-dimensional woven composite material CT (Computed Tomography) image segmentation and reconstruction method. According to the method, a yarn segmentation task of a three-dimensional woven composite material image is redesigned, and through combination of a semantic segmentation-based technology and a gradient flow-based instance segmentation technology, boundaries of adhered yarns can be accurately segmented on a low-contrast CT image. Finally, a test result shows that the method can effectively segment yarns with different shapes and sizes, yarn segmentation with clear boundaries is realized in a segmented sample, and the reliability of yarn segmentation is remarkably improved.

Description

Technical Field

[0001] The invention belongs to the technical field of three-dimensional composite woven material segmentation and reconstruction, and in particular relates to a method for fully automatic three-dimensional composite material CT image segmentation and reconstruction. Background Art

[0002] The physical properties of woven materials (such as mechanical strength and durability) are closely related to their microstructure. Establishing a detailed microscopic model of textiles can more accurately predict their performance in practical applications and is a hot topic in current research. Modeling of composite materials can be divided into two categories: predictive modeling and descriptive modeling. Predictive modeling mainly relies on numerical methods and aims to predict the morphology of textile materials under specific conditions. Descriptive modeling is based on high-resolution X-ray computed tomography (μ-CT) images, and virtual models are derived from the image volume of existing materials through image segmentation technology. Micro-CT scanning is currently the mainstream choice as a technology that can directly obtain cross-sectional views of three-dimensional fabric structures in composite materials. Once the original micro-CT images of different cross-sections of the scanned fabric are obtained, the yarn and matrix can be easily distinguished by grayscale, but the segmentation of yarns with similar grayscale is not so easy and requires the use of other more detailed image features. Traditional image segmentation methods for woven materials usually rely on extracting grayscale information at the pixel level of the image, and then use simple threshold segmentation, clustering, morphological operations or variational methods to distinguish different regions such as yarns, matrix and pores. These methods are only suitable for processing high-contrast images and have high computational complexity.

[0003] In recent years, thanks to the rapid development of artificial intelligence (AI), especially convolutional neural networks (CNN), some research teams have begun to use CNN-based image segmentation algorithms to automatically separate individual yarn cross-sections in microCT slices of woven composites, which has achieved significant performance improvements over traditional methods. Regarding the analysis of CT tomography, we will distinguish two types of image segmentation tasks. First, semantic segmentation requires accurate finding of object boundaries and aims to divide the pixels (or voxels) of the image into several categories (e.g., yarn, matrix, pores). All regions belonging to the same category are labeled with the same label, but different objects in these regions are not distinguished. Instance segmentation goes a step further and requires identifying and labeling each independent object in the same category, for example, distinguishing each yarn in the image instead of just labeling all yarns as "yarn". Based on semantic segmentation, [1] combined variational models and deep learning algorithms to extract and reconstruct the warp and weft yarns of woven composites. [2] ResNet is used to segment the two-dimensional slices of CT images, compose three-dimensional yarn bodies, and then post-process them into finite element (FE) meshes. Based on instance segmentation, [3] proposed ResL-U-Net to process low-contrast CT images, and combined it with a concave point detection algorithm and an improved watershed algorithm to solve the problem of tight fiber bundle connection positions and avoid mutual penetration between yarns. [4] The virtual fabric composite model generated by finite element simulation was converted into a large number of "pseudo" CT scan slices containing yarn cross-sectional profiles, and these "pseudo" scan slices were used to train Mask R-CNN to segment individual yarn cross sections in real CT scan slices of fabric composite samples. Although researchers have begun to apply CNN-based image segmentation algorithms to the mesoscopic geometric reconstruction of fabric composites, most existing methods have limited effectiveness in dealing with low contrast and tightly adhered yarns. Recently, instance segmentation methods based on gradient flow have gradually attracted attention. This method has shown excellent performance in the field of cell segmentation [5], can effectively segment overlapping cells, and performs well in dealing with complex cell morphology. However, in the segmentation task of woven materials, methods and research based on gradient flow are still lacking. In view of the problem of yarn adhesion in woven materials, there is an urgent need to invent a robust algorithm that can adapt to various types of woven materials and effectively segment tightly adhered yarns in low-contrast CT images.

[0004]

References

[0005] [1]Sinchuk, Yuriy, et al. "Variational and deep learning segmentation of very-low-contrast X-ray computed tomography images of carbon / epoxy woven composites." Materials 13.4 (2020): 936.

[0006] [2]Ali, Muhammad A., et al. "Efficient processing of μCT images using deep learning tools for generating digital material twins of woven fabrics." Composites Science and Technology 217 (2022): 109091.

[0007] [3]Song, Yingying, et al. "Material twins generation of woven polymer composites based on ResL-U-Net convolutional neural networks." Composite Structures 307 (2023): 116672.

[0008] [4]Mendoza, Arturo, et al. "Descriptive modeling of textiles using FE simulations and deep learning." Composites Science and Technology 213 (2021): 108897.

[0009] [5]Stringer, Carsen, et al. "Cellpose: a generalist algorithm for cellular segmentation." Nature methods 18, no. 1 (2021): 100-106. Summary of the Invention

[0010] Purpose of the invention: Aiming at the difficulties faced by image segmentation in descriptive modeling of woven material micro-models, including multi-type materials, low contrast of CT scan slice images, tightly adhered yarns, etc. In this invention, the woven material segmentation task is redesigned, and a method for fully automatic 3D composite material CT image segmentation and reconstruction is proposed. It aims to combine semantic segmentation methods with gradient flow-based instance segmentation technology to accurately identify and separate yarns in multi-type materials, low-contrast images, and micro-CT slices with yarn adhesion, thereby optimizing the subsequent 3D modeling process.

[0011] Technical solution: The present invention proposes a method for fully automatic three-dimensional composite material CT image segmentation and reconstruction, which includes the following 7 steps:

[0012] a) Literature research: In-depth collection of literature and invention patents in related fields, including but not limited to the research status of woven material image segmentation and reconstruction; the research status of semantic segmentation and instance segmentation methods.

[0013] b) Data collection: In cooperation with engineers from Nanjing Fiberglass Research Institute, data of 9 single-cell woven materials of various types were collected, using three parameter scans.

[0014] c) Construction of semantic segmentation model training set: 600 slices of different types of woven materials were selected for annotation, and the annotated data was expanded to 2,400. The collected annotated slice data set was randomly divided into training set and test set according to a certain ratio, and corresponding preprocessing was performed;

[0015] d) Model building and training: A three-stage segmentation framework is constructed, including the semantic segmentation stage, the instance segmentation pre-training stage, and the instance segmentation fine-tuning stage. In stage 1, a basic semantic segmentation model is trained using the training set constructed in step c) to perform preliminary semantic segmentation of various types of woven materials (warp and weft) in CT images; in stage 2, based on the semantic segmentation results of yarns, an input dataset of the instance segmentation pre-training model is constructed, and a gradient flow-based segmentation method is used on the dataset for pre-training; in stage 3, manual modification is performed for the yarn adhesion in the semantic segmentation results; based on the manual modification results, the instance segmentation pre-training model is used to fine-tune the dataset of similar types of woven materials; a post-processing module is designed to correct the instance segmentation results.

[0016] e) After training, the network model obtained in d) is tested on the test set and non-quantitatively evaluated.

[0017] f) 3D reconstruction: The 2D slice segmentation results of the cross-section of the warp and weft yarns are reconstructed into 3D visualization.

[0018] Specifically, the CT images in step b) are high-resolution X-ray computed tomography scans, and the size of each two-dimensional image varies from about 2300×2300. These high-resolution CT images provide detailed information for subsequent yarn segmentation.

[0019] Specifically, in step c), the collected data is randomly divided into a training set and a validation set in a ratio of 4:1. The preprocessing process of the data set is implemented as follows: First, the foreground and background are distinguished by the threshold segmentation method, and the foreground part of the image is cropped. Then, histogram equalization is applied to enhance the contrast of the image. In addition, random rotation, horizontal flip, vertical flip and grayscale change operations are performed to increase the diversity of the data. Finally, the images in the training set are resized to 512×512 to improve training efficiency.

[0020] Specifically, the three-stage segmentation framework for woven material image yarns described in step d) is divided into three main parts: a semantic segmentation stage, an instance segmentation pre-training stage, and an instance segmentation fine-tuning stage. The semantic segmentation network used in stage 1 includes but is not limited to the nnUNet network, and the loss function for training the nnUNet network includes but is not limited to the Dice+ cross entropy (CE) loss function, which is specifically calculated as follows:

[0021]

[0022] L=L Dice +L CE

[0023] Where L Dice is the Dice loss function term, L CE is the CE loss function term, L is the overall network loss function, N is the total number of voxels, s i and g i are the segmentation results and labels at voxel point i, p i For i The SoftMax function output relative to all voxels.

[0024] Specifically, in step d) of stage 2, the input data set of the instance segmentation pre-training model is constructed by the semantic segmentation result of the yarn, that is, the semantic segmentation result of the warp yarn is used as the original image and label data at the same time, and the pre-training allows the model to learn more yarn shapes to avoid segmentation errors or over-segmentation problems; the gradient flow-based segmentation method includes but is not limited to the Cellpose framework, specifically, the network predicts the final segmentation label by learning the direction field and probability map of each pixel to obtain the flow field of the yarn, the direction field represents the vector of each pixel pointing to the center of the yarn, the probability map represents the probability of each pixel belonging to the yarn, the flow field is used to describe the shrinkage of the pixel from the yarn boundary to the center, and the vector field guides the pixel to the global yarn center through the Euler integral, thereby segmenting the yarn according to the point where the pixel is gathered. The network architecture includes but is not limited to the U-Net architecture, the input is an image dataset constructed based on the semantic segmentation result, the output is the direction field and probability map of each pixel, and the training loss function of the network includes but is not limited to the L2 loss for the direction field and the cross entropy (CE) loss for the probability map. The L2 loss is specifically calculated as follows:

[0025]

[0026] L=L2+L CE

[0027] Where N is the total number of voxels, v i and t i are the predicted directional gradient and the true directional gradient at voxel point i, respectively.

[0028] Specifically, the manually modified annotations in step d) of stage 3 are mainly focused on the situation of yarn adhesion after semantic segmentation. Similar types of woven materials refer to materials with similar yarn cross-sectional areas and yarn shapes. During fine-tuning, the semantic segmentation results are used as the original images, and the manually modified results are used as labels to input the instance segmentation pre-training model. The post-processing module is used to correct the yarns that are not correctly identified in the instance segmentation, by differentiating the semantic segmentation results from the instance segmentation results, and detecting the unidentified yarns through the threshold method, and adding them to the instance segmentation results.

[0029] Specifically, the test process described in step e) is implemented as follows: the data in the test set are passed through the three stages in d) one by one. First, semantic segmentation is performed through the semantic segmentation module, and the result after semantic segmentation is sent to the instance segmentation module for instance segmentation. The final result is corrected by the post-processing module, and the weft yarn semantic segmentation result is supplemented. In this way, the obtained prediction results are comprehensively analyzed to evaluate the performance and effect of the entire segmentation framework.

[0030] Compared with the technology reported in the existing literature, the present invention has achieved the following significant progress: the present invention redesigns the yarn segmentation task of woven material images by integrating the instance segmentation method of gradient flow and the semantic segmentation technology, and proposes a method for fully automatic three-dimensional composite material CT image segmentation and reconstruction, aiming to deal with the problems of multi-type woven materials, low-contrast images and tight adhesion of yarns. The present invention uses X-ray computed tomography (CT) images and corresponding semantic segmentation results, integrates the gradient flow-based method, and uses pre-training, fine-tuning and segmentation result repair technology to achieve significant results for the yarn segmentation task of woven material images. For multiple types of materials, in the semantic segmentation stage, our method as a whole achieved a Dice score of 0.95. After instance segmentation processing, compared with the results of semantic segmentation, we can effectively solve the problem of tight adhesion of yarns of multiple material types. This part of the results shows high interpretability and reliability after non-quantitative analysis. Through the method of the present application, the limitations of existing methods in the case of multiple types of woven materials, low image contrast and tight adhesion of yarns are successfully overcome, providing a more robust and efficient solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is the overall framework diagram of the method proposed in the present invention;

[0032] Figure 2 An example of application of the gradient flow method in the method framework proposed in the present invention in yarn segmentation;

[0033] Figure 3 Visualization of the semantic segmentation and instance segmentation results of warp yarns in the method framework proposed by the present invention;

[0034] Figure 4 Visualization of the segmentation results of multiple types of woven materials in the framework of the method proposed in the present invention;

[0035] Figure 5 Visualize the segmentation results by 3D reconstruction. DETAILED DESCRIPTION

[0036] The objectives, technical solutions and advantages of the present invention are described in detail below in conjunction with the accompanying drawings and specific embodiments of the present invention.

[0037] The present invention proposes a method for fully automatic 3D composite material CT image segmentation and reconstruction. Specifically, by integrating the instance segmentation method of gradient flow and the semantic segmentation technology, the yarn segmentation task of 3D woven materials is redesigned, which can effectively solve the problems of diverse material types, low image contrast, and tight adhesion of yarns in the yarn segmentation task of woven materials. The present invention specifically includes the following steps:

[0038] a) Data collection: In cooperation with engineers from Nanjing Fiberglass Research Institute, we collected 9 unit cells of different types, using three parameter scans respectively;

[0039] b) Construction of semantic segmentation model training set: 600 slices of woven materials with different structures were selected, and the size of each two-dimensional image varied from about 2300×2300 for preprocessing, that is, firstly, the foreground and background were distinguished by the threshold segmentation method, and the foreground part of the image was cropped. Then, histogram equalization was applied to enhance the contrast of the image. Next, these 600 images were annotated, and the annotated data was expanded to 2400, that is, the diversity of the data was increased by random rotation, horizontal flipping, vertical flipping and grayscale change operations, and then the images in the training set were adjusted to 512×512 to improve the training efficiency. Finally, the collected annotated slice data set was randomly divided into training set and test set in a ratio of 4:1.

[0040] c) Model building and training: Construct a three-stage segmentation framework, as shown in the flowchart below Figure 1 As shown, it includes a semantic segmentation stage, an instance segmentation pre-training stage and an instance segmentation fine-tuning stage. In the first stage, a basic semantic segmentation model is trained using the training set constructed in step b) to perform preliminary semantic segmentation on various types of woven materials (warp and weft) in CT images; in the second stage, based on the semantic segmentation results of the yarns, an input data set for the instance segmentation pre-training model is constructed, that is, the warp semantic segmentation results are used as both the original image and the label data, and a gradient flow-based segmentation method is used on the data set for pre-training, so that the model can extract more features of the shaped yarns and avoid segmentation errors and over-segmentation. The gradient flow-based segmentation method is shown in Figure 2 As shown in the figure, the Cellpose framework is used. Specifically, the neural network learns the direction field (Vertical flow and Horizontal flow) and probability map (Fiber probability) of each pixel to obtain the flow field (Flow Field) of the yarn.

[0041] To predict the final segmentation label, the final segmentation label is achieved through flow field tracking, that is, the vector field guides the pixel to the global yarn center through Euler integration, so as to segment the yarn according to the point where the pixel is gathered. The entire network architecture adopts the U-Net model, the input is the CT image dataset constructed from the semantic segmentation result, and the output is the direction field and probability map of each pixel. The L2 loss for the direction field and the cross entropy (CE) loss for the probability map are used in the network training process to optimize the segmentation accuracy and effect of the model. At the same time, both models use the Adam algorithm to optimize the network parameters. The hyperparameters in the training process are shown in Table 1. In the third stage, manual modification is performed for the situation of yarn adhesion in the semantic segmentation results. The semantic segmentation result is used as the original image, and the manual modification result is used as the label input. The instance segmentation pre-training model is fine-tuned on the dataset of similar types of woven materials. Similar types of woven materials refer to materials with similar yarn cross-sectional areas and yarn shapes.

[0042] Table 1 Model training hyperparameters

[0043]

[0044] d) After the training is completed, the data in the test set are passed through the three stages in c) one by one. First, semantic segmentation is performed through the semantic segmentation module, and the results of warp semantic segmentation are sent to the instance segmentation module for instance segmentation. The final result is corrected by the post-processing module, that is, the semantic segmentation result is subtracted from the instance segmentation result, and the threshold method is used to identify and supplement the individual unidentified yarns in the instance segmentation, and then the weft semantic segmentation result is supplemented.

[0045] e) 3D reconstruction: The 2D slice segmentation results of the cross-section of the warp and weft yarns are reconstructed into 3D visualization.

[0046] In order to more clearly illustrate the feasibility and superiority of the three-stage segmentation framework proposed in this invention, a visualization diagram of the semantic segmentation results and the instance segmentation results is shown, as shown in Figure 3 As shown in the figure, the present invention can still achieve good segmentation effect when the image contrast is low and the yarns are closely adhered. Semantic segmentation can identify the yarn category, and instance segmentation can further distinguish each individual yarn.

[0047] In order to evaluate the semantic segmentation effect and robustness of the model, the Dice index performance of image segmentation and the changes in the loss function in different training rounds are shown, as shown in Table 2. Finally, the Dice index of semantic segmentation reached 0.95, indicating the effectiveness of the model in the segmentation of yarns of multiple types of woven materials.

[0048] Table 2 Semantic segmentation evaluation results

[0049] Epoch Dice Val_loss 2 0.7659 -0.6426 25 0.9152 -0.7859 50 0.9375 -0.8169 100 0.9464 -0.8414 200 0.9534 -0.8662 300 0.9588 -0.8722

[0050] To further illustrate the feasibility and superiority of the three-stage segmentation framework proposed in this paper for segmenting yarns of multiple types of woven materials, we show the semantic segmentation results on yarns of different shapes and sizes, the instance segmentation results, and the visualization of the results of only the warp yarns, such as Figure 4 As shown, the results show that instance segmentation can effectively distinguish different yarn individuals.

[0051] The present invention also shows the visualization result of 3D reconstruction after segmentation of a small piece of fiber material, such as Figure 5 As shown, the application potential of the method of the present invention in complex material structures is further verified.

Claims

1. A method for fully automatic three-dimensional composite material CT image segmentation and reconstruction, characterized in that: The steps include: a) performing preliminary semantic segmentation of yarns in various types of woven materials in CT images, wherein the yarns include warp yarns and weft yarns; b) Based on the semantic segmentation results of YARN, an input dataset of the instance segmentation pre-training model is constructed, and a gradient flow-based segmentation method is used for pre-training on the dataset; c) According to the semantic segmentation results of the woven material, the situation where yarn adhesion exists is selected for manual modification; d) Based on the manual modification results, the instance segmentation pre-trained model is fine-tuned on a dataset of similar types of woven materials; e) Design a post-processing module to correct the instance segmentation results.

2. The method for fully automatic 3D composite material CT image segmentation and reconstruction according to claim 1, characterized in that: The semantic segmentation network used in step (a) includes but is not limited to the nnUNet network, and the loss function for training the nnUNet network includes but is not limited to the Dice+cross entropy CE loss function, which is specifically calculated as follows: L=L Dice +L CE Where L Dice is the Dice loss function term, L CE is the CE loss function term, L is the overall network loss function, N is the total number of voxels, s i and g i are the segmentation results and labels at voxel point i, p i For i The SoftMax function output relative to all voxels.

3. The method for fully automatic 3D composite material CT image segmentation and reconstruction according to claim 1, characterized in that: In the step (b), the input data set of the instance segmentation pre-training model is constructed by the semantic segmentation result of the yarn, that is, the warp semantic segmentation result is used as the original image and label data at the same time; the gradient flow-based segmentation method includes but is not limited to the Cellpose framework, including the following strategies and variants thereof, specifically, the network predicts the final segmentation label by learning the direction field and probability map of each pixel to obtain the flow field of the yarn, the direction field represents the vector of each pixel pointing to the center of the yarn, the probability map represents the probability of each pixel belonging to the yarn, and the flow field is used to describe the shrinkage of the pixel from the boundary of the yarn to the center; the network architecture includes but is not limited to the U-Net architecture, the input is an image data set constructed based on the semantic segmentation result, the output is the direction field and probability map of each pixel, and the training loss function of the network includes but is not limited to the L2 loss for the direction field and the cross entropy CE loss for the probability map; the L2 loss is specifically calculated as follows: L=L2+L CE Where N is the total number of voxels, v i and t i are the predicted directional gradient and the true directional gradient at voxel point i, respectively, CE is the CE loss function term, and L is the overall loss function of the network.

4. The method for fully automatic 3D composite material CT image segmentation and reconstruction according to claim 1, characterized in that: The manually modified annotations in step (c) focus on the situation of yarn adhesion after semantic segmentation.

5. The method for fully automatic 3D composite material CT image segmentation and reconstruction according to claim 1, characterized in that: In the step (d), similar types of woven materials refer to materials with similar yarn cross-sectional areas and yarn shapes. During fine-tuning, the semantic segmentation result is used as the original image, and the manually modified result is used as a label to input into the instance segmentation pre-training model.

6. The method for fully automatic 3D composite material CT image segmentation and reconstruction according to claim 1, characterized in that: In the step (e), the post-processing module is used to correct the warp yarns that are not correctly identified in the instance segmentation by differentiating the semantic segmentation results of the warp yarns from the instance segmentation results, detecting the unidentified yarns through a threshold method, and adding them to the instance segmentation results, while also supplementing the semantic segmentation results of the weft yarns.

Images