Implementation method and system based on industrial CT image three-dimensional reconstruction

Abstract

The invention relates to the technical field of three-dimensional reconstruction, in particular to an industrial CT image three-dimensional reconstruction-based implementation method and system, and the method comprises the following steps: CT image preprocessing, edge detection, image frame insertion, two-dimensional image coordinate three-dimensional processing, and mesh outer surface 3D visualization. The three-dimensional reconstruction implementation method and system based on the industrial CT image have the beneficial effects that a series of processing is carried out on the CT section scanning data of a plurality of objects, and the three-dimensional grid model of the objects is finally restored and generated and is used for subsequent application such as industrial detection and recognition. Compared with a traditional medical reconstruction method, the reconstruction efficiency is higher, and the resource demand of a machine is reduced by more than 50%; the middle process of the image is estimated through frame insertion based on motion estimation through deep learning, and compared with a traditional mathematical algorithm, the reconstruction precision of the three-dimensional model can be improved, so that the precision requirement of subsequent industrial detection is met.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and particularly to a method and system for realizing three-dimensional reconstruction based on industrial CT images. Background Art

[0002] CT three-dimensional reconstruction: CT three-dimensional reconstruction technology is an advanced imaging technology. It uses computer graphics and image processing technologies to reconstruct a sequence of two-dimensional tomographic images obtained by CT scanning into three-dimensional image data in a computer and vividly display a three-dimensional stereoscopic view on the screen. This technology has a wide range of applications in the medical field, including but not limited to helping doctors better understand the nature of lesions and the three-dimensional structural relationship of the surrounding tissues, etc. At present, it also has a very broad application prospect in the industrial field and can play an important role in multiple links such as the design, manufacturing, inspection, and quality control of industrial products.

[0003] RIFE (Real-Time Intermediate Flow Estimation for Video Frame Interpolation) is an advanced video frame interpolation algorithm. This algorithm uses deep learning technology and can generate realistic intermediate frames between two video frames, thereby significantly increasing the frame rate of the video and making the video playback more smooth and natural. The core principle of RIFE is to learn the optical flow field between two adjacent frames through a convolutional neural network (CNN), that is, the movement trajectory of pixel points between different frames. Then, using the information of the optical flow field and the original frames, intermediate frames are generated. Its core idea is to estimate the movement information of pixel points to achieve smooth transition between frames, thereby generating realistic intermediate frames.

[0004] Poisson reconstruction is a method used in computer graphics to generate smooth surfaces. It is based on the Poisson equation to solve the surface reconstruction problem. The Poisson equation is a partial differential equation, usually used to describe diffusion processes in physical phenomena, such as heat conduction, current flow, etc. The key advantage of Poisson reconstruction is that it can generate very smooth surfaces while maintaining the accuracy of data points. It is particularly effective in dealing with noisy data and filling missing data regions. In addition, Poisson reconstruction can also handle complex topological structures, such as multiply connected domains. Poisson reconstruction has applications in many fields, including computer-aided design (CAD), medical imaging, terrain generation, etc. It is a powerful tool that can help us recover continuous geometric shapes from discrete data.

[0005] At present, there is no unified industry standard or unified methodology for CT three-dimensional reconstruction in the industrial field. The reconstructed results are uneven, which makes it difficult to meet the increasingly urgent needs of industrial inspection and defect analysis. Therefore, how to achieve a method for industrial CT image three-dimensional reconstruction with high adaptability, high reconstruction efficiency and high accuracy has become one of the problems that must be solved in the current industrial CT image three-dimensional reconstruction. Summary of the invention

[0006] The purpose of the present invention is to provide a method and system for realizing three-dimensional reconstruction based on industrial CT images to solve the problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solution: a method for implementing three-dimensional reconstruction based on industrial CT images, the method comprising the following steps:

[0008] CT image preprocessing: convert the dat format image obtained by CT scanning into JPEG format and grayscale it, then use the mean filter to denoise it, and then use the bicubic interpolation algorithm to improve the image resolution;

[0009] Edge detection: perform edge detection on the preprocessed image, use the Canny algorithm or binarization processing to obtain the edge point coordinates, and select the appropriate method based on the image features;

[0010] Image interpolation, based on deep learning interpolation algorithms such as the FlowNet network, calculates the optical flow information between images to generate intermediate frames. For discrete point images that do not have motion coherence, the bicubic interpolation algorithm is used;

[0011] Convert the 2D image coordinates to 3D coordinates, convert the interpolated image coordinates to 3D coordinates, determine the z-axis coordinates of each pixel point according to the scanning speed and image sequence, and save the color information;

[0012] 3D visualization of the mesh outer surface, restore the mesh from the 3D point cloud through the Poisson reconstruction algorithm, use the Taubin smoothing algorithm to smooth the surface, and map the color information to the mesh surface based on the KD search tree, and finally generate and display a colored 3D mesh model.

[0013] Preferably, in the CT image preprocessing step, the bicubic interpolation algorithm selects 16 nearest neighbor pixels around the target pixel for horizontal and vertical interpolation to improve image accuracy.

[0014] Preferably, in the edge detection step, the Canny algorithm or binarization processing is selected according to the degree of prominence of the image edge to ensure accurate acquisition of the edge point coordinates.

[0015] Preferably, in the image interpolation step, the FlowNet network is used to estimate the optical flow between consecutive frames, and intermediate interpolation frames are generated based on the optical flow information to meet the requirements of restoring a wide range of three-dimensional models of ultrasonic images.

[0016] Preferably, in the 3D visualization step of the grid outer surface, it also includes mapping the color information on the point cloud to each triangular face of the 3D grid model through a KD search tree, and separating the color material and the model body of the model through UV unwrapping and a model baking engine to achieve high-precision three-dimensional visualization.

[0017] An implementation system for three-dimensional reconstruction based on industrial CT images, the system includes:

[0018] A CT image preprocessing module, used to convert the CT-scanned dat format image into JPEG format, perform grayscale processing and denoising, and then improve the image resolution through a bicubic interpolation algorithm;

[0019] An image edge detection module, used to perform edge detection on the preprocessed image, and select to use the Canny algorithm or binary processing according to the obviousness of the image edge to obtain the edge point coordinates;

[0020] An image interpolation module, including a traditional interpolation algorithm based on motion estimation and a deep learning interpolation algorithm, such as using the FlowNet network to calculate the optical flow and generate intermediate interpolation frames to enrich the object surface information in the image sequence;

[0021] A two-dimensional image coordinate three-dimensionalization module, used to convert the coordinates of the interpolated image into three-dimensional coordinates, and calculate the z-axis coordinates of each image edge pixel according to the scanning speed parameter to generate three-dimensional point cloud data;

[0022] A three-dimensional grid reconstruction and visualization module, including a grid body recovery algorithm based on Poisson reconstruction, used to recover the grid body from the three-dimensional point cloud, perform surface smoothing through the Taubin smoothing algorithm, and finally through color mapping, UV unwrapping and model baking, present the three-dimensional grid model and its color material on a 3D visualization platform, or combine the grid with the point cloud for high-precision visualization.

[0023] Preferably, in the CT image preprocessing module, the bicubic interpolation algorithm performs interpolation in the horizontal and vertical directions by selecting 16 nearest neighbor pixels around the target pixel to improve the calculation efficiency while ensuring accuracy.

[0024] Preferably, in the image edge detection module, the choice of the edge detection algorithm is based on the clarity of the image edge. For images with obvious edges, the Canny algorithm is used, and for images with unclear edges, binary processing is used.

[0025] Preferably, the deep learning frame interpolation algorithm in the image frame interpolation module calculates the optical flow between consecutive frames using the FlowNet network and generates intermediate interpolated frames based on this optical flow information to meet the wide range of requirements for 3D reconstruction of ultrasonic images.

[0026] Preferably, in the 3D grid reconstruction and visualization module, for 3D models with a large number of discrete points, a grid point cloud fitting algorithm is adopted. Through the KD search tree and the nearest neighbor search algorithm, the cropped point cloud is fitted with the grid body to accurately restore the physical characteristics of the detected object.

[0027] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0028] The implementation method and system for 3D reconstruction based on industrial CT images proposed by the present invention perform a series of processes on the CT sectional scan data of several objects, and finally restore and generate a 3D grid model of the object for subsequent applications such as industrial inspection and recognition. In terms of reconstruction efficiency, it is higher than that of traditional medical reconstruction methods, and the resource demand for the machine is reduced by more than 50%. By estimating the intermediate process of the image through deep learning-based motion estimation frame interpolation, the reconstruction accuracy of the 3D model can be improved compared with traditional mathematical algorithms to meet the accuracy requirements of subsequent industrial inspections. Description of the Drawings

[0029] Figure 1 It is a flowchart of the method of the present invention. Detailed Embodiments

[0030] In order to clearly and completely describe the objectives, technical solutions of the present invention and make the advantages more clearly understood, the following further details the embodiments of the present invention with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are some embodiments of the present invention, rather than all embodiments, and are only used to explain the embodiments of the present invention, not to limit the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0031] Embodiment 1. Please refer to Figure 1 , the present invention provides a technical solution: an implementation method for 3D reconstruction based on industrial CT images, the method comprising the following steps:

[0032] 1. CT Image Preprocessing

[0033] First, optimize the detected image, mainly using the preprocessing means in the image target detection and recognition stage. During the preprocessing process, first convert the dat format image scanned by CT into a JPEG format image, then grayscale the image, and grayscale it according to the image features by selecting one of the R, G, and B channels. Then use a mean filter to denoise the grayscale image.

[0034] After image denoising, in order to improve the image accuracy without losing the original information of the image, bicubic interpolation is used to process the image pixels. First, according to the position of each pixel in the target image, its corresponding position in the original low-resolution image is found. Then, 16 nearest neighbor pixels around the target pixel are selected, including the target pixel itself and its surrounding neighbor pixels. Next, the bicubic interpolation algorithm calculates the interpolation result of the target pixel based on the gray values of these 16 nearest neighbor pixels. This calculation process includes two steps: First, horizontal interpolation is performed on the row where the target pixel is located to obtain a temporary interpolation result; then, vertical interpolation is performed on the column where the temporary interpolation result is located to obtain the final interpolation result. By repeating the above interpolation process for each pixel in the target image, the calculation efficiency is greatly improved while ensuring the accuracy, and a high-resolution image is generated.

[0035] 2. Image edge detection:

[0036] After obtaining the optimized high-quality image, edge detection needs to be performed on the image to obtain the edge point coordinates. Specifically, it is divided into two cases: If the edge of the ultrasonic image is obvious and there are not too many discrete point blocks, after gray processing of the image, each pixel point in the image is traversed. By fitting the pixel points within a certain area to the same range of colors, the color difference between different regions is increased, and then the Canny algorithm idea of OpenCV is used to perform edge detection on the image. After a series of operations such as denoising, filtering, and gradient calculation, the edge point coordinates of the ultrasonic image are obtained; if the edge of the ultrasonic image is not obvious, only binary processing is performed on the image to obtain the pixel points on the surface and inside of the ultrasonic object.

[0037] 3. Intermediate process of obtaining the ultrasonic image and the image:

[0038] In order to obtain more surface information of the object during the 3D mesh reconstruction process, predictions need to be made on the image after edge detection and the intermediate process of the image. Here, a prediction method based on motion estimation is mainly used to perform image interpolation. First, the average pixel spacing of a single image needs to be calculated. This average spacing is the standard spacing during interpolation. Then, based on data such as the actual spacing between the first image and the last image, and the ultrasonic scanning speed, the actual length, width, and height of the object are calculated, and then equal-proportion interpolation is performed according to the calculated pixel spacing.

[0039] Here, an innovative approach is used to generate the process images of objects based on deep learning frame interpolation algorithms. Compared with traditional medical CT three-dimensional reconstruction, it can better meet the three-dimensional model restoration requirements of a wider range of ultrasound images. Pixel-level motion estimation frame interpolation is achieved through a neural network architecture based on FlowNet. The FlowNet network can estimate the optical flow between two consecutive frames, that is, the information describing the motion direction and speed of pixels over time. RIFE uses the FlowNet network to calculate the optical flow between consecutive frames and then uses this optical flow information to generate intermediate interpolation frames. In this way, the changes between existing ultrasound images can be accurately restored, providing underlying data support for the restoration of the 3D model.

[0040] For a small number of discrete point images that do not have motion coherence, bicubic interpolation algorithms can be used for frame interpolation adaptation. This is a conventional image frame interpolation method in the industry and will not be elaborated here.

[0041] 4. Three-dimensional conversion of two-dimensional image coordinates

[0042] After completing image frame interpolation, the coordinates of the interpolated image are processed three-dimensionally. The specific method is as follows: The first pixel in the lower left corner of the image is used as the origin (0, 0), and then the coordinate points (x, y) of other edge detection pixels in the image are calculated. The horizontal and vertical coordinates (x, y) of the pixels in the first scanned image remain unchanged, and the z-axis coordinate is set to 0. According to the scanning speed of the scanned object, the z-axis coordinates of the edge pixels of the second to the nth images are derived linearly and written into a file in the format of a three-dimensional point cloud. If the target image only undergoes binary processing in the edge detection stage, further processing of the three-dimensional point cloud is required. The outer surface of the three-dimensional point cloud is obtained through a point cloud outer surface extraction algorithm, providing basic materials for the reconstruction of the three-dimensional mesh model. It should be noted that in addition to the position information of the image points, the color information of the ultrasound image can be optionally saved. To restore the color depth changes during object ultrasound, the original image also needs to be frame interpolated, and then the color of the two-dimensional image coordinates on the original image is stored at the corresponding points.

[0043] 5. 3D visualization of the outer surface of the mesh

[0044] The three-dimensional models used in mainstream three-dimensional visualization technologies are in the format of mesh triangles. Discrete point clouds are difficult to restore the three-dimensional outer surface of an object. Therefore, it is necessary to restore a three-dimensional mesh based on the point cloud. Here, mesh body recovery based on Poisson reconstruction is selected. Through a series of processes such as estimating the normal vector of the point cloud, Poisson reconstruction, removing redundant edges, denoising, and filtering, a complete 3D mesh model is finally generated.

[0045] There is a relatively common methodology in the industry for the processing flow of the entire model. Here, only technical recommendations for the key processing environment are provided. Before performing Poisson reconstruction, the normal vectors of the point cloud need to be calculated. Currently, the KD tree is used for nearest neighbor search, and the smoothness and accuracy are adjusted by regulating the coverage range of the nearest neighbors. These points are fitted as closely as possible to a surface to estimate the direction of the normal vectors. After calculating the normal vectors, an octree data structure is constructed, and the point cloud data is organized orderly into an octree space partitioning data structure. The octree can effectively represent the sampling density in space. By using the Poisson equation to solve for the isosurface, the triangular mesh outer surface of the 3D model can be inferred. After obtaining the preliminary outer surface of the model, the outer surface of the model needs to be smoothed. Here, the Taubin smoothing algorithm is used. Taubin smoothing is a surface smoothing technique used in image processing, which was proposed by Gabriel Taubin in 1995. This technique can be used for 3D surface reconstruction. Taubin smoothing smooths the surface in an iterative manner while keeping the feature points (such as corner points and boundaries) of the surface unchanged, so as to improve the surface quality.

[0046] At this time, the mesh model does not have color. The KD search tree needs to be used to color the points on the point cloud to each triangular face of the 3D mesh model. Finally, through UV unwrapping and the model baking engine, the color material of the model and the model body are separated, and finally presented on 3D visualization platforms such as UE and Unity.

[0047] For 3D models with a large number of discrete points, they are visualized in the form of a mesh + point cloud through a mesh point cloud fitting algorithm to improve the reduction accuracy as much as possible. Suppose there is a point cloud A for ultrasonic reconstruction and a mesh body B reconstructed from the mesh surface of the point cloud A. By creating a KD search tree, for each triangular face vertex of B, the nearest neighbor search algorithm is used to search for points in A. In this way, the points on the point cloud A that are far from B are deleted, and the cropped point cloud A is fitted with B to form a 3D model combining a mesh and a point cloud to accurately restore the physical characteristics of the ultrasonic object.

[0048] Example 2, based on Example 1, proposes an implementation system for 3D reconstruction based on industrial CT images. The system includes:

[0049] A CT image preprocessing module, which is used to convert the CT-scanned dat format image into JPEG format, perform grayscale processing and denoising, and then improve the image resolution through the bicubic interpolation algorithm. The bicubic interpolation algorithm interpolates horizontally and vertically by selecting 16 nearest neighbor pixels around the target pixel to improve the calculation efficiency while ensuring accuracy.

[0050] An image edge detection module is used to perform edge detection on the preprocessed image, and select to use the Canny algorithm or binaryzation processing to obtain the edge point coordinates according to the obviousness of the image edge; the selection of the edge detection algorithm is based on the clarity of the image edge. For images with obvious edges, the Canny algorithm is adopted, and for images with unclear edges, binaryzation processing is adopted.

[0051] An image interpolation module includes a traditional interpolation algorithm based on motion estimation and a deep learning interpolation algorithm, such as using the FlowNet network to calculate the optical flow and generate intermediate interpolation frames to enrich the surface information of objects in the image sequence; the deep learning interpolation algorithm in the image interpolation module uses the FlowNet network to calculate the optical flow between consecutive frames and generates intermediate interpolation frames based on this optical flow information to meet the wide range of three-dimensional reconstruction requirements of ultrasonic images.

[0052] A two-dimensional image coordinate three-dimensionalization module is used to convert the coordinates of the interpolated image into three-dimensional coordinates, calculate the z-axis coordinates of each image edge pixel according to the scanning speed parameter, and generate three-dimensional point cloud data;

[0053] A three-dimensional mesh reconstruction and visualization module includes a mesh body recovery algorithm based on Poisson reconstruction, which is used to recover the mesh body from the three-dimensional point cloud, and perform surface smoothing through the Taubin smoothing algorithm. Finally, through color mapping, UV unwrapping and model baking, the three-dimensional mesh model and its color materials are presented on the 3D visualization platform, or high-precision visualization is performed by combining the mesh and the point cloud; for three-dimensional models with more discrete points, a mesh point cloud fitting algorithm is adopted, and the cropped point cloud and the mesh body are fitted through the KD search tree and the nearest neighbor search algorithm to accurately restore the physical characteristics of the detected object.

[0054] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for implementing three-dimensional reconstruction based on industrial CT images, characterized in that: The method comprises the following steps: CT image preprocessing: convert the dat format image obtained by CT scanning into JPEG format and grayscale it, then use the mean filter to denoise it, and then use the bicubic interpolation algorithm to improve the image resolution; Edge detection: perform edge detection on the preprocessed image, use the Canny algorithm or binarization processing to obtain the edge point coordinates, and select the appropriate method based on the image features; Image interpolation, based on deep learning interpolation algorithms such as the FlowNet network, calculates the optical flow information between images to generate intermediate frames. For discrete point images that do not have motion coherence, the bicubic interpolation algorithm is used; Convert the 2D image coordinates to 3D coordinates, convert the interpolated image coordinates to 3D coordinates, determine the z-axis coordinates of each pixel point according to the scanning speed and image sequence, and save the color information; 3D visualization of the mesh outer surface, restore the mesh from the 3D point cloud through the Poisson reconstruction algorithm, use the Taubin smoothing algorithm to smooth the surface, and map the color information to the mesh surface based on the KD search tree, and finally generate and display a colored 3D mesh model.

2. The method for realizing three-dimensional reconstruction based on industrial CT images according to claim 1, characterized in that: In the CT image preprocessing step, the bicubic interpolation algorithm improves image accuracy by selecting 16 nearest neighbor pixels around the target pixel for horizontal and vertical interpolation.

3. The method for realizing three-dimensional reconstruction based on industrial CT images according to claim 1, characterized in that: In the edge detection step, the Canny algorithm or binarization processing is selected according to the obviousness of the image edge to ensure the accurate acquisition of the edge point coordinates.

4. The method for realizing three-dimensional reconstruction based on industrial CT images according to claim 1, characterized in that: In the image interpolation step, the FlowNet network is used to estimate the optical flow between consecutive frames, and intermediate interpolation frames are generated based on the optical flow information to meet the needs of a wide range of ultrasound image three-dimensional model restoration.

5. The method for realizing three-dimensional reconstruction based on industrial CT images according to claim 1, characterized in that: The 3D visualization step of the mesh outer surface also includes mapping the color information on the point cloud to each triangle face of the 3D mesh model through the KD lookup tree, and separating the color material and model body of the model through UV unfolding and model baking engine to achieve high-precision 3D visualization.

6. A system for implementing three-dimensional reconstruction of industrial CT images according to the method for implementing three-dimensional reconstruction of industrial CT images according to any one of claims 1 to 5, characterized in that: The system comprises: CT image preprocessing module, used to convert CT scanned DAT format images into JPEG format, perform grayscale processing and denoise, and then improve image resolution through bicubic interpolation algorithm; The image edge detection module is used to perform edge detection on the preprocessed image and select the Canny algorithm or binarization processing to obtain the edge point coordinates according to the obviousness of the image edge; Image interpolation module, including traditional interpolation algorithms based on motion estimation and deep learning interpolation algorithms, such as using the FlowNet network to calculate optical flow and generate intermediate interpolation frames to enrich the surface information of objects in the image sequence; The 2D image coordinate conversion module is used to convert the interpolated image coordinates into 3D coordinates, and calculate the z-axis coordinates of each image edge pixel according to the scanning speed parameter to generate 3D point cloud data; The 3D mesh reconstruction and visualization module includes a mesh recovery algorithm based on Poisson reconstruction, which is used to recover the mesh from the 3D point cloud and perform surface smoothing through the Taubin smoothing algorithm. Finally, through color mapping, UV unfolding and model baking, the 3D mesh model and its color material are presented on the 3D visualization platform, or the mesh and point cloud are combined for high-precision visualization.

7. The system for realizing three-dimensional reconstruction based on industrial CT images according to claim 6, characterized in that: In the CT image preprocessing module, the bicubic interpolation algorithm selects 16 nearest neighbor pixels around the target pixel for horizontal and vertical interpolation to improve computational efficiency while ensuring accuracy.

8. The system for realizing three-dimensional reconstruction based on industrial CT images according to claim 6, characterized in that: In the image edge detection module, the selection of edge detection algorithm depends on the clarity of the image edge. For images with obvious edges, the Canny algorithm is used, and for images with unclear edges, binarization processing is used.

9. The system for realizing three-dimensional reconstruction based on industrial CT images according to claim 6, characterized in that: The deep learning interpolation algorithm in the image interpolation module uses the FlowNet network to calculate the optical flow between consecutive frames and generate intermediate interpolation frames based on this optical flow information to meet the needs of a wide range of ultrasound image 3D reconstruction.

10. The system for realizing three-dimensional reconstruction based on industrial CT images according to claim 6, characterized in that: In the 3D mesh reconstruction and visualization module, for 3D models with a large number of discrete points, a mesh point cloud fitting algorithm is used to fit the cropped point cloud with the mesh body through the KD search tree and nearest neighbor search algorithm to accurately restore the physical characteristics of the detected object.

Images