Three-dimensional model reconstruction method and system based on interactive segmentation and multi-stage fusion

Abstract

The invention discloses a three-dimensional model reconstruction method and system based on interactive segmentation and multi-stage fusion, and the method comprises the following steps: obtaining a target two-dimensional image of a labeling sequence, and carrying out the preprocessing; obtaining a marking operation result of a target area and a non-target area of the preprocessed two-dimensional image; segmenting all the two-dimensional images after marking operation by adopting an SAM model to obtain masks; morphological operation processing is carried out on the mask, contour extraction is carried out, and a contour list of target area contours is obtained; according to the contour number in the contour list, grid model reconstruction of a single contour stage, a multi-contour stage and / or a transition contour stage is carried out on all the target area contours; and carrying out merging, tail end closing and optimization processing on the reconstructed grid model to obtain a reconstructed target three-dimensional model. The problems that in an existing three-dimensional modeling technology, contour extraction precision is low, efficiency is low, and complex topological structures are difficult to process are solved.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer graphics and medical image processing, and more particularly to a three-dimensional model reconstruction method and system based on interactive segmentation and multi-stage fusion. Background Art

[0002] In the prior art, three-dimensional reconstruction methods based on CT or MRI images mostly rely on traditional segmentation algorithms or fully automatic deep learning segmentation models.

[0003] However, the accuracy of traditional segmentation methods is limited. When dealing with medical images with low contrast, noise pollution or complex shapes, problems such as inaccurate contour extraction, breakage or loss are likely to occur, resulting in low accuracy and rough surface of the reconstructed model. The fully automatic method has insufficient robustness for specific scenarios, is difficult to efficiently process complex targets, and most models in the automatic segmentation method based on convolutional neural networks can only process specific target types, lacking wide applicability and being unable to cope with variable target recognition tasks. Manual segmentation can improve accuracy, but it is time-consuming, laborious and highly subjective.

[0004] In recent years, the Segment Anything Model (SAM) based on deep learning has been proposed. It supports fast and interactive object segmentation and has good versatility, providing new possibilities for three-dimensional model reconstruction.

[0005] Therefore, how to combine the efficient segmentation ability of SAM with the three-dimensional reconstruction process is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a three-dimensional model reconstruction method and system based on interactive segmentation and multi-stage fusion, which can achieve fast and accurate object contour extraction through a small number of marked points, and generate high-quality three-dimensional models by multi-stage fusion.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] In the first aspect, the present invention provides a three-dimensional model reconstruction method based on interactive segmentation and multi-stage fusion, including the following steps:

[0009] S1. Obtain the target two-dimensional image of the annotation sequence and perform preprocessing; and then obtain the marking operation results of the target area and non-target area on the preprocessed two-dimensional image;

[0010] S2. Use the SAM model to segment all the marked two-dimensional images to obtain masks;

[0011] S3. Perform morphological operation on the mask, and perform contour extraction to obtain a contour list of the target region contour;

[0012] S4. Reconstruct the grid model of the single contour stage, multi-contour stage, and / or transition contour stage for all target region contours respectively according to the number of contours in the contour list;

[0013] S5. Merge, close the ends, and optimize the reconstructed grid model to obtain the reconstructed target three-dimensional model.

[0014] Further, the contour extraction in step S3 specifically includes:

[0015] Perform binarization on the mask obtained after morphological operation;

[0016] Scan the image line by line, find the first non-zero value as the contour starting point; search according to the eight-neighborhood rule, find adjacent non-zero values, record and move along the contour boundary until returning to the contour starting point to form a complete contour;

[0017] Continue to scan the image to find unvisited non-zero values until the image scanning is completed to find all contours.

[0018] Further, step S4 specifically includes:

[0019] When the number of contours in the contour lists of adjacent images is 1, reconstruct the single contour grid model of the target region contour;

[0020] When the number of contours in the contour lists of adjacent images is k and k>1, reconstruct the multi-contour grid model of the target region contour;

[0021] When the number of contours in the contour lists of adjacent images is different, reconstruct the transition contour grid model of the target region contour.

[0022] Further, the reconstruction of the single contour grid model includes:

[0023] By resampling, sample the same number of points from all contours with a quantity of 1,

[0024] Find points A and B from the adjacent upper contour and lower contour respectively; among them, the three-dimensional Euclidean distance between A and B is the shortest;

[0025] Take the line segment AB as the initial edge, and add subsequent points and the previous two points to form a triangle alternately up and down according to the contour direction until all points are used as vertices to form a triangle, and the last point and the initial edge AB form a triangle to obtain the triangulation result of two adjacent images in the single contour stage;

[0026] Add adjacent contours and continue with triangulation until all contours with a quantity of 1 are added, obtaining the triangulation result in the single contour stage as the single contour mesh model.

[0027] Furthermore, the reconstruction of the multi - contour mesh model includes:

[0028] Calculate the centroid of all contours in two adjacent images through contour matching;

[0029] Find the corresponding contours in the two images according to the positions of the centroids;

[0030] Triangulate each pair of corresponding contours in the same way as the reconstruction of the single contour mesh model;

[0031] Merge all the triangulation results to obtain the multi - contour mesh model.

[0032] Furthermore, when the number of contours in the contour lists of adjacent images is different, reconstruct the transitional contour mesh model of the target area contour; specifically including:

[0033] When the number of contours in the contour lists of adjacent images is different, it is divided into the transition from the single contour to the multi - contour stage and the transition from the multi - contour to the multi - contour stage;

[0034] Merge the triangulation results of the two transition stages to obtain the transitional contour mesh model.

[0035] Furthermore, the transition from the single contour to the multi - contour stage includes:

[0036] Project the contours in two adjacent images onto the same plane;

[0037] Enlarge the contour of the single - contour image until all contours in the multi - contour image are completely enclosed;

[0038] Use Delauney triangulation between the contours to construct the topological relationship between the contours of the two images;

[0039] After construction, keep the connection relationship of the triangles unchanged, and restore the coordinates of the contour points to the contour coordinates in the three - dimensional space before projection, obtaining the triangulation in the transition stage from the single contour to the multi - contour.

[0040] Furthermore, the transition from the single contour to the multi - contour stage also includes:

[0041] When the shape of the single - contour image cannot completely enclose all contours in the multi - contour image after scaling, segment the single - contour image according to the concave shape;

[0042] Align the segmented contour with each contour in the multi - contour image;

[0043] Project each contour in the multi - contour image to the corresponding position of the contour in the single - contour image, and then scale the single - contour image to achieve complete wrapping.

[0044] Furthermore, the transition from the multi - contour to the multi - contour stage includes:

[0045] Through contour matching and centroid position calculation; convert the transition problem from the multi - contour to the multi - contour stage into the reconstruction problem of a multi - contour grid model with the same number and the transition problem from the single - contour to the multi - contour stage;

[0046] According to the reconstruction of the multi - contour grid model, obtain the partial meshing result of the transition stage;

[0047] According to the transition from the single - contour to the multi - contour stage, obtain another partial meshing result of the transition stage;

[0048] Merge the meshing results to obtain the meshing of the transition stage from the multi - contour to the multi - contour.

[0049] In a second aspect, the present invention provides a three - dimensional model reconstruction system based on interactive segmentation and multi - stage fusion, including the following modules:

[0050] Acquisition module: used to acquire the target two - dimensional image of the annotation sequence and perform pre - processing; and then acquire the marking operation results of the target area and non - target area on the pre - processed two - dimensional image;

[0051] Segmentation module: used to segment all the two - dimensional images after the marking operation by using the SAM model to obtain a mask;

[0052] Extraction module: used to perform morphological operation processing on the mask and perform contour extraction to obtain a contour list of the target area contour;

[0053] Reconstruction module: used to perform grid model reconstruction of the single - contour stage, multi - contour stage, and / or transition contour stage for all the target area contours according to the number of contours in the contour list;

[0054] Optimization module: used to perform merging, end - closing, and optimization processing on the reconstructed grid model to obtain the reconstructed target three - dimensional model.

[0055] For the description of the second aspect of the present invention, reference can be made to the detailed description of the first aspect; and for the beneficial effects of the description of the second aspect, reference can be made to the analysis of the beneficial effects of the first aspect, which will not be elaborated here.

[0056] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a three-dimensional model reconstruction method based on interactive segmentation and multi-stage fusion, which has the following beneficial effects:

[0057] 1. High-precision interactive segmentation: Interactive segmentation only requires providing a small number of hint points to obtain high-precision segmentation results. It can work effectively even in the case of poor image quality and blurred object boundaries, greatly improving the efficiency and accuracy of contour extraction.

[0058] 2. High efficiency and high degree of automation: Interactive segmentation only requires a small amount of input, greatly reducing the workload of manual segmentation and improving the reconstruction efficiency. The subsequent contour analysis, alignment, and three-dimensional reconstruction processes are all automated, further improving the efficiency.

[0059] 3. Can handle complex topological structures: The present invention can handle complex object structures such as branches, bends, and diameter changes, and adaptively adjust the triangulation strategy according to the change in the number of contours, effectively avoiding topological errors and geometric distortions.

[0060] 4. The generated model has higher quality: Through mesh merging and optimization processing, the generated model has high precision, a smooth surface, and is more in line with the shape of the actual object. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0062] Figure 1 It is a flowchart of the three-dimensional model reconstruction method based on interactive segmentation and multi-stage fusion provided by the present invention.

[0063] Figure 2 It is a display diagram of the medical image sequence input in the embodiment of the present invention.

[0064] Figure 3 It is a schematic diagram of interactive segmentation provided in the embodiment of the present invention.

[0065] Figure 4 It is a result display diagram of the contour extraction of the input medical image sequence provided in the embodiment of the present invention.

[0066] Figure 5 It is a schematic diagram of the side simple constraint triangulation of the skeleton based on the contour line provided in the embodiment of the present invention.

[0067] Figure 6Schematic diagram of the relationship of the reconstructed triangular patches in the contour transition stage provided by the embodiment of the present invention.

[0068] Figure 7 Schematic diagram of the concave contour segmentation provided by the embodiment of the present invention.

[0069] Figure 8 Schematic diagrams of the 3D reconstruction results at different stages and the final merging result provided by the embodiment of the present invention.

[0070] Figure 9 Schematic diagrams of the result of closing the end of the 3D model and the final result obtained through mesh optimization provided by the embodiment of the present invention.

[0071] Figure 10 Block diagram of the 3D model reconstruction system based on interactive segmentation and multi-stage fusion provided by the present invention. Detailed implementation manners

[0072] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0073] Embodiment 1

[0074] An embodiment of the present invention discloses a 3D model reconstruction method based on interactive segmentation and multi-stage fusion. Referring to Figure 1 as shown, where STEP1 is the contour extraction stage, and STEP2 is the 3D reconstruction stage, including the following steps:

[0075] S1. Obtain the target two-dimensional image of the annotation sequence and perform preprocessing; and then obtain the marking operation results of the target area and the non-target area on the preprocessed two-dimensional image;

[0076] S2. Use the SAM model to segment all the marked two-dimensional images to obtain a mask;

[0077] S3. Perform morphological operation processing on the mask and perform contour extraction to obtain a contour list of the target area contour;

[0078] S4. Perform mesh model reconstruction on all the target area contours in single contour stage, multi-contour stage and / or transition contour stage respectively according to the number of contours in the contour list;

[0079] S5. Perform merging, end closing and optimization processing on the reconstructed mesh model to obtain the reconstructed target 3D model.

[0080] In medical image processing, a series of two-dimensional medical images can be obtained through CT scans or MRI. In this embodiment, an MRI sequence image of the knee joint part of the human body is obtained by scanning the human knee with a magnetic resonance imaging scanner. First, the target two-dimensional image of the knee joint is obtained from the MRI sequence, and necessary preprocessing operations are performed, such as denoising and contrast adjustment, to improve the accuracy of subsequent processing. The preprocessed image is labeled to clarify which regions are the target regions of interest (such as the knee joint) and which are non-target regions. The SegmentAnythingModel (SAM) is used to accurately segment the labeled two-dimensional image to generate a mask of the target region for distinguishing the target from the background. Further processing is performed on the generated mask, including morphological operations (such as dilation and erosion) to optimize the edges, and then contour extraction is performed to obtain a list containing the contours of all target regions. According to the information in the contour list, three-dimensional mesh model reconstruction is performed on each contour in the single contour stage, multi-contour stage, or transitional contour stage; this step involves converting two-dimensional contour information into three-dimensional space information. Finally, all the reconstructed mesh models are merged, the ends of the models are closed, and the entire model is optimized to ensure that the final obtained three-dimensional model is both accurate and smooth.

[0081] This method based on interactive segmentation and multi-stage fusion shows great potential in medical image processing, especially in the three-dimensional reconstruction of complex structures, such as the detailed reconstruction of the knee joint part of the human body, which helps doctors more accurately evaluate the condition and formulate treatment plans. The present invention can be widely applied to fields such as medical image analysis, surgical planning, organ modeling, and industrial CT reconstruction.

[0082] The following details each of the above steps:

[0083] Step S1, first obtain the target two-dimensional image of the labeled sequence and perform preprocessing;

[0084] In this embodiment, an MRI sequence image of the knee joint part of the human body is first obtained by scanning the human knee with a magnetic resonance imaging scanner; the image type can be in multiple formats, including but not limited to image formats such as ".png", ".jpg", ".jpeg", ".bmp", ".tiff", etc. According to the folder path provided by the user for storing the image sequence, all the images in the folder are automatically read in sequence and converted into a numpy array for preprocessing. Refer to Figure 2 As shown, in this embodiment, the input continuous sequence is the MRI sequence image of the knee joint part from 10 to 21, and the picture format is ".jpg", and then preprocessing is performed on it.

[0085] In this embodiment, the image preprocessing is mainly image denoising. The bilateral filtering method is used. Different from traditional linear filters (such as mean filtering and Gaussian filtering), bilateral filtering not only considers the spatial position of pixels, but also considers the color or grayscale value difference of pixels. Therefore, it can maintain the details and edges of the image while smoothing the image, which is beneficial to improving the subsequent segmentation effect.

[0086] Then, perform marking operations on the target area and non-target area of the preprocessed two-dimensional image;

[0087] In this embodiment, the preprocessed image I is sequentially displayed in the input order of the image sequence I i , where i is the image sequence; referring to Figure 3 As shown, the user can perform marking operations on the foreground points and background points of the image at this stage. The foreground points are used as the target area, and the background points are used as the non-target area.

[0088] Specifically, for each displayed image I i , after observing the image, the user uses the left mouse button to click on the target area to be segmented in the image to mark the foreground point P f . The marking rule is relatively free, as long as there is at least one marking point in each connected target area. In this embodiment, about 3 foreground points are marked in the target area to be segmented, and they are distributed as evenly as possible in the target area to be segmented. For mis-marked or unsatisfactory points, the user can click the right mouse button to cancel the marking operation, and multi-step cancellation operations are supported. After the foreground points are marked, press the "Q" key to enter the next background point marking stage. The operation rule for marking background points is the same as that for marking foreground points. Click the right mouse button in the non-target area to mark it as the background point P b , and cancellation operations are also supported.

[0089] After marking the foreground point P f and the background point P b , the marking work of a single image I i is completed. At this time, press the "Q" key to enter the marking of the next image I i+1 , until all images are marked.

[0090] Step S2, use the SAM model to segment all the two-dimensional images after the marking operations to obtain a mask;

[0091] In this embodiment, after completing the annotation of the entire image sequence I, all the annotated images are automatically sent to the SAM model for segmentation to obtain a mask.

[0092] The SAM model used in this embodiment is an open-source interactive image segmentation network model. First, the input image enters a powerful feature extractor, usually a deep convolutional neural network, which can capture multi-scale information in the image and generate rich feature representations. Then, the user guides the model to focus on specific target areas by providing foreground and background points; these points are added to the feature map as additional information to help the model more accurately identify the target boundaries. Subsequently, the prediction mask is generated; based on the enhanced feature map, the SAM model generates multiple possible object masks. Each mask represents an independent object or part of an object that the model believes may be of interest to the user. Finally, the best mask is selected; after obtaining a series of candidate masks, the model is further optimized according to the points provided by the user, and finally the mask that best meets the user's intention is selected as the output result.

[0093] The SAM model segmentation in this embodiment also has an iterative improvement function; if the initial segmentation result is not satisfactory, the user can refine the operation by adding more indicator points or adjusting the positions of the existing points until a satisfactory segmentation result is obtained.

[0094] Step S3, perform morphological operation processing on the segmented mask, and perform contour extraction to obtain a contour list of the target area contour;

[0095] After the segmentation is completed, since there are a certain number of small holes, noises, and uneven boundaries in the obtained mask, this embodiment performs morphological operation processing on the mask. A 3*3 matrix is used as a window to perform closing operation first and then opening operation on the mask image to remove noise points, fill holes, and smooth the mask boundary, obtaining the processed target area mask M.

[0096] After performing the above operations, the segmented mask M can be subjected to contour extraction to obtain the contour C of the target area in the image.

[0097] The contour extraction part uses the contour tracing algorithm to extract the contour of the mask. First, the mask M obtained through the above processing is binarized, and then the Suzuki algorithm is used for contour tracing. The specific implementation method is to scan the image row by row starting from the upper left corner until the first foreground pixel P (with a value of 255 in this embodiment) is found. This foreground point is determined as the starting point of the contour. After determining the starting point, according to the eight-neighborhood rule, search for surrounding pixels, find adjacent foreground pixels, record this pixel and continue to move along the boundary of the contour until returning to the starting point, forming a complete contour C. After a contour is accessed, continue to scan the unaccessed foreground pixels to find all the contours until the entire image is traversed. All the contour information obtained for the same picture is stored in the same contour list L.

[0098] In this embodiment, the above steps are generally referred to in the appendixFigure 3 As shown, attached Figure 3 From left to right are the input image to be segmented, user markings, and segmentation results. The green dots are the marked foreground points, the red dots are the marked background points, and the blue area is the mask segmented by the SAM model based on the markings; the contour of the target area extracted according to the segmentation mask. Refer to attached Figure 4 As shown, this is the result of contour extraction of the target area in the input image sequence in this embodiment. In this embodiment, the number of contours for sequences 10 to 19 is 1, and the number of contours for 20 to 21 is 2. Figure 2

[0099] Step S4, perform grid model reconstruction on all target area contours in single contour stage, multi contour stage, and / or transitional contour stage respectively according to the number of contours in the contour list obtained in step S3;

[0100] In this embodiment, the contour list of each picture is read in the order of the picture sequence, and the contours are divided into single contour stage, multi contour stage, and transitional stage from single contour to multi contour according to the change of the number of contours in the contour list, and the triangulation strategy is adaptively adjusted according to different stages.

[0101] The specific implementation method is:

[0102] ①. For the triangulation in the single contour stage.

[0103] The number of contours in the contour lists of adjacent images is 1, and the single contour grid model of the target area contour is reconstructed; in this embodiment, the single contour stage is defined as that the number of contours included in two adjacent pictures I i and I i+1 is 1, and a simple constrained triangulation strategy can be used. Specifically, refer to attached Figure 5 shown in a - 5f.

[0104] First, resample all contours to the same number of points, and then find points A and B from the upper contour C i and the lower contour C i+1 respectively, so that the three - dimensional Euclidean distance between A and B is the shortest. Refer to part a of attached Figure 5 ; take AB as the starting edge for triangle construction. Refer to part b of attached Figure 5 ; for the two points C and D adjacent to A and B in the contour direction, compare the magnitudes of ∠ACB and ∠ADB in three - dimensional space. If ∠ACB is larger, then construct a triangle with points A, C, and B as vertices. Refer to part c of attached Figure 5 ; otherwise, construct a triangle with points A, D, and B as vertices. Refer to part d of attached Figure 5 After constructing the first triangle, take the newly added edge as the new starting edge. Refer to attached​Figure 5 For parts e and f, construct subsequent triangles again according to the above rules until all points are used as vertices to form triangles. Finally, construct a triangle with the last point and the initial side AB. In this way, the triangulation result of two adjacent images in the single contour stage is obtained. And so on, continuously add new contours and perform triangulation with the contours in the previous layer to obtain the triangulation result in the single contour stage;

[0105] ②. For the triangulation in the multi - contour stage.

[0106] The number of contours in the contour list of adjacent images is k, and k > 1. Reconstruct the multi - contour grid model of the target area contour; in this embodiment, the multi - contour stage is defined as two adjacent pictures I i and I i+1 where the number of contours contained in both is k and k ≠ 1.

[0107] First, through contour matching, calculate the centroid of all contours in two adjacent images. According to the position of the centroid, find the corresponding contours in the two images. Once the corresponding contours are found, each pair of corresponding contours can be triangulated as in the triangulation method of the single - contour stage in step ① above. Finally, merge all the triangulation results to complete the triangulation in the multi - contour stage.

[0108] ③. For the triangulation in the transition stage from single - contour to multi - contour.

[0109] When the number of contours in the contour list of adjacent images is different, reconstruct the transition contour grid model of the target area contour; in this embodiment, the transition stage is defined as two adjacent pictures I i and I i+1 where the number of contours contained is k i and k i+1 where k i = 1 and k i+1 > 1, for the case of multi - contour to multi - contour, that is, k i+1 > k i and k i ≠ 1, then refine the contour change as in step ② to include the triangulation from the single - contour stage to the multi - contour stage and the multi - contour stage, and handle them separately.

[0110] Generally, the transition stage with a change in the number of contours only includes two adjacent images I i and I i+1 , so the contours in the two images can be projected onto the same plane. Under the condition of aligning the centroids of the two images, enlarge the contours of the single - contour image I i until the contours of the multi - contour image I i+1All the contours in are completely enclosed, and then Delauney triangulation is used between each contour to construct I i and I i+1 The topological relationship between the contours of is shown in part a of the attached Figure 6 where the horizontal and vertical coordinates are the two-dimensional coordinates of the sampling points in pixels; after the construction is completed, the connection relationship of the triangles is kept unchanged, and the coordinates of the points in the contour point set are restored to the contour coordinates in the 3D space before projection. In this way, the triangulation of the transition stage from single contour to multiple contours is completed, and the final result is shown in part b of the attached Figure 6 where the three-dimensional coordinates are based on the two-dimensional coordinates, and the z-axis coordinates are added to the contour point sets in all images. The z-axis coordinates of the point sets in the image with the smallest serial number (i.e., the first image in the sequence) are 0, and the z-axis coordinates of each subsequent image are constructed as the three-dimensional space coordinates of all point sets with the scanning accuracy of the scanned image, that is, the distance d between slices as the z-axis coordinate. It should be noted that for the convenience of observation, the x, y, and z axes can be scaled to different degrees when displaying the image. For better observation of the reconstructed topological structure, the x, y coordinates and the z coordinate are not on the same scale, and the z-axis is appropriately enlarged for easy observation.

[0111] ④. When the shape of the single contour image cannot completely enclose all the contours in the multi-contour image after scaling, the single contour image is segmented according to the concave shape;

[0112] In this embodiment, the shape of the single contour is a concave contour, and it is possible that simply scaling cannot make the contour C i in I i completely enclose all the contours C i+1 in the multi-contour image I i+1 , as shown in part a of the attached Figure 7 . At this time, the contour C i needs to be segmented according to the concave shape, as shown in part b of the attached Figure 7 . The contour with one concave is segmented into left and right parts, and the segmented contours and are aligned with each contour in the multi-contour image. Each contour i+1 in the multi-contour image I is projected onto the corresponding position of the segmented contour i in the single contour image I , and then these segmented partial contours are scaled. In this way, the concave contour can be completely enclosed. The subsequent triangulation steps are the same as those described in step ③.

[0113] In this embodiment, the concavity and convexity of the contour are determined by calculating the convex hull of the contour, and then the area of the contour is compared with the area of the convex hull. If the area of the contour is smaller than the area of the convex hull, it indicates that there is a depression, and different dissection methods are selected according to the concavity and convexity. Similarly, the scaling factor for scaling the contour here is also automatically calculated by the system. The specific steps are as follows: First, the iterative closest point (ICP) algorithm is used to align the two contours, and then the scaling factor scale_factor = 1.0 is initialized. The contour is iteratively scaled with a step size step = 0.01 and a maximum value max_scale = 3.0 until the scaled contour completely encloses the target contour, and the scaling factor scale_factor at this time is returned.

[0114] Refer to the appendix for the contour reconstruction results and mesh merging results at different stages. Figure 8 as shown. Figure 8 Figures a - 8d are respectively the reconstruction results in the single - contour stage, the reconstruction results in the transition stage, the reconstruction results in the multi - contour stage, and the mesh merging results of the reconstruction in all stages.

[0115] Step S5: Merge, close the ends, and optimize the reconstructed mesh model to obtain the reconstructed target three - dimensional model.

[0116] In this embodiment, after the image sequence reaches the end, the ends of the mesh model are closed. Delauney triangulation within the contours of all the contours in the last image is performed to achieve closure, obtaining a complete and closed three - dimensional mesh model, as shown in the left - hand part of the appendix; then vertex welding and Loop subdivision and other optimization processes are carried out to improve the model quality. The final result is shown in the right - hand part of the appendix; finally, the ultimately reconstructed three - dimensional model is output. Figure 9 as shown in the left - hand part of the appendix; then vertex welding and Loop subdivision and other optimization processes are carried out to improve the model quality. The final result is shown in the right - hand part of the appendix; finally, the ultimately reconstructed three - dimensional model is output. Figure 9 Finally, the ultimately reconstructed three - dimensional model is output.

[0117] Embodiment 2

[0118] This embodiment of the present invention discloses a three - dimensional model reconstruction system based on interactive segmentation and multi - stage fusion. Refer to Figure 10 as shown, which includes the following modules:

[0119] Acquisition module: used to acquire the target two - dimensional images of the annotation sequence and perform pre - processing; and then acquire the marking operation results of the target area and non - target area on the pre - processed two - dimensional images.

[0120] Segmentation module: used to segment all the marked two - dimensional images by using the SAM model to obtain a mask.

[0121] Extraction module: used to perform morphological operation processing on the mask and extract contours to obtain a contour list of the target area contours.

[0122] Reconstruction module: used to perform mesh model reconstruction on all target region contours in single contour stage, multi - contour stage, and / or transition contour stage respectively according to the number of contours in the contour list;

[0123] Optimization module: used to perform merging, end - closing, and optimization processing on the reconstructed mesh model to obtain the reconstructed target three - dimensional model.

[0124] This system obtains the target two - dimensional image of the annotation sequence through the acquisition module and performs pre - processing; performs marking operations on the target area and non - target area through the marking module; then, through the segmentation module, uses the SAM model to accurately segment the target organ according to a small number of foreground and background point hints provided, and can obtain high - quality contours even in the case of poor image quality, and at the same time supports multi - target segmentation.

[0125] Performs contour extraction through the extraction module; and automatically analyzes the extracted contours through the reconstruction module, divides the image sequence into different stages (single contour, multi - contour, and transition stage) according to the change in the number of contours, and adopts corresponding processing strategies for different stages to adapt to the complex changes in the shape of the organ, and can well handle the situation where the target object has branch changes, including:

[0126] Contour resampling and alignment: Resample each contour to have the same number of points, and use the Iterative Closest Point (ICP) algorithm for precise alignment to ensure the spatial correspondence between different slice contours. For split contours, use Principal Component Analysis (PCA) for segmentation and alignment.

[0127] Stage - based adaptive triangulation: According to the number and shape of contours, adopt different triangulation strategies, such as column - shaped side reconstruction in the single - contour stage, independent triangulation and merging in the multi - contour stage, and projection - based triangulation and automatic scaling factor determination in the transition stage, to ensure that the generated mesh can have a smooth transition.

[0128] Finally, through the optimization module, perform mesh merging and optimization, merge the triangular meshes of different stages into a complete model, and perform mesh optimization, such as vertex welding to remove duplicate vertices, Loop subdivision for smoothing, etc., to improve the model quality.

[0129] This system supports distributed input data processing, can be applicable to large - scale image sequences; can also perform result visualization and output, and supports the export of models in multiple formats.

[0130] The present system solves the problem that the existing system lacks user interaction, making it difficult for users to quickly intervene in the segmentation results, which affects the accuracy of specific tasks. The present system can handle complex topological structures. For complex organ structures such as branches, bends, and variations, it can achieve accurate reconstruction, and there are no problems of boundary discontinuity and surface roughness in the mesh generation of 3D reconstruction; the alignment and fusion of multi-stage contours avoid problems such as error accumulation.

[0131] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same and similar parts between the various embodiments, reference can be made to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and reference can be made to the description of the method part for the relevant parts.

[0132] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A three-dimensional model reconstruction method based on interactive segmentation and multi-stage fusion, characterized in that: The following steps are involved: S1, obtaining a target two-dimensional image of a labeled sequence and performing preprocessing; and then obtaining the result of marking the target area and non-target area of ​​the preprocessed two-dimensional image; S2, using the SAM model to segment the two-dimensional images after all labeling operations to obtain masks; S3, performing morphological operation processing on the mask and performing contour extraction to obtain a contour list of the target area contour; S4, reconstructing the mesh models of all target area contours in a single contour stage, a multi-contour stage and / or a transition contour stage according to the number of contours in the contour list; S5. The reconstructed mesh models are merged, the ends are closed and optimized to obtain a reconstructed target three-dimensional model.

2. The three-dimensional model reconstruction method based on interactive segmentation and multi-stage fusion according to claim 1, characterized in that: The contour extraction described in step S3 specifically includes: Binarize the mask obtained after morphological operation; Scan the image line by line to find the first non-zero value as the starting point of the contour; search according to the eight-neighborhood rule to find adjacent non-zero values, record and move along the contour boundary until returning to the starting point of the contour to form a complete contour; Continue scanning the image, looking for unvisited non-0 values, until the image is scanned completely and all contours are found.

3. The three-dimensional model reconstruction method based on interactive segmentation and multi-stage fusion according to claim 1, characterized in that: Step S4 specifically includes: When the number of contours in the contour lists of adjacent images is all 1, a single contour mesh model of the target area contour is reconstructed; When the number of contours in the contour lists of adjacent images is k, and k>1, a multi-contour mesh model of the contour of the target area is reconstructed; When the number of contours in the contour lists of adjacent images is different, a transition contour mesh model of the contour of the target area is reconstructed.

4. The three-dimensional model reconstruction method based on interactive segmentation and multi-stage fusion according to claim 3, characterized in that: The reconstruction of the single contour mesh model comprises: By resampling, the same number of points are sampled from all contours with a value of 1. Find point A and point B from the adjacent upper contour and lower contour respectively; among them, the three-dimensional Euclidean distance between A and B is the shortest; Take the AB line as the initial edge, add subsequent points alternately up and down in the direction of the contour to form a triangle with the previous two points, until all points are used as vertices to form a triangle, and the last point and the initial edge AB construct a triangle to obtain the triangulation result of two adjacent images in the single contour stage; Adjacent contours are added and triangulation is continued until all contours with a quantity of 1 are added, and the triangulation result of the single contour stage is obtained as a single contour mesh model.

5. The three-dimensional model reconstruction method based on interactive segmentation and multi-stage fusion according to claim 4, characterized in that: The reconstruction of the multi-contour mesh model comprises: By contour matching, the centroids are calculated for all contours in two adjacent images; Find the corresponding contours in the two images based on the position of the center of gravity; Dividing each pair of corresponding contours according to the reconstruction method of the single contour mesh model; All the segmentation results are combined to obtain a multi-contour mesh model.

6. The three-dimensional model reconstruction method based on interactive segmentation and multi-stage fusion according to claim 5, characterized in that: When the number of contours in the contour lists of adjacent images is different, a transition contour grid model of the contour of the target area is reconstructed; specifically, the following steps are performed: When the number of contours in the contour lists of adjacent images is different, it is divided into transition from single contour to multi-contour stage and transition from multi-contour to multi-contour stage; The segmentation results of the two transition stages are combined to obtain the transition contour mesh model.

7. The three-dimensional model reconstruction method based on interactive segmentation and multi-stage fusion according to claim 6, characterized in that: Transition from single profile to multi-profile phase, including: Project the contours in two adjacent images onto the same plane; The contour of the single contour image is enlarged until all contours in the multi-contour image are completely enclosed; Use Delauney triangulation between contours to construct the topological relationship between the contours of the two images; After the construction is completed, the connection relationship of the triangles is kept unchanged, and the coordinates of the contour points are restored to the contour coordinates in the three-dimensional space before projection, so as to obtain the segmentation of the transition stage from single contour to multiple contours.

8. The three-dimensional model reconstruction method based on interactive segmentation and multi-stage fusion according to claim 7, characterized in that: The transition from single profile to multi-profile phase also includes: When the shape of the single contour image cannot completely enclose all contours in the multi-contour image after scaling, segmenting the single contour image according to the concave shape; Align the segmented contours with each contour in the multi-contour image; Each contour in the multi-contour image is projected to the corresponding position of the contour in the single-contour image, and then the single-contour image is scaled to achieve complete wrapping.

9. The three-dimensional model reconstruction method based on interactive segmentation and multi-stage fusion according to claim 8, characterized in that: Transition from multi-contour to multi-contour phases, including: Through contour matching and center of gravity position calculation, the transition problem from multi-contour to multi-contour stage is converted into a multi-contour mesh model reconstruction problem with the same number of contours and a single contour to multi-contour stage transition problem; Obtaining a partial segmentation result of the transition phase according to the reconstruction of the multi-contour mesh model; According to the transition from the single contour to the multi-contour stage, another part of the segmentation results of the transition stage is obtained; The segmentation results are merged to obtain the segmentation of the transition stage from multi-contour to multi-contour.

10. A three-dimensional model reconstruction system based on interactive segmentation and multi-stage fusion, characterized in that: Includes the following modules: Acquisition module: used to acquire the target two-dimensional image of the labeled sequence and perform preprocessing; and then obtain the marking operation results of the target area and non-target area of ​​the preprocessed two-dimensional image; Segmentation module: used to segment all the two-dimensional images after the marking operation using the SAM model to obtain the mask; Extraction module: used for performing morphological operation on the mask and performing contour extraction to obtain a contour list of the target area contour; Reconstruction module: used for reconstructing mesh models of all target area contours at a single contour stage, a multi-contour stage and / or a transition contour stage according to the number of contours in the contour list; Optimization module: used to merge, close the ends and optimize the reconstructed mesh model to obtain the reconstructed target three-dimensional model.

Images