A visualization method and system for realizing two-dimensional and three-dimensional switching

Abstract

This invention discloses a visualization method and system for achieving 2D / 3D switching, relating to the field of medical image processing technology. The method includes: acquiring CT sequence images; performing combined image segmentation on the CT sequence images based on image morphology operations to obtain a binarized image of the lesion site; inputting the binarized image into a pre-trained keypoint selection model to select keypoints; establishing a grid based on the binarized image, and constructing an index coordinate system by combining the grid with the index positions of each keypoint; performing 3D reconstruction of the lesion site and surrounding area based on the index coordinate system and the CT sequence images to obtain a 3D model; and using mixed reality technology to overlay the real lesion location with the 3D model to obtain a superimposed virtual-real image, assisting in surgical procedures. This invention, based on keypoint-based 3D modeling, can accurately locate the lesion site and perform 3D reconstruction, providing doctors with intuitive spatial information.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and more particularly to a visualization method and system for realizing two-dimensional and three-dimensional switching. Background Art

[0002] CT (Computed Tomography) is a computer tomography scan that uses precisely collimated X-ray beams, gamma rays, ultrasound, etc., together with highly sensitive detectors to scan sections of a certain part of the human body one by one. It has the characteristics of fast scanning time and clear images, and can be used to examine a variety of diseases. According to different imaging methods, it can generally be divided into planar scanning imaging and three-dimensional reconstruction imaging.

[0003] In the prior art, in order to achieve a better diagnostic effect, planar scanning imaging and three-dimensional reconstruction imaging are usually used in conjunction to achieve a better positioning effect of the lesion. This type of diagnostic process usually uses a planar CT image to observe the approximate location of the lesion, and then completes the three-dimensional reconstruction of the lesion on the enhanced CT image, so that the doctor can more accurately determine the location of the lesion, thereby improving the success rate of the operation.

[0004] However, in the actual implementation process, it was found that in the existing technology, different scanning devices are used to respectively collect the patient's plain CT images and enhanced CT images for this type of scanning needs. The specifications of the two are obviously not unified, and the plain CT image only corresponds to a certain frame in the Y-axis of the enhanced CT image. Therefore, it is difficult for the computer to process it, resulting in poor three-dimensional reconstruction effect and inability to assist in the precise implementation of surgery. Summary of the invention

[0005] In view of this, the present invention provides a visualization method and system for realizing two-dimensional and three-dimensional switching. By segmenting and extracting CT images, delineating key points, and performing three-dimensional modeling based on the key points, the lesion site can be accurately located and three-dimensional reconstruction can be performed, providing doctors with intuitive spatial information.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A visualization method for realizing two-dimensional and three-dimensional switching, comprising:

[0008] Acquire CT sequence images, and perform combined image segmentation on the CT sequence images based on image morphology operation to obtain a binary image of the lesion site;

[0009] Inputting the binarized image into a pre-trained key point selection model to select key points and obtain key points;

[0010] Establishing a square grid according to the binary image, and constructing an index coordinate system by combining the square grid with the index position of each key point; performing three-dimensional reconstruction of the lesion site and the surrounding area according to the index coordinate system and the CT sequence image to obtain a three-dimensional model;

[0011] Using mixed reality technology, the real lesion location is superimposed on the three-dimensional model to obtain a virtual and real superimposed image to assist in the implementation of the surgery.

[0012] Preferably, the method of obtaining a binary image of the lesion site specifically includes: combining threshold segmentation and region segmentation as a combined image segmentation method for the lesion site, first performing threshold segmentation on the CT sequence images using a variance method, and performing region segmentation on the image after threshold segmentation using a region growing method to obtain the binary image.

[0013] Preferably, before performing combined image segmentation on the CT sequence images based on image morphology operation, the method further includes performing histogram correction and image filtering on the CT sequence images.

[0014] Preferably, the key point selection model specifically includes training the key point selection model by using a CT sequence image with a ratio of the lesion key point coordinates to the lesion image side length as a sample set;

[0015] Obtain CT sequence images with the ratio of the coordinates of the lesion key points to the side length of the lesion image as a sample set, and divide the sample set into a training set and a test set;

[0016] Construct key point selection model and loss function;

[0017] The training set is input into the key point selection model for convolution training to obtain the key point coordinate mean and the key point coordinate variance, and the loss value between the key point coordinate mean and the key point coordinate variance is calculated using the loss function to determine whether the loss value remains unchanged;

[0018] If the loss value changes, the parameters of the key point selection model are adjusted, and the training set is input into the key point selection model for convolution training to obtain the key point coordinate mean and the key point coordinate variance;

[0019] If the loss value remains unchanged, the test set is input into the key point selection model for convolution test to obtain a test result, and whether the test result meets the condition is determined;

[0020] If the test result does not meet the conditions, the parameters of the key point selection model are adjusted until the test result meets the conditions.

[0021] Preferably, the key point selection model includes five convolutional layers, and the head of the convolutional neural network is two fully connected layers with 10 output nodes.

[0022] Preferably, the establishment of the grid specifically includes: according to the major axis length A, the minor axis length B and the height L of the cross section of the lesion site, calculating the number of grids x required for the major axis length A, the minor axis length B and the height L in the three directions of x, y and z respectively. A ,y B and z L , establish a square grid, the origin position of the square grid is (0,0,0), and the length, width and height of the square grid are smaller than A, B and L respectively.

[0023] In a specific embodiment, the construction of the index coordinate system specifically includes: when establishing the square grid, establishing the index of each of the key points to form an index coordinate system; the indexing method of the index coordinate system is that the index x-coordinate positions of the two outermost layers of key points at both ends of the square grid in the x-axis direction are the same, the index y-coordinate positions of the two outermost layers of key points at both ends of the y-axis direction are the same, and the index z-coordinate positions of the two outermost layers of key points at both ends of the z-axis direction are the same.

[0024] A visualization system for realizing two-dimensional and three-dimensional switching, comprising:

[0025] An image segmentation module is used to obtain a CT sequence image, and to perform combined image segmentation on the CT sequence image based on image morphology operation to obtain a binary image of the lesion site;

[0026] A key point acquisition module, inputting the binary image into a pre-trained key point selection model to select key points and acquire key points;

[0027] A three-dimensional model acquisition module is used to establish a grid based on the binary image, and the grid is combined with the index position of each key point to construct an index coordinate system; and three-dimensional reconstruction of the lesion site and the surrounding area is performed based on the index coordinate system and the CT sequence image to obtain a three-dimensional model;

[0028] The implementation module uses mixed reality technology to superimpose the actual lesion location with the three-dimensional model to obtain a virtual and real superimposed image to assist in the implementation of the surgery.

[0029] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a visualization method and system for realizing two-dimensional and three-dimensional switching. By performing combined image segmentation, key point selection, and establishing a grid and index coordinate system on CT sequence images, the lesion site can be accurately located and three-dimensional reconstruction can be performed to provide doctors with intuitive spatial information; the real lesion position is superimposed on the three-dimensional model by using mixed reality technology to realize a virtual and real superimposed picture, which can effectively assist doctors in performing surgery and improve the accuracy and safety of surgery; through detailed image processing processes such as threshold segmentation, region segmentation, and image filtering, the clarity and quality of the image can be improved, and the convolutional neural network is used for training and testing, which can effectively select key points and improve the accuracy and stability of the selection. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0031] Figure 1 A schematic diagram of the method steps provided by the present invention;

[0032] Figure 2 This is a schematic diagram of the system structure provided by the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] The embodiment of the present invention discloses a visualization method for realizing two-dimensional and three-dimensional switching. Figure 1 As shown, including:

[0035] Acquire CT sequence images, and perform combined image segmentation on the CT sequence images based on image morphology operations to obtain a binary image of the lesion site;

[0036] Input the binary image into the pre-trained key point selection model to select key points and obtain key points;

[0037] A square grid is established based on the binary image, and an index coordinate system is constructed by combining the square grid with the index position of each key point; a three-dimensional reconstruction of the lesion site and the surrounding area is performed based on the index coordinate system and the CT sequence images to obtain a three-dimensional model;

[0038] Using mixed reality technology, the real lesion location is superimposed on the three-dimensional model to obtain a virtual and real superimposed image to assist in the implementation of the surgery.

[0039] In a specific embodiment, when mixed reality technology is used to assist in the implementation of surgery, mixed reality glasses are used to guide the surgery through real-time CT to obtain changes in the lesion site and adjust the three-dimensional model in real time.

[0040] Among them, superimposing the actual lesion location with the three-dimensional model specifically includes:

[0041] The SIFT operator is used to calculate the features of each pixel point of the CT image to form multiple images with physical points. The CT images are named according to the physical points, and the correspondence between the pixel feature points and the physical points is established. The correspondence is presented in a binary group: <pixel feature point, physical point>;

[0042] The obtained pixel feature points and 3D model key points are presented in a binary pair: <pixel feature point, key point>;

[0043] According to <pixel feature points, physical points> and <pixel feature points, key points>, a corresponding relationship among the three is constructed: <pixel feature points, key points, physical points>; according to the above-constructed <pixel feature points, key points, physical points> relationship, the physical points and the three-dimensional model are matched.

[0044] In order to avoid the incompleteness of the virtual and real superimposed images caused by network factors, a defect repair network can be set up inside the mixed reality glasses to repair the incompleteness of the superimposed images. The architecture of the model is as follows:

[0045] The incomplete patching network contains 17 layers of 3D convolutions and adopts a U-shape architecture to fuse multi-scale features. It is a U-shaped network, which includes an encoder and a decoder. The decoder and the encoder each include 4 groups of convolution blocks, and each convolution block includes 2 convolution layers; in the encoder stage, the 3D input is downsampled by 2 times each time it passes through a convolution block; in the decoder stage, it is upsampled by 2 times each time it passes through a convolution block, and the output of the corresponding block of the encoder is fused at the same time; the incomplete virtual and real superimposed images are sent to the incomplete patching network, and the output of the incomplete patching network is compared with the complete three-dimensional model to calculate the loss; the loss function is the L2 loss of the output of the incomplete patching network and the complete three-dimensional model; the incomplete patching network is optimized according to the loss function.

[0046] In a specific embodiment, the process further includes converting the three-dimensional model into a two-dimensional image through an index coordinate system.

[0047] In a specific embodiment, obtaining a binary image of the lesion site specifically includes: combining threshold segmentation and region segmentation as a combined image segmentation method for the lesion site, first performing threshold segmentation on the CT sequence image using the variance method, and then performing region segmentation on the image after threshold segmentation using the region growing method to obtain a binary image.

[0048] Assume that the lesion site obtained by variance segmentation is C, and the average grayscale W of site C is calculated. C , the calculation formula is as follows:

[0049]

[0050] Among them, f(i,j) is the gray value of each pixel in the part C, N C is the total number of pixels in part C;

[0051] Calculate the average grayscale W of part C C The difference ΔP from the optimal threshold P;

[0052] Region segmentation, with gray value W C The pixel point is used as the initial "seed" of the region growing method, and ΔP is used as the threshold of the similarity criterion, that is, the pixels whose grayscale value differs from the "seed" by less than ΔP are considered to be similar to the "seed" and are merged with the "seed" region. The image is segmented twice using the region growing method, and the combined segmentation method is used to segment the CT sequence images.

[0053] In a specific embodiment, before performing combined image segmentation on the CT sequence images based on image morphology operation, the method further includes performing histogram correction and image filtering on the CT sequence images.

[0054] Histogram correction is to perform image enhancement processing on the image based on histogram equalization. The specific steps are as follows:

[0055] 1. List the grayscale levels r in the grayscale image of the original CT image k , k = 0, 1, 2..., L-1, where L is the total number of gray levels in the image;

[0056] 2. Count the number of pixels n at each gray level k , k=0,1,2...,L-1;

[0057] 3. For an image f(x,y), the total number of pixels is N, use r k Indicates the grayscale corresponding to the kth grayscale level, n k Represents grayscale r k The number of pixels, according to the size of the gray value, count the frequency of each gray level p(r k), expressed mathematically as:

[0058]

[0059] 4. Calculate the cumulative distribution function of gray level k=0,1,2...L-1;

[0060] 5. Determine the grayscale level s of the output image k , k=0,1,2...,L-1, the calculation formula is as follows:

[0061] s k = int[(L-1)*C(r k )+0.5];

[0062] 6. Determine the gray level r of the input image k To the output image gray level s k The corresponding relationship (r k →s k );

[0063] 7. Count the number of pixels at each gray level in the new histogram n k , k=0,1,2...,L-1;

[0064] 8. Calculate the histogram according to step 3, and according to the corresponding relationship in step 6 (r k →s k ) obtain an output image;

[0065] Image filtering is median filtering. First, determine a template of size m×n. The pixels covered by the template are sorted from small to large in grayscale value, and the middle grayscale value is used to replace the original grayscale value. The specific steps are as follows:

[0066] 1. Set a template with an odd size and traverse the entire image in a certain order;

[0067] 2. Read the grayscale value of each pixel in the image sub-block corresponding to the template;

[0068] 3. Arrange these gray values ​​from small to large;

[0069] 4. Assign the gray value in the middle to the pixel corresponding to the center of the template. The mathematical expression is:

[0070]

[0071] Among them, s is a neighborhood centered on the pixel point (a, b);

[0072] In a specific embodiment, the key point selection model specifically includes training the key point selection model by using a CT sequence image with a ratio of the lesion key point coordinates to the lesion image side length as a sample set;

[0073] Obtain CT sequence images with the ratio of the coordinates of the lesion key points to the side length of the lesion image as a sample set, and divide the sample set into a training set and a test set;

[0074] Construct key point selection model and loss function;

[0075] The training set is input into the key point selection model for convolution training to obtain the key point coordinate mean and key point coordinate variance. The loss value between the key point coordinate mean and key point coordinate variance is calculated using the loss function to determine whether the loss value remains unchanged.

[0076] If the loss value changes, adjust the parameters of the key point selection model, and input the training set into the key point selection model for convolution training to obtain the key point coordinate mean and key point coordinate variance;

[0077] If the loss value remains unchanged, the test set is input into the key point selection model for convolution test to obtain the test result and determine whether the test result meets the conditions;

[0078] If the test result does not meet the conditions, the parameters of the key point selection model are adjusted until the test result meets the conditions.

[0079] In a specific embodiment, the key point selection model includes five convolutional layers, and the head of the convolutional neural network is two fully connected layers with 10 output nodes. Only a simple 5-layer convolutional network can achieve a good quality assessment effect, and the convolutional neural network converges very quickly. Taking 5 key points as an example, considering the horizontal and vertical coordinates, 10 means and 10 variances need to be learned. In order to stabilize the gradient, log(σ 2 ) instead of variance σ 2 , so the network head is two pairs of fully connected layers, and the number of output nodes in each pair of fully connected layers is 10.

[0080] In a specific embodiment, establishing a grid specifically includes: calculating the number of grids x required for the major axis length A, the minor axis length B and the height L in the x, y and z directions respectively according to the major axis length A, the minor axis length B and the height L of the cross section of the lesion site. A ,y B and z L , establish a square grid with the origin of the grid at (0,0,0) and the length, width and height of the grid smaller than A, B and L respectively.

[0081] In a specific embodiment, constructing an index coordinate system specifically includes: when establishing a square grid, establishing an index of each key point thereof to form an index coordinate system; the indexing method of the index coordinate system is that the index x-coordinate positions of the two outermost key points at both ends of the square grid in the x-axis direction are the same, the index y-coordinate positions of the two outermost key points at both ends in the y-axis direction are the same, and the index z-coordinate positions of the two outermost key points at both ends in the z-axis direction are the same.

[0082] A visualization system that realizes two-dimensional and three-dimensional switching, such as Figure 2 As shown, including:

[0083] An image segmentation module obtains CT sequence images, and performs combined image segmentation on the CT sequence images based on image morphology operations to obtain a binary image of the lesion site;

[0084] The key point acquisition module inputs the binary image into the pre-trained key point selection model to select key points and obtain key points;

[0085] The three-dimensional model acquisition module establishes a grid based on the binary image, and constructs an index coordinate system by combining the grid with the index position of each key point; the three-dimensional reconstruction of the lesion site and the surrounding area is performed based on the index coordinate system and the CT sequence images to obtain a three-dimensional model;

[0086] The implementation module uses mixed reality technology to superimpose the actual lesion location with the three-dimensional model to obtain a virtual and real superimposed image to assist in the implementation of the surgery.

[0087] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0088] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A visualization method for realizing two-dimensional and three-dimensional switching, characterized in that: include: Acquire CT sequence images, and perform combined image segmentation on the CT sequence images based on image morphology operation to obtain a binary image of the lesion site; Inputting the binarized image into a pre-trained key point selection model to select key points and obtain key points; Establishing a square grid according to the binary image, and constructing an index coordinate system by combining the index position of each key point with the square grid; Perform three-dimensional reconstruction of the lesion site and surrounding areas according to the index coordinate system and the CT sequence images to obtain a three-dimensional model; Using mixed reality technology, the real lesion location is superimposed on the three-dimensional model to obtain a virtual and real superimposed image to assist in the implementation of the surgery.

2. A visualization method for realizing two-dimensional and three-dimensional switching according to claim 1, characterized in that: The method of obtaining a binary image of the lesion site specifically includes: combining threshold segmentation and region segmentation as a combined image segmentation method for the lesion site, first performing threshold segmentation on the CT sequence images using a variance method, and performing region segmentation on the image after threshold segmentation using a region growing method to obtain the binary image.

3. The method for realizing two-dimensional and three-dimensional switching visualization according to claim 1, characterized in that: Before performing combined image segmentation on the CT sequence images based on image morphology operation, the method further includes performing histogram correction and image filtering on the CT sequence images.

4. The method for realizing two-dimensional and three-dimensional switching visualization according to claim 1, characterized in that: The key point selection model specifically includes training the key point selection model by using a CT sequence image with a ratio of the lesion key point coordinates to the lesion area side length as a sample set; Obtain CT sequence images with the coordinates of the lesion key points and the ratio of the side length of the lesion area as a sample set, and divide the sample set into a training set and a test set; Construct key point selection model and loss function; The training set is input into the key point selection model for convolution training to obtain the key point coordinate mean and the key point coordinate variance, and the loss value between the key point coordinate mean and the key point coordinate variance is calculated using the loss function to determine whether the loss value remains unchanged; If the loss value changes, the parameters of the key point selection model are adjusted, and the training set is input into the key point selection model for convolution training to obtain the key point coordinate mean and the key point coordinate variance; If the loss value remains unchanged, the test set is input into the key point selection model for convolution test to obtain a test result, and whether the test result meets the condition is determined; If the test result does not meet the conditions, the parameters of the key point selection model are adjusted until the test result meets the conditions.

5. The method for realizing two-dimensional and three-dimensional switching visualization according to claim 4, characterized in that: The key point selection model includes five convolutional layers, and the head of the convolutional neural network is two fully connected layers with 10 output nodes.

6. The method for realizing two-dimensional and three-dimensional switching visualization according to claim 1, characterized in that: The establishment of the grid specifically includes: calculating the number of grids required for the major axis length A, the minor axis length B and the height L in the x, y and z directions respectively according to the major axis length A, the minor axis length B and the height L of the cross section of the lesion site; , and , establish a square grid, the origin position of the square grid is (0,0,0), and the length, width and height of the square grid are smaller than A, B and L respectively.

7. The method for realizing two-dimensional and three-dimensional switching visualization according to claim 6, characterized in that: The construction of the index coordinate system specifically includes: when establishing the square grid, establishing the index of each of the key points to form an index coordinate system; the indexing method of the index coordinate system is that the index x-coordinate positions of the two outermost key points at both ends of the square grid in the x-axis direction are the same, the index y-coordinate positions of the two outermost key points at both ends of the y-axis direction are the same, and the index z-coordinate positions of the two outermost key points at both ends of the z-axis direction are the same.

8. A visualization system for realizing two-dimensional and three-dimensional switching, applied to a visualization method for realizing two-dimensional and three-dimensional switching according to any one of claims 1 to 7, characterized in that: include: An image segmentation module is used to obtain a CT sequence image, and to perform combined image segmentation on the CT sequence image based on image morphology operation to obtain a binary image of the lesion site; A key point acquisition module, inputting the binary image into a pre-trained key point selection model to select key points and acquire key points; A three-dimensional model acquisition module is used to establish a square grid based on the binary image, and the square grid is combined with the index position of each key point to construct an index coordinate system; Perform three-dimensional reconstruction of the lesion site and surrounding areas according to the index coordinate system and the CT sequence images to obtain a three-dimensional model; The implementation module uses mixed reality technology to superimpose the actual lesion location with the three-dimensional model to obtain a virtual and real superimposed image to assist in the implementation of the surgery.

Images