A method for automatic recognition and model reconstruction of anatomical features in medical images

Through segmentation network and shape model comparison technology, the anatomical structure in two-dimensional images is automatically identified and the three-dimensional model is reconstructed, which solves the errors of two-dimensional image recognition and three-dimensional reconstruction and frequent CT scans in the existing technology, and improves the clinical application Efficiency and accuracy.

CN112037200BActive Publication Date: 2024-11-22SHANGHAI TAOIMAGE MEDICAL TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202010896593.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-08-31
Publication Date
2024-11-22
Estimated Expiration
2040-08-31

AI Technical Summary

Technical Problem

In the prior art, the automatic identification and three-dimensional model reconstruction of two-dimensional images have dependencies on key point detection and shape constraint steps, resulting in large errors in the result, and the two-dimensional images lack depth information, making it difficult to directly reconstruct an accurate three-dimensional model, especially in Frequent use of CT scans is not suitable for clinical applications.

Method used

The pixel-level classification of two-dimensional images through a pre-trained segmentation network, extract key points and contour lines of anatomical structures, and compare shape models using images from different perspectives to generate accurate three-dimensional models, reducing manual marking and CT scans need.

Benefits of technology

It realizes automatic identification of two-dimensional images and accurate key point extraction, which reduces the work burden of doctors, reduces the frequency of CT scans, improves the reconstruction efficiency and accuracy of the three-dimensional model, and is suitable for clinical and operating room environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112037200B_ABST
    Figure CN112037200B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of intelligent analysis of medical images, and provides a method for automatic recognition and model reconstruction of anatomical features in medical images, including: S1: inputting a two-dimensional image into a segmentation network for prediction, and outputting a prediction result map corresponding to the two-dimensional image; S2: performing morphological analysis on each anatomical structure segmented in the prediction result map, and extracting the position and contour of key points; S3: reading the contour line, and comparing it with the shape model. When the projection contour simulated by the shape model is similar to the contour line, the current shape model is a reconstruction of the real anatomical structure; S4: using at least two two-dimensional images from different perspectives, executing the above steps S1-S3, and reconstructing a three-dimensional model of the anatomical structure. The above technical solution can automatically identify the anatomical feature points and contour lines of a two-dimensional image, and can reconstruct the best three-dimensional model through at least two two-dimensional images from different perspectives.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent analysis of medical images, and in particular to a method for automatic recognition and model reconstruction of anatomical features in medical images. Background Art

[0002] Two-dimensional images, including high-precision X-ray images, are very important image references in clinical diagnosis and scientific research. With the help of two-dimensional anteroposterior images, doctors can quantitatively calculate some joint angles, joint distances, and perform alignment analysis by manually circling some anatomical key points and contour lines, thereby providing preoperative planning for prosthetic replacement surgery and providing a numerical basis for postoperative evaluation of effects. However, manually marking a large number of key points and lines on the image is a huge burden for doctors, and the calculation based on the results of manual marking is also very time-consuming, labor-intensive, and prone to errors. If an algorithm can be designed to automate the above process, it will greatly reduce the burden on doctors.

[0003] like Figure 1 As shown in the figure, in the prior art, segmentation of two-dimensional images generally includes two steps: key point detection and shape constraint. After inputting an image, we first detect the positions of some predetermined key points on the contour line, and then use the detected key points as constraint factors of the shape model to predict the position and shape of the contour line using the statistical shape model. Such a strategy is a cascade structure. The main problem is that the output of the previous step will affect the result of the second step. That is, if the key point prediction in the first step is slightly biased, the shape output in the end will have a large error.

[0004] Furthermore, since two-dimensional images are images in a single direction, the depth information in the projection direction will be compressed. Therefore, there is bound to be a problem of information loss in two-dimensional images, and the measurement results based on two-dimensional images are only a simplification of the real information in three-dimensional space. However, doctors are still very willing to understand the angular distance relationship between joints, or the relationship between the curvature of the joint contour line and the gravity line in the real three-dimensional space. In the existing processing methods, if a relatively accurate three-dimensional image is needed, a CT scan of the patient is required. However, the CT machine is relatively large and is not suitable for occasions such as operating rooms. The radiation dose of taking a CT image is huge, and it is not suitable for application scenarios similar to those in surgery that require multiple scans. If a three-dimensional image can be directly reconstructed from a two-dimensional image, it will bring great convenience to doctors.

[0005] In fact, the identification of key points and contours in a two-dimensional plane is closely related to the problem of three-dimensional reconstruction. Correctly identified contours and key points are important inputs for a good three-dimensional reconstruction algorithm. Therefore, if a computer algorithm can be developed that can automatically identify key points and contours and reconstruct a three-dimensional model based on them without user intervention, it will be of great clinical value. However, the main problems now are:

[0006] (1) Mainstream algorithm research often focuses on one of the two problems: two-dimensional plane key point recognition and three-dimensional reconstruction, but does not explore the effect of the final reconstruction model in the case of joint reconstruction.

[0007] (2) Some algorithm studies use membrane imaging data. This type of image is less noisy than real bones, is cleaner and clearer, and is easier to process. However, the distortion of the image itself determines the limitations of the algorithm.

[0008] (3) Most algorithm studies focus on feature recognition and reconstruction of a single bone, lacking in versatility and coverage. Summary of the invention

[0009] In view of the above problems, the purpose of the present invention is to provide a method for automatic recognition and model reconstruction of anatomical features in medical images, which can automatically recognize anatomical feature points and contour lines on two-dimensional images, and can reconstruct the best three-dimensional model through at least two two-dimensional images at different viewing angles. It is particularly suitable for the recognition and three-dimensional reconstruction of lower limbs, which account for a relatively high proportion in clinical research in orthopedics, and can perfectly cover the femur, tibia, and fibula of the lower limbs, and has strong clinical practicality.

[0010] The above-mentioned object of the present invention is achieved through the following technical solutions:

[0011] A method for automatically identifying anatomical features and reconstructing models in medical images comprises the following steps:

[0012] S1: Input the two-dimensional image to be segmented into the segmentation network for prediction, agree in advance on the type of anatomical structure to be identified, and make classification prediction for each pixel of the input two-dimensional image to form a prediction result graph, wherein the segmentation network is obtained by pre-collecting a large number of the two-dimensional images, annotating them at the pixel level, and then training the neural network;

[0013] S2: performing morphological analysis on each anatomical structure segmented in the prediction result image, extracting the positions of key points in the anatomical structure using an image processing algorithm, and extracting boundary pixels of the anatomical structure as the contour line of the anatomical structure;

[0014] S3: reading the contour line and comparing it with a plurality of groups of shape models generated using different weight factors. When the projection contour simulated by the shape model is close to the contour line, the current shape model is a reconstruction of the real anatomical structure, wherein the generation of the shape model needs to rely on a model database based on CT artificial reconstruction;

[0015] S4: Using at least two of the two-dimensional images at different viewing angles, execute the above steps S1-S3 to reconstruct a three-dimensional model of the anatomical structure.

[0016] Furthermore, in step S1, the segmentation network is obtained by pre-collecting a large number of the two-dimensional images and performing pixel-level annotation and neural network training. The specific training process is:

[0017] Setting different labels for different anatomical structures in the two-dimensional image;

[0018] Using pixel annotation tools, each pixel in each collected two-dimensional image is labeled differently according to different anatomical structures, and each different label is overlaid on the pixel with a layer of different color, forming a two-dimensional annotated image that uses different colors to represent the type of anatomical structure to which each pixel belongs;

[0019] The data set of the two-dimensional annotated images is classified and divided into a training set and a validation set. The training set is used to train the neural network to form the segmentation network, and the validation set is used to train the network to perform well in the face of unknown data.

[0020] Furthermore, the dataset of the two-dimensional annotated image is classified, specifically:

[0021] Checking the image quality of the two-dimensional annotated image, classifying the image into multiple types including clear, general, and blurred according to the clarity of the image, and classifying each of the two-dimensional annotated images;

[0022] For the two-dimensional annotated images of each definition, randomly select some images to add to the training set, and the rest to add to the verification set;

[0023] Using the two-dimensional annotated images in the training set to perform training to generate the segmentation network;

[0024] The segmentation network is validated using the two-dimensional annotated images in the validation set.

[0025] Furthermore, in step S1, the prediction result graph is further processed, specifically:

[0026] De-noising the predicted result image, and performing connectivity analysis on all anatomical structure types except the background type and all pixels classified into this type. If multiple connected areas are analyzed, find and retain the largest connected area, and remove other connected areas;

[0027] Morphological operations are used on the connected areas to remove small jagged points, which is used as the final prediction result graph.

[0028] Furthermore, in step S3, the specific generation process of the shape model is as follows:

[0029] For each collected 3D image, a structural model of each anatomical structure is reconstructed to form a structural model database specific to each anatomical structure;

[0030] Each structural model is resampled according to the standard sampling model, so that the structural models reconstructed by different people can be normalized to a standard shape model;

[0031] The coordinates of all nodes of each structural model are combined into a node vector for describing the structural model, a covariance matrix is ​​constructed for the node vectors of all structural models corresponding to each anatomical structure, and the eigenvalues ​​and eigenvectors of the covariance matrix are calculated;

[0032] The shape model is generated using the average model and eigenvalues ​​and eigenvectors calculated from all structure models of each anatomical structure, using different weighting factors.

[0033] Furthermore, each structural model is resampled according to the standard sampling model. The specific steps are as follows:

[0034] Each part of the structural model is resampled using hemisphere fitting. Rays are emitted from the center of the hemisphere fitting and intersect with the surface patches of the corresponding part. The intersection point is a resampling node.

[0035] Furthermore, the coordinates of all nodes of each structural model are combined into a node vector for describing the structural model, a covariance matrix is ​​constructed for the node vectors of all structural models corresponding to each anatomical structure, and the eigenvalues ​​and eigenvectors of the covariance matrix are calculated, specifically:

[0036] All node coordinates of each structural model are combined into a (3*n,1)-dimensional vector;

[0037] The covariance matrix is ​​constructed for the node vectors of all structural models corresponding to each anatomical structure:

[0038]

[0039] Where p is the number of structural models corresponding to the anatomical structure, n is the number of nodes in each structural model, S i is a vector representation of a structural model, is the average model of all structural models;

[0040] By calculating the eigenvalues ​​and eigenvectors of the covariance matrix, we can obtain:

[0041]

[0042] Among them, λ is the eigenvalue, arranged from large to small; P is the eigenvector, and s is the number of eigenvalues ​​and eigenvectors used.

[0043] Furthermore, the shape model is generated using the average model, eigenvalues ​​and eigenvectors calculated using all the structural models of each anatomical structure, and different weight factors are used, specifically:

[0044] ,

[0045] Among them, a j is a weight factor for generating the shape model.

[0046] Further, in step S3, the contour line is read and compared with a plurality of groups of shape models generated using different weight factors. When the projection contour simulated by the shape model is close to the contour line, the current shape model is a reconstruction of the real anatomical structure, specifically:

[0047] reading the contour line;

[0048] Reading environmental parameters at the time of photography from the original two-dimensional image;

[0049] In a photographic environment corresponding to the environmental parameters, using the shape model to simulate a projection contour;

[0050] The simulated projection contour is compared with the contour line segmented from the actual image. When the projection contour is close to the contour line, the current shape model is a reconstruction of the real anatomical structure. The optimization process is expressed by the following formula:

[0051]

[0052] Wherein, L1 is the simulated projection contour, L2 is the contour line segmented from the actual image, and when F is close to 0, the current shape model is a reconstruction of the real anatomical structure.

[0053] Furthermore, when reconstructing the 3D model of the anatomical structure, the optimization process is formulated as follows:

[0054]

[0055] Wherein, k represents the viewing angle, n represents the number of viewing angles, and when F is close to 0, the current shape model is a reconstruction of the real anatomical structure.

[0056] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0057] (1) By inputting the two-dimensional image to be segmented into the segmentation network for prediction, the type of anatomical structure to be identified is agreed in advance, and each pixel of the input two-dimensional image is classified and predicted to form a prediction result map. Furthermore, by performing morphological analysis on each anatomical structure segmented in the prediction result map, the key points and contours in the anatomical structure are extracted. The above technical solution realizes the automatic recognition of two-dimensional images, and can automatically obtain key points and contours, saving doctors' time and reducing their burden. The calculation results are more accurate, leaving doctors with more time to complete more important work. For application scenarios where time is very precious, such as surgery, after taking the two-dimensional image, the key points and contours can be seen in a very short time through automatic recognition by the computer, which wins precious time for treating patients.

[0058] (2) Based on the technical solution described in (1), the prior art has gotten rid of the problem that the segmentation of two-dimensional images is generally carried out through two steps: key point detection and shape constraint. The output result of the previous step will affect the result of the second step. If the key point prediction of the first step is slightly deviated, the output shape will have a large error. The technical solution of the present invention first obtains a prediction result map containing different labels by labeling each anatomical structure, and then performs morphological analysis on each anatomical structure distinguished by the label to obtain key points and contour lines. The acquisition of key points and contour lines can be carried out independently and simultaneously, and the two steps will not affect each other, ensuring the accuracy of the key points and contour lines obtained. At the same time, obtaining accurate key points and contour lines provides important input for the subsequent reconstruction of the three-dimensional model.

[0059] (3) By establishing a shape model, the shape model is compared with the contour extracted from the two-dimensional image. When the projection contour simulated by a certain group of shape models is close to the contour, the current shape model can be considered to be a reconstruction of the real anatomical structure. Furthermore, by using at least two two-dimensional images from different perspectives, a three-dimensional model of the anatomical structure can be reconstructed. The above technical solution, by using at least two two-dimensional images from different perspectives, can quickly reconstruct a three-dimensional model through an algorithm, which will have great significance in clinical applications. For example: when it is necessary to obtain a three-dimensional image, the existing processing method is generally to perform a CT scan on the patient. However, the CT machine is relatively large and requires a dedicated CT room for CT scanning. During the scan, other operations cannot be performed, which is not suitable for the scene of the operating room. At the same time, the radiation received by the human body during a CT scan is huge. If the scan is too frequent, it will cause harm to the human body. For the above problem, through the three-dimensional reconstruction method of the present invention, only at least two two-dimensional images need to be scanned to reconstruct the three-dimensional model, which can be applied to scenes such as surgery and outpatient clinics. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A diagram showing the steps of segmenting a two-dimensional image in the prior art;

[0061] Figure 2 This is an overall flow chart of a method for automatic recognition and model reconstruction of anatomical features in medical images according to the present invention;

[0062] Figure 3 It is a schematic diagram of an example of marking the sciatic tuberosity node of the pelvis in the key point marking of the present invention;

[0063] Figure 4 It is a schematic diagram of an example of marking the greater trochanter node of the femur in the key point marking of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0066] For two-dimensional images, doctors usually manually mark key points and lines on the images, which is time-consuming, labor-intensive, and prone to errors. In the prior art, two-dimensional image segmentation is generally performed through two steps: key point detection and shape constraint. The output result of the first step will affect the result of the second step. If the key point prediction of the first step is slightly deviated, the output shape will have a large error. Furthermore, in the prior art, if three-dimensional images need to be obtained, they need to be obtained through CT scanning. The CT machine is large in size and has a huge amount of radiation, so it cannot be scanned frequently.

[0067] Based on the above application scenarios and technical problems, the core idea of ​​the present invention is: to provide a method for automatic recognition and model reconstruction of anatomical features in medical images, to identify the two-dimensional image to be segmented through a pre-built segmentation network, and the segmentation network automatically outputs a prediction result map, and extracts key points and contour lines for each anatomical structure segmented in the prediction result map. A shape model is established and compared with the contour lines extracted from the two-dimensional image to achieve reconstruction of the real anatomical structure. Furthermore, a three-dimensional model can be reconstructed through at least two two-dimensional images from different perspectives.

[0068] The significant features of the present invention are: (1) without the need to segment the two-dimensional image through the two steps of key points and shape constraints in the prior art, the key points and contours can be extracted simultaneously from the prediction result map output by the segmentation network. (2) Through at least two two-dimensional images at different perspectives, the reconstruction of the three-dimensional space can be achieved.

[0069] First embodiment

[0070] like Figure 2 As shown, this embodiment provides a method for automatic recognition and model reconstruction of anatomical features in medical images, comprising the following steps:

[0071] S1: Input the two-dimensional image to be segmented into the segmentation network for prediction, agree in advance on the type of anatomical structure to be identified, make classification prediction for each pixel of the input two-dimensional image, and form a prediction result graph, wherein the segmentation network is obtained by pre-collecting a large number of the two-dimensional images, annotating them at the pixel level, and then training the neural network.

[0072] 1. Predict and mark the two-dimensional image to be segmented into the segmentation network

[0073] Specifically, in this embodiment, the purpose of step S1 is to predict and mark each different type of anatomical structure in the two-dimensional image with different labels, so that each anatomical structure can be distinguished at a glance on the processed prediction result map. Among them, each bone and each soft tissue structure in the two-dimensional image is an anatomical structure, which needs to be distinguished with different labels, and even the empty area without bones and soft tissues needs to be distinguished with a separate different label.

[0074] It should be noted that the purpose of this step is to distinguish anatomical structures. The label mentioned in this embodiment is only a preferred solution. All solutions that can distinguish anatomical structures in two-dimensional images belong to the protection scope of the present invention. For labels used for marking, the present invention does not limit the form of expression of the labels. Different labels can be distinguished by using different colors, different grayscales of the same color, different symbols, etc., which will not be elaborated in the present invention. Preferably, in clinical applications, since colors are more clearly distinguished in two-dimensional images, different colors can be used to correspond to different labels.

[0075] For labels, when annotating anatomical structures in two-dimensional images, since pixels are the smallest units in an image, each pixel can be annotated, the anatomical structure to which the current pixel belongs can be determined, the corresponding label can be obtained for annotation, and different label representations can be used to overlay the pixel. Since the pixels of each anatomical structure are concentrated together, after the two-dimensional image is annotated, each anatomical structure can be easily distinguished on the prediction result map through different label representations.

[0076] Furthermore, the pixels may be labeled using integers, letters, etc., and a correspondence may be formed between the labels and the labels actually used, and stored in a separate file or a table in a database to facilitate query and result display.

[0077] For the labeling method, a pixel labeling tool can be used to label the category of each pixel in the two-dimensional image with different labels. The present invention does not impose any restrictions on the pixel labeling tool, and any existing and subsequently developed tools that can perform pixel labeling can be applied to the present invention. For example, the pixel labeling tool built into the imageLabeler app of matlab can be used to label pixels.

[0078] 2. Training of Segmentation Network

[0079] Furthermore, how to correctly determine the category of each pixel and label each pixel correctly is achieved by training the segmentation network. When the two-dimensional image to be segmented is input into the trained segmentation network, the category of each pixel in the two-dimensional image to be segmented can be accurately distinguished through the data in the trained segmentation network. The specific training steps of the segmentation network are:

[0080] S11: Using the label setting method mentioned above, different labels are set for different anatomical structures in the two-dimensional image.

[0081] S12: Use the pixel annotation tool to label each pixel in each collected two-dimensional image differently according to different anatomical structures, and overlay each different label on the pixel with a layer of different color, so as to form a two-dimensional annotated image that uses different colors to represent the type of anatomical structure to which each pixel belongs.

[0082] Specifically, for the two-dimensional image of the part to be segmented of the human body, as many existing two-dimensional images of the same part to be segmented as possible are collected, and each of the collected two-dimensional images is marked and analyzed.

[0083] S13: Classify the data set of the two-dimensional annotated image and divide it into a training set and a validation set. The training set is used to train the neural network to form the segmentation network, and the validation set is used to train the network's performance in the face of unknown data.

[0084] (1) Classifying the dataset of the two-dimensional annotated image, specifically:

[0085] The image quality of the two-dimensional annotated image is checked, and each of the two-dimensional annotated images is classified into multiple types including clear, general, and blurred according to the clarity of the image; for the two-dimensional annotated images of each clarity, some images are randomly selected to add to the training set, and the rest are added to the verification set; the two-dimensional annotated images in the training set are used for training to generate the segmentation network; and the two-dimensional annotated images in the verification set are used to verify the segmentation network.

[0086] According to the experiments that have been conducted, for each definition of the two-dimensional annotated images, 60% of the images are selected to be added to the training set, and the rest are added to the validation set, and the trained segmentation network is the best. Of course, the value of 60% is only an optimal solution. In actual applications, different proportions of images can be selected to be added to the training set according to one's own choice, and the rest are added to the validation set. The present invention does not impose any restrictions.

[0087] The purpose of this classification is to reasonably divide the collected two-dimensional images, one part for training and one part for verification, to ensure the accuracy of the final segmentation network.

[0088] (2) About the process of training the segmentation network

[0089] Step S13 also includes the process of training the segmentation network. Generally, the segmentation network is trained through a neural network. Of course, the present invention is not limited to the training method of the neural network. All methods that can train a large amount of existing data to achieve the training purpose of the present invention can be applied to the present invention.

[0090] The present invention does not limit the method of training the segmentation network by a neural network. For example, a hybrid neural network of Unet and deeplabV3 can be built, a mixed loss function of diceLoss and activeContourLoss can be used, the segmentation network can be trained using the adam optimizer, and an early stopping strategy can be used to decide when to stop training.

[0091] Furthermore, each time the segmentation network is used to segment the two-dimensional image to be segmented, it is not only a process of segmenting the two-dimensional image, but also a process of continuously completing the segmentation network. As the number of two-dimensional images used for training in the segmentation network continues to increase, the subsequent segmentation process will become more and more accurate.

[0092] When a new two-dimensional image needs to be segmented, the segmentation network is input, and the segmentation network can output a corresponding prediction result map.

[0093] S2: Performing morphological analysis on each anatomical structure segmented in the prediction result image, extracting the positions of key points in the anatomical structure using an image processing algorithm, and extracting boundary pixels of the anatomical structure as the contour line of the anatomical structure.

[0094] Specifically, for the prediction result map predicted by step S1, key points and contour lines can be extracted respectively, which saves the trouble of manual labeling by doctors and provides important input for the subsequent reconstruction of the three-dimensional model.

[0095] 1. Extract key points

[0096] For each segmented anatomical structure, the position is generally relatively fixed, and the position of the key points can be further extracted through basic image processing methods.

[0097] For example, when different colors are used as labels, the segmentation result of each anatomical structure is a binary image, and the position of the key points can be extracted using Hough circle detection.

[0098] Furthermore, the positions of key points are not fixed, and different key points will be marked corresponding to different scenes and different anatomical structures.

[0099] Here are two examples of key point marking:

[0100] 1. Pelvic ischial tuberosity

[0101] like Figure 3 As shown in FIG. 1 , the sciatic node of the pelvis is the lowest end point of the pelvis in the frontal view and has the property of a convex corner point. The convex corner point detection is used on the pelvic binary image and the corner point with the largest y coordinate is selected to locate this marker point.

[0102] 2. Greater trochanter of femur

[0103] like Figure 4 As shown, the greater trochanter of the femur is the outermost corner point of the femur. The convex corner point detection is performed on the binary images of the femurs on both sides, and the outermost result is selected to locate this marking point.

[0104] 2. Extracting Contour Lines

[0105] By extracting the boundary pixels representing the projection boundary of each anatomical structure, the contour line of each anatomical structure is formed, which can help the subsequent reconstruction of the three-dimensional model.

[0106] For example, using different colors as labels, in the context of labels marked with different colors, each anatomical structure is a binary image, which can be tracked by the Moore neighbor tracking algorithm to obtain the contour of the specified anatomical structure.

[0107] S3: Read the contour line and compare it with multiple groups of shape models generated using different weight factors. When the projection contour simulated by the shape model is close to the contour line, the current shape model is a reconstruction of the real anatomical structure, wherein the generation of the shape model needs to rely on a model database based on CT manual reconstruction.

[0108] 1. Training the shape model

[0109] Before reconstructing the 3D model, the shape model needs to be trained in advance, specifically:

[0110] S311: For each collected three-dimensional image, a structural model of each anatomical structure is reconstructed to form a structural model database specific to each anatomical structure.

[0111] First, a large number of three-dimensional images of the same part of the human body as the three-dimensional image to be reconstructed need to be collected. The three-dimensional image can be in any form including CT, and the present invention does not impose any limitation.

[0112] Secondly, when the anatomical structure of a two-dimensional image at a certain viewing angle needs to be reconstructed, a database of structural models at the same viewing angle as the two-dimensional image to be reconstructed is established for a large number of pre-collected three-dimensional images. And all shape models in the database can be iterated together using the Iterative Closest Point.

[0113] S312: resampling the structural model of each anatomical structure according to a standard sampling model, so that the structural models reconstructed by different people can be normalized to a standard shape model;

[0114] Specifically, all structural models corresponding to each anatomical structure obtained from the pre-collected three-dimensional images are resampled, and as many evenly distributed nodes as possible are collected for each structural model. For evenly distributed nodes, it is necessary to ensure that the same nodes on different structural models corresponding to the same anatomical structure are in similar positions. In this way, the positions of the resampled points of different anatomical structures are similar.

[0115] In order to ensure that the same nodes on different structural models corresponding to the same anatomical structure are at similar positions, the specific method of the present invention is as follows:

[0116] Each part of the structural model is resampled using a hemispherical fitting method, and a ray is emitted from the center of the hemispherical fitting to intersect with the surface patch of the corresponding part, and the intersection point is a resampling node. It should be noted that hemispherical fitting is only a preferred method of the present invention, and there are many other methods for obtaining nodes, which will not be described in detail in the present invention.

[0117] For example, taking the resampling of the distal femur as an example, the distal femur is divided into the medial malleolus, lateral malleolus and the diaphysis. A hemisphere is used to fit each part. A ray is emitted from the center of the hemisphere fitting sphere and the intersection point with the surface patch of the corresponding part is taken as a resampling point.

[0118] S313: The coordinates of all nodes of each structural model are combined into a node vector for describing the structural model, a covariance matrix is ​​constructed for the node vectors of all structural models corresponding to each anatomical structure, and the eigenvalues ​​and eigenvectors of the covariance matrix are calculated.

[0119] Specifically, through the resampling in step S32, each structural model forms a model represented by a fixed number of nodes. Further, all nodes are composed of a node vector, and the entire structural model can be represented by the node vector.

[0120] Then, the covariance matrix is ​​constructed for all node vectors, and the eigenvalues ​​and eigenvectors of the covariance matrix are calculated, specifically:

[0121] All node coordinates of each structural model are combined into a (3*n,1)-dimensional vector;

[0122] The covariance matrix is ​​constructed for the node vectors of all structural models corresponding to each anatomical structure:

[0123]

[0124] Where p is the number of structural models corresponding to the anatomical structure, n is the number of nodes in each structural model, S i is a vector representation of a structural model, is the average model of all structural models;

[0125] By calculating the eigenvalues ​​and eigenvectors of the covariance matrix, we can obtain:

[0126]

[0127] Among them, λ is the eigenvalue, arranged from large to small; P is the eigenvector, and s is the number of eigenvalues ​​and eigenvectors used.

[0128] S314: Using the average model, eigenvalues ​​and eigenvectors calculated from all the structural models of each anatomical structure, and using different weight factors to generate the shape model, specifically:

[0129]

[0130] Among them, a j To generate the weight factor of the shape model, different shape models are generated by different weight factors, specifically:

[0131] First, select the positive and negative value range for each weight factor (usually ± Standard deviation), randomly select a value within the range, substitute s weight factors into the formula in S314 to calculate the shape model vector under this weight factor combination, extract the coordinate values ​​of the corresponding nodes for each group of three coordinates of this long vector, and sort out a point cloud. Re-dividing the point cloud into triangular meshes can generate a shape model.

[0132] 2. Model Reconstruction

[0133] The contour line is read and compared with a plurality of groups of shape models generated using different weight factors. When the projection contour simulated by the shape model is close to the contour line, the current shape model is a reconstruction of the real anatomical structure, specifically:

[0134] S321: Read the contour line extracted in step S2.

[0135] S322: Read the environmental parameters during photography from the original two-dimensional image.

[0136] Specifically, in order to make the projection of the model more accurate, it is necessary to simulate the photographic environment of the two-dimensional image at that time.

[0137] S323: In a photographic environment corresponding to the environmental parameters, the shape model is used to simulate a projection outline.

[0138] S324: Compare the simulated projection contour with the contour line segmented from the actual image. When the projection contour is close to the contour line, the current shape model is a reconstruction of the real anatomical structure. The optimization process is expressed by the following formula:

[0139]

[0140] Wherein, L1 is the simulated projection contour, L2 is the contour line segmented from the actual image, and when F is close to 0, the current shape model is a reconstruction of the real anatomical structure.

[0141] S4: Using at least two of the two-dimensional images at different viewing angles, execute the above steps S1-S3 to reconstruct a three-dimensional model of the anatomical structure.

[0142] Specifically, since two 2D images at different viewing angles can generally cover the structure of the entire space, when a 3D model needs to be constructed from 2D images, at least two 2D images at different viewing angles need to be selected and the above steps need to be performed respectively to simulate a 3D model.

[0143] Preferably, a front two-dimensional image and a side two-dimensional image can be selected, and a three-dimensional model can be constructed through the two two-dimensional images of the front and the side.

[0144] When reconstructing a 3D model of an anatomical structure, the optimization process is formulated as follows:

[0145]

[0146] Wherein, k represents the viewing angle, n represents the number of viewing angles, and when F is close to 0, the current shape model is a reconstruction of the real anatomical structure.

[0147] Second embodiment

[0148] The steps of this embodiment are basically the same as those of the first embodiment, except that, in step S1, the prediction result graph is further processed, specifically:

[0149] 1. Get rid of irritability

[0150] The prediction result image is denoised, and connectivity analysis is performed on all anatomical structure types except the background class and all pixels classified into this category. If multiple connected areas are analyzed, the largest connected area is found and retained, and the other connected areas are removed.

[0151] Specifically, for each pixel corresponding to a label, find the largest connected area and delete other connected areas. The largest connected area is an anatomical structure corresponding to each label.

[0152] 2. Remove jagged points

[0153] Morphological operations are used on the connected areas to remove small jagged points, which is used as the final prediction result graph.

[0154] Based on the prediction result image after denoising, morphological calculation is further used to remove small jagged points. The prediction result image after these two steps will have better accuracy and clarity.

[0155] Third embodiment

[0156] This embodiment combines the methods in the first embodiment and the second embodiment, and takes the lower limbs, which account for a relatively high proportion in clinical research in orthopedics, as an example to explain the content of the invention through a specific example.

[0157] 1. Training of Segmentation Network

[0158] (1) 90 full-length Dicom images of the lower limbs were collected and converted into 16-bit PNG format after anonymization.

[0159] Among them, Dicom image is a medical image format that can be used for data exchange and has quality that meets clinical needs, and the use of Dicom image and conversion into png format for processing are all for the convenience of image processing and are not used to limit the present invention. At the same time, collecting 90 full-length films of lower limbs in this embodiment is only a specific example, and it is not necessarily 90 in practical applications. Any number mentioned in this embodiment is an example, used to illustrate the inventive point of the present invention, not to limit the present invention, and will not be repeated in this embodiment.

[0160] (2) Import the 90 collected PNG images into the imageLabeler app of Matlab, and use the built-in pixel labeling tool of the app to label the category to which each pixel in the image belongs. In the visual interface of the app, different labels will be overlaid on the pixels with layers of different colors. A total of 10 labels are set for the full-length lower limb film: right hip, left hip, right femur, left femur, right tibia, left tibia, right fibula, left fibula, soft tissue and empty area. The labeling results will be saved as PNG images, and this result PNG image has the same size as the original image PNG. Each pixel value in the labeling result PNG image is an integer from 1 to 10, representing the category to which the pixel at the corresponding position in the original image belongs. The correspondence between the integer and the actual label will be stored in a separate file for easy query and result display.

[0161] (3) Check the image quality of the 90 images and classify them into three categories: clear, normal, and blurry. Classify each image into one of the categories. Extract 60% of the images from each category and add them to the training set, and the remaining 40% to the validation set.

[0162] (4) Build a hybrid neural network of Unet and deeplabV3+, use the mixed loss function of diceLoss and activeContourLoss, use the adam optimizer to train this segmentation network, and use the early stopping strategy to decide when to stop training. The network input is 756*2268.

[0163] The specific refinement steps are exemplified as follows (for example only, and the present invention is not limited thereto):

[0164] 1. Download the deeplabV3+ pre-trained network from the Matlab deep learning library. The pre-trained network uses resnet18 as the backbone network and the downsampling is 16.

[0165] 2. Change the size of the pre-trained network input layer to (756, 2268).

[0166] The upsampling part of the original and pre-trained network is upsampled twice, each upsampling is 4 times. The original upsampling part is removed and replaced with 4 upsamplings of Unet, each upsampling is 2 times.

[0167] 3. Dice Loss is based on the Sorensen-Dice similarity coefficient and is used to measure the overlap between two segmentation results. The formula for the loss function L that measures the output segmentation result Y and the true segmentation result T is:

[0168]

[0169] Where K is the number of categories, M is the number of pixels in the image, is the weight factor of the category, which is used to control the contribution of different categories to the loss function. is inversely proportional to the area occupied by the species, that is,

[0170]

[0171] 4. The actual segmentation and predicted segmentation are recorded as ,Active Contour Loss can be written as:

[0172]

[0173] in:

[0174]

[0175] Written in pixel form:

[0176]

[0177] +

[0178] in Generally take 1, Generally it is set to 0.

[0179] 5. Hybrid loss function , where α is the coefficient that balances the two losses and is generally set to 1.

[0180] 6. Adam is the most commonly used adaptive optimization algorithm, and its iteration formula is:

[0181]

[0182] in:

[0183]

[0184]

[0185] L is the number of iterations, is the gradient decay factor, is the squared gradient decay factor.

[0186] 7. Early stopping: Monitor the error of the validation set. When the error of the validation set rises for several iterations or epochs, stop training to prevent data overfitting and return the parameters with the lowest error of the validation set. The actual threshold is 10 epochs.

[0187] 2.2D Image Segmentation

[0188] (5) When the image corresponding to the new two-dimensional image needs to be segmented, the image is first scaled to a size of 756*2268. After entering the segmentation network, the network will output a prediction result image of the same size, where each pixel is a number from 1 to 10, representing the network's prediction of the category to which the pixel belongs. There will be some noise in the prediction result, and the image processing algorithm needs to be used to post-process the result. The method is: for each pixel category, the result image is analyzed for connectivity, and the largest connected area is found. The remaining connected areas are considered to be noise. The largest connected area is retained. The morphological opening operation is used to remove some small jagged points on this connected area. After post-processing, the prediction result is scaled to the original input image size as the final prediction result.

[0189] (6) For the segmentation results of (5), further morphological analysis is performed on the segmentation results of each bone. The segmentation result of each bone is a binary image. Based on the relatively fixed bone shape, the location of key points can be further extracted using basic image processing algorithms such as Hough circle detection.

[0190] (7) For the segmentation results of (5), for each bone segmentation result, use the Moore-Neighbor tracing algorithm to extract the boundary pixels of the binary image. These pixels represent the boundary of the bone projection and can guide the generation of the subsequent shape model.

[0191] 3. Training the Shape Model

[0192] (8) 150 lower limb CT scans were collected and the shape models of the hip, femur, tibia, and fibula were reconstructed using Mimics. The 150 shape models of each bone correspond to the shape model database of that side.

[0193] (9) Consider a unilateral shape model database of bones. 150 bones are aligned together using Iterative Closest Point.

[0194] (10) Resample each bone model so that each model samples 4000 evenly distributed nodes and the same nodes in different models are in similar positions. The method is as follows:

[0195] Taking the resampling of the distal femur as an example, the distal femur is divided into the medial condyle, lateral condyle and the diaphysis. A hemisphere is used to fit each part. The intersection of the ray emitted from the center of the hemisphere fitting and the surface patch of the corresponding part is taken as a resampling point. This ensures that the positions of the resampling points of different bones are similar.

[0196] (11) Each bone is transformed into a fixed number of nodes by the S10 method. By putting the three coordinates of all nodes into a vector, a (3*n, 1)-dimensional vector can be used to describe a bone model in a database. The covariance matrix of the representation vectors of 150 bones is constructed:

[0197]

[0198] Where p is the number of bones in the database, here 150, and n is the number of nodes in each bone model. Si is a vector representation of a bone model. is the average model of all bones.

[0199] Calculating the eigenvalues ​​and eigenvectors of the covariance matrix Cov yields:

[0200]

[0201] Where λ is the eigenvalue, arranged from large to small. P is the eigenvector, corresponding to the eigenvalue.

[0202] Using the average model and eigenvalue eigenvectors, a new shape model can be combined:

[0203]

[0204] Where: s is the number of eigenvectors used, a j is the weight factor used to generate the new shape model.

[0205] 4. Model Reconstruction

[0206] S12: Read the contour result of S7. Read the environmental parameters of X-ray photography from the dicom image. jWhen the shape model generated by the parameters makes the projection contour simulated in the photographic environment close to the contour segmented by the actual image, this shape model is a reconstruction of the real bone model. This is equivalent to optimizing the following problem:

[0207]

[0208] Where L1 is the simulated projection contour, and L2 is the contour segmented from the actual image. If there are more than one image from one perspective involved in the reconstruction, the problem becomes the following form:

[0209]

[0210] Where K represents the viewing angle.

[0211] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

[0212] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for automatic recognition and model reconstruction of anatomical features in medical images, It is characterized in that The following steps are involved: S1: Input the two-dimensional image to be segmented into the segmentation network for prediction, agree in advance on the type of anatomical structure to be identified, and make classification prediction for each pixel of the input two-dimensional image to form a prediction result graph, wherein the segmentation network is obtained by pre-collecting a large number of the two-dimensional images, annotating them at the pixel level, and then training the neural network; S2: performing morphological analysis on each anatomical structure segmented in the prediction result image, extracting the positions of key points in the anatomical structure using an image processing algorithm, and extracting boundary pixels of the anatomical structure as the contour line of the anatomical structure; S3: reading the contour line and comparing it with a plurality of groups of shape models generated using different weight factors. When the projection contour simulated by the shape model is close to the contour line, the current shape model is a reconstruction of the real anatomical structure, wherein the generation of the shape model needs to rely on a model database based on CT artificial reconstruction; S4: Using at least two of the two-dimensional images at different viewing angles, executing the above steps S1-S3, so as to reconstruct a three-dimensional model of the anatomical structure; In step S3, the specific generation process of the shape model is as follows: For each collected 3D image, a structural model of each anatomical structure is reconstructed to form a structural model database unique to each anatomical structure; Each structural model is resampled according to the standard sampling model, so that the structural models reconstructed by different people can be normalized to a standard shape model; The coordinates of all nodes of each structural model are combined into a node vector for describing the structural model, a covariance matrix is ​​constructed for the node vectors of all structural models corresponding to each anatomical structure, and the eigenvalues ​​and eigenvectors of the covariance matrix are calculated; The shape model is generated using the average model and eigenvalues ​​and eigenvectors calculated from all structure models of each anatomical structure, using different weighting factors.

2. The method for automatic recognition and model reconstruction of anatomical features in medical images according to claim 1, It is characterized in that In step S1, the segmentation network is obtained by pre-collecting a large number of the two-dimensional images and performing pixel-level annotation and neural network training. The specific training process is: Setting different labels for different anatomical structures in the two-dimensional image; Using pixel annotation tools, each pixel in each collected two-dimensional image is labeled differently according to different anatomical structures, and each different label is overlaid on the pixel with a layer of different color, forming a two-dimensional annotated image that uses different colors to represent the type of anatomical structure to which each pixel belongs; The data set of the two-dimensional annotated images is classified and divided into a training set and a validation set. The training set is used to train the neural network to form the segmentation network, and the validation set is used to train the network to perform well in the face of unknown data.

3. The method for automatic recognition and model reconstruction of anatomical features in medical images according to claim 2, It is characterized in that The data set of the two-dimensional annotated image is classified as follows: Checking the image quality of the two-dimensional annotated image, classifying the image into multiple types including clear, general, and blurred according to the clarity of the image, and classifying each of the two-dimensional annotated images; For the two-dimensional annotated images of each definition, randomly select some images to add to the training set, and the rest to add to the verification set; Using the two-dimensional annotated images in the training set to perform training to generate the segmentation network; The segmentation network is validated using the two-dimensional annotated images in the validation set.

4. The method for automatic recognition and model reconstruction of anatomical features in medical images according to claim 1, It is characterized in that In step S1, the prediction result graph is further processed, specifically: De-noising the predicted result image, and performing connectivity analysis on all anatomical structure types except the background type and all pixels classified into this type. If multiple connected areas are analyzed, find and retain the largest connected area, and remove other connected areas; Morphological operations are used on the connected areas to remove small jagged points, which is used as the final prediction result graph.

5. The method for automatic recognition and model reconstruction of anatomical features in medical images according to claim 1, It is characterized in that Each structural model is resampled according to the standard sampling model. The specific steps are: Each part of the structural model is resampled using hemisphere fitting. Rays are emitted from the center of the hemisphere fitting and intersect with the surface patches of the corresponding part. The intersection point is a resampling node.

6. The method for automatic recognition and model reconstruction of anatomical features in medical images according to claim 1, It is characterized in that The coordinates of all nodes of each structural model are combined into a node vector for describing the structural model, a covariance matrix is ​​constructed for the node vectors of all structural models corresponding to each anatomical structure, and the eigenvalues ​​and eigenvectors of the covariance matrix are calculated, specifically: All node coordinates of each structural model are combined into a (3*n,1)-dimensional vector; The covariance matrix is ​​constructed for the node vectors of all structural models corresponding to each anatomical structure: Where p is the number of structural models corresponding to the anatomical structure, n is the number of nodes in each structural model, S i is a vector representation of a structural model, is the average model of all structural models; By calculating the eigenvalues ​​and eigenvectors of the covariance matrix, we can obtain: Among them, λ is the eigenvalue, arranged from large to small; P is the eigenvector, and s is the number of eigenvalues ​​and eigenvectors used.

7. The method for automatic recognition and model reconstruction of anatomical features in medical images according to claim 6, It is characterized in that The shape model is generated using the average model, eigenvalues ​​and eigenvectors calculated from all the structural models of each anatomical structure, using different weight factors, specifically: Among them, a j is a weight factor for generating the shape model.

8. The method for automatic recognition and model reconstruction of anatomical features in medical images according to claim 7, It is characterized in that In step S3, the contour line is read and compared with a plurality of groups of shape models generated using different weight factors. When the projection contour simulated by the shape model is close to the contour line, the current shape model is a reconstruction of the real anatomical structure, specifically: reading the contour line; Reading environmental parameters at the time of photography from the original two-dimensional image; In a photographic environment corresponding to the environmental parameters, using the shape model to simulate a projection contour; The simulated projection contour is compared with the contour line segmented from the actual image. When the projection contour is close to the contour line, the current shape model is a reconstruction of the real anatomical structure. The optimization process is expressed by the following formula: Wherein, L1 is the simulated projection contour, L2 is the contour line segmented from the actual image, and when F is close to 0, the current shape model is a reconstruction of the real anatomical structure.

9. The method for automatic recognition and model reconstruction of anatomical features in medical images according to claim 8, It is characterized in that When reconstructing a 3D model of an anatomical structure, the optimization process is formulated as follows: Wherein, k represents the viewing angle, n represents the number of viewing angles, and when F is close to 0, the current shape model is a reconstruction of the real anatomical structure.

Citation Information

Patent Citations

  • Three-dimensional modeling method based on organization structure, system and three-dimensional model

    CN107599412A

  • Multi-view reconstruction method based on deep learning contour network

    CN108305229A

  • Systems and methods for obtaining 3-d images from x-ray information

    US20170323443A1