A method and system for designing a maxillofacial defect repair implant

Abstract

The application discloses a design method and system of a maxillofacial defect repair implant, comprising: CT or MR scanning of a patient's head to obtain scanning data; threshold segmentation and non-skeletal part removal processing of the scanning data to obtain a binary image; partial registration of the binary image and a skull binary image template, adjustment of the pose according to the registration result, and determination of the defect type according to the registration missing part; local cutting and scaling processing of the binary image of the patient's skull according to the defect type; input of the processed image into a generation network to generate a preliminary implant design result; import of the preliminary implant design result and the original defect skull image into a mixed reality device, projection into a real space coordinate for fine adjustment and modification to obtain a correction result; transmission of the modification result into a 3D printer for production to obtain a final implant entity. The application can accurately design a suitable implant shape according to medical images of a defect site.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image processing and medical technology, and particularly relates to a design method and system of a maxillofacial defect repair implant. BACKGROUND

[0002] Patient-specific implants (PSIs) are often used to treat maxillofacial defects, especially defects of bone parts caused by accidents, tumor resection, etc. In order to be aesthetically pleasing and to protect internal tissues, doctors need to reconstruct the original bone shape according to the defect of the patient's bone.

[0003] Computer-aided design (CAD) is the step that has the greatest impact on the final repair quality, and is a lengthy and challenging task for both doctors and PSI design engineers. In order to ensure the protection effect of the implant on human tissues and the aesthetic appearance, the shape of the implant should be as reasonable and natural as possible, and should ensure good fit with the surrounding residual tissues. At present, this requires a high level of design from doctors and engineers. Therefore, there is an urgent need for an automatic design method or system for maxillofacial defect repair implants. SUMMARY

[0004] To achieve the above object, the present application provides the following scheme: a design method of a maxillofacial defect repair implant, comprising:

[0005] CT scanning or MR scanning is performed on the head of a patient to obtain scanning data; threshold segmentation processing and non-bone part removal processing are performed on the scanning data to obtain a binary image of the skull of the patient;

[0006] Partial registration is performed on the binary image of the skull of the patient and a binary image template of a skull selected by a person to obtain a registration result; after adjusting the pose according to the registration result, a registration missing part is obtained, and a defect type is determined according to the registration missing part;

[0007] Local cutting and scaling processing are performed on the binary image of the skull of the patient according to the defect type; and the processed image is input into a generation network to generate a preliminary implant design result;

[0008] After the preliminary implant design result and the original defect skull image are imported into a mixed reality device, the result is projected into a real space coordinate for fine tuning and adjustment to obtain a correction result;

[0009] The adjustment result is transmitted to a 3D printer for production to obtain a final implant entity.

[0010] Preferably, the threshold segmentation processing of the scanning data comprises,

[0011] A threshold value is set according to a gray value range of the bone part in the scan image, and the scan image is segmented based on a size relationship between a gray value of each part in the scan image and the threshold value, to obtain a segmented image.

[0012] Preferably, the process of performing non-bone part removal processing on the scan data to obtain a binary image of the patient's skull includes,

[0013] The segmented image is subjected to three-dimensional reconstruction, and the segmented image after three-dimensional reconstruction is filtered using a maximum connected domain algorithm to remove non-bone parts, to obtain a maximum connected part, and a complete skull binary image is saved.

[0014] Preferably, the process of performing partial registration of the binary image of the patient's skull with a manually selected skull binary image template to obtain a registration result includes,

[0015] Feature points of the binary image of the patient's skull and the manually selected skull binary image template are extracted based on a key point detection network, and connections between the feature points are generated according to an adjacency rule, to obtain a graph network.

[0016] The binary image of the patient's skull and the manually selected skull binary image template are subjected to partial registration by a point set registration and graph registration method, a rigid transformation matrix of a corresponding pose is calculated according to successfully registered parts, a scan image is subjected to rigid transformation, and the type of a defect is determined according to parts that are not successfully registered.

[0017] Preferably, the process of extracting feature points based on a key point detection network includes,

[0018] An ordered sequence point set of key points is defined on a standard skull, a key point regression network is built to learn the key points, three-dimensional binary images of the binary image of the patient's skull and the manually selected skull binary image template are input, a convolutional neural network is used to extract features, and a feature map is obtained.

[0019] The feature map is unfolded, a fully connected layer with learnable parameters is set, the feature map is input into the fully connected layer for calculation to obtain a position regression result of a feature point and a corresponding confidence, and points with a confidence greater than a preset threshold value are taken as predicted key points.

[0020] Preferably, the process of calculating a rigid transformation matrix of a corresponding pose according to successfully registered parts and subjecting a scan image to rigid transformation includes,

[0021] The ICP registration algorithm is used to register the binary image of the patient's skull and all corresponding feature points in the artificially selected skull binary image template, nearest point selection is performed, and a rotation matrix and a translation vector are calculated by using principal component analysis until the iteration termination condition is met, the calculation is ended, and the shape of the skull is obtained.

[0022] Preferably, the process of local cutting and scaling processing of the binary image of the patient's skull according to the defect type comprises,

[0023] After the defect type is determined according to the registration missing part, a fixed size image block is cut out with the defect part as the center, and then uniform sampling and cubic linear interpolation are used for scaling processing.

[0024] Preferably, the process of inputting the processed image into the generation network to generate a preliminary implant design result comprises,

[0025] The generation network is built based on a codec architecture, high-level semantic features of the image are extracted layer by layer through a plurality of parameter-learnable convolution modules, then convolution and deconvolution modules are used to start sampling from the deepest semantic feature map, low-level features are combined for continuous optimization, until the original resolution implant design result is generated, and the optimization method is to learn the distribution relationship between the optimization target and the latent variable, so that the network can generate a target implant shape based on the input skull.

[0026] In another aspect, a design system for a maxillofacial defect repair implant is also provided, comprising,

[0027] An image acquisition module is configured to perform CT scanning or MR scanning on the head of the patient to obtain a scanning image;

[0028] An image processing module is connected to the image acquisition module and configured to perform threshold segmentation processing and non-skeletal part removal processing on the scanning data to obtain a binary image of the patient's skull;

[0029] An image registration module is connected to the image processing module and configured to perform partial registration of the binary image of the patient's skull and an artificially selected skull binary image template to obtain a registration result;

[0030] A registration repair module is connected to the image registration module and configured to adjust the pose according to the registration result, obtain a registration missing part, and determine a defect type according to the registration missing part; and perform local cutting and scaling processing of the binary image of the patient's skull according to the defect type.

[0031] The implant manufacturing module, connected to the registration and repair module, is used to input the processed image into the generation network to generate a preliminary implant design result; it is also used to import the preliminary implant design result and the original defective skull image into a mixed reality device, project them onto real space coordinates for fine-tuning and repair, and obtain a correction result; and then input the repair result into a 3D printer for manufacturing to obtain the final implant entity.

[0032] The present invention discloses the following technical effects:

[0033] This invention provides a design method and system for maxillofacial defect repair implants, which can quickly generate relatively accurate PSI design candidates based on the defect situation. This allows doctors to fine-tune the implant candidate design in a mixed reality space, based on the binary image of the original defective skull or even the real patient's body, using virtual erasing, cutting, and filling tools. This enables the rapid design of clinically usable implant shapes, thereby accelerating the design process while ensuring the safety and usability of the final implant design results, and creating a reasonable human-machine collaborative process. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of the generated network architecture according to an embodiment of the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] like Figure 1As shown, the present application provides a design method of maxillofacial defect repair implant, comprising: CT or MR scanning of the patient's head to obtain medical image data of the patient; using medical image processing software to perform threshold segmentation and non-skeletal part removal on the scanning data of the patient to obtain a binary image of the skull part; performing partial registration of the binary image of the patient's skull and the artificially selected standard skull binary image, adjusting the pose according to the registration result, and determining the defect type according to the registration missing part; performing local cutting and scaling processing on the skull binary image according to the defect type; inputting the processed image into the generation network to generate a preliminary implant design result; importing the preliminary implant design result and the original defect skull image into a mixed reality device and projecting it into a mixed reality space, so that the doctor can adapt the repair result to the defect skull as needed, and make on-site fine tuning to generate a shape fully adapted to the defect site; finally using 3D printing technology to produce a clinically usable implant with high adaptation degree. The present application enables doctors to quickly generate a relatively reasonable and beautiful implant design based on the image of the defect skull, and can use a mixed reality device to quickly trim the generated implant design as needed. This system saves a lot of time and effort for the shape design process of the implant, and also improves the clinical practicability.

[0040] Further optimization scheme, the present application is further described in detail:

[0041] The design method of the maxillofacial defect repair implant of the present application comprises the following steps:

[0042] CT or MR scanning of the patient's head to obtain three-dimensional medical image data of the head including the maxillofacial part;

[0043] Threshold segmentation of the three-dimensional image data according to the pixel gray value of the image to obtain the part with gray value greater than the specified threshold in the image, which includes the skull part image and the object with high density in the imaging area around the patient, using medical image processing software to perform three-dimensional reconstruction on the segmentation result, using the maximum connected domain algorithm to remove the non-skeletal part, and by finding the maximum connected part, that is, the skull part, the complete skull binary image is obtained;

[0044] Taking the patient's skull as the source image and the artificially selected skull template as the target image, extracting feature points from both sides and generating connections between the features according to the adjacent rule to form a graph network, then using point set registration and graph registration method to perform partial registration on both sides, calculating the rigid transformation matrix of the pose according to the successfully registered part, performing rigid transformation on the source image, and determining the defect type according to the unsuccessfully registered part;

[0045] Cutting the patient's skull according to the defect type and scaling to a fixed size;

[0046] The cut and scaled image is input into the implant design network, feature extraction is performed on the image based on a codec architecture, high-level semantic information is gradually extracted as the level deepens, then the feature map is gradually restored to the original resolution by combining the encoder output information of each layer, and finally a preliminary implant design result is output;

[0047] The generated result is saved as a three-dimensional data file, imported into a mixed reality device, reconstructed and projected into a mixed reality space, and used by a doctor to interact with a handle, move, rotate, scale, and other operations on the three-dimensional image. The preliminary implant design result is adapted to the original defect image, and a virtual tool is used to trim the shape. If it is in a surgical room environment, it can be directly adapted to the patient's body. The trimmed image can be used as the final implant result;

[0048] The final determined implant shape is imported into a 3D printer for production to obtain a clinically usable implant;

[0049] Further, the partial registration used includes the following processes:

[0050] The patient's skull is used as the source image, and the skull template is used as the target image. Key point detection network is used to extract key points from both sides to obtain point sets V1 and V2, and prior knowledge of skull key points is used to screen and register the two point sets to obtain the same position key points V1' and V2'.

[0051] According to the key point set, the corresponding relationship of the corresponding points in V1 and V2 is calculated, the rigid transformation matrix of both sides is calculated, the source image is rigidly transformed, and the type of the defect is determined according to the position of the missing key points in V1.

[0052] The specific implementation of the key point detection method used in the embodiment is as follows:

[0053] A point set P = {p1, …, pn} is defined on a standard skull, and P is an ordered sequence of key points. A key point regression network is built to learn the regression of key points, and the input is a three-dimensional binary image of the skull A convolutional neural network is used for feature extraction to obtain a feature map The feature map is unfolded to a size of 1xd'w'h', a parameter-learnable fully connected layer W is set, the feature map is input into the fully connected layer for calculation, and finally the position regression result of n feature points and the corresponding confidence are obtained The points with a confidence greater than a threshold s are used as the predicted key points.

[0054] The specific implementation of the rigid transformation matrix calculation method used in the embodiment is as follows:

[0055] ICP registration algorithm is used for registration, for any point vi in the point set V2', find the spatial nearest point xj in X, after all points are determined, the principal component analysis method is used to calculate the rotation matrix R k and translation vector T k , calculate the new coordinates P1' = R k ·P1' + T k , repeat the nearest point selection and rotation matrix and translation vector calculation steps until the final iteration termination condition is met, end the calculation, and obtain the skull shape.

[0056] Further, the establishment of the generation network includes the following processes:

[0057] Extract defect data and original shape of corresponding defect part from the database as a training set;

[0058] Establish a parameter trainable conditional generation network based on a codec architecture, the structure of the entire codec is as follows: the encoder is used to extract features from the input image and generate feature maps, each layer of feature map is down-sampled using convolution and sent to the next layer of encoder for extraction of higher layer semantic features, until the lowest layer feature map of the preset size is generated, the structure of the decoder is similar to that of the encoder, the difference lies in that the transpose convolution is used to expand the size of the feature map, and the feature map of each layer of the decoder is spliced with the feature map of the same level of the encoder in the channel dimension as the input of the next layer of the decoder.

[0059] The generation network belongs to a probability model, and the architecture is as shown in Figure 2 The expected result y of the desired generation is Y, P(Y) is the possible distribution of y, the hidden space Z of the lowest layer feature map of the encoder is taken as the sampling space, it is assumed that the data distribution in the sampling space satisfies a certain distribution, for example, Gaussian distribution, the network needs to learn to generate the corresponding Z distribution in the hidden space from the Y distribution, that is, to learn P(Y') = ∫ Z P θ (Y'|Z)P(Z), θ represents the network parameters of the generator, and the result Y' generated according to Z still needs to satisfy the distribution P(Y), each y needs to find the corresponding z, which needs the network to learn P(Z|Y), and the generation result takes the input skull shape X as the condition, and needs to learn P(X|Y,Z),

[0060]

[0061] L V = -D KL (q(z|x,y)||p(z|x))+F q(z∣x,y) (log(p(y|z,x))

[0062] L is a fixed value, q is a distribution represented by an inference network, p is an actual distribution, the first term D KL is a KL divergence calculation function, ensuring that the whole conforms to a Gaussian distribution, and the second term E q(z∣x,y) optimizes the generation network directly, ensuring that the generator can be well restored from the hidden variable to the data, and L V is updated as a loss function for the network.

[0063] The output y of the last layer of the decoder is used as the repair result, that is, the generated implant shape, and the loss function value is calculated with the original missing part shape, and the trainable parameters in the network are optimized according to the loss function value, so that the generated result is closer to the shape of the original missing part.

[0064] Embodiment two

[0065] The application also provides a maxillofacial defect repair implant design system, comprising,

[0066] An image acquisition module is configured to perform CT scanning or MR scanning on the head of the patient to obtain scanning images.

[0067] An image processing module is connected to the image acquisition module and is configured to perform threshold segmentation processing and non-skeletal part removal processing on the scanning data to obtain a binary image of the skull of the patient.

[0068] An image registration module is connected to the image processing module and is configured to perform partial registration of the binary image of the skull of the patient and a binary image template of a skull selected by a person to obtain a registration result.

[0069] A registration repair module is connected to the image registration module and is configured to obtain a registration missing part after adjusting the pose according to the registration result, and determine a defect type according to the registration missing part; and perform local cutting and scaling processing on the binary image of the skull of the patient according to the defect type.

[0070] An implant manufacturing module is connected to the registration repair module and is configured to input the processed image into a generation network to generate a preliminary implant design result; and is further configured to project the preliminary implant design result and the original defect skull image into a real space coordinate after importing them into a mixed reality device for fine tuning and trimming to obtain a correction result; and transmit the trimming result to a 3D printer for manufacturing to obtain a final implant entity.

[0071] Further, in the maxillofacial defect repair implant design system, the hardware devices include: a desktop computer installed with a windows / Ubuntu operating system and a graphics processing unit (GPU), a high-resolution display, a mixed reality device, and a biological material 3D printer.

[0072] The desktop computer is used for image preprocessing, registration, repair, post-processing and other operations, and provides computing power for the mixed reality device; the high-resolution display is used to show the repair process and the results of each stage and to fine-tune the results by the doctor; the mixed display device includes mixed reality glasses and a handle, which are used to fine-tune the implant design in the mixed reality space, interact with the graphics and adapt, and the 3D printer is used to make the final available implant.

[0073] The above-described embodiments are only to describe the preferred modes of the present application, and not to limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by those skilled in the art shall fall within the protection scope determined by the claims of the present application.

Claims

1. A method for designing implants for repairing maxillofacial defects, characterized in that, include: A CT or MR scan of the patient's head is performed to obtain scan images; The scanned image is subjected to threshold segmentation and non-skeletal part removal processing to obtain a binary image of the patient's skull; The binary image of the patient's skull is partially registered with a manually selected binary image template of the skull to obtain the registration result; after adjusting the pose according to the registration result, the missing part of the registration is obtained, and the defect type is determined according to the missing part of the registration. The binary image of the patient's skull is locally cut and scaled according to the type of defect. The processed images are input into the generative network to generate preliminary implant design results; After importing the preliminary implant design results and the original defective skull image into the mixed reality device, they are projected onto real space coordinates for fine-tuning and correction to obtain the corrected result. The correction results are fed into a 3D printer for fabrication to obtain the final implant entity; The process of partially registering the binary image of the patient's skull with a manually selected binary image template of the skull to obtain the registration result includes: Based on the key point detection network, feature points are extracted from the binary image of the patient's skull and the manually selected binary image template of the skull, and connections between the feature points are generated according to the adjacency rule to obtain a graph network. The binary image of the patient's skull is partially registered with a manually selected binary image template of the skull using point set registration and graph registration methods. The rigid body transformation matrix of the corresponding pose is calculated based on the successfully registered part, and the scanned image is subjected to rigid body transformation. The type of defect is then determined based on the unregistered part. The process of extracting feature points based on a keypoint detection network includes, An ordered sequence of key points is defined on a standard skull. A key point regression network is built to perform regression learning on the key points. The binary image of the patient's skull and a three-dimensional binary image of a manually selected skull binary image template are input. A convolutional neural network is used to extract features and obtain a feature map. The feature map is expanded, and a fully connected layer with learnable parameters is set. The feature map is input into the fully connected layer to calculate the location regression results of the feature points and their corresponding confidence scores. Points with confidence scores greater than a preset threshold are used as predicted key points. The rigid body transformation matrix of the corresponding pose is calculated based on the successfully registered portion. The process of performing rigid body transformation on the scanned image includes... The ICP registration algorithm is used to register the binary image of the patient's skull with all corresponding feature points in a manually selected binary image template of the skull. The nearest point selection is performed and the rotation matrix and translation vector are calculated using principal component analysis until the iteration termination condition is met, and the calculation ends to obtain the aligned skull shape.

2. The design method for a maxillofacial defect repair implant according to claim 1, characterized in that, The process of thresholding the scanned image includes, A threshold is set based on the grayscale value range of the skeletal part in the scanned image. Based on the relationship between the grayscale value of each part in the scanned image and the threshold, the scanned image is segmented to obtain a segmented image.

3. The design method for a maxillofacial defect repair implant according to claim 2, characterized in that, The process of removing non-skeletal parts from the scanned image to obtain a binary image of the patient's skull includes: The segmented image is reconstructed in three dimensions. The maximum connected component algorithm is used to filter the reconstructed segmented image, remove non-skeletal parts, obtain the maximum connected component, and save the complete skull binarized image.

4. The design method of a maxillofacial defect repair implant according to claim 1, characterized in that, The process of locally cutting and scaling the binary image of the patient's skull according to the defect type includes, Based on the defect type determined by the registration missing part, after cutting out a fixed-size image block centered on the defect area, uniform sampling and cubic linear interpolation are used for scaling.

5. The design method of a maxillofacial defect repair implant according to claim 1, characterized in that, The process of inputting the processed images into the generative network to generate preliminary implant design results includes: The generative network is built based on an encoder-decoder architecture. High-level semantic features of the image are extracted layer by layer through multi-layer parameter-learnable convolutional modules. Then, starting from the deepest semantic feature map, convolutional and deconvolutional modules are used to upsample layer by layer, and low-level features are combined to continuously optimize until the implant design result with the original resolution is generated.

6. A design system for maxillofacial defect repair implants, characterized in that, include, The image acquisition module is used to perform CT or MR scans on the patient's head to obtain scan images; An image processing module, connected to the image acquisition module, is used to perform threshold segmentation and non-skeletal part removal processing on the scanned image to obtain a binary image of the patient's skull. An image registration module, connected to the image processing module, is used to partially register the binary image of the patient's skull with a manually selected binary image template of the skull to obtain a registration result. The registration and repair module, connected to the image registration module, is used to adjust the pose according to the registration result, obtain the registration missing part, and determine the defect type according to the registration missing part; and to perform local cutting and scaling processing on the binary image of the patient's skull according to the defect type. The implant manufacturing module, connected to the registration and repair module, is used to input the processed image into the generation network to generate a preliminary implant design result; it is also used to import the preliminary implant design result and the original defective skull image into the mixed reality device, project them onto real space coordinates for fine-tuning and repair, and obtain a corrected result. The correction results are then fed into a 3D printer for fabrication to obtain the final implant entity; The process of partially registering the binary image of the patient's skull with a manually selected binary image template of the skull to obtain the registration result includes: Based on the key point detection network, feature points are extracted from the binary image of the patient's skull and the manually selected binary image template of the skull, and connections between the feature points are generated according to the adjacency rule to obtain a graph network. The binary image of the patient's skull is partially registered with a manually selected binary image template of the skull using point set registration and graph registration methods. The rigid body transformation matrix of the corresponding pose is calculated based on the successfully registered part, and the scanned image is subjected to rigid body transformation. The type of defect is then determined based on the unregistered part. The process of extracting feature points based on a keypoint detection network includes, An ordered sequence of key points is defined on a standard skull. A key point regression network is built to perform regression learning on the key points. The binary image of the patient's skull and a three-dimensional binary image of a manually selected skull binary image template are input. A convolutional neural network is used to extract features and obtain a feature map. The feature map is expanded, and a fully connected layer with learnable parameters is set. The feature map is input into the fully connected layer to calculate the location regression results of the feature points and their corresponding confidence scores. Points with confidence scores greater than a preset threshold are used as predicted key points. The rigid body transformation matrix of the corresponding pose is calculated based on the successfully registered portion. The process of performing rigid body transformation on the scanned image includes... The ICP registration algorithm is used to register the binary image of the patient's skull with all corresponding feature points in a manually selected binary image template of the skull. The nearest point selection is performed and the rotation matrix and translation vector are calculated using principal component analysis until the iteration termination condition is met, and the calculation ends to obtain the aligned skull shape.

Images