A sacrum three-dimensional model matching method and device based on deep learning

Abstract

The application relates to the technical field of medical image processing and deep learning, in particular to a sacrum three-dimensional model matching method and device based on deep learning, which can solve the problem that the difference between a standard sacrum three-dimensional model and a patient three-dimensional model is relatively large, and it is difficult to compare the two, thereby affecting the judgment of the sacroiliac screw implantation position. The sacrum three-dimensional model matching method based on deep learning comprises the following steps: acquiring a CT image comprising a normal sacrum, performing three-dimensional reconstruction on the CT image through a deep learning neural network, and generating a three-dimensional point cloud model representing a normal sacrum three-dimensional model; acquiring a feature part of the normal sacrum, and selecting corresponding feature points from the feature part by using a deep learning neural network; and according to the feature points, superimposing the three-dimensional point cloud model and a to-be-detected three-dimensional model with to-be-detected feature points, to obtain a superimposed image of the three-dimensional point cloud model and the to-be-detected three-dimensional model.

Description

Technical Field

[0001] This application relates to the fields of medical image processing and deep learning technology, and more specifically, to a method and apparatus for matching a three-dimensional model of the sacrum based on deep learning. Background Technology

[0002] The pelvic girdle consists of the sacrum, coccyx, ilium, ischium, and pubis. The sacrum, triangular in shape, is located at the base of the lumbar vertebrae, forming the posterior aspect of the pelvic cavity. As a crucial structure for stress transmission in the pelvis, the sacrum is highly susceptible to fragility fractures. Sacral fragility fractures are typically treated with percutaneous sacroiliac screws, a technique proven effective and safe, with studies reporting a screw misalignment rate of less than 3%. However, in elderly patients with osteoporosis, up to 14% experience sacroiliac screw loosening. This is because the cancellous bone provides poor holding force after screw placement in an osteoporotic sacrum, leading to screw loosening and misalignment. Therefore, the placement of sacroiliac screws must be individualized.

[0003] Typically, a CT scan of the pelvic region is performed during the preoperative preparation phase to obtain a three-dimensional model of the patient, including the sacrum. The surgeon compares this standard sacral 3D model with the patient's 3D model to determine the placement of the sacroiliac screw. However, due to differences in age, physical characteristics, physical condition, and lifestyle among patients with sacral fractures, the size, height, and bone density of the sacrum vary, leading to significant differences between the standard sacral 3D model and the patient's 3D model. Therefore, comparing the two models presents challenges for the surgeon, thus affecting the determination of the sacroiliac screw placement. Summary of the Invention

[0004] To address the challenge of comparing standard sacral 3D models with patient 3D models, which makes determining the sacroiliac screw placement difficult, this application provides a deep learning-based sacral 3D model matching method and apparatus.

[0005] The embodiments of this application are implemented as follows:

[0006] This application provides a deep learning-based method for matching a three-dimensional model of the sacrum, including:

[0007] A CT image including a normal sacrum is acquired, and the CT image is reconstructed in three dimensions using a deep learning neural network to generate a three-dimensional point cloud model representing a three-dimensional model of a normal sacrum. The three-dimensional point cloud model is used to characterize information including the height and bone density of the normal sacrum.

[0008] The characteristic regions of the normal sacrum are obtained, and corresponding feature points are selected from the characteristic regions using a deep learning neural network. The characteristic regions are used to characterize the compositional structure of the normal sacrum.

[0009] Based on the feature points, the three-dimensional point cloud model and the three-dimensional model to be tested marked with the feature points to be tested are superimposed to obtain a superimposed image of the three-dimensional point cloud model and the three-dimensional model to be tested. The three-dimensional model to be tested is a three-dimensional point cloud model generated by CT images of the patient's sacrum. The feature points to be tested correspond to the feature points. The superimposed image is used to characterize the difference between the three-dimensional model to be tested and the three-dimensional point cloud model.

[0010] In some embodiments, the deep learning neural network includes an encoder network, a fully connected layer, a decoder network, and a pixel classification layer. The deep learning neural network performs three-dimensional reconstruction of the CT image to generate a three-dimensional point cloud model representing a normal sacrum, further including:

[0011] Based on the CT image, a set of feature maps is generated using the convolution of the encoder network, and then the feature maps are input into the fully connected layer.

[0012] After adding features containing sacral location information to the feature map through the fully connected layer, the feature map is input into the decoder network;

[0013] The feature map is sampled by the decoder network to generate a sparse feature map, which is then used to generate a dense feature map through convolution. The dense feature map is then used as input to the pixel classification layer.

[0014] The pixel classification layer predicts the largest category corresponding to the probability of each pixel in the dense feature map, and the category includes the sacrum and other body parts besides the sacrum;

[0015] The CT image is semantically segmented based on the category to obtain a two-dimensional array. The two-dimensional array is used to represent the gray value of each pixel in the CT image, wherein the gray value of the pixels that make up the sacrum is 255, and the gray value of the pixels that make up the other parts is 0.

[0016] Stacking the two-dimensional array generates a three-dimensional point cloud model representing a three-dimensional model of the sacrum. The two-dimensional array includes the layer position, image height, and pixel density information in the CT image. The image height corresponds to the height, and the pixel density information corresponds to the bone density information.

[0017] In some embodiments, after predicting the largest class corresponding to the probability of each pixel in the dense feature map through the pixel classification layer, the method further includes:

[0018] Set the optimal learning rate, initial momentum, model loss function, and optimizer type for training the deep learning neural network, wherein the optimizer is set to Adam and the model loss function is set to the cross-entropy loss function.

[0019] The CT images are labeled according to the largest category, and the deep learning neural network is trained using a first training set to verify the category to which the feature region belongs. The labeled CT images constitute a first dataset, which includes a first training set and a first test set.

[0020] In some embodiments, obtaining the characteristic region of the normal sacrum further includes:

[0021] The normal sacrum is classified to obtain the characteristic locations of the normal sacrum;

[0022] The characteristic parts of the normal sacrum include: the superior endplate of the sacrum, the sacral foramen, the spinal canal, the lateral articular surface of the sacral wing, the apex of the midline of the spinous process, and the apex of the lower part of the midline of the spinous process.

[0023] In some embodiments, the step of selecting corresponding feature points from the feature region using a deep learning neural network further includes:

[0024] The three-dimensional point cloud data of the feature region is input into a deep learning neural network to obtain a two-dimensional array with preset rows. The deep learning neural network is used to perform input transformation and feature transformation on the three-dimensional point cloud data. The preset rows correspond to the number of the feature regions.

[0025] The two-dimensional array in each row is analyzed using a classification function to obtain the maximum probability in each row of the two-dimensional array. The point cloud corresponding to the maximum probability is the feature point.

[0026] In some embodiments, after selecting corresponding feature points from the feature regions using a deep learning network, the method further includes:

[0027] Set the optimal learning rate, initial momentum, model loss function, and optimizer type for training the deep learning neural network, wherein the optimizer is set to Adam and the model loss function is set to the cross-entropy loss function.

[0028] The 3D point cloud data is labeled according to the said feature parts, and the deep learning neural network is trained using the second training set to verify the category to which the feature points belong. The labeled 3D point cloud data is used as the second dataset, which includes a second training set and a second test set.

[0029] In some embodiments, the step of superimposing the 3D point cloud model and the 3D model to be tested marked with the feature points to obtain a superimposed image of the 3D point cloud model and the 3D model to be tested, based on the feature points, further includes:

[0030] The scaling ratio is determined based on the feature points, and the 3D model to be tested is scaled according to the scaling ratio until the first connecting line in the 3D point cloud model coincides with the second connecting line in the 3D model to be tested, thus obtaining the scaled 3D model to be tested. The first connecting line is the line between the first feature point and the second feature point in the 3D point cloud model, and the second connecting line is the line between the corresponding first feature point and the second feature point in the 3D model to be tested.

[0031] A face-to-face matching is performed between the first plane in the three-dimensional point cloud model and the second plane in the scaled three-dimensional model to be tested. The first plane is composed of the first connecting line and the third feature point, and the second plane is composed of the second connecting line and the corresponding third feature point to be tested.

[0032] Obtain the standard points of the third, fourth, fifth, and sixth feature points to be tested. The standard point is equal to the quotient between the sum of the corresponding feature point matrices in each of the three-dimensional models to be tested and the number of three-dimensional models to be tested.

[0033] Based on the standard points, the 3D point cloud model and the 3D model to be tested are superimposed to obtain a superimposed image of the 3D point cloud model and the 3D model to be tested.

[0034] In some embodiments, based on the feature points, a scaling ratio is determined according to the feature points, and the 3D model to be tested is scaled according to the scaling ratio until the first connecting line in the 3D point cloud model coincides with the second connecting line in the 3D model to be tested, further including:

[0035] Calculate the first length of the first line connecting any two feature points in the three-dimensional point cloud model, where the first length is represented by a first matrix.

[0036] Calculate the second length of the second line connecting two feature points in the three-dimensional model to be tested, and the second length is represented by a second matrix;

[0037] Calculate the scaling ratio based on the first and second matrices;

[0038] The 3D model to be tested is adjusted based on the scaling ratio so that the second connecting line coincides with the first connecting line.

[0039] In some embodiments, the overlaying of the 3D point cloud model and the 3D model to be measured based on the standard points further includes:

[0040] The standard points corresponding to the fourth, fifth, and sixth feature points to be tested are combined to form a 3*3 feature matrix;

[0041] The 3*3 feature matrix is ​​converted into a 4*4 homogeneous coordinate matrix;

[0042] The transformation matrix is ​​obtained by summing the 4*4 homogeneous coordinate matrices of each of the three-dimensional models to be tested and taking the average value.

[0043] Based on the transformation matrix, the three-dimensional model to be tested and the three-dimensional point cloud model are superimposed.

[0044] This application also provides a deep learning-based sacral 3D model matching device, comprising:

[0045] The generation module is used to acquire CT images including a normal sacrum, and to perform three-dimensional reconstruction of the CT images using a deep learning neural network to generate a three-dimensional point cloud model representing a three-dimensional model of a normal sacrum. The three-dimensional point cloud model is used to characterize information including the height and bone density of the normal sacrum.

[0046] A selection module is used to obtain the characteristic regions of the normal sacrum, and to select corresponding feature points from the characteristic regions using a deep learning neural network. The characteristic regions are used to characterize the compositional structure of the normal sacrum.

[0047] The overlay module is used to overlay the three-dimensional point cloud model and the three-dimensional model to be tested marked with the feature points according to the feature points, so as to obtain an overlay image of the three-dimensional point cloud model and the three-dimensional model to be tested. The three-dimensional model to be tested is a three-dimensional point cloud model generated by CT image of the patient's sacrum. The feature points to be tested correspond to the feature points. The overlay image is used to characterize the difference between the three-dimensional model to be tested and the three-dimensional point cloud model.

[0048] The beneficial effects of this application are as follows: By constructing a three-dimensional point cloud model of a normal sacrum, the height and bone density of the normal sacrum can be intuitively characterized; furthermore, by using a deep learning neural network to select corresponding feature points from the characteristic parts of the sacrum, it is convenient to perform feature point matching in the future; furthermore, based on the feature points, the three-dimensional point cloud model is superimposed with the three-dimensional model to be tested marked with the feature points to be tested, to obtain a superimposed image of the three-dimensional point cloud model and the three-dimensional model to be tested. The superimposed image intuitively shows the difference between the normal sacrum and the patient's sacrum, and the physician can more easily determine the implantation position of the sacroiliac screw based on this difference. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart of a deep learning-based 3D model matching method for the sacrum according to one or more embodiments of this application;

[0051] Figure 2 A diagram showing the distribution of various characteristic parts of a normal sacrum according to one or more embodiments of this application;

[0052] Figure 3 This is a schematic diagram of an overlay image of a 3D point cloud model and a 3D model under test according to one or more embodiments of this application;

[0053] Figure 4 This is a block diagram of a deep learning-based sacral 3D model matching device according to one or more embodiments of this application. Detailed Implementation

[0054] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0055] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0056] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0057] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0058] The terms "setup" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of these terms in this application based on the specific circumstances.

[0059] like Figure 1 As shown in the figure, this application provides a flowchart of a deep learning-based three-dimensional sacral model matching method. The deep learning-based three-dimensional sacral model matching method of this application includes the following steps:

[0060] Step 110: Obtain CT images including the normal sacrum, and perform three-dimensional reconstruction of the CT images using a deep learning neural network to generate a three-dimensional point cloud model representing the three-dimensional model of the normal sacrum. The three-dimensional point cloud model is used to characterize information including the height and bone density of the normal sacrum.

[0061] In some embodiments, clinical CT images are obtained by performing thin-slice CT scans of the pelvis using a 64-slice or 128-slice spiral CT scanner. Patient privacy information contained in the CT image DICOM format file is removed to protect patient privacy, and the file with the removed patient privacy information is converted into a JPG format file.

[0062] The height of the sacrum varies with a person's height, and the bone density of the sacrum varies with a person's age. The point cloud density parameter in the three-dimensional point cloud model is used to characterize bone density.

[0063] In some embodiments, the deep learning neural network includes an encoder network, a fully connected layer, a decoder network, and a pixel classification layer.

[0064] The encoder network consists of 13 convolutional layers, each corresponding to a corresponding decoder convolutional layer; therefore, the decoder network also consists of 13 convolutional layers. A fully connected layer separates the encoder and decoder networks. The main function of the fully connected layer is to incorporate the location information of the sacrum from the CT image into the image. The pixel classification layer is constructed using a softmax classification function.

[0065] CT images typically capture the sacrum and surrounding cartilage and synovium. Since the 3D point cloud model only contains the sacrum, semantic segmentation of the overall 3D point cloud model is required first, dividing it into the sacrum and other areas (such as cartilage and synovium). Then, a 3D point cloud model containing only the sacrum is selected from the 3D point cloud model. This is achieved through the following steps:

[0066] Step 111: Based on the CT image, a set of feature maps is generated using the convolution of the encoder network. The output is downsampled by multiples of 2 through batch normalization, ReLU activation function, and max pooling. The feature maps are then input into the fully connected layer.

[0067] Step 112: After adding features containing sacral location information to the feature map through a fully connected layer, the feature map is input into the decoder network.

[0068] Step 113: Sample the feature map through the decoder network to generate a sparse feature map. The sparse feature map is used to generate a dense feature map through convolution. The dense feature map is used as input to the pixel classification layer.

[0069] When sampling the feature map through the decoder network, the maximum value pooling stored in the corresponding encoder is used as the index for sampling. The feature map generated by the decoder network has the same size and number of channels as the corresponding encoder input.

[0070] Step 114: Predict the largest category corresponding to the probability of each pixel in the dense feature map through the pixel classification layer. The categories include the sacrum and other body parts besides the sacrum.

[0071] Step 115: Perform semantic segmentation on the CT image based on the category to obtain a two-dimensional array. The two-dimensional array is used to represent the gray value of each pixel in the CT image, where the gray value of the pixels that make up the sacrum is 255, and the gray value of the pixels that make up other parts is 0.

[0072] Step 116: Stack the two-dimensional arrays to generate a three-dimensional point cloud model representing the three-dimensional model of the sacrum. The two-dimensional arrays include the slice position, image height, and pixel density information from the CT image. The image height corresponds to the height of a normal sacrum, and the pixel density information corresponds to the bone density information. The three-dimensional point cloud model uses the STL format.

[0073] Following step 114, the following steps are also included:

[0074] Configure the optimal learning rate, initial momentum, model loss function, and optimizer type for training the deep learning neural network, where the optimizer is set to Adam and the model loss function is set to the cross-entropy loss function;

[0075] CT images are labeled according to the largest category. A deep learning neural network is trained using the first training set to verify the category to which the feature parts belong. The labeled CT images constitute the first dataset, which includes the first training set and the first test set, with 80% being the first training set and 20% being the first test set.

[0076] If the verification result is consistent with the result obtained in step 114, proceed to step 115; if the verification result is inconsistent with the result obtained in step 114, proceed to step 111.

[0077] Step 120: Obtain the characteristic regions of a normal sacrum, and use a deep learning neural network to select corresponding feature points from these regions. These characteristic regions are used to characterize the constituent structures of a normal sacrum, and this is achieved through the following steps:

[0078] Step 121: Classify normal sacrum, obtain the characteristic parts of normal sacrum, and use the characteristic parts to mark the three-dimensional point cloud model.

[0079] The characteristic features of a normal sacrum include: the apex of the midline of the spinous process, the lateral articular surface of the sacral wing, the apex of the midline of the spinous process, the spinal canal, the superior endplate of the sacrum, and the sacral foramen. In this embodiment, the sacrum is divided into six characteristic features. For example... Figure 2 The diagram shows the distribution of characteristic parts of a normal sacrum. The marked parts are, in order: the apex of the midline of the spinous process, the lateral articular surface of the sacral wing, the apex of the midline of the spinous process, the spinal canal, the superior endplate of the sacrum, and the sacral foramen.

[0080] Step 122: Input the 3D point cloud data of the feature region into the deep learning neural network to obtain a 2D array of preset rows. The deep learning neural network is used to perform input transformation and feature transformation on the 3D point cloud data. The preset rows correspond to the number of feature regions.

[0081] The input transformation involves transforming the 3D point cloud data using a T-Net network to obtain a 3D transformation matrix, which is then used to perform orientation correction transformation on the 3D point cloud data.

[0082] The feature transformation consists of two MLP layers. The first MLP layer outputs an N×64 two-dimensional array, and the second MLP layer outputs an N×1024 two-dimensional array. Max pooling outputs a 1×1024 global feature array.

[0083] Concatenate the N×64 two-dimensional array with the 1×1024 global feature array (by copying the 1×1024 global feature array into an N×1024 two-dimensional array and then concatenating them) to obtain an N×1088 two-dimensional array.

[0084] Step 123: Output an N×6 two-dimensional array through MLP, and analyze each row of the two-dimensional array using a classification function (such as the softmax function) to obtain the maximum probability in each row of the two-dimensional array. The point cloud corresponding to the maximum probability is the feature point, thus achieving the goal of determining a corresponding feature point in each feature region.

[0085] After steps 123, it is also necessary to verify the category to which the feature points belong, which is achieved through the following steps:

[0086] Configure the optimal learning rate, initial momentum, model loss function, and optimizer type for training the deep learning neural network, where the optimizer is set to Adam and the model loss function is set to the cross-entropy loss function;

[0087] The three-dimensional point cloud data is labeled according to the characteristic parts as follows: the superior endplate of the sacrum, the sacral foramen, the spinal canal, the lateral articular surface of the sacral wing, the apex of the midline of the spinous process, and the apex of the midline of the spinous process.

[0088] The deep learning neural network is trained using a second training set to verify the category of the feature points. The labeled 3D point cloud data is used as the second dataset, which includes a second training set and a second test set, with a ratio of 8:2.

[0089] Step 130: Based on the feature points, the three-dimensional point cloud model and the three-dimensional model to be tested marked with the feature points to be tested are superimposed to obtain the superimposed image of the three-dimensional point cloud model and the three-dimensional model to be tested. The three-dimensional model to be tested is a three-dimensional point cloud model generated by the CT image of the sacrum to be tested. The feature points to be tested correspond to the feature points. The superimposed image is used to characterize the difference between the three-dimensional model to be tested and the three-dimensional point cloud model.

[0090] The 3D model to be tested was obtained based on CT images including the patient's sacrum. The method for obtaining the 3D model is the same as the method for obtaining the 3D point cloud model. Six feature regions were obtained by semantic segmentation of the normal sacrum. Therefore, the same method was used to perform semantic segmentation of the patient's sacrum to obtain six feature regions. The process of determining the feature points to be tested in each feature region of the patient is also the same as the process of determining feature points based on the feature regions of the normal sacrum, and will not be described in detail here.

[0091] In some embodiments, feature points and feature points to be measured are numbered respectively, and the feature parts to which the numbered points belong are corresponding.

[0092] Based on feature points, the 3D point cloud model is superimposed on the 3D model to be measured, which is marked with the feature points to be measured, to obtain a superimposed image of the 3D point cloud model and the 3D model to be measured. This is achieved through the following steps:

[0093] Step 131: Determine the scaling ratio based on the feature points, and scale the 3D model to be tested according to the scaling ratio until the first connecting line in the 3D point cloud model coincides with the second connecting line in the 3D model to be tested, thus obtaining the scaled 3D model to be tested. The first connecting line is the line between the first feature point and the second feature point in the 3D point cloud model, and the second connecting line is the line between the corresponding first feature point and the second feature point in the 3D model to be tested.

[0094] The scaling factor is represented by D. t / D s D t Let be a two-dimensional matrix, representing the length of the second connection in the 3D model to be tested, and t represent the number of samples in the 3D model to be tested. s It is also a two-dimensional matrix, representing the length of the first connecting line in the three-dimensional point cloud model.

[0095] Step 132: Perform face-to-face matching between the first plane in the 3D point cloud model and the second plane in the scaled 3D model to be tested. The first plane is composed of the first connecting line and the third feature point, and the second plane is composed of the second connecting line and the corresponding third feature point to be tested.

[0096] When performing face-to-face matching, the position, size and other information of the sacral 3D model are ignored, and only the topological information is retained.

[0097] Step 133: Obtain the standard points of the third, fourth, fifth, and sixth feature points to be tested. The standard point is equal to the quotient between the sum of the corresponding feature point matrices in each 3D model to be tested and the number of 3D models to be tested.

[0098] Let the third feature point be a three-dimensional matrix P. 3_t The standard point of the third feature point is obtained by summing the three-dimensional matrices of the third feature point in each three-dimensional model and then averaging them.

[0099] The standard points for the fourth, fifth, and sixth feature points to be measured are calculated using the same method.

[0100] Step 134: Based on standard points, overlay the 3D point cloud model and the 3D model to be measured to obtain an overlaid image of the 3D point cloud model and the 3D model to be measured. This is achieved through the following steps:

[0101] The standard points corresponding to the fourth, fifth, and sixth feature points to be tested are used to form a 3*3 feature matrix, which is represented as M. f ;

[0102] The 3x3 characteristic matrix M fConvert to a 4x4 homogeneous coordinate matrix, represented as M. f_h ;

[0103] The 4*4 homogeneous coordinate matrix M of each 3D model to be tested f_h After adding them together, we take the mean to obtain the transformation matrix.

[0104] Based on the transformation matrix The 3D model to be tested and the 3D point cloud model are overlaid. For example... Figure 3 The image shown is an overlay of the 3D point cloud model and the 3D model under test.

[0105] Figure 3 The first row is the three-dimensional model of patient 1 to be tested, the second row is the three-dimensional point cloud model of the normal sacrum, and the third row is the superimposed image of the three-dimensional point cloud model and the three-dimensional model to be tested. Figure 3 The fourth row is the three-dimensional model of patient 2 to be tested, the fifth row is the three-dimensional point cloud model of the normal sacrum, and the sixth row is the superimposed image of the three-dimensional point cloud model and the three-dimensional model to be tested.

[0106] from Figure 3 As can be seen from the first and third rows of the test 3D models, the size of the sacrum is different. However, the method of this application can fit and superimpose the two patient sacral models with the 3D point cloud model of the normal sacrum to obtain superimposed images. The difference between the patient's sacrum and the normal sacrum can be intuitively shown by the color or grayscale of the superimposed image. Then, the doctor can determine the implantation position of the sacroiliac screw based on the difference.

[0107] By constructing a three-dimensional point cloud model of a normal sacrum, the height and bone density of the normal sacrum can be intuitively characterized. Furthermore, by using a deep learning neural network, corresponding feature points are selected from the characteristic areas of the sacrum, which facilitates subsequent feature point matching. Based on the feature points, the three-dimensional point cloud model is superimposed with the three-dimensional model to be tested, which is marked with the feature points to be tested, to obtain a superimposed image of the three-dimensional point cloud model and the three-dimensional model to be tested. The superimposed image intuitively shows the difference between the normal sacrum and the patient's sacrum, and based on this difference, doctors can more easily determine the implantation position of the sacroiliac screw.

[0108] In some embodiments, Figure 4 A block diagram of a deep learning-based sacral 3D model matching device is shown, as follows: Figure 4 As shown, a deep learning-based sacral 3D model matching device includes a generation module 401, a selection module 401, and an overlay module 403.

[0109] The generation module 401 is used to acquire CT images including the normal sacrum, and to perform three-dimensional reconstruction of the CT images through a deep learning neural network to generate a three-dimensional point cloud model representing the three-dimensional model of the normal sacrum. The three-dimensional point cloud model is used to characterize information including the height and bone density of the normal sacrum.

[0110] The selection module 402 is used to obtain the characteristic parts of a normal sacrum, and to select corresponding feature points from the characteristic parts using a deep learning neural network. The characteristic parts are used to characterize the compositional structure of a normal sacrum.

[0111] The overlay module 403 is used to overlay the three-dimensional point cloud model and the three-dimensional model to be tested marked with the feature points according to the feature points, so as to obtain the overlay image of the three-dimensional point cloud model and the three-dimensional model to be tested. The three-dimensional model to be tested is a three-dimensional point cloud model generated by CT image of the patient's sacrum. The feature points to be tested correspond to each other. The overlay image is used to characterize the difference between the three-dimensional model to be tested and the three-dimensional point cloud model.

[0112] Each module in the aforementioned deep learning-based sacral 3D model matching device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0113] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the discussion in some embodiments above is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A method for matching a three-dimensional model of the sacrum based on deep learning, characterized in that, include: A CT image including a normal sacrum is acquired, and the CT image is reconstructed in three dimensions using a deep learning neural network to generate a three-dimensional point cloud model representing a three-dimensional model of a normal sacrum. The three-dimensional point cloud model is used to characterize information including the height and bone density of the normal sacrum. The characteristic regions of the normal sacrum are obtained, and corresponding feature points are selected from the characteristic regions using a deep learning neural network. The characteristic regions are used to characterize the compositional structure of the normal sacrum. Based on the feature points, the three-dimensional point cloud model and the three-dimensional model to be tested marked with the feature points to be tested are superimposed to obtain the superimposed image of the three-dimensional point cloud model and the three-dimensional model to be tested. The three-dimensional model to be tested is a three-dimensional point cloud model generated by CT images of the patient's sacrum. The feature points to be tested correspond to the feature points. The superimposed image is used to characterize the difference between the three-dimensional model to be tested and the three-dimensional point cloud model. The step of superimposing the 3D point cloud model with the 3D model to be tested marked with the feature points to be tested, based on the feature points, to obtain a superimposed image of the 3D point cloud model and the 3D model to be tested, further includes: The scaling ratio is determined based on the feature points, and the 3D model to be tested is scaled according to the scaling ratio until the first connecting line in the 3D point cloud model coincides with the second connecting line in the 3D model to be tested, thus obtaining the scaled 3D model to be tested. The first connecting line is the line between the first feature point and the second feature point in the 3D point cloud model, and the second connecting line is the line between the corresponding first feature point and the second feature point in the 3D model to be tested. A face-to-face matching is performed between the first plane in the three-dimensional point cloud model and the second plane in the scaled three-dimensional model to be tested. The first plane is composed of the first connecting line and the third feature point, and the second plane is composed of the second connecting line and the corresponding third feature point to be tested. Obtain the standard points of the third, fourth, fifth, and sixth feature points to be tested. The standard point is equal to the quotient between the sum of the corresponding feature point matrices in each of the three-dimensional models to be tested and the number of three-dimensional models to be tested. Based on the standard points, the three-dimensional point cloud model and the three-dimensional model to be tested are superimposed to obtain a superimposed image of the three-dimensional point cloud model and the three-dimensional model to be tested. Based on the feature points, a scaling ratio is determined according to the feature points, and the 3D model to be tested is scaled according to the scaling ratio until the first connecting line in the 3D point cloud model coincides with the second connecting line in the 3D model to be tested, further including: Calculate the first length of the first line connecting any two feature points in the three-dimensional point cloud model, where the first length is represented by a first matrix. Calculate the second length of the second line connecting two feature points in the three-dimensional model to be tested, and the second length is represented by a second matrix; Calculate the scaling ratio based on the first and second matrices; The three-dimensional model to be tested is adjusted based on the scaling ratio so that the second connecting line coincides with the first connecting line; Based on the standard points, the three-dimensional point cloud model and the three-dimensional model to be measured are superimposed, further including: The standard points corresponding to the fourth, fifth, and sixth feature points to be tested are combined to form a 3*3 feature matrix; The 3*3 feature matrix is ​​converted into a 4*4 homogeneous coordinate matrix; The transformation matrix is ​​obtained by summing the 4*4 homogeneous coordinate matrices of each of the three-dimensional models to be tested and taking the average value. Based on the transformation matrix, the three-dimensional model to be tested and the three-dimensional point cloud model are superimposed.

2. The deep learning-based sacral 3D model matching method as described in claim 1, characterized in that, The deep learning neural network includes an encoder network, a fully connected layer, a decoder network, and a pixel classification layer. It performs three-dimensional reconstruction of the CT image using the deep learning neural network to generate a three-dimensional point cloud model representing a normal sacrum, further including: Based on the CT image, a set of feature maps is generated using the convolution of the encoder network, and then the feature maps are input into the fully connected layer. After adding features containing sacral location information to the feature map through the fully connected layer, the feature map is input into the decoder network; The feature map is sampled by the decoder network to generate a sparse feature map, which is then used to generate a dense feature map through convolution. The dense feature map is then input into the pixel classification layer. The pixel classification layer predicts the largest category corresponding to the probability of each pixel in the dense feature map, and the category includes the sacrum and other body parts besides the sacrum; The CT image is semantically segmented based on the category to obtain a two-dimensional array. The two-dimensional array is used to represent the gray value of each pixel in the CT image, wherein the gray value of the pixels that make up the sacrum is 255, and the gray value of the pixels that make up the other parts is 0. Stacking the two-dimensional array generates a three-dimensional point cloud model representing a three-dimensional model of the sacrum. The two-dimensional array includes the layer position, image height, and pixel density information in the CT image. The image height corresponds to the height, and the pixel density information corresponds to the bone density information.

3. The deep learning-based sacral 3D model matching method as described in claim 2, characterized in that, After predicting the largest class corresponding to the probability of each pixel in the dense feature map through the pixel classification layer, the method further includes: Set the optimal learning rate, initial momentum, model loss function, and optimizer type for training the deep learning neural network, wherein the optimizer is set to Adam and the model loss function is set to the cross-entropy loss function. The CT images are labeled according to the largest category, and the deep learning neural network is trained using a first training set to verify the category to which the feature region belongs. The labeled CT images constitute a first dataset, which includes a first training set and a first test set.

4. The deep learning-based sacral 3D model matching method as described in claim 1, characterized in that, The step of obtaining the characteristic location of the normal sacrum further includes: The normal sacrum is classified to obtain the characteristic locations of the normal sacrum; The characteristic parts of the normal sacrum include: the superior endplate of the sacrum, the sacral foramen, the spinal canal, the lateral articular surface of the sacral wing, the apex of the midline of the spinous process, and the apex of the lower part of the midline of the spinous process.

5. The deep learning-based sacral 3D model matching method as described in claim 1, characterized in that, The step of selecting corresponding feature points from the feature region using a deep learning neural network further includes: The three-dimensional point cloud data of the feature region is input into a deep learning neural network to obtain a two-dimensional array with preset rows. The deep learning neural network is used to perform input transformation and feature transformation on the three-dimensional point cloud data. The preset rows correspond to the number of the feature regions. The two-dimensional array in each row is analyzed using a classification function to obtain the maximum probability in each row of the two-dimensional array. The point cloud corresponding to the maximum probability is the feature point.

6. The deep learning-based sacral 3D model matching method as described in claim 5, characterized in that, After selecting corresponding feature points from the feature region using a deep learning neural network, the method further includes: Set the optimal learning rate, initial momentum, model loss function, and optimizer type for training the deep learning neural network, wherein the optimizer is set to Adam and the model loss function is set to the cross-entropy loss function. The 3D point cloud data is labeled according to the said feature parts, and the deep learning neural network is trained using the second training set to verify the category to which the feature points belong. The labeled 3D point cloud data is used as the second dataset, which includes a second training set and a second test set.

7. A sacral three-dimensional model matching device based on deep learning, characterized in that, include: The generation module is used to acquire CT images including a normal sacrum, and to perform three-dimensional reconstruction of the CT images using a deep learning neural network to generate a three-dimensional point cloud model representing a three-dimensional model of a normal sacrum. The three-dimensional point cloud model is used to characterize information including the height and bone density of the normal sacrum. A selection module is used to obtain the characteristic parts of the normal sacrum, and to select corresponding feature points from the characteristic parts using a deep learning neural network. The characteristic parts are used to characterize the compositional structure of the normal sacrum. The overlay module is used to overlay the three-dimensional point cloud model and the three-dimensional model to be tested marked with the feature points according to the feature points, so as to obtain an overlay image of the three-dimensional point cloud model and the three-dimensional model to be tested. The three-dimensional model to be tested is a three-dimensional point cloud model generated by CT image of the patient's sacrum. The feature points to be tested correspond to the feature points. The overlay image is used to characterize the difference between the three-dimensional model to be tested and the three-dimensional point cloud model. The step of superimposing the 3D point cloud model with the 3D model to be tested marked with the feature points to be tested, based on the feature points, to obtain a superimposed image of the 3D point cloud model and the 3D model to be tested, further includes: The scaling ratio is determined based on the feature points, and the 3D model to be tested is scaled according to the scaling ratio until the first connecting line in the 3D point cloud model coincides with the second connecting line in the 3D model to be tested, thus obtaining the scaled 3D model to be tested. The first connecting line is the line between the first feature point and the second feature point in the 3D point cloud model, and the second connecting line is the line between the corresponding first feature point and the second feature point in the 3D model to be tested. A face-to-face matching is performed between the first plane in the three-dimensional point cloud model and the second plane in the scaled three-dimensional model to be tested. The first plane is composed of the first connecting line and the third feature point, and the second plane is composed of the second connecting line and the corresponding third feature point to be tested. Obtain the standard points of the third, fourth, fifth, and sixth feature points to be tested. The standard point is equal to the quotient between the sum of the corresponding feature point matrices in each of the three-dimensional models to be tested and the number of three-dimensional models to be tested. Based on the standard points, the three-dimensional point cloud model and the three-dimensional model to be tested are superimposed to obtain a superimposed image of the three-dimensional point cloud model and the three-dimensional model to be tested. Based on the feature points, a scaling ratio is determined according to the feature points, and the 3D model to be tested is scaled according to the scaling ratio until the first connecting line in the 3D point cloud model coincides with the second connecting line in the 3D model to be tested, further including: Calculate the first length of the first line connecting any two feature points in the three-dimensional point cloud model, where the first length is represented by a first matrix. Calculate the second length of the second line connecting two feature points in the three-dimensional model to be tested, and the second length is represented by a second matrix; Calculate the scaling ratio based on the first and second matrices; The three-dimensional model to be tested is adjusted based on the scaling ratio so that the second connecting line coincides with the first connecting line; Based on the standard points, the three-dimensional point cloud model and the three-dimensional model to be measured are superimposed, further including: The standard points corresponding to the fourth, fifth, and sixth feature points to be tested are combined to form a 3*3 feature matrix; The 3*3 feature matrix is ​​converted into a 4*4 homogeneous coordinate matrix; The transformation matrix is ​​obtained by summing the 4*4 homogeneous coordinate matrices of each of the three-dimensional models to be tested and taking the average value. Based on the transformation matrix, the three-dimensional model to be tested and the three-dimensional point cloud model are superimposed.

Images