Training method and device of to-be-detected part feature point extraction model, and operation planning parameter automatic generation method and device

Abstract

The invention discloses a training method and device of a to-be-detected part feature point extraction model and an automatic surgical planning parameter generation method and device.The method comprises the steps that an obverse and lateral 2D image containing a complete to-be-detected part and an X-ray image containing the to-be-detected part in a normal posture are obtained; x-ray images of a to-be-detected part in various pathological states are used as training data, so that the trained to-be-detected part feature point extraction model can pay attention to other parts which affect the patient part and are balanced in stress, and feature points of a coronal plane (normal position) or a sagittal plane (lateral position) in different postures are considered; therefore, the model is high in generalization when facing various lesion data, and the recognition accuracy of the position and the bending posture of the to-be-detected part (such as the spine) is good.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and particularly to a training method and device for a feature point extraction model of a part to be detected, and a method and device for automatically generating surgical planning parameters. Background Art

[0002] Planning spinal surgery and correcting spinal deformities are complex. Any change in one or more vertebrae may affect the entire spinal column distribution and range of motion, so this "chain effect" must be considered during surgical planning. In addition, since the spine is physically connected to the pelvis, and thus physically connected to the femoral head and lower limbs, any surgical intervention on the spine will also affect the balance of other parts of the patient's body and the whole body. If only considering local surgical planning methods for specific vertebrae in the potential surgical area without considering all the characteristics affected by the surgery in the patient (such as all vertebrae of the spine, pelvis, and lower limbs), it may be inappropriate and may even have negative results.

[0003] In order to achieve a personalized surgical planning scheme for the overall balance of the spine-pelvis-lower limbs, various parameters need to be measured based on image data: such as thoracolumbar cobb angle, sagittal vertical axis deviation SVA, pelvic incidence PI, pelvic tilt angle PT, sacral slope SS, femoral shaft, etc. Conducting a comprehensive clinical index analysis of DR images in clinical practice is time-consuming and laborious, reducing the diagnosis and treatment efficiency; in addition, there are often problems in DR images such as blurred bone boundaries or even partial missing, and it is only by relying on the doctor's experience for imagination and subjective judgment, and the standards cannot be unified.

[0004] With the increasingly in-depth application research of deep learning technology in segmentation and landmark detection, its application in the field of medical imaging has become more and more mature. In the prior art, although the existing parameter calculation methods reduce the dependence on doctors' experience and human errors in diagnosis based on AI detection technology, there are still the following disadvantages: the detection range is limited, for example, spinal detection only focuses on the lumbar part, or only focuses on the pelvic part, or only focuses on the coronal plane or sagittal plane, etc.; the model has poor generalization ability when facing various lesion data, and the recognition accuracy of the position and bending posture of the spine is poor.

[0005] In the related art, a skeletal sagittal balance parameter detection system is provided, but this technology only considers sagittal skeletal balance parameters, and the method of detecting bone contours based on semantic segmentation has uncertainty in the generalization ability and accuracy of the model in the case of spinal deformities where the vertebral segments cannot be clearly distinguished. Summary of the Invention

[0006] The object of the present invention is to overcome the above technical deficiencies and provide a training method and device for a feature point extraction model of a part to be detected, as well as a method and device for automatically generating surgical planning parameters, so as to solve the problems in the related art that due to the limited detection range of X-ray images, the generalization ability of the model is poor when facing various lesion data, and the recognition accuracy of the position and bending posture of the spine is poor.

[0007] To achieve the above technical object, the present invention adopts the following technical solutions:

[0008] According to the first aspect of the present invention, there is provided a training method for a feature point extraction model of a part to be detected, including:

[0009] Obtain training data, where the training data is multiple anteroposterior and lateral 2D images including the complete part to be detected;

[0010] Construct a model trainer, which is used to extract feature points from the training data and output feature points of the part to be detected according to the feature points;

[0011] Input the training data into the model trainer for training until the model converges, and determine the model at this time as the feature point extraction model of the part to be detected.

[0012] Preferably, the anteroposterior and lateral 2D images cover X-ray images of the part to be detected in a normal body posture and X-ray images of the part to be detected in various pathological conditions. If the part to be detected is the spine and pelvis, the obtaining of the training data includes:

[0013] Obtain anteroposterior and lateral 2D images including the complete part to be detected, and the anteroposterior and lateral 2D images at least include: the spine, the pelvis, and the femoral part including the complete femoral head;

[0014] Cut the anteroposterior and lateral 2D images into multiple image blocks from bottom to top or from top to bottom. Each image block contains complete information of the feature points to be detected. Among them, the image block of the pelvis part contains all information of the pelvis feature points, and the image block of the spine contains complete information of the spine feature points, and calculate the heat map of each image block;

[0015] Determine the image blocks and the heat map of each image block as the training data, and input the image blocks and the heat map of each image block into the constructed model trainer in pairs.

[0016] Preferably, the constructing of the model trainer includes:

[0017] Based on the neural network model HRNet, a first model trainer is constructed; the input of the first model trainer is the image patches of the cervical vertebra and pelvic regions and the heat map of each image patch, and the output is the feature points of the cervical vertebra and pelvic regions; the first model trainer is used to predict the feature points of the input multiple image patches, generate the heat map of the corresponding image patches, and compare the generated heat map with the input heat map until the loss function between the two meets the preset requirements;

[0018] Based on the residual neural network model ResNet, a second model trainer is constructed; the input of the second model trainer is the image patches of the spine and the heat map of each image patch, and the output is the feature points of each vertebral body in the spine region; the second model trainer is used to determine the upper and lower intervals of the spine according to the feature points of the cervical vertebra and pelvic regions output by the first model trainer, and determine the feature points of each vertebral body on the spine within the upper and lower regions.

[0019] Preferably, the first model trainer includes:

[0020] Construct n neural network layers of different scales, and the input image patch of each scale layer is W / 2 n 、H / 2 n in size, representing different scale information; each scale layer is composed of m - 2n convolutional blocks, representing the depth information of the network at each scale; where W is the width of the input image patch, H is the height of the input image patch, n is the number of scales of the network, m is the network depth at the first scale, m - 2n is the network depth at each scale, n≥0, m≥0 are integers;

[0021] The feature fusion of each scale neural network layer is performed at the intermediate feature expression layer and the final feature expression layer, and the final output heat map size is W×H×K, where K is the number of feature points, and the positions of K feature points are obtained by detecting the extreme points on the heat map.

[0022] Preferably, the second model trainer includes:

[0023] An encoder and a neural network convolutional layer cascaded at the output end of the encoder. The encoder is a residual neural network model ResNet structure. The input of the encoder is the image patches of the spine and the heat map of each image patch. There is mask information of the four corner points of each vertebral body in the spine and mask information of the center point of each vertebral body on the heat map; the neural network convolutional layer includes:

[0024] The first neural network convolutional layer is used to output the heat map of the mask center point of each vertebral body in the spine;

[0025] The second neural network convolutional layer is used to output the position offset between the center point of each vertebral body and the mask center point of the vertebral body;

[0026] The third neural network convolutional layer is used to output the position offsets of the four corner points of each vertebral body and the center point of the vertebral body mask.

[0027] The neural network convolutional layer outputs the position vectors of the center points of each vertebral body of the spine and the position vectors of the four corner points of each vertebral body.

[0028] According to the second aspect of the present invention, there is provided a method for automatically generating surgical planning parameters, including:

[0029] Obtain the anteroposterior and lateral 2D images containing the complete part to be detected;

[0030] Input the anteroposterior and lateral 2D images into the feature point extraction model of the part to be detected trained by the above method to obtain the feature points of the part to be detected;

[0031] Based on the relative smooth continuity of the spinal curve, perform outlier detection on the feature points and remove the outliers;

[0032] Calculate various surgical planning parameters for the anteroposterior and lateral positions according to the feature points after removing the outliers.

[0033] Preferably, the performing outlier detection on the feature points and removing the outliers based on the relative smooth continuity of the spinal curve includes:

[0034] From bottom to top or from top to bottom, for each image block on the spine, sequentially perform the following steps, including:

[0035] Predict the vertebral body center curve on the first image block according to the feature points on the first image block;

[0036] Determine whether there are outliers in the vertebral body center curve on the first image block. If so, predict the vertebral body center curve on the second image block according to the feature points on the second image block; otherwise, output the vertebral body center curve on the first image block;

[0037] Determine whether there are outliers in the vertebral body center curve on the second image block. If so, fit the vertebral body center curves on the first image block and the second image block to remove the outliers; otherwise, output the vertebral body center curve on the second image block.

[0038] According to the third aspect of the present invention, there is provided a training device for a feature point extraction model of a part to be detected, including:

[0039] An acquisition module for acquiring training data, where the training data is multiple anteroposterior and lateral 2D images containing the complete part to be detected, and the anteroposterior and lateral 2D images cover X-ray images of the part to be detected in normal postures and X-ray images of the part to be detected in various pathological conditions;

[0040] A building module for building a model trainer, which is used to extract feature points from the training data and output the feature points of the part to be detected according to the feature points;

[0041] A training module for inputting the training data into the model trainer for training until the model converges, and determining the model at this time as a feature point extraction model for the part to be detected.

[0042] According to the fourth aspect of the present invention, there is provided a surgical planning parameter automatic generation device, including:

[0043] An acquisition module for acquiring anterior-posterior and lateral 2D images containing the complete part to be detected;

[0044] An input module for inputting the anterior-posterior and lateral 2D images into the feature point extraction model for the part to be detected trained by the above method to obtain the feature points of the part to be detected;

[0045] A detection module for performing abnormal point detection on the feature points based on the relative smooth continuity of the spinal curve and removing the abnormal points;

[0046] A calculation module for calculating various surgical planning parameters of the anterior-posterior and lateral positions according to the feature points after removing the abnormal points.

[0047] According to the fifth aspect of the present invention, there is provided an electronic device, including:

[0048] A processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus;

[0049] A memory for storing a computer program;

[0050] A processor for implementing the above method when executing the program stored in the memory.

[0051] According to the sixth aspect of the present invention, there is provided a non-transitory computer-readable storage medium storing computer instructions, and the computer instructions are used to cause a computer to execute the above method.

[0052] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0053] By obtaining the anterior-posterior and lateral 2D images that include the complete part to be detected and the X-ray images of the part to be detected under normal postures, as well as the X-ray images of the part to be detected under various pathological conditions as training data, the trained feature point extraction model for the part to be detected can pay attention to other parts that affect the force balance of the diseased part, and take into account the feature points under different postures in the coronal plane (anterior-posterior view) or sagittal plane (lateral view), making the model have strong generalization ability when facing various lesion data and good recognition accuracy for the position and bending posture of the part to be detected (for example, the spine).

[0054] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flowchart of a training method for a feature point extraction model of a part to be detected shown according to an exemplary embodiment;

[0056] Figures 2A - 2B is a schematic diagram of annotating the anterior-posterior and lateral 2D images of the part to be detected shown according to an exemplary embodiment;

[0057] Figures 3A - 3C is a schematic diagram of a kind of training data shown according to an exemplary embodiment;

[0058] Figure 4 is a flowchart of constructing a model trainer shown according to an exemplary embodiment;

[0059] Figure 5 is a schematic structural diagram of a first model trainer shown according to an exemplary embodiment;

[0060] Figure 6 is a schematic structural diagram of a second model trainer shown according to an exemplary embodiment;

[0061] Figure 7 is a flowchart of a method for automatically generating surgical planning parameters shown according to an exemplary embodiment;

[0062] Figure 8 is a schematic diagram of the extracted feature points having outliers shown according to an exemplary embodiment;

[0063] Figure 9 is a comparison graph of the change of the spinal center point coordinate curve with or without outliers (outlier) shown according to an exemplary embodiment;

[0064] Figure 10 is a vertebral body center curve graph fitted when there are outliers under the center point coordinates of different vertebral bodies shown according to an exemplary embodiment;

[0065] Figure 11 It is a vertebral center curve graph fitted after removing abnormal points under the center point coordinates of different vertebrae shown according to an exemplary embodiment;

[0066] Figure 12 It is a schematic block diagram of a training device for a feature point extraction model of a part to be detected shown according to an exemplary embodiment;

[0067] Figure 13 It is a schematic block diagram of a device for automatically generating surgical planning parameters shown according to an exemplary embodiment;

[0068] Figure 14 A schematic block diagram of an electronic device shown according to an exemplary embodiment. Detailed implementation manners

[0069] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0070] As described in the background art above, in the related art, due to the limited detection range of X-ray images, the model has poor generalization ability when facing various lesion data, and has poor recognition accuracy for the position and bending posture of the spine.

[0071] In order to effectively solve the problems in the related art, the present invention provides a training method and device for a feature point extraction model of a part to be detected, and a method and device for automatically generating surgical planning parameters, which will be specifically described below.

[0072] Embodiment 1

[0073] Figure 1 It is a flowchart of a training method for a feature point extraction model of a part to be detected shown according to an exemplary embodiment. As Figure 1 shown, the method includes:

[0074] Step S11: Obtain training data, where the training data is multiple pairs of anterior-posterior and lateral 2D images including the complete part to be detected, and the anterior-posterior and lateral 2D images cover X-ray images of the part to be detected under normal postures and X-ray images of the part to be detected under various morbid states;

[0075] Step S12: Construct a model trainer, where the model trainer is used to extract feature points from the training data and output feature points of the part to be detected according to the feature points;

[0076] Step S13: Input the training data into the model trainer for training until the model converges, and determine the model at this time as the feature point extraction model for the part to be detected.

[0077] It should be noted that the technical solution provided in this embodiment runs in the controller of the medical device in specific practice, or is loaded and runs in the electronic device connected to the controller. The controller of the medical device executes the corresponding method by calling the program stored in the electronic device.

[0078] In specific practice, the technical solution provided in this embodiment can be applied to different surgical scenarios. For example, the extraction of feature points of the diseased spine and the extraction of feature points of the injured knee joint.

[0079] It can be understood that the technical solution provided in this embodiment, by obtaining the anteroposterior and lateral 2D images containing the complete part to be detected and the X-ray images covering the part to be detected in the normal body posture, as well as the X-ray images of the part to be detected in various pathological conditions as the training data, enables the trained feature point extraction model for the part to be detected to pay attention to the other parts affecting the force balance of the diseased part, and takes into account the feature points in different body postures in the coronal plane (anteroposterior) or sagittal plane (lateral), making the model have strong generalization ability when facing various pathological data and good recognition accuracy for the position and bending posture of the part to be detected (such as the spine).

[0080] In specific practice, if the part to be detected is the spine and pelvis, obtaining the training data in step S11 includes:

[0081] Obtain the anteroposterior and lateral 2D images containing the complete part to be detected, and the anteroposterior and lateral 2D images at least include: the spine, the pelvis, and the femoral part containing the complete femoral head;

[0082] Cut the anteroposterior and lateral 2D images into multiple image blocks from bottom to top or from top to bottom. Each image block contains the complete information of the feature points to be detected. Among them, the image block of the pelvis part contains all the information of the pelvis feature points, and the image block of the spine contains the complete information of the spine feature points, and calculate the heat map of each image block;

[0083] Determine the image blocks and the heat map of each image block as the training data, and input the image blocks and the heat map of each image block into the constructed model trainer in pairs.

[0084] If the part to be detected is the spine, obtain the anteroposterior and lateral 2D images obtained by X-ray scanning, which at least include the part from the head to the femur.

[0085] See Figure 2A (anteroposterior) and Figure 2B(Lateral view), mark the feature points of each vertebral body of the spine: Mark the position information of the four corner points of each vertebral body on the anteroposterior and lateral view diagrams, and the position of the center point of the vertebral body (which can be obtained by calculation). In the data with severe spinal deformities, refer to the anteroposterior and lateral images for marking.

[0086] Mark the feature points of other parts except the spine: The center of the femoral head, the two end points of the upper edge of the sacrum, the midpoint of C7, etc. on the lateral view; the center of the femoral head (the femoral head is spherical, and the center of the femoral head is the center of the fitted circle in the anteroposterior and lateral view diagrams), the outer edge point of the acetabulum, the teardrop point, etc. on the anteroposterior view.

[0087] See Figures 3A - 3C , the training data diversity coverage: The data sources cover the lesions of spinal coronal deformities, sagittal deformities, and the pathology of the pelvis and spine parts containing implants, etc.

[0088] See Figure 4 , in specific practice, in step S12, construct a model trainer, including:

[0089] Based on the neural network model HRNet, construct a first model trainer; the input of the first model trainer is the image patches of the cervical vertebra and pelvis parts and the heat map of each image patch, and the output is the feature points of the cervical vertebra and pelvis parts; the first model trainer is used to predict the feature points of the input multiple image patches, generate the heat map corresponding to the image patches, and compare the generated heat map with the input heat map until the loss function between the two meets the preset requirements;

[0090] Based on the residual neural network model ResNet, construct a second model trainer; the input of the second model trainer is the image patches of the spine and the heat map of each image patch, and the output is the surgical planning parameters for the force balance of the spine part; the second model trainer is used to determine the upper and lower intervals of the spine according to the feature points of the cervical vertebra and pelvis parts output by the first model trainer, and determine the feature points of each vertebral body on the spine within the upper and lower regions.

[0091] Among them, see Figure 5 , the first model trainer includes:

[0092] Construct n neural network layers with different scales, and the input image patch of each scale layer is W / 2 n 、H / 2 n in size, representing different scale information; each scale layer is composed of m - 2n convolutional blocks, representing the network depth information at each scale; where W is the width of the input image patch, H is the height of the input image patch, n is the number of network scales, m is the network depth at the first scale, m - 2n is the network depth at each scale, n≥0, m≥0 are integers;

[0093] The neural network layers at each scale perform feature fusion at the intermediate feature expression layer and the final feature expression layer. The final output heatmap has a size of W×H×K, where K is the number of feature points. The positions of the K feature points are obtained by detecting the extreme points on the heatmap.

[0094] It should be noted that the first model trainer mainly trains the model for the feature points of the head and neck (the cervical vertebra belongs to a part of the spine) and the pelvis. The distribution of the feature points in this part is sparse, unclear, and the pose changes greatly. In this embodiment, a neural network structure with high resolution, High Resolution Network (HRNet), is selected for model training, and at the same time, a data augmentation method with larger deformation is used to make the model better in detection accuracy and generalization performance.

[0095] See Figure 5 , in the first model trainer, the vertical direction represents the neural network convolutional layers at multiple scales, the horizontal direction represents the depth of each neural network convolutional layer, and the cross-interconnection of different scales up and down represents multi-scale feature fusion. The first model trainer selects a single-channel grayscale image as the input. The width and height of the image block sample are W and H respectively. If there are K feature points to be detected, the dimension of the input image block in model training is W×H, and the dimension of the heatmap of the K feature points is W×H×K. For example, if the size of the input image block is 512×512 and there are 6 feature points to be detected, the output heatmap size of the first-scale neural network convolutional layer is 512×512×6; the size of the input image block of the second-scale neural network convolutional layer is 256×256, and the output heatmap is 256×256×6... and so on. The neural network convolutional layers at each scale not only perform feature fusion at the final feature expression layer, but also perform feature fusion at the intermediate feature expression layer. The final output heatmap size is W×H×K. During the training process, the calculation of the loss function can be obtained by comparing the mean square error between the predicted heatmap (the heatmap output by the first model trainer) and the ground truth heatmap (the heatmap input to the first model trainer, which is an artificially annotated heatmap). During the prediction process, the positions of the K feature points are obtained by detecting the extreme points of the heatmap.

[0096] Among them, see Figure 6 , the second model trainer includes:

[0097] An encoder and a neural network convolutional layer cascaded at the output end of the encoder. The encoder is a ResNet structure of a residual neural network model. The input of the encoder is an image block of the spine and the heatmap of each image block. There is mask information of the four corner points of each vertebral body of the spine and mask information of the center point of each vertebral body on the heatmap; the neural network convolutional layer includes:

[0098] The first neural network convolutional layer is used to output the heatmap of the mask center point of each vertebral body of the spine;

[0099] The second neural network convolutional layer is used to output the position offset between the center point of each vertebral body and the center point of the vertebral body mask;

[0100] The third neural network convolutional layer is used to output the position offset between the four corner points of each vertebral body and the center point of the vertebral body mask;

[0101] The neural network convolutional layer outputs the position vector of the center point of each vertebral body of the spine and the position vectors of the four corner points of each vertebral body.

[0102] See Figure 6 , due to the highly similar and closely connected characteristics of each vertebral body of the spine, it is not suitable to detect the feature points of each vertebral body separately. Therefore, the second model trainer adopts a detection mode of detecting the center point of the vertebral body plus the offset of the four corner points from the center point of the vertebral body to train the neural network.

[0103] The second model trainer uses ResNet as the encoder backbone network and adopts a network architecture similar to U-net. If the input image size is 512x256 dimensions, the ground truth includes the center point position mask map of each vertebral body segment, which is also 512x256 dimensions. It outputs the center point of the mask of each vertebral body segment, the offset (dx, dy) of the center point from the center point of the segment mask, and the offset of the four corner points of each vertebral body from the center point of the segment mask; the loss function is the weighted sum of three prediction errors. The three prediction errors are the deviation between the predicted center point position mask and the ground truth center point position mask, the deviation between the predicted center point position offset and the ground truth center point position offset, and the deviation between the predicted offset of the four corner points and the ground truth offset of the four corner points; finally, it outputs the position vectors of all vertebral body center points and the position vectors of the four corner points of each vertebral body.

[0104] In summary, for the technical solution provided in this embodiment, the constructed model trainer, for any input coronal (frontal) or sagittal (lateral) image, first takes multiple image patches from bottom to top or from top to bottom through the first model trainer for prediction, and uses the point with the highest confidence in the heat map as the feature point, and outputs the detection position of the feature point. The feature points at least include: the center of C7 of the cervical vertebra. According to the position of the feature points predicted by the first model trainer, such as the positions of the C7 of the cervical vertebra and the sacral vertebra feature points, the thoracic and lumbar regions can be roughly located and used as the input of the second model trainer for spine feature point detection.

[0105] It can be understood that for the technical solution provided in this embodiment, different models are used for feature point extraction training for different parts. For the pelvic and lower limb parts, the key points to be detected are sparse, some feature points are blurred and unclear, and the posture changes greatly. Therefore, a first model trainer with high resolution is selected for model training; for the characteristics that each vertebral body of the spine is closely connected and has high similarity, a second model trainer applicable to the offset of the center point and four corner points of the vertebral body is selected for model training, so that the feature point extraction model of the part to be detected trained can pay attention to other parts that affect the force balance of the diseased part, and consider the feature points in different postures of the coronal plane (frontal view) or sagittal plane (lateral view), so that the model has strong generalization ability when facing various lesion data, and has good recognition accuracy for the position and bending posture of the part to be detected (for example, the spine).

[0106] Embodiment 2

[0107] Figure 7 is a flowchart of a method for automatically generating surgical planning parameters shown according to an exemplary embodiment, as Figure 7 shown, the method includes:

[0108] Step S21, obtain the frontal and lateral 2D images containing the complete part to be detected;

[0109] Step S22, input the frontal and lateral 2D images into the feature point extraction model of the part to be detected trained by the above method to obtain the feature points of the part to be detected;

[0110] Step S23, based on the relatively smooth continuity of the spinal curve, perform outlier detection on the feature points and remove the outliers;

[0111] Step S24, calculate various surgical planning parameters of the frontal and lateral views according to the feature points after removing the outliers.

[0112] It should be noted that the technical solution provided in this embodiment, in specific practice, runs in the controller of the medical device, or is loaded and runs in an electronic device connected to the controller. The controller of the medical device executes the corresponding method by calling the program stored in the electronic device.

[0113] In specific practice, the technical solution provided in this embodiment can be applied to different surgical scenarios. For example, it can be used for feature point extraction of a diseased spine and feature point extraction of an injured knee joint.

[0114] It can be understood that, for the technical solution provided in this embodiment, by obtaining the anteroposterior and lateral 2D images including the complete part to be detected for feature point extraction of the part to be detected, the extracted feature points can focus on other parts that affect the force balance of the diseased part. Based on this, various surgical planning parameters in the anteroposterior and lateral positions are more accurate, with strong generalization ability when facing various lesion data, and good recognition accuracy for the position and bending posture of the part to be detected (for example, the spine).

[0115] In specific practice, in step S23, based on the relative smooth continuity of the spine curve, abnormal point detection is performed on the feature points to remove abnormal points, including:

[0116] From bottom to top or from top to bottom, for each image block on the spine, the following steps are sequentially executed, including:

[0117] According to the feature points on the first image block, predict the vertebral body center curve on the first image block;

[0118] Judge whether there are abnormal points on the vertebral body center curve of the first image block. If so, according to the feature points on the second image block, predict the vertebral body center curve on the second image block; otherwise, output the vertebral body center curve on the first image block;

[0119] Judge whether there are abnormal points on the vertebral body center curve of the second image block. If so, fit the vertebral body center curves on the first image block and the second image block to remove the abnormal points; otherwise, output the vertebral body center curve on the second image block.

[0120] See Figure 8 , when detecting the feature points of the spine area, under severe deformity or the image occlusion of other bones in the upper thoracic vertebrae, problems such as feature point dislocation are likely to occur (such as Figure 8 the part circled in the circle in). According to the overall morphological smoothness of the human spine curve in physiology, if there are abnormal points, there will be high-frequency interference in the spine curve. As Figure 9 shows the comparison chart of the change of the spine center point coordinate curve with or without outliers (abnormal points). If there are interference points, it is necessary to obtain the correct vertebral body center curve after removing the abnormal points by fusing the detection results of multiple image blocks. Since the spine includes multiple vertebral body segments and the center point coordinates of each vertebral body segment are different, therefore, taking the center point coordinates of each vertebral body as a reference, the vertebral body center curve of the entire spine is fitted as Figure 10 shown. Figure 10 For the vertebral body center curve with abnormal points in, after removing the abnormal points through high-frequency abnormal point detection and the fusion of the detection results of multiple image blocks Figure 10 , the correct vertebral body center curve shown in Figure 11 is obtained.

[0121] It is understandable that only after removing the outliers can the correct surgical planning parameters for the anteroposterior view be calculated. In the clinical study of the overall balance of the spine, whether it is sagittal or coronal balance, various angle and parameter measurements and analyses are required, including parameters of the spine, pelvis, lower limbs, overall parameters, and various compensatory parameters, etc. By using a pre-trained feature point extraction model for the part to be detected to detect the feature points of the part to be detected, automatic measurement of any parameter can be achieved, and the three-dimensional morphology of the spine can also be analyzed to realize personalized surgical plan planning.

[0122] It should be noted that the various surgical planning parameters for the anteroposterior view include, but are not limited to:

[0123] 1. Sagittal C7 to sacrum distance ratio (Barrey index): The ratio of the horizontal distance between the C7 plumb line and the posterior superior margin of the sacrum to the horizontal distance between the C7 plumb line and the midpoint of the femoral head center line;

[0124] 2. Sagittal C7-S1 sagittal plane axial distance (SVA): The distance between the posterior superior margin of S1 and the plumb line passing through the center of the C7 vertebral body;

[0125] 3. Sagittal sacral tilt angle (SS): The angle between the upper endplate line of S1 and the horizontal line;

[0126] 4. Sagittal pelvic incidence angle (PI): The angle between the line connecting the midpoint of the upper endplate of S1 and the centers of the bilateral femoral heads and the perpendicular line of the upper endplate of S1;

[0127] 5. Sagittal pelvic tilt angle (PT): The angle between the line connecting the midpoint of the upper endplate of S1 and the centers of the bilateral femoral heads and the plumb line;

[0128] 6. Sagittal sacropelvic angle (SPA): The angle between the line connecting the midpoint of the femoral head connection and the posterior upper corner of the sacrum and the perpendicular line of the upper endplate of the sacrum;

[0129] 7. Sagittal global tilt angle (GT): The angle between the line connecting the center of the C7 vertebral body and the midpoint of the femoral head and the line connecting the midpoint of the femoral head and the midpoint of the upper endplate of the sacrum;

[0130] 8. Sagittal C7 vertical tilt angle (C7VT): The angle between the line connecting the center of the C7 vertebral body and the midpoint of the femoral head and the plumb line passing through the center of the C7 vertebral body;

[0131] 9. Sagittal lumbar pelvic angle (LPA): The angle between the line connecting the center of the L1 vertebral body and the midpoint of the femoral head and the line connecting the midpoint of the femoral head and the midpoint of the upper endplate of the sacrum;

[0132] 10. Sagittal spinosacral angle (SSA): The angle between the line connecting the center of the C7 vertebral body and the midpoint of the upper endplate of S1 and the tangent line of the upper endplate of S1;

[0133] 11. Sagittal T1 pelvic angle: The angle between the line connecting the center of the T1 vertebra and the midpoint of the femoral head, and the line connecting the midpoint of the femoral head and the midpoint of the superior endplate of the sacrum;

[0134] 12. Sagittal T1 spinal pelvic inclination (T1SPI): The angle between the midpoint of the line connecting the centers of the two femoral heads and the center of the T1 vertebra and the plumb line;

[0135] 13. Sagittal lumbar lordosis angle (LL): The angle between the superior endplate of L1 and the superior endplate of S1;

[0136] 14. Sagittal lumbosacral angle (LSA): The angle formed by the superior endplate line of S1 and the inferior endplate line of L5;

[0137] 15. Sagittal thoracic kyphosis angle (TK): The angle between the superior endplate of T4 and the inferior endplate of T12;

[0138] 16. Sagittal L4-S1 lordosis: The angle between the superior endplate of L4 and the superior endplate of S1;

[0139] 17. Sagittal L4 incidence: The angle between the line connecting the midpoint of the superior endplate of L4 and the midpoint of the femoral head and the line passing through the midpoint of the superior endplate of L5 and perpendicular to the superior endplate of L4;

[0140] 18. Sagittal L5 incidence: The angle between the line connecting the midpoint of the superior endplate of L5 and the midpoint of the femoral head and the line passing through the midpoint of the superior endplate of L5 and perpendicular to the superior endplate of L5;

[0141] 19. Sagittal global thoracic kyphosis angle (GTK): The angle between the superior endplate of T1 and the inferior endplate of T12;

[0142] 20. Sagittal main thoracic kyphosis angle (MTK): The angle between the superior endplate of T5 and the inferior endplate of T12;

[0143] 21. Sagittal upper thoracic kyphosis angle (UTK): The angle between the superior endplate of T1 and the inferior endplate of T5;

[0144] 22. Coronal spinal balance parameter (CVA): The distance between the center sacral vertical line (CSVL) passing through the midpoint of the superior endplate of S1 and the C7 vertical line (C7VL);

[0145] 23. Coronal sacroacetabular angle (SAA): The angle between the superior endplate of S1 and the line connecting the two outer edges of the acetabulum;

[0146] 24. Coronal pelvic acetabular angle (PAA): The angle between the line connecting the midpoint of the superior endplate of the sacrum and the midpoint of the two outer edges of the acetabulum and the line connecting the two outer edges of the acetabulum;

[0147] 25. Coronal apical vertebral translation (AVT): The distance between the center point of the apical vertebra and the mid-vertical line of the sacrum.

[0148] 26. Coronal Cobb angle: The angle between the upper endplate of the selected vertebra and the lower endplate of another selected vertebra.

[0149] Wherein, C represents the cervical vertebra, T represents the thoracic vertebra, L represents the lumbar vertebra, and S represents the sacrum.

[0150] Cn represents the nth cervical vertebra counted from top to bottom, Tn represents the nth thoracic vertebra counted from top to bottom, Ln represents the nth lumbar vertebra counted from top to bottom, and n is a positive integer.

[0151] Embodiment III

[0152] Figure 12 It is a schematic block diagram of a training device 100 for a feature point extraction model of a part to be detected shown according to an exemplary embodiment. As Figure 12 shown, the device 100 includes:

[0153] An acquisition module 101, configured to acquire training data, where the training data is multiple pairs of anterior-posterior 2D images including the complete part to be detected, and the anterior-posterior 2D images cover X-ray images of the part to be detected in a normal body posture and X-ray images of the part to be detected in various pathological conditions.

[0154] A construction module 102, configured to construct a model trainer, where the model trainer is used to extract feature points from the training data and output feature points of the part to be detected according to the feature points.

[0155] A training module 103, configured to input the training data into the model trainer for training until the model converges, and determine the model at this time as the feature point extraction model of the part to be detected.

[0156] It can be understood that the technical solution provided in this embodiment, by acquiring anterior-posterior 2D images including the complete part to be detected and covering X-ray images of the part to be detected in a normal body posture and X-ray images of the part to be detected in various pathological conditions as training data, enables the trained feature point extraction model of the part to be detected to pay attention to other parts that affect the force balance of the diseased part, and takes into account the feature points in different body postures in the coronal plane (frontal view) or sagittal plane (lateral view), making the model have strong generalization ability when facing various lesion data and good recognition accuracy for the position and bending posture of the part to be detected (for example, the spine).

[0157] Embodiment IV

[0158] Figure 13 It is a schematic block diagram of an automatic surgical planning parameter generation device 200 shown according to an exemplary embodiment. As Figure 13As shown, the device 200 includes:

[0159] An acquisition module 201, configured to acquire the anteroposterior and lateral 2D images including the complete part to be detected;

[0160] An input module 202, configured to input the anteroposterior and lateral 2D images into the feature point extraction model of the part to be detected trained by the above method to obtain the feature points of the part to be detected;

[0161] A detection module 203, configured to perform outlier detection on the feature points based on the relatively smooth continuity of the spinal curve and remove the outliers;

[0162] A calculation module 204, configured to calculate various surgical planning parameters of the anteroposterior and lateral positions according to the feature points after removing the outliers.

[0163] It can be understood that the technical solution provided in this embodiment extracts the feature points of the part to be detected by acquiring the anteroposterior and lateral 2D images including the complete part to be detected, so that the extracted feature points can pay attention to other parts that affect the force balance of the diseased part. Based on this, the various surgical planning parameters of the anteroposterior and lateral positions calculated are more accurate, with strong generalization when facing various lesion data, and good recognition accuracy for the position and bending posture of the part to be detected (for example, the spine).

[0164] Embodiment Five

[0165] Refer to Figure 14 , an electronic device shown according to an exemplary embodiment includes:

[0166] A processor 701, a communication interface 702, a memory 703, and a communication bus 704, where the processor 701, the communication interface 702, and the memory 703 complete mutual communication through the communication bus 704;

[0167] The memory 703 is used to store a computer program;

[0168] The processor 701 is configured to implement the above method when executing the program stored on the memory.

[0169] It can be understood that the technical solution provided in this embodiment extracts the feature points of the part to be detected by acquiring the anteroposterior and lateral 2D images including the complete part to be detected, so that the extracted feature points can pay attention to other parts that affect the force balance of the diseased part. Based on this, the various surgical planning parameters of the anteroposterior and lateral positions calculated are more accurate, with strong generalization when facing various lesion data, and good recognition accuracy for the position and bending posture of the part to be detected (for example, the spine).

[0170] Embodiment Six

[0171] A non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause a computer to execute the above method.

[0172] It can be understood that for the technical solution provided in this embodiment, by obtaining the anterior-posterior and lateral 2D images including the complete part to be detected for feature point extraction of the part to be detected, the extracted feature points can pay attention to other parts that affect the force balance of the diseased part. Based on this, various surgical planning parameters of the anterior-posterior and lateral positions are calculated more accurately, with strong generalization ability in the face of various lesion data, and good recognition accuracy for the position and bending posture of the part to be detected (for example, the spine).

[0173] Of course, those of ordinary skill in the art can understand that all or part of the processes in implementing the methods of the above embodiments can be completed by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the above method embodiments. The storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0174] The specific embodiments of the present invention described above do not constitute a limitation on the protection scope of the present invention. Any other corresponding changes and deformations made according to the technical concept of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A training method for a feature point extraction model of a part to be detected, characterized in that, Including: Obtain training data, where the training data is multiple 2D anterior-posterior and lateral images containing the complete part to be detected. Construct a model trainer, which is used to extract feature points from the training data and output the feature points of the part to be detected according to the feature points. Input the training data into the model trainer for training until the model converges, and determine the model at this time as the feature point extraction model of the part to be detected.

2. The method according to claim 1, characterized in that, The 2D anterior-posterior and lateral images cover the X-ray images of the part to be detected in the normal body posture and the X-ray images of the part to be detected in various pathological conditions. If the part to be detected is the spine and pelvis, the obtaining of the training data includes: Obtain 2D anterior-posterior and lateral images containing the complete part to be detected, and the 2D anterior-posterior and lateral images at least include: the spine, the pelvis, and the femoral part containing the complete femoral head. Cut the 2D anterior-posterior and lateral images into multiple image blocks from bottom to top or from top to bottom. Each image block contains the complete information of the feature points to be detected. Among them, the image block of the pelvis part contains all the information of the pelvis feature points, and the image block of the spine contains the complete information of the spine feature points, and calculate the heat map of each image block. Determine the image blocks and the heat map of each image block as the training data, and input the image blocks and the heat map of each image block into the constructed model trainer in pairs.

3. The method according to claim 2, characterized in that, The constructing of the model trainer includes: Based on the neural network model HRNet, construct a first model trainer; the input of the first model trainer is the image blocks of the cervical vertebra and pelvis parts and the heat map of each image block, and the output is the feature points of the cervical vertebra and pelvis parts; the first model trainer is used to predict the feature points of the input multiple image blocks, generate the heat map of the corresponding image blocks, and compare the generated heat map with the input heat map until the loss function between the two meets the preset requirements. Based on the residual neural network model ResNet, construct a second model trainer; the input of the second model trainer is the image blocks of the spine and the heat map of each image block, and the output is the feature points of each vertebral body of the spine part; the second model trainer is used to determine the upper and lower intervals of the spine according to the feature points of the cervical vertebra and pelvis parts output by the first model trainer, and determine the feature points of each vertebral body on the spine within the upper and lower regions.

4. The method according to claim 3, characterized in that, The first model trainer includes: Construct n neural network layers of different scales, and the input image patches of each scale layer are W / 2 n 、H / 2 n in size, representing different scale information; each scale layer is composed of m - 2n convolutional blocks, representing the network depth information at each scale; where W is the width of the input image patch, H is the height of the input image patch, n is the number of network scales, m is the network depth at the first scale, m - 2n is the network depth at each scale, n≥0, m≥0 are integers; Each scale neural network layer performs feature fusion in the intermediate feature expression layer and the final feature expression layer, and finally outputs a heat map with a size of W×H×K, where K is the number of feature points, and K feature point positions are obtained by detecting the extreme points on the heat map.

5. The method according to claim 3, characterized in that, The second model trainer includes: An encoder and a neural network convolutional layer cascaded at the output end of the encoder. The encoder is a residual neural network model ResNet structure. The input of the encoder is the image blocks of the spine and the heat map of each image block. There is mask information of the four corner points of each vertebral body of the spine and mask information of the center point of each vertebral body on the heat map; the neural network convolutional layer includes: A first neural network convolutional layer for outputting the heat map of the mask center point of each vertebral body of the spine. The second neural network convolutional layer is used to output the position offset between the center point of each vertebral body and the center point of the vertebral body mask; The third neural network convolutional layer is used to output the position offset between the four corner points of each vertebral body and the center point of the vertebral body mask; The neural network convolutional layer outputs the position vector of the center point of each vertebral body of the spine and the position vectors of the four corner points of each vertebral body.

6. A method for automatically generating surgical planning parameters, characterized in that, It includes: Obtain the anteroposterior and lateral 2D images containing the complete part to be detected; Input the anteroposterior and lateral 2D images into the feature point extraction model of the part to be detected trained by the method according to any one of claims 1 to 5 to obtain the feature points of the part to be detected; Based on the relative smooth continuity of the spinal curve, perform outlier detection on the feature points and remove the outliers; According to the feature points after removing the outliers, calculate various surgical planning parameters of the anteroposterior and lateral positions.

7. The method according to claim 6, characterized in that, The performing outlier detection on the feature points based on the relative smooth continuity of the spinal curve and removing the outliers includes: From bottom to top or from top to bottom, for each image block on the spine, sequentially perform the following steps, including: According to the feature points on the first image block, predict the vertebral body center curve on the first image block; Judge whether there are outliers on the vertebral body center curve of the first image block. If so, according to the feature points on the second image block, predict the vertebral body center curve on the second image block; otherwise, output the vertebral body center curve on the first image block; Judge whether there are outliers on the vertebral body center curve of the second image block. If so, fit the vertebral body center curves on the first image block and the second image block to remove the outliers; otherwise, output the vertebral body center curve on the second image block.

8. A training device for a feature point extraction model of a part to be detected, characterized in that, It includes: An acquisition module for acquiring training data, where the training data is multiple anteroposterior and lateral 2D images containing the complete part to be detected, and the anteroposterior and lateral 2D images cover X-ray images of the part to be detected in a normal body posture and X-ray images of the part to be detected in various pathological conditions; A construction module for constructing a model trainer, where the model trainer is used to extract the feature points from the training data and output the feature points of the part to be detected according to the feature points; A training module for inputting the training data into the model trainer for training until the model converges, and determining the model at this time as the feature point extraction model of the part to be detected.

9. An automatic surgical planning parameter generation device, characterized in that, It includes: An acquisition module for acquiring the anteroposterior and lateral 2D images containing the complete part to be detected; An input module for inputting the anteroposterior and lateral 2D images into the feature point extraction model of the part to be detected trained by the method according to any one of claims 1 to 5 to obtain the feature points of the part to be detected; A detection module for performing outlier detection on the feature points based on the relative smooth continuity of the spinal curve and removing the outliers; A calculation module for calculating various surgical planning parameters of the anteroposterior and lateral positions according to the feature points after removing the outliers.

10. An electronic device, characterized in that, It includes: A processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored on the memory.

11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause a computer to perform the method according to any one of claims 1-7.

Images