A bone drilling force prediction method based on CT and drilling working conditions

Abstract

The application discloses a bone drilling force prediction method based on CT and drilling working conditions, which comprises the following steps: step 1, extracting the contact surface image sequence of the drill bit and the bone tissue on the bone drilling path from the CT image and performing pretreatment; step 2, inputting the pretreated contact surface image and the drilling working condition parameters into a multi-physical factor fusion module to obtain an image sequence of the fused working condition physical information; step 3, inputting the image of the fused working condition physical information into an image coding network to extract the drilling force features in the section; and step 4, inputting the sequence composed of the drilling force features of all sections on the drilling path into a sequence regression network ATP-UNeXt to obtain the prediction of the drill bit stress sequence on the drilling path. The application has the advantages that the drilling force related physical factors can be comprehensively considered, more accurate preoperative stress prediction can be provided, the model reasoning speed is fast, the application is suitable for a clinical operation auxiliary system, and the cost of the operation auxiliary system is reduced.

Description

Technical Field

[0001] This invention relates to the field of drilling force prediction technology in orthopedic surgery, and in particular to a method for predicting bone drilling force based on CT (Computed Tomography) and drilling conditions. Background Technology

[0002] In orthopedic surgery, precise control and perception of the interaction forces between surgical tools and bone tissue are crucial for achieving accurate surgery. Traditional surgical robot systems mainly rely on real-time force sensors to monitor force signals during surgery. However, this method has limitations, such as the inability to predict force changes in advance, resulting in slow robot responses to complex surgical tasks and an inability to analyze and predict based on preoperative images and real-time perception information like human surgeons.

[0003] Existing physical model-based methods for predicting drilling forces mainly include analytical methods, voxel methods, and finite element models. While these methods can simulate and predict drilling forces to some extent, they are computationally inefficient and cannot fully consider drilling conditions and various physical properties of bone tissue.

[0004] In recent years, the development of artificial intelligence technology has brought new solutions to the field of surgical robots. Neural network models have attracted attention due to their superior ability to approximate the actual drilling physics process at the data level and their high computational speed. However, existing deep learning-based methods for predicting bone drilling forces only model the relationship between working parameters and drilling forces, and cannot learn the influence of different bone tissue morphologies and material properties on drilling forces. In addition, existing methods still cannot uniformly extract features from drilling working information and bone tissue morphology and density information reflected in images within the neural network, resulting in compromised force prediction accuracy. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, this invention provides a method for predicting bone drilling force based on CT and drilling conditions. This method incorporates drilling conditions and bone tissue images into a deep learning-based bone drilling force prediction network, which more comprehensively considers the physical factors related to drilling force and provides more accurate preoperative stress prediction for bone drilling.

[0006] To solve the above problems, the present invention adopts the following technical solution:

[0007] This invention provides a method for predicting bone drilling force based on CT scans and drilling conditions. The specific steps are as follows:

[0008] Step 1: Extract and preprocess the image sequence of the contact surface between the drill bit and bone tissue along the bone drilling path from CT images;

[0009] Step 2: Input the preprocessed contact surface image and drilling condition parameters into the multi-physical factor fusion module to obtain an image sequence of fused physical information of the working condition;

[0010] Step 3: Input the image with fused working condition physical information into the image encoding network to extract the drilling force features within the cross section;

[0011] Step 4: Input the sequence of drilling force features of all cross surfaces on the drilling path into the sequence regression network ATP-UNeXt to obtain the prediction of the drill bit force sequence on the drilling path;

[0012] The training data for the multi-physics factor fusion module and the image coding network comes from three-dimensional images generated by the simulation program and the force calculated by the physical model based on the compression and cutting effects; the training data for the sequence regression network ATP-UNeXt comes from CT images and force sensor signals scanned in real bone drilling experiments.

[0013] Step 1 includes:

[0014] Step 1-1: Before the bone drilling surgery, perform a MicroCT scan on the bone tissue and set drilling parameters such as feed rate F, rotation speed S, and drill diameter D; during the training phase, the bone tissue needs to be scanned again after the surgery to register the preoperative and postoperative images.

[0015] Steps 1-2: Extract the contact surface image sequence V = {I} of the drill bit and bone tissue from the preoperative images. 1 ,I 2 ,…,I T}. Where T is the number of images extracted from the drilling path, and I t ∈R H*W Let H represent the t-th slice image in the image sequence V, where H and W are the height and width of the image, respectively.

[0016] Steps 1-3, for each cross-sectional image I in image sequence V. t The MinMax normalization process is performed separately, as shown in the following formula: Where x is the pixel grayscale value in the image before normalization, x max With x min , , represent the maximum and minimum pixel grayscale values ​​in the image, respectively, and x′ represents the pixel grayscale value in the normalized image.

[0017] Steps 1-4: Convert the operating parameters into the cross-sectional image I. t Multi-channel two-dimensional matrix of the same size Where C M This refers to the number of types of operating parameters.

[0018] Step 2 includes:

[0019] Step 2-1: Generate a spatial weighted graph based on drilling conditions using a multi-level gated linear unit (GMLP). This process is represented by the following formula:

[0020] M C =GMLP(M)

[0021] in, For a spatial weighted graph based on drilling conditions, C N This represents the number of channels in the weighted graph.

[0022] Step 2-2, Weighted graph M C With each image I in the contact surface image sequence t A pixel-by-pixel dot product operation is performed to generate an image sequence that fuses the physical information of the working conditions. This process is represented by the following formula:

[0023]

[0024] in, V is an image that incorporates physical information of the working condition after point multiplication. C For all of the drilling paths The image sequence is composed of multiple images. A spatial weight map is generated by GMLP, and the working condition parameters (such as feed rate and rotation speed) are fused with CT images pixel by pixel, which solves the limitation of existing methods that only independently model working conditions or image features.

[0025] The image coding network Enc in step 3 S Cross-sectional images that integrate physical information of working conditions Converted to in-plane drilling force characteristic u t This process is represented by the following formula:

[0026]

[0027] The image encoding network in step 3 uses multiple sets of ConvNeXt Blocks for feature extraction. Each ConvNeXt Block consists of a convolutional layer, layer normalization, and a GELU activation function. Upsampling is performed between each set of ConvNeXt Blocks using a PixelUnShuffle operation, reducing the image dimension from C1 to C2. in *H in *W in Transform into C out *H out *W out ,satisfy:

[0028] C out =C in×downsample_factor 2

[0029] H out =H in ÷downsample_factor

[0030] W out =W in ÷downsample_factor

[0031] Where downsample_factor is the downsampling factor. By pre-training with simulation data (based on the physical model) and fine-tuning with experimental data, we can both utilize prior physical knowledge to make the model more closely match actual physical processes and verify the robustness of the model in real-world scenarios, thereby improving the model's generalization ability.

[0032] The overall data processing procedure of the sequence regression network in step 4 is represented by the following formula:

[0033]

[0034] Where UNX is a sequence regression network, U = {u 1 ,u 2 ,…,u T} represents the drilling force characteristic u of all cross-sections along the drilling path. t The sequence formed, This is the network's predicted sequence of drill bit force sequence along the drilling path, where... For the force prediction of the drill bit at the t-th cutting plane position, C f The number of types of drilling forces to be predicted.

[0035] The sequence regression network in step 4 adopts the UNet architecture, which is suitable for processing multi-channel one-dimensional data. It includes a multi-layer encoder to extract features from the sequence at different scales. This process is expressed by the following formula:

[0036]

[0037] Among them, Encoder i Let N be the encoder of layer i, and N be the number of layers. This is the input to the i-th layer encoder. This is the output of the encoder at layer i.

[0038] Correspondingly, sequence regression networks also include multi-layer decoders that receive skip connections from encoders at the same level and the output of decoders at the previous level as outputs, progressively decoding deep semantic features into force sequence predictions of the same length as the original input sequence. This process is represented by the following formula:

[0039]

[0040] Among them, Decoder i For the i-th layer decoder, X is the output of the i-th layer decoder. bottle Res is the output of the bottleneck layer, and Res is the residual block on the skip connection path.

[0041] The sequence regression network in step 4 introduces a module called ATP-Block, which has a multi-frequency feature extraction and fusion mechanism. This module performs semantic fusion on shallow and deep feature sequences in the encoder under different receptive fields. This process is expressed by the following formula:

[0042]

[0043] Among them, W b With b b For the parameters of the fully connected layer, Conv Di Let be a convolutional block with a di-rate, and n be the number of different di-rate combinations. This is the output of the Nth layer encoder. It directly outputs the complete force sequence on the drilling path, avoiding the accumulated error of point-by-point prediction in traditional single-step prediction methods, and improving the overall prediction efficiency by more than 30%.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] This invention combines physical models and deep learning technology to more accurately predict bone drilling force from CT images and surgical plans, improving prediction accuracy compared to either a single physical model or a single deep learning model.

[0046] In the network structure designed in this invention, the multi-physical factor fusion module and the image encoder based on the cutting physical model can learn the physical features within the cross section that conform to the physical model, and the sequence regression network can supplement the temporal features between cross sections, thereby improving the robustness of the model.

[0047] The model inference speed of this invention is significantly improved compared to the calculation speed of the physical model, making it more suitable for clinical surgical assistance systems.

[0048] The predictions of this invention are based on preoperative images and information on the bone drilling procedure, and do not rely on real-time force sensor signals during the operation, thus reducing the cost of surgical assistance systems. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0050] Figure 1 This is a flowchart illustrating a method for predicting bone drilling force based on CT scans and drilling conditions, as shown in an embodiment of the present invention.

[0051] Figure 2 This is a structural diagram of the multi-physical factor fusion module in an embodiment of the present invention.

[0052] Figure 3 This is a structural diagram of the image encoder in an embodiment of the present invention.

[0053] Figure 4 This is a structural diagram of the sequence regression network in an embodiment of the present invention.

[0054] Figure 5 These are a set of comparative experimental results figures from examples of this invention. Detailed Implementation

[0055] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0056] Example: The bone drilling force prediction method based on CT and drilling conditions described in this embodiment of the invention refers to... Figure 1 The steps include:

[0057] Step 1: Extract and preprocess the image sequence of the contact surface between the drill bit and bone tissue along the bone drilling path from CT images;

[0058] Step 1-1: Before the bone drilling surgery, perform a MicroCT scan on the bone tissue and set drilling parameters such as feed rate F, rotation speed S, and drill diameter D; during the training phase, the bone tissue needs to be scanned again after the surgery to register the preoperative and postoperative images.

[0059] Steps 1-2: Extract the contact surface image sequence V = {I} of the drill bit and bone tissue from the preoperative images. 1 ,I 2 ,…,I T}. Where T is the number of images extracted from the drilling path, and I t ∈R H*W Let H represent the t-th slice image in the image sequence V, where H and W are the height and width of the image, respectively.

[0060] Steps 1-3, for each cross-sectional image I in image sequence V. t The MinMax normalization process is performed separately, as shown in the following formula: Where x is the pixel grayscale value in the image before normalization, x maxWith x min , , represent the maximum and minimum pixel grayscale values ​​in the image, respectively, and x′ represents the pixel grayscale value in the normalized image.

[0061] Steps 1-4: Convert the operating parameters into the cross-sectional image I. t Multi-channel two-dimensional matrix of the same size Where C M This refers to the number of types of operating parameters.

[0062] Step 2: Input the preprocessed contact surface image and drilling condition parameters into the multi-physical factor fusion module to obtain an image sequence of fused physical information of the working condition;

[0063] Step 2-1, refer to Figure 2 As shown, a spatial weight map based on drilling conditions is generated using multi-level gated linear element (GMLP) to model the influence weights of physical factors included in the drilling conditions on the cutting micro-element. This process is expressed by the following formula:

[0064] M C =GMLP(M)

[0065] in, For a spatial weighted graph based on drilling conditions, C N This represents the number of channels in the weighted graph.

[0066] Step 2-2, Weighted graph M C With each image I in the contact surface image sequence t A pixel-by-pixel dot product operation is performed to generate an image sequence that fuses the physical information of the working conditions. This process is represented by the following formula:

[0067]

[0068] in, V is an image that incorporates physical information of the working condition after point multiplication. C For all of the drilling paths The image sequence that is composed of images.

[0069] Step 3: Input the image with fused working condition physical information into an image encoding network to extract the drilling force features within the cross-section. (Refer to...) Figure 3 As shown, the image coding network Enc s Cross-sectional images that integrate physical information of working conditions Converted to in-plane drilling force characteristic u t This process is represented by the following formula:

[0070]

[0071] The image encoding network in step 3 uses multiple sets of ConvNeXt Blocks for feature extraction. Each ConvNeXt Block consists of a convolutional layer, layer normalization, and a GELU activation function. Upsampling is performed between each set of ConvNeXt Blocks using a PixelUnShuffle operation, reducing the image dimension from C1 to C2. in *H in *W in Transform into C out *H out *W out ,satisfy:

[0072] C out =C in ×downsample_factor 2

[0073] H out =H in ÷downsample_factor

[0074] W out =W in ÷downsample_factor

[0075] Where downsample_factor is the downsampling factor.

[0076] After processing through N sets of ConvNeXt Blocks, global max pooling is used to transform the feature map into a feature vector u. t As the full-path drilling force characteristic sequence U={u 1 ,u 2 ,…,u T One of the elements in}. During the pre-training phase, a fully connected (FC) layer is also needed to connect u. t The transformation involves calculating the drilling force prediction from the physical model, enabling the image encoder to fit the cutting physical model.

[0077] Step 4: Input the sequence of drilling force characteristics from all cross-sections along the drilling path into a sequence regression network to obtain a predicted drill bit force sequence along the drilling path. The overall data processing in this step is represented by the following formula:

[0078]

[0079] Where UNX is a sequence regression network, U = {u 1 ,u 2 ,…,u T} represents the drilling force characteristic u of all cross-sections along the drilling path. t The sequence formed, This is the network's predicted sequence of drill bit force sequence along the drilling path, where... For the force prediction of the drill bit at the t-th cutting plane position, C f The number of types of drilling forces to be predicted.

[0080] Reference Figure 4 As shown in (a), the sequence regression network in step 4 adopts the UNet architecture, which is suitable for processing multi-channel one-dimensional data. It includes a multi-layer encoder to extract features from the sequence at different scales. This process is expressed by the following formula:

[0081]

[0082] Among them, Encoder i Let N be the encoder of layer i, and N be the number of layers. This is the input to the i-th layer encoder. This is the output of the encoder at layer i.

[0083] Correspondingly, sequence regression networks also include multi-layer decoders that receive skip connections from encoders at the same level and the output of decoders at the previous level as outputs, progressively decoding deep semantic features into force sequence predictions of the same length as the original input sequence. This process is represented by the following formula:

[0084]

[0085] Among them, Decoder i For the i-th layer decoder, X is the output of the i-th layer decoder. bottle Res is the output of the bottleneck layer, and Res is the residual block on the skip connection path.

[0086] Reference Figure 4 As shown in (b), the sequence regression network in step 4 introduces a deep separable convolution module, DWCBlock, which reduces the computational cost and parameter count while maintaining a large receptive field. Its operation process is represented by the following formula:

[0087] X D_out =FC 2 (GELU(FC 1 (LN((DC(X D_in ))))))+X D_in

[0088] Among them, X D_in For the input data of the DWC Block, X D_outThis is the output data of the DWC Block, where DC is the depthwise convolution kernel, LN is the layer normalization operation, and FC is the layer normalization operation. 1 FC 2 These are fully connected layers that represent channel expansion and contraction, respectively.

[0089] Reference Figure 4 As shown in (c), the sequence regression network in step 4 introduces a module ATP-Block with a multi-frequency feature extraction and fusion mechanism to perform semantic fusion of shallow and deep feature sequences in the encoder under different receptive fields. This process is expressed by the following formula:

[0090]

[0091] Among them, W b With b b For the parameters of the fully connected layer, Conv Di Let be a convolutional block with a di-rate, and n be the number of different di-rate combinations. This is the output of the Nth layer encoder.

[0092] The training data for the multi-physics factor fusion module and the image coding network comes from three-dimensional images generated by the simulation program and the force calculated by the physical model based on the compression and cutting effects; the training data for the sequence regression network ATP-UNeXt comes from CT images and force sensor signals scanned in real bone drilling experiments.

[0093] In the training and testing of the sequence regression network, a dataset containing 14 drilling paths from 3 cow bones was constructed. Ten paths from 2 bones were used as the training and validation set for 10-fold cross-validation, and the remaining 4 paths from the bones were used as the test set. During training, SmoothL1 was used as the loss function, calculated as follows:

[0094]

[0095] Where y is the truth value. These are the model's predicted values.

[0096] Choose to use MAE, MAPE, R 2 The formula for calculating the model's performance is as follows:

[0097]

[0098] Where N is the number of samples in the batch for calculating the loss function.

[0099]

[0100] in, Calculate the average of the true values ​​in the batch for the loss function.

[0101] The parameter settings during model training are shown in Table 1.

[0102] Table 1 Experimental hyperparameter settings

[0103]

[0104]

[0105] In this embodiment, the method proposed in this invention is compared with existing methods through experiments. Table 2 shows the MAE and MAPE indices of the sequence regression network proposed in this invention, existing cutting physics models, and other deep learning models on the test set. The results show that ATP-UNeXt not only significantly exceeds the prediction accuracy of the physics model, but also has the lowest MAE and MAPE error indices among all the deep learning models tested.

[0106] Table 2 Model Performance Comparison

[0107]

[0108] Regarding the system in the above embodiments, the specific steps for each module to perform operations have been described in detail in the embodiments related to the method, and will not be elaborated further here. Each module of the above-described deep learning-based multi-stage detection method for CT spinal bone lesions can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0109] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0110] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of the inventive concept are included in this invention and are protected by the appended claims.

Claims

1. A method for predicting the bone drilling force based on CT and drilling conditions, characterized in that: The method comprises the following steps: Step 1: Extract the contact surface image sequence of the drill bit and the bone tissue on the bone drilling path from the CT image and pre-process; Step 2: Input the pre-processed contact surface image and drilling condition parameters into a multi-physical factor fusion module to obtain an image sequence of fused condition physical information; Step 3: Input the image of fused condition physical information into an image coding network to extract the drilling force features in the section; Step 4: Input the sequence composed of the drilling force features of all sections on the drilling path into a sequence regression network ATP(Atrous Temporal Pyramid)-UNeXt to obtain the prediction of the drill bit force sequence on the drilling path. The training data of the multi-physical factor fusion module and the image coding network come from the three-dimensional images generated by the simulation program and the force calculated based on the extrusion and cutting effect; the training data of the sequence regression network ATP-UNeXt come from the CT images scanned in the real bone drilling experiment and the force sensor signals.

2. The bone drilling force prediction method based on CT and drilling conditions according to claim 1, wherein: Step 1 comprises: Step 1-1, before the bone drilling surgery, perform MicroCT scanning on the bone tissue and set the drilling condition parameters of feed rate F, rotation speed S and drill bit diameter D; in the training stage, also need to scan the bone tissue again after the surgery, and register the preoperative image with the postoperative image, Steps 1-2, extracting the sequence of contact surface images of the drill bit and bone tissue from the preoperative images where T is the number of images extracted from the drilling path, denotes the sequence of images the t-th cross-sectional image, H, W are the height and width of the image, respectively, Steps 1-3: For each slice image in image sequence V The MinMax normalization process is performed separately, as shown in the following formula: ,in These are the pixel grayscale values ​​in the image before normalization. and These represent the maximum and minimum grayscale values ​​of the pixels in the image, respectively. The pixel grayscale values ​​in the normalized image. Step 1-4, converting the operating parameter into a cross-sectional image Multichannel two-dimensional matrix of the same size wherein is the number of classes of operating parameters.

3. The method according to claim 1, wherein: Step 2, the multi-physics factor fusion module, includes a Gated-Multilayer Perceptron (GMLP) to generate a spatial weight map based on drilling conditions. , This refers to the number of channels in the weighted graph; weighted graph With each image in the contact surface image sequence Perform pixel-by-pixel dot product operations to generate an image sequence that fuses the physical information of the working conditions. The process is represented by the following formula: wherein, is the image sequence composed of the images of the physical information of the post-fusion working condition, is the image sequence composed of the images of the physical information of the post-fusion working condition, is the image sequence composed of the images of the physical information of the post-fusion working condition.

4. The method according to claim 3, wherein: In step 3, the image encoder comprises a convolutional neural network with a ConvNeXt architecture, uses a PixelUnShuffle pixel reorganization operation instead of a convolutional pooling operation for downsampling, and encodes the image of fused condition physical information into drilling force features in the section; the network uses a phased training strategy: first pre-train using simulation data, and then fine-tune using experimental measurement data.

5. The bone drilling force prediction method based on CT and drilling conditions according to claim 4, wherein: Image encoding network in step 3 Cutting plane image fusing with physical information of working condition Convert to drilling force characteristics in cutting plane The process is represented by the following equation: The image encoding network in step 3 uses multiple groups of ConvNeXt Blocks for feature extraction, each ConvNeXt Block consisting of a convolutional layer, layer normalization, and a GELU activation function. Between each group of ConvNeXt Blocks, a PixelUnShuffle operation is used for upsampling, increasing the image dimensions from to , satisfying: wherein is the down-sampling rate.

6. The method according to claim 5, wherein: In step 4, the sequence regression network ATP-UNeXt takes the image sequence of fused condition physical information as input, models the time sequence features between sections; the network uses a multi-frequency feature extraction module in the stem layer and the bottleneck layer, and uses a depth separable convolution kernel in the feature extraction module.

7. The method according to claim 6, wherein, The overall data processing process of the sequence regression network in step 4 is represented by the following formula: ； wherein, is a sequence regression network, is a drilling force feature of all the cutting surfaces on the drilling path is a sequence, is a predicted sequence of the force sequence on the drilling path by the network, wherein is a force prediction when the drill bit is at the tth cutting surface position, is the number of drilling forces to be predicted, The sequence regression network in step 4 adopts a UNet architecture suitable for processing multi-channel one-dimensional data, which contains multiple encoders that extract features from sequences at different scales, which is represented by the following formula: wherein, N is the number of layers, is the input to the i-th layer encoder, is the output of the i-th layer encoder; The sequence regression network also contains multiple decoders that receive the skip connection from the same level encoder and the output of the previous level decoder as the output, and gradually decodes the deep semantic features into the force sequence prediction with the same length as the original input sequence, which is represented by the following formula: wherein, is the i-th layer decoder, is the output of the i-th layer decoder, is the output of the bottleneck layer, is the residual block on the skip connection path.

8. The method according to claim 7, wherein the method is characterized by: In step 4, the sequence regression network introduces a module ATP-Block with multi-frequency feature extraction and fusion mechanism, which performs semantic fusion on the shallow and deep feature sequences in the encoder at different receptive fields, which is represented by the following formula: wherein, is the output of the ATP-Block module, are the parameters of the fully connected layer, is the convolutional block with the dilation rate of n is the number of different dilation rates, is the output of the Nth layer encoder.​​

Images