2D / 3D detection segmentation method, device and terminal equipment for orthopedic surgery registration

Abstract

The present invention discloses a 2D / 3D detection and segmentation method, device and terminal equipment for orthopedic surgery registration, the method steps include: step S01. obtaining preoperative CT data and intraoperative perspective X-ray images; step S02. inputting the preoperative CT data into a 3D detection network for three-dimensional landmark detection and bone tissue segmentation, wherein the 3D image coding features of the input CT data are first detected, edge information is extracted, and decoding is performed to obtain preliminary detection and segmentation results, and the confidence map is calculated to obtain the intermediate segmentation results, which are input into a 3D prompt encoder to generate prompt information; step S03. inputting the intraoperative perspective X-ray image into a TransUNet network for two-dimensional landmark detection and bone tissue segmentation; step S04. outputting the landmark detection and segmentation results for registration. The present invention has the advantages of simple implementation method, strong real-time and interactivity, high detection and segmentation accuracy and efficiency.

Description

Technical Field

[0001] The present invention relates to the field of medical device technology, and in particular to a 2D (two-dimensional) / 3D (three-dimensional) detection and segmentation method, device and terminal equipment for orthopedic surgery registration. Background Art

[0002] 2D / 3D image registration plays a key role in orthopedic surgical navigation. It is achieved by registering preoperative 3D volume data with intraoperative 2D image data so that a real-time three-dimensional spatial position relationship of surgical instruments relative to the patient's lesion can be obtained, thereby achieving accurate and safe surgical operations. In the 2D / 3D image registration process, it is first necessary to perform three-dimensional landmark detection and three-dimensional bone tissue segmentation on the preoperative CT data to be registered, and to perform two-dimensional landmark detection and two-dimensional bone tissue segmentation on the intraoperative two-dimensional perspective X-ray image. With regard to 2D / 3D landmark detection and bone tissue segmentation in the image registration process, the prior art usually performs them as two independent tasks, that is, bone tissue segmentation and landmark detection and recognition are performed separately as two tasks, which is not only complex to implement, but also affects the detection and segmentation efficiency, resulting in low real-time registration.

[0003] At the same time, in order to achieve 2D / 3D landmark detection and bone tissue segmentation in different scenarios, the existing technology usually uses deep learning to train each scenario separately, lacks continuous learning of the entire surgical process, and thus ignores the fine-grained interaction details during the surgery, and can only achieve detection and segmentation at a coarse-grained level. And it usually needs to rely on strong supervision and expensive manual labeling, which is costly to implement. When new incremental detection classes or new training data sets need to be added, the model needs to be retrained, so the actual detection and segmentation efficiency is low, and 2D / 3D landmark detection and bone tissue segmentation cannot be completed in real time. Summary of the invention

[0004] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides a 2D / 3D detection and segmentation method, device and terminal equipment for orthopedic surgery registration, which has a simple implementation method, low cost, strong real-time and interactivity, high detection and segmentation accuracy and efficiency.

[0005] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0006] A 2D / 3D detection and segmentation method for orthopedic surgery registration, comprising the following steps:

[0007] Step S01. Obtaining CT (Computed Tomography) data of the bone tissue to be registered obtained before the operation and the fluoroscopic X-ray image obtained during the operation;

[0008] Step S02. Input the preoperative CT data into the 3D detection network for 3D landmark detection and 3D bone tissue segmentation, wherein the 3D image encoding features of the input CT data are first detected by the 3D image encoder in the 3D detection network, and the edge information is extracted by the edge perception module, and then the 3D image encoding features are decoded by the 3D mask decoder in the 3D detection network to obtain preliminary 3D landmark detection and bone tissue segmentation results, and the abnormality perception and confidence learning module calculates the confidence map according to the preliminary bone tissue segmentation result to obtain an intermediate segmentation result, and the intermediate segmentation result and the preliminary 3D landmark detection result are input into the 3D prompt encoder of the 3D detection network to generate prompt information, and the generated prompt information is fed back to the 3D mask decoder and the edge perception module, and after multiple iterations, the final 3D landmark detection result and segmentation result output are obtained;

[0009] Step S03. Input the intraoperative fluoroscopic X-ray image into the constructed TransUNet network for two-dimensional landmark point detection and two-dimensional bone tissue segmentation, wherein the TransUNet network includes an encoder and a decoder, and a segmentation path for two-dimensional bone tissue segmentation and a landmark point heat map network path for two-dimensional landmark point detection are arranged between the encoder and the decoder, and the TransUNet network is pre-trained using a loss obtained by synthesizing a segmentation loss and a landmark point detection loss function, wherein the segmentation loss is a loss function corresponding to the segmentation path, and the landmark point detection loss is a loss function corresponding to the landmark point heat map network path;

[0010] Step S04: Output the three-dimensional landmark point detection and three-dimensional bone tissue segmentation results of the preoperative CT data and the two-dimensional landmark point detection and two-dimensional bone tissue segmentation results of the intraoperative fluoroscopic X-ray image for registration.

[0011] Furthermore, in step S02, the energy function used in the three-dimensional landmark detection is E Landmark =α1·E Region +β1·E Edge +γ1∑w 2 , the energy function used for three-dimensional bone tissue segmentation is:

[0012] E Segmentation =α2·E Region +β2·E Edge +γ2·E Outlier +λ1∑w 2

[0013] Among them, E Region represents the output result of the 3D detection network, E Edge represents the output result of the edge perception module, EOutlier represents the output result of the anomaly perception and confidence learning module, ∑w 2 is the regularization term, and α1, β1, γ1 and α2, β2, γ2, λ1 are the preset coefficients respectively.

[0014] Furthermore, the 3D detection network is sequentially provided with a 3D image encoder, a 3D mask decoder and a 3D hint encoder to form a continuous learning framework, and the continuous learning framework obtains the prediction result according to the following formula

[0015]

[0016] Where N is the number of datasets under the same scene, M is the total number of voxel positions, and T is the total number of sequence datasets containing the same / different scenes. Denotes the dataset D t The predicted map of voxel position j in , j represents the voxel position, X t and Y t Represents the data set D t The input image and the corresponding bone tissue label in f e 、f d Respectively represent the CNN / Transformer functions in the encoder and decoder, c t Indicates the output category, W e , W d are the parameters of the encoding and decoding paths, H t (j) is the data set D t The confidence map of voxel position j in , is the tth data set;

[0017] A dual-position Transformer module is also provided at the end of the 3D image encoder and the 3D hint encoder to extract the learning position code and the pixel space position code respectively. The edge perception module extracts the input image data X 3D The learned position encoding feature extracted from is sent to the multi-head attention module, and the pixel spatial position extracted from the feature of the region of interest in the input image data is used as the key position encoding feature X spacial Add to the target feature of interest, and finally input image data X 3D , the output of the multi-head attention module X attention And the key position encoding feature X spacial The output result E of the edge perception module is obtained through the multi-layer perceptron MLP Edge , that is, E Edge =MLP(ReLU(X 3D +Xattention +w*X spacial )), where w is the edge weight and ReLU represents the activation function.

[0018] Furthermore, the abnormality perception and confidence learning module obtains the CT bone tissue part map according to the relationship between the slices of the CT preliminary segmentation result output by the 3D mask decoder, overlaps the bounding boxes of all the real bone tissue labels on the bone tissue part map, and calculates the volume overlap ratio between the bounding box and the bone tissue part map, generates the bone tissue part distribution map and takes the average to obtain the average bone tissue distribution map P t , according to the formula Calculate the tth data set D t The weighted graph of voxel position j in P t (j) represents the data set D t The average bone tissue distribution map at voxel position j in , Represents the prediction of different bone tissue abnormalities at voxel position j, where J represents a matrix with elements of 1; the data set D is calculated by the entropy function t Confidence map of voxel position j in To merge overlapping organ predictions from different decoders to get the final prediction result in, Denotes the dataset D t The predicted map of voxel position j in , Represents element-wise multiplication.

[0019] Further, in step S03, the segmentation path and the landmark point heat map network path are set at the decoder end of the TransUNet network, and the segmentation path is calculated by the following formula: seg (w):

[0020]

[0021]

[0022] Among them, L dice (i,c,w) is the Dice loss, L cross-entropy (i, k, w) is the cross entropy loss, w is the network weight, i is the model training batch_size value, C is the number of segmentation categories, k is the number of specific segmentation categories for guiding landmark detection, α is the weight coefficient, y true is the true segmentation category label, y predicted is the predicted segmentation category label.

[0023] Further, in step S03, the landmark point detection loss function L of the landmark point heat map network path in the TransUNet network is landmark (w) Using the landmark heat map loss Lheat (w) and spatial position constraint L position (w) is calculated as:

[0024] L landmark (w) = L heat (w)+βL position (w)

[0025] The landmark heat map loss L heat (w) Use the true heat map h of each landmark point true (i) and corresponding prediction heat map The average normalized cross-correlation NCC coefficient between is calculated as the evaluation index, and the calculation expression is:

[0026]

[0027] Among them, (x i ,y i ) is the initial position of the i-th landmark, σ is the standard deviation, N is the number of heatmaps / landmarks, represents a symbolic function with a value of 0 or 1, and β is the weight coefficient.

[0028] Furthermore, the spatial position constraint L position The acquisition step of (w) includes: limiting the positioning range of the predicted landmark points according to the spatial position of the bone tissue, and jointly evaluating the landmark point positions according to the image segmentation results and the landmark point heat map results, wherein the candidate position of the femoral head center is first calculated by the landmark point heat map, and the candidate position is limited to the pixels marked as corresponding to the femur in the bone tissue segmentation result, and then the remaining landmark points are marked as corresponding to the bone tissue part;

[0029] When the orthopedic surgery is periacetabular osteotomy, the spatial position constraint L position The steps of obtaining (w) include:

[0030] The left and right candidate positions L_FH and R_FH of the center of the femoral head are calculated by the landmark point heat map, and the left and right candidate positions L_FH and R_FH are limited to the pixels marked as corresponding to the femur, that is, the segmentation results LF_seg(w) or RF_seg(w), and the position of the center of the femoral head is determined according to the multi-pose image, and all the remaining landmark points are marked as corresponding to the hemipelvis and osteotomy block parts, that is, the positions of LHP_seg(w), RHP_seg(w) and PAO_seg(w).

[0031]

[0032] Among them, I is the multi-pose X-ray image taken by the intraoperative C-arm dynamic fluoroscopy, n is the number of intraoperative multi-pose X-ray images, fit represents the multi-pose fitting operation, and Mean represents the mean of the (n-1) femoral head candidate center coordinates.

[0033] Furthermore, in step S03, when training the TransUNet network, an L in the training process is sought according to the following formula: overall (w) is minimized to make the weights of the segmentation loss function and the landmark detection loss function equal:

[0034] L overall (w) = λL seg (w)+(1-λ)L landmark (w)

[0035] Among them, L seg (w) is the segmentation loss function, L landmark (w) is the landmark detection loss function.

[0036] A 2D / 3D detection and segmentation device for orthopedic surgery registration, comprising:

[0037] A data acquisition module is used to acquire the CT data of the bone tissue to be registered obtained before the operation and the fluoroscopic X-ray image obtained during the operation;

[0038] A 3D detection and segmentation module, used for inputting the preoperative CT data into a 3D detection network for 3D landmark detection and 3D bone tissue segmentation, wherein the 3D image encoding features of the input CT data are first detected by a 3D image encoder in the 3D detection network, and edge information is extracted by an edge perception module, and then the 3D image encoding features are decoded by a 3D mask decoder in the 3D detection network to obtain preliminary 3D landmark detection and bone tissue segmentation results, and the abnormality perception and confidence learning module calculates a confidence map based on the preliminary bone tissue segmentation result to obtain an intermediate segmentation result, and the intermediate segmentation result and the preliminary 3D landmark detection result are input into a 3D prompt encoder of the 3D detection network to generate prompt information, and the prompt information generated by the 3D prompt encoder is fed back to the 3D mask decoder and the edge perception module, and after multiple iterations, the final 3D landmark detection result and segmentation result output are obtained;

[0039] A 2D detection and segmentation module, for inputting the intraoperative fluoroscopic X-ray image into a constructed TransUNet network for two-dimensional landmark detection and two-dimensional bone tissue segmentation, wherein the TransUNet network includes an encoder and a decoder, and a segmentation path for two-dimensional bone tissue segmentation and a landmark point heat map network path for two-dimensional landmark detection are arranged between the encoder and the decoder, wherein the TransUNet network is pre-trained using a loss obtained by synthesizing a segmentation loss and a landmark point detection loss function, wherein the segmentation loss is a loss function corresponding to the segmentation path, and the landmark point detection loss is a loss function corresponding to the landmark point heat map network path;

[0040] The detection output module is used to output the three-dimensional landmark point detection and three-dimensional bone tissue segmentation results of the preoperative CT data and the two-dimensional landmark point detection and two-dimensional bone tissue segmentation results of the intraoperative perspective X-ray image for alignment.

[0041] A terminal device comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0042] Compared with the prior art, the advantages of the present invention are:

[0043] 1. The present invention constructs a continuous learning framework by utilizing a 3D detection network, an edge perception module, and an anomaly perception and confidence learning module, and uses a continuous learning method to simultaneously perform three-dimensional landmark detection and three-dimensional bone tissue segmentation. This can not only improve the real-time and efficiency of three-dimensional landmark detection and bone tissue segmentation, but also continuously learn the entire surgical process, retain the fine-grained interaction details during the surgery, achieve fine-grained level detection and segmentation, and solve the problem of landmark and bone tissue occlusion during orthopedic surgery registration.

[0044] 2. The present invention constructs the TransUNet network to detect and segment intraoperative fluoroscopic X-ray images. In the multi-task deep learning process, the combination of segmentation loss and landmark detection loss is used as the final loss function to train the model. It can simultaneously realize two-dimensional landmark detection and two-dimensional bone tissue segmentation, improve the execution efficiency and accuracy of two-dimensional landmark detection and segmentation, and ensure the real-time performance of two-dimensional landmark detection and segmentation during orthopedic surgery registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the implementation flow of the 2D / 3D detection and segmentation method for orthopedic surgery registration in this embodiment.

[0046] Figure 2It is a schematic diagram of the principle of the continuous learning framework for realizing three-dimensional landmark point detection and three-dimensional bone tissue segmentation construction in this embodiment.

[0047] Figure 3 It is a schematic diagram of the principle of realizing two-dimensional landmark point detection and two-dimensional bone tissue segmentation based on the TransUNet network in this embodiment. DETAILED DESCRIPTION

[0048] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0049] like Figure 1 As shown, the steps of the 2D / 3D detection and segmentation method for orthopedic surgery registration in this embodiment include:

[0050] Step S01: Obtain CT data of the bone tissue to be registered obtained before the operation and a perspective X-ray image obtained during the operation.

[0051] CT is an imaging technology that uses X-rays to image the internal structure of the body. By performing X-ray scans on the human body at multiple angles, and then using a computer to process and reconstruct these data, a cross-sectional image of the body's interior is generated. Fluoroscopic X-ray imaging is an imaging technology that uses X-rays to penetrate the human body and form images on a fluorescent screen or digital detector. In this embodiment, the preoperative CT data to be registered and the intraoperative C-arm fluoroscopic X-ray image are obtained and preprocessed respectively. The preprocessing steps include adjusting the window width and window position to select the area of ​​interest, rotating part of the CT data to ensure that the CT data and the X-ray image are in the same direction, and cropping a specified number of pixels from the boundary of each X-ray image to remove artifact effects. The specific preprocessing method can be configured according to actual needs.

[0052] Step S02. The preoperative CT data is input into the 3D detection network for 3D landmark detection and 3D bone tissue segmentation, wherein the 3D image encoding features of the input CT data are first detected by the 3D image encoder in the 3D detection network, and the edge information is extracted by the edge perception module. Then, the 3D image encoding features are decoded by the 3D mask decoder in the 3D detection network to obtain preliminary 3D landmark detection and bone tissue segmentation results. The abnormality perception and confidence learning module calculates the confidence map according to the preliminary bone tissue segmentation result to obtain an intermediate segmentation result. The intermediate segmentation result and the preliminary 3D landmark detection result are input into the 3D prompt encoder of the 3D detection network to generate prompt information. The prompt information generated by the 3D prompt encoder is fed back to the 3D mask decoder and the edge perception module. After multiple iterations, the final 3D landmark detection result and segmentation result output are obtained.

[0053] In this embodiment, a continuous learning framework is constructed using a 3D detection network, an edge perception module, and an anomaly perception and confidence learning module. A continuous learning method is used to perform 3D landmark detection and 3D bone tissue segmentation, so that the entire surgical process can be continuously learned, and the fine-grained interaction details during the surgical process can be retained. Fine-grained level detection and segmentation can be achieved, and catastrophic forgetting problems in continuous learning can be avoided. At the same time, the edge perception module can also solve the landmark and bone tissue occlusion problems. Specifically, the constructed continuous learning framework is expressed as This framework can avoid the mutual occlusion of landmarks and bone tissues. t and Y t Denote the input image and dataset D respectively t The corresponding bone tissue label in , j represents the predicted map of voxel position, is the tth data set, c t Indicates the output category, represents the union of all previous predictions, f e 、f d is the parameter of CNN / Transformer function, W e , W d They are the parameters of the encoding and decoding paths respectively.

[0054] Specifically, assuming that the dataset is a set of CT sequence data sets that are continuously added incrementally. When the tth sub-dataset When training a continuous learning model, the first t-1 data set Unable to access the model training, the constructed continuous learning framework is expressed as This framework can avoid catastrophic forgetting in continuous learning.

[0055] N is the number of datasets with the same scene, T is the total number of sequential datasets containing the same / different scenes, C1 is the number of bone tissue categories with the same scene, C2 is the number of bone tissue categories with different scenes, C is the number of overall segmentation categories for periacetabular osteotomy, V(i; seg) is the i-th bone tissue segmentation in the 3D CT volume, geo(R x ,R y ,R z ,t x ,t y ,t z ) represents the six-degree-of-freedom operation including rotation and translation on the XYZ axis of the CT coordinate system. Denotes the dataset D t The predicted map of voxel position j in , j represents the voxel position, X t and Y t Represents the data set Dt The input image and the corresponding bone tissue label in c t represents the output category, f e 、f d Respectively represent the encoder ( Figure 2 3D image encoder and 3D hint encoder in ), decoder ( Figure 2 CNN / Transformer function in 3D mask decoder in , W e , W d are parameters of the encoding and decoding paths respectively. e Encoder( Figure 2 The 3D image encoder and 3D hint encoder in are both frozen encoders, which are used as the general encoder for the sequence dataset D. d Decoder ( Figure 2 3D mask decoder in Figure 2 The anomaly perception and confidence learning modules in are dynamically incrementally expanded. t They are trained separately, and their number is equal to or proportional to the total number T of sequence datasets D. The final prediction result of the continuous learning framework is the union of all previous predictions Where M is the number of voxels, ∪ is the union, and H t (j) is the data set D t The confidence map of voxel position j in , and M represents the total number of voxel positions.

[0056] In this embodiment, a continuous learning framework is constructed to realize 3D landmark detection and 3D bone tissue segmentation. Figure 2 As shown in the figure, the continuous learning framework Contains 3D detection network (E Region ), edge perception module (E Edge ) and the anomaly perception and confidence learning module (E Outlier ), where the 3D detection network is sequentially provided with a 3D image encoder, a 3D mask decoder and a 3D hint encoder, an edge perception module (E Edge) is arranged between the 3D image encoder and the 3D mask decoder. By setting a dual-position Transformer module at the end of the 3D image encoder and the 3D hint encoder to extract the learning position code and the pixel space position code respectively, the spatial positioning ability and the edge information extraction ability can be improved. The 3D image encoder includes a connected 3D embedding unit, a 3D absolute position coding unit and a 3D attention module. After the 3D image encoder inputs the preoperative CT data, it sequentially undergoes 3D embedding and 3D absolute position coding to obtain a 3D absolute position code, and then extracts the 3D image coding features after passing through the 3D attention module. The 3D attention module includes a layer normalization layer, a 3D multi-head self-attention mechanism layer, a layer normalization layer and a feedforward network layer, wherein the 3D multi-head self-attention mechanism calculates the Q (query), K (key) and V (value) of multiple heads in parallel to capture the features of the input data from different angles and representation subspaces to obtain the 3S true position code. After the activation value of each layer is normalized by the layer normalization layer, the output of the attention mechanism is further processed and refined by the feedforward network.

[0057] like Figure 2 As shown, in the edge perception module of this embodiment, the position embedding module is first learned by the dual-position Transformer module from the input image data X 3D Extract the learning position encoding features and send them to the multi-head attention module (X attention ), and then the edge weight guidance module uses the edge weight guidance (w) to extract edge information from shallow-grained low-level features, and inputs the features of the region of interest into the dual-position Transformer module. The pixel spatial position embedding module learns the pixel spatial position of the target region and uses it as the key position encoding X spacial , the key position encoding feature X spacial Add to the target feature of interest, and finally normalize and convert the input image data X 3D , the output of the multi-head attention module X attention And the key position encoding feature X spacial The output result E of the edge perception module is obtained through the multi-layer perceptron MLP Edge , that is, E Edge =MLP(ReLU(X 3D +X attention +w*X spacial )), where w is the edge weight and ReLU represents the activation function. The 3D mask decoder includes an image embedding unit, two Transformer modules, two Transformer convolutional layers, and a multi-layer perceptron MLP connected in sequence. The 3D mask decoder receives the output result E of the edge perception module. Edge, it is processed by image embedding, Transformer module, Transposed 3D convolution and multi-layer perceptron in sequence, and outputs the preliminary predicted segmentation results and landmark detection results, which are then output to the anomaly perception and confidence learning module.

[0058] In this embodiment, the anomaly perception and confidence learning module calculates the average bone tissue distribution map P for each dataset t by overlapping the average bounding box with the bone tissue label t , the confidence map is calculated by the entropy function To merge overlapping organ predictions from different decoders, M t Represents a weighted graph, represented by Sure, represents element-wise multiplication, represents different bone tissue predictions, and J represents a matrix with elements equal to 1. Figure 2 As shown, the anomaly perception and confidence learning module includes multiple 3D convolutional layers, projection 3D convolutional layers, multiple decoding modules and multiple projection modules, and outputs intermediate segmentation results.

[0059] In this embodiment, the abnormality perception and confidence learning module learns the relationship between the slices of the CT preliminary segmentation result output by the 3D mask decoder according to an unsupervised model to obtain the CT bone tissue part map, and then overlaps the bounding boxes of all the real bone tissue labels on the bone tissue part map, and calculates the volume overlap ratio between the bounding box and the bone tissue part map, generates the bone tissue part distribution map of all patients and further calculates the mean to obtain the average bone tissue distribution map P t .pass Calculate the data set D t The weighted graph of voxel position j in P t (j) represents the data set D t The average bone tissue distribution map at voxel position j in , Represents different bone tissue abnormality predictions at voxel position j (including but not limited to bone tissue damage, bone tumors, etc.), where J represents a matrix with elements of 1. The data set D is calculated by the entropy function t Confidence map of voxel position j in The final prediction result is obtained by merging overlapping organ predictions from different decoders (whose number is equal to or proportional to the total number T of sequence data sets D) in, Denotes the dataset D t The predicted map of voxel position j in , Represents element-wise multiplication.

[0060] In a specific application embodiment, The abnormal weighted map can be obtained by adding an abnormal segmentation head (including but not limited to pre-trained networks for abnormal situations such as bone tissue damage and bone tumors) to the abnormal perception and confidence learning module. The unsupervised model for obtaining CT bone tissue part map can use typical components such as CNN convolution, ReLU layer, maximum pooling layer, and fully connected layer, and use sequence loss and distance loss to obtain the abnormal weighted map in the dataset D. t The preliminary segmentation results are trained to learn the relationship between CT slices, and the model loss function is Loss = L order +L distance .

[0061] Specifically, we first randomly select b CT volumes (batch size) from the preliminary segmentation results, and randomly select k equidistant slices from the selected volumes. Each axial slice outputs a continuous score Score through the fully connected layer, which represents the normalized position of the bone tissue in the slice. Then, we use the deep learning regressor as a hard constraint to make the CT slices obey the spatial hierarchical ordering, and the order loss is In addition, the numerical difference between the predicted two slice scores is proportional to the spatial distance between the slice images, and the distance loss is used. Constrain the slice score to grow linearly. Where Score(v,s) is the slice score corresponding to the sth slice in the vth CT volume, f sigmoid is the sigmoid activation function, f L1 is the L1 loss.

[0062] Furthermore, the decoder can be constructed and optimized by using the combination function F of continuous operations to speed up The prediction is as follows:

[0063]

[0064] The combination function F includes batch normalization, linear units, optional convolutional layers (possible kernels) and decoder pruning, where conv_projection in the optional convolutional layer is a 1x1x1 projection convolution, conv2D is a 3x3x1 2D convolution, conv3D is a 3x3x3 3D ​​convolution, and convP3D is a 3x3x1 pseudo 3D convolution. Decoder pruning is as follows: the original decoder is matched with the projection block that is not path-connected using the mean square error loss to generate 2 T Decoding paths are selected based on the Dice coefficient (abnormal perception + normal segmentation), decoders are pruned, and speed is increased. predict.

[0065] In this embodiment, the 3D hint encoder receives the intermediate segmentation result output by the abnormality perception and confidence learning module and the preliminary landmark point detection result output by the 3D mask decoder, and processes them respectively, wherein the intermediate segmentation result is subjected to 3D convolution, 3D normalization and activation function GELU processing, and the landmark point detection result is subjected to 3D absolute position encoding and learning embedding processing to obtain hint embedding information, which is fed back to the edge perception module and the 3D mask decoder respectively. The edge perception module uses the hint embedding information to locate the precise edge information, and the 3D mask decoder uses the hint embedding information to obtain more accurate landmark point detection results and segmentation results.

[0066] It is understandable that the above 3D detection network (E Region ), edge perception module (E Edge ) and the anomaly perception and confidence learning module (E Outlier ) can also be implemented in other structural forms according to actual needs, so as to respectively realize 3D image feature extraction, edge information extraction, abnormal position perception and confidence learning, and can be configured according to actual needs.

[0067] In a specific application embodiment, the detailed steps of implementing 3D landmark detection and 3D bone tissue segmentation using a continuous learning framework are as follows:

[0068] (1) Input the preoperative CT data output in step S01 into the 3D detection network (E Region )’s 3D image encoder obtains 3D image encoding features through 3D block embedding, 3D absolute position encoding and 3D attention module;

[0069] (2) Input the 3D image encoding features into the edge perception module (E Edge ) to extract edge information in order to improve spatial positioning and edge information extraction capabilities;

[0070] (3) The edge perception module (E Edge ) The extracted edge information is input into the 3D detection network (E Region )’s 3D mask decoder is processed by image embedding, Transformer module, Transposed 3D convolution and multi-layer perceptron to obtain preliminary segmentation results and landmark detection results;

[0071] (4) Input the preliminary segmentation results into the anomaly perception and confidence learning module (E Outlier ), and obtain the intermediate segmentation result;

[0072] (5) Input the intermediate segmentation results and preliminary landmark detection results into the 3D detection network (E Region), performs 3D convolution, 3D normalization and activation function processing on the intermediate segmentation results, performs 3D absolute position encoding and learning embedding processing on the landmark point detection results, and obtains prompt information output;

[0073] (6) Feedback the hint information output by the 3D hint encoder to the 3D mask decoder and edge perception module (E Edge );

[0074] (7) Repeat steps (1) to (6), and after multiple iterations, output the final segmentation result and landmark point detection result.

[0075] This embodiment constructs the above-mentioned continuous learning framework to simultaneously perform 3D landmark detection and 3D bone tissue segmentation, and constructs a new energy function to measure the consistency between the model prediction results and the actual labels. The energy function used for 3D landmark detection is:

[0076] E Landmark =α1·E Region +β1·E Edge +γ1∑w 2 (1)

[0077] The energy function used for three-dimensional bone tissue segmentation is:

[0078] E Segmentation =α2·E Region +β2·E Edge +γ2·E Outlier +λ1∑w 2 (2)

[0079] Among them, E Region Represents the output result of the 3D detection network, represented by E Region Directly measure the 3D landmark prediction error and the original region pixel segmentation error, E Edge represents the output result of the edge perception module, E Outlier represents the output of the anomaly perception and confidence learning module, E Edge Measuring the mutual occlusion of bone tissue edges and edge pixel segmentation error, E Outlier Measuring the spatial correlation between the structure where the abnormal pixel is segmented and the surrounding tissue, ∑w 2 is the regularization term, and α1, β1, γ1 and α2, β2, γ2, λ1 are the preset coefficients respectively.

[0080] Step S03. Input the intraoperative fluoroscopic X-ray image into the constructed TransUNet network for two-dimensional landmark point detection and two-dimensional bone tissue segmentation. The TransUNet network includes an encoder and a decoder, and a segmentation path for two-dimensional bone tissue segmentation and a landmark point heat map network path for two-dimensional landmark point detection are set at the ends of the encoder and the decoder. The TransUNet network is pre-trained using the loss obtained by combining the segmentation loss and the landmark point detection loss function. The segmentation loss is the loss function corresponding to the segmentation path, and the landmark point detection loss is the loss function corresponding to the landmark point heat map network path.

[0081] The TransUNet network is an image segmentation network that combines Transformer and U-Net. Figure 3 As shown, in this embodiment, the TransUNet network includes a CNN network and a Transformer module composed of multiple Transformer layers to form a CNN-Transformer hybrid encoder. CNN learns implicit features and performs linear projection to extract high-resolution feature maps. Patch is extracted from the feature maps and output to the Transformer module. Each Transformer layer in the Transformer module sequentially normalizes the embedded sequence through layer normalization, multi-head self-attention mechanism, and multi-layer perceptron to obtain encoded feature outputs. After a series of upsampling and convolution operations, the features encoded by the Transformer module are gradually enlarged to the size of the original image, and are fused with the high-resolution feature maps extracted by CNN to obtain landmark detection results and bone tissue segmentation results, respectively, which can simultaneously realize multi-task deep learning of two-dimensional landmark detection and two-dimensional bone tissue segmentation.

[0082] This embodiment adds a segmentation path and a landmark point heat map network path at the end of the TransUNet encoder-decoder. After the intraoperative fluoroscopic X-ray image is processed by the TransUNet encoder-decoder module, the segmentation result is calculated using the standard method. On the landmark point heat map path, the segmentation features before the soft-max of the segmentation path are connected with the features output by the TransUNet encoder-decoder, and the landmark point heat map is estimated by using 1x1 convolution. The TransUNet network is pre-trained using the loss obtained by combining the segmentation loss and the landmark point detection loss function.

[0083] In this embodiment, for the segmentation path in the TransUNet network, the segmentation loss L is calculated by the following formula: seg (w):

[0084]

[0085]

[0086]

[0087] Among them, L dice (i,c,w) is the Dice loss, L cross-entropy (i, k, w) is the cross entropy loss, w is the network weight, i is the model training batch_size value, C is the number of segmentation categories, k is the number of specific segmentation categories for guiding landmark detection, α is the weight coefficient, y true is the true segmentation category label, y predicted is the predicted segmentation category label.

[0088] Taking periacetabular osteotomy as an example, the number of segmentation categories C can be configured to be 8, including osteotomy block (PAO), left hemipelvis (LHP), right hemipelvis (RHP), left femur (LF), right femur (RF), vertebrae, sacrum and background. The number of specific segmentation categories k for guiding landmark point detection can be set to 5, namely osteotomy block segmentation (PAO_seg(w)), left hemipelvis segmentation (LHP_seg(w)), right hemipelvis segmentation (RHP_seg(w)), left femur segmentation (LF_seg(w)), and right femur segmentation (RF_seg(w)).

[0089] In this embodiment, for the landmark point heat map network path in the TransUNet network, the landmark point detection loss function L landmark (w) Using the landmark heat map loss L heat (w) and spatial position constraint L position (w) is calculated, and the calculation expression is:

[0090] L landmark (w) = L heat (w)+βL position (w) (6)

[0091] Among them, the landmark heat map loss L heat (w) Use the true heat map h of each landmark point true (i) and corresponding prediction heatmap The average normalized cross-correlation NCC coefficient between is calculated as the evaluation index, and the calculation expression is:

[0092]

[0093] Among them, (x i ,y i ) is the initial position of the i-th landmark, σ is the standard deviation, N is the number of heatmaps / landmarks, represents a symbolic function with a value of 0 or 1, and β is the weight coefficient.

[0094] In a specific application embodiment, a symmetrical two-dimensional normal distribution is first used to define the real heat map h of the landmark point. true (i) When no landmark is detected or the landmark is not located in the image region of interest (ROI), h true (i) is set to 0; the position of each landmark point is determined as the position with the maximum heat map intensity, and then the true heat map h of each landmark point is calculated true (i) and the corresponding prediction heatmap The average normalized cross-correlation (NCC) coefficient between them is used as the evaluation index to calculate the heat map loss L heat (w).

[0095] In this embodiment, the spatial position constraint L position The acquisition step of (w) includes: limiting the positioning range of the predicted landmark points according to the spatial position of the bone tissue, and jointly evaluating the landmark point positions according to the image segmentation results and the landmark point heat map results. By limiting the positioning range of the landmark points, and jointly evaluating the landmark point positions in combination with the image segmentation results and the landmark point heat map results, the candidate position of the femoral head center is first calculated through the landmark point heat map, and the candidate position is limited to the pixels marked as corresponding to the femur in the bone tissue segmentation results, and then the remaining landmark points are marked as corresponding to the bone tissue part, which can effectively distinguish between landmark point detection, landmark point non-detection and false heat map, and accurately locate the landmark point position.

[0096] Taking periacetabular osteotomy as an example, the spatial position constraint L position The specific steps for obtaining (w) are:

[0097] The left and right candidate positions L_FH and R_FH of the femoral head center are calculated through the landmark heat map, and the left and right candidate positions L_FH and R_FH are limited to the pixels marked as the corresponding femur, that is, the segmentation results LF_seg(w) or RF_seg(w), and the position of the femoral head center is determined according to the multi-pose image. Then all the remaining landmarks are marked as the corresponding hemipelvis and osteotomy parts, that is, LHP_seg(w), RHP_seg(w) and PAO_seg(w). The spatial position constraint L position The specific expression of (w) can be expressed as:

[0098]

[0099] Where I is the multi-pose X-ray image taken by the dynamic fluoroscopic C-arm during surgery, n is the number of multi-pose X-ray images during surgery, fit represents the multi-pose fitting operation, and Mean represents the mean of the (n-1) candidate center coordinates of the femoral head. (x i ,yi ) is the i-th landmark point, in Mean[fit(I1,I2...I n )], I is the multi-pose X-ray image taken by dynamic fluoroscopy of the intraoperative C-arm, n is the number of multi-pose X-ray images during the operation, and fit represents the multi-pose fitting operation: by first aligning the pelvis and then rotating the femur, (n-1) candidate femoral head centers are determined based on the landmarks or markers on the femur, and Mean takes the average of the coordinates of the (n-1) candidate femoral head centers.

[0100] In this embodiment, when the TransUNet network is trained, a minimum value in the training process is sought according to the following formula to make the weights of the segmentation loss function and the landmark detection loss function equal:

[0101] L overall (w) = λL seg (w)+(1-λ)L landmark (w) (9)

[0102] Among them, L seg (w) is the segmentation loss function, L landmark (w) is the landmark detection loss function.

[0103] Specifically, the intraoperative two-dimensional X-ray images and perspective simulation images are used as sample data to train the multi-task deep learning model. The final loss function of the model is shown in the above formula (9), so that the loss function L of the multi-task deep learning algorithm is overall (w) Merge segmentation term L seg (w) and landmark detection term L landmark (w), thereby being able to simultaneously realize two-dimensional landmark point detection and two-dimensional bone tissue segmentation results, and improving the real-time performance of landmark point detection and segmentation.

[0104] Step S04: Output the three-dimensional landmark point detection and three-dimensional bone tissue segmentation results of the preoperative CT data and the two-dimensional landmark point detection and two-dimensional bone tissue segmentation results of the intraoperative fluoroscopic X-ray image for registration.

[0105] The 2D / 3D detection and segmentation device for orthopedic surgery registration in this embodiment includes:

[0106] A data acquisition module is used to acquire the CT data of the bone tissue to be registered obtained before the operation and the perspective X-ray image obtained during the operation;

[0107] A 3D detection and segmentation module is used to input preoperative CT data into a 3D detection network and perform 3D landmark detection and 3D bone tissue segmentation in a continuous learning manner, wherein the 3D image encoding features of the input CT data are first detected by a 3D image encoder in the 3D detection network, and edge information is extracted by an edge perception module. Then, the 3D image encoding features are decoded by a 3D mask decoder in the 3D detection network to obtain preliminary 3D landmark detection and bone tissue segmentation results. The abnormality perception and confidence learning module calculates a confidence map based on the preliminary bone tissue segmentation results to obtain an intermediate segmentation result. The intermediate segmentation result and the preliminary 3D landmark detection result are input into a 3D prompt encoder of the 3D detection network to generate prompt information. The prompt information generated by the 3D prompt encoder is fed back to the 3D mask decoder and the edge perception module. After multiple iterations, the final 3D landmark detection result and segmentation result output are obtained.

[0108] A 2D detection and segmentation module, which is used to input the intraoperative fluoroscopic X-ray image into the constructed TransUNet network for two-dimensional landmark detection and two-dimensional bone tissue segmentation. The TransUNet network includes an encoder and a decoder, and a segmentation path for two-dimensional bone tissue segmentation and a landmark point heat map network path for two-dimensional landmark detection are set between the encoder and the decoder. The TransUNet network is pre-trained using a loss obtained by synthesizing a segmentation loss and a landmark point detection loss function. The segmentation loss is a loss function corresponding to the segmentation path, and the landmark point detection loss is a loss function corresponding to the landmark point heat map network path.

[0109] The detection output module is used to output the three-dimensional landmark point detection and three-dimensional bone tissue segmentation results of the preoperative CT data and the two-dimensional landmark point detection and two-dimensional bone tissue segmentation results of the intraoperative perspective X-ray image for alignment.

[0110] In this embodiment, the structure of the 3D detection and segmentation module is as follows: Figure 2 As shown, including 3D detection network (E Region ), edge perception module (E Edge ) and the anomaly perception and confidence learning module (E Outlier ), the 3D detection network is sequentially equipped with a 3D image encoder, a 3D mask decoder and a 3D hint encoder, an edge perception module (E Edge ) is set between the 3D image encoder and the 3D mask decoder, and a dual-position Transformer module is set at the end of the 3D image encoder and the 3D hint encoder to extract the learning position encoding and the pixel space position encoding respectively. The structure of the 2D detection and segmentation module is as follows Figure 3As shown, two-dimensional landmark detection and two-dimensional bone tissue segmentation are performed simultaneously in the TransUNet network. The 2D / 3D detection and segmentation device for orthopedic surgery registration in this embodiment corresponds to the above-mentioned 2D / 3D detection and segmentation method for orthopedic surgery registration, and will not be described one by one here.

[0111] This embodiment also provides a terminal device, including a processor and a memory, the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0112] It is understandable that the above method of this embodiment can be executed by a single device, such as a computer or server, etc., and can also be applied to a distributed scenario and completed by multiple devices in cooperation with each other. In the case of a distributed scenario, one of the multiple devices can only execute one or more steps in the above method of this embodiment, and multiple devices interact to complete the above method. The processor can be implemented in the form of a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, etc., for executing related programs to implement the above method of this embodiment. The memory can be implemented in the form of a read-only memory ROM, a random access memory RAM, a static storage device, and a dynamic storage device. The memory can store an operating system and other applications. When the above method of this embodiment is implemented by software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0113] This embodiment further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0114] Those skilled in the art will appreciate that the above-mentioned embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0115] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A 2D / 3D detection and segmentation method for orthopedic surgery registration, characterized in that the steps include: Step S01. Acquire CT data of the bone tissue to be registered obtained before the operation and a perspective X-ray image obtained during the operation; Step S02. Input the CT data obtained before the operation into the 3D detection network for 3D landmark point detection and 3D bone tissue segmentation, wherein the 3D image encoding features of the input CT data are first detected by the 3D image encoder in the 3D detection network, and the edge information is extracted by the edge perception module, and then the 3D image encoding features are decoded by the 3D mask decoder in the 3D detection network to obtain preliminary 3D landmark point detection and preliminary bone tissue segmentation results, and the abnormality perception and confidence learning module calculates the confidence map according to the preliminary bone tissue segmentation result to obtain an intermediate segmentation result, and the intermediate segmentation result and the preliminary 3D landmark point detection result are input into the 3D prompt encoder of the 3D detection network to generate prompt information, and the generated prompt information is fed back to the 3D mask decoder and the edge perception module, and after multiple iterations, the final 3D landmark point detection result and bone tissue segmentation result are output; Step S03. Input the fluoroscopic X-ray image obtained during the operation into the constructed TransUNet network to perform two-dimensional landmark point detection and two-dimensional bone tissue segmentation, wherein the TransUNet network includes an encoder and a decoder, and a segmentation path for two-dimensional bone tissue segmentation and a landmark point heat map network path for two-dimensional landmark point detection are set between the encoder and the decoder. The TransUNet network is pre-trained using a loss obtained by synthesizing a segmentation loss and a landmark point detection loss function, wherein the segmentation loss is a loss function corresponding to the segmentation path, and the landmark point detection loss function is a loss function corresponding to the landmark point heat map network path; Step S04. Output the three-dimensional landmark point detection results and bone tissue segmentation results of the preoperative CT data and the two-dimensional landmark point detection and two-dimensional bone tissue segmentation results of the intraoperative perspective X-ray image for registration; In step S02, the energy function used for three-dimensional landmark detection is E Landmark =α1·E Region +β1·E Edge +γ1∑w 2 , the energy function used for three-dimensional bone tissue segmentation is: E Segmentation =α2·E Region +β2·E Edge +γ2·E Outlier +λ1∑w 2 Among them, E Region represents the output result of the 3D detection network, E Edge represents the output result of the edge perception module, E Outlier represents the output result of the anomaly perception and confidence learning module, ∑w 2 is the regularization term, α1, β1, γ1 and α2, β2, γ2, λ1 are the preset coefficients respectively; A continuous learning framework is constructed using the 3D detection network, edge perception module, and anomaly perception and confidence learning modules. The continuous learning framework obtains the prediction result according to the following formula Where N is the number of datasets under the same scene, M is the total number of voxel positions, and T is the total number of sequence datasets containing the same / different scenes. Denotes the dataset D t The predicted map of voxel position j in , j represents the voxel position, X t and Y t Represents the data set D t The input image and the corresponding bone tissue label in f e 、f d Respectively represent the CNN / Transformer functions in the encoder and decoder, c t Indicates the output category, W e , W d are the parameters of the encoding and decoding paths, H t (j) is the data set D t The confidence map of voxel position j in , is the tth data set; The 3D detection network is sequentially provided with a 3D image encoder, a 3D mask decoder and a 3D hint encoder. A dual-position Transformer module is also provided at the end of the 3D image encoder and the 3D hint encoder to extract learning position encoding features and pixel space position encoding respectively. The edge perception module will be used to extract the input image data X 3D The learned position encoding feature extracted from is sent to the multi-head attention module, and the pixel spatial position encoding extracted from the feature of the region of interest in the input image data is used as the key position encoding feature X spacial Add to the target feature of interest, and finally input image data X 3D , the output of the multi-head attention module X attention And the key position encoding feature X spacial The output result E of the edge perception module is obtained through the multi-layer perceptron MLP Edge , that is, E Edge =MLP(ReLU(X 3D +X attention +w*X spacial )), where w is the edge weight and ReLU represents the activation function.

2. The 2D / 3D detection and segmentation method for orthopedic surgery registration according to claim 1, characterized in that: The abnormality perception and confidence learning module obtains the CT bone tissue part map according to the relationship between the slices of the CT preliminary segmentation result output by the 3D mask decoder, overlaps the bounding boxes of all the real bone tissue labels on the bone tissue part map, and calculates the volume overlap ratio between the bounding box and the bone tissue part map, generates the bone tissue part distribution map and takes the average to obtain the average bone tissue distribution map P t , according to the formula Calculate the tth data set D t The weighted graph of voxel position j in P t (j) represents the data set D t The average bone tissue distribution map at voxel position j in , Represents the prediction of different bone tissue abnormalities at voxel position j, where J represents a matrix with elements of 1; the data set D is calculated by the entropy function t Confidence map of voxel position j in To merge overlapping organ predictions from different decoders to get the final prediction result in, Denotes the dataset D t The predicted map of voxel position j in , Represents element-wise multiplication.

3. The 2D / 3D detection and segmentation method for orthopedic surgery registration according to claim 1, characterized in that: In step S03, the segmentation path and the landmark heat map network path are set at the decoder end of the TransUNet network, and the segmentation path is calculated by the following formula: seg (w): Among them, L dice (i',c,w) is the Dice loss, L cross-entropy (i', k, w) is the cross entropy loss, w is the network weight, i' is the model training batch_size value, C is the number of segmentation categories, k is the number of specific segmentation categories for guiding landmark detection, α is the weight coefficient, y true is the true segmentation category label, y predicted is the predicted segmentation category label, LF_seg(w) represents left femur segmentation, RF_seg(w) represents right femur segmentation, LHP_seg(w) represents left hemipelvis segmentation, RHP_seg(w) represents right hemipelvis segmentation, and PAO_seg(w) represents osteotomy segmentation.

4. The 2D / 3D detection and segmentation method for orthopedic surgery registration according to claim 3, characterized in that: In step S03, the landmark point detection loss function L of the landmark point heat map network path in the TransUNet network is landmark (w) Using the landmark heat map loss L heat (w) and spatial position constraint L position (w) is calculated as: L landmark (w)=L heat (w)+βL position (w) The landmark heat map loss L heat (w) Use the true heat map h of each landmark point true (i) and corresponding prediction heatmap The average normalized cross-correlation NCC coefficient between is calculated as the evaluation index, and the calculation expression is: Among them, (x i ,y i ) is the initial position of the i-th landmark, σ is the standard deviation, N is the number of heatmaps / landmarks, represents a symbolic function with a value of 0 or 1, and β is the weight coefficient.

5. The 2D / 3D detection and segmentation method for orthopedic surgery registration according to claim 4, characterized in that: The spatial position constraint L position The acquisition step of (w) includes: limiting the positioning range of the predicted landmark points according to the spatial position of the bone tissue, and jointly evaluating the landmark point positions according to the image segmentation results and the landmark point heat map results, wherein the candidate position of the femoral head center is first calculated by the landmark point heat map, and the candidate position is limited to the pixels marked as corresponding to the femur in the bone tissue segmentation result, and then the remaining landmark points are marked as corresponding to the bone tissue part; When the orthopedic surgery is periacetabular osteotomy, the spatial position constraint L position The steps of obtaining (w) include: The left and right candidate positions L_FH and R_FH of the center of the femoral head are calculated by the landmark point heat map, and the left and right candidate positions L_FH and R_FH are limited to the pixels marked as corresponding to the femur, that is, the segmentation results LF_seg(w) or RF_seg(w), and the position of the center of the femoral head is determined according to the multi-pose image, and all the remaining landmark points are marked as corresponding to the hemipelvis and osteotomy block parts, that is, the positions of LHP_seg(w), RHP_seg(w) and PAO_seg(w). Among them, I is the multi-pose X-ray image taken by the intraoperative C-arm dynamic fluoroscopy, n is the number of intraoperative multi-pose X-ray images, fit represents the multi-pose fitting operation, and Mean represents the mean of the (n-1) femoral head candidate center coordinates.

6. The 2D / 3D detection and segmentation method for orthopedic surgery registration according to any one of claims 1 to 5, characterized in that: In step S03, when training the TransUNet network, an L in the training process is sought according to the following formula: overall (w) is minimized to make the weights of the segmentation loss function and the landmark detection loss function equal: L overall (w)=λL seg (w)+(1-λ)L landmark (w) Among them, L seg (w) is the segmentation loss function, L landmark (w) is the landmark detection loss function.

7. A 2D / 3D detection and segmentation device for orthopedic surgery registration used in the 2D / 3D detection and segmentation method for orthopedic surgery registration according to any one of claims 1 to 6, characterized in that: include: A data acquisition module is used to acquire the CT data of the bone tissue to be registered obtained before the operation and the fluoroscopic X-ray image obtained during the operation; A 3D detection and segmentation module, used for inputting the CT data obtained before the operation into a 3D detection network for 3D landmark detection and 3D bone tissue segmentation, wherein the 3D image encoding features of the input CT data are first detected by a 3D image encoder in the 3D detection network, and edge information is extracted by an edge perception module, and then the 3D image encoding features are decoded by a 3D mask decoder in the 3D detection network to obtain preliminary 3D landmark detection and preliminary bone tissue segmentation results, and the abnormality perception and confidence learning module calculates a confidence map based on the preliminary bone tissue segmentation result to obtain an intermediate segmentation result, and the intermediate segmentation result and the preliminary 3D landmark detection result are input into a 3D prompt encoder of the 3D detection network to generate prompt information, and the prompt information generated by the 3D prompt encoder is fed back to the 3D mask decoder and the edge perception module, and after multiple iterations, the final 3D landmark detection result and bone tissue segmentation result are output; A 2D detection and segmentation module, for inputting the intraoperative fluoroscopic X-ray image into a constructed TransUNet network for two-dimensional landmark detection and two-dimensional bone tissue segmentation, wherein the TransUNet network includes an encoder and a decoder, and a segmentation path for two-dimensional bone tissue segmentation and a landmark point heat map network path for two-dimensional landmark detection are arranged between the encoder and the decoder, wherein the TransUNet network is pre-trained using a loss obtained by synthesizing a segmentation loss and a landmark point detection loss function, wherein the segmentation loss is a loss function corresponding to the segmentation path, and the landmark point detection loss function is a loss function corresponding to the landmark point heat map network path; The detection output module is used to output the three-dimensional landmark point detection and three-dimensional bone tissue segmentation results of the preoperative CT data and the two-dimensional landmark point detection and two-dimensional bone tissue segmentation results of the intraoperative perspective X-ray image for alignment.

8. A terminal device, comprising a processor and a memory, wherein the memory is used to store a computer program, characterized in that: The processor is configured to execute the computer program to perform the method according to any one of claims 1 to 6.

Images