Percutaneous lumbar intervertebral disc puncture surgery navigation method and device based on multi-modal image

Abstract

The invention provides a percutaneous lumbar intervertebral disc puncture surgery navigation method and device based on a multi-modal image, and the method comprises the steps: obtaining a preoperative MR image and a CT image, and sequentially inputting a multi-contrast multi-modal image segmentation model and a multi-modal image rigid-elastic mixed registration model to obtain a first registration image; calling the multi-contrast multi-modal image segmentation model and the multi-modal image rigid-elastic hybrid registration model again based on the intraoperative three-dimensional CBCT image to obtain a second registration image; and constructing a spine three-dimensional visual model based on the second registration image and generating a puncture navigation path. According to the technical scheme provided by the embodiment of the invention, the mask can be utilized to perform automatic registration and alignment on the MR image, the CT image and the intraoperative three-dimensional CBCT image, and the preoperative MR image is introduced into a fat-pressing sequence and a non-fat-pressing sequence, so that the second registration image can distinguish the bone-flesh and spine structure, the accuracy of the spine three-dimensional visualization model is improved, and the accuracy of the spine three-dimensional visualization model is improved. The precision of a navigation path and the feasibility of an operation are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of surgical navigation, and in particular to a percutaneous lumbar disc puncture surgical navigation method and device based on multimodal images. Background Art

[0002] At present, during the transforaminal puncture, percutaneous lumbar disc puncture is mostly performed under the guidance of two-dimensional C-arm CT (CBCT) images. During the operation, multiple fluoroscopy and repeated adjustment of the puncture needle to the position of the disc lesion may be required. The above process not only increases the radiation exposure, but also increases the risk of nerve root, dura mater and blood vessel damage, and has the disadvantages of low radiation exposure, safety or accuracy. At the same time, the anatomical structure of the lumbar intervertebral foramen is complex, containing important anatomical structures such as fat, nerve roots, arteries and veins, intervertebral foraminal ligaments, and sinus nerves. The inner side is the dura mater and running nerve roots. The thickening of the yellow ligament and hyperplasia of the articular process joints caused by degeneration can further change the morphology of the intervertebral foramen. These factors all cause difficulties and risks to the operation of percutaneous lumbar disc puncture.

[0003] In scenarios with complex spinal structures, current percutaneous lumbar disc puncture surgery guided by two-dimensional CBCT images, magnetic resonance (MR) images, and computed tomography (CT) images has difficulty distinguishing between bone and flesh, image guidance is not intuitive, and the surgeon is highly dependent on experience. In addition, tissue damage such as nerve roots, dura mater, vertebral end plates, and blood vessels may occur during the operation, and the accuracy and feasibility of surgical navigation are insufficient. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a percutaneous lumbar disc puncture surgical navigation method and device based on multimodal images, which can clearly display multiple structures of the spine in all directions during surgical navigation, realize intuitive and accurate image guidance, and improve the accuracy and feasibility of surgical navigation.

[0005] In a first aspect, an embodiment of the present invention provides a percutaneous lumbar disc puncture surgical navigation method based on multimodal images, which is applied to a surgical navigation system. The surgical navigation system is pre-configured with a multi-contrast multimodal image segmentation model and a multimodal image rigid-elastic hybrid registration model. The percutaneous lumbar disc puncture surgical navigation method based on multimodal images includes:

[0006] Acquire a preoperative spinal image set, wherein the preoperative spinal image set includes a preoperative thin-layer fat-suppressed MR image, a preoperative thin-layer non-fat-suppressed MR image, and a preoperative CT image, wherein the preoperative thin-layer fat-suppressed MR image is acquired based on a preset fat-suppressed sequence, and the preoperative thin-layer non-fat-suppressed MR image is acquired based on a preset non-fat-suppressed sequence;

[0007] Inputting the preoperative spinal image set into the multi-contrast multi-modal image segmentation model for image segmentation, and fusing the image segmentation results to obtain a first mask, wherein the first mask is used to indicate a spinal structure or a preoperative lesion target;

[0008] Inputting the preoperative spinal image set and the first mask into a multimodal image rigid-elastic hybrid registration model, performing a registration and alignment operation, and obtaining a first registered image, wherein the first registered image includes the preoperative thin-layer fat-suppressed MR image, the preoperative thin-layer non-fat-suppressed MR image, and the preoperative CT image that are registered with each other;

[0009] Acquire an intraoperative three-dimensional CBCT image, and input the intraoperative three-dimensional CBCT image into the multi-contrast multi-modal image segmentation model for image segmentation to obtain a second mask, wherein the second mask is used to indicate a vertebral structure;

[0010] Inputting the second mask, the intraoperative three-dimensional CBCT image and the first registered image into the multimodal image rigid-elastic hybrid registration model, performing the registration and alignment operation, and obtaining a second registered image, wherein the second registered image includes the first registered image and the intraoperative three-dimensional CBCT image that are registered with each other;

[0011] A three-dimensional visualization model of the spine is constructed based on the second registered image, and when a target lesion target point marked on the three-dimensional visualization model of the spine is acquired, a puncture navigation path is generated based on the target lesion target point.

[0012] According to some embodiments of the present invention, the multi-contrast multi-modal image segmentation model includes a first segmentation branch and a second segmentation branch, the first segmentation branch includes a first encoder and a first decoder, the second segmentation branch includes a second encoder and a second decoder, the preoperative spinal image set is input into the multi-contrast multi-modal image segmentation model for image segmentation, and the image segmentation results are fused to obtain a first mask, including:

[0013] Inputting the preoperative thin layer fat-suppressed MR image into the first segmentation branch, extracting a first regional feature of the preoperative thin layer fat-suppressed MR image through the first encoder, extracting first semantic information of the first regional feature through the first decoder, and determining a first segmentation result based on the first semantic information, wherein the first regional feature is used to indicate the spinal structure or the preoperative lesion target;

[0014] Inputting the preoperative thin-layer non-fat-suppressed MR image into the first segmentation branch, and inputting the CT image into the second segmentation branch;

[0015] Extracting a second regional feature of the preoperative thin-layer non-fat-suppressed MR image by the first encoder, extracting second semantic information of the second regional feature by the first decoder, and determining a second segmentation result based on the second semantic information, wherein the second regional feature is used to indicate the spinal structure or the preoperative lesion target;

[0016] Inputting the second semantic information into the second segmentation branch, and extracting a third region feature of the CT image through the second encoder;

[0017] Performing comparative learning based on the second semantic information and the third region feature to obtain a third segmentation result;

[0018] The first segmentation result, the second segmentation result, and the third segmentation result are fused to obtain the first mask.

[0019] According to some embodiments of the present invention, the second decoder includes a plurality of attention-aware fusion modules, and the performing comparative learning based on the second semantic information and the third region feature to obtain the third segmentation result includes:

[0020] Inputting the second semantic information into the attention perception fusion module to obtain an attention feature, and upsampling the attention feature and fusing it into the third region feature;

[0021] extracting third semantic information of the third region feature, and determining a third segmentation result based on the third semantic information and a preset contrast loss function, wherein the third region feature is used to indicate a vertebral structure;

[0022] Among them, the contrast loss function satisfies the following conditions: loss seg =loss a +λ b loss b +λ con loss con , loss a is the cross entropy loss function of the first segmentation branch, loss b is the cross entropy loss function of the second segmentation branch, λ b and λ con is the preset magnification factor, loss con is the contrast loss function, loss seg is the total loss function of the multi-contrast multimodal image segmentation model.

[0023] According to some embodiments of the present invention, the first encoder and the second encoder perform encoding N times, the first decoder and the second decoder perform decoding N times, the number of the attention-aware fusion modules is N, N is a positive integer greater than 2, the second semantic information is input into the attention-aware fusion module to obtain an attention feature, and the attention feature is up-sampled and fused to the third region feature, including:

[0024] Inputting the second semantic information obtained by the Nth decoding of the first decoder into the first attention perception fusion module, performing an attention extraction operation, and obtaining the first attention feature;

[0025] Upsampling the first attention feature, fusing it with the third region feature obtained by the Nth encoding of the second encoder, and then inputting it into the second attention perception fusion module;

[0026] Inputting the second semantic information obtained by the N-1th decoding of the first decoder into the second attention perception fusion module, performing the attention extraction operation, and obtaining the second attention feature;

[0027] When the Nth attention feature is obtained, it is fused with the third region feature obtained by the first encoding of the second encoder to obtain the attention feature.

[0028] According to some embodiments of the present invention, the attention perception fusion module includes a channel attention module and a spatial attention module, the channel attention module includes a first convolution block, a first BN block and a first ReLu layer, a first maximum pooling layer, a first average pooling layer, a multilayer perceptron and a first Sigmoid function, the spatial attention module includes a second convolution block, a second BN block, a third convolution block and a third BN block, and the attention extraction operation includes:

[0029] Determine a segmented image pair, wherein the segmented image pair includes the preoperative thin-layer non-fat-suppressed MR image and the preoperative CT image, or includes the first registered image and the intraoperative three-dimensional CBCT image;

[0030] Obtaining a first transformation feature based on the second semantic information by sequentially processing the first convolution block, the first BN block, and the first ReLu layer;

[0031] Inputting the first transformed features into the first maximum pooling layer and the first average pooling layer respectively, fusing the outputs of the first maximum pooling layer and the first average pooling layer, and sequentially passing through the multi-layer perceptron and the first Sigmoid function to obtain a channel attention map;

[0032] Based on the third region feature, sequentially passing through the second convolution block and the second BN block, a second transformation feature is obtained;

[0033] Inputting the first transformation feature into the spatial attention module, fusing the first transformation feature with the second transformation feature, and then sequentially passing through the third convolution block and the third BN block to obtain a spatial attention map;

[0034] The attention feature is obtained based on the spatial attention map and the channel attention map.

[0035] According to some embodiments of the present invention, the multimodal image rigid-elastic hybrid registration model includes a multiple affine matrix estimation module, an affine elastic fusion module and a local rigid constraint module, and the registration and alignment operation includes:

[0036] Inputting a plurality of images to be registered into the multiple affine matrix estimation module to obtain multi-scale image features and initial deformation fields of each of the images to be registered, wherein the images to be registered include the preoperative thin-layer fat-suppressed MR image, the preoperative thin-layer non-fat-suppressed MR image and the preoperative CT image, or the images to be registered include the first registration image and the intraoperative three-dimensional CBCT image;

[0037] Based on the same image to be registered, the corresponding multi-scale image features and the initial deformation field are input into the affine elastic fusion module, deformation field splicing is performed based on the initial deformation field to obtain a rigid elastic deformation field, and all the rigid elastic deformation fields are fused to output an initial registration image;

[0038] The initial registration image and the registration mask are input into the local rigid constraint module, and the rigid elastic deformation field of the initial registration image is corrected by the local rigid constraint module to obtain a target registration image, wherein, when the image to be registered includes the preoperative spinal image set, the registration mask is the first mask, or, when the image to be registered includes the intraoperative three-dimensional CBCT image, the registration mask is the second mask.

[0039] According to some embodiments of the present invention, the multiple affine matrix estimation module includes 4 fourth convolution blocks, a flat layer, and a rigid transformation estimation module arranged in series, the rigid transformation estimation module includes multiple dense layers and multiple affine matrices, the dense layers uniquely correspond to the affine matrices, and the multi-scale image features and initial deformation fields of each of the images to be registered are obtained, including:

[0040] Subjecting the image to be registered to convolution processing of each of the fourth convolution blocks in sequence to obtain the multi-scale image feature;

[0041] After the multi-scale image features are expanded based on each scale through the flattening layer, the image features of each scale are input into one of the dense layers of the rigid transformation estimation module, and the rigid deformation field is calculated based on the preset rigid transformation estimation parameters;

[0042] The initial deformation field is obtained by taking a weighted average of all the rigid deformation fields.

[0043] According to some embodiments of the present invention, the affine elastic fusion module includes 4 combination units, three sixth convolution blocks and an integrated block, the combination unit includes an upsampling layer and a fifth convolution block, the deformation field is spliced ​​based on the initial deformation field to obtain a rigid elastic deformation field, and all the rigid elastic deformation fields are fused to output an initial registration image, including:

[0044] The image features of each scale are processed by the four combination units in sequence to obtain the structural image features of each scale;

[0045] Splicing the structural image features of the corresponding scale with the initial deformation field to obtain a rigid elastic deformation field;

[0046] After each of the rigid-elastic deformation fields is input into the sixth convolution block for convolution processing, it is input into the integrated block for fusion to obtain the initial registration image.

[0047] According to some embodiments of the present invention, the step of correcting the rigid elastic deformation field of the initial registration image by the local rigid constraint module to obtain a target registration image includes:

[0048] Performing least square regression calculation on the rigid elastic deformation field based on the registration mask to obtain a reference deformation field;

[0049] constructing a rigid constraint cost function based on an error between the rigid elastic deformation field and the reference deformation field;

[0050] The target registration image is obtained by modifying the initial registration image based on the rigid constraint cost function.

[0051] In a second aspect, an embodiment of the present invention provides a percutaneous lumbar disc puncture surgical navigation device based on multimodal images, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the percutaneous lumbar disc puncture surgical navigation method based on multimodal images as described in the first aspect above.

[0052] In a third aspect, an embodiment of the present invention provides an electronic device, comprising the multimodal image-based percutaneous lumbar disc puncture surgical navigation device as described in the second aspect above.

[0053] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the multimodal image-based percutaneous lumbar disc puncture surgical navigation method as described in the first aspect above.

[0054] According to the multimodal image-based percutaneous lumbar disc puncture surgical navigation method of an embodiment of the present invention, there are at least the following beneficial effects: obtaining a preoperative spinal image set, wherein the preoperative spinal image set includes a preoperative thin-layer fat-suppressed MR image, a preoperative thin-layer non-fat-suppressed MR image and a preoperative CT image, the preoperative thin-layer fat-suppressed MR image is acquired based on a preset fat-suppressed sequence, and the preoperative thin-layer non-fat-suppressed MR image is acquired based on a preset non-fat-suppressed sequence; inputting the preoperative spinal image set into the multi-contrast multimodal image segmentation model for image segmentation, fusing the image segmentation results to obtain a first mask, wherein the first mask is used to indicate a spinal structure or a preoperative lesion target; inputting the preoperative spinal image set and the first mask into a multimodal image rigid-elastic hybrid registration model, performing a registration and alignment operation, and obtaining a first registered image, the first registered image including the corresponding The pre-thin layer fat suppression MR image, the pre-operative thin layer non-fat suppression MR image and the pre-operative CT image are mutually registered; an intra-operative three-dimensional CBCT image is acquired, and the intra-operative three-dimensional CBCT image is input into the multi-contrast multi-modal image segmentation model for image segmentation to obtain a second mask, wherein the second mask is used to indicate the vertebral structure; the second mask, the intra-operative three-dimensional CBCT image and the first registration image are input into the multi-modal image rigid-elastic hybrid registration model, and the registration alignment operation is performed to obtain a second registration image, wherein the second registration image includes the mutually registered first registration image and the intra-operative three-dimensional CBCT image; a three-dimensional visualization model of the spine is constructed based on the second registration image, and when the target lesion target marked in the three-dimensional visualization model of the spine is acquired, a puncture navigation path is generated based on the target lesion target. According to the technical solution of the embodiment of the present invention, it is possible to use image segmentation to determine a mask that characterizes the spinal structure or vertebral structure, and use the mask to automatically align preoperative thin-layer MR images, CT images and intraoperative three-dimensional CBCT images, and introduce fat suppression and non-fat suppression sequences into the preoperative MR images, so that the second registered image can clearly display the lumbar vertebral structure at the MR, CT and CBCT levels, improve the effect of distinguishing between bones, flesh and various structures in the second registered image, improve the accuracy of the three-dimensional visualization model of the spine, and thus improve the accuracy of the navigation path and the feasibility of the operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flowchart of a multimodal image-based percutaneous lumbar disc puncture surgical navigation method provided by an embodiment of the present invention;

[0056] Figure 2 is a complete flow chart of a multimodal image-based percutaneous lumbar disc puncture surgical navigation method provided by another embodiment of the present invention;

[0057] Figure 3 is a structural diagram of a multi-contrast multi-modal image segmentation model provided by another embodiment of the present invention;

[0058] Figure 4 is a structural diagram of an attention perception fusion module provided by another embodiment of the present invention;

[0059] Figure 5 is a structural diagram of a multimodal image rigid-elastic hybrid registration model provided by another embodiment of the present invention;

[0060] Figure 6 It is a structural diagram of a multimodal image-based percutaneous lumbar disc puncture surgical navigation device provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0061] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0062] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0063] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0064] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0065] An embodiment of the present invention provides a percutaneous lumbar disc puncture surgical navigation method and device based on multimodal images, wherein the percutaneous lumbar disc puncture surgical navigation method based on multimodal images comprises: obtaining a preoperative spinal image set, wherein the preoperative spinal image set comprises a preoperative thin-layer fat-suppressed MR image, a preoperative thin-layer non-fat-suppressed MR image and a preoperative CT image, wherein the preoperative thin-layer fat-suppressed MR image is obtained based on a preset fat-suppressed sequence, and the preoperative thin-layer non-fat-suppressed MR image is obtained based on a preset non-fat-suppressed sequence; inputting the preoperative spinal image set into the multi-contrast multimodal image segmentation model for image segmentation, and fusing the image segmentation results to obtain a first mask, wherein the first mask is used to indicate a spinal structure or a preoperative lesion target; inputting the preoperative spinal image set and the first mask into a multimodal image rigid-elastic hybrid registration model, performing a registration and alignment operation, and obtaining a first registered image. The first registration image includes the mutually registered anterior thin-layer fat-suppressed MR image, the preoperative thin-layer non-fat-suppressed MR image and the preoperative CT image; an intraoperative three-dimensional CBCT image is acquired, and the intraoperative three-dimensional CBCT image is input into the multi-contrast multi-modal image segmentation model for image segmentation to obtain a second mask, wherein the second mask is used to indicate the vertebral structure; the second mask, the intraoperative three-dimensional CBCT image and the first registration image are input into the multi-modal image rigid-elastic hybrid registration model, and the registration alignment operation is performed to obtain a second registration image, wherein the second registration image includes the mutually registered first registration image and the intraoperative three-dimensional CBCT image; a three-dimensional visualization model of the spine is constructed based on the second registration image, and when the target lesion target marked on the three-dimensional visualization model of the spine is acquired, a puncture navigation path is generated based on the target lesion target. According to the technical solution of the embodiment of the present invention, it is possible to use image segmentation to determine a mask that characterizes the spinal structure or vertebral structure, and use the mask to automatically align preoperative thin-layer MR images, CT images and intraoperative three-dimensional CBCT images, and introduce fat suppression and non-fat suppression sequences into the preoperative MR images, so that the second registered image can clearly display the lumbar vertebral structure at the MR, CT and CBCT levels, improve the effect of distinguishing between bones, flesh and various structures in the second registered image, improve the accuracy of the three-dimensional visualization model of the spine, and thus improve the accuracy of the navigation path and the feasibility of the operation.

[0066] The technical solution of the embodiment of the present invention is further described below based on the accompanying drawings.

[0067] Reference Figure 1 , Figure 1A flowchart of a multimodal image-based percutaneous lumbar disc puncture surgical navigation method is provided in an embodiment of the present invention. The multimodal image-based percutaneous lumbar disc puncture surgical navigation method includes but is not limited to the following steps:

[0068] S10, obtaining a preoperative spinal image set, wherein the preoperative spinal image set includes a preoperative thin-layer fat-suppressed MR image, a preoperative thin-layer non-fat-suppressed MR image, and a preoperative CT image, the preoperative thin-layer fat-suppressed MR image is obtained based on a preset fat-suppressed sequence, and the preoperative thin-layer non-fat-suppressed MR image is obtained based on a preset non-fat-suppressed sequence;

[0069] S20, inputting the preoperative spinal image set into a multi-contrast multi-modal image segmentation model for image segmentation, and fusing the image segmentation results to obtain a first mask, wherein the first mask is used to indicate a spinal structure or a preoperative lesion target;

[0070] S30, inputting the preoperative spinal image set and the first mask into the multimodal image rigid-elastic hybrid registration model, performing a registration and alignment operation, and obtaining a first registered image, wherein the first registered image includes a preoperative thin-layer fat-suppressed MR image, a preoperative thin-layer non-fat-suppressed MR image, and a preoperative CT image that are registered with each other;

[0071] S40, acquiring an intraoperative three-dimensional CBCT image, and inputting the intraoperative three-dimensional CBCT image into a multi-contrast multi-modal image segmentation model for image segmentation to obtain a second mask, wherein the second mask is used to indicate a vertebral structure;

[0072] S50, inputting the second mask, the intraoperative three-dimensional CBCT image and the first registered image into a multimodal image rigid-elastic hybrid registration model, performing a registration and alignment operation, and obtaining a second registered image, wherein the second registered image includes the first registered image and the intraoperative three-dimensional CBCT image that are registered with each other;

[0073] S60, constructing a three-dimensional visualization model of the spine based on the second registered image, and when the target lesion target marked on the three-dimensional visualization model of the spine is obtained, generating a puncture navigation path based on the target lesion target.

[0074] It should be noted that fat suppression is a medical image obtained by taking fat suppression technology, which is specifically defined as a technology well known to those skilled in the art. Preoperative thin-layer fat suppression MR images are obtained by taking fat suppression sequences, and preoperative thin-layer non-fat suppression MR images are obtained by taking non-fat suppression sequences. The patient's posture is the same when taking preoperative thin-layer fat suppression MR images and preoperative thin-layer non-fat suppression MR images, ensuring that the images are aligned in space.

[0075] For example, 3D-SPACE non-fat suppression sequence and fat suppression sequence can be used to obtain two different sequences of transverse imaging data. The specific parameters of the 3D-Space non-fat suppression sequence are: repetition time / echo time is 2800.0ms / 189.0ms, flip angle is 45°, field of view is 240×240mm, matrix is ​​320×320, slice thickness is 0.8mm, bandwidth is 579kHz, and final image resolution is 0.8×0.8×0.8mm; the specific parameters of the 3D-Space fat suppression sequence are: repetition time / echo time is 2800.0ms / 189.0ms, flip angle is 45°, field of view is 240×240mm, matrix is ​​320×320, slice thickness is 0.8mm, bandwidth is 579kHz, and final image resolution is 0.8×0.8×0.8mm. After shooting, the data is exported in DICOM format.

[0076] It should be noted that the preoperative CT image can use the direction perpendicular to the patient's long axis as the scanning baseline, scan the patient layer by layer without interval, with a scanning field of view of 180mm and a layer thickness of 1mm. The patient's transverse CT imaging data is collected and exported in DICOM format.

[0077] It should be noted that the intraoperative three-dimensional CBCT images include multiple spinal structures, which can be scanned three-dimensionally using a three-dimensional C-arm to obtain three-dimensional CBCT imaging data of the L4 / L5 or L5 / S1 lumbar segment and export the data in DICOM format.

[0078] It should be noted that after obtaining the preoperative spinal image set, the spinal structure or preoperative lesion target in each image can be marked first. The spinal structure can be L4 vertebra, L4 / L5 intervertebral disc, L5 vertebra, L4 nerve root, L5 nerve root, dura mater, skin, and the preoperative lesion target in this embodiment is the lesion tissue of intervertebral disc herniation.

[0079] It should be noted that the multi-contrast multi-modal image segmentation model includes multiple segmentation branches, which segment two images at the same time, and fuse the results obtained after the segmentation, so that the first mask can represent the features respectively possessed by the two images, thereby representing the spinal structure or preoperative lesion target through the first mask. For example, after the image is annotated according to the above spinal structure, the preoperative thin-layer non-fat-suppressed MR image and the preoperative CT image are simultaneously input into the multi-contrast multi-modal image segmentation model, and the semantic representation of the spinal structure without fat suppression is obtained by segmenting the preoperative thin-layer non-fat-suppressed MR image, the semantic representation of the spinal structure in the fat-suppressed state is obtained by segmenting the preoperative thin-layer fat-suppressed MR image, and the semantic representation of the vertebral structure is obtained by segmenting the preoperative CT image. The first mask obtained after fusion can represent the semantic representation of the spinal structure or preoperative lesion target at the fat-suppressed, non-fat-suppressed and CT levels, thereby providing structural reference data for image registration and alignment.

[0080] It should be noted that, by registering the MR image and the CT image under the first mask, the same spinal structure or lesion target in the MR image and the CT image can be aligned on the image, and the first mask of this embodiment introduces fat-suppressed MR images and non-fat-suppressed MR images, which can distinguish the flesh boundary by using the difference in images during fat-suppression and non-fat-suppression, so that the first registered image obtained by the registration can effectively distinguish between bones and flesh, thereby realizing bone and flesh visualization in the subsequently established three-dimensional visualization model of the spine, effectively improving the accuracy and feasibility of surgical navigation.

[0081] It should be noted that, after acquiring the intraoperative three-dimensional CBCT image during the operation, this embodiment again calls the multi-contrast multimodal image segmentation model to perform image segmentation, so that the second mask corresponding to the semantic information obtained by the segmentation can represent the vertebral structure in the plane dimension, and then calls the multimodal image rigid-elastic hybrid registration model to perform secondary registration with the first registration image, so as to achieve accurate registration and alignment of the plane vertebral structure and the three-dimensional spinal structure, thereby obtaining a clear MR / CT / three-dimensional CBCT image, and improving the accuracy of the constructed three-dimensional visualization model of the spine.

[0082] It should be noted that after obtaining the clear MR / CT / 3D CBCT images after registration, those skilled in the art are familiar with how to construct a 3D visualization model of the spine, and this embodiment does not elaborate on the model construction technology.

[0083] It should be noted that after the target lesion target is selected in the three-dimensional visualization model of the spine, the target lesion target can be located in the plane and space based on the registration relationship of the second registration image, and multiple available puncture navigation paths can be automatically generated, thereby providing multiple path selections and completing surgical navigation based on the final selected target puncture path. This embodiment does not make any improvements to the specific path planning and surgical navigation process, and will not be elaborated here.

[0084] Through the technical solution of this embodiment, the multimodal image rigid-elastic hybrid registration model can be used to register preoperative high-resolution MR images, CT images, and intraoperative three-dimensional CBCT images, and the registered images can distinguish between bones and flesh based on fat-suppressed and non-fat-suppressed MR images. Through the multi-contrast multimodal image segmentation model, the preoperative CT images, preoperative thin-layer fat-suppressed MR images, and preoperative thin-layer non-fat-suppressed MR images from the same source are fused to extract high-contrast spinal structure feature information, and the multi-category structures and lesion targets of the spine are accurately segmented. Finally, a complete three-dimensional spinal model is constructed in combination with the multimodal registration method to improve the accuracy and feasibility of surgical navigation.

[0085] In addition, in one embodiment, referring to Figure 3 The multi-contrast multimodal image segmentation model includes a first segmentation branch and a second segmentation branch, the first segmentation branch includes a first encoder and a first decoder, and the second segmentation branch includes a second encoder and a second decoder. Figure 2 , Figure 1 The step S20 shown specifically includes but is not limited to the following steps:

[0086] S21, inputting the preoperative thin layer fat suppression MR image into the first segmentation branch, extracting the first regional feature of the preoperative thin layer fat suppression MR image through the first encoder, extracting the first semantic information of the first regional feature through the first decoder, and determining the first segmentation result based on the first semantic information, wherein the first regional feature is used to indicate the spinal structure or the preoperative lesion target;

[0087] S22, inputting the preoperative thin-layer non-fat-suppressed MR image into the first segmentation branch, and inputting the CT image into the second segmentation branch;

[0088] S23, extracting a second regional feature of the preoperative thin-layer non-fat-suppressed MR image by the first encoder, extracting second semantic information of the second regional feature by the first decoder, and determining a second segmentation result based on the second semantic information, wherein the second regional feature is used to indicate a spinal structure or a preoperative lesion target;

[0089] S24, inputting the second semantic information into the second segmentation branch, and extracting a third region feature of the CT image through a second encoder;

[0090] S25, performing comparative learning based on the second semantic information and the third region feature to obtain a third segmentation result;

[0091] S26, fusing the first segmentation result, the second segmentation result, and the third segmentation result to obtain a first mask.

[0092] It should be noted that the multi-contrast multimodal image segmentation model is Figure 3As shown, in this embodiment, the preoperative thin layer fat suppression MR image is first input into the first segmentation branch. In the first decoder, the preoperative thin layer fat suppression MR image is first subjected to a first encoding operation to realize image characterization. The first encoding operation includes convolution (conv), batch normalization (BN) and activation (ReLu) in sequence. Then, based on the first region feature, a second encoding operation, three third encoding operations, a fourth encoding operation and a fifth encoding operation are performed in sequence. The output feature is used as the first region feature. The second encoding operation includes maximum pooling (Maxpool), conv+BN+ReLu in sequence. The third encoding operation includes Maxpool and (conv+BN+ReLu)×2 in sequence. The fourth encoding operation includes Maxpool. The fifth encoding operation includes upsampling (Upsamp le).

[0093] It should be noted that after the encoding is completed, the first region feature is input into the first decoder, the semantic information is obtained through the first decoding operation, the first decoding information is fused with the features of the corresponding encoding layer, and the second decoding operation is performed to obtain further optimized semantic information, and the above operation is continued, and the final output semantic information is subjected to the third decoding operation to obtain the first semantic information. The first decoding operation includes Upsamp le, the second decoding operation includes (conv+BN+ReLu)×2, and the third decoding operation includes conv.

[0094] For example, Figure 3 As shown, encoding is performed five times in the first encoder. After the first decoding operation is performed for the first time, the fused coding layer is the image features of the fifth coding operation, and the second decoding information is obtained. After the first decoding operation is performed on the second decoding information, the fused coding layer is the image features of the fourth coding operation, and so on, until the image features of the first coding operation are fused. Through the image segmentation operation of this embodiment, the coding features of the same layer can be fused during decoding, and features with the same semantics can be continuously extracted, so that the first segmentation result can more clearly reflect the labeled spinal structure or preoperative lesion target.

[0095] It should be noted that the process of extracting the second segmentation result of the preoperative thin-layer non-fat-suppressed MR image through the first segmentation branch can refer to the extraction process of the first segmentation result, which will not be repeated here.

[0096] It should be noted that after completing the segmentation of the preoperative thin-layer fat-suppressed MR image, the first segmentation result is saved as reference information for distinguishing the bone-flesh boundary. Then, the preoperative thin-layer non-fat-suppressed MR image and the preoperative CT image are simultaneously input into the multi-contrast multimodal image segmentation model for contrast learning segmentation. After the second semantic information of the preoperative thin-layer non-fat-suppressed MR image is extracted by the first decoder, it is input into the attention-aware fusion module (AFM) to guide the second segmentation branch to focus on the vertebral structure area when segmenting the preoperative CT image, and focus on extracting the regional features of the CT vertebral structure, so that the third segmentation result can clearly characterize the vertebral structure.

[0097] It should be noted that after obtaining the first segmentation result, the second segmentation result and the third segmentation result, the three segmentation results can be combined with a preset loss function calculation and then fused. The specific fusion process will not be described in detail here.

[0098] In addition, in one embodiment, referring to Figure 3 , the second decoder includes multiple attention-aware fusion modules, referring to Figure 2 Step S25 specifically includes but is not limited to the following steps:

[0099] S251, inputting the second semantic information into the attention perception fusion module to obtain an attention feature, and upsampling the attention feature and fusing it into the third region feature;

[0100] S252, extracting third semantic information of the third region feature, and determining a third segmentation result based on the third semantic information and a preset contrast loss function, wherein the third region feature is used to indicate a vertebral structure;

[0101] Among them, the contrast loss function satisfies the following conditions: loss seg =loss a +λ b loss b +λ con loss con , loss a is the cross entropy loss function of the first segmentation branch, loss b is the cross entropy loss function of the second segmentation branch, λ b and λ con is the preset magnification factor, loss con is the comparison loss function, loss seg is the total loss function of the multi-contrast multimodal image segmentation model.

[0102] It should be noted that after obtaining the second semantic information, refer to Figure 3As shown, the second semantic information of the corresponding scale is input into the AFM of the same scale, and the number of AFMs is the same as the number of encoding times, so that when the features of the same scale are decoded, the attention features can be extracted through the attention mechanism, and the attention features are combined with the third region features, so that the extracted semantic information can be more focused on the vertebral structure.

[0103] It should be noted that by contrasting the loss function, the points of the same category near the vertebral structure are made similar in the feature space, and the features of the vertebral structure area are enhanced, thereby improving the segmentation accuracy of the vertebral structure. According to the above method, the thin layer non-fat-suppressed MR image and thin layer fat-suppressed MR image segmentation group data are input into the multi-contrast multimodal image segmentation model for automatic segmentation to enhance the features of the nerve root structure area and improve the segmentation accuracy of the nerve root structure. During the test, the segmentation results output by the thin layer fat-suppressed MR image segmentation branch are saved. Finally, the segmentation results output by the thin layer fat-suppressed MR image and the CT image are automatically spliced, and the spliced ​​segmentation results are used as the final segmentation results.

[0104] It should be noted that in the multi-structure segmentation process of multi-modal images, the junction area between the vertebral structure and the muscle ligament, and the junction area between the nerve root and the dura mater are not clear, and there are problems such as low recognition and segmentation discontinuity. To solve the above problems, this embodiment designs a contrast loss function to make the points of the same category near the vertebral structure and the nerve root similar in the feature space, aiming to improve the distinction between the vertebral structure and the nerve root area.

[0105] In addition, in one embodiment, referring to Figure 3 , the encoding times of the first encoder and the second encoder are N times, the decoding times of the first decoder and the second decoder are N times, the number of attention perception fusion modules is N, N is a positive integer greater than 2, and the reference Figure 2 Step S251 specifically includes but is not limited to the following steps:

[0106] S2511, inputting the second semantic information obtained by the Nth decoding of the first decoder into the first attention perception fusion module, performing an attention extraction operation, and obtaining a first attention feature;

[0107] S2512, upsampling the first attention feature, fusing it with the third region feature obtained by the Nth encoding of the second encoder, and inputting it into the second attention perception fusion module;

[0108] S2513, inputting the second semantic information obtained by the N-1th decoding of the first decoder into the second attention perception fusion module, performing an attention extraction operation, and obtaining a second attention feature;

[0109] S2514, when the Nth attention feature is obtained, it is fused with the third region feature obtained by the first encoding of the second encoder to obtain the attention feature.

[0110] It should be noted that the first segmentation channel of this embodiment is based on the preoperative thin-layer non-fat-suppressed MR image, which records the spinal structure. The second segmentation channel is based on the preoperative CT image. Since the CT image is obtained based on a planar scan, this embodiment uses an attention mechanism to enhance the spinal structure features of the MR image in the three-dimensional space to the planar vertebral structure, so that the third semantic information can multiple times enhance the vertebral structure of the same scale during the decoding process, thereby improving the accuracy of the third segmentation result.

[0111] like Figure 3 As shown, N=5, the second semantic information obtained by the first decoding is input into the first AFM, the first attention feature is extracted, and the first attention feature is upsampled and fused with the third region feature encoded for the fifth time, so that the features of the vertebral structure are strengthened in the third region feature, and multiple iterations are performed in this way.

[0112] In addition, in one embodiment, referring to Figure 4 The attention perception fusion module includes a channel attention module and a spatial attention module. The channel attention module includes the first convolution block, the first BN block and the first ReLu layer, the first maximum pooling layer, the first average pooling layer, the multi-layer perceptron and the first Sigmoid function. The spatial attention module includes the second convolution block, the second BN block, the third convolution block and the third BN block. Figure 2 , the attention extraction operation in steps S2511 and S2513 includes:

[0113] S71, determining a segmented image pair, where the segmented image pair includes a preoperative thin-layer non-fat-suppressed MR image and a preoperative CT image, or includes a first registered image and an intraoperative three-dimensional CBCT image;

[0114] S72, obtaining a first transformation feature by sequentially processing the second semantic information through a first convolution block, a first BN block, and a first ReLu layer;

[0115] S73, inputting the first transformed features into the first maximum pooling layer and the first average pooling layer respectively, fusing the outputs of the first maximum pooling layer and the first average pooling layer, and sequentially passing through a multi-layer perceptron and a first sigmoid function to obtain a channel attention map;

[0116] S74, sequentially passing through a second convolution block and a second BN block based on the third region feature to obtain a second transformed feature;

[0117] S75, input the first transformed feature into the spatial attention module, fuse the first transformed feature with the second transformed feature, and then pass through the third convolution block and the third BN block in sequence to obtain a spatial attention map;

[0118] S76, obtain attention features based on the spatial attention map and the channel attention map.

[0119] It should be noted that this embodiment uses the segmented image pair of a preoperative thin-layer non-fat-suppressed MR image and a preoperative CT image as an example to illustrate the principle. The segmented image pair of a first registration image and an intraoperative three-dimensional CBCT image is similar, and will not be repeated later.

[0120] It should be noted that AFM includes a channel attention module and a spatial attention module, which are used to extract regional feature information of feature structures. The second semantic information of this embodiment is the image feature of a thin layer non-fat-suppressed MR image, which is input into the first convolution block, the first BN block and the first ReLu layer. Figure 4 The Conv+BN+ReLu operation shown in the figure can perform a convolution operation on the second semantic information to achieve transformation, obtain the first transformed feature, and then input it into the first maximum pooling layer and the first average pooling layer respectively. After the corresponding pooling operation, the two pooling results are fused, and then the channel attention map is obtained through a multi-layer perceptron and the first Sigmoid function.

[0121] It should be noted that the third regional feature of this embodiment is the vertebral structure feature obtained after CT image encoding, which is input into the second convolution block and the second BN block to obtain the second transformation feature. The second convolution block and the second BN block are Figure 4 The Conv+BN operation shown in the figure is performed, and then the first transformed feature is fused with the second transformed feature and the Conv+BN operation is performed again to obtain the spatial attention map. Finally, the spatial attention map and the channel attention map are fused and the fused attention feature is obtained by the ReLu operation.

[0122] For example, the second semantic information is a thin layer non-fat-suppressed MR image feature, the third regional feature is a vertebral structure regional feature of the CT image, and the i-th thin layer non-fat-suppressed MR image feature After the channel attention module, the transformed non-fat-suppressed MR image features are output and channel attention map After the spatial attention module, the spatial attention map is output Where i is a positive integer, f a is the non-fat-suppressed MR image feature, is the transformed non-fat-suppressed MR image feature, A C is the channel attention map, A S is the spatial attention map. The vertebral structural features of the i-th CT image are For example, after conv and BN, the transformed CT image vertebral structure features are obtained. and Multiply point by point to get the attention perception feature and Add point by point and pass through the ReLu activation function to get the fused attention feature These features refer to the multi-structural features of the spine in MR images, with a focus on the enhanced regional features of the vertebral structure.

[0123] In the above example, the thin layer non-fat suppressed MR image feature f i a After convolution, batch normalization, and Relu activation function in the channel attention module, the transformed thin-layer non-fat-suppressed MR image features are obtained. Finally output channel attention map The formula is:

[0124]

[0125] Among them, Maxpool is the maximum pooling, AvgPool is the average pooling, and MLP is the multi-layer perceptron.

[0126] In the above example, Input the spatial attention module to generate a spatial attention map, which can be expressed as follows:

[0127]

[0128] The attention-aware feature is expressed as:

[0129] In addition, in one embodiment, referring to Figure 5 The multimodal image rigid-elastic hybrid registration model includes multiple affine matrix estimation modules, affine elastic fusion modules and local rigid constraint modules. Figure 2 The registration and alignment operations in step S30 and step S50 specifically include but are not limited to the following steps:

[0130] S81, inputting a plurality of images to be registered into a multiple affine matrix estimation module to obtain multi-scale image features and an initial deformation field of each image to be registered, wherein the images to be registered include a preoperative thin-layer fat-suppressed MR image, a preoperative thin-layer non-fat-suppressed MR image, and a preoperative CT image, or the images to be registered include a first registration image and an intraoperative three-dimensional CBCT image;

[0131] S82, based on the same image to be registered, inputting the corresponding multi-scale image features and the initial deformation field into the affine elastic fusion module, performing deformation field splicing based on the initial deformation field to obtain a rigid elastic deformation field, and outputting the initial registration image after fusing all the rigid elastic deformation fields;

[0132] S83, input the initial registration image and the registration mask into the local rigid constraint module, and correct the rigid elastic deformation field of the initial registration image through the local rigid constraint module to obtain the target registration image, wherein, when the image to be registered includes a preoperative spinal image set, the registration mask is a first mask, or, when the image to be registered includes an intraoperative three-dimensional CBCT image, the registration mask is a second mask.

[0133] It should be noted that this embodiment uses the images to be registered as the preoperative thin-layer fat-suppressed MR image, the preoperative thin-layer non-fat-suppressed MR image and the preoperative CT image as examples for illustrative explanation. The images to be registered include the first registration image and the intraoperative three-dimensional CBCT image, which can be obtained by the same logic and will not be repeated later.

[0134] It should be noted that the rigid-elastic hybrid registration model based on the spine multimodal image is composed of a rigid-elastic registration deep network framework (including multiple affine matrix estimation modules and affine elastic fusion modules) and a local rigid constraint module for the spine multimodal image. Its working principle is to first extract the multiscale image features of the image to be registered through the multiple affine matrix estimation module, and use the rigid transformation estimation parameters to obtain the initial deformation field of each scale, and then splice the initial deformation field of each structure obtained by the image features of each scale and the rigid transformation estimation through the affine elastic fusion module, calculate the rigid-elastic deformation field, and output the image after registration. In addition, the present embodiment adds a local rigid constraint module to provide strict local rigid constraints to the vertebral structure, so as to reduce the possibility of unreasonable elastic deformation of the vertebral structure during the spine image registration process.

[0135] In addition, in one embodiment, referring to Figure 5 The multiple affine matrix estimation module includes four consecutively arranged fourth convolution blocks, flat layers, and rigid transformation estimation modules. The rigid transformation estimation module includes multiple dense layers and multiple affine matrices. The dense layers correspond uniquely to the affine matrices. Figure 2 Step S81 also specifically includes but is not limited to the following steps:

[0136] S811, subjecting the image to be registered to convolution processing of each fourth convolution block in sequence to obtain a multi-scale image feature;

[0137] S812, after the multi-scale image features are expanded based on each scale through the flattening layer, the image features of each scale are input into a dense layer of the rigid transformation estimation module, and the rigid deformation field is calculated based on the preset rigid transformation estimation parameters;

[0138] S813, performing weighted averaging on all rigid deformation fields to obtain an initial deformation field.

[0139] It should be noted that the multiple affine matrix estimation module is used to extract the multi-scale image features of the image to be registered and simultaneously estimate the affine matrix of each structure and the overall structure affine matrix. It consists of four fourth convolution blocks, a flat layer and N dense layers (N is the number of multiple structures, corresponding to L4 vertebra, L4 / L5 intervertebral disc, L5 vertebra, L4 nerve root, L5 nerve root, dura mater, ilium, iliac artery and branches, inferior vena cava and branches, psoas major muscle, skin and lesion targets, respectively N1, N2, N.3..., N9, N10, N11). Each convolution block consists of a 3D convolution layer and a LeakyReLU layer. These four convolution blocks are used as image feature extraction modules to extract multi-scale image coding features of the moving image Imoving to be registered and the fixed image Ifixed to be registered (Fu represents the smallest high-level image feature, and F1 has the largest low-level image representation).

[0140] Exemplarily, the initial deformation field obtained in this embodiment is specifically implemented as follows: a multi-resolution network is constructed using a multi-layer convolution + pooling structure; the multimodal image to be registered is input into the network for multi-scale image feature extraction. A flat layer and N pool layers constitute a rigid transformation estimation module, which is used to calculate the initial deformation field of each scale. The function of the flat layer is to project high-level image features to N affine matrices. N dense layers are used to predict the affine matrix Tn of each structure, where n is a positive integer. The affine matrix of the overall structure is obtained by global optimization. N affine dense layers convert the IV affine matrix into N features, and connect them with the u scale to obtain the mixed feature Hu for use in the subsequent affine elastic fusion module. In addition, the registration mask and Smoving of each structure of the moving image to be registered, as well as the registration mask and Sfixed of each structure of the moving image to be registered will be used to weakly supervise the multiple affine matrix estimation module to obtain a more accurate affine matrix for each structure. The specific implementation is as follows: ① Use a fully connected network to estimate the rigid transformation parameters of each structure, and the cost function uses the DICE cost of the segmented area corresponding to each rigid area; calculate the rigid deformation field of N features through the rigid transformation parameters; perform distance-related weighted averaging on the rigid deformation field of N features to obtain the initial deformation field with the lowest resolution; calculate the initial deformation field of each structure through upsampling for subsequent rigid / property deformation field estimation.

[0141] In addition, in one embodiment, referring to Figure 5 The affine elastic fusion module includes 4 combination units, three sixth convolution blocks and integrated blocks. The combination unit includes an upsampling layer and a fifth convolution block. Figure 2 Step S82 also specifically includes but is not limited to the following steps:

[0142] S821, processing the image features of each scale through the four combination units in sequence to obtain the structural image features of each scale;

[0143] S822, combining the structural image features of the corresponding scale with the initial deformation field to obtain a rigid elastic deformation field;

[0144] S823, input each rigid elastic deformation field into the sixth convolution block for convolution processing, and then input it into the integrated block for fusion to obtain an initial registration image.

[0145] It should be noted that the affine elastic fusion module consists of 4 combination units, three sixth convolution blocks and one integrated block. The specific structure is referenced Figure 5 As shown. The affine elastic fusion module is used to estimate the rigid elastic deformation field. For each combination unit, the input is the encoding feature Fu and the mixed feature u of the u-1 scale, and the output is the combined feature Ou-1 of the u-1 scale. Three convolution blocks and one integrated block are used to fuse multi-resolution rigid elastic information and estimate the rigid elastic deformation field. In addition, the registration mask and Smoving of each structure of the moving image to be registered, as well as the registration mask and Sfixed of each structure of the moving image to be registered are used to weakly supervise the affine elastic fusion module, aiming to obtain the optimal rigid elastic deformation field. Specific implementation: multi-layer convolution + upsampling structure to construct a multi-resolution network; splice the image features of each structure output by the feature extraction module with the initial deformation field of each structure output by the rigid transformation estimation module, and output the rigid elastic deformation field; use the spatial transformation network to calculate the registered image, and calculate the mutual information cost and the DICE cost of the segmented area corresponding to each rigid area to supervise the network.

[0146] In addition, in one embodiment, referring to Figure 2 , refer to Figure 2 Step S83 also specifically includes but is not limited to the following steps:

[0147] S831, performing least square regression calculation on the rigid elastic deformation field based on the registration mask to obtain a reference deformation field;

[0148] S832, constructing a rigid constraint cost function based on an error between the rigid elastic deformation field and the reference deformation field;

[0149] S833, based on the rigid constraint cost function, the initial registration image is modified to obtain the target registration image.

[0150] It should be noted that this embodiment is applied to the specific scenario of lumbar intervertebral disc. In the registration process, the vertebral structure is prone to unreasonable elastic deformation, which affects the accuracy of the registered image. Therefore, this embodiment introduces a local rigid constraint module to ensure the reasonable rigid deformation of the bone structure to avoid unreasonable elastic deformation of the vertebral structure during the registration process of the spine image. First, the registration mask of the segmented image is used to perform least squares regression on the rigid structural area of ​​each vertebral structure (the output of the rigid elastic deformation transformation estimation module) to obtain its corresponding rigid deformation field, and then calculate the error between the original rigid elastic deformation field and the obtained ideal rigid deformation field to form a local rigid constraint cost function, perform error feedback, and correct the network parameters.

[0151] like Figure 6 As shown, Figure 6 The present invention also provides a percutaneous lumbar disc puncture surgical navigation device based on multimodal images, comprising:

[0152] The processor 601 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0153] The memory 602 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 602, and the processor 601 calls and executes the percutaneous lumbar disc puncture surgical navigation method based on multimodal images in the embodiments of this application;

[0154] Input / output interface 603, used to implement information input and output;

[0155] Communication interface 604, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0156] Bus 605 , which transmits information between various components of the device (e.g., processor 601 , memory 602 , input / output interface 603 , and communication interface 604 );

[0157] The processor 601 , the memory 602 , the input / output interface 603 and the communication interface 604 are connected to each other in communication within the device via a bus 605 .

[0158] An embodiment of the present application also provides an electronic device, including the multimodal image-based percutaneous lumbar disc puncture surgical navigation device as described above.

[0159] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned percutaneous lumbar disc puncture surgical navigation method based on multimodal images is implemented.

[0160] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and are implemented to be located in one place, or may also be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0161] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically include computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0162] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A multimodal image-based percutaneous lumbar disc puncture surgical navigation method, characterized in that: Applied to a surgical navigation system, the surgical navigation system is pre-configured with a multi-contrast multi-modal image segmentation model and a multi-modal image rigid-elastic hybrid registration model, and the multi-modal image-based percutaneous lumbar disc puncture surgical navigation method includes: Acquire a preoperative spinal image set, wherein the preoperative spinal image set includes a preoperative thin-layer fat-suppressed MR image, a preoperative thin-layer non-fat-suppressed MR image, and a preoperative CT image, wherein the preoperative thin-layer fat-suppressed MR image is acquired based on a preset fat-suppressed sequence, and the preoperative thin-layer non-fat-suppressed MR image is acquired based on a preset non-fat-suppressed sequence; Inputting the preoperative spinal image set into the multi-contrast multi-modal image segmentation model for image segmentation, and fusing the image segmentation results to obtain a first mask, wherein the first mask is used to indicate a spinal structure or a preoperative lesion target; Inputting the preoperative spinal image set and the first mask into a multimodal image rigid-elastic hybrid registration model, performing a registration and alignment operation, and obtaining a first registered image, wherein the first registered image includes the preoperative thin-layer fat-suppressed MR image, the preoperative thin-layer non-fat-suppressed MR image, and the preoperative CT image that are registered with each other; Acquire an intraoperative three-dimensional CBCT image, and input the intraoperative three-dimensional CBCT image into the multi-contrast multi-modal image segmentation model for image segmentation to obtain a second mask, wherein the second mask is used to indicate a vertebral structure; Inputting the second mask, the intraoperative three-dimensional CBCT image and the first registered image into the multimodal image rigid-elastic hybrid registration model, performing the registration and alignment operation, and obtaining a second registered image, wherein the second registered image includes the first registered image and the intraoperative three-dimensional CBCT image that are registered with each other; A three-dimensional visualization model of the spine is constructed based on the second registered image, and when a target lesion target point marked on the three-dimensional visualization model of the spine is acquired, a puncture navigation path is generated based on the target lesion target point.

2. The multimodal image-based percutaneous lumbar disc puncture surgical navigation method according to claim 1, characterized in that: The multi-contrast multi-modal image segmentation model includes a first segmentation branch and a second segmentation branch, the first segmentation branch includes a first encoder and a first decoder, the second segmentation branch includes a second encoder and a second decoder, the preoperative spinal image set is input into the multi-contrast multi-modal image segmentation model for image segmentation, and the image segmentation results are fused to obtain a first mask, including: Inputting the preoperative thin layer fat-suppressed MR image into the first segmentation branch, extracting a first regional feature of the preoperative thin layer fat-suppressed MR image through the first encoder, extracting first semantic information of the first regional feature through the first decoder, and determining a first segmentation result based on the first semantic information, wherein the first regional feature is used to indicate the spinal structure or the preoperative lesion target; Inputting the preoperative thin-layer non-fat-suppressed MR image into the first segmentation branch, and inputting the CT image into the second segmentation branch; Extracting a second regional feature of the preoperative thin-layer non-fat-suppressed MR image by the first encoder, extracting second semantic information of the second regional feature by the first decoder, and determining a second segmentation result based on the second semantic information, wherein the second regional feature is used to indicate the spinal structure or the preoperative lesion target; Inputting the second semantic information into the second segmentation branch, and extracting a third region feature of the CT image through the second encoder; Performing comparative learning based on the second semantic information and the third region feature to obtain a third segmentation result; The first segmentation result, the second segmentation result, and the third segmentation result are fused to obtain the first mask.

3. The multimodal image-based percutaneous lumbar disc puncture surgical navigation method according to claim 2, characterized in that: The second decoder includes a plurality of attention perception fusion modules, and the third segmentation result is obtained by performing comparative learning based on the second semantic information and the third region feature, including: Inputting the second semantic information into the attention perception fusion module to obtain an attention feature, and upsampling the attention feature and fusing it into the third region feature; extracting third semantic information of the third region feature, and determining a third segmentation result based on the third semantic information and a preset contrast loss function, wherein the third region feature is used to indicate a vertebral structure; Among them, the contrast loss function satisfies the following conditions: loss seg =loss a +λ b loss b +λ con loss con , loss a is the cross entropy loss function of the first segmentation branch, loss b is the cross entropy loss function of the second segmentation branch, λ b and λ con is the preset magnification factor, loss con is the contrast loss function, loss seg is the total loss function of the multi-contrast multimodal image segmentation model.

4. The multimodal image-based percutaneous lumbar disc puncture surgical navigation method according to claim 3, characterized in that: The encoding times of the first encoder and the second encoder are N times, the decoding times of the first decoder and the second decoder are N times, the number of the attention-aware fusion modules is N, N is a positive integer greater than 2, the inputting of the second semantic information into the attention-aware fusion module to obtain the attention feature, and the upsampling of the attention feature and the fusion of the attention feature into the third region feature include: Inputting the second semantic information obtained by the Nth decoding of the first decoder into the first attention perception fusion module, performing an attention extraction operation, and obtaining the first attention feature; Upsampling the first attention feature, fusing it with the third region feature obtained by the Nth encoding of the second encoder, and then inputting it into the second attention perception fusion module; Inputting the second semantic information obtained by the N-1th decoding of the first decoder into the second attention perception fusion module, performing the attention extraction operation, and obtaining the second attention feature; When the Nth attention feature is obtained, it is fused with the third region feature obtained by the first encoding of the second encoder to obtain the attention feature.

5. The multimodal image-based percutaneous lumbar disc puncture surgical navigation method according to claim 4, characterized in that: The attention perception fusion module includes a channel attention module and a spatial attention module, the channel attention module includes a first convolution block, a first BN block and a first ReLu layer, a first maximum pooling layer, a first average pooling layer, a multi-layer perceptron and a first Sigmoid function, the spatial attention module includes a second convolution block, a second BN block, a third convolution block and a third BN block, and the attention extraction operation includes: Determine a segmented image pair, wherein the segmented image pair includes the preoperative thin-layer non-fat-suppressed MR image and the preoperative CT image, or includes the first registered image and the intraoperative three-dimensional CBCT image; Obtaining a first transformation feature based on the second semantic information by sequentially processing the first convolution block, the first BN block, and the first ReLu layer; Inputting the first transformed features into the first maximum pooling layer and the first average pooling layer respectively, fusing the outputs of the first maximum pooling layer and the first average pooling layer, and sequentially passing through the multi-layer perceptron and the first Sigmoid function to obtain a channel attention map; Based on the third region feature, sequentially passing through the second convolution block and the second BN block, obtaining a second transformation feature; Inputting the first transformation feature into the spatial attention module, fusing the first transformation feature with the second transformation feature, and then sequentially passing through the third convolution block and the third BN block to obtain a spatial attention map; The attention feature is obtained based on the spatial attention map and the channel attention map.

6. The multimodal image-based percutaneous lumbar disc puncture surgical navigation method according to claim 1, characterized in that: The multimodal image rigid-elastic hybrid registration model includes a multiple affine matrix estimation module, an affine elastic fusion module and a local rigid constraint module. The registration and alignment operation includes: Inputting a plurality of images to be registered into the multiple affine matrix estimation module to obtain multi-scale image features and initial deformation fields of each of the images to be registered, wherein the images to be registered include the preoperative thin-layer fat-suppressed MR image, the preoperative thin-layer non-fat-suppressed MR image and the preoperative CT image, or the images to be registered include the first registration image and the intraoperative three-dimensional CBCT image; Based on the same image to be registered, the corresponding multi-scale image features and the initial deformation field are input into the affine elastic fusion module, deformation field splicing is performed based on the initial deformation field to obtain a rigid elastic deformation field, and all the rigid elastic deformation fields are fused to output an initial registration image; The initial registration image and the registration mask are input into the local rigid constraint module, and the rigid elastic deformation field of the initial registration image is corrected by the local rigid constraint module to obtain a target registration image, wherein, when the image to be registered includes the preoperative spinal image set, the registration mask is the first mask, or, when the image to be registered includes the intraoperative three-dimensional CBCT image, the registration mask is the second mask.

7. The multimodal image-based percutaneous lumbar disc puncture surgical navigation method according to claim 6, characterized in that: The multiple affine matrix estimation module includes four fourth convolution blocks, a flat layer, and a rigid transformation estimation module arranged in series. The rigid transformation estimation module includes multiple dense layers and multiple affine matrices. The dense layers uniquely correspond to the affine matrices. The multi-scale image features and initial deformation fields of each of the images to be registered are obtained, including: Subjecting the image to be registered to convolution processing of each of the fourth convolution blocks in sequence to obtain the multi-scale image feature; After the multi-scale image features are expanded based on each scale through the flattening layer, the image features of each scale are input into one of the dense layers of the rigid transformation estimation module, and the rigid deformation field is calculated based on the preset rigid transformation estimation parameters; The initial deformation field is obtained by taking a weighted average of all the rigid deformation fields.

8. The multimodal image-based percutaneous lumbar disc puncture surgical navigation method according to claim 7, characterized in that: The affine elastic fusion module includes 4 combination units, three sixth convolution blocks and an integrated block, the combination unit includes an upsampling layer and a fifth convolution block, the deformation field is spliced ​​based on the initial deformation field to obtain a rigid elastic deformation field, and all the rigid elastic deformation fields are fused to output an initial registration image, including: The image features of each scale are processed by the four combination units in sequence to obtain the structural image features of each scale; Splicing the structural image features of the corresponding scale with the initial deformation field to obtain a rigid elastic deformation field; After each of the rigid-elastic deformation fields is input into the sixth convolution block for convolution processing, it is input into the integrated block for fusion to obtain the initial registration image.

9. The multimodal image-based percutaneous lumbar disc puncture surgical navigation method according to claim 8, characterized in that: The method of correcting the rigid elastic deformation field of the initial registration image by the local rigid constraint module to obtain a target registration image includes: performing least square regression calculation on the rigid elastic deformation field based on the registration mask to obtain a reference deformation field; constructing a rigid constraint cost function based on an error between the rigid elastic deformation field and the reference deformation field; The target registration image is obtained by modifying the initial registration image based on the rigid constraint cost function.

10. A navigation device for percutaneous lumbar disc puncture surgery based on multimodal images, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to perform the multi-modal image-based percutaneous lumbar disc puncture surgical navigation method as described in any one of claims 1 to 9.

Images