Operation path planning method and device based on multi-modal image registration

Abstract

The invention provides a surgical path planning method and device based on multi-modal image registration, and the method comprises the steps: obtaining multi-modal image data of a target brain structure, the multi-modal image data comprises a plurality of images captured for the target brain structure, and the capturing modes of the plurality of images are different; performing image registration on the plurality of images to obtain registered multi-modal image data; receiving a target position input by a user for the registered multi-modal image data; and based on the registered multi-modal image data, planning a path from the outside of the target craniocerebral structure to the target position to obtain a planned operation path. According to the method and the device, the problems that preoperative path planning is time-consuming, error-prone and low in safety can be solved, the surgical path can be more accurately planned, the precision and the safety of path planning are improved, and help is provided for preoperative preparation of operators.

Description

Technical Field

[0001] The present disclosure relates to the fields of medical image processing and computer-assisted surgery planning, and in particular to a surgical path planning method and device based on multimodal image registration. Background Art

[0002] With the development of computer technology, through computer analysis and processing of medical images, surgical paths can be simulated and planned before surgery. With the help of automated path planning, auxiliary reference information can be provided for surgery, reducing the computational workload of surgical operators in the preoperative preparation stage.

[0003] Specifically, in related technologies, it is generally dependent on manual planning by personnel. It is difficult to fully consider the complex anatomical structure and functional area distribution before surgery. The planning process is time-consuming and prone to errors. In addition, in some automated path planning schemes, only single-modality image data can be used, which cannot provide comprehensive anatomical and functional information, affecting the accuracy and safety of path planning. Summary of the invention

[0004] The present disclosure provides a surgical path planning method and device based on multimodal image registration, in order to at least solve the problems in the related art that preoperative manual path planning is time-consuming and error-prone, and that automated path planning has low accuracy and safety. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of the present disclosure, a surgical path planning method based on multimodal image registration is provided, the surgical path planning method comprising: acquiring multimodal image data of a target cranial and brain structure, wherein the multimodal image data comprises a plurality of images captured for the target cranial and brain structure, and the plurality of images are captured in different ways; performing image registration on the plurality of images to obtain registered multimodal image data; receiving a target position input by a user for the registered multimodal image data; and planning a path from the outside of the target cranial and brain structure to the target position based on the registered multimodal image data to obtain a planned surgical path.

[0006] Optionally, the planned surgical path is obtained in the following manner: under preset optimization conditions, based on the registered multimodal imaging data, determining the cranial entry position on the outer surface of the target cranial structure; taking a straight line path from the cranial entry position to the target position as the planned surgical path, wherein the optimization conditions include at least one of the following items: the path between the cranial entry position and the target position is the shortest; the direction of the path between the cranial entry position and the target position is perpendicular to the skull surface of the target cranial structure; the skull thickness at the cranial entry position is the smallest; the distance between the path between the cranial entry position and the target position and a preset avoidance area is the largest; the distance between the path between the cranial entry position and the target position and a preset approach area is the smallest; the depth of the path between the cranial entry position and the target position through the sulcus area is the smallest.

[0007] Optionally, there are multiple optimization conditions, wherein under the preset optimization conditions, based on the registered multimodal image data, the cranial entry position on the outer surface of the target cranial structure is determined, including: constructing an objective function based on each optimization condition and a preset weight corresponding to each optimization condition, wherein the objective function represents the weighted sum of the scores of the cranial entry position under each optimization condition using the preset weight; solving the optimal solution of the objective function for the position within the preset cranial entry position area on the outer surface of the target cranial structure; and determining the position corresponding to the optimal solution as the cranial entry position.

[0008] Optionally, the distance between the path from the cranial entry position to the target position and the preset avoidance area is determined in the following manner: multiple intermediate points on the straight path from the cranial entry position to the target position are determined; and the minimum distance between the distances from each intermediate point to the avoidance area is used as the distance between the path and the avoidance area, wherein the distance between the path from the cranial entry position to the target position and the preset pathway area is determined in the following manner: multiple intermediate points on the straight path from the cranial entry position to the target position are determined; and the minimum distance between the distances from each intermediate point to the pathway area is used as the distance between the path and the pathway area.

[0009] Optionally, the multiple images include CT images, MRI images and vascular images, wherein the surgical path planning method further includes: determining the vascular surface mesh of the target cranial and brain structure based on the vascular image, and determining the avoidance area based on the vascular surface mesh; and / or determining the skull surface mesh of the target cranial and brain structure based on the CT image, and determining the skull surface and the skull thickness based on the skull surface mesh; and / or determining the gray and white matter surface mesh of the target cranial and brain structure based on the MRI image, and determining the depth of the sulcus area based on the gray and white matter surface mesh.

[0010] Optionally, the multiple images include CT images, MRI images and vascular images, wherein the registered multimodal image data is obtained by: performing a first image registration on the CT image and the MRI image to obtain a first registration relationship; performing the first image registration on the vascular image and the MRI image to obtain a second registration relationship; performing a second image registration on the MRI image and a reference brain atlas template to obtain a third registration relationship; based on the first registration relationship, the second registration relationship and the third registration relationship, the registered multimodal image data is obtained, wherein the first image registration is performed by: determining the rotational transformation between the images to be registered The rotation transformation amount and the translation transformation amount are adjusted to maximize the similarity between the images to be registered, and the registration relationship between the images to be registered is obtained; wherein the second image registration is performed in the following manner: determining a first transformation relationship that maximizes the similarity between the MRI image and the reference brain map template; determining a second transformation relationship that aligns each voxel between the MRI image and the reference brain map template, and determining the third registration relationship based on the first transformation relationship and the second transformation relationship, wherein the third registration relationship represents the correspondence between the MRI image and each brain region of the reference brain map template.

[0011] According to a second aspect of the present disclosure, a surgical path planning device based on multimodal image registration is provided, the surgical path planning device comprising: an acquisition unit, configured to acquire multimodal image data of a target cranial and brain structure, wherein the multimodal image data comprises a plurality of images captured for the target cranial and brain structure, and the plurality of images are captured in different ways; a registration unit, configured to perform image registration on the plurality of images to obtain registered multimodal image data; a receiving unit, configured to receive a target position input by a user for the registered multimodal image data; and a planning unit, configured to plan a path from the outside of the target cranial and brain structure to the target position based on the registered multimodal image data to obtain a planned surgical path.

[0012] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor, wherein when the instructions executable by the processor are executed by the processor, the processor is prompted to execute the surgical path planning method based on multimodal image registration according to the present disclosure.

[0013] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the surgical path planning method based on multimodal image registration according to the present disclosure.

[0014] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising computer executable instructions, which, when executed by at least one processor, implement the surgical path planning method based on multimodal image registration according to the present disclosure.

[0015] The technical solution provided by the present disclosure brings at least the following beneficial effects:

[0016] According to the surgical path planning scheme based on multimodal image registration disclosed in the present invention, by registering the multimodal image data to obtain registered multimodal image data, the path from the outside of the target cranial structure to the target position can be planned based on the registered multimodal image data and the target position input by the user. In this way, automated surgical path planning can be achieved, and because the multimodal image data can contain more information about the target cranial structure, the surgical path can be planned more accurately, thereby improving the accuracy and safety of path planning and providing assistance to the operator's preoperative preparation.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0019] Figure 1 It is a flowchart of a surgical path planning method based on multimodal image registration according to an exemplary embodiment of the present disclosure.

[0020] Figure 2 It is a flowchart of the steps of obtaining registered multimodal image data in a surgical path planning method based on multimodal image registration according to an exemplary embodiment of the present disclosure.

[0021] Figure 3 It is a flowchart of the steps of performing the second image registration in the surgical path planning method based on multimodal image registration according to an exemplary embodiment of the present disclosure.

[0022] Figure 4It is a flowchart of the steps of determining the cranial entry position in the surgical path planning method based on multimodal image registration according to an exemplary embodiment of the present disclosure.

[0023] Figure 5 It is a schematic diagram of an example of a preset cranial entry position area in a surgical path planning method based on multimodal image registration according to an exemplary embodiment of the present disclosure.

[0024] Figure 6 It is a flowchart of an application example of image registration in a surgical path planning method based on multimodal image registration according to an exemplary embodiment of the present disclosure.

[0025] Figure 7 It is a flowchart of an application example of planning a path in a surgical path planning method based on multimodal image registration according to an exemplary embodiment of the present disclosure.

[0026] Figure 8 It is a schematic block diagram of a surgical path planning device based on multi-modal image registration according to an exemplary embodiment of the present disclosure.

[0027] Fig. 9 is a schematic block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the present application will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0030] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0031] As mentioned above, in the related art, there are problems that preoperative manual path planning is time-consuming and error-prone, and the accuracy and safety of automated path planning are low.

[0032] Generally speaking, it may be difficult to directly integrate multimodal imaging data such as CT, MRI, and TOF-MRA in existing image processing software, which is complicated to operate and prone to errors.

[0033] In addition, manual path planning is not only time-consuming and labor-intensive, but also fails to fully and comprehensively consider safety factors such as vascular avoidance, which increases surgical risks.

[0034] In addition, some existing path planning algorithms have high computational complexity and are difficult to achieve real-time interaction, which affects clinical and scientific research applications.

[0035] In view of the above problems, exemplary embodiments of the present disclosure propose a surgical path planning method and device based on multimodal image registration, an electronic device, a computer-readable storage medium, and a computer program product, which can solve or at least alleviate the above problems.

[0036] In a first aspect of an exemplary embodiment of the present disclosure, a surgical path planning method based on multimodal image registration is provided.

[0037] The surgical path planning method according to the exemplary embodiment of the present disclosure can be applied to scenarios where users interact with surgical path planning software. For example, the surgical path planning software can be loaded on a user terminal, and the user can input surgical path planning instructions on the user terminal. The user terminal can obtain the final surgical path by executing the surgical path planning method according to the exemplary embodiment of the present disclosure.

[0038] The user terminal can be a tablet computer, a laptop computer, a digital assistant, a wearable device, etc. However, the implementation scenario of the above method is only an example scenario. The surgical path planning method according to the exemplary embodiment of the present disclosure can also be applied to other application scenarios, for example, the user can request the server to perform surgical path planning, and the server can execute the method. In addition, the server can be an independent server, a server cluster, a cloud computing platform or a virtualization center.

[0039] In addition, the surgical path planning method according to the exemplary embodiment of the present disclosure can be applied to theoretical neuroscience research on large animals such as dogs, pigs, monkeys, etc., such as drawing brain maps, studying the electrophysiological mechanisms of neural circuits, etc.; it can also be applied to preoperative path planning of human neurosurgery such as intracranial surgery to provide auxiliary reference information for clinical decision-making; it can also be used as a medical education and training tool to help doctors and researchers learn and simulate intracranial surgical path planning. In addition, the application scenario of the surgical path planning method is flexible and can be applied according to actual needs. For example, it can also be applied to or included in other planning methods, such as planning schemes for drug delivery systems and planning schemes for electrode implantation. In addition, the image registration method in this method can also be applied to image analysis and processing fields that require precise positioning, such as brain tumor surgery and deep brain stimulation.

[0040] The following will refer to Figures 1 to 7 An example of a surgical path planning method based on multimodal image registration according to an embodiment of the present disclosure is described. Figure 1 As shown, the surgical path planning method may include the following steps:

[0041] In step S110, multimodal image data of the target cranial brain structure may be acquired.

[0042] Here, the multimodal image data may include multiple images captured for the target cranial brain structure, where the multiple images are captured in different ways.

[0043] As an example, the plurality of images may include CT (Computed Tomography) images, MRI (Magnetic Resonance Imaging) images, and vascular images.

[0044] Here, CT images are images of the inside of tissues or structures such as skulls and soft tissues obtained by X-rays; MRI images are images of the inside of tissues or structures obtained by using the nuclear magnetic resonance phenomenon, such as structural MRI (Structural Magnetic Resonance Imaging, sMRI) images, which can provide high-resolution brain tissue structure information; vascular images can be structural images of blood vessels, such as but not limited to TOF-MRA (Time-of-Flight Magnetic Resonance Angiography) images, which can obtain three-dimensional structures such as blood vessels in the skull.

[0045] The above-mentioned image data can be obtained by capturing an image of the head of the target object. Here, the target object can be, for example, a human body or an animal body. CT images can cover the face and the top of the head, which can provide structural information of the skull and soft tissue; MRI images can be T1-weighted or T2-weighted images, which can provide structural information of brain tissue; vascular images such as TOF-MRA can be three-dimensional imaging of vascular structures.

[0046] As an example, before performing image registration, the multimodal image data may be preprocessed, which may include image direction rotation, image cropping, and the like.

[0047] For example, the method may further include: receiving a rotation operation of a user on any image in the multimodal image data; and rotating the image according to a rotation center and a rotation angle specified by the rotation operation.

[0048] In the embodiments of the present disclosure, the user can manually rotate the CT image, MRI image and / or vascular image to ensure that each image is consistent in the anatomical direction (e.g., the three anatomical directions of head-tail, upper-lower and left-right). Here, the rotation operation can be achieved by manually adjusting the image, or by inputting the rotation center, rotation axis and / or rotation angle.

[0049] For example, assuming that the rotation center received from the user is c=[c x ,c y ,c z ] T , the rotation axis is φ, the rotation angle is θ, where [c x ,c y ,c z ] represents the three-dimensional coordinates of the rotation center, T represents transpose, then the rotation matrix can be expressed as R φ(θ), the specific form of the rotation matrix can be obtained by, for example, the Logriggs formula. In this case, assuming that the original direction matrix of the rotated image is R and the origin vector is o, the direction matrix R′ of the image after rotation can be expressed by the following formula (1):

[0050] R′=R φ (θ)R (1)

[0051] The origin o′ of the image after rotation can be expressed by the following formula (2):

[0052] o′=R φ (θ)(oc)+c (2)

[0053] Here, o′ may represent a rotated origin vector, and similarly, in this document, symbols in bold may represent vectors.

[0054] Through the above-mentioned image rotation process, the processed images can meet the standards of neuroscience research and ensure the accuracy of subsequent image registration. In addition, this image rotation method ensures that no interpolation errors are introduced during the rotation process, thereby maintaining the accuracy of the original image.

[0055] In addition, in the above process, image rotation can be achieved by adjusting the direction matrix and the origin, and fast and accurate rotation operations can be achieved without resampling the image, which can maintain the high precision of the original image, so as to improve the accuracy of subsequent path planning.

[0056] For another example, the method may further include: receiving a cropping operation from a user for any image in the multimodal image data; and cropping the image according to a cropping method specified by the cropping operation. The cropping method may include, for example, a cropping size, a cropping shape, and the like.

[0057] Specifically, the user can manually crop the CT image, MRI image and / or vascular image according to actual needs. The purpose of cropping can be, for example, to preserve the skull and brain tissue, thereby reducing the computational burden of subsequent registration and focusing on key brain areas.

[0058] In step S120, image registration may be performed on the multiple images to obtain registered multi-modal image data.

[0059] In this step, the acquired original image or the image pre-processed by the above rotation, cropping, etc. can be registered to obtain registered multimodal image data. As an example, registration can refer to finding a mapping relationship between images so that each point on one image has a unique point corresponding to it on another image, and the two points should correspond to the same anatomical position.

[0060] As an example, in an example where the plurality of images include CT images, MRI images, and vascular images, such as Figure 2 As shown, the registered multimodal image data can be obtained in the following ways:

[0061] In step S210, a first image registration may be performed on the CT image and the MRI image to obtain a first registration relationship.

[0062] In step S220, a first image registration may be performed on the vascular image and the MRI image to obtain a second registration relationship.

[0063] In the above steps, for example, the first image registration can be performed in the following manner: determining the rotational transformation amount and the translational transformation amount between the images to be registered, adjusting the rotational transformation amount and the translational transformation amount to maximize the similarity between the images to be registered, and obtaining the registration relationship between the images to be registered.

[0064] Specifically, in order to ensure the spatial alignment of images of different modalities, the CT image can be registered with the MRI image (such as the sMRI image), and the vascular image can be registered with the MRI image (such as the sMRI image). Here, the optimal transformation can be found by optimizing the similarity measure S so that the two images A and B achieve the best spatial consistency. The rotation transformation amount can be represented by a rotation matrix, and the translation transformation amount can be represented by a translation vector. For example, the first image registration can be expressed as the following formula (3):

[0065]

[0066] Among them, R A→B represents the rotation matrix from image A to B, t A→B Represents the translation vector from image A to B, and max represents the translation vector from image A to image B by adjusting the rotation matrix R A→B and the translation vector t A→B Get the maximum value of the similarity measure S. Here, the similarity measure S may be, for example, but not limited to, mutual information.

[0067] The above registration process can be implemented, for example, by various rigid body registration algorithms. Specifically, for the above formula (3), the transformation that maximizes the similarity measure can be found by continuously adjusting the rotation matrix R and the translation vector t. In this way, the optimal rotation matrix R and the translation vector t can be directly applied to the image's direction matrix and origin coordinates, avoiding resampling of the image data, thereby effectively preventing detail loss. In order to speed up the registration process, the image pyramid technology can be used to gradually optimize the registration results from low resolution to high resolution. Each level contains different downsampling factors and smoothing parameters to balance speed and accuracy.

[0068] In step S230, a second image registration may be performed on the MRI image and the reference brain atlas template to obtain a third registration relationship.

[0069] Here, the brain map can be, for example, a standard brain anatomy and functional partitioning model that can be mapped to individual images for locating and marking brain regions. The reference brain map template can be a preset standard brain map that can characterize the standard brain map of the group to which the target object under study belongs. By aligning the individual MRI image of the target object's cranial brain structure with the reference brain map target, it can be ensured that the anatomical labels of the brain map can be accurately mapped to the individual space, thereby accurately dividing the brain regions in the MRI image.

[0070] As an example, Figure 3 As shown, the second image registration can be performed in the following way:

[0071] In step S310, a first transformation relationship that maximizes the similarity between the MRI image and the reference brain atlas template may be determined.

[0072] Here, the first transformation relationship may be, for example, an affine transformation matrix. Specifically, the optimal affine transformation matrix T may be solved by affine transformation. affjne , so that the similarity measure between the fixed image (such as the sMRI image of the target object) and the moving image (such as the reference brain atlas template) is maximized. The affine transformation matrix T affine The form of can be expressed, for example, by the following formula (4):

[0073]

[0074] Where R1 can represent an n×n (e.g., 3×3) rotation / shear / scaling matrix, t1=[t x ,t y ,t z ] T can represent the translation vector, where [t x ,t y ,t z ] represents the coordinates of the translation vector on the three coordinate axes, and 0 represents [0,0,0].

[0075] However, the transformation method that maximizes the similarity between the MRI image and the reference brain map template is not limited to the above-mentioned affine transformation method. Accordingly, the first transformation relationship is not limited to the above-mentioned affine transformation matrix. Other transformation methods and corresponding transformation relationships may also be selected.

[0076] In step S320, a second transformation relationship for aligning each voxel between the MRI image and the reference brain atlas template may be determined.

[0077] Here, the first transformation relationship may be an overall registration between the MRI image and the reference brain atlas template (e.g., image-level registration), and the second transformation relationship may be a detailed registration between the MRI image and the reference brain atlas template (e.g., voxel-level registration). Specifically, after obtaining the first transformation relationship, the details of the image may be further aligned.

[0078] As an example, the second transformation relationship can be a nonlinear deformation field. Specifically, after completing the affine transformation, the details between the MRI image and the reference brain atlas template can be aligned by estimating the nonlinear deformation field. As shown in the following equation (5), the deformation field can use the displacement vector field D(x) to describe the nonlinear transformation of each voxel:

[0079] x ′ =x+D(x) (5)

[0080] Where x represents the coordinates of any voxel on the moving image, x ′ Represents the coordinates of any voxel on the registered moving image, and the displacement vector field D represents the coordinate offset of x in the anatomical direction, which can be determined by an optimization algorithm such as the Demons algorithm. Here, the deformation field can gradually adjust the displacement of each voxel by an optimization algorithm such as the Demons algorithm to align the fixed image (e.g., the sMRI image of the target object) and the moving image (e.g., the reference brain map template) in detail. Although the process of registering the MRI image as a fixed image and the reference brain map template as a moving image is described here, the embodiments of the present disclosure are not limited to this, and the reference brain map template can also be used as a fixed image and the MRI image as a moving image for registration, as long as the transformation relationship between the two can be determined.

[0081] In step S330, a third registration relationship may be determined based on the first transformation relationship and the second transformation relationship.

[0082] Here, the third registration relationship can represent the correspondence between the MRI image and each brain region of the reference brain atlas template. Specifically, through the above-mentioned double registration process, the first transformation relationship and the second transformation relationship between the template (or group) and the individual space can be used to obtain the transformation between the template (or group) and the individual space.

[0083] In step S330, based on the above-mentioned first transformation relationship and second transformation relationship, the brain map and gray matter / white matter surface grid of the group space (such as a reference brain map template) can be mapped to the individual space (such as an MRI image of the target cranial brain structure).

[0084] Specifically, the MRI image of the target cranial and brain structure can be transformed based on the first transformation relationship to obtain a first registered MRI image; then, based on the second transformation relationship, the first registered MRI image can be transformed to obtain a second registered MRI image, and a brain map label can be marked in the second registered MRI image, so that the brain map and / or gray matter / white matter surface mesh of the target cranial and brain structure can be determined based on the second registered MRI image. In this article, the surface mesh can refer to a surface composed of vertices and edges, so as to represent the morphology of the corresponding structure (such as skull, blood vessels, gray matter, white matter, skin, etc.) through the surface.

[0085] For example, the template and the deformation field of the individual space can be applied to the first brain map and the first gray matter / white matter surface grid in the group space to generate the second brain map and the second gray matter / white matter surface grid in the individual space (i.e., the brain map and gray matter / white matter surface grid of the target cranial structure). For example, the third registration relationship can be expressed by the following formula (6):

[0086]

[0087] Among them, L template represents the brain map label in the template space, T affine Represents the first transformation relation such as the affine transformation matrix, D nonlinear represents the second transformation relationship such as nonlinear deformation field, ° represents the transformation of the matrix using the transformation relationship. Finally, L individual Represents a second brain atlas and a second gray / white matter surface mesh mapped into individual space.

[0088] In addition, optionally, after obtaining the brain map of the target cranial structure, the brain map can be fine-tuned using an individual brain region segmentation algorithm to obtain more accurate functional area positioning.

[0089] Preferably, the individual brain region division algorithm can be, for example, a multimodal connectivity-based individual parcellation algorithm (MCIP). Specifically, the individual brain region division algorithm includes the following steps: functional magnetic resonance imaging (fMRI) data and diffusion magnetic resonance imaging (dMRI) data of the target cranial brain structure can be obtained; the fMRI data can be preprocessed to extract the functional time series, wherein the preprocessing of the fMRI data can include, for example, but not limited to artifact removal, slice time correction, head motion correction, sMRI-fMRI registration, standardization and spatial filtering, etc.; the dMRI data can be preprocessed to extract the anatomical connection spectrum, wherein the preprocessing of the dMRI data can include, for example, but not limited to motion correction, eddy current distortion correction, sMRI-dMRI registration, fiber direction estimation and probabilistic fiber tracking, etc.; based on the functional time series (Resting State Timeseries), the anatomical connectivity profile (Anatomical Connectivity Profile), the spatial domain information and the brain map of the target cranial brain structure, the brain region division of the target cranial brain structure is iteratively updated, so that a more accurate brain region division result can be obtained.

[0090] Specifically, the MCIP algorithm can comprehensively consider the anatomical and functional connectivity information of the group-level atlas and the individuals to be partitioned, with the goal of maximizing the consistency of connections within the brain regions under each modality, while ensuring the spatial continuity of the brain regions and the similarity between the individual and group-level atlases. Specifically, the algorithm integrates the functional time series, anatomical connectivity spectrum, spatial connectivity characteristics of vertices, and second brain atlas information, and iteratively solves an energy function based on the graph cuts algorithm to achieve individual brain region partitioning.

[0091] As an example, the brain region division of the target cranial brain structure can be iteratively updated in the following way: the brain region of the target cranial brain structure can be initialized as a group-level atlas; based on the current brain region division of the target cranial brain structure, the average functional time series and average anatomical connectivity spectrum of each brain region are calculated, and the graph cutting algorithm is used to minimize the energy function to update the brain region division of the target cranial brain structure, and the updated brain region division is used as the brain region division of the target cranial brain structure in the next iteration. The above process can be repeated until the brain region division results converge. The MCIP algorithm can ensure that the division results not only reflect the consistency of anatomical and functional connections, but also capture individual-specific brain region characteristics by fusing multimodal data and group-level atlas information.

[0092] In addition, as an example, before performing the above-mentioned second image registration, inhomogeneous field correction can also be performed on the MRI image (such as the preprocessed MRI image). By performing inhomogeneous field correction, the problem of inconsistent signal strength caused by magnetic field inhomogeneity during magnetic resonance imaging can be eliminated, thereby improving the registration accuracy and ensuring that the spatial alignment of the individual's MRI image with the standard brain map template is more accurate.

[0093] In step S240, registered multimodal image data may be obtained based on the first registration relationship, the second registration relationship, and the third registration relationship.

[0094] For example, based on multiple images including CT images, MRI images, and vascular images, and the first registration relationship, the second registration relationship, and the third registration relationship, the registered multimodal image data can be obtained. Through such registration, the corresponding relationship between the multimodal images can be established, so that the information of each multimodal image can be fused to perform comprehensive path planning.

[0095] As an example, the registered CT image can be determined based on the first registration relationship and the original CT image. For example, the original CT image can be transformed using the first registration relationship to obtain the registered CT image. The registered MRI image can be determined based on the second registration relationship and the original MRI image. For example, the original MRI image can be transformed using the second registration relationship to obtain the registered MRI image. The brain map and / or gray-white matter surface grid corresponding to the original MRI image, that is, the brain map and / or gray-white matter surface grid of the target cranial structure can be determined based on the third registration relationship and the original MRI image. For example, based on the third registration relationship, the position of each brain region on the original MRI image is determined to construct a brain map and / or gray-white matter surface grid.

[0096] Here, the registered CT images, the registered MRI images, the brain atlas of the target cranial structure and / or the gray-white matter surface grid can provide information on different aspects of the target cranial structure, so that the information can be integrated for subsequent path planning.

[0097] In addition, although an example process of registration between images is described above, the embodiments of the present disclosure are not limited thereto, and registration between images may also be achieved in other ways, for example, a registration algorithm based on deep learning may be used to improve the registration speed and automation level.

[0098] In addition, although the registration process is described above in the order from step S210 to step S240, the present disclosure does not limit the execution order between the steps. For example, the execution order of steps S210, S220 and S230 can be arbitrary, or can be executed in parallel.

[0099] In step S130 , a target position input by a user for the registered multi-modal image data may be received.

[0100] As an example, a user interaction interface for a user to input a target position may be provided, and the target position may be, for example, any position in the skull determined according to actual needs. According to an embodiment of the present disclosure, an MRI image in multimodal imaging data may be displayed, and a target position selected by a user on the image may be received. For example, a user may manually select a target position such as a target injection / implantation operation to be performed on an MRI image or an individual brain map through a visualization interface.

[0101] In addition, in the embodiments of the present disclosure, the target position can also be automatically recommended. For example, based on big data and artificial intelligence technology, the optimal target point (as well as the preset cranial insertion position range described below) can be automatically recommended to reduce user manual operations.

[0102] Return to reference Figure 1 In step S140, a path from the outside of the target cranial structure to the target position may be planned based on the registered multimodal image data to obtain a planned surgical path.

[0103] After obtaining the registered multimodal image data and the target position, the path planning can be performed accurately because these images can provide comprehensive information from many aspects.

[0104] As an example, when planning a path, some preset planning criteria or optimization conditions may be followed to automatically optimize to the most preferred path. The registered multimodal image data may be used to determine these planning criteria or optimization conditions.

[0105] Here, the planning criteria may include, for example: the shortest distance principle, which can minimize surgical trauma; the perpendicularity to the skull principle, which can ensure stability during the cranial drilling process (for example, based on the skull surface grid to be described below); the thinnest skull principle, which can reduce damage to the skull (based on the skull surface grid); the blood vessel avoidance principle, which can minimize the risk of intraoperative bleeding (for example, based on the blood vessel surface grid to be described below); the important brain area avoidance principle, which can reduce the risk of postoperative neurological function damage (for example, based on the brain map to be described below); setting the route range area, which can meet specific experimental needs, such as the need to pass through a preset intracranial area (for example, based on the brain map to be described below); the sulcus avoidance principle, which can reduce the risk of damaging microvessels (for example, based on the second gray matter / white matter surface grid to be described below).

[0106] In this step, the planned surgical path can be obtained in the following manner: under preset optimization conditions, based on the registered multimodal imaging data, the cranial entry position on the outer surface of the target cranial structure is determined; and the straight line path from the cranial entry position to the target position is used as the planned surgical path.

[0107] Here, the optimization conditions may include at least one of the following items: the path between the entry position and the target position is the shortest; the direction of the path between the entry position and the target position is perpendicular to the skull surface of the target cranial structure; the skull thickness at the entry position is the smallest; the distance between the path between the entry position and the target position and the preset avoidance area is the largest; the distance from the path between the entry position and the target position to the preset pathway area is the smallest; the depth of the path between the entry position and the target position through the sulcus area is the smallest.

[0108] Specifically, multiple optional optimization conditions can be displayed in the user interaction interface. In response to the user selecting at least one optimization condition, a path from the outside of the target cranial structure to the target position is planned based on the optimization condition selected by the user to obtain a planned surgical path.

[0109] The optimization condition can be used as an evaluation criterion to find the optimal cranial insertion position. Here, the shortest path between the cranial insertion position and the target position can be called the shortest path criterion (L1), and its optimization goal is to minimize the distance between the cranial insertion position and the target position. Here, the distance between the cranial insertion position and the target position can be expressed, for example, by the following formula (7):

[0110] L1(p)=||p-p0||2 (7)

[0111] Where p represents the skull entry position, p0 represents the target position, ||·||2 represents the 2-norm (i.e., Euclidean distance), L1(p) represents the distance between the skull entry position p and the target position, and the optimization goal is to minimize the value of L1(p).

[0112] The direction of the path between the entry position and the target position is perpendicular to the skull surface of the target cranial structure and can be called the perpendicular to the skull criterion (L2). The optimization goal is to ensure the stability of the execution instrument such as the surgical instrument when entering the skull. Here, the distance between the entry position and the target position can be expressed by the following formula (8), for example:

[0113]

[0114] Among them, n prepresents the normal vector at the entry position p on the skull surface, L2(p) represents the angle between the direction of the path between the entry position and the target position and the direction of the skull surface perpendicular to the target cranial structure, and the optimization goal is to minimize the value of L2(p).

[0115] The minimum skull thickness at the entry position can be called the thinnest skull criterion (L3), and its optimization goal is to select the thinnest skull thickness for entry to reduce the difficulty of surgery. Here, the skull thickness at the entry position can be expressed, for example, by the following formula (9):

[0116] L3(p)=||pS(p,p0)||2(9)

[0117] Among them, S(p, p0) represents the intersection of the path (p, p0) and the inner surface of the skull, L3(p) represents the thickness of the skull at the entry position, and the optimization goal is to minimize the value of L3(p).

[0118] The maximum distance between the path between the entry point and the target position and the preset avoidance area can be called the avoidance area criterion (L4). Its optimization goal is to avoid blood vessels and important brain areas and reduce intraoperative risks.

[0119] As an example, the distance between the path from the skull entry position to the target position and the preset avoidance area can be determined in the following manner: determine multiple intermediate points on the straight path from the skull entry position to the target position; and take the minimum distance from each intermediate point to the avoidance area as the distance between the path and the avoidance area.

[0120] Here, the thickness of the skull at the cranial insertion position can be expressed, for example, by the following formula (10):

[0121]

[0122] Among them, SDF A (·) represents the signed distance function of the target avoidance area, A represents the target avoidance area, SDF A A positive value indicates that the input point is outside region A, and a negative value indicates that the input point is inside region A. Its absolute value indicates the shortest distance between the input point and the boundary of region A. A The value can be taken in the range of [0, 1] according to actual needs to find the path between the entry position p and the target position p0 so that the signed distance function SDF A(·) is the smallest point. L4(p) represents the inverse of the distance between the path between the entry position and the target position and the preset avoidance area. The optimization goal is to minimize the value of L4(p). Here, the distance between the path between the entry position and the target position and the preset avoidance area is taken as the inverse number, so that the optimization goals in each optimization condition can be unified (for example, the optimization goals are all to minimize or maximize the value), simplifying the optimization calculation.

[0123] As an example, the minimum distance among the distances from each intermediate point to the avoidance area can be determined by iteratively performing the following operations: determining multiple current sampling points on the current candidate path; determining the current reference sampling point with the shortest distance from the avoidance area among the multiple current sampling points; determining a first path segment on the current candidate path that includes the current reference sampling point (e.g., centered on the reference sampling point); determining a second path segment based on the distances from the two endpoints of the first path segment to the avoidance area and the distance from the current reference sampling point to the avoidance area, wherein the second path segment is the path between one of the two endpoints of the first path segment and the current reference sampling point; using the second path segment as the candidate path for the next iteration until the number of iterations reaches a preset number, and using the distance from the reference sampling point of the last iteration to the avoidance area as the above-mentioned minimum distance. Here, in the first iteration, the candidate path is the path between the cranial entry position p and the target position p0.

[0124] As a preferred implementation of this embodiment, for a specific cranial entry position p, a fast algorithm combining discrete sampling and binary iteration is used to solve L4(p). Compared with discrete sampling, this algorithm has better accuracy and can improve the defect that simple binary iteration falls into a local minimum point and cannot reach the global minimum point. The implementation process of this algorithm is as follows: First, on the line segment (p, p0) (i.e., t A ∈[0,1] interval), specifically, the interval [0,1] is evenly divided into N (such as N = 10) parts, N+1 sampling points are obtained, and the L4 value at each sampling point is calculated respectively, and then the t that makes L4 obtain the minimum value is found. A , denoted as t mid ; then t mid As the center, take a smaller sampling interval to the left and right as the initial upper and lower limits of the binary iteration, for example, t low =t mid -1 / 2N, t high =t mid +1 / 2N, then calculate and compare t low ,t mid and t high L4. If t low The smaller L4 is, the smaller the minimum value is. low , t mid], then let the new t high =t mid ; if t high The smaller the L4 is, the smaller the minimum value is. mid , t high ], then let the new t low =t mid Otherwise, it means t mid L4 is already better than t low and t high smaller, the minimum is in the middle, then let the new t low =(t low +t mid ) / 2, new = (t mid +t high ) / 2. By repeating this process multiple times within the above-mentioned smaller interval, the minimum value of L4 can be accurately obtained.

[0125] The path between the entry point and the target position with the shortest distance to the preset pathway area can be called the pathway area criterion (L5). The optimization goal is to ensure that the path passes through a specific area, meets specific experimental requirements, or passes through a preset safety area.

[0126] As an example, the distance between the path from the skull entry position to the target position and the preset pathway area can be determined in the following manner: determine multiple intermediate points on the straight path from the skull entry position to the target position; and take the minimum distance from each intermediate point to the pathway area as the distance between the path and the pathway area.

[0127] Here, the distance from the path between the cranial entry position and the target position to the preset path area can be expressed, for example, by the following formula (11):

[0128]

[0129] Among them, SDF B (.) represents the signed distance function of the target path area, B represents the target path area, t B The value can be taken in the range of [0, 1] according to actual needs to find the path between the entry position p and the target position p0 so that the signed distance function SDF B (·) is the smallest point. L5(p) represents the distance from the path between the cranial entry position and the target position to the preset path area. The optimization goal is to minimize the value of L5(p).

[0130] As an example, a fast algorithm combining discrete sampling and binary iteration can also be used to solve L5(p). Specifically, the minimum distance among the distances from each intermediate point to the path area can be determined by iteratively performing the following operations: determining multiple current sampling points on the current candidate path; determining the current reference sampling point with the shortest distance from the path area among the multiple current sampling points; determining the first path segment on the current candidate path that includes the current reference sampling point (for example, centered on the reference sampling point); determining the second path segment based on the distances from the two endpoints of the first path segment to the path area and the distance from the current reference sampling point to the path area, wherein the second path segment is the path between one of the two endpoints of the first path segment and the current reference sampling point; using the second path segment as the candidate path for the next iteration until the number of iterations reaches a preset number, and using the distance from the reference sampling point of the last iteration to the path area as the above-mentioned minimum distance. Here, in the first iteration, the candidate path is the path between the cranial entry position p and the target position p0.

[0131] The minimum depth of the path between the entry position and the target position passing through the sulcus region can be called the sulcus avoidance criterion (L6), and its optimization goal is to avoid the sulcus region as much as possible to reduce the risk of microvascular damage in the sulcus region. Here, the depth of the path between the entry position and the target position passing through the sulcus region can be expressed, for example, by the following formula (12):

[0132] L6(p)=SulcDepth(p,p0) (12)

[0133] Among them, SulcDepth(p,p0) represents the depth of the path (p,p0) passing through the sulcus. L6(p) represents the depth of the path between the entry position and the target position passing through the sulcus area. The optimization goal is to minimize the value of L6(p).

[0134] Through the above method, a variety of optimization conditions or evaluation criteria can be provided, so that multiple safety constraints (such as skull thickness, blood vessel distance, brain sulcus depth, etc.) can be set during the path planning process, making the final planned path safer, more accurate, and more valuable for reference.

[0135] In addition, although several optimization conditions are described above in an exemplary manner, the embodiments of the present disclosure are not limited thereto, and other optimization conditions may be added, such as avoiding functional nuclei, considering the direction of white matter fiber bundles (for example, dMRI / fMRI data may be acquired, the fMRI / dMRI data may be registered with the sMRI data, the positions of the functional nuclei may be determined based on the registered fMRI data, the directions of the white matter fiber bundles may be determined based on the registered dMRI data, and the optimization conditions related to the positions of the functional nuclei or the directions of the white matter fiber bundles may be determined), etc., to meet a wider range of application needs.

[0136] As an example, the optimization condition may be one, in which case the optimal solution under the optimization condition may be determined as the final cranial insertion position to plan a path from the cranial insertion position to the target position.

[0137] As another example, the optimization condition may be multiple. In the example of multiple optimization conditions, for example Figure 4 As shown, the cranial insertion position on the outer surface of the target cranial structure can be determined by:

[0138] In step S410, an objective function may be constructed based on each optimization condition and a preset weight corresponding to each optimization condition.

[0139] Here, the objective function may be represented by weighting and summing the scores of the cranial entry positions under various optimization conditions using preset weights.

[0140] Specifically, a weight may be preset for each of the plurality of optimization conditions. Here, the weight of each optimization condition may be set according to actual needs or experience, for example, may be input by a user.

[0141] As an example, the objective function can be expressed by the following equation (13):

[0142]

[0143] in, represents the normalized value of the i-th optimization condition, m represents the total number of optimization conditions, L(p) represents the objective function, and the overall optimization goal is to minimize the value of L(p).

[0144] In step S420, an optimal solution of the objective function may be obtained for a position within a preset cranial entry position region on the outer surface of the target cranial structure.

[0145] Here, the preset cranial entry position area may be a local area on the outer surface of the cranial structure, for example, it may be input by the user or predetermined. By setting this area, the scope of searching for the optimal solution may be narrowed and the amount of calculation may be reduced.

[0146] As an example, the preset cranial entry position area can be determined in the following manner: based on the registered multimodal imaging data, the outer surface of the target cranial structure (such as the outer surface of the skull) is determined; based on the outer surface of the target cranial structure, the target axis and its value range, the preset cranial entry position area is determined, wherein the preset cranial entry position area is the area of ​​the outer surface of the target cranial structure within the value range of the target axis.

[0147] Here, the centroid c of the skull surface vertices can be calculated, and for each vertex p among the multiple vertices on the skull surface, its normal vector n can be calculated. pand the displacement vector pc pointing to the center of gravity, calculate the product of the normal vector and the transpose of the displacement vector (pc) T n p In response to the product being greater than 0, the vertex p can be determined as the inner surface, and in response to the product being less than or equal to 0, the vertex p can be determined as the outer surface. By sequentially determining whether the surfaces on which the above-mentioned multiple vertices are located are the inner surface or the outer surface, the outer surface of the target cranial structure (such as the outer surface of the skull) can be determined based on the set of vertices located on the outer surface.

[0148] As an example, the target axis and its value range can be input by the user or preset. Figure 5 As shown, the preset cranial entry position area can be determined on the outer surface of the target cranial structure according to the target axis and its value range.

[0149] Specifically, the user can select the target axis plane (X / Y / Z) and its value range (for example, 100-200) in the user interaction interface. In the method of the embodiment of the present invention, in response to receiving the target axis and its value range input by the user, a set of vertices Ω on the outer surface of the skull that meets the conditions can be screened out on the outer surface of the target cranial structure as a candidate range for the cranial entry position, i.e., the above-mentioned preset cranial entry position area.

[0150] In this example, the target position and the preset skull entry position area can be manually input by the user. For example, the user can determine the target position and customize the region of interest on the skull surface for the registered multimodal imaging data through the user interaction interface as the automatic planning range of the skull entry point (i.e., the preset skull entry position area). The above describes an example of determining the preset skull entry position area. Returning to step S420, the position that can obtain the optimal solution of the objective function can be solved within the preset skull entry position area.

[0151] As an example, multiple optimization algorithms (or solvers) may be preset, and the user may select a suitable optimization algorithm according to the requirements. In response to receiving the optimization algorithm selected by the user, the optimal solution of the objective function is solved using the selected optimization algorithm.

[0152] Here, the preset optimization algorithm may include, for example, at least one of the traversal search method, the Levenberg-Marquardt algorithm (LM algorithm), and the simulated annealing algorithm. In addition, the preset optimization algorithm may also include, for example, a global optimization algorithm such as a particle swarm optimization algorithm (PSO), a genetic algorithm (GA), etc., so as to further improve the efficiency and accuracy of the solution.

[0153] Specifically, in the traversal search method, the candidate cranial entry point set Ω can be exhaustively enumerated, the objective function value of each cranial entry point can be calculated, and several optimal solutions can be selected.

[0154] The LM algorithm is an optimization algorithm for solving nonlinear problems. It combines the advantages of the gradient descent method and the Gauss-Newton method, and has a good balance between global search and local convergence. The LM algorithm is similar to gradient descent in the initial iterations, which can effectively avoid falling into the local optimum. When it is close to the solution, it will switch to the Gauss-Newton method to accelerate convergence.

[0155] Specifically, multiple initial skull entry positions can be selected from the vertex set Ω of the preset skull entry position area, and the skull entry positions can be iteratively updated to gradually approach the optimal solution. The iterative process can be expressed by the following formula (14), for example:

[0156]

[0157] Where p represents the current step’s entry position, J is the Jacobian matrix of the objective function L(p), is the gradient of the objective function, and λ is an adjustment coefficient, which is used to balance between the gradient descent and the Gauss-Newton method, and can be given in advance. Based on the above formula (14), iteration is performed. If the maximum number of iterations is reached or the gradient is less than a preset threshold, the iteration can be terminated and the optimal solution obtained by the current iteration is output.

[0158] Here, the Levenberg-Marquardt algorithm is an optimization algorithm for nonlinear least squares problems. By using this algorithm, the advantages of the gradient descent method and the Gauss-Newton method can be combined to find the optimal solution more quickly and accurately.

[0159] The simulated annealing algorithm is a random search algorithm for global optimization, which is particularly suitable for solving multi-peak and non-convex optimization problems. The simulated annealing algorithm allows suboptimal solutions to be accepted during the optimization process by introducing randomness, thereby avoiding falling into the local optimum and ultimately approaching the global optimal solution.

[0160] Specifically, multiple initial skull entry positions p0 may be selected by uniformly sampling from the vertex set Ω of the preset skull entry position area, where p0∈Ω. For each initial skull entry position p0, the skull entry position p and the temperature T may be iteratively updated.

[0161] For example, the update can be performed in the following manner: randomly generate a new solution p′ in the neighborhood of the current skull entry position p, and calculate the energy difference ΔL=L(p′)-L(p) between the current skull entry position p and the new skull entry position p′; if ΔL<0, the new solution can be accepted; otherwise, the new solution can be updated with a preset probability (e.g. Among them, T i represents the current temperature) to accept the new solution. Here, accepting the new solution may mean replacing the current cranial entry position with a new cranial entry position. In addition, the temperature may also be updated. For example, the updated temperature may be Ti+1 =αT i , where α is the preset update coefficient.

[0162] Iteration is performed based on the above process. If the maximum number of iterations is reached or the temperature is lower than the threshold, the iteration can be terminated. Otherwise, the above update process can be repeated until the iteration is terminated and the optimal solution is output.

[0163] Here, since the simulated annealing algorithm is a random optimization algorithm that simulates the physical annealing process, it is suitable for solving the global optimal solution, and is particularly suitable for the process of solving the optimal solution for the objective function here.

[0164] In step S430, the position corresponding to the optimal solution may be determined as the cranial entry position.

[0165] In this step, the position corresponding to the above optimal solution can be used as the skull entry position.

[0166] Through the above method, the optimal cranial entry position that meets the set optimization conditions can be obtained to provide a more reasonable and accurate surgical path.

[0167] In addition, although the above description describes determining the cranial entry position under preset optimization conditions, the embodiments of the present disclosure are not limited to this, and the user can also select the cranial entry position completely manually, so that the path from the cranial entry position to the target position can be planned based on the registered multimodal imaging data according to the cranial entry position selected by the user, thereby obtaining the planned surgical path.

[0168] When the cranial entry position is determined, for example, a straight line path from the cranial entry position to the target position can be used as the final planned surgical path. As an example, the final surgical path can be exported into a format readable by a surgical navigation system to support reference positioning and guidance during surgery.

[0169] An example process of surgical path planning according to an embodiment of the present disclosure is described above, and an example of determining a reference position in each optimization condition will be described in detail below.

[0170] Specifically, as described above, the multiple images may include CT images, MRI images, and vascular images. On the one hand, multimodal image data registration may be performed to establish a correspondence between different images; on the other hand, the surgical path planning method may also include: surface mesh reconstruction based on multimodal image data to obtain a vascular surface mesh, a skull surface mesh, and / or a gray matter surface mesh of the target cranial structure. These meshes may be used to determine reference positions in optimization conditions, such as avoidance areas, skull surfaces, skull thickness, brain sulcus areas, etc.

[0171] For example, the surgical path planning method may also include: determining a vascular surface grid of the target cranial structure based on the vascular image; and determining an avoidance area based on the vascular surface grid.

[0172] Specifically, blood vessel segmentation can be performed based on the original blood vessel image or the preprocessed blood vessel image to extract the three-dimensional structure of the intracranial blood vessels of the target cranial and brain structure, and determine the blood vessel surface grid of the target cranial and brain structure. In addition, blood vessel labels can be generated based on the three-dimensional structure of the intracranial blood vessels, and the blood vessel labels can be used to mark the position of the blood vessels in the target cranial and brain structure. Here, the existing blood vessel segmentation method can be used for segmentation. The embodiments of the present disclosure do not impose any special restrictions on the blood vessel segmentation method. For example, an image segmentation algorithm such as U-Net can be used to implement blood vessel segmentation to improve the accuracy of blood vessel segmentation.

[0173] By generating the vascular surface mesh, the location information of the intracranial blood vessels of the target cranial structure can be obtained, providing an accurate anatomical structure reference for path planning, so that key blood vessels can be avoided during path planning and the risk of intraoperative bleeding can be reduced. For example, based on the vascular surface mesh, the avoidance area in the optimization condition can be determined, such as the target avoidance area A in the above formula (10), to avoid important blood vessels.

[0174] Alternatively or additionally, the surgical path planning method may further include: determining a skull surface mesh of the target cranial and brain structure based on the CT image; and determining a skull surface and skull thickness based on the skull surface mesh.

[0175] Here, since the CT image may contain skin surface information and bone information, the skin surface mesh and / or skull surface mesh of the target cranial structure may be reconstructed based on the CT image.

[0176] Here, the position and thickness of the skull can be determined by performing bone segmentation on the CT image to obtain the skull surface mesh. The skull surface mesh can provide an accurate anatomical structure reference for path planning. For example, the skull surface and skull thickness under the optimization condition can be determined based on the skull surface mesh, such as determining the skull thickness in the above formula (9) and the skull surface used to determine the position of the skull entry. In addition, the skin surface mesh can be used for navigation in subsequent surgery to achieve image-physical space alignment.

[0177] In the above process, the existing bone segmentation method can be used for bone segmentation. The embodiments of the present disclosure do not impose any special restrictions on the segmentation method of the CT image. For example, the existing image segmentation algorithm can be used to perform bone segmentation on the CT image to obtain the skull surface grid.

[0178] Alternatively or additionally, the surgical path planning method may further include: determining the gray-white matter surface grid of the target cranial structure based on the MRI image; and determining the depth of the sulcus region based on the gray-white matter surface grid.

[0179] Here, software such as FreeSurfer can be used to extract the gray and white matter surface mesh of the target cranial structure. The gray and white matter surface mesh can be used to calculate the depth of the sulcus. For example, the depth of the sulcus region in the above formula (12) can be determined.

[0180] In addition, the brain map of the target cranial structure can also be determined based on the MRI images. For example, it can be obtained by image registration between the MRI images and the reference brain map template, or it can be obtained based on the individual brain region division algorithm of fMRI / dMRI.

[0181] Through the above-mentioned method, the surgical path planning method of the embodiment of the present disclosure can integrate multimodal images such as sMRI structural images, CT skull data, vascular imaging (such as TOF-MRA) data, and personalized brain maps, and define the multimodal input data required for path planning (such as sMRI, skull surface mesh, vascular surface mesh, etc.), thereby realizing a comprehensive and comprehensive path planning solution.

[0182] According to the surgical path planning method of the embodiment of the present disclosure, multimodal imaging data can be combined to automatically plan one or more candidate paths based on the optimization conditions manually determined by the user and the importance weight of each optimization condition. These paths can provide real-time previews to users through a three-dimensional visual user interaction interface, and support users to further correct and optimize the paths based on professional experience.

[0183] Specifically, during the execution of the surgical path planning method of the embodiment of the present disclosure, the planned surgical path can be displayed in real time, so that the user can observe the cranial entry position, target position and path in three-dimensional space. In response to the user's editing operation on the displayed surgical path, the surgical path can be adjusted and optimized. In this way, while realizing automated path planning, it also supports the intervention of expert experience to improve the reliability and accuracy of the final generated path.

[0184] The following will refer to Figure 6 and Figure 7 An application example of an image registration process and a path planning process in a surgical path planning method based on multimodal image registration according to an exemplary embodiment of the present disclosure is described.

[0185] like Figure 6As shown, in step S601, multimodal image data, such as a first CT image, a first sMRI image, and a first TOF-MRA image, can be read. In step S602, a user's manual rotation operation on the first CT image, the first sMRI image, and the first TOF-MRA image can be received to rotate each image to the correct anatomical direction. In step S603, a user's manual cropping operation on the first CT image, the first sMRI image, and the first TOF-MRA image can be received to obtain a cropped second CT image, a second sMRI image, and a second TOF-MRA image. Here, step S602 and step S603 are optional steps, and rotation and cropping may not be performed.

[0186] In step S604, cross-modality image registration may be performed for the current individual, for example, image registration of the second CT image to the second sMRI image, image registration of the second TOF-MRA image to the second sMRI image may be performed, for example, corresponding image transformation relationships may be obtained respectively.

[0187] In step S605, a skull surface mesh may be reconstructed based on the second CT image, and optionally, a skin surface mesh may also be reconstructed.

[0188] In step S606, blood vessel segmentation may be performed based on the second TOF-MRA image to obtain blood vessel labels; in step S607, blood vessel surface meshes may be reconstructed based on the blood vessel labels.

[0189] Optionally, in step S608, inhomogeneous field correction may be performed on the second sMRI image.

[0190] In step S609, the second sMRI image can be image registered with the reference brain map template, thereby obtaining a second brain map and a second gray matter / white matter surface grid located in the individual space (corresponding to the second sMRI image) based on the first brain map and the first gray matter / white matter surface grid located in the group space (corresponding to the reference brain map template).

[0191] In step S610, based on the second brain map and the second sMRI image, a target position (or target point) manually input by a user can be received, and in step S611, an area of ​​interest on the skull surface manually defined by the user can also be received as a range for automatic planning of the skull entry position.

[0192] In step S612, one or more candidate paths can be automatically planned based on the skull surface mesh, the blood vessel surface mesh, and the second gray matter / white matter surface mesh, through the evaluation criteria manually determined by the user (i.e., the above-mentioned optimization conditions) and the importance weight of each evaluation criterion, such as determining the skull entry position.

[0193] In step S613 , the user's selection and / or fine-tuning of the candidate path may be received, and the final planned surgical path may be saved.

[0194] like Figure 7 As shown, in step S701, the individual brain map (such as the above-mentioned second brain map) and the sMRI image (such as the above-mentioned second sMRI image) can be determined through the above-mentioned image registration; in step S702, the target position selected by the user for the individual brain map and the sMRI image can be received, and in step S703, the evaluation criteria set by the user (i.e., the above-mentioned optimization conditions) and their weights can be received.

[0195] Here, the evaluation criteria may include, for example, the shortest path criterion, the perpendicularity to the skull criterion, the thinnest skull criterion, the criterion of avoiding blood vessels and / or specific areas, the criterion of passing through specific areas, the criterion of avoiding brain grooves, etc., wherein, referring to the above Figure 6 As a result of the image registration in , the criterion perpendicular to the skull and the thinnest criterion through the skull can be determined based on the skull surface grid, the criterion for avoiding blood vessels and / or specific areas can be determined based on the blood vessel surface grid and the individual brain atlas, the criterion for approaching specific areas can be determined based on the individual brain atlas, and the criterion for avoiding sulci can be determined based on the gray matter / white matter surface grid.

[0196] Based on the set evaluation criteria and their weights, the skull outer surface identified in step S704, and the skull entry position range selected by the user received in step S705, the solver selected by the user received in step S706 is used to optimize the objective function through the solver to obtain multiple candidate paths. In step S707, the path selected by the user and the user's adjustment and optimization of the path can be received to obtain the final planned path.

[0197] According to the surgical path planning method of the embodiment of the present disclosure, a variety of medical imaging data such as structural magnetic resonance imaging (sMRI), CT, TOF-MRA, etc. can be used to provide comprehensive anatomical and functional information. For example, image data such as the skull surface mesh (generated by CT images) and the blood vessel surface mesh (generated by TOF-MRA or other vascular imaging technologies) can be obtained to achieve the fusion of multimodal imaging data and support high-precision path planning.

[0198] In addition, according to the surgical path planning method of the embodiment of the present disclosure, multiple optimization conditions such as the shortest distance, perpendicular to the skull, the thinnest skull, avoiding blood vessels and important brain areas, passing through specific areas, and avoiding brain grooves can be comprehensively considered, so that the final path helps to improve the safety, success rate and efficiency of the operation. In addition, the method also allows the user to adjust the weight of each criterion according to needs to achieve personalized path planning.

[0199] In addition, according to the surgical path planning method of the embodiment of the present disclosure, efficient optimization methods such as the Levenberg-Marquardt algorithm and the simulated annealing algorithm can be used to achieve fast and accurate path planning, generate the optimal path, and meet real-time interaction requirements.

[0200] In addition, according to the surgical path planning method of the embodiment of the present disclosure, a friendly and intuitive three-dimensional visual user interface can be provided, which supports manual selection of the target position and the range of the cranial entry position, and supports real-time preview and adjustment of the planning results, thereby improving the user experience and control.

[0201] In addition, according to the surgical path planning method of the embodiment of the present disclosure, in the process of multimodal image registration, high-precision image registration is achieved by directly adjusting the direction matrix and origin of the image, thereby retaining the detailed information of the original image without the need for resampling, ensuring high-precision alignment of the image and avoiding the problem of accuracy loss.

[0202] In addition, according to the surgical path planning method of the embodiment of the present disclosure, an end-to-end workflow can be achieved. Specifically, a complete set of automated processes can be formed from the reading, processing, and registration of multimodal images to path planning, thereby reducing manual intervention and operational complexity and improving work efficiency.

[0203] The surgical path planning method according to the embodiment of the present disclosure relates to the fields of neuroscience, medical image processing and computer-assisted surgical planning, especially the preoperative path planning method for intracranial drug administration and implantation surgery such as large animal experiments. The method integrates multimodal brain images (such as CT, MRI, TOF-MRA, etc.) to achieve high-precision brain structure registration and personalized surgical path planning, comprehensively considers multiple evaluation criteria, and automatically generates the optimal surgical path, thereby improving the accuracy and safety of the surgery.

[0204] According to a second aspect of an embodiment of the present disclosure, a surgical path planning device based on multimodal image registration is provided, such as Figure 8 As shown, the surgical path planning device may include an acquisition unit, a registration unit, a receiving unit and a planning unit.

[0205] The acquisition unit 810 is configured to acquire multimodal image data of a target cranial structure, wherein the multimodal image data includes a plurality of images captured for the target cranial structure, and the plurality of images are captured in different ways.

[0206] The registration unit 820 is configured to perform image registration on multiple images to obtain registered multi-modal image data.

[0207] The receiving unit 830 is configured to receive a target position input by a user for the registered multi-modal image data.

[0208] The planning unit 840 is configured to plan a path from the outside of the target cranial structure to the target position based on the registered multimodal image data to obtain a planned surgical path.

[0209] As an example, the planning unit 840 is also configured to obtain a planned surgical path in the following manner: under preset optimization conditions, based on the registered multimodal imaging data, determine the cranial entry position on the outer surface of the target cranial structure; use the straight line path from the cranial entry position to the target position as the planned surgical path, wherein the optimization conditions include at least one of the following items: the path between the cranial entry position and the target position is the shortest; the direction of the path between the cranial entry position and the target position is perpendicular to the skull surface of the target cranial structure; the skull thickness at the cranial entry position is the smallest; the distance between the path between the cranial entry position and the target position and the preset avoidance area is the largest; the distance between the path between the cranial entry position and the target position and the preset approach area is the smallest; the depth of the path between the cranial entry position and the target position through the sulcus area is the smallest.

[0210] As an example, there are multiple optimization conditions, wherein the planning unit 840 is further configured to: construct an objective function based on each optimization condition and a preset weight corresponding to each optimization condition, wherein the objective function represents the weighted sum of the scores of the cranial entry position under each optimization condition using the preset weights; solve the optimal solution of the objective function for the position within the preset cranial entry position area on the outer surface of the target cranial structure; and determine the position corresponding to the optimal solution as the cranial entry position.

[0211] As an example, the planning unit 840 is also configured to determine the distance between the path from the cranial entry position to the target position and the preset avoidance area in the following manner: determining multiple intermediate points on the straight path from the cranial entry position to the target position; taking the minimum distance from each intermediate point to the avoidance area as the distance between the path and the avoidance area, wherein the distance between the path from the cranial entry position to the target position and the preset pathway area is determined in the following manner: determining multiple intermediate points on the straight path from the cranial entry position to the target position; taking the minimum distance from each intermediate point to the pathway area as the distance between the path and the pathway area.

[0212] As an example, the multiple images include CT images, MRI images and vascular images, wherein the surgical path planning device also includes a surface mesh determination unit, and the surface mesh determination unit is configured to: determine the vascular surface mesh of the target cranial and brain structure based on the vascular image, and determine the avoidance area based on the vascular surface mesh; and / or, determine the skull surface mesh of the target cranial and brain structure based on the CT image, and determine the skull surface and skull thickness based on the skull surface mesh; and / or, determine the gray and white matter surface mesh of the target cranial and brain structure based on the MRI image, and determine the depth of the sulcus area based on the gray and white matter surface mesh.

[0213] As an example, the multiple images include CT images, MRI images, and vascular images, and the registration unit 820 is further configured to obtain registered multimodal image data in the following manner: performing a first image registration on the CT image and the MRI image to obtain a first registration relationship; performing a first image registration on the vascular image and the MRI image to obtain a second registration relationship; performing a second image registration on the MRI image and the reference brain atlas template to obtain a third registration relationship; based on the first registration relationship, the second registration relationship, and the third registration relationship, obtaining the registered multimodal image data, wherein the first image registration is performed in the following manner: determining the The rotation transformation amount and the translation transformation amount between the images are adjusted to maximize the similarity between the images to be registered, and obtain the registration relationship between the images to be registered; wherein the second image registration is performed in the following manner: determining a first transformation relationship that maximizes the similarity between the MRI image and the reference brain map template; determining a second transformation relationship that aligns each voxel between the MRI image and the reference brain map template, and determining a third registration relationship based on the first transformation relationship and the second transformation relationship, wherein the third registration relationship represents the correspondence between the MRI image and each brain region of the reference brain map template.

[0214] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method. The training system can execute the corresponding steps in the method according to the method embodiments of the first and second aspects above, which will not be elaborated here.

[0215] Fig. 9 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. Fig. 9 As shown, the electronic device 10 includes a processor 101 and a memory 102 for storing processor executable instructions. Here, when the processor executable instructions are executed by the processor, the processor is prompted to execute the image segmentation model training method or image segmentation method as described in the above exemplary embodiments.

[0216] As an example, the electronic device 10 is not necessarily a single device, but may be any collection of devices or circuits that can execute the above instructions (or instruction sets) individually or in combination. The electronic device 10 may also be part of an integrated control system or system manager, or may be configured as a server that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0217] In the electronic device 10, the processor 101 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not a limitation, the processor 101 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0218] The processor 101 may execute instructions or codes stored in the memory 102, wherein the memory 102 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0219] The memory 102 may be integrated with the processor 101, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 102 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 102 and the processor 101 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 101 can read files stored in the memory 102.

[0220] In addition, the electronic device 10 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 10 may be connected to each other via a bus and / or a network.

[0221] In an exemplary embodiment, a computer-readable storage medium may also be provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the image segmentation model or the image segmentation method as described in the above exemplary embodiments. The computer-readable storage medium may be, for example, a memory including instructions. Optionally, the computer-readable storage medium may be: a read-only memory (ROM), a random access memory (RAM), a random access programmable read-only memory (PROM), an electrically erasable programmable read-only memory (EEPROM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a non-volatile memory, a CD-ROM, a CD-R, a CD+R, a CD-RW, a CD+RW, a DVD-ROM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, a DVD-RAM, a BD-ROM, a BD-R, a BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0222] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, the computer program product comprising computer executable instructions, which, when executed by at least one processor, implement the image segmentation model training method or image segmentation method according to the exemplary embodiment of the present disclosure.

[0223] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0224] In addition, it should be noted that although several examples of each step are described above with reference to specific drawings, it should be understood that the embodiments of the present disclosure are not limited to the combinations given in the examples, and the steps appearing in different drawings may be combined, and the order of execution of the steps may be changed, which is not exhaustive here.

[0225] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A surgical path planning method based on multimodal image registration, characterized in that: The surgical path planning method comprises: Acquiring multimodal image data of a target cranial brain structure, wherein the multimodal image data includes a plurality of images captured for the target cranial brain structure, and the plurality of images are captured in different ways; Performing image registration on the multiple images to obtain registered multimodal image data; receiving a target position input by a user for the registered multimodal image data; Based on the registered multimodal image data, a path from the outside of the target cranial structure to the target position is planned to obtain a planned surgical path.

2. The surgical path planning method according to claim 1, characterized in that: The planned surgical path is obtained by: Under preset optimization conditions, based on the registered multimodal image data, determining a cranial entry position on the outer surface of the target cranial brain structure; The straight line path from the cranial entry position to the target position is used as the planned surgical path. The optimization condition includes at least one of the following: The path between the cranial entry position and the target position is the shortest; The direction of the path between the cranial entry position and the target position is perpendicular to the skull surface of the target cranial brain structure; The thickness of the skull at the cranial entry position is the smallest; The distance between the path between the cranial entry position and the target position and the preset avoidance area is the greatest; The distance between the path between the cranial entry position and the target position and the preset path area is the shortest; The depth of the path between the cranial entry position and the target position passing through the brain sulcus region is minimal.

3. The surgical path planning method according to claim 2, characterized in that: There are multiple optimization conditions, wherein under the preset optimization conditions, based on the registered multimodal image data, determining the cranial entry position on the outer surface of the target cranial structure includes: Based on each optimization condition and a preset weight corresponding to each optimization condition, construct an objective function, wherein the objective function represents a weighted sum of the scores of the cranial entry position under each optimization condition using the preset weight; Solving the optimal solution of the objective function for a position within a preset cranial entry position area on the outer surface of the target cranial structure; The position corresponding to the optimal solution is determined as the cranial entry position.

4. The surgical path planning method according to claim 2 or 3, characterized in that: The distance between the path from the cranial entry position to the target position and the preset avoidance area is determined by: Determine a plurality of intermediate points on a straight line path from the cranial insertion position to the target position; The minimum distance between each intermediate point and the avoidance area is used as the distance between the path and the avoidance area. The distance between the path from the cranial entry position to the target position and the preset path area is determined by: Determine a plurality of intermediate points on a straight line path from the cranial insertion position to the target position; The minimum distance between each intermediate point and the path area is taken as the distance between the path and the path area.

5. The surgical path planning method according to claim 2, characterized in that: The multiple images include CT images, MRI images and vascular images, wherein the surgical path planning method further includes: Based on the vascular image, determining the vascular surface grid of the target cranial structure, Determining the avoidance area based on the blood vessel surface grid; and / or, Based on the CT image, determining the skull surface grid of the target cranial structure, Based on the skull surface mesh, determining the skull surface and the skull thickness; and / or, Based on the MRI image, the gray and white matter surface grid of the target cranial structure is determined, Based on the gray-white matter surface mesh, the depth of the sulcal region is determined.

6. The surgical path planning method according to claim 1, characterized in that: The multiple images include CT images, MRI images and vascular images, Among them, the registered multimodal image data is obtained by the following method: Performing a first image registration on the CT image and the MRI image to obtain a first registration relationship; Performing the first image registration on the vascular image and the MRI image to obtain a second registration relationship; Performing a second image registration on the MRI image and the reference brain atlas template to obtain a third registration relationship; Based on the first registration relationship, the second registration relationship and the third registration relationship, the registered multimodal image data is obtained, The first image registration is performed in the following manner: determining a rotational transformation amount and a translational transformation amount between the images to be registered, adjusting the rotational transformation amount and the translational transformation amount to maximize the similarity between the images to be registered, and obtaining a registration relationship between the images to be registered; The second image registration is performed in the following manner: Determining a first transformation relationship that maximizes the similarity between the MRI image and the reference brain atlas template; determining a second transformation relationship that aligns each voxel between the MRI image and the reference brain atlas template, Based on the first transformation relationship and the second transformation relationship, the third registration relationship is determined, wherein the third registration relationship represents the correspondence between the MRI image and each brain region of the reference brain atlas template.

7. A surgical path planning device based on multimodal image registration, characterized in that: The surgical path planning device comprises: An acquisition unit is configured to acquire multimodal image data of a target cranial brain structure, wherein the multimodal image data includes a plurality of images captured for the target cranial brain structure, and the plurality of images are captured in different ways; a registration unit, configured to perform image registration on the multiple images to obtain registered multimodal image data; A receiving unit, configured to receive a target position input by a user for the registered multimodal image data; The planning unit is configured to plan a path from the outside of the target cranial structure to the target position based on the registered multimodal image data to obtain a planned surgical path.

8. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor, Wherein, when the processor executable instructions are executed by the processor, the processor is prompted to execute the surgical path planning method based on multimodal image registration according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the surgical path planning method based on multimodal image registration according to any one of claims 1 to 6.

10. A computer program product comprising computer executable instructions, characterized in that: When the computer executable instructions are executed by at least one processor, the surgical path planning method based on multimodal image registration according to any one of claims 1 to 6 is implemented.

Images