Interventional guidance method and device based on multi-modal image fusion

Abstract

The invention discloses an intervention guiding method and device based on multi-modal image fusion. The method comprises the following steps: acquiring a multi-modal image; the multi-modal image comprises a CT image, a magnetic resonance image and an ultrasonic image; the multi-modal image is preprocessed, and a preprocessed multi-modal image is obtained; fusing the preprocessed multi-modal image to obtain a fused image; and processing the fused image to obtain interventional guidance path information. According to the method, rich image information is obtained by fusing multi-modal images, and segmented images are obtained by performing image segmentation on the fused image; performing three-dimensional reconstruction on the segmented image to obtain a three-dimensional reconstruction model; and performing path planning on the three-dimensional reconstruction model to obtain interventional guidance path information. The method can assist a doctor in accurately positioning a focus area, and the interventional treatment effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and applications, and particularly to an intervention guidance method and device based on multimodal image fusion. Background Art

[0002] Interventional surgery is the most commonly used method for disease diagnosis and treatment in clinical practice at present. However, during the surgery, the patient needs to be scanned continuously to determine the relative position between the lesion and the interventional needle, resulting in multiple pauses during the surgery and low interventional accuracy. Long-term surgery will also increase the radiation dose. Therefore, accurate positioning of the lesion and planning of the needle insertion path of the interventional needle are of great significance.

[0003] By combining image information of different sensors or modalities (such as infrared and visible light, MRI and CT), and enhancing complementarity through feature-level or decision-level fusion, an information-rich fused image can be generated. The problems existing in the existing fusion methods include: insufficient processing of feature differences, the traditional algorithm lacks the adaptive ability to the feature differences of multi-source images, and forcibly adopts the same transformation to extract features, resulting in limited expression ability; noise and distortion, the spatial domain method is prone to spectral distortion, and the transform domain method may introduce noise or loss of high-frequency information during the fusion process; the fusion strategy is rough, the traditional feature fusion relies on artificial rules (such as weighted average), and it is difficult to achieve fine-grained fusion in complex scenarios. Therefore, it is of great significance to study multimodal image fusion methods and plan the path of the interventional needle based on the fused image. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an intervention guidance method and device based on multimodal image fusion, which fuse CT, MRI (magnetic resonance imaging), and ultrasound images of the liver to obtain a fused image, perform image segmentation on the fused image to obtain a segmented image, perform three-dimensional reconstruction on the segmented image to obtain a three-dimensional reconstruction model, perform path planning on the three-dimensional reconstruction model to obtain the needle insertion path of the interventional needle, and assist in guiding the interventional treatment of liver lesions according to the needle insertion path.

[0005] To solve the above technical problems, a first aspect of an embodiment of the present invention discloses an intervention guidance method based on multimodal image fusion, and the method includes:

[0006] S1, obtaining multimodal images; the multimodal images include CT images, magnetic resonance images, and ultrasound images;

[0007] S2, preprocessing the multimodal images to obtain preprocessed multimodal images;

[0008] S3, fusing the preprocessed multimodal images to obtain a fused image;

[0009] S4. Process the fused image to obtain intervention guidance path information.

[0010] As an optional implementation manner, in the first aspect of the embodiments of the present invention, the preprocessing of the multimodal image to obtain a preprocessed multimodal image includes:

[0011] S21. Filter the multimodal image to obtain a filtered multimodal image;

[0012] S22. Enhance the filtered multimodal image to obtain an enhanced multimodal image;

[0013] S23. Perform data registration on the enhanced multimodal image to obtain a preprocessed multimodal image.

[0014] As an optional implementation manner, in the first aspect of the embodiments of the present invention, the filtering of the multimodal image to obtain a filtered multimodal image includes:

[0015] S211. Perform Gaussian filtering on the multimodal image to obtain a first multimodal image;

[0016] S212. Perform grayscale conversion on the first multimodal image to obtain a second multimodal image;

[0017] S213. Denoise the second multimodal image to obtain a third multimodal image;

[0018] S214. Perform morphological closing operation on the third multimodal image to obtain a filtered multimodal image.

[0019] As an optional implementation manner, in the first aspect of the embodiments of the present invention, the enhancement of the filtered multimodal image to obtain an enhanced multimodal image includes:

[0020] S221. Obtain a high-resolution reference image;

[0021] S222. Process the high-resolution reference image to obtain a low-resolution reference image;

[0022] S223. Extract features from the filtered multimodal image, the high-resolution reference image, and the low-resolution reference image to obtain multimodal feature information, high-resolution feature information, and low-resolution feature information;

[0023] S224. Process the multimodal feature information and the low-resolution feature information to obtain reference information;

[0024] S225. Process the high-resolution feature information and the multi-modal feature information according to the reference information to obtain an enhanced multi-modal image.

[0025] As an optional implementation manner, in the first aspect of the embodiments of the present invention, the data registration of the enhanced multi-modal image to obtain a preprocessed multi-modal image includes:

[0026] S231. Process the enhanced multi-modal image to obtain a set of matching feature point pairs;

[0027] S232. Process the set of matching feature point pairs to obtain an optimal transformation matrix;

[0028] S233. Process the optimal transformation matrix to obtain a projection matrix;

[0029] S234. Use the projection matrix to perform data registration on the enhanced multi-modal image to obtain a preprocessed multi-modal image; the preprocessed multi-modal image includes a preprocessed CT image, a preprocessed magnetic resonance image, and a preprocessed ultrasound image.

[0030] As an optional implementation manner, in the first aspect of the embodiments of the present invention, the fusion of the preprocessed multi-modal image to obtain a fused image includes:

[0031] S31. Use a feature extraction model to extract features from the preprocessed multi-modal image to obtain shallow multi-modal image features and deep multi-modal image features;

[0032] S32. Use a feature filtering model to process the shallow multi-modal image features and the deep multi-modal image features to obtain fused features;

[0033] S33. Use a fusion model to process the fused features to obtain a fused image.

[0034] As an optional implementation manner, in the first aspect of the embodiments of the present invention, the processing of the fused image to obtain intervention guidance path information includes:

[0035] S41. Perform image segmentation on the fused image to obtain a segmented image;

[0036] S42. Perform three-dimensional reconstruction on the segmented image to obtain a three-dimensional reconstruction model;

[0037] S43. Perform path planning on the three-dimensional reconstruction model to obtain intervention guidance path information.

[0038] The second aspect of the embodiments of the present invention discloses an intervention guidance device based on multi-modal image fusion, and the device includes:

[0039] An image acquisition module for acquiring multimodal images; the multimodal images include CT images, magnetic resonance images, and ultrasound images;

[0040] A preprocessing module for preprocessing the multimodal images to obtain preprocessed multimodal images;

[0041] An image fusion module for fusing the preprocessed multimodal images to obtain a fused image;

[0042] An interventional guidance path planning module for processing the fused image to obtain interventional guidance path information.

[0043] As an optional implementation manner, in the second aspect of the embodiments of the present invention, the preprocessing of the multimodal images to obtain preprocessed multimodal images includes:

[0044] S21, filtering the multimodal images to obtain filtered multimodal images;

[0045] S22, enhancing the filtered multimodal images to obtain enhanced multimodal images;

[0046] S23, performing data registration on the enhanced multimodal images to obtain preprocessed multimodal images.

[0047] As an optional implementation manner, in the second aspect of the embodiments of the present invention, the filtering of the multimodal images to obtain filtered multimodal images includes:

[0048] S211, performing Gaussian filtering on the multimodal images to obtain first multimodal images;

[0049] S212, performing grayscale conversion on the first multimodal images to obtain second multimodal images;

[0050] S213, performing denoising processing on the second multimodal images to obtain third multimodal images;

[0051] S214, performing morphological closing operation on the third multimodal images to obtain filtered multimodal images.

[0052] As an optional implementation manner, in the second aspect of the embodiments of the present invention, the enhancing of the filtered multimodal images to obtain enhanced multimodal images includes:

[0053] S221, obtaining a high-resolution reference image;

[0054] S222, processing the high-resolution reference image to obtain a low-resolution reference image;

[0055] S223. Extract features from the filtered multi-modal image, the high-resolution reference image, and the low-resolution reference image to obtain multi-modal feature information, high-resolution feature information, and low-resolution feature information;

[0056] S224. Process the multi-modal feature information and the low-resolution feature information to obtain reference information;

[0057] S225. Process the high-resolution feature information and the multi-modal feature information according to the reference information to obtain an enhanced multi-modal image.

[0058] As an optional implementation manner, in the second aspect of the embodiments of the present invention, the step of performing data registration on the enhanced multi-modal image to obtain a preprocessed multi-modal image includes:

[0059] S231. Process the enhanced multi-modal image to obtain a set of matching feature point pairs;

[0060] S232. Process the set of matching feature point pairs to obtain an optimal transformation matrix;

[0061] S233. Process the optimal transformation matrix to obtain a projection matrix;

[0062] S234. Use the projection matrix to perform data registration on the enhanced multi-modal image to obtain a preprocessed multi-modal image; the preprocessed multi-modal image includes a preprocessed CT image, a preprocessed magnetic resonance image, and a preprocessed ultrasound image.

[0063] As an optional implementation manner, in the second aspect of the embodiments of the present invention, the step of fusing the preprocessed multi-modal image to obtain a fused image includes:

[0064] S31. Use a feature extraction model to extract features from the preprocessed multi-modal image to obtain shallow multi-modal image features and deep multi-modal image features;

[0065] S32. Use a feature filtering model to process the shallow multi-modal image features and the deep multi-modal image features to obtain fused features;

[0066] S33. Use a fusion model to process the fused features to obtain a fused image.

[0067] As an optional implementation manner, in the second aspect of the embodiments of the present invention, the step of processing the fused image to obtain intervention guidance path information includes:

[0068] S41. Perform image segmentation on the fused image to obtain a segmented image;

[0069] S42. Perform three-dimensional reconstruction on the segmented image to obtain a three-dimensional reconstruction model;

[0070] S43. Perform path planning on the three-dimensional reconstruction model to obtain intervention guidance path information.

[0071] The third aspect of the present invention discloses another intervention guidance device based on multi-modal image fusion. The device includes:

[0072] A memory storing executable program code;

[0073] A processor coupled to the memory;

[0074] The processor calls the executable program code stored in the memory and executes some or all of the steps in the intervention guidance method based on multi-modal image fusion disclosed in the first aspect of the embodiments of the present invention.

[0075] The fourth aspect of the present invention discloses a computer-readable storage medium. The computer-readable storage medium stores computer instructions, which are used to execute some or all of the steps in the intervention guidance method based on multi-modal image fusion disclosed in the first aspect of the embodiments of the present invention when called.

[0076] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0077] The method of the present invention fuses multi-modal images to obtain rich image information, performs image segmentation on the fused image to obtain a segmented image, performs three-dimensional reconstruction on the segmented image to obtain a three-dimensional reconstruction model, and performs path planning on the three-dimensional reconstruction model to obtain intervention guidance path information. The method of the present invention can assist doctors in accurately positioning the lesion area and improving the intervention treatment effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0079] Figure 1 is a flowchart of an intervention guidance method based on multi-modal image fusion disclosed in the embodiments of the present invention;

[0080] Figure 2 is a schematic diagram of the fusion model structure disclosed in the embodiments of the present invention;

[0081] Figure 3 is a schematic diagram of the segmentation model structure disclosed in the embodiments of the present invention;

[0082] Figure 4 It is a schematic structural diagram of an intervention guidance device based on multi-modal image fusion disclosed in an embodiment of the present invention;

[0083] Figure 5 It is a schematic structural diagram of another intervention guidance device based on multi-modal image fusion disclosed in an embodiment of the present invention. Detailed implementation manners

[0084] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope of protection of the present invention.

[0085] The terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or equipment.

[0086] Referring to "embodiments" herein means that a particular feature, structure, or characteristic described in connection with the embodiments can be included in at least one embodiment of the present invention. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0087] The present invention discloses an intervention guidance method and device based on multimodal image fusion. The method includes: acquiring multimodal images; the multimodal images include CT images, magnetic resonance images, and ultrasound images; preprocessing the multimodal images to obtain preprocessed multimodal images; fusing the preprocessed multimodal images to obtain a fused image; and processing the fused image to obtain intervention guidance path information. The method of the present invention obtains rich image information by fusing multimodal images, performs image segmentation on the fused image to obtain a segmented image; performs three-dimensional reconstruction on the segmented image to obtain a three-dimensional reconstruction model; and performs path planning on the three-dimensional reconstruction model to obtain intervention guidance path information. The method of the present invention can assist doctors in accurately positioning the lesion area and improving the intervention treatment effect. The following will be described in detail respectively.

[0088] Embodiment 1

[0089] Please refer to Figure 1 , Figure 1 which is a flowchart of an intervention guidance method based on multimodal image fusion disclosed in an embodiment of the present invention. Among them, Figure 1 the described intervention guidance method based on multimodal image fusion is applied to the field of image processing and application technologies, and the embodiment of the present invention does not make any limitations. As Figure 1 shown, the intervention guidance method based on multimodal image fusion may include the following operations:

[0090] S1. Acquire multimodal images; the multimodal images include CT images, magnetic resonance images, and ultrasound images;

[0091] The multimodal images include liver CT images, liver magnetic resonance images, and liver ultrasound images;

[0092] S2. Preprocess the multimodal images to obtain preprocessed multimodal images;

[0093] The preprocessed multimodal images include preprocessed CT images, preprocessed magnetic resonance images, and preprocessed ultrasound images;

[0094] S3. Fuse the preprocessed multimodal images to obtain a fused image;

[0095] S4. Process the fused image to obtain intervention guidance path information.

[0096] Optionally, the preprocessing the multimodal images to obtain preprocessed multimodal images includes:

[0097] S21. Filter the multimodal images to obtain filtered multimodal images;

[0098] The filtered multimodal images include filtered CT images, filtered magnetic resonance images, and filtered ultrasound images;

[0099] S22. Enhance the filtered multimodal images to obtain enhanced multimodal images;

[0100] The enhanced multimodal images include enhanced CT images, enhanced magnetic resonance images, and enhanced ultrasound images;

[0101] S23. Perform data registration on the enhanced multimodal images to obtain preprocessed multimodal images.

[0102] Optionally, the filtering of the multimodal images to obtain filtered multimodal images includes:

[0103] S211. Perform Gaussian filtering on the multimodal images to obtain first multimodal images;

[0104] S212. Perform gray-scale conversion on the first multimodal images to obtain second multimodal images;

[0105] S213. Denoise the second multimodal images to obtain third multimodal images;

[0106] In this embodiment, the denoising method is the pyramid denoising method;

[0107] S214. Perform morphological closing operations on the third multimodal images to obtain filtered multimodal images.

[0108] Optionally, the enhancing of the filtered multimodal images to obtain enhanced multimodal images includes:

[0109] S221. Obtain a high-resolution reference image;

[0110] When acquiring multimodal images in step S1, multiple CT images, magnetic resonance images, and ultrasound images can be collected; one of them is selected as the high-resolution reference image;

[0111] S222. Process the high-resolution reference image to obtain a low-resolution reference image;

[0112] The method is: perform bicubic downsampling on the high-resolution reference image to reduce its resolution, and then scale it to the same size as the high-resolution reference image to obtain a low-resolution reference image;

[0113] S223. Extract features from the filtered multimodal images, the high-resolution reference image, and the low-resolution reference image to obtain multimodal feature information, high-resolution feature information, and low-resolution feature information;

[0114] Using VGG19 as the network for image feature extraction, multi-modal feature information F1, high-resolution feature information F2, and low-resolution feature information F3 are obtained;

[0115] S224, process the multi-modal feature information and the low-resolution feature information to obtain reference information;

[0116] The multi-modal feature information and the low-resolution feature information are partitioned. Each of the multi-modal feature information and the low-resolution feature information obtains M square blocks. Calculate the similarity of the corresponding square blocks of the multi-modal feature information and the low-resolution feature information, and the similarity is the reference information.

[0117] L i (i ∈ [1, H LR × W LR ) and R2 j (j ∈ [1, H R2 × W R2 ) represent the square blocks of the multi-modal feature information and the low-resolution feature information respectively;

[0118] The similarity between the square blocks is denoted as r i,j :

[0119]

[0120] r i,j The larger the value of r indicates the higher the similarity degree of the two square blocks;

[0121] S225, according to the reference information, process the high-resolution feature information and the multi-modal feature information to obtain an enhanced multi-modal image;

[0122] Replace the corresponding area in the multi-modal feature information with the square blocks whose similarity is greater than the preset threshold to obtain an enhanced multi-modal image.

[0123] Optionally, the data registration of the enhanced multi-modal image to obtain a preprocessed multi-modal image includes:

[0124] S231, process the enhanced multi-modal image to obtain a set of matching feature point pairs;

[0125] The pixel points in the image form a pixel set, which is the set of matching feature point pairs;

[0126] S232, process the set of matching feature point pairs to obtain the optimal transformation matrix H;

[0127] The method is:

[0128] Assume x = [x y] TFor a point in the set I of matching feature point pairs, its matching point in the preset enhanced multimodal image I′ is x′ = [x′ y′] T , and the corresponding relationship between them is:

[0129]

[0130] where H is the optimal transformation matrix between the two images, and ∼ means equal at a certain scale. and are the homogeneous coordinate forms of x and x′ respectively. Two linear equations can be obtained from the above formula:

[0131]

[0132] where h1 ∼ h8 are the 8 degrees of freedom of the optimal transformation matrix. Using direct linear transformation, the above formula can be further written as:

[0133] ah = 0

[0134] where is the linear parameter matrix of the known variable relationship of the feature matching point pair and , and h = (h1 h2... h8 1) T is the vector form of the optimal transformation matrix H.

[0135] S233, process the optimal transformation matrix to obtain the projection matrix;

[0136] Transform the set I of matching feature point pairs from the (x, y) coordinate system to the (u, v) coordinate system, and the transformation relationship is:

[0137]

[0138] θ = atan2(-h8, -h7)

[0139] After the coordinate transformation, the projection matrix from (u, v) to (x′, y′) can be obtained:

[0140]

[0141] where

[0142]

[0143] is the vector form of the optimal transformation matrix .

[0144] S234. Use the projection matrix to perform data registration on the enhanced multimodal image to obtain a preprocessed multimodal image; the preprocessed multimodal image includes a preprocessed CT image, a preprocessed magnetic resonance image, and a preprocessed ultrasound image.

[0145] Use two straight lines of u = u1 and u = u2 to divide the enhanced multimodal image to be registered into an overlapping region R H ={(u, v)|u ≤ u1}, a transition region R T ={(u, v)|u1 < u < u2} and a non-overlapping region R S ={(u, v)|u2 ≤ u}. Define the partition transformation function w(u, v) as:

[0146]

[0147] Among them, the projection method of the transition region transformation function T(u, v) gradually changes from H(u, v) to S(u, v). Among them

[0148]

[0149] And S x (u, v), S y (u, v), T x (u, v), T y (u, v) all conform to the general formula of the ruled surface, so the w(u, v) function also satisfies the general formula of the ruled surface:

[0150]

[0151] Among them

[0152]

[0153]

[0154] F y (u) and G y (u) can be obtained by a similar method. To ensure the continuity of w(u, v), it is required that F x (u), G x (u), F y (u) and G y (u) are all continuous. According to the continuity of F x (u), the following equation can be obtained:

[0155]

[0156] Solve the overdetermined equation to obtain f x and β. Similarly, according to G x (u), F y(u), G y Solve for the remaining parameters by the continuity of (u).

[0157] When determining u1 and u2, it is mainly based on the characteristic of maintaining the shape of the transformation function, making the transformation function as close as possible to the similarity transformation. For this purpose, define each image I i The cost function E i , which measures the deviation of its transformation function w i from the nearest similarity transformation.

[0158]

[0159] where Ω i is a rectangular area in the image I i ; J i (x, y; u1, u2) is the Jacobian matrix of w i at (x, y); when splicing two images, the total deviation E(u1, u2) = E1(u1, u2) + E2(u1, u2), where w2 = w.

[0160] Optionally, the fusion of the preprocessed multimodal image to obtain a fused image includes:[[]]

[0161] S31, using a feature extraction model to extract features from the preprocessed multimodal image to obtain shallow multimodal image features and deep multimodal image features;

[0162] Specifically:[[]]

[0163]

[0164] d j = u j-1 - u j , (j = 1, …, N - 1)

[0165]

[0166] d j = u j-1 - u j , (j = N)

[0167] where u j represents the j-level filtered image, RGF represents the guided filter, Gaussian is the Gaussian filter, d j represents the j-level detail layer, N represents the number of decomposition layers, and the initial image u 0 represents the input image I, and the base layer B (i.e., u N ) is obtained by Gaussian filtering, set An increasingly rough image can be obtained. The input preprocessed multi-modal image I1 can be decomposed into shallow features u of the multi-modal image 1 , u 2 ,..., u N and deep features d of the multi-modal image 1 , d 2 ,..., d N .

[0168] S32. Using a feature filtering model, process the shallow features and deep features of the multi-modal image to obtain fused features;

[0169] Subtract the shallow features and deep features of the multi-modal image element by element to obtain a difference feature. The difference feature after subtraction is the complementary feature of the shallow features and deep features of the multi-modal image, representing their respective dominant features. Then multiply this difference feature by the corresponding shallow features and deep features of the multi-modal image to obtain fused features, including shallow fused multi-modal image features and deep fused multi-modal image features.

[0170] S33. Using a fusion model, process the fused features to obtain a fused image.

[0171] The fusion model is as Figure 2 shown.

[0172] Optionally, processing the fused image to obtain intervention guidance path information includes:

[0173] S41. Perform image segmentation on the fused image to obtain a segmented image;

[0174] The image segmentation is implemented by an image segmentation model, which consists of two parts: a contracting path and an expanding path. The contracting path downsamples to extract local features of the image, and the expanding path accurately locates the features of the image based on context information. The segmentation model is as Figure 3 shown. The convolutional block attention module (CBAM) assigns weights to each feature map in the contracting path. The expanding path also includes four modules, each of which includes a transposed convolution of size 2×2 and is connected to the feature map weighted from the contracting path through the CBAM module. Then, after passing through the ARB module, and finally through a convolutional layer and a Sigmoid function to obtain the segmentation map of the model.

[0175] S42. Perform three-dimensional reconstruction on the segmented image to obtain a three-dimensional reconstruction model;

[0176] Process the segmented image to obtain a three-dimensional point cloud map; perform three-dimensional reconstruction on the three-dimensional point cloud map to obtain a three-dimensional reconstruction model;

[0177] The three-dimensional reconstruction method is an existing technology in the art, and this embodiment does not impose any limitations.

[0178] S43. Perform path planning on the three-dimensional reconstruction model to obtain intervention guidance path information.

[0179] S431. Analyze the three-dimensional reconstruction model to obtain starting point coordinate information and ending point coordinate information.

[0180] S432. Set the cost function and constraint conditions for the path planning problem.

[0181] F tr = ω4f fu + ω5f hi + ω6f da

[0182] In the formula, ω4, ω5, and ω6 are the weight factors of the path length cost f fu , the path height cost f hi , and the risk cost f da , respectively, and ω4 + ω5 + ω6 = 1. F tr is the cost function.

[0183] S433. Generate the initial population of the particle swarm using the Logistic chaotic map.

[0184] S434. Perform iterative search using the artificial bee colony algorithm, and update the individual positions to move in the direction of lower fitness values.

[0185] S435. Search the local solution space using the chaotic map to jump out of the local optimum.

[0186] S436. Repeat S434 and S435 until the maximum number of iterations is reached or the fitness value change threshold is reached.

[0187] S437. Output the optimal fitness value to obtain the intervention guidance path information.

[0188] It can be seen that the method of the present invention fuses multi-modal images to obtain rich image information, performs image segmentation on the fused image to obtain a segmented image, performs three-dimensional reconstruction on the segmented image to obtain a three-dimensional reconstruction model, and performs path planning on the three-dimensional reconstruction model to obtain intervention guidance path information. The method of the present invention can assist doctors in accurately locating the lesion area and improving the intervention treatment effect.

[0189] Embodiment 2

[0190] Please refer to Figure 4 , Figure 4It is a schematic structural diagram of an intervention guidance device based on multimodal image fusion disclosed in an embodiment of the present invention. Among them, Figure 4 The described intervention guidance device based on multimodal image fusion is applied to the field of image processing and application technologies, which is not limited in the embodiments of the present invention. As Figure 4 shown, the intervention guidance device based on multimodal image fusion may include the following operations:

[0191] S301, an image acquisition module, configured to acquire multimodal images; the multimodal images include CT images, magnetic resonance images, and ultrasonic images;

[0192] S302, a preprocessing module, configured to preprocess the multimodal images to obtain preprocessed multimodal images;

[0193] S303, an image fusion module, configured to fuse the preprocessed multimodal images to obtain a fused image;

[0194] S304, an intervention guidance path planning module, configured to process the fused image to obtain intervention guidance path information.

[0195] Embodiment Three

[0196] Please refer to Figure 5 , Figure 5 It is a schematic structural diagram of another intervention guidance device based on multimodal image fusion disclosed in an embodiment of the present invention. Among them, Figure 5 The described intervention guidance device based on multimodal image fusion is applied to the field of image processing and application technologies, which is not limited in the embodiments of the present invention. As Figure 5 shown, the intervention guidance device based on multimodal image fusion may include the following:

[0197] A memory 401 storing executable program code;

[0198] A processor 402 coupled to the memory 401;

[0199] The processor 402 calls the executable program code stored in the memory 401 and is configured to execute the steps in the intervention guidance method based on multimodal image fusion described in Embodiment One.

[0200] Embodiment Four

[0201] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps in the intervention guidance method based on multimodal image fusion described in Embodiment One.

[0202] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0203] Through the above specific descriptions of the embodiments, those skilled in the art can clearly understand that each implementation can be achieved by means of software plus a necessary general hardware platform, and of course, it can also be achieved by hardware. Based on such an understanding, the essence of the above technical solution or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium. The storage medium includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc memories, magnetic disk memories, tape memories, or any other computer-readable medium capable of carrying or storing data.

[0204] Finally, it should be noted that: The disclosure of an intervention guidance method and device based on multimodal image fusion disclosed in the embodiments of the present invention is only the preferred embodiments of the present invention, and is only used to illustrate the technical solutions of the present invention, rather than limiting it; Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intervention guidance method based on multimodal image fusion, characterized in that The method includes: S1. Obtain multimodal images; the multimodal images include CT images, magnetic resonance images, and ultrasound images; S2. Preprocess the multimodal images to obtain preprocessed multimodal images; S3. Fuse the preprocessed multimodal images to obtain a fused image; S4. Process the fused image to obtain interventional guidance path information.

2. The intervention guidance method based on multi-modal image fusion according to claim 1, wherein The preprocessing of the multimodal images to obtain preprocessed multimodal images includes: S21. Filter the multimodal images to obtain filtered multimodal images; S22. Enhance the filtered multimodal images to obtain enhanced multimodal images; S23. Perform data registration on the enhanced multimodal images to obtain preprocessed multimodal images.

3. The intervention guidance method based on multimodal image fusion according to claim 2, wherein The filtering of the multimodal images to obtain filtered multimodal images includes: S211. Perform Gaussian filtering on the multimodal images to obtain first multimodal images; S212. Perform gray-scale conversion on the first multimodal images to obtain second multimodal images; S213. Denoise the second multimodal images to obtain third multimodal images; S214. Perform morphological closing operations on the third multimodal images to obtain filtered multimodal images.

4. The intervention guidance method based on multi-modal image fusion according to claim 2, wherein, The enhancement of the filtered multimodal images to obtain enhanced multimodal images includes: S221. Obtain a high-resolution reference image; S222. Process the high-resolution reference image to obtain a low-resolution reference image; S223. Extract features from the filtered multimodal images, the high-resolution reference image, and the low-resolution reference image to obtain multimodal feature information, high-resolution feature information, and low-resolution feature information; S224. Process the multimodal feature information and the low-resolution feature information to obtain reference information; S225. According to the reference information, process the high-resolution feature information and the multimodal feature information to obtain enhanced multimodal images.

5. The intervention guidance method based on multi-modal image fusion according to claim 2, wherein The data registration of the enhanced multimodal images to obtain preprocessed multimodal images includes: S231. Process the enhanced multimodal images to obtain a set of matching feature point pairs; S232. Process the set of matching feature point pairs to obtain an optimal transformation matrix; S233. Process the optimal transformation matrix to obtain a projection matrix; S234. Use the projection matrix to perform data registration on the enhanced multimodal images to obtain preprocessed multimodal images; the preprocessed multimodal images include preprocessed CT images, preprocessed magnetic resonance images, and preprocessed ultrasound images.

6. The intervention guidance method based on multi-modal image fusion according to claim 1, characterized in that, The fusion of the preprocessed multimodal images to obtain a fused image includes: S31. Use a feature extraction model to extract features from the preprocessed multimodal images to obtain shallow multimodal image features and deep multimodal image features; S32. Use a feature filtering model to process the shallow multimodal image features and the deep multimodal image features to obtain fused features; S33. Use a fusion model to process the fused features to obtain a fused image.

7. The intervention guidance method based on multimodal image fusion according to claim 1, characterized in that Processing the fused image to obtain intervention guidance path information includes: S41. Performing image segmentation on the fused image to obtain a segmented image; S42. Performing three-dimensional reconstruction on the segmented image to obtain a three-dimensional reconstruction model; S43. Performing path planning on the three-dimensional reconstruction model to obtain intervention guidance path information.

8. An interventional guidance device based on multimodal image fusion, characterized in that, The device includes: An image acquisition module for acquiring multi-modal images; the multi-modal images include CT images, magnetic resonance images, and ultrasound images; A preprocessing module for preprocessing the multi-modal images to obtain preprocessed multi-modal images; An image fusion module for fusing the preprocessed multi-modal images to obtain a fused image; An intervention guidance path planning module for processing the fused image to obtain intervention guidance path information.

9. An interventional guidance device based on multimodal image fusion, characterized in that, The device includes: A memory storing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory and executes the intervention guidance method based on multi-modal image fusion according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions which, when called, are used to execute the intervention guidance method based on multi-modal image fusion according to any one of claims 1-7.

Images