Intervention guidance method and device based on multimodal image fusion

Abstract

The present invention discloses an interventional guidance method and device based on multimodal image fusion, the method comprising: acquiring multimodal images; the multimodal images comprising CT images, magnetic resonance images, and ultrasound images; preprocessing the multimodal images to obtain preprocessed multimodal images; fusing the preprocessed multimodal images to obtain a fused image; and processing the fused image to obtain interventional guidance path information. The method of the present invention obtains rich image information by fusing multimodal images, obtains a segmented image by segmenting the fused image, performs three-dimensional reconstruction on the segmented image to obtain a three-dimensional reconstruction model, and performs path planning on the three-dimensional reconstruction model to obtain interventional guidance path information. The method of the present invention can assist doctors in accurately locating the lesion area and improve the effectiveness of interventional treatment.

Description

Technical Field

[0001] The present invention relates to the field of image processing and application technology, and in particular to an intervention guidance method and device based on multimodal image fusion. Background Art

[0002] Interventional surgery is currently the most common method for clinical diagnosis and treatment of diseases. However, during surgery, the patient must be continuously scanned to determine the relative position of the lesion and the interventional needle. This results in multiple pauses during the procedure and low interventional accuracy. Longer procedures also increase radiation exposure. Therefore, precise lesion location and needle insertion path planning are crucial.

[0003] Combining image information from different sensors or modalities (such as infrared and visible light, MRI and CT) and enhancing complementarity through feature-level or decision-level fusion can produce information-rich fused images. Problems with existing fusion methods include: insufficient processing of feature differences. Traditional algorithms lack the ability to adapt to feature differences in multi-source images and enforce the use of the same transform to extract features, resulting in limited expressiveness; noise and distortion. Spatial domain methods are prone to spectral distortion, while transform domain methods may introduce noise or lose high-frequency information during the fusion process; and crude fusion strategies. Traditional feature fusion relies on artificial rules (such as weighted averaging), making it difficult to achieve refined fusion in complex scenarios. Therefore, it is of great significance to study multimodal image fusion methods and use fused images for interventional needle path planning. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an interventional guidance method and device based on multimodal image fusion, which fuses CT, MRI (magnetic resonance imaging), and ultrasound images of the liver to obtain a fused image, performs image segmentation on the fused image to obtain a segmented image, performs three-dimensional reconstruction on the segmented image to obtain a three-dimensional reconstructed model, performs path planning on the three-dimensional reconstructed model to obtain the needle insertion path of the interventional needle, and assists in guiding the interventional treatment of liver lesions according to the needle insertion path.

[0005] In order to solve the above technical problems, the first aspect of the embodiments of the present invention discloses an intervention guidance method based on multimodal image fusion, the method comprising:

[0006] S1, acquiring a multimodal image; the multimodal image includes a CT image, a magnetic resonance image, and an ultrasound image;

[0007] S2, preprocessing the multimodal image to obtain a preprocessed multimodal image;

[0008] S3, fusing the preprocessed multimodal images to obtain a fused image;

[0009] S4: Process the fused image to obtain intervention guidance path information.

[0010] As an optional implementation manner, in the first aspect of the embodiments of the present invention, preprocessing the multimodal image to obtain a preprocessed multimodal image includes:

[0011] S21, filtering the multimodal image to obtain a filtered multimodal image;

[0012] S22, enhancing the filtered multimodal image to obtain an enhanced multimodal image;

[0013] S23, performing data registration on the enhanced multimodal image to obtain a preprocessed multimodal image.

[0014] As an optional implementation manner, in the first aspect of the embodiments of the present invention, filtering the multimodal image to obtain a filtered multimodal image includes:

[0015] S211, performing Gaussian filtering on the multimodal image to obtain a first multimodal image;

[0016] S212, performing grayscale conversion on the first multimodal image to obtain a second multimodal image;

[0017] S213, performing denoising processing on the second multimodal image to obtain a third multimodal image;

[0018] S214: Perform a morphological closing operation on the third multimodal image to obtain a filtered multimodal image.

[0019] As an optional implementation manner, in the first aspect of the embodiments of the present invention, enhancing the filtered multimodal image to obtain an enhanced multimodal image includes:

[0020] S221, acquiring a high-resolution reference image;

[0021] S222, processing the high-resolution reference image to obtain a low-resolution reference image;

[0022] S223, performing feature extraction on the filtered multimodal image, the high-resolution reference image, and the low-resolution reference image to obtain multimodal feature information, high-resolution feature information, and low-resolution feature information;

[0023] S224, processing the multimodal feature information and the low-resolution feature information to obtain reference information;

[0024] S225 : Process the high-resolution feature information and the multimodal feature information according to the reference information to obtain an enhanced multimodal image.

[0025] As an optional implementation manner, in the first aspect of the embodiment of the present invention, performing data registration on the enhanced multimodal image to obtain a preprocessed multimodal image includes:

[0026] S231, processing the enhanced multimodal image to obtain a set of matching feature point pairs;

[0027] S232, processing the set of matching feature point pairs to obtain an optimal transformation matrix;

[0028] S233, processing the optimal transformation matrix to obtain a projection matrix;

[0029] S234, using the projection matrix, performing data registration on the enhanced multimodal image to obtain a preprocessed multimodal image; the preprocessed multimodal image includes a preprocessed CT image, a preprocessed magnetic resonance image, and a preprocessed ultrasound image.

[0030] As an optional implementation manner, in the first aspect of the embodiments of the present invention, fusing the preprocessed multimodal images to obtain a fused image includes:

[0031] S31, using a feature extraction model to perform feature extraction on the preprocessed multimodal image to obtain shallow features and deep features of the multimodal image;

[0032] S32, using a feature filtering model, processing the shallow features and deep features of the multimodal image to obtain fused features;

[0033] S33: Process the fusion features using a fusion model to obtain a fusion image.

[0034] As an optional implementation manner, in the first aspect of the embodiment of the present invention, processing the fused image to obtain intervention guidance path information includes:

[0035] S41, performing image segmentation on the fused image to obtain a segmented image;

[0036] S42, performing three-dimensional reconstruction on the segmented image to obtain a three-dimensional reconstructed model;

[0037] S43: Perform path planning on the three-dimensional reconstructed model to obtain intervention guidance path information.

[0038] A second aspect of an embodiment of the present invention discloses an intervention guidance device based on multimodal image fusion, the device comprising:

[0039] An image acquisition module, configured to acquire multimodal images, including CT images, magnetic resonance images, and ultrasound images;

[0040] A preprocessing module, configured to preprocess the multimodal image to obtain a preprocessed multimodal image;

[0041] An image fusion module, configured to fuse the pre-processed multimodal images to obtain a fused image;

[0042] The intervention guidance path planning module is used to process the fused image to obtain intervention guidance path information.

[0043] As an optional implementation manner, in the second aspect of the embodiments of the present invention, preprocessing the multimodal image to obtain a preprocessed multimodal image includes:

[0044] S21, filtering the multimodal image to obtain a filtered multimodal image;

[0045] S22, enhancing the filtered multimodal image to obtain an enhanced multimodal image;

[0046] S23, performing data registration on the enhanced multimodal image to obtain a preprocessed multimodal image.

[0047] As an optional implementation manner, in the second aspect of the embodiments of the present invention, filtering the multimodal image to obtain a filtered multimodal image includes:

[0048] S211, performing Gaussian filtering on the multimodal image to obtain a first multimodal image;

[0049] S212, performing grayscale conversion on the first multimodal image to obtain a second multimodal image;

[0050] S213, performing denoising processing on the second multimodal image to obtain a third multimodal image;

[0051] S214: Perform a morphological closing operation on the third multimodal image to obtain a filtered multimodal image.

[0052] As an optional implementation manner, in the second aspect of the embodiments of the present invention, the enhancing the filtered multimodal image to obtain the enhanced multimodal image includes:

[0053] S221, acquiring a high-resolution reference image;

[0054] S222, processing the high-resolution reference image to obtain a low-resolution reference image;

[0055] S223, performing feature extraction on the filtered multimodal image, the high-resolution reference image, and the low-resolution reference image to obtain multimodal feature information, high-resolution feature information, and low-resolution feature information;

[0056] S224, processing the multimodal feature information and the low-resolution feature information to obtain reference information;

[0057] S225 : Process the high-resolution feature information and the multimodal feature information according to the reference information to obtain an enhanced multimodal image.

[0058] As an optional implementation manner, in the second aspect of the embodiment of the present invention, performing data registration on the enhanced multimodal image to obtain a preprocessed multimodal image includes:

[0059] S231, processing the enhanced multimodal image to obtain a set of matching feature point pairs;

[0060] S232, processing the set of matching feature point pairs to obtain an optimal transformation matrix;

[0061] S233, processing the optimal transformation matrix to obtain a projection matrix;

[0062] S234, using the projection matrix, performing data registration on the enhanced multimodal image to obtain a preprocessed multimodal image; the preprocessed multimodal image includes a preprocessed CT image, a preprocessed magnetic resonance image, and a preprocessed ultrasound image.

[0063] As an optional implementation manner, in the second aspect of the embodiments of the present invention, fusing the preprocessed multimodal images to obtain a fused image includes:

[0064] S31, using a feature extraction model to perform feature extraction on the preprocessed multimodal image to obtain shallow features and deep features of the multimodal image;

[0065] S32, using a feature filtering model, processing the shallow features and deep features of the multimodal image to obtain fused features;

[0066] S33: Process the fusion features using a fusion model to obtain a fusion image.

[0067] As an optional implementation manner, in the second aspect of the embodiment of the present invention, processing the fused image to obtain intervention guidance path information includes:

[0068] S41, performing image segmentation on the fused image to obtain a segmented image;

[0069] S42, performing three-dimensional reconstruction on the segmented image to obtain a three-dimensional reconstructed model;

[0070] S43: Perform path planning on the three-dimensional reconstructed model to obtain intervention guidance path information.

[0071] A third aspect of the present invention discloses another intervention guidance device based on multimodal image fusion, the device comprising:

[0072] a memory storing executable program code;

[0073] a processor coupled to the memory;

[0074] The processor calls the executable program code stored in the memory to execute part or all of the steps in the intervention guidance method based on multimodal image fusion disclosed in the first aspect of the embodiment of the present invention.

[0075] The fourth aspect of the present invention discloses a computer-storable medium, which stores computer instructions. When the computer instructions are called, they are used to execute some or all of the steps in the intervention guidance method based on multimodal image fusion disclosed in the first aspect of an embodiment of the present invention.

[0076] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0077] The method of the present invention fuses multimodal images to obtain rich image information. The fused image is then segmented to obtain a segmented image. The segmented image is then 3D reconstructed to obtain a 3D reconstructed model. Path planning is then performed on the 3D reconstructed model to obtain interventional guidance path information. This method can assist physicians in accurately locating lesions and improve the effectiveness of interventional treatments. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0079] Figure 1 This is a flow chart of an intervention guidance method based on multimodal image fusion disclosed in an embodiment of the present invention;

[0080] Figure 2 is a schematic diagram of the fusion model structure disclosed in an embodiment of the present invention;

[0081] Figure 3 is a schematic diagram of the segmentation model structure disclosed in an embodiment of the present invention;

[0082] Figure 4 1 is a schematic structural diagram of an intervention guidance device based on multimodal image fusion disclosed in an embodiment of the present invention;

[0083] Figure 5 It is a structural schematic diagram of another intervention guidance device based on multimodal image fusion disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0084] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0085] The terms "first," "second," and so on, in the description and claims of the present invention and the accompanying drawings are used to distinguish between different objects, not to describe a specific order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or device.

[0086] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0087] The present invention discloses an intervention guidance method and device based on multimodal image fusion, the method comprising: acquiring multimodal images; the multimodal images comprising CT images, magnetic resonance images and ultrasound images; preprocessing the multimodal images to obtain preprocessed multimodal images; fusing the preprocessed multimodal images to obtain a fused image; processing the fused image to obtain intervention guidance path information. The method of the present invention obtains rich image information by fusing multimodal images, obtains segmented images by performing image segmentation on the fused image; performs three-dimensional reconstruction on the segmented image to obtain a three-dimensional reconstruction model; performs path planning on the three-dimensional reconstruction model to obtain intervention guidance path information. The method of the present invention can assist doctors in accurately locating the lesion area and improve the effect of intervention treatment. The following are detailed descriptions of each.

[0088] Example 1

[0089] See also Figure 1 , Figure 1 This is a flow chart of an intervention guidance method based on multimodal image fusion disclosed in an embodiment of the present invention. Figure 1 The described intervention guidance method based on multimodal image fusion is applied to the field of image processing and application technology, and is not limited in the embodiments of the present invention. Figure 1 As shown, the intervention guidance method based on multimodal image fusion may include the following operations:

[0090] S1, acquiring a multimodal image; the multimodal image includes a CT image, a magnetic resonance image, and an ultrasound image;

[0091] Multimodal images include liver CT images, liver magnetic resonance images, and liver ultrasound images;

[0092] S2, preprocessing the multimodal image to obtain a preprocessed multimodal image;

[0093] Preprocessing multimodal images includes preprocessing CT images, preprocessing magnetic resonance images, and preprocessing ultrasound images;

[0094] S3, fusing the preprocessed multimodal images to obtain a fused image;

[0095] S4: Process the fused image to obtain intervention guidance path information.

[0096] Optionally, preprocessing the multimodal image to obtain a preprocessed multimodal image includes:

[0097] S21, filtering the multimodal image to obtain a filtered multimodal image;

[0098] The filtered multimodal images include filtered CT images, filtered magnetic resonance images, and filtered ultrasound images;

[0099] S22, enhancing the filtered multimodal image to obtain an enhanced multimodal image;

[0100] Enhanced multimodal images include enhanced CT images, enhanced magnetic resonance images, and enhanced ultrasound images;

[0101] S23, performing data registration on the enhanced multimodal image to obtain a preprocessed multimodal image.

[0102] Optionally, filtering the multimodal image to obtain a filtered multimodal image includes:

[0103] S211, performing Gaussian filtering on the multimodal image to obtain a first multimodal image;

[0104] S212, performing grayscale conversion on the first multimodal image to obtain a second multimodal image;

[0105] S213, performing denoising processing on the second multimodal image to obtain a third multimodal image;

[0106] In this embodiment, the denoising method is a pyramid denoising method;

[0107] S214: Perform a morphological closing operation on the third multimodal image to obtain a filtered multimodal image.

[0108] Optionally, the enhancing the filtered multimodal image to obtain an enhanced multimodal image includes:

[0109] S221, acquiring a high-resolution reference image;

[0110] When acquiring a multimodal image in step S1, multiple CT images, magnetic resonance images, and ultrasound images may be acquired; one of them is selected as a high-resolution reference image;

[0111] S222, processing the high-resolution reference image to obtain a low-resolution reference image;

[0112] The method comprises: performing bicubic downsampling on the high-resolution reference image to reduce its resolution, and then scaling it to the same size as the high-resolution reference image to obtain a low-resolution reference image;

[0113] S223, performing feature extraction on the filtered multimodal image, the high-resolution reference image, and the low-resolution reference image to obtain multimodal feature information, high-resolution feature information, and low-resolution feature information;

[0114] Using VGG19 as the network for image feature extraction, we obtain multimodal feature information F1, high-resolution feature information F2, and low-resolution feature information F3;

[0115] S224, processing the multimodal feature information and the low-resolution feature information to obtain reference information;

[0116] The multimodal feature information and the low-resolution feature information are divided into blocks, and M square blocks are obtained for each of the multimodal feature information and the low-resolution feature information. The similarity between the square blocks corresponding to the multimodal feature information and the low-resolution feature information is calculated, and the similarity is used as reference information.

[0117] L i (i∈[1,H LR ×W LR ]) and R2 j (j∈[1,H R2 ×W R2 ]) represent square blocks of multimodal feature information and low-resolution feature information respectively;

[0118] The similarity between square blocks is recorded as r i,j :

[0119]

[0120] r i,j The larger the value of , the higher the similarity between the two square blocks;

[0121] S225, processing the high-resolution feature information and the multimodal feature information according to the reference information to obtain an enhanced multimodal image;

[0122] The corresponding areas in the multimodal feature information are replaced with square blocks whose similarity is greater than a preset threshold to obtain an enhanced multimodal image.

[0123] Optionally, performing data registration on the enhanced multimodal image to obtain a preprocessed multimodal image includes:

[0124] S231, processing the enhanced multimodal image to obtain a set of matching feature point pairs;

[0125] The pixels in the image are formed into a pixel set, which is a set of matching feature point pairs;

[0126] S232, processing the set of matching feature point pairs to obtain an optimal transformation matrix H;

[0127] The method is:

[0128] Assume x = [xy] TTo match a point in the feature point pair set I, its matching point in the preset enhanced multimodal image I′ is x′=[x′y′] T , the corresponding relationship between them is:

[0129]

[0130] Where H is the optimal transformation matrix between the two images, and ~ indicates equality on a certain scale. and are the homogeneous coordinate forms of x and x′ respectively. From the above formula, we can get two linear equations:

[0131]

[0132] Among them, h1~h8 are the 8 degrees of freedom of the optimal transformation matrix. Using direct linear transformation, the above formula can be further written as:

[0133] ah=0

[0134] in Matching point pairs for features and The linear parameter matrix of the known variable relationship, h=(h1h2...h8 1) T is the vector form of the optimal transformation matrix H.

[0135] S233, processing the optimal transformation matrix to obtain a projection matrix;

[0136] Transform the matching feature point pair set I from the (x, y) coordinate system to the (u, v) coordinate system. The transformation relationship is:

[0137]

[0138] θ=atan2(-h8,-h7)

[0139] After coordinate transformation, the projection matrix from (u,v) to (x′,y′) can be obtained:

[0140]

[0141] in

[0142]

[0143] is the optimal transformation matrix in vector form.

[0144] S234, using the projection matrix, performing data registration on the enhanced multimodal image to obtain a preprocessed multimodal image; the preprocessed multimodal image includes a preprocessed CT image, a preprocessed magnetic resonance image, and a preprocessed ultrasound image.

[0145] Use the two straight lines u=u1 and u=u2 to divide the multimodal image to be registered into overlapping regions R H ={(u,v)|u≤u1}, transition region R T ={(u,v)|u1<u<u2} and non-overlapping region R S ={(u,v)|u2≤u}. Define the partition transformation function w(u,v) as:

[0146]

[0147] The transition region transformation function T(u,v) gradually transforms its projection mode from H(u,v) to S(u,v).

[0148]

[0149] And S x (u,v),S y (u,v),T x (u,v),T y (u,v) all conform to the ruled surface formula, so the w(u,v) function also satisfies the ruled surface formula:

[0150]

[0151] in

[0152]

[0153]

[0154] F y (u) and G y (u) can be obtained in a similar way. In order to ensure the continuity of w(u,v), it is required that F x (u), G x (u), F y (u) and G y (u) are continuous, according to F x (u) continuity, we can get the following equation:

[0155]

[0156] By solving the overdetermined equation, we can get f x and β, similarly, according to G x (u), F y(u), G y The continuity of (u) is used to solve the remaining parameters.

[0157] When determining u1 and u2, the main consideration is to make the transformation function as close to the similarity transformation as possible based on the shape preservation characteristics of the transformation function. To this end, define each image I i The cost function E i , measure its transformation function w i Deviation from the nearest similarity transformation.

[0158]

[0159] where Ω i is image I i A rectangular area in J i (x,y;u1,u2) is w i Jacobian matrix at (x, y); when stitching two images, the total deviation E(u1,u2) = E1(u1,u2) + E2(u1,u2), where w2=w.

[0160] Optionally, fusing the preprocessed multimodal images to obtain a fused image includes:

[0161] S31, using a feature extraction model to perform feature extraction on the preprocessed multimodal image to obtain shallow features and deep features of the multimodal image;

[0162] Specifically:

[0163]

[0164] d j =u j-1 -u j ,(j=1,…,N-1)

[0165]

[0166] d j =u j-1 -u j ,(j=N)

[0167] where u j represents the j-level filtered image, RGF represents guided filtering, Gaussian represents Gaussian filtering, d j represents the j-level detail layer, N represents the number of decomposition layers, and the initial image u 0 Represents the input image I, the base layer B (i.e. u N ) is obtained by Gaussian filtering, setting The input preprocessed multimodal image I1 can be decomposed into the shallow features u of the multimodal image 1 ,u 2 ,...,u N and multimodal image deep features d 1 ,d 2 ,...,d N .

[0168] S32, using a feature filtering model, processing the shallow features and deep features of the multimodal image to obtain fused features;

[0169] The shallow features and deep features of the multimodal image are subtracted element by element to obtain a difference feature. The difference feature after subtraction is a complementary feature of the shallow features and deep features of the multimodal image, representing the dominant features of each. This difference feature is then multiplied by the corresponding shallow features and deep features of the multimodal image to obtain a fused feature, including the shallow features and deep features of the multimodal fused image.

[0170] S33: Process the fusion features using a fusion model to obtain a fusion image.

[0171] Fusion models such as Figure 2 shown.

[0172] Optionally, processing the fused image to obtain intervention guidance path information includes:

[0173] S41, performing image segmentation on the fused image to obtain a segmented image;

[0174] Image segmentation is achieved by the image segmentation model, which consists of two parts: the contraction path and the expansion path. The contraction path downsamples to extract local features of the image, and the expansion path accurately locates the features of the image based on context information. Figure 3 As shown in the figure, the convolutional block attention module (CBAM) assigns weights to each feature map in the contraction path. The expansion path also contains four modules, each of which includes a 2×2 transposed convolution and is connected to the weighted feature map from the contraction path through the CBAM module. After passing through the ARB module, it finally passes through the convolution layer and the Sigmoid function to obtain the segmentation map of the model.

[0175] S42, performing three-dimensional reconstruction on the segmented image to obtain a three-dimensional reconstructed model;

[0176] Processing the segmented image to obtain a three-dimensional point cloud image; performing three-dimensional reconstruction on the three-dimensional point cloud image to obtain a three-dimensional reconstructed model;

[0177] The three-dimensional reconstruction method is a prior art in this field and is not limited in this embodiment.

[0178] S43: Perform path planning on the three-dimensional reconstructed model to obtain intervention guidance path information.

[0179] S431, analyzing the three-dimensional reconstructed model to obtain starting point coordinate information and ending point coordinate information;

[0180] S432, setting the cost function and constraints of the path planning problem;

[0181] F tr =ω4f fu +ω5f hi +ω6f da

[0182] Where ω4, ω5, and ω6 are the path length costs f fu , path height cost f hi , risk cost f da The weight factor, and ω4+ω5+ω6=1, F tr is the cost function.

[0183] S433, using Logistic chaos mapping to generate the initial population of particle swarm;

[0184] S434, using the artificial bee colony algorithm to perform iterative search and update the individual position so that it moves towards a lower fitness value;

[0185] S435, using chaotic mapping to search the local solution space so as to make it jump out of the local optimal value;

[0186] S436, repeat S434 and S435 until the maximum number of iterations is reached or the fitness value change threshold is reached;

[0187] S437: Output the optimal fitness value and obtain intervention guidance path information.

[0188] As can be seen, the method of the present invention obtains rich image information by fusing multimodal images, then performs image segmentation on the fused image to obtain a segmented image; performs 3D reconstruction on the segmented image to obtain a 3D reconstructed model; and performs path planning on the 3D reconstructed model to obtain interventional guidance path information. This method of the present invention can assist physicians in accurately locating lesion areas and improve the effectiveness of interventional treatment.

[0189] Example 2

[0190] See also Figure 4 , Figure 4: is a schematic diagram of the structure of an intervention guidance device based on multimodal image fusion disclosed in an embodiment of the present invention. Figure 4 The described intervention guidance device based on multimodal image fusion is applied to the field of image processing and application technology, and the embodiments of the present invention do not limit this. Figure 4 As shown, the intervention guidance device based on multimodal image fusion may include the following operations:

[0191] S301, an image acquisition module, configured to acquire a multimodal image; the multimodal image includes a CT image, a magnetic resonance image, and an ultrasound image;

[0192] S302, a preprocessing module, configured to preprocess the multimodal image to obtain a preprocessed multimodal image;

[0193] S303, an image fusion module, configured to fuse the pre-processed multimodal images to obtain a fused image;

[0194] S304, an intervention guidance path planning module, configured to process the fused image to obtain intervention guidance path information.

[0195] Example 3

[0196] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of another intervention guidance device based on multimodal image fusion disclosed in an embodiment of the present invention. Figure 5 The described intervention guidance device based on multimodal image fusion is applied to the field of image processing and application technology, and the embodiments of the present invention do not limit this. Figure 5 As shown, the intervention guidance device based on multimodal image fusion may include the following operations:

[0197] A memory 401 storing executable program code;

[0198] a processor 402 coupled to the memory 401;

[0199] The processor 402 calls the executable program code stored in the memory 401 to execute the steps of the intervention guidance method based on multimodal image fusion described in the first embodiment.

[0200] Example 4

[0201] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps of the intervention guidance method based on multimodal image fusion described in the first embodiment.

[0202] The device embodiments described above are merely illustrative. Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0203] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus the necessary general hardware platform, or of course, by means of hardware. Based on this understanding, the above technical solution, in essence, or the portion that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0204] Finally, it should be noted that the interventional guidance method and device based on multimodal image fusion disclosed in the embodiments of the present invention are only preferred embodiments of the present invention, and are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features therein may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An intervention guidance method based on multimodal image fusion, characterized in that: The method comprises: S1, acquiring a multimodal image; the multimodal image includes a CT image, a magnetic resonance image, and an ultrasound image; S2, preprocessing the multimodal image to obtain a preprocessed multimodal image, including: S21, filtering the multimodal image to obtain a filtered multimodal image; S22, enhancing the filtered multimodal image to obtain an enhanced multimodal image, including: S221, acquiring a high-resolution reference image; S222, processing the high-resolution reference image to obtain a low-resolution reference image; S223, performing feature extraction on the filtered multimodal image, the high-resolution reference image, and the low-resolution reference image to obtain multimodal feature information, high-resolution feature information, and low-resolution feature information; S224, processing the multimodal feature information and the low-resolution feature information to obtain reference information; Divide the multimodal feature information and the low-resolution feature information into blocks, obtaining M square blocks of the multimodal feature information and the low-resolution feature information, and calculate the similarity between the corresponding square blocks of the multimodal feature information and the low-resolution feature information, where the similarity serves as reference information; L i and R2 j The square blocks represent multimodal feature information and low-resolution feature information respectively; The similarity between square blocks is recorded as r i,j : r i,j The larger the value of , the higher the similarity between the two square blocks; S225, processing the high-resolution feature information and the multimodal feature information according to the reference information to obtain an enhanced multimodal image; Replacing the corresponding area in the multimodal feature information with square blocks whose similarity is greater than a preset threshold to obtain an enhanced multimodal image; S23, performing data registration on the enhanced multimodal image to obtain a preprocessed multimodal image; S3, fusing the preprocessed multimodal images to obtain a fused image; S4: Process the fused image to obtain intervention guidance path information.

2. The intervention guidance method based on multimodal image fusion according to claim 1, characterized in that: The filtering of the multimodal image to obtain a filtered multimodal image includes: S211, performing Gaussian filtering on the multimodal image to obtain a first multimodal image; S212, performing grayscale conversion on the first multimodal image to obtain a second multimodal image; S213, performing denoising processing on the second multimodal image to obtain a third multimodal image; S214: Perform a morphological closing operation on the third multimodal image to obtain a filtered multimodal image.

3. The intervention guidance method based on multimodal image fusion according to claim 1, characterized in that: The performing data registration on the enhanced multimodal image to obtain a preprocessed multimodal image includes: S231, processing the enhanced multimodal image to obtain a set of matching feature point pairs; S232, processing the set of matching feature point pairs to obtain an optimal transformation matrix; S233, processing the optimal transformation matrix to obtain a projection matrix; S234, using the projection matrix, performing data registration on the enhanced multimodal image to obtain a preprocessed multimodal image; the preprocessed multimodal image includes a preprocessed CT image, a preprocessed magnetic resonance image, and a preprocessed ultrasound image.

4. The intervention guidance method based on multimodal image fusion according to claim 1, characterized in that: The fusing the pre-processed multimodal images to obtain a fused image includes: S31, using a feature extraction model to perform feature extraction on the preprocessed multimodal image to obtain shallow features and deep features of the multimodal image; S32, using a feature filtering model, processing the shallow features and deep features of the multimodal image to obtain fused features; S33: Process the fusion features using a fusion model to obtain a fusion image.

5. The intervention guidance method based on multimodal image fusion according to claim 1, characterized in that: The processing of the fused image to obtain intervention guidance path information includes: S41, performing image segmentation on the fused image to obtain a segmented image; S42, performing three-dimensional reconstruction on the segmented image to obtain a three-dimensional reconstructed model; S43: Perform path planning on the three-dimensional reconstructed model to obtain intervention guidance path information.

6. An intervention guidance device based on multimodal image fusion, characterized in that: The device comprises: An image acquisition module, configured to acquire multimodal images, including CT images, magnetic resonance images, and ultrasound images; A preprocessing module, configured to preprocess the multimodal image to obtain a preprocessed multimodal image, comprising: S21, filtering the multimodal image to obtain a filtered multimodal image; S22, enhancing the filtered multimodal image to obtain an enhanced multimodal image, including: S221, acquiring a high-resolution reference image; S222, processing the high-resolution reference image to obtain a low-resolution reference image; S223, performing feature extraction on the filtered multimodal image, the high-resolution reference image, and the low-resolution reference image to obtain multimodal feature information, high-resolution feature information, and low-resolution feature information; S224, processing the multimodal feature information and the low-resolution feature information to obtain reference information; Divide the multimodal feature information and the low-resolution feature information into blocks, obtaining M square blocks of the multimodal feature information and the low-resolution feature information, and calculate the similarity between the corresponding square blocks of the multimodal feature information and the low-resolution feature information, where the similarity serves as reference information; L i and R2 j The square blocks represent multimodal feature information and low-resolution feature information respectively; The similarity between square blocks is recorded as r i,j : r i,j The larger the value of , the higher the similarity between the two square blocks; S225, processing the high-resolution feature information and the multimodal feature information according to the reference information to obtain an enhanced multimodal image; Replacing the corresponding area in the multimodal feature information with square blocks whose similarity is greater than a preset threshold to obtain an enhanced multimodal image; S23, performing data registration on the enhanced multimodal image to obtain a preprocessed multimodal image; An image fusion module, configured to fuse the pre-processed multimodal images to obtain a fused image; The intervention guidance path planning module is used to process the fused image to obtain intervention guidance path information.

7. An intervention guidance device based on multimodal image fusion, characterized in that: The device comprises: a memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the intervention guidance method based on multimodal image fusion according to any one of claims 1 to 5.

8. A computer storable medium, characterized in that The computer storable medium stores computer instructions, and when the computer instructions are called, they are used to execute the intervention guidance method based on multimodal image fusion according to any one of claims 1 to 5.

Images