Multimodal medical image registration methods, devices, equipment and media

Abstract

This invention relates to the field of medical image processing, and provides a multimodal medical image registration method, apparatus, device, and medium. The method includes: S3, calculating the target point deviation between a first boundary point set and a second boundary point set; moving the target points of the second boundary point set to the target points of the first boundary point set according to the target point deviation, to obtain a third boundary point set; S4, defining a cost function, iteratively calculating the transformation dataset from the third boundary point set to the first boundary point set to minimize the cost function; S5, performing the transformation in S3 on all pixels in the second target image to obtain a transformation point set; selecting boundary points in the transformation point set and then performing the transformation dataset in S4; performing interpolation transformation based on boundary point transformation on the non-boundary points in the non-boundary point set to obtain the processing result; mapping the processing result onto a multimodal fused image to obtain a registered third target image. This method is used to reduce the degree of image mismatch during multimodal image registration.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a multimodal medical image registration method, apparatus, device, and medium. Background Technology

[0002] Individual imaging modalities may have limitations; lesions may be indistinct in one imaging modality but clearly visible in another. Computed tomography (CT) excels at displaying anatomical details and skeletal structures, while magnetic resonance imaging (MRI) can reveal soft tissue and functional information, and ultrasound is adept at displaying muscles, organs, fluids, and cystic structures.

[0003] Multimodal registration integrates image information from different sources into a single spatial coordinate system, forming a more comprehensive view of lesions or anatomical structures. However, living tissues may undergo complex deformations under physiological motion or pathological conditions, leading to mismatches in multimodal image registration during dynamic processes or multi-period imaging scenarios with large deformations. Therefore, there is an urgent need for a multimodal medical image registration method, device, equipment, and medium to address these issues. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal medical image registration method, apparatus, device, and medium, which reduces the degree of image mismatch during multimodal image registration.

[0005] In a first aspect, the present invention provides a multimodal medical image registration method, comprising: S1, acquiring a first modal image and extracting a first target image; constructing a first boundary point set based on the target boundary pixel coordinates in the first target image; S2, acquiring a second modal image and extracting a second target image; constructing a second boundary point set and a non-boundary point set based on the target boundary pixel coordinates in the second target image; S3, calculating the target point deviation between the first boundary point set and the second boundary point set; moving the target points of the second boundary point set to the target points of the first boundary point set according to the target point deviation to obtain a third boundary point set; S4, defining a cost function and iteratively calculating the transformation dataset from the third boundary point set to the first boundary point set to minimize the cost function; S5, performing the transformation in S3 on all pixels in the second target image to obtain a transformation point set; selecting the boundary points in the transformation point set and then performing the transformation dataset in S4 on the non-boundary points in the non-boundary point set, and performing an interpolation transformation based on the boundary point transformation to obtain a processing result; mapping the processing result onto a multimodal fused image to obtain a registered third target image.

[0006] Optionally, calculating the target point deviation between the first boundary point set and the third boundary point set includes: calculating the coordinates of a first target point based on the first boundary point set; calculating the coordinates of a second target point based on the second boundary point set; calculating the coordinate deviation of the target point based on the coordinates of the first target point and the coordinates of the second target point; the coordinates of the first target point are set as the mean coordinates of the first boundary point set, the centroid coordinates of the first boundary point set, or the feature point coordinates of the first boundary point set; the coordinates of the second target point are set as the mean coordinates of the second boundary point set, the centroid coordinates of the second boundary point set, or the feature point coordinates of the second boundary point set.

[0007] Optionally, S4 includes: defining a cost function C, setting the number of coordinates in the first boundary point set to m, and the number of coordinates in the third boundary point set to n, where m and n are both positive integers; setting the weights of the first boundary point set and the third boundary point set; and normalizing the first boundary point set to obtain the first coordinate set (y1…y2). m Normalizing the coordinates of the third boundary point set yields the second coordinate set (x1…x2). n ).

[0008] Optionally, the cost function C satisfies:

[0009]

[0010] Among them, y j Let x be the j-th coordinate in the first coordinate set. i Let y be the i-th coordinate in the second coordinate set. j and x i Both belong to the three-dimensional coordinate space R 3 , Let be an element in the transformed dataset, representing the vector transformed from the i-th coordinate to the j-th coordinate, and σ be the coordinate space R. 3 The average sampling distance in the coordinate space, τ is the largest possible offset in the given coordinate space; KL represents the Kullback-Leibler divergence, which satisfies:

[0011]

[0012] Where a and b are discrete probability distributions, and i is the index of each element in the distribution.

[0013] Optionally, S5 includes: defining a vector field v i And confidence weight w i To obtain the weighted vector field ((w1,v1)...(w) of the first dimension of the third boundary point set. n ,v n )),satisfy:

[0014]

[0015]

[0016] The free deformation of splines is defined using kernel function interpolation, where the kernel function k satisfies:

[0017] k(x,y)>0, (x,y)∈R 3 ×R 3

[0018] Where y represents the coordinates of a point in the first coordinate set, and x represents the coordinates of a point in the second coordinate set; the model based on the spline free deformation defines the transformation method of the third boundary point set to the first boundary point set, so that the second coordinate set is transformed into the coordinate system of the first coordinate set, satisfying:

[0019]

[0020] Where g is the point coordinate under the normalized first target domain coordinates.

[0021] Optionally, the first modal image is set as an ultrasound reconstruction image containing the target object; the ultrasound reconstruction image is generated by three-dimensional reconstruction of an ultrasound scan image; the second modal image is set as an MRI image containing the target object; the multimodal fusion image is set as an MRI image containing the target object with ultrasound coordinates consistent; when extracting the first target image, a deep learning-based ultrasound target segmentation network is used to remove non-target regions on the ultrasound reconstruction image; the ultrasound target segmentation network includes an encoder and a decoder; both the encoder and the decoder are provided with several converter modules.

[0022] Optionally, each of the converter modules includes a first feature layer and a second feature layer; both the first and second feature layers include a convolution operator, a normalization operator, and an activation function; the first feature layer includes a global attention layer, and the second feature layer includes a local attention layer; at least one of the first and second feature layers includes a cross-layer connection for inter-layer fusion; the global attention layer is used to generate a global query matrix, a key matrix, and a value matrix for all feature layers; the local attention layer is used to limit the size of a window and slide the window to generate a local query matrix, a key matrix, and a value matrix.

[0023] In a second aspect, the present invention provides a multimodal medical image registration device for use with the method described in any one of the first aspects, comprising: an acquisition unit, a processing unit, and a storage unit; the acquisition unit and the storage unit are both electrically connected to the processing unit; the acquisition unit is used to acquire a first modal image and a second modal image; the processing unit is used to execute the method described in any one of the first aspects; and the storage unit is used to store the third target image.

[0024] Thirdly, the present invention provides an electronic device including a memory and a processor, wherein the memory stores a program for running on the processor, and when the program is executed by the processor, the electronic device performs the method described in any one of the first aspects.

[0025] Fourthly, the present invention provides a readable storage medium storing a program, which, when executed, implements the method described in any one of the first aspects.

[0026] The beneficial effects of this invention are as follows: By calculating the target point deviation between the first boundary point set and the second boundary point set, image alignment based on target points is achieved, thereby initially correcting the position of the second modality image. By establishing a cost function and then iteratively optimizing the transformation dataset, the third boundary point set is made as close as possible to the first boundary point set after transformation, thereby reducing the overall mismatch of the image. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a multimodal medical image registration method provided by the present invention.

[0028] Figure 2 A schematic diagram of a first target image provided by the present invention;

[0029] Figure 3 A schematic diagram of a second target image provided by the present invention;

[0030] Figure 4 This is a schematic diagram of the structure of an ultrasonic target segmentation network provided by the present invention;

[0031] Figure 5 This is a schematic diagram of the structure of a converter module provided by the present invention;

[0032] Figure 6 A schematic diagram illustrating the composition of pixels in a first boundary point set provided by the present invention;

[0033] Figure 7 A schematic diagram illustrating the composition of pixels in a second boundary point set provided by the present invention;

[0034] Figure 8 This invention provides a schematic diagram of the pixel composition after the fusion of a second coordinate set and a first coordinate set.

[0035] Figure 9 A schematic diagram of a third target image provided by the present invention;

[0036] Figure 10 This is a schematic diagram of the structure of a multimodal medical image registration device provided by the present invention;

[0037] Figure 11 This is a schematic diagram of the structure of an electronic device provided by the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art. The terms "comprising" and similar expressions used herein mean that the element or object preceding the word covers the element or object listed following the word and its equivalents, but do not exclude other elements or objects.

[0039] In response to the problems existing in the current technology, such as Figure 1 As shown, the first embodiment of the present invention provides a multimodal medical image registration method, including: S1, acquiring a first modal image and extracting a first target image; constructing a first boundary point set based on the target boundary pixel coordinates in the first target image; S2, acquiring a second modal image and extracting a second target image; constructing a second boundary point set and a non-boundary point set based on the target boundary pixel coordinates in the second target image; S3, calculating the target point deviation between the first boundary point set and the second boundary point set; moving the target points of the second boundary point set to the target points of the first boundary point set according to the target point deviation to obtain a third boundary point set; S4, defining a cost function and iteratively calculating the transformation dataset from the third boundary point set to the first boundary point set to minimize the cost function; S5, performing the transformation in S3 on all pixels in the second target image to obtain a transformation point set; selecting the boundary points in the transformation point set and then performing the transformation dataset in S4; performing interpolation transformation based on boundary point transformation on the non-boundary points in the non-boundary point set to obtain a processing result; mapping the processing result onto a multimodal fused image to obtain a registered third target image.

[0040] Specifically, the target object is set as the prostate; the first target image is as follows: Figure 2 The image shown is an ultrasound reconstruction image excluding non-prostate areas.

[0041] In other embodiments, the target object may be an abdominal organ, such as the liver, gallbladder, spleen, or kidney; the first target image is an ultrasound reconstruction image containing only the abdominal organ; the second target image is an MRI image containing only the abdominal organ; and the third target image is used to provide a navigation target for puncture biopsy or focal ablation surgery targeting the abdominal organ.

[0042] In some embodiments, S3 further includes normalizing the first boundary point set and the third boundary point set. Correspondingly, S5 further includes performing a thin plate spline (TPS) interpolation transformation based on boundary point transformation on the non-boundary points in the non-boundary point set to obtain a normalized point set. The normalized point set is then de-normalized to obtain the processing result.

[0043] In some embodiments, the first modal image is set as an ultrasound reconstruction image containing the target object; the ultrasound reconstruction image is generated by three-dimensional reconstruction of an ultrasound scan image. The second modal image is set as an MRI image containing the target object. The multimodal fusion image is set as an MRI image containing the target object with ultrasound coordinates consistent. When extracting the first target image, a deep learning-based ultrasound target segmentation network is used to remove non-target regions on the ultrasound reconstruction image; the ultrasound target segmentation network includes an encoder and a decoder; both the encoder and the decoder are provided with several converter modules.

[0044] After acquiring an MRI image containing the target object, non-target regions are removed from the MRI image to obtain a second target image, such as... Figure 3 The image shown is an MRI image with non-prostate regions removed. NMR target segmentation can be performed using a deep learning-based NMR target segmentation network, or traditional image processing or manual segmentation methods to obtain the NMR segmentation binary image. To improve the accuracy of NMR target segmentation, the NMR image is first nonlinearly interpolated to obtain a smooth interpolated image. The NMR image segmentation model is obtained by training a deep learning-based NMR image segmentation network, whose structure is similar to that of an ultrasound segmentation network. Non-target regions are removed from the NMR interpolated image based on the NMR binary image.

[0045] like Figure 4As shown, specifically, the encoder 7 and decoder 8 constitute a three-dimensional U-Net structured ultrasonic target segmentation network. The encoder 7 includes an input layer 1, an embedding layer 2, four interleaved converter modules 3, and three fusion layers 4; the decoder 8 includes an output layer 6, four interleaved converter modules 3, and four extension layers 5. Following the direction of traversing the segmentation network from the input layer 1 to the output layer 6, a skip connection 9 is provided between the embedding layer 2 and the last extension layer 5; a skip connection 9 is also provided between the last fusion layer 4 and the first extension layer 5.

[0046] It is worth noting that the segmentation network includes an M-layer fusion layer 4, an M+1-layer extension layer 5, and a 2M+2-layer converter module 3, where M is a positive integer. A skip connection 9 is provided between the L-th fusion layer 4 and the M-L+1-th layer, where L is a positive integer in the range [1, M].

[0047] More specifically, each of the converter modules 3 includes a first feature layer and a second feature layer; both the first and second feature layers include a convolution operator 31, a normalization operator 32, and an activation function 33; the first feature layer includes a global attention layer 34, and the second feature layer includes a local attention layer 35; at least one of the first and second feature layers includes a cross-layer connection for inter-layer fusion; the global attention layer 34 is used to generate a global query matrix, a key matrix, and a value matrix for all feature layers; the local attention layer 35 is used to limit the size of a window and slide the window to generate a local query matrix, a key matrix, and a value matrix.

[0048] In another example, the encoder input layer is first connected to an embedding layer. The embedding layer first segments the image into several image patches, then performs linear mapping to convert them into high-dimensional vectors (tokens) which are then fed into the converter module. A fusion layer is provided between adjacent converter modules in the encoder; an expansion layer is provided between adjacent converter modules in the decoder. This expansion layer first performs channel augmentation, then uses deconvolution for aspect ratio upsampling and channel reduction; a skip connection is provided between the encoder and decoder.

[0049] like Figure 5 As shown, in some embodiments, the converter module 3 includes: a first feature layer and a second feature layer, both of which include a convolution operator 31, a normalization operator 32 and an activation function 33; the first feature layer includes a global attention layer 34 and the second feature layer includes a local attention layer 35; at least one of the first feature layer and the second feature layer includes a cross-layer connection for inter-layer fusion.

[0050] It is worth noting that the segmentation network is implemented based on the U-Net, which retains its lightweight nature while enhancing the extraction of global and local features, resulting in significant advantages for accurate target segmentation. The segmentation network can also acquire preset contour information as auxiliary input. This contour information is used to delineate the contours of the prostate's starting, ending, and intermediate layers to assist in prostate segmentation constraints and verification, thereby improving the accuracy of the segmentation network.

[0051] Specifically, the input of the first feature layer is provided with a normalization operator 32 and a global attention layer 34; the output of the global attention layer 34 and the input of the normalization operator 32 are fused together for the first time; the result of the first inter-layer fusion is passed through an activation function 33 and then through two normalization operators 32 and two convolution operators 31 to obtain a convolution result; the convolution result is fused together with the result of the first inter-layer fusion for the second time.

[0052] In other specific embodiments, the input end of the second feature layer is provided with a normalization operator 32 and a local attention layer 35; the output end of the local attention layer 35 and the input end of the normalization operator 32 are fused for the third time; the result of the third inter-layer fusion is passed through an activation function 33 and then through two normalization operators 32 and two convolution operators 31 to obtain a convolution result; the convolution result and the result of the third inter-layer fusion are fused for the fourth time; the result of the fourth inter-layer fusion is used as the output of the converter module 3.

[0053] The activation function used is the GELU activation function. The GELU activation function combines the probability of the activation parameter being 0 or 1 with the activation value of the neural network, enabling the neural network to make self-deterministic decisions and generate feature data with better sparsity. That is, the smaller the activation value of the neural network, the lower the probability that the activation parameter it is multiplied by is 1, which can better highlight the input features. This is achieved using a sparse matrix with most elements being 0. The first feature layer includes a global attention layer, and the second feature layer includes a local attention layer; at least one of the first and second feature layers includes a cross-layer connection for inter-layer fusion.

[0054] In some specific embodiments, prior to the three-dimensional reconstruction, the ultrasound scan image is further subjected to image denoising processing, and the denoising algorithm used includes at least one of the following: three-dimensional block matching algorithm, wavelet transform, principal component analysis-based algorithm, and bilateral filtering algorithm.

[0055] The process involves acquiring an ultrasound scan image containing the target object, traversing each voxel in the three-dimensional imaging space based on the ultrasound scan image, finding one or more corresponding pixel values ​​in the two-dimensional ultrasound scan image sequence through transformation, and then using an interpolation method to calculate the corresponding pixel value and assigning the pixel value to the voxel; the interpolation method can be a distance-weighted algorithm, a linear interpolation algorithm, or a trajectory tracking algorithm.

[0056] In some embodiments, calculating the target point deviation between the first boundary point set and the third boundary point set includes: calculating the coordinates of a first target point based on the first boundary point set; calculating the coordinates of a second target point based on the second boundary point set; calculating the coordinate deviation of the target point based on the coordinates of the first target point and the coordinates of the second target point; wherein the coordinates of the first target point are set as the mean coordinates of the first boundary point set, the centroid coordinates of the first boundary point set, or the feature point coordinates of the first boundary point set; and the coordinates of the second target point are set as the mean coordinates of the second boundary point set, the centroid coordinates of the second boundary point set, or the feature point coordinates of the second boundary point set.

[0057] In some embodiments, the boundary map for obtaining the first boundary map can be implemented by edge extraction based on a binary segmentation mask. Specifically, boundary tracking, contour extraction, region growing, or edge detection methods can be used. The coordinates of the generated boundary map are extracted to generate the first boundary point set, and the boundary pixels corresponding to the first boundary point set are composed as follows: Figure 6 The diagram shown is shown in the image.

[0058] In some embodiments, in S2, a boundary map of the second target image is obtained, and the coordinates of boundary pixels are extracted from the boundary map of the second target as a second boundary point set. The boundary pixels corresponding to the second boundary point set are composed as follows: Figure 7 The diagram shown illustrates how obtaining the boundary map of the second target map can be achieved through edge extraction based on a binary segmentation mask. The centroids of the first and second boundary point sets are calculated by taking their mean values, calculating their deviation, and then moving the second boundary point set to the first boundary point set according to the deviation to obtain a new second boundary point set.

[0059] In another specific embodiment, when the tissue corresponding to the volume block is a homogeneous symmetrical geometry, the mean coordinates coincide with the centroid coordinates. In other specific embodiments, when the tissue corresponding to the volume block is a heterogeneous or asymmetrical geometry, the mean coordinates and the centroid coordinates separate. For example, if most of the tumor's volume is concentrated on one side of the image, while only a small portion protrudes on the other side, the tumor's centroid may be biased towards the side with the larger volume, rather than simply falling at the tumor's geometric center, i.e., the mean coordinate position. Furthermore, if there is density heterogeneity within the tumor, the centroid will also be affected by the density distribution, further deviating from the geometric center.

[0060] This embodiment, by distinguishing between mean coordinates and centroid coordinates, can describe the morphological and mass distribution characteristics of an object in greater detail. This is of great significance for medical image analysis, biological structural research, and centroid calculation in physics, helping researchers and doctors to more accurately understand the intrinsic properties and structural differences of target objects. Understanding the mean and centroid locations of lesions can help doctors more accurately locate treatment targets, ensuring treatment precision and reducing the impact on surrounding healthy tissues.

[0061] In some embodiments, S4 includes: defining a cost function C, setting the number of coordinates in the first boundary point set to m, and setting the number of coordinates in the third boundary point set to n, where m and n are both positive integers; setting the weights of the first boundary point set and the third boundary point set; and normalizing the first boundary point set to obtain a first coordinate set (y1…y2). m Normalizing the coordinates of the third boundary point set yields the second coordinate set (x1…x2). n ).

[0062] In some embodiments, the cost function C satisfies:

[0063]

[0064]

[0065] Among them, y j Let x be the j-th coordinate in the first coordinate set. i Let y be the i-th coordinate in the second coordinate set. j and x i Both belong to the three-dimensional coordinate space R 3 , Let be an element in the transformed dataset, representing the vector transformed from the i-th coordinate to the j-th coordinate, and σ be the coordinate space R. 3 The average sampling distance in the coordinate space, τ is the largest possible offset in the given coordinate space; KL represents the Kullback-Leibler divergence, which satisfies:

[0066]

[0067] Where a and b are discrete probability distributions, and i is the index of each element in the distribution. This embodiment transforms the registration problem into finding a movement scheme from the second coordinate set to the first coordinate set that minimizes the cost function C. A minimum optimization problem estimation method is used to solve this problem; the limited-memory Broyden-Fletche-Goldfarb-Shanno (L-BFGS-B) optimization method can be employed. In other embodiments, the unlimited-memory Broyden-Fletche-Goldfarb-Shanno (L-BFGS-B) optimization method can also be used to solve for the parameters.

[0068] It is worth noting that, for the sake of preserving the continuity and integrity of biological tissues, it is desirable to maintain the spatial structure and tissue information of the original image during image registration, preventing local breaks or unnatural deformations. Allowing point sets to be arbitrarily split during movement could lead to severe image distortion and loss of original anatomical significance, which is unacceptable for medical applications (such as surgical navigation, lesion localization, and treatment planning).

[0069] For the reasons mentioned above, the transition from the second coordinate set to the first coordinate set is constrained by the fact that the second coordinate set cannot be separated during the movement. Therefore, S5 includes: defining a vector field v. i And confidence weight w i To obtain the weighted vector field ((w1,v1)...(w) of the first dimension of the third boundary point set. n ,v n )),satisfy:

[0070]

[0071]

[0072] The free deformation of splines is defined using kernel function interpolation, where the kernel function k satisfies:

[0073] k(x,y)>0, (x,y)∈R 3 ×R 3

[0074] Where y represents the coordinates of a point in the first coordinate set, and x represents the coordinates of a point in the second coordinate set; the model based on the spline free deformation defines the transformation method of the third boundary point set to the first boundary point set, so that the second coordinate set is transformed into the coordinate system of the first coordinate set, satisfying:

[0075]

[0076] Where g represents the point coordinates in the normalized first target domain coordinate system. Transforming the second coordinate set to the first coordinate set coordinate system, the resulting pixel composition after fusing the second and first coordinate sets is as follows: Figure 8 The diagram shown. The third target image is as follows. Figure 9 As shown, this is used to provide navigation targets for prostate biopsy procedures.

[0077] In some embodiments, each of the converter modules includes a first feature layer and a second feature layer; both the first and second feature layers include a convolution operator 31, a normalization operator 32, and an activation function 33; the first feature layer includes a global attention layer 34, and the second feature layer includes a local attention layer 35; at least one of the first and second feature layers includes a cross-layer connection for inter-layer fusion; the global attention layer 34 is used to generate a global query matrix, a key matrix, and a value matrix for the size of all feature layers; the local attention layer 35 is used to limit the size of a window and slide the window to generate a local query matrix, a key matrix, and a value matrix. Here, the normalization operator 32 adopts inter-layer normalization, avoiding multi-dimensional normalization processing of channels, height, and width on the training subset (batch), which can avoid internal covariate shift. To improve the accuracy of edge detection, the loss function is implemented using a weighted average of DICE and Hausdorff boundary distances, and its weights can be dynamically adjusted according to requirements.

[0078] like Figure 10As shown, the second embodiment provides a multimodal medical image registration device 100 for the method described in any one of the above embodiments, comprising: an acquisition unit 101 for acquiring a first modal image and a second modal image; a processing unit 102 for extracting a first target image based on the first modal image; constructing a first boundary point set based on the target boundary pixel coordinates in the first target image; extracting a second target image based on the second modal image; constructing a second boundary point set based on the target boundary pixel coordinates in the second target image; calculating the target point deviation between the first boundary point set and the second boundary point set; and moving the target points of the second boundary point set to the first boundary point set according to the target point deviation. The target point is obtained by acquiring a third set of boundary points; the first and third set of boundary points are normalized; a cost function is defined, and the transformation dataset from the third set of boundary points to the first set of boundary points is iteratively calculated to minimize the cost function; all pixels on the second target image are transformed in step S3 to obtain a transformation point set; the boundary points are then transformed in step S4, and the non-boundary points are transformed by thin-plate spline interpolation based on the boundary point transformation to obtain a normalized point set; the normalized point set is de-normalized, and the processing result is mapped onto a multimodal fusion image to obtain a registered third target image; the third target image is used to provide a navigation target for medical operations. A storage unit 103 is used to store the third target image. Both the acquisition unit 101 and the storage unit 103 are electrically connected to the processing unit 102.

[0079] Specifically, the acquisition unit 101 is configured with an MRI probe for acquiring MRI images, an ultrasound probe for acquiring ultrasound images, and a computed tomography (CT) machine for acquiring CT images. The processing unit 102 is configured as a processor. The storage unit 103 is configured as a memory.

[0080] like Figure 11 As shown, the third embodiment provides an electronic device 200, including a memory 202 and a processor 201. The memory 202 stores a program for running on the processor 201. When the program is executed by the processor 201, the electronic device 200 implements the method described in any of the above embodiments.

[0081] Specifically, in one possible embodiment, the electronic device 200 further includes: an output interface 203 for outputting results; a communication interface 204 for transmitting communication signals; and an antenna 205 for transmitting or receiving signals.

[0082] It should be noted that the processor 201 in this embodiment can be an image processing chip or an integrated circuit chip, capable of processing image signals. In implementation, each step of the above method embodiment can be completed by the integrated logic circuitry in the processor's hardware or by software instructions. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices. It can implement or execute the methods, steps, and logic block diagrams disclosed in this embodiment. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this embodiment can be directly implemented by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0083] It is understood that the memory 202 in this embodiment can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0084] The fourth embodiment provides a readable storage medium storing a program, characterized in that, when the program is executed, it implements the method described in any one of the above embodiments.

[0085] It is worth noting that if the method is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0086] While embodiments of the present invention have been described in detail above, it will be apparent to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations fall within the scope and spirit of the invention as set forth in the claims. Furthermore, the invention described herein may have other embodiments and can be implemented or carried out in various ways.

Claims

1. A multimodal medical image registration method, characterized in that, include: S1, acquire the first modality image and extract the first target image; A first boundary point set is constructed based on the target boundary pixel coordinates in the first target image; S2, acquire the second modality image, extract the second target image; construct a second boundary point set and a non-boundary point set based on the target boundary pixel coordinates in the second target image; S3, calculate the target point deviation between the first boundary point set and the second boundary point set; move the target point of the second boundary point set to the target point of the first boundary point set according to the target point deviation to obtain the third boundary point set; S4, Define a cost function, iteratively calculate the transformed dataset from the third boundary point set to the first boundary point set to minimize the cost function; set the number of coordinates in the first boundary point set to m, and the number of coordinates in the third boundary point set to n, where m and n are both positive integers; set the weights of the first boundary point set and the third boundary point set, and normalize the first boundary point set to obtain the first coordinate set. The second coordinate set is obtained by normalizing the coordinates of the third boundary point set. The cost function C satisfies: in, Let j be the j-th coordinate in the first coordinate set. Let i be the i-th coordinate in the second coordinate set. and Both belong to three-dimensional coordinate space , For each element in the transformed dataset, a vector is represented that transforms the i-th coordinate to the j-th coordinate. For the coordinate space The average sampling distance in the middle, Take the largest possible offset in the given coordinate space; KL represents the Kullback-Leibler divergence, which satisfies: in, and It is a discrete probability distribution. It is the index of each element in the distribution; S5, Define a vector field and confidence weight To obtain the weighted vector field of the first dimension of the third boundary point set. ,satisfy: The free deformation of splines is defined using kernel function interpolation, where the kernel function k satisfies: Where y represents the coordinates of a point in the first coordinate set, and x represents the coordinates of a point in the second coordinate set; the model based on the spline free deformation defines the transformation method of the third boundary point set to the first boundary point set, so that the second coordinate set is transformed into the coordinate system of the first coordinate set, satisfying: Where g is the point coordinate under the normalized first target domain coordinates; the non-boundary points in the non-boundary point set are subjected to interpolation transformation based on boundary point transformation to obtain the processing result; the processing result is mapped onto the multimodal fusion image to obtain the registered third target image.

2. The method according to claim 1, characterized in that, Calculating the target point deviation between the first boundary point set and the third boundary point set includes: Calculate the coordinates of the first target point based on the first set of boundary points; calculate the coordinates of the second target point based on the second set of boundary points; Calculate the coordinate deviation of the target point based on the coordinates of the first target point and the coordinates of the second target point; The coordinates of the first target point are set to the mean coordinates of the first boundary point set, the centroid coordinates of the first boundary point set, or the feature point coordinates of the first boundary point set; the coordinates of the second target point are set to the mean coordinates of the second boundary point set, the centroid coordinates of the second boundary point set, or the feature point coordinates of the second boundary point set.

3. The method according to claim 1, characterized in that, The first modal image is set as an ultrasound reconstruction image containing the target object; the ultrasound reconstruction image is generated by three-dimensional reconstruction of the ultrasound scan image. The second modal image is set as an MRI image containing the target object; The multimodal fusion image is set as an NMR image containing the target object and consistent with the ultrasound coordinates; When extracting the first target image, a deep learning-based ultrasound target segmentation network is used to remove non-target regions from the ultrasound reconstruction image; The ultrasonic target segmentation network includes an encoder and a decoder; both the encoder and the decoder are equipped with several converter modules.

4. The method according to claim 3, characterized in that, Each of the converter modules includes a first feature layer and a second feature layer; both the first feature layer and the second feature layer include a convolution operator, a normalization operator, and an activation function; The first feature layer includes a global attention layer, and the second feature layer includes a local attention layer; at least one of the first feature layer and the second feature layer includes a cross-layer connection for inter-layer fusion. The global attention layer is used to generate the global query matrix, key matrix, and value matrix for all feature layers; the local attention layer is used to limit the size of the window and slide the window to generate the local query matrix, key matrix, and value matrix.

5. A multimodal medical image registration device, characterized in that, include: Acquisition unit, processing unit, and storage unit; Both the acquisition unit and the storage unit are electrically connected to the processing unit; The acquisition unit is used to acquire the first modal image and the second modal image; The processing unit is configured to perform the method according to any one of claims 1 to 4; The storage unit is used to store the third target image.

6. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a program for running on the processor, which, when executed by the processor, causes the electronic device to perform the method of any one of claims 1 to 4.

7. A readable storage medium storing a program, characterized in that, When the program is executed, it implements the method of any one of claims 1 to 4.

Images