Registration method and apparatus for incomplete medical image, electronic device, and medium

Abstract

The present invention provides a registration method and apparatus for an incomplete medical image, an electronic device, and a medium. The method comprises: acquiring an ultrasonic scanning image containing an incomplete target object, and performing three-dimensional reconstruction to obtain an ultrasonic reconstructed image; removing a non-target area on the ultrasonic reconstructed image to obtain a first target image; acquiring a nuclear magnetic image containing a target object, and removing a non-target area to obtain a second target image; on the basis of the first target image and the second target image, performing rigid registration to obtain a transformation matrix; on the basis of the transformation matrix, converting the second target image into an ultrasonic coordinate system to obtain a third target image; performing related processing on the third target image and the first target image to obtain a fourth target image; and on the basis of the first target image and the fourth target image, performing elastic registration to obtain a deformation field, and on the basis of the deformation field, converting the fourth target image into the ultrasonic coordinate system to obtain a fifth target image. The method is used for improving the registration precision of an incomplete medical image.

Description

Registration method, device, electronic device and medium for incomplete medical images

[0001] Cross-references

[0002] This application claims priority to Chinese patent application No. 2023112502718, filed on September 25, 2023. The contents of the above application are incorporated herein by reference. Technical Field

[0003] The present invention relates to the technical field of medical image processing, and in particular to a registration method, device, electronic equipment and medium for incomplete medical images. Background Art

[0004] Prostate biopsy and puncture are important screening methods for prostate cancer and other lesions. Before the biopsy, the doctor first performs an MRI on the patient to obtain the morphology and location of the prostate and lesion. During the biopsy, the doctor uses preoperative and intraoperative information to guide the needle to the prostate lesion.

[0005] To obtain clear ultrasound scans, conventional techniques typically involve inserting an ultrasound probe into the anus to scan the prostate. However, due to the deep prostate location in some individuals, the ultrasound probe is not long enough, and the imaging range of the ultrasound probe is limited. Consequently, the anal ultrasound probe's scanning depth is insufficient to cover the prostate's depth, resulting in an incomplete prostate image in the scanned ultrasound image. This makes it difficult to align the MRI image with the incomplete ultrasound image, increasing registration errors and even leading to registration failure. Therefore, a new registration method, apparatus, electronic device, and medium for incomplete medical image registration is urgently needed to address these issues.

[0006] Summary of the Invention

[0007] The object of the present invention is to provide a registration method, device, electronic device and medium for incomplete medical images, which are used to improve the registration accuracy of incomplete medical images.

[0008] In a first aspect, the present invention provides a registration method for incomplete medical images, comprising: obtaining an ultrasound scanning image containing an incomplete target object, performing three-dimensional reconstruction based on the ultrasound scanning image to obtain an ultrasound reconstruction image; eliminating non-target areas on the ultrasound reconstruction image to obtain a first target image; obtaining a nuclear magnetic resonance image containing the target object, eliminating non-target areas on the nuclear magnetic resonance image to obtain a second target image; performing rigid registration based on the first target image and the second target image to obtain a transformation matrix; converting the second target image to an ultrasound coordinate system based on the transformation matrix to obtain a third target image; performing correlation processing on the third target image and the first target image to obtain a fourth target image; the target contour in the fourth target image is consistent with the target contour in the first target image; performing elastic registration based on the first target image and the fourth target image to obtain a deformation field, and converting the fourth target image to an ultrasound coordinate system based on the deformation field to obtain a fifth target image; the fifth target image is used to provide a navigation target for medical operations.

[0009] The method of the present invention has the following beneficial effects: the present invention performs correlation processing on the third target image and the first target image to obtain a fourth target image; the target contour in the fourth target image is consistent with the target contour in the first target image; and the fourth target image is ensured to be successfully registered with the first target image, avoiding registration errors caused by direct registration due to different contours between the second target image and the first target image. Furthermore, elastic fine-tuning based on rigid registration can effectively improve registration accuracy. Even if an incomplete ultrasound image is obtained, the present invention can still align it with the nuclear magnetic resonance image, which is beneficial for reducing registration errors and improving the registration success rate.

[0010] Optionally, the correlation processing of the third target image and the first target image to obtain the fourth target image includes: calculating the start and end layers of the first target image, eliminating the part outside the start and end layers on the third target image, so that the part outside the start and end layers on the rigidly aligned nuclear magnetic resonance image is set to zero, to obtain the fourth target image.

[0011] Optionally, the acquiring of an ultrasonic scanning image containing an incomplete target object and performing three-dimensional reconstruction based on the ultrasonic scanning image to obtain an ultrasonic reconstruction image include: before performing the three-dimensional reconstruction, performing image denoising on the ultrasonic scanning image, the denoising algorithm including at least one of a three-dimensional block matching algorithm (Block Matching 3D, BM3D), a wavelet transform, an algorithm based on principal component analysis, and a bilateral filtering algorithm; when performing the three-dimensional reconstruction, traversing each voxel in the three-dimensional imaging space, finding one or more corresponding pixel values ​​in a two-dimensional ultrasonic scanning image sequence through transformation, and then using an interpolation method to calculate the corresponding pixel values, and assigning the pixel values ​​to the voxels; the interpolation method includes a distance weighted algorithm, a linear interpolation algorithm, or a trajectory tracking algorithm.

[0012] Optionally, the distance weighted algorithm satisfies:

[0013] Among them, I(V c ) represents the current voxel pixel value, is the pixel value of the kth voxel in the nth neighborhood of the current voxel, W k is the weight corresponding to the pixel value of the k-th voxel, d k is the distance of the kth voxel relative to the current voxel.

[0014] Optionally, non-target areas on the ultrasound reconstruction image are eliminated to obtain a first target image, including: inputting the ultrasound reconstruction image into an ultrasound image segmentation model to obtain an ultrasound binary image; the ultrasound image segmentation model is obtained by training an ultrasound image segmentation network based on deep learning; non-target areas on the ultrasound reconstruction image are eliminated according to the ultrasound binary image; the segmentation network includes an encoder and a decoder; the encoder and the decoder are both provided with a number of transformer blocks; a fusion layer is provided between adjacent transformer modules in the encoder; an expansion layer is provided between adjacent transformer modules in the decoder; and a jump connection is provided between the encoder and the decoder.

[0015] Optionally, the converter module includes: a first feature layer and a second feature layer, both of which include a convolutional operator (conv), a normalization operator (normlization) and an activation function (Mish activation); the first feature layer includes a global attention layer, and the second feature layer includes a local (shift) attention layer; at least one of the first feature layer and the second feature layer includes a cross-layer connection for inter-layer fusion.

[0016] Optionally, a nuclear magnetic resonance image containing a target object is obtained, and non-target areas on the nuclear magnetic resonance image are eliminated to obtain a second target image, including: performing nonlinear interpolation on the nuclear magnetic resonance image to obtain a smooth interpolation image, and inputting the smooth interpolation image into a nuclear magnetic resonance image segmentation model to obtain a nuclear magnetic resonance binary image; the nuclear magnetic resonance image segmentation model is obtained by training a nuclear magnetic resonance image segmentation network based on deep learning; and non-target areas on the nuclear magnetic resonance interpolation image are eliminated based on the nuclear magnetic resonance binary image.

[0017] Optionally, the elastic registration adopts a pyramid multi-resolution multi-metric elastic registration method; the pyramid multi-resolution represents the number of multi-resolution pyramids set according to the size of the first target image; the multi-metric includes a normalized mutual information metric and an enhanced correlation coefficient metric; the normalized mutual information metric NMI(A,B) satisfies:

[0018] Among them, H(A) is the information entropy of image A, H(B) is the information entropy of image B, H(A,B) is the joint entropy of images A and B, and the information entropy H(A) of image A satisfies:

[0019] The joint entropy H(A,B) of image A and image B satisfies:

[0020] Where N is the number of grayscale values ​​in the image, p i is the probability of the pixel with gray value i appearing, p AB (a, b) is the probability that the pixel value in image A is a and the pixel value in image B is b; the enhanced correlation coefficient metric ECC(A, B) satisfies:

[0021] The evaluation index metric of the elastic registration method satisfies: metirc(A,B)=α*NMI(A,B)+β*ECC(A,B)

[0022] Where: α+β=1

[0023] α is the mutual information weight, and β is the enhanced correlation coefficient weight. α and β can be adjusted according to actual conditions. The elastic registration method uses deformation field transformation. The interpolator uses at least one of linear interpolation and B-spline interpolation. The evaluation index is optimized using at least one of an adaptive stochastic gradient descent algorithm, an adaptive stochastic LBFGS (Limited-memory Broyden-Fletcher-Goldfarb-Shanno), and a quasi-Newton LBFGS.

[0024] In a second aspect, the present invention provides a registration device for incomplete medical images, which is used for the method described in any one of the first aspects, and includes: a three-dimensional reconstruction unit, which is used to obtain an ultrasound scanning image containing an incomplete target object, and perform three-dimensional reconstruction based on the ultrasound scanning image to obtain an ultrasound reconstruction image; an ultrasound image segmentation unit, which is used to eliminate non-target areas on the ultrasound reconstruction image to obtain a first target image; a nuclear magnetic resonance image segmentation unit, which is used to obtain a nuclear magnetic resonance image containing a target object, and eliminate non-target areas on the nuclear magnetic resonance image to obtain a second target image; a registration unit, which is used to perform rigid registration based on the first target image and the second target image to obtain a transformation matrix; convert the second target image into an ultrasound coordinate system according to the transformation matrix to obtain a third target image; perform correlation processing on the third target image and the first target image to obtain a fourth target image; the target contour in the fourth target image is consistent with the target contour in the first target image; perform elastic registration based on the first target image and the fourth target image to obtain a deformation field, and convert the fourth target image into an ultrasound coordinate system according to the deformation field to obtain a fifth target image; the fifth target image is used to provide a navigation target for medical operations.

[0025] Optionally, the three-dimensional reconstruction unit is used to perform image noise reduction processing on the ultrasonic scanning image before performing the three-dimensional reconstruction, and the noise reduction algorithm includes at least one of a three-dimensional block matching algorithm, a wavelet transform, a principal component analysis algorithm and a bilateral filtering algorithm; the three-dimensional reconstruction unit is also used to traverse each voxel in the three-dimensional imaging space when performing the three-dimensional reconstruction, find the corresponding one or more pixel values ​​in the two-dimensional ultrasonic scanning image sequence through transformation, and then use an interpolation method to calculate the corresponding pixel values ​​and assign the pixel values ​​to the voxels.

[0026] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, wherein the memory stores a program that can be run on the processor, and when the program is executed by the processor, the electronic device implements any one of the methods described in the first aspect.

[0027] In a fourth aspect, the present invention provides a readable storage medium, wherein the readable storage medium stores a program, and when the program is executed, the method according to any one of the first aspects is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] FIG1 is a schematic diagram of a flow chart of a registration method for incomplete medical images provided by the present invention;

[0029] FIG2 is a three-view diagram of a first target image provided by the present invention;

[0030] FIG3 is a three-view diagram of a second target image provided by the present invention;

[0031] FIG4 is a three-view diagram of a third target image provided by the present invention;

[0032] FIG5 is a three-view diagram of a fourth target image provided by the present invention;

[0033] FIG6 is a three-view diagram of a fifth target image provided by the present invention;

[0034] FIG7 is a schematic diagram of the structure of an ultrasound image segmentation network provided by the present invention;

[0035] FIG8 is a schematic structural diagram of a converter module provided by the present invention;

[0036] FIG9 is a schematic structural diagram of a registration device for incomplete medical images provided by the present invention;

[0037] FIG10 is a schematic structural diagram of an electronic device provided by the present invention.

[0038] Numbers in the figure: 1. Input layer; 2. Embedding layer; 3. Converter module; 4. Fusion layer; 5. Expansion layer; 6. Output layer; 7. Encoder; 8. Decoder; 9. Skip connection; 31. Convolution operator; 32. Normalization operator; 33. Activation function; 34. Global attention layer; 35. Local attention layer; 100. Registration device for incomplete medical images; 101. Three-dimensional reconstruction unit; 102. Ultrasound image segmentation unit; 103. MRI image segmentation unit; 104. Registration unit; 200. Electronic device; 201. Processor; 202. Memory; 203. Interface. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be the common meanings understood by people with ordinary skills in the field to which the present invention belongs. The words "including" and similar words used in this article mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0040] In response to the problems existing in the prior art, as shown in Figure 1, the present invention provides a registration method for incomplete medical images, including: S101, obtaining an ultrasound scanning image containing an incomplete target object, and performing three-dimensional reconstruction based on the ultrasound scanning image to obtain an ultrasound reconstruction image; S102, eliminating non-target areas on the ultrasound reconstruction image to obtain a first target image; S103, obtaining a nuclear magnetic resonance image containing the target object, and eliminating non-target areas on the nuclear magnetic resonance image to obtain a second target image; S104, performing rigid registration based on the first target image and the second target image to obtain a transformation matrix; converting the second target image into an ultrasound coordinate system according to the transformation matrix to obtain a third target image; S105, performing correlation processing on the third target image and the first target image to obtain a fourth target image; the target contour in the fourth target image is consistent with the target contour in the first target image; S106, performing elastic registration based on the first target image and the fourth target image to obtain a deformation field, and converting the fourth target image into an ultrasound coordinate system according to the deformation field to obtain a fifth target image; the fifth target image is used to provide a navigation target for medical operations.

[0041] Specifically, the target object is the prostate; the first target image is shown in Figure 2, which is an ultrasound reconstruction image with non-prostate areas removed; the second target image is shown in Figure 3, which is a nuclear magnetic image with non-prostate areas removed; the third target image is shown in Figure 4, which is a nuclear magnetic image converted to an ultrasonic coordinate system, and the non-prostate area is removed from the nuclear magnetic image; the fourth target image is shown in Figure 5, which is a nuclear magnetic image obtained after correlation processing, and the outline of the nuclear magnetic image is consistent with the outline of the ultrasound reconstruction image with non-prostate areas removed; the fifth target image is shown in Figure 6, which is used to provide a navigation target for prostate puncture surgery.

[0042] It is worth noting that the A direction represents the front side of the body, the P direction represents the back side of the body, the S direction represents the top side of the body, the I direction represents the bottom side of the body, the L direction represents the left side of the body, and the R direction represents the right side of the body.

[0043] In some other embodiments, the target object may be an abdominal organ, such as the liver, gallbladder, spleen or kidney; the first target image is an ultrasound reconstruction image that only contains the abdominal organ; the second target image is a nuclear magnetic image that only contains the abdominal organ; the third target image is a nuclear magnetic image converted to an ultrasound coordinate system, which only contains the abdominal organ; the fourth target image is a nuclear magnetic image obtained after correlation processing, and the outline of the nuclear magnetic image is consistent with the outline of the ultrasound reconstruction image that only contains the abdominal organ; the fifth target image is used to provide a navigation target for a puncture biopsy or focal ablation surgery on the abdominal organ.

[0044] It is worth noting that this embodiment performs correlation processing on the third target image and the first target image to obtain a fourth target image; the target contour in the fourth target image is consistent with the target contour in the first target image; ensuring that the fourth target image and the first target image can be successfully registered, avoiding registration errors caused by direct registration due to different contours between the second target image and the first target image. Furthermore, elastic fine-tuning based on rigid registration can effectively improve registration accuracy. Even if an incomplete ultrasound image is obtained, the present invention can still align it with the nuclear magnetic resonance image, which is beneficial for reducing registration errors and improving the registration success rate.

[0045] Specifically, performing rigid registration on the first target image and the second target image to obtain a transformation matrix includes using an Euler transform to translate and rotate the first target image relative to the second target image. Four different resolutions are used as inputs to the Euler transform during image pyramid construction. The transformation matrix is ​​optimized with maximizing mutual information, using B-spline interpolation, and employing an adaptive stochastic gradient descent method for optimization.

[0046] In other embodiments, the number of pyramid resolutions can be customized according to the image size, the interpolation method can be set to Lagrange interpolation or spline interpolation, and the optimization method can be set to Powell method or conjugate gradient method.

[0047] In some other embodiments, the correlation processing of the third target image and the first target image to obtain the fourth target image includes: calculating the start and end layers of the first target image, eliminating the part outside the start and end layers on the third target image, so that the part outside the start and end layers on the rigidly aligned nuclear magnetic resonance image is set to zero, to obtain the fourth target image.

[0048] In some embodiments, the acquiring of an ultrasonic scanning image containing a target object and performing three-dimensional reconstruction based on the ultrasonic scanning image to obtain an ultrasonic reconstruction image include: before performing the three-dimensional reconstruction, performing image noise reduction processing on the ultrasonic scanning image, the noise reduction algorithm including at least one of a three-dimensional block matching algorithm, a wavelet transform, an algorithm based on principal component analysis, and a bilateral filtering algorithm; when performing the three-dimensional reconstruction, traversing each voxel in the three-dimensional imaging space, finding one or more corresponding pixel values ​​in the two-dimensional ultrasonic scanning image sequence through transformation, and then using an interpolation method to calculate the corresponding pixel value, and assigning the pixel value to the voxel; the interpolation method includes a distance weighted algorithm, a linear interpolation algorithm, or a trajectory tracking algorithm.

[0049] Specifically, the image denoising based on principal component analysis and bilateral filtering is performed on the two-dimensional image, which can remove noise while effectively retaining the image details. The interpolation method adopts a distance weighted algorithm, which satisfies:

[0050] Among them, I(V c ) represents the current voxel pixel value, is the pixel value of the kth voxel in the nth neighborhood of the current voxel, W k is the weight corresponding to the pixel value of the k-th voxel, d k is the distance of the kth voxel relative to the current voxel.

[0051] As shown in Figure 7, in some embodiments, non-target areas on the ultrasound reconstruction image are eliminated to obtain a first target image, including: inputting the ultrasound reconstruction image into an ultrasound image segmentation model to obtain an ultrasound binary image; the ultrasound image segmentation model is obtained by training an ultrasound image segmentation network based on deep learning; non-target areas on the ultrasound reconstruction image are eliminated according to the ultrasound binary image; the segmentation network includes an encoder 7 and a decoder 8; the encoder 7 and the decoder 8 are both provided with a number of converter modules 3; a fusion layer 4 is provided between adjacent converter modules 3 in the encoder 7; an expansion layer 5 is provided between adjacent converter modules 3 in the decoder 8; and a jump connection 9 is provided between the encoder 7 and the decoder 8.

[0052] Specifically, the encoder 7 and decoder 8 form an ultrasound prostate image segmentation network with a three-dimensional U-Net structure. The encoder 7 includes an input layer 1, an embedding layer 2, four transformer modules 3 arranged at interleaved intervals, and three fusion layers 4. The decoder 8 includes an output layer 6, four transformer modules 3 arranged at interleaved intervals, and four expansion layers 5. Following the direction of traversing the segmentation network from the input layer 1 to the output layer 6, a jump connection 9 is provided between the embedding layer 2 and the last expansion layer 5; and a jump connection 9 is provided between the last fusion layer 4 and the first expansion layer 5.

[0053] It is worth noting that the segmentation network includes M fusion layers 4, M+1 expansion layers 5, and 2M+2 converter modules 3, where M is a positive integer. A skip connection 9 is provided between the Lth fusion layer 4 and the M-L+1th layer, where L is a positive integer between [1, M].

[0054] As shown in Figure 8, in some embodiments, the converter module 3 includes: a first feature layer and a second feature layer, each of the first feature layer and the second feature layer includes a convolution operator 31, a normalization operator 32 and an activation function 33; the first feature layer includes a global attention layer 34, and the second feature layer includes a local attention layer 35; at least one of the first feature layer and the second feature layer includes a cross-layer connection for inter-layer fusion.

[0055] It is worth noting that the segmentation network is implemented based on U-Net, retaining its lightweight nature while enhancing the extraction of global and local features, significantly improving the accuracy of target segmentation. The segmentation network can also obtain preset contour information as auxiliary input. This contour information is used to outline the starting, ending, and intermediate layers of the prostate, assisting in prostate segmentation constraint and verification, thereby improving the accuracy of the segmentation network.

[0056] Specifically, a normalization operator 32 and a global attention layer 34 are provided at the input end of the first feature layer; the output end of the global attention layer 34 and the input end of the normalization operator 32 undergo a first inter-layer fusion, and the first inter-layer fusion result passes through an activation function 33 and then undergoes two normalization operators 32 and two convolution operators 31 to obtain a convolution result, and the convolution result is subjected to a second inter-layer fusion with the first inter-layer fusion result.

[0057] In other specific embodiments, a normalization operator 32 and a local attention layer 35 are provided at the input end of the second feature layer; the output end of the local attention layer 35 and the input end of the normalization operator 32 are subjected to a third inter-layer fusion, and the result of the third inter-layer fusion passes through an activation function 33 and then undergoes two normalization operators 32 and two convolution operators 31 to obtain a convolution result, and the convolution result is subjected to a fourth inter-layer fusion with the result of the third inter-layer fusion, and the result of the fourth inter-layer fusion is used as the output of the converter module 3.

[0058] In some embodiments, a nuclear magnetic resonance image containing a target object is obtained, and non-target areas on the nuclear magnetic resonance image are eliminated to obtain a second target image, including: performing nonlinear interpolation on the nuclear magnetic resonance image to obtain a smooth interpolation image, and inputting the smooth interpolation image into a nuclear magnetic resonance image segmentation model to obtain a nuclear magnetic resonance binary image; the nuclear magnetic resonance image segmentation model is obtained by training a nuclear magnetic resonance image segmentation network based on deep learning; and non-target areas on the nuclear magnetic resonance interpolation image are eliminated based on the nuclear magnetic resonance binary image.

[0059] It is worth noting that in other embodiments, a threshold processing method, a region growing method, an edge detection method, or a level set method can also be used to segment ultrasound images or nuclear magnetic resonance images. The threshold processing and region growing methods include: first performing preprocessing such as image enhancement, separating the prostate from surrounding tissues based on a preset or adaptive threshold and region growing, thereby converting the nuclear magnetic resonance image into a prostate binary image. The enhancement method includes an edge detection method, which includes: detecting the edge of the prostate using an edge detection algorithm, adding it to the original image to obtain an enhanced image, and the edge detection algorithm includes a Sobel operator and a Canny operator. The level set method includes: deducing the target edge curve based on the grayscale gradient information of the image edge and regional features, which can automatically update the parameters during the curve evolution process to achieve accurate segmentation of the prostate.

[0060] Specifically, based on the above-mentioned ultrasound prostate segmentation network, the training set is replaced with a nuclear magnetic resonance image containing the prostate, and the nuclear magnetic resonance image segmentation model is retrained. Since the resolution of nuclear magnetic resonance images is lower than that of ultrasound images, the nuclear magnetic resonance images are interpolated by orthogonal wavelet transform to separate the high- and low-frequency information of the image. The separated low-frequency information contains most of the energy of the image, and the high-frequency information corresponds to the edge information of the image. In this embodiment, the high-frequency information is processed separately by wavelet transform to obtain the high-frequency details of the image. The reconstruction process of the wavelet transform is used to superimpose the obtained high-frequency details with the original low-frequency details, and then a high-resolution nuclear magnetic resonance image is obtained by inverse discrete wavelet transform. It is worth noting that the resolution of nuclear magnetic resonance images can also be improved by using a super-resolution method based on deep learning.

[0061] In some embodiments, the elastic registration adopts a pyramid multi-resolution multi-metric elastic registration method; the pyramid multi-resolution represents the number of multi-resolution pyramids set according to the size of the first target image; the multi-metric includes a normalized mutual information metric and an enhanced correlation coefficient metric; the normalized mutual information metric NMI(A,B) satisfies:

[0062] Among them, H(A) is the information entropy of image A, H(B) is the information entropy of image B, H(A,B) is the joint entropy of images A and B, and the information entropy H(A) of image A satisfies:

[0063] The joint entropy H(A,B) of image A and image B satisfies:

[0064] Where N is the number of grayscale values ​​in the image, p i is the probability of the pixel with gray value i appearing, p AB(a, b) is the probability that the pixel value in image A is a and the pixel value in image B is b; the enhanced correlation coefficient metric ECC(A, B) satisfies:

[0065] The evaluation index metric of the elastic registration method satisfies: metirc(A,B)=α*NMI(A,B)+β*ECC(A,B)

[0066] Where: α+β=1

[0067] α is the mutual information weight, and β is the enhanced correlation coefficient weight. α and β can be adjusted according to actual conditions. The elastic registration method uses deformation field transformation. The interpolator uses at least one of linear interpolation and B-spline interpolation. The evaluation index is optimized using at least one of an adaptive stochastic gradient descent algorithm, an adaptive stochastic LBFGS algorithm, and a quasi-Newton LBFGS algorithm. This method has the advantage of enhancing the accuracy and robustness of registration at multiple resolutions and metrics.

[0068] As shown in FIG9 , a second embodiment provides a registration device 100 for incomplete medical images, which is used in any of the methods described in the above embodiments and includes: a three-dimensional reconstruction unit 101 for acquiring an ultrasound scan image containing an incomplete target object and performing three-dimensional reconstruction based on the ultrasound scan image to obtain an ultrasound reconstruction image; an ultrasound image segmentation unit 102 for eliminating non-target areas on the ultrasound reconstruction image to obtain a first target image; a nuclear magnetic resonance image segmentation unit 103 for acquiring a nuclear magnetic resonance image containing the target object and eliminating non-target areas on the nuclear magnetic resonance image to obtain a second target image; a registration unit 104 for performing rigid registration on the first target image and the second target image to obtain a transformation matrix; transforming the second target image into an ultrasound coordinate system according to the transformation matrix to obtain a third target image; correlating the third target image with the first target image to obtain a fourth target image; wherein the target contour in the fourth target image is consistent with the target contour in the first target image; performing elastic registration on the first target image and the fourth target image to obtain a deformation field, and transforming the fourth target image into an ultrasound coordinate system according to the deformation field to obtain a fifth target image; and the fifth target image is used to provide a navigation target for a medical operation.

[0069] In some embodiments, the three-dimensional reconstruction unit 101 is used to perform image noise reduction processing on the ultrasonic scanning image before performing the three-dimensional reconstruction, and the noise reduction algorithm includes at least one of a three-dimensional block matching algorithm, a wavelet transform, a principal component analysis algorithm and a bilateral filtering algorithm; the three-dimensional reconstruction unit 101 is also used to traverse each voxel in the three-dimensional imaging space when performing the three-dimensional reconstruction, find one or more corresponding pixel values ​​in the two-dimensional ultrasonic scanning image sequence through transformation, and then use an interpolation method to calculate the corresponding pixel value and assign the pixel value to the voxel.

[0070] As shown in Figure 10, the third embodiment provides an electronic device 200, including a memory 202 and a processor 201, wherein the memory 202 stores a program that can be run on the processor 201. When the program is executed by the processor 201, the electronic device 200 implements any one of the methods in the above embodiments.

[0071] Specifically, the processor 201 is further connected to an interface 203, which is used to connect to a puncture needle navigation device and provide the third target image to the puncture needle navigation device. The puncture needle navigation device adjusts the position of the puncture needle according to the third target image so that the tip of the puncture needle reaches the prostate.

[0072] It should be noted that the processor 201 in this embodiment can be an image processing chip or an integrated circuit chip capable of processing image signals. During implementation, each step of the above-described method embodiment can be completed by hardware integrated logic circuits in the processor or by software instructions. The above-described processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in this embodiment can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this embodiment can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above-described method.

[0073] It is understood that the memory 202 in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0074] A fourth embodiment provides a readable storage medium, wherein the readable storage medium stores a program, and when the program is executed, the method according to any one of the above embodiments is implemented.

[0075] It is worth noting that if the method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing an electronic device to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0076] This embodiment also provides a program product, which, when executed by an electronic device, implements the method described in any of the above method embodiments.

[0077] While the embodiments of the present invention have been described in detail above, it will be apparent to those skilled in the art that various modifications and variations of these embodiments are possible. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as set forth in the claims. Furthermore, the invention described herein is susceptible to other embodiments and may be practiced or implemented in a variety of ways.

Claims

1. A registration method for incomplete medical images, characterized in that: include: Acquire an ultrasonic scanning image containing a defective target object, and perform three-dimensional reconstruction based on the ultrasonic scanning image to obtain an ultrasonic reconstruction image; Eliminating non-target areas on the ultrasound reconstruction image to obtain a first target image; Acquire a nuclear magnetic resonance image containing a target object, and remove non-target areas on the nuclear magnetic resonance image to obtain a second target image; Performing rigid registration according to the first target image and the second target image to obtain a transformation matrix; The second target image is converted into an ultrasonic coordinate system according to the transformation matrix to obtain a third target image; Correlation processing is performed on the third target image and the first target image to obtain a fourth target image; the target contour in the fourth target image is consistent with the target contour in the first target image; elastically registering the first target image and the fourth target image to obtain a deformation field, and converting the fourth target image into an ultrasonic coordinate system according to the deformation field to obtain a fifth target image; The fifth target image is used to provide a navigation target for a medical operation.

2. The method according to claim 1, characterized in that The correlating process of the third target image and the first target image to obtain the fourth target image includes: Calculate the start and end layers of the first target image, remove the part outside the start and end layers on the third target image, so that the part outside the start and end layers on the nuclear magnetic resonance image after rigid registration is set to zero, and obtain the fourth Target image.

3. The method according to claim 1, characterized in that The step of acquiring an ultrasonic scanning image containing a defective target object and performing three-dimensional reconstruction according to the ultrasonic scanning image to obtain an ultrasonic reconstruction image includes: Before performing the three-dimensional reconstruction, the ultrasonic scanning image is subjected to image noise reduction processing, wherein the noise reduction algorithm includes at least one of a three-dimensional block matching algorithm, a wavelet transform, an algorithm based on principal component analysis, and a bilateral filtering algorithm; When performing the three-dimensional reconstruction, each voxel in the three-dimensional imaging space is traversed, and the corresponding one or more pixel values ​​in the two-dimensional ultrasonic scanning image sequence are found through transformation, and then the corresponding pixel values ​​are calculated using an interpolation method, and the pixel values ​​are assigned to the voxels; the interpolation method includes a distance weighted algorithm, a linear interpolation algorithm or a trajectory tracking algorithm.

4. The method according to claim 3, characterized in that The distance weighted algorithm satisfies: Among them, I(V c ) represents the current voxel pixel value, is the pixel value of the kth voxel in the nth neighborhood of the current voxel, W k is the weight corresponding to the pixel value of the kth voxel, d k is the distance of the kth voxel relative to the current voxel.

5. The method according to claim 1, characterized in that Eliminating the non-target area on the ultrasound reconstruction image to obtain the first target image, including: Inputting the ultrasound reconstruction image into the ultrasound image segmentation model to obtain an ultrasound binary image; the ultrasound image segmentation model is obtained by training an ultrasound image segmentation network based on deep learning; and removing non-target areas on the ultrasound reconstruction image according to the ultrasound binary image; The segmentation network includes an encoder and a decoder; the encoder and the decoder are both provided with a plurality of converter modules; a fusion layer is provided between adjacent converter modules in the encoder; an expansion layer is provided between adjacent converter modules in the decoder; and a jump connection is provided between the encoder and the decoder.

6. The method according to claim 5, characterized in that The converter module comprises: A first feature layer and a second feature layer, wherein the first feature layer and the second feature layer both include a convolution operator, a normalization operator, and an activation function; The first feature layer includes a global attention layer, and the second feature layer includes a local attention layer; At least one of the first feature layer and the second feature layer includes a cross-layer connection for inter-layer fusion.

7. The method according to claim 1, characterized in that Acquiring a nuclear magnetic resonance image containing a target object and removing a non-target area on the nuclear magnetic resonance image to obtain a second target image includes: Nonlinear interpolation is performed on the nuclear magnetic image to obtain a smooth interpolation image, and the smooth interpolation image is input into a nuclear magnetic image segmentation model to obtain a nuclear magnetic binary image; the nuclear magnetic image segmentation model is obtained by training a nuclear magnetic image segmentation network based on deep learning; according to the nuclear magnetic The binary image removes non-target areas on the NMR interpolation image.

8. The method according to claim 1, characterized in that The elastic registration adopts a pyramid multi-resolution multi-metric elastic registration method; the pyramid multi-resolution represents the number of multi-resolution pyramids set according to the size of the first target image; the multi-metric includes a normalized mutual information metric and an enhanced correlation coefficient metric; The normalized mutual information metric NMI(A,B) satisfies: Among them, H(A) is the information entropy of image A, H(B) is the information entropy of image B, H(A,B) is the joint entropy of images A and B, and the information entropy H(A) of image A satisfies: The joint entropy H(A,B) of image A and image B satisfies: Where N is the number of grayscale values ​​in the image, p i is the probability of the pixel with gray value i appearing, p AB (a,b) is the probability that the pixel value in image A is a and the pixel value in image B is b; The enhanced correlation coefficient metric ECC(A,B) satisfies: The evaluation index metric of the elastic registration method satisfies: metirc(A,B)=α*NMI(A,B)+β*ECC(A,B) in: α+β＝1 α is the mutual information weight, β is the enhanced correlation coefficient weight, and α and β can be adjusted according to actual conditions. The elastic registration method adopts deformation field transformation. The interpolator adopts at least one of linear interpolation and b-spline interpolation. The evaluation index is optimized by at least one method of adaptive stochastic gradient descent algorithm, adaptive stochastic LBFGS, and quasi-Newton LBFGS.

9. A registration device for incomplete medical images, used in the method according to any one of claims 1 to 8, characterized in that: include: A three-dimensional reconstruction unit, used for acquiring an ultrasonic scanning image containing a defective target object, and performing three-dimensional reconstruction according to the ultrasonic scanning image to obtain an ultrasonic reconstruction image; An ultrasonic image segmentation unit, used for removing non-target areas on the ultrasonic reconstruction image to obtain a first target image; A nuclear magnetic image segmentation unit is used to obtain a nuclear magnetic image containing a target object and remove non-target areas on the nuclear magnetic image to obtain a second target image; A registration unit, used for performing rigid registration according to the first target image and the second target image to obtain a transformation matrix; The second target image is converted into the ultrasound coordinate system according to the transformation matrix to obtain the third target image; and the third target image and the first target image are correlated. Obtaining a fourth target image; wherein the target contour in the fourth target image is consistent with the target contour in the first target image; elastically registering the first target image and the fourth target image to obtain a deformation field, and converting the fourth target image into an ultrasonic coordinate system according to the deformation field to obtain a fifth target image; The fifth target image is used to provide a navigation target for a medical operation.

10. The device according to claim 9, characterized in that The three-dimensional reconstruction unit is used to perform image noise reduction processing on the ultrasonic scanning image before performing the three-dimensional reconstruction, and the noise reduction algorithm includes at least one of a three-dimensional block matching algorithm, a wavelet transform, an algorithm based on principal component analysis, and a bilateral filtering algorithm; The three-dimensional reconstruction unit is also used to traverse each voxel in the three-dimensional imaging space when performing the three-dimensional reconstruction, find out the corresponding one or more pixel values ​​in the two-dimensional ultrasonic scanning image sequence through transformation, and then use the interpolation method to calculate the corresponding pixel value and assign the pixel value to the voxel.

11. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory stores a program executable on the processor, and when the program is executed by the processor, the electronic device implements the method according to any one of claims 1 to 8.

12. A readable storage medium, wherein a program is stored in the readable storage medium, characterized in that: When the program is executed, the method according to any one of claims 1 to 8 is implemented.

Images