Medical image fusion method and device based on hole repair
By generating a repair matrix and directly repairing the fusion medical images, the problem of hollowness after fusion of CT images and MRI images is solved, and high-quality fusion images can be obtained without secondary processing.
Patent Information
- Application Number
- CN202411538307.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2025-03-25
AI Technical Summary
CT图像和MRI图像融合后容易出现空洞，需要进行二次处理。
By acquiring CT and MRI tag images, generating a repair matrix, and inputting it into the fusion structure, the fusion medical images are directly repaired to solve the hollow problem.
Without secondary processing of the fusion image, a repair image without holes is obtained directly, which improves the accuracy and efficiency of image fusion.
Smart Images
Figure CN119693245A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of medical image recognition, and in particular to a medical image fusion method and device based on cavity repair. Background Art
[0002] CT (computed tomography) and MRI (magnetic resonance imaging) are two common imaging technologies in the field of medical imaging, each with different characteristics and applications. CT provides high-resolution X-ray imaging, while MRI provides more information about tissue structure and biological tissue properties. The registration and fusion of CT images and MRI images is crucial for computer-aided diagnosis or surgical navigation positioning.
[0003] However, when CT images and MRI images are fused, holes are likely to appear in the fused image due to the differences between different modalities such as CT images and MRI images, and secondary processing is required. Summary of the invention
[0004] The problem solved by this application is that the fusion of CT images and MRI images requires secondary processing.
[0005] To solve the above problems, the first aspect of the present application provides a medical image fusion method based on hole repair, comprising:
[0006] Acquire a CT medical image and an MRI medical image, wherein the CT medical image and the MRI medical image are medical images of the same part of the same object;
[0007] Inputting the CT medical image and the MRI medical image into a registration structure to obtain a registration matrix of the CT medical image and the MRI medical image, a CT label image, and an MRI label image;
[0008] Inputting the CT label image and the MRI label image into an edge repair structure to obtain a repair matrix of the CT label image and the MRI label image;
[0009] The CT medical image, the MRI medical image, the registration matrix and the repair matrix are input into the fusion structure to obtain the repaired fused medical image.
[0010] The second aspect of the present application provides a medical image fusion device based on cavity repair, which includes:
[0011] An image acquisition module, which is used to acquire a CT medical image and an MRI medical image, wherein the CT medical image and the MRI medical image are medical images of the same part of the same object;
[0012] An image registration module, which is used to input the CT medical image and the MRI medical image into a registration structure to obtain a registration matrix of the CT medical image and the MRI medical image, a CT label image, and an MRI label image;
[0013] An image restoration module, which is used to input the CT label image and the MRI label image into an edge restoration structure to obtain restoration matrices of the CT label image and the MRI label image;
[0014] The image fusion module is used to input the CT medical image, the MRI medical image, the registration matrix and the repair matrix into the fusion structure to obtain the repaired fused medical image.
[0015] A third aspect of the present application provides an electronic device, comprising: a memory and a processor;
[0016] The memory is used to store programs;
[0017] The processor, coupled to the memory, is configured to execute the program to:
[0018] Acquire a CT medical image and an MRI medical image, wherein the CT medical image and the MRI medical image are medical images of the same part of the same object;
[0019] Inputting the CT medical image and the MRI medical image into a registration structure to obtain a registration matrix of the CT medical image and the MRI medical image, a CT label image, and an MRI label image;
[0020] Inputting the CT label image and the MRI label image into an edge repair structure to obtain a repair matrix of the CT label image and the MRI label image;
[0021] The CT medical image, the MRI medical image, the registration matrix and the repair matrix are input into the fusion structure to obtain the repaired fused medical image.
[0022] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the above-mentioned medical image fusion method based on hole repair.
[0023] In the present application, a repair branch is set. After obtaining the registration matrix based on the registration structure, the repair matrix is generated by the CT label image and the MRI label image, so as to directly repair the fused medical image to obtain a repaired fused medical image; thus, there is no need to perform secondary processing on the fused image, and a repaired image without holes can be directly obtained.
[0024] In the present application, the annotated CT images and MRI images are processed to eliminate the interference of non-registration area data and improve the accuracy of registration and fusion. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 is a flow chart of a medical image fusion method based on hole repair according to an embodiment of the present application;
[0026] Figure 2 A model architecture diagram of a medical image fusion method based on hole repair according to an embodiment of the present application;
[0027] Figure 3 It is an architecture diagram of a registration unit of a medical image fusion method based on hole repair according to an embodiment of the present application;
[0028] Figure 4 It is an architecture diagram of residual processing of a medical image fusion method based on hole repair according to an embodiment of the present application;
[0029] Figure 5 It is an architectural diagram of the edge repair structure of the medical image fusion method based on hole repair according to an embodiment of the present application;
[0030] Figure 6 It is an architectural diagram of the fusion structure of the medical image fusion method based on hole repair according to an embodiment of the present application;
[0031] Figure 7 A flowchart of a model training of a medical image fusion method based on hole repair according to an embodiment of the present application;
[0032] Figure 8 is a structural block diagram of a medical image fusion device based on cavity repair according to an embodiment of the present application;
[0033] Fig. 9 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0034] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below in conjunction with the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.
[0035] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.
[0036] In response to the above problems, the present application provides a new medical image fusion solution based on void repair, which can use the branches of CT label images and MRI label images for repair, solving the problem of requiring secondary processing after the fusion of CT images and MRI images.
[0037] The present application embodiment provides a medical image fusion method based on hole repair, and the specific scheme of the method is as follows: Figure 1-Figure 7 As shown, the method can be performed by a medical image fusion device based on hole repair, and the medical image fusion device based on hole repair can be integrated in electronic devices such as computers, servers, computers, server clusters, and data centers. Figure 1 , Figure 2 As shown, it is a flow chart of a medical image fusion method based on hole repair according to an embodiment of the present application; wherein the medical image fusion method based on hole repair includes:
[0038] S101, acquiring a CT medical image and an MRI medical image, wherein the CT medical image and the MRI medical image are medical images of the same part of the same object;
[0039] S102, inputting the CT medical image and the MRI medical image into a registration structure to obtain a registration matrix of the CT medical image and the MRI medical image, a CT label image, and an MRI label image;
[0040] In the present application, the label image is a segmentation result image, that is, a medical image containing only bones.
[0041] In this application, the CT label image is the bone segmentation result of the CT original image; the MRI label image is the bone segmentation result of the MRI original image.
[0042] S103, inputting the CT label image and the MRI label image into an edge repair structure to obtain a repair matrix of the CT label image and the MRI label image;
[0043] S104, inputting the CT medical image, the MRI medical image, the registration matrix and the restoration matrix into the fusion structure to obtain a restored fused medical image.
[0044] In the present application, a repair branch is set. After obtaining the registration matrix based on the registration structure, the repair matrix is generated by the CT label image and the MRI label image, so as to directly repair the fused medical image to obtain a repaired fused medical image; thus, there is no need to perform secondary processing on the fused image, and a repaired image without holes can be directly obtained.
[0045] In the present application, the annotated CT images and MRI images are processed to eliminate the interference of non-registration area data and improve the accuracy of registration and fusion.
[0046] In one embodiment, acquiring a CT medical image and an MRI medical image further includes:
[0047] The step of adjusting the dimension of the MRI medical image based on the CT medical image includes:
[0048] Acquiring three-dimensional data of the CT medical image;
[0049] Adjusting the sagittal MRI medical image to an axial MRI medical image;
[0050] The axial MRI medical image is interpolated, and the three-dimensional dimensional data of the MRI medical image after the interpolation processing is consistent with the three-dimensional dimensional data of the CT medical image.
[0051] In medical imaging, especially scenes involving three-dimensional data (such as MRI and CT scans), three main planes are usually used to describe different sections of human anatomy. These three main planes are the axial plane, sagittal plane, and coronal plane.
[0052] Among them, the axial plane, also called the transverse section or cross section, is a plane that cuts the body horizontally. It divides the body into upper and lower parts. The sagittal plane is a plane that cuts the body longitudinally, which divides the body into left and right halves.
[0053] Among them, CT images generally obtain axial data through tomography, and aggregate multiple toms into three-dimensional data; MRI images generally obtain sagittal data, and aggregate multiple sagittal data into three-dimensional data.
[0054] In the present application, the CT medical image is an axial image, and the MRI medical image is a sagittal image; wherein, the three-dimensional data format of the CT medical image is W1 / H1 / D1, and the three-dimensional data format of the MRI medical image is D2 / W2 / H2; by changing the length, width and height of the MRI medical image, the sagittal MRI medical image is adjusted to the axial MRI medical image, that is, W2 / H2 / D2.
[0055] In the present application, the axial MRI medical image is interpolated through interpolation processing to make it consistent with the three-dimensional data of the CT medical image.
[0056] That is, the MRI medical image of W2 / H2 / D2 is interpolated to obtain the MRI medical image of W1 / H1 / D1.
[0057] In the present application, the dimensionality adjustment of the MRI medical image is essentially the dimensionality adjustment of the MRI original image based on the CT original image.
[0058] In one embodiment, in combination Figure 7 As shown, before acquiring the CT medical image and the MRI medical image, the method further includes:
[0059] S201, acquiring a CT training image and an MRI training image; the CT training image includes a CT original image and a CT sample image, and the MRI training image includes an MRI original image and an MRI sample image, and is annotated with a fusion result;
[0060] S202, inputting the CT original image and the MRI original image into the registration structure to obtain a predicted registration matrix;
[0061] S203, inputting the CT sample image and the MRI sample image into an edge repair structure to obtain a predicted repair matrix;
[0062] S204, inputting the CT original image, the MRI original image, the predicted registration matrix and the predicted restoration matrix into the fusion structure to obtain a predicted fusion image;
[0063] S205, calculating a first loss according to the predicted registration matrix, the CT sample image, and the MRI sample image;
[0064] S206, calculating a second loss according to the predicted registration matrix, the CT original image, and the MRI original image;
[0065] S207, calculating a third loss according to the predicted fusion image and the fusion result;
[0066] S208, determining an overall loss of the deep convolutional model based on the first loss, the second loss, and the third loss;
[0067] S209, iterating the deep convolutional model based on the overall loss until the loss converges.
[0068] In this application, the first loss is calculated based on the predicted registration matrix, the CT sample image and the MRI sample image. This loss measures the accuracy of the registration structure for aligning CT and MRI images. The purpose of the first loss is to optimize the registration capability of the model so that the alignment of CT and MRI images is more accurate.
[0069] In this application, the second loss is calculated based on the predicted registration matrix, the CT original image and the MRI original image to evaluate the alignment effect of the original image under the action of the registration matrix; the second loss is used to evaluate the generalization ability of the registration effect in non-sample images, thereby improving the robustness of the model.
[0070] In this application, the third loss is calculated based on the predicted fusion image and the annotated fusion result to measure the difference between the predicted fusion image and the actual fusion result. The third loss is mainly used to optimize the fusion structure so that the final output fusion image is closer to the ideal annotated result.
[0071] In this application, the overall loss of the model is calculated based on the first loss, the second loss and the third loss. The overall loss can combine different losses in a weighted manner to ensure the effectiveness of the model in terms of registration, restoration and fusion.
[0072] In one embodiment, in combination Figure 2 As shown, the step of inputting the CT medical image and the MRI medical image into a registration structure to obtain a registration matrix of the CT medical image and the MRI medical image, a CT label image, and an MRI label image includes:
[0073] Inputting the CT medical image into a first segmentation unit to obtain a CT label image;
[0074] Inputting the MRI medical image into a first segmentation unit to obtain an MRI label image;
[0075] The CT label image and the MRI label image are input into a registration unit to obtain a registration matrix of the CT label image and the MRI label image, where the registration matrix is a registration matrix of the CT medical image and the MRI medical image.
[0076] In the present application, the CT medical image is input into the first segmentation unit, which is a deep learning model structure (such as U-Net, FCN, etc.) for automatically identifying and segmenting anatomical structures of interest such as bones.
[0077] In the present application, the first segmentation unit for MRI medical image input and the first segmentation unit for CT medical image input have the same structure but different parameters, thereby increasing the homology of the segmented label images.
[0078] In this application, the characteristic that CT label images and MRI label images have no data except the marked areas is utilized to avoid the interference of data with low correlation with the registration fusion on the deformation field, thereby improving the accuracy of the registration.
[0079] In one embodiment, in combination Figure 3As shown, the CT label image and the MRI label image are input into the registration unit to obtain the registration matrix of the CT label image and the MRI label image, including:
[0080] Splicing the CT label image and the MRI label image to obtain a spliced label image;
[0081] Perform multiple rounds of convolution processing, residual processing, convolution processing and downsampling processing on the spliced label image to obtain multiple downsampled feature maps;
[0082] Perform multiple rounds of upsampling, concatenation, residual processing, and convolution processing on the downsampled feature map with the smallest size to obtain multiple upsampled feature maps;
[0083] The up-sampled feature map with the largest size is convolved to obtain the registration matrix.
[0084] In this application, the CT label image and the MRI label image are spliced along the channel dimension (or specific axis) to form a new spliced label image. This spliced image contains the feature information of the two label images so that the network can learn registration in the same space. The splicing operation combines the anatomical information of CT and MRI so that subsequent convolution operations can refer to both information at the same time.
[0085] In this application, the concatenated label image is first subjected to multiple rounds of convolution processing to extract low-level and high-level features. The size and number of convolution kernels can be adjusted as needed. In this application, downsampling processing: downsampling (usually achieved by pooling or stride convolution) is performed to gradually reduce the size of the feature map and increase the receptive field. The downsampled feature map contains higher-level semantic information.
[0086] In this application, the upsampling process is as follows: starting from the smallest downsampled feature map, multiple rounds of upsampling operations are performed to gradually restore the spatial resolution of the feature map. During the upsampling process, the upsampled feature map is spliced on the channel with the corresponding downsampled feature map (usually called a "skip connection" or "skip connection"), combining low-level detail information and high-level semantic information. The concatenated feature map is convoluted to extract the integrated features and generate multiple upsampled feature maps of increasing size.
[0087] In this application, the largest upsampled feature map is input into the convolutional layer for processing, and the final registration matrix is output. This matrix contains the transformation parameters required to align the CT and MRI label images.
[0088] In this application, the registration matrix is used to spatially align CT and MRI medical images in subsequent steps, thereby providing a basis for image fusion.
[0089] In this application, through the multi-step processing of convolution, residual, downsampling and upsampling, the network can effectively extract, integrate and generate a registration matrix for CT and MRI image registration. The matrix utilizes the anatomical structure and edge information of the CT and MRI label images, thereby improving the registration accuracy and robustness.
[0090] In one embodiment, in combination Figure 4 As shown, the residual processing process includes:
[0091] After continuous convolution processing on the input feature map, it is added to the input feature map to obtain the first convolution feature map;
[0092] After continuously convolution processing is performed on the first convolution feature map, the first convolution feature map is added to the first convolution feature map to obtain a second convolution feature map;
[0093] After continuously convolving the second convolution feature map, the third convolution feature map is obtained by adding the second convolution feature map and the input feature map.
[0094] After pooling, upsampling and ReLU processing of the input feature map, the convolution coefficient is obtained;
[0095] After multiplying the third convolution feature map by the convolution coefficient, it is added to the input feature map to obtain the output feature map.
[0096] In this application, the input feature map is processed through two consecutive convolutional layers (possibly with nonlinear activation functions) to extract deeper features. In this application, the convolved feature map is added element by element to the original input feature map to form a first convolutional feature map. This operation is used to retain the information in the input feature map while adding new features.
[0097] In this application, the first convolution feature map is passed through two consecutive convolution layers to further extract deep features. In this application, the convolution result is added to the first convolution feature map to generate a second convolution feature map. This operation allows more deep features to be superimposed on the basic features.
[0098] In this application, the second convolution feature map is subjected to continuous convolution processing again to extract richer feature information. In this application, the convolution result is added to the second convolution feature map and the original input feature map to generate a third convolution feature map. Multiple additions enhance the flow of information and ensure that both the initial features and the intermediate features can affect the final output.
[0099] In this application, the input feature map is pooled to reduce its spatial resolution to capture a wider range of contextual information. In this application, the pooled feature map is upsampled to restore it to the original size of the input feature map. In this application, the upsampled feature map is activated by ReLU to generate convolution coefficients. ReLU ensures that the coefficients are non-negative and provides activated feature information for subsequent multiplication operations.
[0100] In this application, the third convolution feature map is element-wise multiplied by the convolution coefficient to enhance or suppress a specific area in the third convolution feature map. In this application, the multiplication result is added to the input feature map to generate an output feature map. Through this multiplication-addition operation, the model can adaptively integrate features of different depths to obtain enhanced feature expression.
[0101] In this application, the residual processing process uses multi-level convolution addition, pooling and convolution coefficient multiplication to enable the feature map to efficiently retain input information and introduce deep features. The final output feature map not only contains basic features, but also has enhanced representation capabilities through residuals and adaptive coefficients.
[0102] In one embodiment, in combination Figure 5 As shown, the step of inputting the CT label image and the MRI label image into an edge repair structure to obtain a repair matrix of the CT label image and the MRI label image includes:
[0103] Perform multiple convolution processing on the CT label image to obtain a first multi-convolution feature map;
[0104] Perform multiple convolutions on the MRI label image to obtain a second multi-convolution feature map;
[0105] Inputting the first multi-convolution feature map into a first edge detection module to obtain a first edge feature map;
[0106] Inputting the second multi-convolution feature map into a second edge detection module to obtain a second edge feature map;
[0107] After the first edge feature map and the second edge feature map are added, multiple convolution processes are performed to obtain restoration matrices of the CT label image and the MRI label image.
[0108] In this application, multiple convolutional layers (usually 3-5 layers) are used to perform convolution operations on the CT label image to extract features at different levels and form a first multi-convolutional feature map. An activation function (such as ReLU) and a normalization layer can be added after the convolutional layer to further enhance the feature expression.
[0109] In this application, multiple convolution layers are also used to process the MRI label image to generate a second multi-convolution feature map. In this application, as with CT processing, activation functions and normalization layers are added to optimize feature learning.
[0110] In this application, the first multi-convolution feature map is input into a specially designed edge detection module, which is usually based on a convolutional neural network edge detection algorithm. The module extracts edge features through the gradient information in the feature map and outputs a first edge feature map to highlight the edge information in the CT image.
[0111] In the present application, the second multi-convolution feature map is input into the second edge detection module, and a similar edge detection algorithm is used. In the present application, the second edge feature map is output to emphasize the edge information in the MRI image.
[0112] In the present application, the first edge feature map and the second edge feature map are added element by element, and the edge information of the two is merged to form a comprehensive edge feature map. In the present application, the synthesized edge feature map is subjected to multiple rounds of convolution processing to further extract and enhance features and generate a final repair matrix. In the present application, the repair matrix describes how to repair missing or blurred parts in CT label images and MRI label images based on edge information, providing support for subsequent image fusion.
[0113] In this application, through multi-layer convolution and edge detection, the edge features in CT and MRI label images are effectively extracted, and the repair matrix is generated by addition and further convolution. This process provides strong support for the fusion and analysis of medical images, making the details and edges in the image clearer and more accurate.
[0114] In one embodiment, in combination Figure 6 As shown, the CT medical image, the MRI medical image, the registration matrix and the repair matrix are input into the fusion structure to obtain the repaired fused medical image, including:
[0115] Multiplying the MRI medical image and the registration matrix to obtain a registration feature map;
[0116] Multiply the registration feature map and the repair matrix to obtain the repair feature map;
[0117] The restoration feature map and the CT medical image are added to obtain a fused medical image.
[0118] In the present application, the sizes of the MRI medical image and the registration matrix are ensured to match so as to perform matrix multiplication. In the present application, spatial alignment is achieved by multiplying the MRI medical image with the registration matrix to generate a registration feature map.
[0119] In the present application, the registration feature map retains the anatomical features of the MRI image and is adjusted to the coordinate system of the CT image in preparation for subsequent processing.
[0120] In this application, the registration feature map is multiplied with the repair matrix to enhance and fill the features. In this application, this operation helps to improve the details and edges in the registration feature map, thereby improving the image quality.
[0121] In this application, the repair feature map is added element by element to the CT medical image to fuse the information of the two. In this application, the final generated fused medical image combines the CT and repaired MRI information to provide a more comprehensive anatomical and pathological view.
[0122] In the present application, the bone edge of CT and the cartilage edge of MRI are used to determine where the cartilage needs to be supplemented based on the holes between the cartilage and the bone.
[0123] In the present application, MRI is mapped to CT according to the first registration matrix and fused with the area where the cartilage needs to be completed to obtain the final fusion result.
[0124] An embodiment of the present application provides a medical image fusion device based on hole repair, which is used to execute a medical image fusion method based on hole repair described in the above content of the present application. The medical image fusion device based on hole repair is described in detail below.
[0125] like Figure 8 As shown, the medical image fusion device based on hole repair includes:
[0126] An image acquisition module 101 is used to acquire a CT medical image and an MRI medical image, wherein the CT medical image and the MRI medical image are medical images of the same part of the same object;
[0127] An image registration module 102, which is used to input the CT medical image and the MRI medical image into a registration structure to obtain a registration matrix of the CT medical image and the MRI medical image, a CT label image, and an MRI label image;
[0128] An image restoration module 103, which is used to input the CT label image and the MRI label image into an edge restoration structure to obtain restoration matrices of the CT label image and the MRI label image;
[0129] The image fusion module 104 is used to input the CT medical image, the MRI medical image, the registration matrix and the restoration matrix into the fusion structure to obtain a restored fused medical image.
[0130] In one embodiment, a model training module is also included, which is used to:
[0131] Acquire a CT training image and an MRI training image; the CT training image includes a CT original image and a CT sample image, and the MRI training image includes an MRI original image and an MRI sample image, and is annotated with a fusion result; input the CT original image and the MRI original image into the registration structure to obtain a predicted registration matrix; input the CT sample image and the MRI sample image into the edge repair structure to obtain a predicted repair matrix; input the CT original image, the MRI original image, the predicted registration matrix and the predicted repair matrix into the fusion structure to obtain a predicted fused image; calculate a first loss based on the predicted registration matrix, the CT sample image and the MRI sample image; calculate a second loss based on the predicted registration matrix, the CT original image and the MRI original image; calculate a third loss based on the predicted fused image and the fusion result; determine the overall loss of the deep convolution model based on the first loss, the second loss and the third loss; iterate the deep convolution model based on the overall loss until the loss converges.
[0132] In one embodiment, the image registration module 102 is further configured to:
[0133] The CT medical image is input into a first segmentation unit to obtain a CT label image; the MRI medical image is input into a first segmentation unit to obtain an MRI label image; the CT label image and the MRI label image are input into a registration unit to obtain a registration matrix of the CT label image and the MRI label image, and the registration matrix is a registration matrix of the CT medical image and the MRI medical image.
[0134] In one embodiment, the image registration module 102 is further configured to:
[0135] The CT label image and the MRI label image are spliced to obtain a spliced label image; the spliced label image is subjected to multiple rounds of convolution processing, residual processing, convolution processing and down-sampling processing to obtain multiple down-sampling feature maps; the down-sampling feature map with the smallest size is subjected to multiple rounds of up-sampling processing, splicing processing, residual processing and convolution processing to obtain multiple up-sampling feature maps; the up-sampling feature map with the largest size is subjected to convolution processing to obtain the registration matrix.
[0136] In one embodiment, the image registration module 102 is further configured to:
[0137] After performing continuous convolution processing on the input feature map, the input feature map is added to obtain a first convolution feature map; after performing continuous convolution processing on the first convolution feature map, the first convolution feature map is added to obtain a second convolution feature map; after performing continuous convolution processing on the second convolution feature map, the second convolution feature map and the input feature map are added to obtain a third convolution feature map; after performing pooling processing, upsampling processing and Relu processing on the input feature map, the convolution coefficient is obtained; after multiplying the third convolution feature map by the convolution coefficient, the third convolution feature map is added to the input feature map to obtain an output feature map.
[0138] In one implementation, the image restoration module 103 is further configured to:
[0139] The CT label image is subjected to multiple convolution processing to obtain a first multi-convolution feature map; the MRI label image is subjected to multiple convolution processing to obtain a second multi-convolution feature map; the first multi-convolution feature map is input into a first edge detection module to obtain a first edge feature map; the second multi-convolution feature map is input into a second edge detection module to obtain a second edge feature map; the first edge feature map and the second edge feature map are added together, and multiple convolution processing is performed to obtain a repair matrix of the CT label image and the MRI label image.
[0140] In one implementation, the image fusion module 104 is further configured to:
[0141] The MRI medical image and the registration matrix are multiplied to obtain a registration feature map; the registration feature map and the repair matrix are multiplied to obtain a repair feature map; the repair feature map and the CT medical image are added to obtain a fused medical image.
[0142] The medical image fusion device based on hole repair provided in the above-mentioned embodiment of the present application and the medical image fusion method based on hole repair provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.
[0143] The above describes the internal functions and structure of a medical image fusion device based on void repair. Fig. 9 As shown, in practice, the medical image fusion device based on hole repair can be realized as an electronic device, including: a memory 301 and a processor 303.
[0144] The memory 301 may be configured to store programs.
[0145] In addition, the memory 301 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.
[0146] The memory 301 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0147] The processor 303 is coupled to the memory 301 and is configured to execute a program in the memory 301 to:
[0148] Acquire a CT medical image and an MRI medical image, wherein the CT medical image and the MRI medical image are medical images of the same part of the same object;
[0149] Inputting the CT medical image and the MRI medical image into a registration structure to obtain a registration matrix of the CT medical image and the MRI medical image, a CT label image, and an MRI label image;
[0150] Inputting the CT label image and the MRI label image into an edge repair structure to obtain a repair matrix of the CT label image and the MRI label image;
[0151] The CT medical image, the MRI medical image, the registration matrix and the repair matrix are input into the fusion structure to obtain the repaired fused medical image.
[0152] In one implementation, the processor 303 is further configured to:
[0153] Acquire a CT training image and an MRI training image; the CT training image includes a CT original image and a CT sample image, and the MRI training image includes an MRI original image and an MRI sample image, and is annotated with a fusion result; input the CT original image and the MRI original image into the registration structure to obtain a predicted registration matrix; input the CT sample image and the MRI sample image into the edge repair structure to obtain a predicted repair matrix; input the CT original image, the MRI original image, the predicted registration matrix and the predicted repair matrix into the fusion structure to obtain a predicted fused image; calculate a first loss based on the predicted registration matrix, the CT sample image and the MRI sample image; calculate a second loss based on the predicted registration matrix, the CT original image and the MRI original image; calculate a third loss based on the predicted fused image and the fusion result; determine the overall loss of the deep convolution model based on the first loss, the second loss and the third loss; iterate the deep convolution model based on the overall loss until the loss converges.
[0154] In one implementation, the processor 303 is further configured to:
[0155] The CT medical image is input into a first segmentation unit to obtain a CT label image; the MRI medical image is input into a first segmentation unit to obtain an MRI label image; the CT label image and the MRI label image are input into a registration unit to obtain a registration matrix of the CT label image and the MRI label image, and the registration matrix is a registration matrix of the CT medical image and the MRI medical image.
[0156] In one implementation, the processor 303 is further configured to:
[0157] The CT label image and the MRI label image are spliced to obtain a spliced label image; the spliced label image is subjected to multiple rounds of convolution processing, residual processing, convolution processing and down-sampling processing to obtain multiple down-sampling feature maps; the down-sampling feature map with the smallest size is subjected to multiple rounds of up-sampling processing, splicing processing, residual processing and convolution processing to obtain multiple up-sampling feature maps; the up-sampling feature map with the largest size is subjected to convolution processing to obtain the registration matrix.
[0158] In one implementation, the processor 303 is further configured to:
[0159] After performing continuous convolution processing on the input feature map, the input feature map is added to obtain a first convolution feature map; after performing continuous convolution processing on the first convolution feature map, the first convolution feature map is added to obtain a second convolution feature map; after performing continuous convolution processing on the second convolution feature map, the second convolution feature map and the input feature map are added to obtain a third convolution feature map; after performing pooling processing, upsampling processing and Relu processing on the input feature map, the convolution coefficient is obtained; after multiplying the third convolution feature map by the convolution coefficient, the third convolution feature map is added to the input feature map to obtain an output feature map.
[0160] In one implementation, the processor 303 is further configured to:
[0161] The CT label image is subjected to multiple convolution processing to obtain a first multi-convolution feature map; the MRI label image is subjected to multiple convolution processing to obtain a second multi-convolution feature map; the first multi-convolution feature map is input into a first edge detection module to obtain a first edge feature map; the second multi-convolution feature map is input into a second edge detection module to obtain a second edge feature map; the first edge feature map and the second edge feature map are added together, and multiple convolution processing is performed to obtain a repair matrix of the CT label image and the MRI label image.
[0162] In one implementation, the processor 303 is further configured to:
[0163] The MRI medical image and the registration matrix are multiplied to obtain a registration feature map; the registration feature map and the repair matrix are multiplied to obtain a repair feature map; the repair feature map and the CT medical image are added to obtain a fused medical image.
[0164] In this application, Fig. 9 Only some components are shown schematically, which does not mean that the electronic device only includes Fig. 9 Components shown.
[0165] The electronic device provided in this embodiment is based on the same inventive concept as the medical image fusion method based on hole repair provided in the embodiment of the present application, and has the same beneficial effects as the method adopted, run or implemented by the application stored in it.
[0166] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0167] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0168] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0170] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0171] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0172] The present application also provides a computer-readable storage medium corresponding to a medical image fusion method based on hole repair provided in the aforementioned embodiment, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute a medical image fusion method based on hole repair provided in any of the aforementioned embodiments.
[0173] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0174] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the medical image fusion method based on hole repair provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.
[0175] It should be noted that in the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.
[0176] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0177] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A medical image fusion method based on void repair, characterized in that: include: Acquire a CT medical image and an MRI medical image, wherein the CT medical image and the MRI medical image are medical images of the same part of the same object; Inputting the CT medical image and the MRI medical image into a registration structure to obtain a registration matrix of the CT medical image and the MRI medical image, a CT label image, and an MRI label image; Inputting the CT label image and the MRI label image into an edge repair structure to obtain a repair matrix of the CT label image and the MRI label image; The CT medical image, the MRI medical image, the registration matrix and the repair matrix are input into the fusion structure to obtain the repaired fused medical image.
2. The medical image fusion method based on hole repair according to claim 1, characterized in that: Before acquiring the CT medical image and the MRI medical image, the method further includes: Acquire a CT training image and an MRI training image; the CT training image includes a CT original image and a CT sample image, and the MRI training image includes an MRI original image and an MRI sample image, and is annotated with a fusion result; Inputting the CT original image and the MRI original image into the registration structure to obtain a predicted registration matrix; Inputting the CT sample image and the MRI sample image into an edge repair structure to obtain a predicted repair matrix; Input the CT original image, the MRI original image, the predicted registration matrix and the predicted repair matrix into the fusion structure to obtain the predicted fusion image; Calculate a first loss based on the predicted registration matrix, the CT sample image, and the MRI sample image; Calculate the second loss based on the predicted registration matrix, the CT original image and the MRI original image; Calculating a third loss according to the predicted fused image and the fusion result; Determining an overall loss of the deep convolutional model based on the first loss, the second loss, and the third loss; The deep convolutional model is iterated based on the overall loss until the loss converges.
3. The medical image fusion method based on hole repair according to claim 1, characterized in that: The step of inputting the CT medical image and the MRI medical image into a registration structure to obtain a registration matrix of the CT medical image and the MRI medical image, a CT label image, and an MRI label image includes: Inputting the CT medical image into a first segmentation unit to obtain a CT label image; Inputting the MRI medical image into a first segmentation unit to obtain an MRI label image; The CT label image and the MRI label image are input into a registration unit to obtain a registration matrix of the CT label image and the MRI label image, where the registration matrix is a registration matrix of the CT medical image and the MRI medical image.
4. The medical image fusion method based on hole repair according to claim 3 is characterized in that: The step of inputting the CT label image and the MRI label image into a registration unit to obtain a registration matrix of the CT label image and the MRI label image includes: Splicing the CT label image and the MRI label image to obtain a spliced label image; Perform multiple rounds of convolution processing, residual processing, convolution processing and downsampling processing on the spliced label image to obtain multiple downsampled feature maps; Perform multiple rounds of upsampling, concatenation, residual processing, and convolution processing on the downsampled feature map with the smallest size to obtain multiple upsampled feature maps; The up-sampled feature map with the largest size is convolved to obtain the registration matrix.
5. The medical image fusion method based on hole repair according to claim 4 is characterized in that: The residual processing process includes: After continuous convolution processing on the input feature map, it is added to the input feature map to obtain the first convolution feature map; After continuously convolution processing is performed on the first convolution feature map, the first convolution feature map is added to the first convolution feature map to obtain a second convolution feature map; After continuously convolving the second convolution feature map, the third convolution feature map is obtained by adding the second convolution feature map and the input feature map. After pooling, upsampling and ReLU processing of the input feature map, the convolution coefficient is obtained; After multiplying the third convolution feature map by the convolution coefficient, it is added to the input feature map to obtain the output feature map.
6. The medical image fusion method based on hole repair according to any one of claims 1 to 5, characterized in that: The step of inputting the CT label image and the MRI label image into an edge repair structure to obtain a repair matrix of the CT label image and the MRI label image includes: Perform multiple convolution processing on the CT label image to obtain a first multi-convolution feature map; Perform multiple convolutions on the MRI label image to obtain a second multi-convolution feature map; Inputting the first multi-convolution feature map into a first edge detection module to obtain a first edge feature map; Inputting the second multi-convolution feature map into a second edge detection module to obtain a second edge feature map; After the first edge feature map and the second edge feature map are added, multiple convolution processes are performed to obtain restoration matrices of the CT label image and the MRI label image.
7. The medical image fusion method based on hole repair according to any one of claims 1 to 5, characterized in that: The step of inputting the CT medical image, the MRI medical image, the registration matrix and the repair matrix into the fusion structure to obtain the repaired fused medical image includes: Multiplying the MRI medical image and the registration matrix to obtain a registration feature map; Multiply the registration feature map and the repair matrix to obtain the repair feature map; The restoration feature map and the CT medical image are added to obtain a fused medical image.
8. A medical image fusion device based on cavity repair, characterized in that: include: An image acquisition module, which is used to acquire a CT medical image and an MRI medical image, wherein the CT medical image and the MRI medical image are medical images of the same part of the same object; An image registration module, which is used to input the CT medical image and the MRI medical image into a registration structure to obtain a registration matrix of the CT medical image and the MRI medical image, a CT label image, and an MRI label image; An image restoration module, which is used to input the CT label image and the MRI label image into an edge restoration structure to obtain restoration matrices of the CT label image and the MRI label image; The image fusion module is used to input the CT medical image, the MRI medical image, the registration matrix and the repair matrix into the fusion structure to obtain the repaired fused medical image.
9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program to: Acquire a CT medical image and an MRI medical image, wherein the CT medical image and the MRI medical image are medical images of the same part of the same object; Inputting the CT medical image and the MRI medical image into a registration structure to obtain a registration matrix of the CT medical image and the MRI medical image, a CT label image, and an MRI label image; Inputting the CT label image and the MRI label image into an edge repair structure to obtain a repair matrix of the CT label image and the MRI label image; The CT medical image, the MRI medical image, the registration matrix and the repair matrix are input into the fusion structure to obtain the repaired fused medical image.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement a medical image fusion method based on hole repair as described in any one of claims 1-7.