Image registration fusion method, device, system, computer device and storage medium

Abstract

The application relates to the field of medical image processing, in particular to an image registration and fusion method, device and system, computer equipment and a storage medium, the method comprising the following steps: acquiring first modality image data and second modality image of a target object; reconstructing and correcting the motion of the first modality image data to obtain a first modality image; performing modality conversion processing on the first modality image to obtain a new first modality image; wherein the new first modality image is a second modality image; performing registration on the new first modality image and the second modality image to obtain registration information; and fusing the first modality image and the second modality image based on the registration information. Since the new first modality image is a second modality image, the registration efficiency and registration accuracy are higher, thereby improving the efficiency and accuracy of image fusion.

Description

Technical Field

[0001] This application relates to the field of medical image processing, and in particular to an image registration and fusion method, apparatus, system, computer equipment, and storage medium. Background Technology

[0002] During PET-CT imaging, CT images are used to attenuate and correct PET image data. Furthermore, the body movements of the target subject between CT and PET scans (such as respiratory movements) can cause discrepancies between the PET and CT images. Therefore, PET and CT images need to be fused together for later display to aid in diagnosis. Similarly, PET-MR imaging and PET-attenuation-corrected image imaging also require registration and fusion.

[0003] Taking PET-CT imaging as an example, in existing technologies, PET images and CT images are usually directly registered and fused. Since PET images and CT images are generated from two different modalities, the registration efficiency and registration accuracy are low. Summary of the Invention

[0004] Therefore, it is necessary to provide an image registration and fusion method, apparatus, system, computer equipment, and storage medium to address the aforementioned technical problems.

[0005] In a first aspect, embodiments of the present invention propose an image registration and fusion method, the method comprising:

[0006] Acquire the first modality image data and the second modality image of the target object;

[0007] The first modal image data is reconstructed and motion corrected to obtain the first modal image;

[0008] The first modal image is subjected to modal transformation to obtain a new first modal image; wherein, the new first modal image is a second modal image.

[0009] The new first modal image is registered with the second modal image to obtain registration information;

[0010] Based on the registration information, the first modal image and the second modal image are fused.

[0011] In one embodiment, the step of reconstructing and motion-correcting the first modal image data to obtain the first modal image includes:

[0012] Based on the first modality image data, a reconstructed image is obtained;

[0013] Based on the reconstructed image, a first motion vector field is determined;

[0014] Based on the reconstructed image and the first motion vector field, the first modal image is obtained using a motion correction algorithm.

[0015] In one embodiment, the step of reconstructing and motion-correcting the first modal image data to obtain the first modal image includes:

[0016] Based on the first modality image data, a reconstructed image is obtained;

[0017] Based on the reconstructed image, a first motion vector field is determined;

[0018] Based on the first modal image data and the first motion vector field, the first modal image is obtained using a motion-compensated image reconstruction algorithm.

[0019] In one embodiment, obtaining a new first modal image by modal transformation of the first modal image includes:

[0020] The first modal image is input into the modality transformation model, and the new first modal image is output.

[0021] The modality conversion model is trained using a first modality image set as input and a second modality image set as output.

[0022] In one embodiment, registering the new first modality image with the second modality image to obtain registration information includes:

[0023] The new first modal image is registered with the second modal image using a registration method to obtain a second motion vector field.

[0024] In one embodiment, the first modal image is a PET image, and the second modal image is a CT image or an MR image.

[0025] Secondly, embodiments of the present invention provide an image registration and fusion apparatus, the apparatus comprising:

[0026] The acquisition module is used to acquire the first modality image data and the second modality image of the target object;

[0027] The reconstruction and correction module is used to reconstruct and perform motion correction on the first modal image data to obtain the first modal image;

[0028] A modality conversion module is used to obtain a new first modality image by modality conversion processing of the first modality image; wherein the new first modality image is a second modality-like image;

[0029] The registration module is used to register the new first modal image with the second modal image to obtain registration information;

[0030] The fusion module is used to fuse the first modal image and the second modal image based on the registration information.

[0031] Thirdly, embodiments of the present invention provide an image scanning system, characterized in that the system includes a scanning device for scanning a target object to obtain first modal image data and second modal image data, and an image registration and fusion device as described in the second aspect connected to the scanning device, wherein the second modal image data is used to generate a second modal image.

[0032] Fourthly, embodiments of the present invention provide a computer device including a memory and a processor, wherein the memory stores a computer program and the processor executes the steps described in the first aspect.

[0033] Fifthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the processor executes the computer program to implement the steps described in the first aspect.

[0034] Compared to existing technologies, the above-mentioned methods, apparatus, systems, computer equipment, and storage media improve the accuracy of subsequent registration and fusion of the first modality image and the second modality image by reconstructing and motion correcting the first modality image data. The first modality image is processed through modality transformation to obtain a new first modality image. The new first modality image is then registered with the second modality image to obtain registration information. Based on the registration information, the first modality image and the second modality image are fused. Since the new first modality image is similar to the second modality image, the registration efficiency and accuracy of the new first modality image and the second modality image are higher, thereby improving the efficiency and accuracy of image fusion between the first modality image and the second modality image. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the structure of an image scanning system in one embodiment;

[0036] Figure 2 This is a flowchart illustrating an image registration and fusion method in one embodiment;

[0037] Figure 3 This is a flowchart illustrating the reconstruction and motion correction method in one embodiment;

[0038] Figure 4 This is a flowchart illustrating the reconstruction and motion correction method in another embodiment;

[0039] Figure 5This is a schematic diagram of the module connections of an image registration and fusion device in an example embodiment;

[0040] Figure 6 This is a schematic diagram of the structure of a computer device in one embodiment. Detailed Implementation

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of the present invention. For those skilled in the art, the present invention can be applied to other similar scenarios based on these drawings without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0042] As indicated in this invention and the claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0043] While this invention makes various references to certain modules in systems according to embodiments of the invention, any number of different modules can be used and run on computing devices and / or processors. Modules are merely illustrative, and different aspects of the system and method may use different modules.

[0044] It should be understood that when a unit or module is described as "connected" or "coupled" to other units, modules, or blocks, it may refer to a direct connection or coupling, or communication with other units, modules, or blocks, or the presence of intermediate units, modules, or blocks, unless the context explicitly indicates otherwise. The term "and / or" as used herein may include any and all combinations of one or more of the related listed items.

[0045] like Figure 1 As shown, the image scanning system may include a scanning device 110, an image registration and fusion device 120, a storage device 140, and a display device 150. The devices in the image scanning system can be interconnected or communicate with each other via a network 130.

[0046] The scanning device 110 can scan an object. The object can be a physical object, human body, organ, tissue, etc. The scanning device can be a medical imaging device. In some embodiments, the scanning device 110 can be a magnetic resonance imaging (MRI) scanner, a computed tomography (CT) scanner, a positron emission tomography (PET) scanner, a b-scan ultrasonography (B-scan ultrasonography) scanner, a thermal texture map (TTM) scanner, a medical electronic endoscope (MEE), etc. In some embodiments, the scanning device 110 can also be a combination of multiple devices, such as a PET-CT scanner, a PET-MRI scanner, etc. The scanning device 110 can generate image data corresponding to the object after scanning. Further, the scanning device 110 can send the acquired image data to an image registration and fusion device 120, a storage device 140, or a display device 150 via a network 130.

[0047] The image registration and fusion apparatus 120 can process image data. The image data can be obtained by scanning with the scanning device 110 or from the storage device 140. In some embodiments, the image data can be two-dimensional or three-dimensional image data representing anatomical and / or functional information of the scanned object. The processing can include reconstructing the image data to generate an image. The reconstruction method can include, but is not limited to, one or more combinations of filtered backprojection (FBP), iterative reconstruction, deep learning-based reconstruction, multiplanar reformatting (MPR), volume rendering (VR), multiplanar volume reformation (MPVR), curved planar reformatting (CPR), maximum intensity projection (MIP), and shaded surface display (SSD). The processing can also include registration and fusion processing of the image data or the generated image. The image registration and fusion apparatus 120 can send the registered and fused image to the storage device 140 for storage.

[0048] Network 130 can be any connection method that connects two or more devices. For example, network 130 can be a wired network or a wireless network. In some embodiments, network 130 can be a single network or a combination of multiple networks. For example, network 130 may include one or more of the following: local area network (LAN), wide area network (WAN), public network, private network, wireless LAN, virtual network, public telephone network, etc. The modules in the image scanning system can interact with each other by connecting to network 130.

[0049] Storage device 140 can store data and / or information. For example, storage device 140 can store image data generated by scanning device 110, images obtained by image registration and fusion device 120, and user input or instructions received by display device 150. In some embodiments, storage device 140 can be local storage, external storage, cloud storage, etc.

[0050] Display device 150 can be used to display images. Display device 150 may include a display screen, touch screen, etc. In some embodiments, display device 150 may include an interactive interface that can receive input from a user or doctor. In some embodiments, display device 150 may include an input device, such as a touchpad, touch screen, mouse, keyboard, microphone, etc. Display device 150 can send user input to image registration and fusion device 120 for processing or to storage device 140 for storage.

[0051] Figure 2 This is a flowchart of an image registration and fusion process according to an embodiment of the present invention. In some embodiments, the process can be implemented by an image registration and fusion device 120. In one embodiment, such as Figure 2 As shown, an image registration and fusion method is proposed, including the following steps:

[0052] S201: Obtain the first modal image data and the second modal image of the target object.

[0053] It is understandable that the target object refers to a certain part of the object being scanned, such as a certain part of the human body, and the first modality image data and the second modality image refer to the image data obtained by scanning the same part.

[0054] In step 201, the first modal image data can be obtained through the scanning device 110 or from the storage device 140. The second modal image can be obtained by obtaining the second modal image data through the scanning device 110 and then processing it to obtain the second modal image, or it can be directly obtained from the storage device 140. In some embodiments, the first modal image data is PET image data, and the second modal image is a CT image or an MR image.

[0055] S202: Reconstruct and motion correct the first modality image data to obtain the first modality image.

[0056] Considering that head movement, limb / trunk movement, and breathing movements may cause mismatch between the first modality image and the second modality image during the acquisition of the first modality image data, step S202 is used to perform motion correction on the first modality image data to improve the accuracy of subsequent registration and fusion of the first modality image and the second modality image.

[0057] S203: The first modal image is processed by modal transformation to obtain a new first modal image; wherein the new first modal image is a second modal image.

[0058] The new first modality image is a CT image, an MR image, or an attenuation-corrected image.

[0059] Taking the first modal image as a PET image and the second modal image as a CT image as an example, the PET image is processed by modal conversion to obtain the CT image.

[0060] Considering the low registration efficiency and accuracy of images of different modalities, such as the registration of PET images with CT images, or the registration of PET images with MR images, in this embodiment, the new first modal image obtained after modal conversion processing of the first modal image is a second modal image (for example, both the new first modal image and the second modal image are CT images). Therefore, the registration efficiency and accuracy of the new first modal image and the second modal image are relatively higher.

[0061] Specifically, images of different modalities refer to medical images that can provide information from multiple levels due to different imaging mechanisms.

[0062] S204: Register the new first modality image with the second modality image to obtain registration information.

[0063] The new first modal image and the second modal image are registered using existing registration methods. The registration information includes offset information between the new first modal image and the second modal image. In some embodiments, the offset information can be represented by a motion vector field.

[0064] S205: Based on the registration information, fuse the first modality image and the second modality image.

[0065] Based on the registration information obtained in step S04, the first modality image and the second modality image can be fused by the image fusion algorithm through deformation operation.

[0066] Based on the above steps S201-S205, the first modality image data is reconstructed and motion corrected to improve the accuracy of subsequent registration and fusion of the first modality image and the second modality image. The first modality image is processed by modality transformation to obtain a new first modality image. The new first modality image is registered with the second modality image to obtain registration information. Based on the registration information, the first modality image and the second modality image are fused. Since the new first modality image is similar to the second modality image, the registration efficiency and registration accuracy are higher, thereby improving the efficiency and accuracy of image fusion.

[0067] In some embodiments, such as Figure 3 As shown, the process of reconstructing and motion-correcting the first modality image data to obtain the first modality image includes the following steps:

[0068] S301: Based on the first modality image data, obtain the reconstructed image;

[0069] S302: Determine the first motion vector field based on the reconstructed image;

[0070] S303: Based on the reconstructed image and the first motion vector field, the first modal image is obtained using a motion correction algorithm.

[0071] First, based on multiple frames of first modality image data, multiple reconstructed images are obtained by using image reconstruction algorithms or deep learning algorithms. Then, the first motion vector field is determined by using the multiple reconstructed images through image registration methods. Finally, the motion correction algorithm is used to correct the motion of the reconstructed images after multiple frames are superimposed to obtain the first modality image.

[0072] Motion correction methods are based on image motion deformation or by incorporating motion vector fields into the image reconstruction process to achieve motion correction.

[0073] For different movements, the specific methods for motion detection and motion vector field calculation are as follows: For head movements, an adaptive motion detection method and a rigid body registration method based on the similarity of the sum of squared errors are used to obtain the motion vector field for head movements. For limb / trunk movements, an adaptive motion detection method and a non-rigid body registration method are used to obtain the motion vector field for limb / trunk movements. For breathing movements, an adaptive motion detection method or an external device-based motion detection method and a non-rigid body registration method are used to obtain the motion vector field for breathing movements.

[0074] In other embodiments, motion-compensated image reconstruction algorithms can be used to simultaneously perform motion correction and reconstruction on the first modality image data to obtain the first modality image, such as the MCIR (motion compensated image reconstruction) algorithm.

[0075] Specifically, such as Figure 4 As shown, it includes the following steps:

[0076] S401: Based on the first modality image data, obtain the reconstructed image;

[0077] S402: Determine the first motion vector field based on the reconstructed image;

[0078] S403: Based on the first modal image data and the first motion vector field, the first modal image is obtained using a motion compensation image reconstruction algorithm.

[0079] Based on the above steps S401-S403, motion correction and reconstruction of the first modality image data are performed simultaneously to obtain the first modality image.

[0080] In some embodiments, obtaining a new first modal image by modal transformation of the first modal image includes the following steps:

[0081] The first modal image is input into the modality transformation model, and a new first modal image is output.

[0082] Specifically, the modality conversion model is trained by using the first modality image set as the input of the model and the second modality image set as the output of the model.

[0083] In one example embodiment, using PET images as the first modality and CT images as the second modality, 300 sets of clinical PET / CT images were selected, with no obvious motion-induced mismatch between the PET and CT images. Then, using PET images without attenuation correction as model input and CT images as model output, a U-NET network with residual modules was trained to obtain a regression network model for generating CT images from PET images.

[0084] The training process for modality conversion models is basically the same for image combinations of other modalities, so it will not be described in detail here.

[0085] In some embodiments, registering the new first modality image with the second modality image to obtain registration information includes:

[0086] The new first modal image is registered with the second modal image using a registration method to obtain the second motion vector field.

[0087] It should be noted that the method for calculating the motion vector field using the registration method has been described in the above embodiments and will not be repeated here.

[0088] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0089] In some embodiments, such as Figure 5 As shown, the present invention provides an image registration and fusion apparatus, the apparatus comprising:

[0090] The acquisition module 501 is used to acquire the first modality image data and the second modality image of the target object;

[0091] The reconstruction and correction module 502 is used to reconstruct and perform motion correction on the first modal image data to obtain the first modal image;

[0092] The modality conversion module 503 is used to obtain a new first modality image by modality conversion processing of the first modality image; wherein the new first modality image is a second modality-like image;

[0093] The registration module 504 is used to register the new first modal image with the second modal image to obtain registration information;

[0094] The fusion module 505 is used to fuse the first modal image and the second modal image based on the registration information.

[0095] In some embodiments, the reconstruction correction module includes:

[0096] The reconstruction module is used to obtain a reconstructed image based on the first modality image data;

[0097] A determining module is used to determine a first motion vector field based on the reconstructed image;

[0098] The reconstruction correction submodule is used to obtain the first modal image based on the reconstructed image and the first motion vector field using a motion correction algorithm.

[0099] In some embodiments, the reconstruction correction module includes:

[0100] The reconstruction module is used to obtain a reconstructed image based on the first modality image data;

[0101] A determining module is used to determine a first motion vector field based on the reconstructed image;

[0102] The reconstruction correction submodule is used to obtain the first modal image based on the reconstructed image and the first motion vector field using a motion correction algorithm.

[0103] In some embodiments, the modality conversion submodule is specifically used to input the first modality image into the modality conversion model and output the new first modality image.

[0104] In some embodiments, the training method of the modality conversion model includes: using a first modality image set as the input of the model and a second modality image set as the output of the model to train the modality conversion model.

[0105] In some embodiments, the registration module is specifically used to: register the new first modal image with the second modal image using a registration method to obtain a second motion vector field.

[0106] In some embodiments, the first modal image is a PET image, and the second modal image is a CT image or an MR image.

[0107] Specific limitations regarding the image registration and fusion device can be found in the limitations of the image registration and fusion method described above, and will not be repeated here. Each module in the aforementioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0108] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores motion detection data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements the steps in any of the above-described image registration and fusion method embodiments.

[0109] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0110] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in any of the above-described image registration and fusion method embodiments.

[0111] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in any of the above-described image registration and fusion method embodiments.

[0112] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0113] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0114] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image registration and fusion method, characterized in that, The method includes: Acquire the first modality image data and the second modality image of the target object; The first modal image data is reconstructed and motion corrected to obtain the first modal image; The first modal image is subjected to modal transformation to obtain a new first modal image; wherein, the new first modal image is a second modal image. The new first modal image is registered with the second modal image to obtain registration information; Based on the registration information, the first modal image and the second modal image are fused together; The step of registering the new first modality image with the second modality image to obtain registration information includes: The new first modal image is registered with the second modal image using a registration method to obtain a second motion vector field.

2. The method according to claim 1, characterized in that, The process of reconstructing and motion-correcting the first modal image data to obtain the first modal image includes: Based on the first modality image data, a reconstructed image is obtained; Based on the reconstructed image, a first motion vector field is determined; Based on the reconstructed image and the first motion vector field, the first modal image is obtained using a motion correction algorithm.

3. The method according to claim 1, characterized in that, The process of reconstructing and motion-correcting the first modal image data to obtain the first modal image includes: Based on the first modality image data, a reconstructed image is obtained; Based on the reconstructed image, a first motion vector field is determined; Based on the first modal image data and the first motion vector field, the first modal image is obtained using a motion-compensated image reconstruction algorithm.

4. The method according to claim 1, characterized in that, The step of obtaining a new first modal image by modal conversion of the first modal image includes: The first modal image is input into the modality transformation model, and the new first modal image is output. The modality conversion model is trained using a first modality image set as input and a second modality image set as output.

5. The method according to claim 1, characterized in that, The first modal image is a PET image, and the second modal image is a CT image or an MR image.

6. An image registration and fusion device, characterized in that, The device includes: The acquisition module is used to acquire the first modality image data and the second modality image of the target object; The reconstruction and correction module is used to reconstruct and perform motion correction on the first modal image data to obtain the first modal image; A modality conversion module is used to obtain a new first modality image by modality conversion processing of the first modality image; wherein the new first modality image is a second modality-like image; The registration module is used to register the new first modal image with the second modal image to obtain registration information; specifically, the registration module is used to register the new first modal image with the second modal image using a registration method to obtain a second motion vector field. The fusion module is used to fuse the first modal image and the second modal image based on the registration information.

7. An image scanning system, characterized in that, The system includes a scanning device for scanning a target object to obtain first modal image data and second modal image data, and an image registration and fusion device as described in claim 6 connected to the scanning device, wherein the second modal image data is used to generate a second modal image.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images