Image registration method, image fusion method, device and electronic equipment

Abstract

The present invention discloses an image registration method, an image fusion method, a device and an electronic device. The image registration method comprises: based on a reference image and a floating image, obtaining relative deformation data between the reference image and the floating image; decomposing the relative deformation data to obtain first deformation data and second deformation data; registering the reference image using the first deformation data to obtain a registered reference image; registering the floating image using the second deformation data to obtain a registered floating image. The present invention improves the accuracy and effect of image registration by registering the reference image and the floating image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image registration method, an image fusion method, a device and an electronic device. Background Art

[0002] In some scenarios, it is necessary to collect images of related objects. Although the images are collected for the same part of the related objects, there will be certain differences in the collected images under different collection conditions. For example, the same part of the related object will have certain differences in different images. Therefore, it is necessary to register the images collected under different conditions through image registration technology to align the matching points between different images.

[0003] Image registration technology has a wide range of applications in the field of medical imaging. More medical information can be obtained through registered images, thereby improving the level of medical care. For example, due to the limitation of the scanning length of CT equipment in related technologies, the images obtained at different time sequences can sometimes only reflect part of the human body, not the complete human body, and the influence of the movement of the human body parts will inevitably lead to misalignment between images at different time sequences. Image registration technology can be used to register and fuse images at different time sequences to obtain more accurate and complete images.

[0004] The image registration technology of the related art has the problems of low accuracy, poor effect and lack of intelligence when performing image registration. Summary of the invention

[0005] The embodiments of this specification are intended to solve at least one of the technical problems in the related art to a certain extent. To this end, the first purpose of the embodiments of this specification is to propose an image registration method, image fusion method, device, equipment, storage medium and program product.

[0006] An embodiment of the present specification provides an image registration method, the method comprising:

[0007] Based on a reference image and a floating image, obtaining relative deformation data between the reference image and the floating image;

[0008] Decomposing the relative deformation data to obtain first deformation data and second deformation data;

[0009] registering the reference image using the first deformation data to obtain a registered reference image; and

[0010] The floating image is registered using the second deformation data to obtain a registered floating image.

[0011] The present specification provides an image fusion method, the method comprising:

[0012] Execute the steps of the above-mentioned image registration method to obtain a registered reference image and a registered floating image;

[0013] The registered reference image and the registered floating image are fused to obtain a fused image.

[0014] An embodiment of the present specification provides an image registration device, the device comprising:

[0015] An acquisition module, used for acquiring relative deformation data between the reference image and the floating image based on the reference image and the floating image;

[0016] A decomposition module, used for decomposing the relative deformation data to obtain first deformation data and second deformation data;

[0017] A first registration module, configured to register the reference image using the first deformation data to obtain a registered reference image; and

[0018] The second registration module is used to register the floating image using the second deformation data to obtain a registered floating image.

[0019] The present specification provides an image fusion device, the device comprising:

[0020] The module of the above-mentioned image registration device is used to obtain a registered reference image and a registered floating image; and

[0021] The fusion module is used to fuse the registered reference image and the registered floating image to obtain a fused image.

[0022] An embodiment of the present specification provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in any one of the above embodiments are implemented.

[0023] The embodiments of this specification provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0024] An embodiment of the present specification provides a computer program product, wherein the computer program product includes instructions, and when the instructions are executed by a processor of a computer device, the computer device is enabled to perform the steps of the method described in any one of the above embodiments.

[0025] In the above-mentioned embodiment of the specification, based on the reference image and the floating image, relative deformation data between the reference image and the floating image is obtained; the relative deformation data is decomposed to obtain the first deformation data and the second deformation data; the reference image is registered using the first deformation data to obtain the registered reference image; the floating image is registered using the second deformation data to obtain the registered floating image. The reference image and the floating image are registered by the above-mentioned image registration method, thereby improving the accuracy and effect of image registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1-Figure 3 A schematic diagram of image registration and image fusion provided in an embodiment of this specification.

[0027] Figure 4 A flowchart of an image registration method provided in an embodiment of this specification.

[0028] Figure 5 A schematic diagram of image acquisition provided in an embodiment of this specification.

[0029] Figure 6 A flowchart of the image fusion method provided in the embodiment of this specification.

[0030] Figure 7 A schematic diagram of an image registration device provided in an embodiment of this specification.

[0031] Figure 8 A schematic diagram of an image fusion device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0032] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0033] Image registration technology has a wide range of applications in the field of medical imaging. Through the registered images, we can obtain various information about patients, facilitate quantitative comparative analysis based on images, and improve the level of medical diagnosis and treatment.

[0034] Most registration techniques are used to register images of the same part with different modalities or different time sequences. Different modalities, for example, represent image data obtained from different signal sources. For example, images obtained from different scanning devices, energy spectrum images, and 4D perfusion scans need to be registered.

[0035] During scanning, the human body inevitably has various movements, such as breathing and heart movements, which can easily cause image inconsistencies, so image registration is required. CT tomography is a commonly used scanning mode of CT equipment. It has the advantages of low radiation dose through multiple step-by-step scanning. However, since it is not a continuous scan, the adjacent images between the two scanning sequences are discontinuous, especially for prospective cardiac tomography. In addition, affected by human breathing and the movement of the heart itself, the image is prone to staggered layer artifacts, which are mainly manifested in staggered layers in cardiac VR images and the disconnection of the coronary artery between two scans in cardiac coronary CTA scanning, which makes it difficult to extract a complete coronary curved planar reconstruction (CPR) image. Among them, CTA is also called angiography technology, which is the abbreviation of vascular imaging technology using CT.

[0036] In addition, most CT devices have a limited detector width coverage, and it is difficult to cover the entire heart in one scan, so multiple scans are inevitably required. In view of this situation, it is necessary to perform registration and fusion between different scanned images to obtain a continuous and complete heart image.

[0037] Image registration requires a reference image and a floating image (the floating image is also called the image to be registered). The reference image and the floating image need to have a certain overlapping area (for example, the reference image and the floating image include the same part) before registration can be performed. Tomography can achieve the purpose of increasing the overlapping area by shortening the bed step for each scan.

[0038] In some image registration techniques, the floating image completely overlaps with the reference image, and after the floating image is registered, it is made as consistent as possible with the reference image in morphological structure. In another image registration technique, the floating image partially overlaps with the reference image, and the floating image and the reference image are spliced ​​together through registration, which is generally a rigid registration. Complete overlap, for example, refers to an image obtained by scanning the target object based on the same scanning position of the floating image and the reference image, and the target object contained in the floating image is consistent with the target object contained in the reference image. Partial overlap, for example, refers to an image obtained by scanning the target object based on different scanning positions of the floating image and the reference image, and a part of the target object contained in the floating image is consistent with a part of the target object contained in the reference image, and the other part of the target object contained in the floating image is inconsistent with the other part of the target object contained in the reference image. The consistent part (overlapping area) and the inconsistent part (non-overlapping area) of the floating image and the reference image are generally arranged in the scanning direction, and the scanning direction can be the extension direction of the scanning cavity axis (Z axis described below) or the feeding direction of the scanning bed. The floating image and the reference image can both be three-dimensional images, or both are image sequences, and the image sequence includes multiple slices or tomographic images arranged in the scanning direction. This article mainly takes the partial overlap of the floating image and the reference image as an example to illustrate.

[0039] In the embodiment of the present specification, the image at the overlapping position of the reference image and the floating image is a fused image of the half-scan image and the full-scan image. Of course, the image at the non-overlapping position of the reference image and the floating image can also be a fused image of the half-scan image and the full-scan image. In other words, the entire reference image and the entire floating image can both be fused images of the half-scan image and the full-scan image.

[0040] The reference image and the floating image can be obtained in the following manner: obtain the first scan data and the second scan data of the region of interest, the first scan data includes the full scan data, the second scan data includes the half scan data, the reconstruction field of view of the first scan data is larger than the reconstruction field of view of the second scan data, and the scanning time of the first scan data is larger than the scanning time of the second scan data. Reconstruct the first part of the target reconstructed image according to the first scan data, reconstruct the second part of the target reconstructed image according to the second scan data, the first part includes the difference part of the reconstruction field of view of the first scan data and the reconstruction field of view of the second scan data, and the second part includes the reconstruction field of view part of the second scan data, and then fuse the first part and the second part to obtain the target reconstructed image. The target reconstructed image is a reference image or a floating image.

[0041] The reference image and floating image are obtained by the above-mentioned medical image reconstruction method, and the advantage of the full scan reconstructed image with a large coverage is fully utilized. Without increasing the number of scans, the expanded full scan partial image is used for registration processing, and the radiation dose of the patient is not increased. It can be understood that in addition to obtaining the reference image and floating image by the above-mentioned medical image reconstruction method, the reference image and floating image can also be obtained by other methods. This specification does not specifically limit the method for obtaining the reference image and floating image. The reference image and floating image can be obtained by other methods according to the actual use situation, and the reference image and floating image include not only medical images, but also images that need to be registered in other fields.

[0042] In other embodiments, both the floating image and the reference image are acquired using full scan data.

[0043] At least one of the floating image and the reference image is completely obtained based on the full-scan partial image and the half-scan partial image, or at least one of the overlapping area of ​​the floating image with the reference image and the overlapping area of ​​the reference image with the floating image is obtained based on the full-scan partial image and the half-scan partial image.

[0044] Figure 1-Figure 3 A schematic diagram of image registration and image fusion provided in an embodiment of this specification.

[0045] like Figure 1 As shown, the dotted shaded portion 101 represents the viewing angle of the image scan using the half-scan method, which can cover part of the target object. The oblique shaded portion 102 indicates that the image scan using the full-scan method can cover a wider viewing angle than the image scan using the half-scan method. By combining the half-scan method and the full-scan method, image reconstruction can be performed to obtain an expanded full-scan image. It can be seen that, combining the advantages of the large reconstruction range of the full-scan image and the high temporal resolution of the half-scan image, the reconstructed image includes the roof parts on both sides (the oblique shaded portion 102), which combines the advantages of the full-scan image and the half-scan image. In an example, the reference image and the floating image of the embodiments of this specification are, for example, both reconstructed full-scan images.

[0046] like Figure 2 As shown, the Z axis represents the depth direction of the image scan, and in some cases the Z axis may also represent the direction in which the CT scanning device steps during scanning. Images scanned at different depths include, for example, overlapping portions 201. The embodiment of this specification aims to register images with overlapping portions 201 obtained by scanning at different depths.

[0047] like Figure 3As shown, after the images are registered, the registered images are fused, for example, images with overlapping parts 301 are fused.

[0048] In the embodiment of the present description, the floating image and the reference image overlap at least partially. When registering the images, it is necessary to ensure the continuity of the non-overlapping part and the overlapping registration part, and to consider the inconsistency of the change of the human body's motion state. Therefore, it is necessary to consider, on the basis of the traditional non-rigid registration, and without shortening the bed stepping, changing the number of scans, and increasing the radiation dose of the patient, to achieve the registration of the overlapping part of the image and the smooth transition between the overlapping part and the non-overlapping part, so that the image between the two scans is fused into a whole, avoiding the occurrence of staggered layer artifacts, and improving the continuity of the image between multiple scans.

[0049] In view of this, the present specification provides an image registration method, for details, please refer to Figure 4 .

[0050] Figure 4 A flowchart of an image registration method provided in an embodiment of this specification.

[0051] like Figure 4 As shown, the image registration method 400 provided in the embodiment of this specification includes, for example, steps S410-S440.

[0052] S410 : Based on the reference image and the floating image, obtain relative deformation data between the reference image and the floating image.

[0053] Exemplarily, the reference image and the floating image are both images of the target object, including but not limited to the heart, lungs, etc. The reference image can be used as a benchmark image, and the floating image can be used as an image to be registered. The similarity between the reference image and the floating image is a first similarity, for example, the reference image and the floating image overlap at least in part, that is, the reference image and the floating image can reflect the same part of the target object. The reference image and the floating image are registered and calculated to obtain relative deformation data, which characterizes the difference between the reference image and the floating image.

[0054] S420: Decompose the relative deformation data to obtain first deformation data and second deformation data.

[0055] For example, the relative deformation data is decomposed into two parts, one part is the first deformation data, and the other part is the second deformation data. The first deformation data is, for example, deformation data for the reference image, and the second deformation data is, for example, deformation data for the floating image. In the embodiments of this specification, in order to improve the registration effect, it is necessary not only to register the floating image but also to register the reference image.

[0056] S430 , registering the reference image using the first deformation data to obtain a registered reference image.

[0057] Exemplarily, the registered reference image obtained by registering the reference image using the first deformation data is closer to the floating image, that is, the similarity between the registered reference image and the floating image is generally greater than the similarity between the reference image and the floating image.

[0058] S440: Register the floating image using the second deformation data to obtain a registered floating image.

[0059] Exemplarily, the registered floating image obtained by registering the floating image using the second deformation data is closer to the reference image, that is, the similarity between the registered floating image and the reference image is generally greater than the similarity between the floating image and the reference image.

[0060] It can be seen that the reference image is aligned toward the floating image based on the first deformation data, and the floating image is aligned toward the reference image based on the second deformation data, and the second similarity between the aligned reference image and the aligned floating image is greater than the first similarity.

[0061] It can be understood that the example of this specification decomposes the relative deformation data between the reference image and the floating image to obtain the first deformation data and the second deformation data, and aligns the reference image in a direction more similar to the floating image based on the first deformation data, and aligns the floating image in a direction more similar to the reference image based on the second deformation data, so that the similarity between the registered reference image and the registered floating image is higher. It can be seen that the embodiment of this specification improves the accuracy and effect of image registration by aligning the reference image and the floating image to each other at the same time.

[0062] In some embodiments, the relative deformation data may be decomposed in the following manner.

[0063] For example, the relative deformation data is decomposed by the first decomposition weight to obtain the first deformation data, and the relative deformation data is decomposed by the second decomposition weight to obtain the second deformation data, wherein the deformation directions indicated by the first deformation data and the second deformation data are opposite. Specifically, the first decomposition weight and the second decomposition weight can be obtained based on the reference decomposition weight. The sum of the absolute value of the first decomposition weight and the absolute value of the second decomposition weight is, for example, equal to the first preset value, and the embodiment of this specification is described with the first preset value being 1. It can be understood that the first preset threshold can be set according to actual needs. In one example, when the relative deformation data (deformation field) is the deformation data of the floating image relative to the reference image, the first decomposition weight is negative and the second decomposition weight is positive. In another example, when the relative deformation data (deformation field) is the deformation data of the reference image relative to the floating image, the first decomposition weight is positive and the second decomposition weight is negative. After obtaining the first decomposition weight and the second decomposition weight, the relative deformation data is processed by the first decomposition weight to obtain the first deformation data, and the relative deformation data is processed by the second decomposition weight to obtain the second deformation data.

[0064] This article takes the case where the relative deformation data (deformation field) is the deformation data of the floating image relative to the reference image, the first decomposition weight is a negative number, and the second decomposition weight is a positive number as an example for explanation.

[0065] For example, the first decomposition weight is used to negatively weight the relative deformation data to obtain the first deformation data. Negative weighting indicates that the reference image is deformed toward the floating image based on the first deformation data.

[0066] For example, the relative deformation data is forward weighted using the second decomposition weight to obtain the second deformation data. The forward weighting indicates that the floating image is deformed toward the reference image based on the second deformation data.

[0067] It can be understood that the embodiments of the present specification utilize the first decomposition weight to negatively weight the relative deformation data, utilize the second decomposition weight to positively weight the relative deformation data, and based on the first deformation data, align the reference image in a direction more similar to the floating image, and based on the second deformation data, align the floating image in a direction more similar to the reference image, so that the similarity between the registered reference image and the registered floating image is higher, thereby improving the accuracy and effect of image registration.

[0068] The embodiments of this specification do not specifically limit the number of reference images and floating images. The reference images may include one or more images, and the floating images may also include one or more images. In order to explain the method of the embodiments of this specification in more detail, the following content is described by taking the reference images including multiple images and the floating images including multiple images as examples.

[0069] Figure 5Schematic diagram of image acquisition provided by the embodiments of this specification.

[0070] As Figure 5 shown, the reference image includes an overlapping area 511 that overlaps with the floating image, a first non-overlapping area 512 that does not overlap with the floating image, and the floating image includes an overlapping area 521 that overlaps with the reference image and a second non-overlapping area 522 that does not overlap with the reference image. For example, the reference image and the floating image are acquired by performing image acquisition on the target object along the target direction. Exemplarily, the target direction is, for example, the depth direction, and the target direction is described by taking the Z-axis as an example, and each acquired image is, for example, in the plane where the X-axis and the Y-axis are located.

[0071] The first non-overlapping area 512 is, for example, for the first part of the target object, the overlapping area 511 where the reference image overlaps with the floating image and the overlapping area 521 where the floating image overlaps with the reference image are, for example, for the second part of the target object, and the second non-overlapping area 522 is, for example, for the third part of the target object. In other words, both the overlapping area 511 and the overlapping area 521 are images for the same second part, indicating that there is an overlap between the reference image and the floating image. The image registration and image fusion of the reference image and the floating image described in the embodiments of this specification, for example, include registering and performing image fusion on the overlapping area 511 and the overlapping area 521.

[0072] For example, the overlapping area 511 of the reference image includes N reference sub-images, and the overlapping area 521 of the floating image includes N floating sub-images, where N is an integer greater than or equal to 1, that is, N represents the number of overlapping sub-images in the reference image and the floating image. The k-th reference sub-image is denoted as Img1 i,j,k , and the k-th floating sub-image is denoted as Img2 i,j,k , where 0 ≤ k < N. 0 ≤ i < Width, 0 ≤ j < Length, Width represents the width of the sub-image pixels, Length represents the length of the sub-image pixels, i and j are the coordinate indices of the pixels in the X and Y directions, and k is the index in the Z direction.

[0073] Among them, the first deformation data is configured to act on the overlapping area 511 of the reference image and the floating image, and the absolute value of the weight value of the first decomposition weight close to the first non-overlapping area 512 is less than the absolute value of the weight value far from the first non-overlapping area 512. The second deformation data is configured to act on the overlapping area 521 of the floating image and the reference image, and the absolute value of the weight value of the second decomposition weight close to the second non-overlapping area 522 is less than the absolute value of the weight value far from the second non-overlapping area 522.

[0074] Performing registration calculations on the N reference sub-images and the N floating sub-images to obtain the relative deformation data DVF i,j,k(X, Y, Z), relative deformation data can also be called deformation field. i,j,k (X, Y, Z) can indicate that each pixel in each floating sub-image corresponds to an offset (X, Y, Z), and the pixel can be aligned or consistent with the corresponding pixel in the reference image after being deformed based on the offset, and the offset corresponding to the pixel in the kth floating sub-image is not necessarily only related to the pixel of the kth reference sub-image, but may also be related to the pixels of some or all of the N reference sub-images.

[0075] In one example, the relative deformation data may be obtained by performing registration calculation using the Demons registration algorithm, which is a medical image registration algorithm and one of the grayscale-based image registration algorithms in medical image registration.

[0076] It is understood that the embodiments of this specification are not limited to using the Demons registration algorithm for image registration, and other image registration techniques can be used for image registration according to actual application conditions. After performing the registration calculation to obtain the relative deformation data, the relative deformation data can be smoothed to remove noise in the relative deformation data, and the smoothing method includes but is not limited to the Gaussian filtering method.

[0077] For example, the first decomposition weight and the second decomposition weight are obtained based on the reference decomposition weight, and both the first decomposition weight and the second decomposition weight are linearly related or sinusoidally related to the distance between the deformation position and the connection position, and the connection position is the position where the overlapping area and the non-overlapping area are connected, that is, the connection position is the connection position of the overlapping area 511 and the first non-overlapping area 512 ( Figure 5 ) or the connection position of the overlapping area 521 and the second non-overlapping area 522 ( Figure 5 The deformation position is, for example, any position from k=0 to k=N-1.

[0078] Exemplarily, the reference decomposition weight is expressed as RegistrationWeight. The reference decomposition weight RegistrationWeight is, for example, related to N. Different reference sub-images and floating sub-images have different reference decomposition weights RegistrationWeight.

[0079] In one example, the reference decomposition weight RegistrationWeight can be expressed as the following formula (1) or formula (2). It can be seen that the reference decomposition weight and the distance between the deformation position and the connection position are linearly related as shown in formula (1) or sinusoidally related as shown in formula (2).

[0080] RegistrationWeight[k]=(k+1) / (N+1),0≤k <N (1)

[0081]

[0082] Based on the reference decomposition weight RegistrationWeight, the first decomposition weight and the second decomposition weight can be obtained. The first decomposition weight is, for example, expressed as -RegistrationWeight, and the second decomposition weight is, for example, expressed as 1-RegistrationWeight. In the case where the reference decomposition weight is linearly or sinusoidally related to the distance between the deformation position and the connection position, the first decomposition weight and the second decomposition weight obtained based on the reference decomposition weight are also linearly or sinusoidally related to the distance between the deformation position and the connection position. In addition, based on formula (1) and formula (2), it can be seen that the absolute value of the first decomposition weight and the absolute value of the second decomposition weight are symmetrical about the center plane of the overlapping area.

[0083] The first decomposition weight includes N first decomposition sub-weights, and the N first decomposition sub-weights correspond to the N reference sub-images one by one. The kth first decomposition sub-weight among the N first decomposition sub-weights is, for example, represented as -RegistrationWeight[k].

[0084] The second decomposition weight includes N second decomposition sub-weights, and the N second decomposition sub-weights correspond to the N floating sub-images one by one. The kth second decomposition sub-weight among the N second decomposition sub-weights is, for example, represented by 1-RegistrationWeight[k].

[0085] Combining formula (1) or formula (2) and Figure 5 , the absolute value of the first decomposition weight is positively correlated with the distance between the deformed position and the first non-overlapping area 512. Specifically, the absolute value of the kth first decomposition sub-weight is positively correlated with the distance between the kth reference sub-image (i.e., the deformed position) and the first non-overlapping area 512 in the target direction Z, or it can also be expressed as the absolute value of the kth first decomposition sub-weight is negatively correlated with the distance between the kth reference sub-image (i.e., the deformed position) and the second non-overlapping area 522 in the target direction. Among them, the positive correlation means that the absolute value of the first decomposition sub-weight corresponding to the reference sub-image closer to the first non-overlapping area 512 in the overlapping area 511 of the reference image is smaller, and the negative correlation means that the absolute value of the first decomposition sub-weight corresponding to the reference sub-image closer to the second non-overlapping area 522 in the overlapping area 511 of the reference image is larger.

[0086] Similarly, the absolute value of the second decomposition weight is positively correlated with the distance between the deformation position and the second non-overlapping area 522. Specifically, the absolute value of the k-th second decomposition sub-weight is positively correlated with the distance between the k-th floating sub-image (i.e., the deformation position) and the second non-overlapping area 522 in the target direction, or it can also be represented that the absolute value of the k-th second decomposition sub-weight is negatively correlated with the distance between the k-th floating sub-image (i.e., the deformation position) and the first non-overlapping area 512 in the target direction. The positive correlation means that the absolute value of the second decomposition sub-weight corresponding to the floating sub-image closer to the second non-overlapping area 522 in the overlapping area 521 of the floating image is smaller, and the negative correlation means that the absolute value of the second decomposition sub-weight corresponding to the floating sub-image closer to the first non-overlapping area 512 in the overlapping area 521 of the floating image is larger.

[0087] In other words, the overlapping area 511 and the first non-overlapping area 512 of the reference image belong to the same sequence of images, and the overlapping area 521 and the second non-overlapping area 522 of the floating image belong to the same sequence of images. For each reference sub-image (or floating sub-image), the closer to the non-overlapping area of ​​the own side (the same time sequence), the smaller the absolute value of the corresponding first decomposition sub-weight (or the absolute value of the second decomposition sub-weight).

[0088] After obtaining N first decomposition sub-weights and N second decomposition sub-weights, the relative deformation data DVF is calculated using the N first decomposition sub-weights. i,j,k (X, Y, Z) is decomposed and calculated to obtain the first deformation data. The relative deformation data DVF is calculated using the N second decomposition sub-weights. i,j,k (X, Y, Z) is decomposed and calculated to obtain the second deformation data. The decomposition calculation includes, for example, a dot product calculation.

[0089] For example, the first deformation data is represented by -RegistrationWeight[k]DVF i,j,k (X, Y, Z). The second deformation data is represented by (1-RegistrationWeight[k])DVF i,j,k (X, Y, Z).

[0090] After obtaining the first deformation data and the second deformation data, the N reference sub-images are registered using the first deformation data to obtain a registered reference image RegImg1 i,j,k , as shown in formula (3). Similarly, the N floating sub-images are registered using the second deformation data to obtain the registered floating image RegImg2 i,j,k , as shown in formula (4).

[0091]

[0092]

[0093] in, It represents that a corresponding transformation is performed based on the first deformation data or the second deformation data, that is, an interpolation transformation is performed.

[0094] Exemplarily, the methods for acquiring relative deformation data include but are not limited to traditional non-rigid registration and artificial intelligence methods. The key to the embodiments of this specification lies in the decomposition of relative deformation data and its forward and reverse use.

[0095] It can be understood that for each reference sub-image (or floating sub-image), the closer to the non-overlapping area of ​​the self, the smaller the absolute value of the corresponding first decomposition sub-weight (or the absolute value of the second decomposition sub-weight). By setting the first decomposition sub-weight (or the second decomposition sub-weight) corresponding to the reference sub-image (or floating sub-image) close to the non-overlapping area to a smaller value, the first deformation data (or the second deformation data) obtained after decomposing the relative deformation data based on the first decomposition sub-weight (or the second decomposition sub-weight) is smaller, so that when the reference image and the floating image are respectively registered based on the first deformation data and the second deformation data, the transition between the first non-overlapping area and the overlapping area of ​​the reference image, or between the second non-overlapping area and the overlapping area of ​​the floating image is smoother and the connection is more natural, thereby improving the image continuity after image registration.

[0096] It can be seen that the embodiments of the present specification decompose the relative deformation data and weighted transition, and apply the relative deformation data to the reference image and the floating image respectively in reverse and forward, which is different from the related art that only aligns the floating image toward the reference image. The embodiments of the present specification align both the reference image and the floating image toward each other, and the deformation amplitude is symmetrical relative to the center of the reference image or the floating image. The closer to the non-overlapping area of ​​the other party, the greater the deformation amplitude, and the closer to the non-overlapping area of ​​the own party, the smaller the deformation amplitude, and the sum of the absolute values ​​of the first deformation data and the second deformation data corresponding to the overlapping sub-images at the same Z position is the absolute value of the relative deformation data before decomposition. The embodiments of the present specification respectively obtain the registered reference image and the registered floating image, which not only ensures that the two can complete the registration, but also maintains the natural connection between the two and their respective non-overlapping areas.

[0097] By rationally using tomography to obtain images of overlapping parts and performing registration processing without increasing the patient's radiation dose, the deformation field is weightedly decomposed and acted reversely on the reference image and forwardly on the floating image respectively, and the two images are fused together, thus solving the image staggered problem existing in different scanning sequences in tomography technology and reducing the adverse effects caused by human breathing and heart movement.

[0098] In one example, the floating image and the reference image may both be two-dimensional images, and the relative deformation data may be two-dimensional deformation data.

[0099] Figure 6 A flowchart of the image fusion method provided in the embodiment of this specification.

[0100] like Figure 6 As shown, the image registration method 600 provided in the embodiment of this specification includes, for example, steps S610-S650.

[0101] S610: Obtain relative deformation data between the reference image and the floating image based on the reference image and the floating image.

[0102] S620: Decompose the relative deformation data to obtain first deformation data and second deformation data.

[0103] S630: Use the first deformation data to register the reference image to obtain a registered reference image.

[0104] S640: Register the floating image using the second deformation data to obtain a registered floating image.

[0105] Exemplarily, steps S610-S640 are the same as or similar to steps S410-S440 described above, and are not described in detail herein.

[0106] S650: Fusing the registered reference image and the registered floating image to obtain a fused image.

[0107] Exemplarily, after obtaining the registered reference image and the registered floating image, the registered reference image and the registered floating image are fused to obtain a fused image, which integrates the information of the reference image and the floating image and removes the deviation information existing in the reference image or the floating image. The registered reference image and the registered floating image include overlapping areas, and when performing image fusion, the overlapping areas can be fused. The obtained fused image constitutes a complete and continuous overall time-series image, which is convenient for obtaining more information about the target object based on the overall time-series image.

[0108] In some embodiments, the registered reference image and the registered floating image may be fused in the following manner.

[0109] The registered reference image, for example, includes a registered overlapping area (corresponding to the overlapping area 511 mentioned above) and a first non-overlapping area, and the registered floating image, for example, includes a registered overlapping area (corresponding to the overlapping area 521 mentioned above) and a second non-overlapping area. This embodiment explains the fusion by taking the example that the registered overlapping area of ​​the registered reference image includes N registered reference sub-images, and the registered overlapping area of ​​the registered floating image includes N registered floating sub-images.

[0110] First, a reference fusion weight is obtained, and a first fusion weight and a second fusion weight are obtained based on the reference fusion weight.

[0111] Exemplarily, the reference image includes an overlapping area with the floating image and a first non-overlapping area that does not overlap with the floating image. The first fusion weight is configured to act on the overlapping area between the reference image and the floating image, and the weight of the first fusion weight close to the first non-overlapping area is greater than the weight far from the first non-overlapping area. The relationship between the weight of the first fusion weight and its distance from the first non-overlapping area is opposite to the first decomposition weight described above.

[0112] Exemplarily, the floating image includes an overlapping area with the reference image and a second non-overlapping area that does not overlap with the reference image. The second fusion weight is configured to act on the overlapping area between the floating image and the reference image, and the weight of the second fusion weight close to the second non-overlapping area is greater than the weight far from the first non-overlapping area. The relationship between the weight of the second fusion weight and the distance between it and the second non-overlapping area is opposite to the second decomposition weight described above.

[0113] The sum of the weight of the first fusion weight and the weight of the second fusion weight is equal to the second preset value, and the weight of the first fusion weight and the weight of the second fusion weight are symmetrical about the center plane of the overlapping area. The second preset value is 1, for example.

[0114] Exemplarily, the reference fusion weight is expressed as MergeWeight, for example, and the reference fusion weight MergeWeight is related to N. Different registered reference sub-images or registered floating sub-images have different corresponding reference fusion weights MergeWeight.

[0115] In one example, the reference fusion weight MergeWeight can be expressed as the following formula (5):

[0116] MergeWeight[k]=(k+1) / (N+1),0≤k <N (5)

[0117] Exemplarily, the first fusion weight is, for example, 1-MergeWeight, and the second fusion weight is, for example, MergeWeight.

[0118] Next, the overlapping area RegImg1 in the registered reference image is processed using the first fusion weight 1-MergeWeight to obtain the overlapping area RegImg1 in the processed reference image i,j,k (1-MergeWeight[k]). The overlapping region RegImg2 in the registered floating image is processed using the second fusion weight MergeWeight to obtain the overlapping region RegImg2 in the processed floating image. i,j,k MergeWeight[k] The processing here includes, for example, dot product calculation.

[0119] Then, the overlapping area in the processed reference image and the overlapping area in the processed floating image are fused to obtain a fused image FinalImg of the overlapping area. i,j,k , as shown in formula (6):

[0120] FinalImg i,j,k =RegImg1 i,j,k (1-MergeWeight[k])+RegImg2 i,j,k MergeWeight[k] (6)

[0121] Exemplarily, the overlapping area in the processed reference image includes N processed reference sub-images, the first fusion weight includes N first fusion sub-weights, and the N first fusion sub-weights correspond one-to-one to the N processed reference sub-images. The overlapping area in the processed floating image includes N processed floating sub-images, the second fusion weight includes N second fusion sub-weights, and the N second fusion sub-weights correspond one-to-one to the N processed floating sub-images.

[0122] In one example, the absolute value of the kth first fusion sub-weight is negatively correlated with the distance between the kth reference sub-image and the first non-overlapping image in the target direction, or it can also be expressed that the absolute value of the kth first fusion sub-weight is positively correlated with the distance between the kth reference sub-image and the second non-overlapping image in the target direction.

[0123] Similarly, the absolute value of the kth second fusion sub-weight is negatively correlated with the distance between the kth floating sub-image and the second non-overlapping image in the target direction, or it can also be expressed that the absolute value of the kth second fusion sub-weight is positively correlated with the distance between the kth floating sub-image and the first non-overlapping image in the target direction.

[0124] In addition, the sum of the absolute value of the kth first fusion sub-weight and the absolute value of the kth second fusion sub-weight is a second preset value, and the second preset value is, for example, 1, or the second preset value can be set according to actual application conditions.

[0125] It can be understood that for each reference sub-image (or floating sub-image), the closer to the non-overlapping image of the own party, the greater the absolute value of the corresponding first fusion sub-weight (or the absolute value of the second fusion sub-weight). By setting the first fusion sub-weight (or the second fusion sub-weight) corresponding to the reference sub-image (or floating sub-image) close to the non-overlapping image to a larger value, when image fusion is performed based on the first fusion sub-weight (or the second fusion sub-weight), more information of the sub-image closer to the non-overlapping image participates in the fusion process, making the transition between the first non-overlapping image and the reference image, or between the second non-overlapping image and the floating image smoother and the connection more natural, thereby improving the continuity of the fused image.

[0126] In another example, the absolute value of the kth first fusion sub-weight is negatively correlated with the kth first decomposition sub-weight, and the absolute value of the kth second fusion sub-weight is negatively correlated with the kth second decomposition sub-weight. In addition, the sum of the absolute value of the kth first fusion sub-weight and the absolute value of the kth second fusion sub-weight is a second preset value, and the second preset value is, for example, 1, or the second preset value can be set according to actual application conditions. The two values ​​referred to here as being negatively correlated means that the larger one value is, the smaller the other value is.

[0127] It can be understood that the absolute value of the kth first fusion sub-weight is negatively correlated with the kth first decomposition sub-weight, and the absolute value of the kth second fusion sub-weight is negatively correlated with the kth second decomposition sub-weight. This is because when the kth first decomposition sub-weight (or the kth second decomposition sub-weight) is large, the corresponding kth reference sub-image (or the kth floating sub-image) is deformed to a large extent during registration. At this time, the absolute value of the corresponding kth first fusion sub-weight (or the absolute value of the kth second fusion sub-weight) is reduced accordingly, so that the fusion weight of the sub-image with a large deformation degree during image fusion is small, and the error caused by excessive introduction of large deformation during image fusion is reduced, thereby improving the effect of image fusion, and obtaining a higher accuracy of the final fused image.

[0128] This embodiment combines the advantages of a large full scan coverage and a high half scan time resolution. Without increasing the patient's radiation dose, it increases the coverage of the first and last images and the image range of the overlapping part between the two scans, making reasonable and efficient use of the overlapping part of the tomographic scan. If the image is scanned by tomographic full scan, the range of the overlapping area can be adjusted by adjusting the bed stepping, etc., to obtain the overlapping image range required for image registration.

[0129] The reference image after reverse registration and the floating image after forward registration are weighted and fused together according to their Z positions, which not only completes the registration of the overlapping area and avoids the original staggered phenomenon, but also maintains a smooth transition with the respective non-overlapping images, so that the two scanned images are naturally fused into a whole, wherein the first image sequence obtained by the first scan includes the first non-overlapping area and the overlapping area of ​​the reference image, and the second image sequence obtained by the second scan includes the overlapping area of ​​the floating image and the second non-overlapping area. The embodiments of this specification are described with reference to the registration and fusion between two image sequences, but can be expanded to the image registration and image fusion between multiple image sequences according to actual conditions.

[0130] In one example, the floating image and the reference image may both be two-dimensional images, the relative deformation data may be two-dimensional deformation data, and the fused image of the processed reference image and the processed floating image may be a two-dimensional image.

[0131] Figure 7 A schematic diagram of an image registration device provided in an embodiment of this specification.

[0132] This specification embodiment provides an image registration device 700, see Figure 7 The image registration device 700 includes: an acquisition module 710, a decomposition module 720, a first registration module 730 and a second registration module 740.

[0133] Exemplarily, the obtaining module 710 is used to obtain relative deformation data between the reference image and the floating image based on the reference image and the floating image.

[0134] Exemplarily, the decomposition module 720 is used to decompose the relative deformation data to obtain the first deformation data and the second deformation data.

[0135] Exemplarily, the first registration module 730 is used to register the reference image using the first deformation data to obtain a registered reference image.

[0136] Exemplarily, the second registration module 740 is used to register the floating image using the second deformation data to obtain a registered floating image.

[0137] It can be understood that for the specific description of the image registration device 700, reference can be made to the above description of the image registration method, which will not be repeated here.

[0138] Figure 8 A schematic diagram of an image fusion device provided in an embodiment of this specification.

[0139] This specification provides an image fusion device 800. Figure 8The image fusion device 800 includes: an acquisition module 810, a decomposition module 820, a first registration module 830, a second registration module 840 and a fusion module 850.

[0140] Exemplarily, the obtaining module 810 is used to obtain relative deformation data between the reference image and the floating image based on the reference image and the floating image.

[0141] Exemplarily, the decomposition module 820 is used to decompose the relative deformation data to obtain the first deformation data and the second deformation data.

[0142] Exemplarily, the first registration module 830 is used to register the reference image using the first deformation data to obtain a registered reference image.

[0143] Exemplarily, the second registration module 840 is used to register the floating image using the second deformation data to obtain a registered floating image.

[0144] Exemplarily, the fusion module 850 is used to perform a fusion process on the registered reference image and the registered floating image to obtain a fused image.

[0145] It can be understood that for the specific description of the image fusion device 800, reference can be made to the description of the image fusion method above, which will not be repeated here.

[0146] An embodiment of the present specification provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method in any of the above embodiments are implemented.

[0147] An embodiment of the present specification provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method in any of the above embodiments are implemented.

[0148] One embodiment of the present specification provides a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device can perform the steps of the method of any of the above embodiments.

[0149] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0150] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0151] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0152] In addition, the terms "first", "second", etc. used in the embodiments of the present invention are only used for descriptive purposes and should not be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated in the present embodiment. Therefore, the features defined by the terms "first", "second", etc. in the embodiments of the present invention can explicitly or implicitly indicate that the embodiment includes at least one of the features. In the description of the present invention, the word "multiple" means at least two or two or more, such as two, three, four, etc., unless otherwise clearly and specifically defined in the embodiments.

[0153] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. An image registration method, characterized in that: The method comprises: Based on a reference image and a floating image, relative deformation data between the reference image and the floating image is obtained, wherein the reference image includes a first non-overlapping area that does not overlap with the floating image, and the first deformation data is configured to act on an overlapping area between the reference image and the floating image, and the overlapping area includes N reference sub-images; the floating image includes a second non-overlapping area that does not overlap with the reference image, and the second deformation data is configured to act on an overlapping area between the floating image and the reference image, and the overlapping area includes N floating sub-images; Decomposing the relative deformation data to obtain the first deformation data and the second deformation data includes: decomposing the relative deformation data using a first decomposition weight to obtain the first deformation data, and decomposing the relative deformation data using a second decomposition weight to obtain the second deformation data, wherein the first deformation data and the second deformation data indicate opposite deformation directions; registering the N reference sub-images in the reference image using the first deformation data to obtain a registered reference image; and registering the N floating sub-images in the floating image using the second deformation data to obtain a registered floating image; The first decomposition weight includes N first decomposition sub-weights, and the N first decomposition sub-weights correspond to the N reference sub-images one by one. The second decomposition weight includes N second decomposition sub-weights, and the N second decomposition sub-weights correspond to the N floating sub-images one by one. The absolute value of the weight of the first decomposition sub-weight close to the first non-overlapping area among the N first decomposition sub-weights is smaller than the absolute value of the weight of the first decomposition sub-weight far from the first non-overlapping area, and the absolute value of the weight of the second decomposition sub-weight close to the second non-overlapping area among the N second decomposition sub-weights is smaller than the absolute value of the weight of the second decomposition sub-weight far from the second non-overlapping area, The first decomposition weight and the second decomposition weight are obtained based on the benchmark decomposition weight, and both the first decomposition weight and the second decomposition weight are linearly or sinusoidally related to the distance between the deformation position and the connection position, and the connection position is the position where the overlapping area and the non-overlapping area are connected.

2. The method according to claim 1, characterized in that The sum of the absolute value of the first decomposition weight and the absolute value of the second decomposition weight is equal to a first preset value; the first decomposition weight is a negative number and the second decomposition weight is a positive number, or the first decomposition weight is a positive number and the second decomposition weight is a negative number.

3. The method according to claim 1, characterized in that The absolute value of the first decomposition weight is positively correlated with the distance between the deformation position and the first non-overlapping area, and the absolute value of the second decomposition weight is positively correlated with the distance between the deformation position and the second non-overlapping area.

4. The method according to claim 1, characterized in that The absolute value of the first decomposition weight and the absolute value of the second decomposition weight are symmetrical about a central plane of the overlapping area.

5. The method according to claim 1, characterized in that The image at the overlapping position of the reference image and the floating image is a fused image of the half-scan image and the full-scan image.

6. An image fusion method, characterized in that: The method comprises: Execute the steps of the method according to any one of claims 1 to 5 to obtain a registered reference image and a registered floating image; and The registered reference image and the registered floating image are fused to obtain a fused image.

7. The method according to claim 6, characterized in that The fusing the registered reference image and the registered floating image to obtain a fused image comprises: Obtain a first fusion weight and a second fusion weight; Processing the registered reference image using the first fusion weight to obtain a processed reference image; Processing the registered floating image using the second fusion weights to obtain a processed floating image; and fusing the processed reference image and the processed floating image to obtain the fused image, The sum of the weight of the first fusion weight and the weight of the second fusion weight is equal to a second preset value, and the weight of the first fusion weight and the weight of the second fusion weight are symmetrical about the central plane of the overlapping area.

8. The method according to claim 7, characterized in that: The reference image includes a first non-overlapping area that does not overlap with the floating image, the first fusion weight is configured to act on the overlapping area between the reference image and the floating image, and the weight of the first fusion weight close to the first non-overlapping area is greater than the weight far from the first non-overlapping area; and / or The floating image includes a second non-overlapping area that does not overlap with the reference image, and the second fusion weight is configured to act on the overlapping area between the floating image and the reference image, and the weight of the second fusion weight close to the second non-overlapping area is greater than the weight far away from the first non-overlapping area.

9. An image registration device, characterized in that: The device comprises: an acquisition module, configured to acquire relative deformation data between the reference image and the floating image based on a reference image and a floating image, wherein the reference image includes a first non-overlapping area that does not overlap with the floating image, and the first deformation data is configured to act on an overlapping area between the reference image and the floating image, and the overlapping area includes N reference sub-images; the floating image includes a second non-overlapping area that does not overlap with the reference image, and the second deformation data is configured to act on an overlapping area between the floating image and the reference image, and the overlapping area includes N floating sub-images; a decomposition module, configured to decompose the relative deformation data to obtain the first deformation data and the second deformation data, comprising: decomposing the relative deformation data using a first decomposition weight to obtain the first deformation data, and decomposing the relative deformation data using a second decomposition weight to obtain the second deformation data, wherein the first deformation data and the second deformation data indicate opposite deformation directions; A first registration module, configured to register the N reference sub-images in the reference image using the first deformation data to obtain a registered reference image; and A second registration module, configured to register the N floating sub-images in the floating image using the second deformation data to obtain a registered floating image; The first decomposition weight includes N first decomposition sub-weights, and the N first decomposition sub-weights correspond to the N reference sub-images one by one. The second decomposition weight includes N second decomposition sub-weights, and the N second decomposition sub-weights correspond to the N floating sub-images one by one. The absolute value of the weight of the first decomposition sub-weight close to the first non-overlapping area among the N first decomposition sub-weights is smaller than the absolute value of the weight of the first decomposition sub-weight far from the first non-overlapping area, and the absolute value of the weight of the second decomposition sub-weight close to the second non-overlapping area among the N second decomposition sub-weights is smaller than the absolute value of the weight of the second decomposition sub-weight far from the second non-overlapping area, The first decomposition weight and the second decomposition weight are obtained based on the benchmark decomposition weight, and both the first decomposition weight and the second decomposition weight are linearly or sinusoidally related to the distance between the deformation position and the connection position, and the connection position is the position where the overlapping area and the non-overlapping area are connected.

10. An image fusion device, characterized in that: The device comprises: The module of the image registration device of claim 9, used to obtain a registered reference image and a registered floating image; and The fusion module is used to fuse the registered reference image and the registered floating image to obtain a fused image.

11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images