Pet imaging method and apparatus based on optical flow registration, and device and storage medium

Abstract

The present application relates to a PET imaging method and apparatus based on optical flow registration, and a device and a storage medium. The method comprises: correcting the position of an MR image, and extracting a first feature map of the PET image and a second feature map of a corrected MR image; performing feature enhancement processing of a spatial dimension and a channel dimension on both the first feature map and the second feature map, and fusing feature enhancement processing results of the spatial dimension and the channel dimension to generate a first multi-modal enhanced feature map of the PET image and a second multi-modal enhanced feature map of the corrected MR image; and fusing the first multi-modal enhanced feature map and the second multi-modal enhanced feature map by means of a cross-modal feature fusion module and using a multi-head cross-attention mechanism, so as to generate a multi-modal fused feature map, and on the basis of the multi-modal fused feature map, generating a reconstructed PET image. By means of the present application, the features of PET and MR can be effectively fused to obtain a clearer reconstructed PET image.

Description

PET imaging method, device, equipment and storage medium based on optical flow registration Technical Field

[0001] The present application belongs to the field of medical imaging technology, and in particular relates to a PET imaging method, apparatus, device, and storage medium based on optical flow registration. Background Art

[0002] PET (Positron Emission Tomography) is an important medical imaging technology used to obtain functional and metabolic information inside organisms. It reveals the metabolic activities of tissues and organs by detecting the distribution and concentration of radioactive isotope-labeled biological molecules in the body. During PET imaging, the patient receives a labeled radioactive drug, usually a glucose molecule with a positron-emitting radioactive isotope (fluorine-18 labeled glucose, FDG). After these drugs are injected into the patient's body, they are metabolized and absorbed by the body's tissues and organs. When the positron decays, two high-energy positrons are released. These positrons meet and annihilate with electrons in the body. During the annihilation process, two gamma photons in opposite directions are produced, which can be captured by the detector.

[0003] The use of radioisotopes during PET scanning and the associated radiation dose are potential risks to patients and medical staff, and the issue of PET radiation dose is receiving increasing attention. However, as radiation dose and scan time decrease, more noise is introduced into the imaging process, resulting in poorer PET image quality.

[0004] In order to solve the above problems, Yuru He et al. published an article entitled "Dynamic PET Image Denoising With Deep Learning-Based Joint Filtering" in the IEEE Access journal in 2021. This study improved the network framework based on the structure of the spatial variant linear representation model (SVLRM). The study used a CNN with 12 convolutional layers to denoise PET noisy images. At the network input, the article fuses PET and MR (Magnetic Resonance) images, and combines the information of the two modalities by splicing the channel dimension. Subsequently, the fused input is processed by a network model with shared weights, and the final denoised PET image is output. However, due to the possible misalignment problem between the PET and MR modalities, this article did not effectively solve this problem in the multimodal fusion process; in addition, there is room for improvement in the information fusion process.

[0005] At the same time, existing PET imaging technology still has the following shortcomings:

[0006] 1. Long PET scanning time: During the PET scanning process, continuous image acquisition of the patient is required, which makes the entire scanning process relatively long, causing inconvenience to the patient and may increase discomfort during the scanning process.

[0007] 2. Slow image reconstruction speed: The amount of data collected by full sampling is large, and complex image reconstruction algorithms are required, resulting in slow image reconstruction speed and prolonging the diagnosis process.

[0008] 3. Motion artifacts: The long scanning process is easily affected by the patient's movement, resulting in image artifacts, thereby reducing image quality and affecting the doctor's accurate judgment of the patient's condition.

[0009] 4. Lack of multimodal data utilization and registration: Most algorithms perform denoising based on only a single PET modality, ignoring the possibility of using MR prior knowledge to guide PET image denoising, which limits the algorithm's ability to utilize multimodal information.

[0010] 5. Simplified multimodal fusion process: Existing algorithms for multimodal fusion typically simply concatenate multimodal data along the channel dimension, lacking in-depth exploration of the effective interaction of multimodal features. This simple concatenation approach may fail to fully exploit the correlation and complementarity between multimodal data, leading to unsatisfactory fusion results.

[0011] Summary of the Invention

[0012] The present application provides a PET imaging method, apparatus, device and storage medium based on optical flow registration, aiming to solve at least one of the above-mentioned technical problems in the prior art to a certain extent.

[0013] In order to solve the above problems, this application provides the following technical solutions:

[0014] A PET imaging method based on optical flow registration, comprising:

[0015] PET images and MR images of the irradiated area were acquired separately;

[0016] Obtaining optical flow information of the PET image and the MR image through an optical flow registration network, correcting the position of the MR image according to the optical flow information, and extracting a first feature map and a second feature map of the PET image and the corrected MR image, respectively;

[0017] performing feature enhancement processing in spatial dimensions and channel dimensions on the first feature map and the second feature map respectively by a spatial channel feature enhancement module, and fusing the feature enhancement processing results in spatial dimensions and channel dimensions to generate a first multimodal enhanced feature map of the PET image and a second multimodal enhanced feature map of the corrected MR image;

[0018] The first multimodal enhanced feature map and the second multimodal enhanced feature map are fused by a cross-modal feature fusion module using a multi-head cross-attention mechanism to generate a multimodal fusion feature map, and a PET reconstructed image is generated based on the multimodal fusion feature map.

[0019] The technical solution adopted in the embodiment of the present application further includes: before obtaining the optical flow information of the PET image and the MR image through the optical flow registration network, it also includes:

[0020] The Canny operator is used to extract contour information from the PET image and the MR image respectively.

[0021] The technical solution adopted in the embodiment of the present application also includes: the use of the Canny operator to extract contour information from the PET image and the MR image respectively is specifically as follows:

[0022] Morphological image processing technology is used to perform image corrosion and dilation processing on the PET image and the MR image, thereby eliminating small objects and thin lines in the PET image and the MR image, retaining the basic contour shape of the radiation part, and finally generating a binary mask composed of 0 and 1, which is used to represent the contour information of the PET image and the MR image respectively.

[0023] The technical solution adopted in the embodiment of the present application also includes: the optical flow registration network is a U-net-based model, including an encoder and a decoder. The optical flow information of the PET image and the MR image is obtained through the optical flow registration network. After correcting the position of the MR image according to the optical flow information, the first feature map and the second feature map of the PET image and the corrected MR image are respectively extracted. Specifically,

[0024] The contour information of the PET image and the MR image is input into the optical flow registration network. The encoder part of the optical flow registration network uses three 1x1 convolution layers to extract feature maps and uses a 3×3 convolution layer to downsample the extracted feature maps. The decoder part of the optical flow registration network uses upsampling and 1x1 convolution layers to expand and compress the extracted feature maps to restore the size of the optical flow map and generate a weight mask with 64 channels. The weight mask is generated by applying a Sigmoid activation function and outputs the first feature map F of the PET image and the corrected MR image. pet and the second feature map F mr .

[0025] The technical solution adopted in the embodiment of the present application further includes: performing feature enhancement processing in the spatial dimension and the channel dimension on the first feature map and the second feature map respectively by the spatial channel feature enhancement module, and fusing the feature enhancement processing results in the spatial dimension and the channel dimension to generate the first multimodal enhanced feature map of the PET image and the second multimodal enhanced feature map of the corrected MR image. Specifically,

[0026] The spatial channel feature enhancement module includes a spatial enhancement module and a channel enhancement module. The spatial enhancement module is implemented by two convolutional layers and the first feature map F pet and the second feature map F mr Compress and obtain two matrices S1 and S2, connect the matrices S1 and S2 in the spatial dimension to obtain a single-dimensional matrix S containing the attention information of the PET and MR image streams, and then perform the first feature map F pet and the second feature map F mr Perform Sigmoid function activation and element-by-element operation to obtain the first spatial enhancement feature map F of the PET image and the corrected MR image in the spatial dimension. s-pet and the second spatial enhanced feature map F s-mr ;

[0027] The channel enhancement module uses global average pooling to transform the first feature map F pet and the second feature map F mr Compression is performed and a channel descriptor is created. The encoding vectors C1 and C2 of the PET image and the MR image are calculated by the channel descriptor. The encoding vectors C1 and C2 are connected and a fused embedding vector is created through a fully connected layer. The embedding vector is then activated by a Sigmoid function and element-by-element operation is performed to obtain the first channel enhanced feature map F of the PET image and the corrected MR image, respectively. c-pet and the second channel enhanced feature map F c-mr ;

[0028] The first spatial enhancement feature map and the first channel enhancement feature map of the PET image and the second spatial enhancement feature map and the second channel enhancement feature map of the corrected MR image are fused to generate a first multimodal enhancement feature map F of the PET image. sc-pet and the second multimodal enhanced feature map F of the corrected MR image sc-mr .

[0029] The technical solution adopted in the embodiment of the present application also includes: after the cross-modal feature fusion module adopts a multi-head cross attention mechanism to fuse the first multimodal enhanced feature map and the second multimodal enhanced feature map, the multimodal fusion feature map is generated specifically as follows:

[0030] The cross-modal feature fusion module adopts a multi-head cross attention mechanism and a Transformer architecture to transform the first multimodal enhanced feature map F sc-pet and the second multimodal enhanced feature map F sc-mr Perform linear projection operations to generate the query matrix Q, key matrix K, and value matrix V for PET images and MR images, respectively, and perform attention operations between the vectorized features of the two modalities:

[0031] In the above formula, σ represents the sigmoid function, represents the square root of the feature dimension, K pet , V pet Represent the query matrix Q, key matrix K and value matrix V of PET images respectively, K mr , V mr They represent the query matrix Q, key matrix K and value matrix V of MR images respectively;

[0032] After compressing the attention operation results of the two modalities through the average pooling operation, the attention maps of the two modalities are spliced ​​in the channel dimension to generate a multimodal fusion feature map.

[0033] The technical solution adopted in the embodiment of the present application also includes: generating a PET reconstructed image according to the multimodal fusion feature map is specifically as follows:

[0034] The multimodal fusion feature map is input into a decoder through a skip connection, and the decoder outputs a PET reconstructed image.

[0035] Another technical solution adopted in the embodiment of the present application is: a PET imaging device based on optical flow registration, comprising:

[0036] Image acquisition module: used to acquire PET images and MR images of the radiation site respectively;

[0037] An optical flow registration module is configured to obtain optical flow information of the PET image and the MR image through an optical flow registration network, correct the position of the MR image according to the optical flow information, and extract a first feature map and a second feature map of the PET image and the corrected MR image, respectively;

[0038] A feature enhancement module is configured to perform feature enhancement processing in the spatial dimension and the channel dimension on the first feature map and the second feature map respectively through the spatial channel feature enhancement module, and fuse the feature enhancement processing results in the spatial dimension and the channel dimension to generate a first multimodal enhanced feature map of the PET image and a second multimodal enhanced feature map of the corrected MR image;

[0039] Feature fusion module: used to fuse the first multimodal enhanced feature map and the second multimodal enhanced feature map through a cross-modal feature fusion module using a multi-head cross-attention mechanism to generate a multimodal fusion feature map, and generate a PET reconstructed image based on the multimodal fusion feature map.

[0040] Another technical solution adopted by the embodiment of the present application is: a device, the device comprising a processor and a memory coupled to the processor, wherein:

[0041] The memory stores program instructions for implementing the PET imaging method based on optical flow registration;

[0042] The processor is configured to execute the program instructions stored in the memory to control a PET imaging method based on optical flow registration.

[0043] Another technical solution adopted in the embodiment of the present application is: a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute the PET imaging method based on optical flow registration.

[0044] Compared with the prior art, the beneficial effects produced by the embodiments of the present application are: the PET imaging method, apparatus, device and storage medium based on optical flow registration in the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] FIG1 is a flow chart of a PET imaging method based on optical flow registration according to an embodiment of the present application;

[0046] FIG2 is a schematic structural diagram of a PET imaging device based on optical flow registration according to an embodiment of the present application;

[0047] FIG3 is a schematic diagram of the device structure of an embodiment of the present application;

[0048] FIG4 is a schematic diagram of the structure of a storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0050] The terms "first," "second," and "third" in this application are used only for descriptive purposes and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of such features. In the description of this application, "multiple" means at least two, for example, two, three, etc., unless otherwise specifically defined. All directional indications in the embodiments of this application (such as up, down, left, right, front, back...) are only used to explain the relative positional relationship, movement, etc. between the components under a specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications also change accordingly. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products, or devices.

[0051] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0052] Please refer to Figure 1, which is a flow chart of the PET imaging method based on optical flow registration according to an embodiment of the present application. The PET imaging method based on optical flow registration according to an embodiment of the present application comprises the following steps:

[0053] S100: Acquire PET images and MR images of the irradiated area respectively;

[0054] S110: Use the Canny operator to extract contour information from the PET image and the MR image respectively;

[0055] In this step, the lower and upper thresholds for contour information extraction are set to 20 and 50, respectively. The contour information extraction process is as follows: First, morphological image processing techniques are used to perform image erosion and dilation on the PET and MR images to eliminate small objects and thin lines in the PET and MR images while retaining the basic contour shape of the radiation site. During the processing, a 5×5 circular kernel is used to ensure isotropic erosion and dilation to maintain the integrity of the contour shape and prevent unnecessary distortion. Finally, a binary mask consisting of 0 and 1 is generated to represent the contour information I of the PET and MR images, respectively. pet and I mr .

[0056] S120: Inputting the contour information of the PET image and the MR image into the optical flow registration network, the optical flow registration network calculates optical flow information based on the contour information of the PET image and the MR image, corrects the position of the MR image based on the optical flow information, and extracts a first feature map and a second feature map of the PET image and the corrected MR image, respectively;

[0057] In this step, the optical flow registration network is a U-net-based model that uses the PET image as a reference and uses inverse image transformation to correct the position of the MR image to achieve the effect of PET / MR dual-modality registration and alignment. Specifically, the optical flow registration network consists of two parts: an encoder and a decoder. The encoder part uses three 1x1 convolutional layers to extract feature maps and uses a 3×3 convolutional layer to downsample the extracted feature maps by 2 times, thereby gradually reducing the size of the extracted feature maps from 256×256 to 32×32. The decoder part uses upsampling and 1x1 convolutional layers to expand and compress the extracted feature maps to restore the size of the optical flow map and ultimately generate a weight mask with 64 channels. Finally, the weight mask is generated by applying the Sigmoid activation function and outputs the first feature map F of the PET image and the corrected MR image. pet and the second feature map F mr Specifically, the optical flow registration network parameter settings are shown in Table 1 below:

[0058] Table 1 PWC optical flow registration network parameters

[0059] The encoder parameters are shown in Table 2 below:

[0060] Table 2 Encoder parameter settings

[0061] S130: Inputting the first feature map and the second feature map into a spatial channel feature enhancement module, performing feature enhancement processing on the first feature map and the second feature map in the spatial dimension and the channel dimension respectively by the spatial channel feature enhancement module, thereby obtaining a first spatial enhancement feature map and a first channel enhancement feature map in the spatial dimension of the PET image, and a second spatial enhancement feature map and a second channel enhancement feature map in the spatial dimension of the corrected MR image, respectively;

[0062] In this step, the spatial channel feature enhancement module (SC-FEM) includes a spatial enhancement module and a channel enhancement module. The spatial enhancement module can improve the network's attention to important areas such as high-density noise. In the spatial enhancement module, the first feature map F of the input is processed through two convolutional layers K1 and K2. pet and the second feature map F mr Compression is performed to obtain two matrices S1 and S2. Matrices S1 and S2 are connected in the spatial dimension to obtain a single-dimensional matrix S containing the attention information of the PET and MR image streams. By activating the input feature map with a Sigmoid function and performing element-by-element operations, the first spatial enhancement feature map F of the PET image and the corrected MR image in the spatial dimension is finally obtained. s-pet and the second spatial enhanced feature map F s-mr The parameter settings of the spatial enhancement module are shown in Table 3 below:

[0063] Table 3 Parameter settings of spatial enhancement module

[0064] In the channel enhancement module, global average pooling is used to compress each H×W input feature map into a 1×1 value, and a channel descriptor is created. The encoding vectors C1 and C2 of the PET image and MR image are calculated through the channel descriptor. The encoding vectors C1 and C2 are then connected and a fused embedding vector is created through a fully connected layer. Finally, the first channel enhanced feature map F of the PET image and the corrected MR image is obtained by activating the embedding vector with a Sigmoid function and performing element-by-element operations. c-pet and the second channel enhanced feature map F c-mr Specifically, the channel enhancement module parameter settings are shown in Table 4 below:

[0065] Table 4 Channel enhancement module parameter settings

[0066] S140: fusing the first spatial enhancement feature map and the first channel enhancement feature map of the PET image and the second spatial enhancement feature map and the second channel enhancement feature map of the corrected MR image, respectively, to generate a first multimodal enhancement feature map of the PET image and a second multimodal enhancement feature map of the corrected MR image;

[0067] In this step, the first spatial enhancement feature map and the first channel enhancement feature map of the PET image and the second spatial enhancement feature map and the second channel enhancement feature map of the corrected MR image are fused to generate a first multimodal enhancement feature map F that combines the two modal information of PET and MR. sc-pet and the second multimodal enhanced feature map F sc-mr , thus comprehensively considering the two modal information, making the feature enhancement process more effective. Specifically, the first multimodal enhanced feature map F sc-pet and the second multimodal enhanced feature map F sc-mr It can be defined by the following formula: sc-pet =F pet +F s-pet +F c-pet (1) F sc-mr =F mr +F s-mr +F c-mr (2)

[0068] S150: Inputting the first multimodal enhanced feature map and the second multimodal enhanced feature map into a cross-modal feature fusion module, and the cross-modal feature fusion module fuses the first multimodal enhanced feature map and the second multimodal enhanced feature map using a multi-head cross attention mechanism to generate a multimodal fusion feature map;

[0069] In this step, the cross-modal feature fusion module adopts the multi-head cross attention mechanism (CM-FFM) and Transformer architecture to obtain the query matrix Q (queries), key matrix K (keys) and value matrix V (values) by calculating the dot product between PET and MR images, thereby estimating the correlation between the multimodal enhanced feature maps, and calculating the cross-attention weight based on the softmax function through scaling and normalization. Specifically, the first multimodal enhanced feature map F sc-pet and the second multimodal enhanced feature map F sc-mr First, a linear projection operation is performed to generate the query matrix Q, key matrix K, and value matrix V of the PET image and MR image. Then, an attention operation is performed between the vectorized features of the two modalities. The calculation formula is as follows:

[0070] In the above formula, σ represents the sigmoid function, which is used to map the input to the range of 0 to 1, ensuring that the attention weight is within a reasonable range to represent the correlation between the two modalities and prevent the weight from deviating too much from the normal range. Represents the square root of the feature dimension and is used to scale the attention weight to ensure that the attention score is not too large when calculating the dot product, thereby maintaining the stability of the model. K pet , V pet Represent the query matrix Q, key matrix K and value matrix V of PET images respectively, K pet , V pet Both are enhanced by the first multimodal feature map F sc-pet The projection is obtained, which represents the information extracted from the PET image. K mr , V mr Represent the query matrix Q, key matrix K and value matrix V of MR images respectively, K mr , V mr Both are enhanced by the second multimodal feature map F sc-mr The projection is obtained, which represents the information extracted from the MR image.

[0071] After compressing the attention results of the two modalities through the average pooling operation, the attention maps of the two modalities are spliced ​​in the channel dimension to generate a multimodal fusion feature map, which provides rich and beneficial image details for PET image denoising. Specifically, the parameters of the multi-head cross attention mechanism are set as shown in Table 5 below:

[0072] Table 5 Multi-head cross attention mechanism parameter settings

[0073] S160: Input the multimodal fusion feature map into the decoder through the skip connection, and output the denoised PET reconstructed image through the decoder;

[0074] In this step, in the decoder part, considering that transposed convolution often produces checkerboard artifacts during the upsampling process and involves a large number of trainable parameters, the embodiment of the present application selects a bilinear upsampling method with a scaling factor of 2, and uses a convolution layer at each stage, which helps to restore the multi-level feature map to the original image resolution and generate a relatively smooth feature map. In order to enhance the generalization ability of the model and improve the expression ability of the activation function, the embodiment of the present application introduces batch normalization in each convolution layer during the downsampling and upsampling of the image, and introduces jump connections at the encoding and decoding ends to retain the most favorable image details from the two modalities. It can be understood that in other embodiments of the present application, CT images can also be used as a substitute for MR images to denoise PET images. Specifically, the decoder parameter settings are shown in Table 6 below:

[0075] Table 6 Decoder parameter settings

[0076] Based on the above, the PET imaging method based on optical flow registration in the embodiment of the present application utilizes the contour information of the PET image and the MR image, and aligns the PET image and the MR image through the optical flow correction network, so that the multimodal data can be better aligned, effectively reducing the image distortion and artifacts caused by the registration problem, and improving the clarity and accuracy of the image; by introducing the spatial channel feature enhancement module, the module uses matrices and vectors to represent spatial attention and channel attention, and calculates not only based on the information of the current modality, but also in combination with the influence of other modalities to ensure the effectiveness of feature extraction at each downsampling stage, which can better capture important features in the image and improve the accuracy and robustness of feature extraction. By adopting a multi-head cross-attention mechanism to capture the diversity of feature information across modalities, it can make fuller use of the correlation and complementarity between multimodal data, and can effectively fuse the features of PET and MR to obtain clearer PET reconstructed images.

[0077] Please refer to FIG2 , which is a schematic diagram of the structure of a PET imaging device based on optical flow registration according to an embodiment of the present application. The PET imaging device 40 based on optical flow registration according to an embodiment of the present application comprises:

[0078] Image acquisition module 41: used to respectively acquire PET images and MR images of the radiation site;

[0079] An optical flow registration module 42 is configured to obtain optical flow information of the PET image and the MR image through an optical flow registration network, correct the position of the MR image according to the optical flow information, and extract a first feature map and a second feature map of the PET image and the corrected MR image, respectively.

[0080] The feature enhancement module 43 is configured to perform feature enhancement processing in the spatial dimension and the channel dimension on the first feature map and the second feature map respectively through the spatial channel feature enhancement module, and fuse the feature enhancement processing results in the spatial dimension and the channel dimension to generate a first multimodal enhanced feature map of the PET image and a second multimodal enhanced feature map of the corrected MR image;

[0081] Feature fusion module 44: used to fuse the first multimodal enhanced feature map and the second multimodal enhanced feature map through a cross-modal feature fusion module using a multi-head cross-attention mechanism to generate a multimodal fusion feature map, and generate a PET reconstructed image based on the multimodal fusion feature map.

[0082] Please refer to Figure 3, which is a schematic diagram of the device structure of an embodiment of the present application. The device 50 includes:

[0083] A memory 51 storing executable program instructions;

[0084] a processor 52 connected to the memory 51;

[0085] The processor 52 is used to call the executable program instructions stored in the memory 51 and perform the following steps: respectively collect PET images and MR images of the radiation site; obtain optical flow information of the PET image and the MR image through the optical flow registration network, correct the position of the MR image according to the optical flow information, and respectively extract the first feature map and the second feature map of the PET image and the corrected MR image; respectively perform feature enhancement processing of the spatial dimension and the channel dimension on the first feature map and the second feature map through the spatial channel feature enhancement module, and fuse the feature enhancement processing results of the spatial dimension and the channel dimension to generate a first multimodal enhanced feature map of the PET image and a second multimodal enhanced feature map of the corrected MR image; after fusing the first multimodal enhanced feature map and the second multimodal enhanced feature map through the cross-modal feature fusion module using a multi-head cross-attention mechanism, a multimodal fusion feature map is generated, and a PET reconstructed image is generated based on the multimodal fusion feature map.

[0086] The processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip having signal processing capabilities. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or any conventional processor.

[0087] Please refer to FIG4 , which is a schematic diagram of the structure of the storage medium of an embodiment of the present application. The storage medium of the embodiment of the present application stores program instructions 61 capable of implementing the following steps: respectively acquiring a PET image and an MR image of the radiation site; obtaining optical flow information of the PET image and the MR image through an optical flow registration network, correcting the position of the MR image according to the optical flow information, and respectively extracting a first feature map and a second feature map of the PET image and the corrected MR image; performing feature enhancement processing on the first feature map and the second feature map in the spatial dimension and the channel dimension respectively through a spatial channel feature enhancement module, and fusing the feature enhancement processing results in the spatial dimension and the channel dimension to generate a first multimodal enhanced feature map of the PET image and a second multimodal enhanced feature map of the corrected MR image; fusing the first multimodal enhanced feature map and the second multimodal enhanced feature map through a cross-modal feature fusion module using a multi-head cross-attention mechanism to generate a multimodal fusion feature map, and generating a PET reconstructed image based on the multimodal fusion feature map. Among them, the program instructions 61 can be stored in the above-mentioned storage medium in the form of a software product, including a number of instructions for enabling a device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the various implementation methods of the present application. The aforementioned storage medium includes: various media that can store program instructions, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, or a tablet. Among them, the server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0088] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0089] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. The above is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the content of the description and drawings of this application, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A PET imaging method based on optical flow registration, characterized in that: include: PET images and MR images of the irradiated area were collected separately; Acquire optical flow information of the PET image and the MR image through an optical flow registration network, correct the position of the MR image according to the optical flow information, and then extract a first feature map and a second feature map of the PET image and the corrected MR image respectively; Performing feature enhancement processing of the spatial dimension and the channel dimension on the first feature map and the second feature map respectively by a spatial channel feature enhancement module, and fusing the feature enhancement processing results of the spatial dimension and the channel dimension to generate a first multimodal enhanced feature map of the PET image and a second multimodal enhanced feature map of the corrected MR image; The first multimodal enhanced feature map and the second multimodal enhanced feature map are fused by a cross-modal feature fusion module using a multi-head cross-attention mechanism to generate a multimodal fusion feature map, and a PET reconstructed image is generated according to the multimodal fusion feature map.

2. The PET imaging method based on optical flow registration according to claim 1, characterized in that: Before acquiring the optical flow information of the PET image and the MR image through the optical flow registration network, the method further includes: The Canny operator is used to extract contour information from the PET image and the MR image respectively.

3. The PET imaging method based on optical flow registration according to claim 2, characterized in that: The use of the Canny operator to extract contour information from the PET image and the MR image is specifically as follows: The PET image and the MR image are subjected to image corrosion and dilation processing by using morphological image processing technology, small objects and thin lines in the PET image and the MR image are eliminated, the basic contour shape of the radiation part is retained, and finally a binary mask composed of 0 and 1 is generated to represent the contour information of the PET image and the MR image respectively.

4. The PET imaging method based on optical flow registration according to any one of claims 1 to 3, characterized in that: The optical flow registration network is a U-net based model, including an encoder and a decoder. The optical flow information of the PET image and the MR image is obtained through the optical flow registration network. After the position of the MR image is corrected according to the optical flow information, the first feature map and the second feature map of the PET image and the corrected MR image are respectively extracted as follows: The contour information of the PET image and the MR image is input into the optical flow registration network. The encoder part of the optical flow registration network uses three 1x1 convolution layers to extract feature maps, and uses a 3×3 convolution layer to downsample the extracted feature maps; the decoder part of the optical flow registration network uses upsampling and 1x1 convolution layers to expand and compress the extracted feature maps to restore the size of the optical flow map, and generates a weight mask with a channel number of 64. The weight mask is generated by applying a Sigmoid activation function, and the first feature map F of the PET image and the corrected MR image is output. pet and the second feature map F mr .

5. The PET imaging method based on optical flow registration according to claim 4, characterized in that: The spatial channel feature enhancement module performs feature enhancement processing of the first feature map and the second feature map in the spatial dimension and the channel dimension respectively, and fuses the feature enhancement processing results in the spatial dimension and the channel dimension to generate the first multimodal enhanced feature map of the PET image and the second multimodal enhanced feature map of the corrected MR image, specifically: The spatial channel feature enhancement module includes a spatial enhancement module and a channel enhancement module. The spatial enhancement module is implemented by two convolutional layers and the first feature map F pet and the second feature map F mr Compress and obtain two matrices S1 and S2. Connect the matrices S1 and S2 in the spatial dimension to obtain a single-dimensional matrix S containing the attention information of the PET and MR image streams, and then perform the first feature map F pet and the second feature map F mr Sigmoid function activation and element-by-element operation are performed to obtain the first spatial enhancement feature map F of the PET image and the corrected MR image in the spatial dimension. s-pet and the second spatial enhanced feature map F s-mr ; The channel enhancement module uses global average pooling to transform the first feature map F pet and the second feature map F mr The PET image and the MR image are compressed and a channel descriptor is created. The encoding vectors C1 and C2 of the PET image and the MR image are calculated by the channel descriptor. The encoding vectors C1 and C2 are connected and a fused embedding vector is created through a fully connected layer. The embedding vector is then activated by a Sigmoid function and element-by-element operations are performed to obtain the first channel enhanced feature map F of the PET image and the corrected MR image, respectively. c-pet and the second channel enhanced feature map F c-mr ; The first spatial enhancement feature map and the first channel enhancement feature map of the PET image and the second spatial enhancement feature map and the second channel enhancement feature map of the corrected MR image are fused to generate a first multimodal enhancement feature map F of the PET image. sc-pet and the second multimodal enhanced feature map F of the corrected MR image sc-mr .

6. The PET imaging method based on optical flow registration according to claim 5, characterized in that: After fusing the first multimodal enhanced feature map and the second multimodal enhanced feature map using a multi-head cross attention mechanism through the cross-modal feature fusion module, the multimodal fusion feature map is generated as follows: The cross-modal feature fusion module adopts a multi-head cross attention mechanism and a Transformer architecture to transform the first multimodal enhanced feature map F sc-pet and the second multimodal enhanced feature map F sc-mr A linear projection operation is performed to generate the query matrix Q, key matrix K, and value matrix V of the PET image and MR image, respectively, and an attention operation is performed between the vectorized features of the two modalities: In the above formula, σ represents the sigmoid function. represents the square root of the feature dimension, K pet , V pet They represent the query matrix Q, key matrix K and value matrix V of the PET image respectively. K mr , V mr They represent the query matrix Q, key matrix K and value matrix V of MR images respectively; After compressing the attention operation results of the two modalities through the average pooling operation, the attention maps of the two modalities are spliced ​​in the channel dimension to generate a multimodal fusion feature map.

7. The PET imaging method based on optical flow registration according to claim 6, characterized in that: The generating of the PET reconstructed image according to the multimodal fusion feature map is specifically as follows: The multimodal fusion feature map is input into a decoder through a jump connection, and a PET reconstructed image is output through the decoder.

8. A PET imaging device based on optical flow registration, characterized in that: include: Image acquisition module: used to respectively acquire PET images and MR images of the radiation site; An optical flow registration module: used for acquiring optical flow information of the PET image and the MR image through an optical flow registration network, and extracting a first feature map and a second feature map of the PET image and the corrected MR image respectively after correcting the position of the MR image according to the optical flow information; A feature enhancement module is used to perform feature enhancement processing of the first feature map and the second feature map in the spatial dimension and the channel dimension respectively through the spatial channel feature enhancement module, and fuse the feature enhancement processing results in the spatial dimension and the channel dimension to generate a first multimodal enhanced feature map of the PET image and a second multimodal enhanced feature map of the corrected MR image; Feature fusion module: used to fuse the first multimodal enhanced feature map and the second multimodal enhanced feature map through a cross-modal feature fusion module using a multi-head cross-attention mechanism to generate a multimodal fusion feature map, and generate a PET reconstructed image based on the multimodal fusion feature map.

9. A device, characterized in that: The device comprises a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the PET imaging method based on optical flow registration according to any one of claims 1 to 7; The processor is used to execute the program instructions stored in the memory to control a PET imaging method based on optical flow registration.

10. A storage medium, characterized in that: Program instructions executable by a processor are stored, and the program instructions are used to execute the PET imaging method based on optical flow registration as described in any one of claims 1 to 7.

Images