Model training method and device, image correction method and device, equipment and storage medium

Abstract

The invention discloses a model training method and device, an image correction method and device, equipment and a storage medium, and the method comprises the steps: obtaining a plurality of first training samples, each first training sample comprises a CT image obtained by photographing a first target object under a first dose, a first PET image obtained by photographing the first target object under a second dose, and a CT image obtained by photographing the first target object under the second dose; and a second PET image obtained by shooting the first target object under a third dose, wherein the third dose is greater than the first dose and greater than the second dose; a plurality of first training samples are adopted to train the diffusion model to obtain a target model, the model is used for correcting the input PET image which is not subjected to attenuation correction according to the input CT image, in the process, the CT image with the relatively low dose, the first PET image and the second PET image with the relatively high dose are adopted to train the model, and the target model is obtained. And the PET image is corrected by adopting the target model, so that the correction accuracy of the PET image can be improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a model training method, an image correction method, a device, a equipment and a storage medium. Background Art

[0002] PET / CT equipment is one of the current high-end molecular imaging devices. It efficiently integrates PET functional information with CT anatomical information, and can image the metabolism, function, blood flow, cell proliferation and receptor distribution of tissue cells at the molecular level. It is widely used in the medical field.

[0003] Attenuation correction technology is particularly important in medical imaging, especially in PET / CT imaging, which can correct the attenuation effect caused by gamma photons passing through human tissues, ensuring the accuracy of qualitative and quantitative diagnosis of PET images. At present, deep learning methods are commonly used to directly generate attenuation-corrected PET (AC PET) images from non-attenuation-corrected PET (NAC PET).

[0004] Although the above method eliminates the radiation of CT scanning, it lacks the process of attenuation correction of CT images, which affects the accuracy of image correction. Summary of the invention

[0005] The embodiments of the present application provide a model training method, an image correction method, an apparatus, a device and a storage medium, which use relatively low-dose CT images, PET images that have not been attenuation corrected, and relatively high-dose PET images that have been attenuation corrected to train the model. The obtained target model can correct the PET image, thereby improving the accuracy of PET image correction.

[0006] In a first aspect, an embodiment of the present application provides a model training method, the method comprising:

[0007] Acquire a plurality of first training samples, each of the first training samples comprising a CT image obtained by photographing a first target object at a first dose, a first PET image obtained by photographing the first target object at a second dose, and a second PET image obtained by photographing the first target object at a third dose, the third dose being greater than the first dose, the third dose being greater than the second dose, the first PET image being a PET image without attenuation correction, and the second PET image being a PET image with attenuation correction;

[0008] The image processing model is trained using a plurality of the first training samples to obtain a target model, wherein the target model is used to correct an input PET image that has not been attenuation corrected according to an input CT image to obtain an attenuation corrected PET image.

[0009] In an embodiment of the present application, the step of using the plurality of the first training samples to train the image processing model to obtain the target model includes:

[0010] The following processing is performed for each of the first training samples:

[0011] According to the first PET image in the first training sample, a dose corresponding to the first PET image is obtained; and the first PET image and the CT image are fused to obtain a fused image;

[0012] The image processing model is trained using the fused image, the dose, and the second PET image corresponding to each of the first training samples to obtain a target model.

[0013] In an embodiment of the present application, obtaining a dose corresponding to the first PET image according to the first PET image in the first training sample includes:

[0014] The first PET image in the first training sample is input into a neural network model to obtain a dose corresponding to the first PET image. The neural network model is trained using multiple second training samples. The second training samples include a third PET image and a dose corresponding to the third PET image. The third PET image is a PET image that has not been attenuated corrected.

[0015] In one embodiment of the present application, the fusing process of the first PET image and the CT image to obtain a fused image includes:

[0016] Acquire image features of the first PET image and the CT image respectively, to obtain a first image feature of the first PET image and a second image feature of the CT image;

[0017] fusing the first image feature and the second image feature to obtain a fused feature;

[0018] The fused features are subjected to noise superposition processing to obtain the fused image.

[0019] In an embodiment of the present application, the fusing process of the first image feature and the second image feature to obtain the fused feature includes:

[0020] Using a cross-attention mechanism to perform feature encoding on the first image feature to obtain a first feature vector, where the first feature vector includes a key vector and a value vector;

[0021] Using the cross attention mechanism to perform feature encoding on the second image feature to obtain a second feature vector, where the second feature vector includes a query vector;

[0022] The fusion feature is obtained by performing calculation according to the first image feature, the key vector, the value vector, and the query vector.

[0023] In a second aspect, an embodiment of the present application provides an image correction method, the method comprising:

[0024] Acquire a target CT image obtained by photographing the second target object at a fourth dose, and a first target PET image obtained by photographing the second target object at a fifth dose, wherein the first target PET image is a PET image that has not been attenuated corrected;

[0025] The target CT image and the target PET image are input into the target model to obtain a second target PET image at a third dose, wherein the second target PET image is an attenuation-corrected PET image, the third dose is greater than the fourth dose, and the third dose is greater than the fifth dose. The target model is obtained by the model training method described in the first aspect.

[0026] In a third aspect, an embodiment of the present application provides an image correction device, the device comprising:

[0027] A first acquisition module is used to acquire a plurality of first training samples, each of which includes a CT image obtained by photographing a first target object at a first dose, a first PET image obtained by photographing the first target object at a second dose, and a second PET image obtained by photographing the first target object at a third dose, wherein the third dose is greater than the first dose, the third dose is greater than the second dose, the first PET image is a PET image without attenuation correction, and the second PET image is a PET image with attenuation correction;

[0028] A training module is used to use multiple of the first training samples to train the image processing model to obtain a target model, and the target model is used to correct the input PET image that has not been attenuation corrected according to the input CT image to obtain an attenuation corrected PET image.

[0029] In a fourth aspect, an embodiment of the present application provides an image correction device, the device comprising:

[0030] a second acquisition module, configured to acquire a target CT image obtained by photographing a second target object at a fourth dose, and a first target PET image obtained by photographing the second target object at a fifth dose, wherein the first target PET image is a PET image that has not been attenuated corrected;

[0031] A correction module is used to input the target CT image and the target PET image into a target model to obtain a second target PET image at a third dose, wherein the second target PET image is a PET image corrected for attenuation, the third dose is greater than the fourth dose, and the third dose is greater than the fifth dose. The target model is obtained by the model training method described in the first aspect.

[0032] In a fifth aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory storing computer program instructions;

[0033] When the processor executes the computer program instructions, it implements the model training method as described in the first aspect, or the image correction method as described in the second aspect.

[0034] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the model training method as described in the first aspect, or the image correction method as described in the second aspect is implemented.

[0035] In the seventh aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the model training method as described in the first aspect, or the image correction method as described in the second aspect.

[0036] The model training method, image correction method, device, equipment and storage medium of the embodiments of the present application obtain multiple first training samples, each first training sample includes a CT image obtained by photographing a first target object at a first dose, a first PET image obtained by photographing the first target object at a second dose, and a second PET image obtained by photographing the first target object at a third dose, the third dose is greater than the first dose and greater than the second dose, and a relatively low-dose CT image, a first PET image and a relatively high-dose second PET image are used to train the model, and the target model is used to correct the PET image, which can improve the accuracy of PET image correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 It is a flow chart of the model training method provided in the embodiment of the present application;

[0039] Figure 2 is a flow chart of an image correction method provided by an embodiment of the present application;

[0040] Figure 3 A standard dose attenuation-corrected PET image provided in an embodiment of the present application;

[0041] Figure 4 A low-dose PET image without attenuation correction provided in an embodiment of the present application;

[0042] Figure 5 Another standard dose attenuation-corrected PET image provided in an embodiment of the present application;

[0043] Figure 6 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;

[0044] Figure 7 is a structural schematic diagram of an image correction device provided in an embodiment of the present application;

[0045] Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0047] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0048] In each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the disclosed embodiment will be obtained.

[0049] In order to solve the problems of the prior art, the embodiments of the present application provide a model training method, an image correction method, an apparatus, a device and a storage medium. The image correction method provided by the embodiments of the present application is first introduced below.

[0050] Figure 1 A schematic diagram of a process flow of a model training method provided in an embodiment of the present application is shown. Figure 1 As shown, the model training method provided in the embodiment of the present application is applied to an electronic device, such as a server, and includes the following steps 101-102, wherein:

[0051] Step 101, obtaining a plurality of first training samples, each of the first training samples comprising a CT image obtained by photographing a first target object at a first dose, a first PET image obtained by photographing the first target object at a second dose, and a second PET image obtained by photographing the first target object at a third dose, wherein the third dose is greater than the first dose, and the third dose is greater than the second dose, the first PET image is a PET image without attenuation correction, and the second PET image is a PET image with attenuation correction.

[0052] In this embodiment, multiple first training samples are obtained, each of which includes a CT image obtained by photographing a first target object at a first dose, where the CT image is an attenuation-corrected CT image, i.e., an AC CT image; a first PET image obtained by photographing the target object at a second dose, where the first PET image is a PET image without attenuation correction, i.e., a NAC PET image; and a second PET image obtained by photographing the target object at a third dose, where the second PET image is an attenuation-corrected PET image, i.e., an AC PET image. The second PET image is used as a standard for training the image processing model.

[0053] The first target object mentioned above refers to a user, and considering the influence of radiation, it may be a plurality of different users.

[0054] The first dose corresponds to a CT image, the first dose may be the same dose or multiple different doses, and the CT image may be a CT image obtained by photographing the first target object at the same dose, or a CT image obtained by photographing the first target object at multiple different doses.

[0055] The second dose corresponds to the first PET image, the second dose may be the same dose or multiple different doses, the first PET image may be the first PET image obtained by photographing the first target object at the same dose, or the first PET image obtained by photographing the first target object at multiple different doses. The CT image and the first PET image are in one-to-one correspondence, for example, the CT image and the first PET image need to be photographed for the same user in a consent posture.

[0056] The third dose is greater than the second dose and greater than the first dose. The third dose corresponds to the second PET image. The third dose may be the second PET image obtained by photographing the first target object at a standard dose.

[0057] The PET images and CT images in the embodiments of the present disclosure are derived from clinical PET / CT equipment. It should be noted that the user's separate permission or consent is obtained. After the user's separate permission or consent is clearly obtained, the necessary user-related data for the normal operation of the embodiments of the present disclosure are obtained.

[0058] Step 102: Use a plurality of the first training samples to train an image processing model to obtain a target model. The target model is used to correct an input PET image that has not been attenuation corrected according to an input CT image to obtain an attenuation corrected PET image.

[0059] In this embodiment, a plurality of first training samples are used to train the image processing model to obtain a target model, and the target model is used to correct the input PET image without attenuation correction according to the input CT image to obtain the attenuation corrected PET image.

[0060] By acquiring multiple first training samples, each first training sample includes a CT image obtained by photographing a first target object at a first dose, a first PET image obtained by photographing the first target object at a second dose, and a second PET image obtained by photographing the first target object at a third dose, the third dose is greater than the first dose and greater than the second dose, using relatively low-dose CT images, first PET images and relatively high-dose second PET images to train the model, and using the target model to correct the PET image, the accuracy of PET image correction can be improved and the clarity of the image can be improved.

[0061] In one embodiment of the present application, specifically, step 102, using a plurality of the first training samples to train the image processing model to obtain a target model, includes:

[0062] The following processing is performed for each of the first training samples:

[0063] According to the first PET image in the first training sample, a dose corresponding to the first PET image is obtained; and the first PET image and the CT image are fused to obtain a fused image;

[0064] The image processing model is trained using the fused image, the dose, and the second PET image corresponding to each of the first training samples to obtain a target model.

[0065] In this embodiment, the following processing is performed for each first training sample: based on the first PET image in the first training sample, a dose corresponding to the first PET image is obtained, and the first PET image and the CT image are fused to obtain a fused image.

[0066] The image processing model is trained using the fused image, dose and second PET image corresponding to each first training sample to obtain a target model, and the second PET image is used as a standard for image processing model training.

[0067] Fusion can make up for the shortcomings of a single image feature and provide support for subsequent image processing and analysis.

[0068] In one embodiment of the present application, obtaining a dose corresponding to the first PET image according to the first PET image in the first training sample includes:

[0069] The first PET image in the first training sample is input into a neural network model to obtain a dose corresponding to the first PET image. The neural network model is trained using multiple second training samples. The second training samples include a third PET image and a dose corresponding to the third PET image. The third PET image is a PET image that has not been attenuated corrected.

[0070] The training data includes first PET images of multiple doses. By observing the first PET images of different doses, the lower the dose, the greater the noise level of the first PET image, and the greater the difficulty of attenuation correction. Based on this, a neural network model is constructed, and the neural network model is used to perceive the dose of the first PET image. The neural network model includes N basic units, such as N can be set to 5. Each basic unit includes a convolution layer, an activation layer, and a pooling layer connected in sequence. The input and output of the i-th unit are expressed as:

[0071]

[0072] Among them, x i is the output of the basic unit, x i-1 is the input of the basic unit, w j and b j represents the weight and bias corresponding to the convolution operation, and g(.) represents the activation layer.

[0073] In order to accelerate the convergence of the model, an instance normalization layer (Instance Norm) is added after the convolution layer of each basic unit. After the Nth basic unit, the fully connected layer and the softmax layer are connected in sequence. The output of the neural network model can be expressed as:

[0074] C dose =F(x input )=R(f L (…f1(x input ))) (2)

[0075] Among them, x input represents the first PET image input, C dose represents the estimated dose, and R represents the fully connected layer and the softmax layer.

[0076] The neural network model is trained using multiple second training samples to obtain a trained neural network model, wherein the multiple second training samples include multiple doses, and the same third PET image can correspond to one dose, or to multiple different doses, and the same dose can correspond to multiple third PET images.

[0077] The first PET image in the first training sample is input into the trained neural network model to obtain the dose corresponding to the first PET image, where the dose represents the radiation dose of the first PET image. The dose is used as a variable to participate in the image processing model training to adapt to the first PET images with different doses.

[0078] In one embodiment of the present application, the fusing process of the first PET image and the CT image to obtain a fused image includes:

[0079] Acquire image features of the first PET image and the CT image respectively, to obtain a first image feature of the first PET image and a second image feature of the CT image;

[0080] fusing the first image feature and the second image feature to obtain a fused feature;

[0081] The fused features are subjected to noise superposition processing to obtain the fused image.

[0082] In the above steps, the encoder is used to acquire the image features of the first PET image to obtain the first image features of the first PET image, and the encoder is used to acquire the image features of the CT image to obtain the second image features of the CT image.

[0083] The encoder may be a variational autoencoder (VAE), specifically, a variational autoencoder network is used as a basic network framework, including an encoder and a decoder, and the first PET image and the CT image are used as inputs of the encoder, and the output of the encoder is used as input of the decoder to realize encoding and decoding of its own features. After the variational autoencoder network training is completed, the encoder in the variational autoencoder network is selected to extract the image features of the first PET image and the image features of the CT image.

[0084] Furthermore, the first image feature and the second image feature are fused to obtain a fused feature, and the feature of the first image feature can be enhanced through the fusion.

[0085] The image processing model can be a diffusion model, including a forward diffusion process and a reverse generation process, wherein the forward diffusion process is to gradually add noise to the data to transform it into pure noise. This process is usually a Markov chain, with a small amount of Gaussian noise added at each step. The reverse generation process is to learn to gradually remove noise from the noise and restore the original data. It is also a Markov chain, but in the opposite direction, and gradually remove noise.

[0086] Specifically, the fusion feature is processed in the shallow space to obtain the high-dimensional fusion feature f0, and the diffusion step size is pre-set. Based on the pre-set diffusion step size, the noise is gradually superimposed on the high-dimensional fusion feature f0, and the noise image f is gradually obtained by continuously superimposing Gaussian noise. t , the noise intensity is expressed as:

[0087]

[0088] Among them, N represents the standard normal distribution, α t For noise scheduling, I is the unit matrix of the same dimension as the input high-dimensional fusion feature f0. After the diffusion step t, the original high-dimensional fusion feature f0 is gradually transformed into a noise image f that obeys the standard Gaussian distribution. t .

[0089] The reverse process is to get the noise image f t The noise is gradually removed from the image, thereby restoring the original high-dimensional fusion feature f0. However, it is difficult to directly restore the original image from the noisy image. Therefore, the process of decoupling noise is based on the noise image at the previous moment, that is, the noise image f t-1 is the noise image f at the previous moment t By analogy, it can be expressed as:

[0090] p(f t-1 |f t )=N(f t-1 ;μ θ (f t ,t),σI) (4)

[0091] Among them, μ θ (f t ,t) is the noise image f t The mean of the noise image f t The variance of .

[0092] Among them, the noise image f t The corresponding expression of the mean is:

[0093]

[0094] Among them, the noise image f t The corresponding expression of the variance is:

[0095]

[0096] In order to facilitate calculation, β t =1-α t .

[0097] Specifically, the image processing model is trained using the fused image, the dose, and the second PET image corresponding to each first training sample to obtain a target model, including:

[0098] The image processing model can be a U-Net neural network model, expressed as:

[0099]

[0100] in, is the attenuation-corrected PET image output by the model, F(·) represents the image processing model, c represents the external condition input, including fusion features and dose, and t is the preset diffusion step size.

[0101] In one embodiment of the present application, the fusing process of the first image feature and the second image feature to obtain the fused feature includes:

[0102] Using a cross-attention mechanism to perform feature encoding on the first image feature to obtain a first feature vector, where the first feature vector includes a key vector and a value vector;

[0103] Performing feature encoding on the second image feature using the cross attention mechanism to obtain a second feature vector, where the second feature vector includes a query vector;

[0104] The fusion feature is obtained by performing calculation according to the first image feature, the key vector, the value vector, and the query vector.

[0105] In this embodiment, a cross-attention mechanism is used to feature encode the source data. Specifically, the first image feature is feature encoded to obtain a first feature vector, which includes a key vector K1, a query vector Q1, and a value vector V1.

[0106] The cross-attention mechanism is used to perform feature encoding on the second image features to obtain a second feature vector, which includes a key vector K2, a query vector Q2, and a value vector V2.

[0107] Substitute the first image feature, key vector K1, value vector V1, and query vector Q2 into formula (8) to calculate and obtain the fusion feature, which is as follows:

[0108]

[0109] in, represents the fusion feature, Q2 is the query vector in the second feature vector, K1 is the key vector in the first feature vector, V1 is the value vector in the first feature vector, C1 is the scale factor, and f PET Represents the first image feature.

[0110] The cross-attention mechanism can more effectively integrate features from different modalities, allowing the model to capture global and local information at the same time, accelerate learning and reduce interference, and can reduce subsequent model training time.

[0111] Figure 2 A schematic diagram of a process flow of an image correction method provided by an embodiment of the present application is shown. Figure 2 As shown, the image correction method provided in the embodiment of the present application is applied to an electronic device, such as a server, and includes the following steps 201-202, wherein:

[0112] Step 201 , obtaining a target CT image obtained by photographing a second target object at a fourth dose, and a first target PET image obtained by photographing the second target object at a fifth dose, wherein the first target PET image is a PET image without attenuation correction.

[0113] The second target object may be different from the first target object or the same as the first target object, which is not limited here. A target CT image obtained by photographing the second target object at a fourth dose and a first target PET image obtained by photographing the second target object at a fifth dose are obtained, wherein the first target image is a PET image without attenuation correction.

[0114] Step 202, input the target CT image and the target PET image into the target model to obtain a second target PET image at a third dose, the second target PET image is an attenuation-corrected PET image, the third dose is greater than the fourth dose, and the third dose is greater than the fifth dose. The target model is obtained based on the above-mentioned model training method.

[0115] In this embodiment, the target CT image and the target PET are input into the target model, and the target model outputs a second target PET image at a third dose, wherein the third dose is greater than the fourth dose and greater than the fifth dose, and the second target PET image is an attenuation-corrected PET image. By correcting the PET image through the target model, a clear PET image can be obtained. Moreover, the user only needs to take low-dose PET images and CT images, and a high-dose attenuation-corrected PET image can be obtained through the target model, thereby reducing radiation to the user.

[0116] The following is an example of the image correction method provided in the embodiment of the present application.

[0117] Step 0: Construct a training sample set.

[0118] In this embodiment, the training sample set is represented by D = {(x1, y1), (x2, y2), ..., (x n ,y n)}, where {(x1,x2,...,x n )} is represented as an ultra-low-dose ACCT image (ie, a CT image obtained by photographing the first target object at the first dose included in each first training sample mentioned above).

[0119] {(y1,y2,...,y n )} represents NAC PET images of different dose levels (i.e., each first training sample in the above text includes a first PET image obtained by photographing the first target object at the second dose. For example, the second dose in the above text can be the same dose or multiple different doses. For the convenience of distinction, the dose corresponding to the PET image without attenuation correction is called the second dose).

[0120] z={z1,z2,…,z n} is an AC PET image of standard dose (ie, the second PET image obtained by photographing the first target object at the third dose included in each training sample mentioned above), and n is the total number of training sample sets.

[0121] The constructed training sample set includes NAC PET images of various dose levels, each of which can be extracted from the standard raw data by randomly selecting a certain proportion of counting events. The PET image size is 192×192, and the ACCT image size is 512×512.

[0122] Step 1: Construct a priori information encoding module.

[0123] In this embodiment, an image processing model is constructed and equipped with a prior information encoding module, a multimodal feature extraction module, and a cross-modal feature fusion module. The above image processing model, prior information encoding module, multimodal feature extraction module, and cross-modal feature fusion module constitute a dose-sensitive diffusion model that fuses multimodal features.

[0124] The above training sample set contains NAC PET images of multiple doses. By observing NAC PET images of different dose levels, it can be found that the lower the dose level, the greater the noise level of the NAC PET image, and the greater the difficulty of attenuation correction. Based on this, a priori information encoding module (the neural network model constructed above) is constructed, which can perceive the different dose levels of NAC PET images and promote the PET attenuation correction process as a priori. The priori information encoding includes 5 basic units. Each basic unit includes a convolution layer, an activation layer and a pooling layer (that is, the neural network model in the above includes N basic units, such as N can be set to 5, and each basic unit includes a convolution layer, an activation layer, and a pooling layer connected in sequence). The input and output of the i-th unit can be expressed by formula (1).

[0125] In order to accelerate the convergence of the model, an instance normalization layer (Instance Norm) is added after each convolution layer. The fully connected layer and the softmax layer are connected after the fifth basic unit. The output of the final model can be expressed as formula (2).

[0126] Step 2: Construct a multimodal feature extraction module.

[0127] For low-dose NAC PET images and ultra-low-dose ACCT images, feature extraction is achieved using a multimodal feature extraction module. When constructing the multimodal feature extraction module, a variational autoencoder network (VAE) framework is used as the basic network framework, including an encoder and a decoder. The NAC PET image and the ACCT image are used as the input of the encoder, and the output of the encoder is used as the input of the decoder to achieve the encoding and decoding of their own features. After the variational autoencoder network training is completed, the encoder of the VAE is selected as the multimodal feature extraction module to extract features of the low-dose NAC PET image and the ultra-low-dose ACCT image, respectively, to obtain the first image feature of the NAC PET image, and the second image feature of the ACCT image (that is, the image features of the first PET image and the CT image are obtained respectively in the above text, to obtain the first image feature of the first PET image, and the second image feature of the CT image), to achieve compression of the multimodal features.

[0128] Step 3: Construct a cross-modal feature fusion module.

[0129] In order to integrate multimodal information, a cross-modal feature fusion module based on the cross-attention mechanism is constructed. In the attention allocation calculation process, the source data features are encoded into three vectors, named key (K) vector, query (Q) vector and value (V) vector, and the source data features are enhanced according to these three vectors:

[0130]

[0131] Among them, X represents the source data characteristics, represents the feature after attention enhancement, d k Represents the dimension of K.

[0132] In this embodiment, the first image feature of the NAC PET feature and the second image feature of the ACCT image extracted by the multimodal feature extraction module are used to obtain the KQV vector of the first image feature by convolution, including the key vector K1, the query vector Q1, and the value vector V1, and the KQV vector of the second characteristic feature is obtained by convolution, including the key vector K2, the query vector Q2, and the value vector V2. Q2 is used to replace Q1, and the attention weight of the PET feature at the pixel level is obtained by calculation, that is, the NAC PET feature after attention enhancement is obtained (that is, the fusion feature is obtained by calculating according to the first image feature, the key vector, the value vector, and the query vector in the above text), and the calculation formula is shown in formula (8).

[0133] Step 4: Construct a diffusion model.

[0134] In this embodiment, the diffusion model (i.e., the image processing model mentioned above) is a type of probabilistic generation model that can reverse the diffusion process through training. The diffusion model is in the form of p θ (x0): =∫p θ (x 0:T )dx 1:T A latent variable model where x1 to x T is a latent variable with the same dimension as x0, where x0~q(x0). The joint distribution p θ (x 0:T ) is called the inverse process and is a learnable Markov chain with Gaussian transformation as its feature.

[0135] Specifically, in the forward process, the attention-enhanced NAC PET features are processed in the shallow space (i.e., the fusion features are processed in the shallow space to obtain the high-dimensional fusion features f0), and the high-dimensional features f0 (i.e., the high-dimensional fusion features in the above) are obtained. By continuously superimposing Gaussian noise, the noise image f t , the noise image f t In the diffusion step t, there is a predefined noise schedule α t , the noise intensity at time step t is shown in formula (3).

[0136] The reverse process attempts to extract the noise f T The noise is gradually removed to restore the original high-dimensional feature f0 (that is, the reverse process is to get the noise image f t The noise is gradually removed from the image, thereby restoring the original high-dimensional fusion feature f0). However, it is difficult to directly restore the original image from the noisy image. Therefore, the process of decoupling noise is based on the noise image at the previous moment, that is, the noise image f t-1 is the noise image f at the previous moment t Then, it can be expressed by formula (4).

[0137] As shown in equations (5)-(6), the variance is a quantitative value (the diffusion process parameters are fixed), while the mean is a value that depends on f0 and f t The specific noise added in each step of the actual reasoning process is unknown. The diffusion model can be a U-Net neural network model (that is, the image processing model in the above text can be a U-Net neural network model). The diffusion model is shown in formula (7).

[0138] Step 5: Train the diffusion model to obtain the target model.

[0139] In this example, low-dose NAC PET images i and ultra-low-dose AC CT images x i As input to the diffusion model, a standard dose AC PET image z i As a standard, see Figure 3 , Figure 3 A standard dose attenuation-corrected PET image is provided in an embodiment of the present application. A diffusion model is trained using a training sample set to obtain a target model (i.e., a plurality of first training samples are used to train the diffusion model to obtain a target model), and an Adam optimizer is used for optimization during the training process.

[0140] Step 6: Application of target model.

[0141] See also Figure 4 As shown, Figure 4 A low-dose PET image without attenuation correction is provided in the embodiment of the present application. The low-dose NAC PET image and the ultra-low-dose AC CT image are input into the target model, and the target model outputs a standard-dose AC PET image, such as Figure 5 As shown, Figure 5 Another standard dose attenuation-corrected PET image provided in the embodiment of the present application is compared with Figure 4 and Figure 5 ,By correcting the PET images, the quality of the PET images is improved, the clarity of the images is improved, and it is convenient for subsequent use.

[0142] Low-dose NAC PET images and ultra-low-dose ACCT images are used for attenuation correction, which significantly reduces the radiation dose during PET imaging. The dual low-dose strategy is aimed at clinical PET / CT equipment and has certain universality and clinical practicability. A prior information encoding module is constructed, and the network will have different responses to NAC PET images of different dose levels. A dose-sensitive diffusion model that integrates multimodal features is constructed, which integrates multimodal features and combines a stable generative model framework. Dual low doses can reduce CT scanning time and avoid respiratory artifacts during CT scanning as much as possible.

[0143] Figure 6 The structure diagram of the model training device provided in the embodiment of the present application is shown. Figure 6 As shown, the model training device 600 includes:

[0144] A first acquisition module 601 acquires a plurality of first training samples, each of the first training samples includes a CT image obtained by photographing a first target object at a first dose, a first PET image obtained by photographing the first target object at a second dose, and a second PET image obtained by photographing the first target object at a third dose, the third dose is greater than the first dose, the third dose is greater than the second dose, the first PET image is a PET image without attenuation correction, and the second PET image is a PET image with attenuation correction;

[0145] The training module 602 is used to use multiple of the first training samples to train the image processing model to obtain a target model, and the target model is used to correct the input PET image without attenuation correction according to the input CT image to obtain the attenuation corrected PET image.

[0146] In one embodiment of the present application, the training module 602 includes:

[0147] an acquisition submodule, configured to obtain a dose corresponding to the first PET image according to the first PET image in the first training sample;

[0148] A fusion submodule, used for fusing the first PET image and the CT image to obtain a fused image;

[0149] The training submodule is used to train the image processing model using the fused image, the dose and the second PET image corresponding to each of the first training samples to obtain a target model.

[0150] In one embodiment of the present application, the acquisition submodule includes a first acquisition unit;

[0151] The first acquisition unit is used to input the first PET image in the first training sample into the neural network model to obtain the dose corresponding to the first PET image. The neural network model is trained using multiple second training samples. The second training samples include a third PET image and the dose corresponding to the third PET image. The third PET image is a PET image that has not been attenuated corrected.

[0152] In one embodiment of the present application, the fusion submodule includes:

[0153] A second acquisition unit is used to respectively acquire image features of the first PET image and the CT image to obtain a first image feature of the first PET image and a second image feature of the CT image;

[0154] a fusion unit, configured to fuse the first image feature and the second image feature to obtain a fused feature;

[0155] The third acquisition unit is used to perform noise superposition processing on the fusion feature to obtain the fusion image.

[0156] In one embodiment of the present application, the fusion unit includes:

[0157] A first acquisition subunit is used to perform feature encoding on the first image feature by adopting a cross attention mechanism to obtain a first feature vector, where the first feature vector includes a key vector and a value vector;

[0158] A second acquisition subunit is used to perform feature encoding on the second image feature by using the cross attention mechanism to obtain a second feature vector, where the second feature vector includes a query vector;

[0159] The third acquisition subunit is used to perform calculation according to the first image feature, the key vector, the value vector, and the query vector to obtain the fusion feature.

[0160] The model training device provided in the embodiment of the present application can implement the various processes implemented in the aforementioned model training method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0161] Figure 7 FIG. 1 shows a structural diagram of an image correction device provided in an embodiment of the present application. Figure 7 As shown, the image correction device 700 includes:

[0162] A second acquisition module 701 is configured to acquire a target CT image obtained by photographing a second target object at a fourth dose, and a first target PET image obtained by photographing the second target object at a fifth dose, wherein the first target PET image is a PET image without attenuation correction;

[0163] The correction module 702 is used to input the target CT image and the target PET image into the target model to obtain a second target PET image at a third dose, wherein the second target PET image is a PET image corrected by attenuation, the third dose is greater than the fourth dose, and the third dose is greater than the fifth dose. The target model is obtained based on the above-mentioned model training method.

[0164] The image correction device provided in the embodiment of the present application can implement the various processes implemented in the aforementioned image correction method embodiment and achieve the same technical effect. To avoid repetition, it will not be described here.

[0165] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0166] The electronic device may include a processor 801 and a memory 802 storing computer program instructions.

[0167] Specifically, the processor 801 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0168] The memory 802 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 802 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 802 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 802 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 802 is a non-volatile solid-state memory.

[0169] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect or the second aspect of the present disclosure.

[0170] The processor 801 implements any one of the information auditing methods in the above embodiments by reading and executing computer program instructions stored in the memory 802 .

[0171] In one example, the electronic device may further include a communication interface 803 and a bus 810. Figure 8 As shown, the processor 801, the memory 802, and the communication interface 803 are connected via a bus 810 and communicate with each other.

[0172] The communication interface 803 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0173] Bus 810 includes hardware, software or both, and the parts of information audit method or verification equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 810 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0174] In addition, in combination with the model training method or image correction method in the above embodiments, the embodiments of the present application may provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the image correction methods in the above embodiments is implemented.

[0175] In addition, the embodiments of the present application may be implemented by providing a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device implements any one of the model training methods or image correction methods in the above embodiments.

[0176] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiment, several specific steps are described as examples. However, the method process of the present application is not limited to the specific steps described, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0177] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0178] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0179] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0180] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a plurality of first training samples, each of the first training samples comprising a CT image obtained by photographing a first target object at a first dose, a first PET image obtained by photographing the first target object at a second dose, and a second PET image obtained by photographing the first target object at a third dose, the third dose being greater than the first dose, and the third dose being greater than the second dose, the first PET image being a PET image without attenuation correction, and the second PET image being a PET image with attenuation correction; The image processing model is trained using a plurality of the first training samples to obtain a target model, wherein the target model is used to correct an input PET image that has not been attenuation corrected according to an input CT image to obtain an attenuation corrected PET image.

2. The model training method according to claim 1, characterized in that: The step of using the plurality of the first training samples to train the image processing model to obtain a target model includes: The following processing is performed for each of the first training samples: According to the first PET image in the first training sample, a dose corresponding to the first PET image is obtained; and the first PET image and the CT image are fused to obtain a fused image; The image processing model is trained using the fused image, the dose, and the second PET image corresponding to each of the first training samples to obtain a target model.

3. The model training method according to claim 2, characterized in that: The step of obtaining a dose corresponding to the first PET image according to the first PET image in the first training sample includes: The first PET image in the first training sample is input into a neural network model to obtain a dose corresponding to the first PET image. The neural network model is trained using multiple second training samples. The second training samples include a third PET image and a dose corresponding to the third PET image. The third PET image is a PET image that has not been attenuated corrected.

4. The model training method according to claim 2, characterized in that: The fusing the first PET image and the CT image to obtain a fused image includes: Acquire image features of the first PET image and the CT image respectively, to obtain a first image feature of the first PET image and a second image feature of the CT image; fusing the first image feature and the second image feature to obtain a fused feature; The fused features are subjected to noise superposition processing to obtain the fused image.

5. The model training method according to claim 4, characterized in that: The fusing the first image feature and the second image feature to obtain a fused feature includes: Using a cross-attention mechanism to perform feature encoding on the first image feature to obtain a first feature vector, where the first feature vector includes a key vector and a value vector; Using the cross attention mechanism to perform feature encoding on the second image feature to obtain a second feature vector, where the second feature vector includes a query vector; The fusion feature is obtained by performing calculation according to the first image feature, the key vector, the value vector, and the query vector.

6. An image correction method, characterized in that: include: Acquire a target CT image obtained by photographing the second target object at a fourth dose, and a first target PET image obtained by photographing the second target object at a fifth dose, wherein the first target PET image is a PET image that has not been attenuated corrected; The target CT image and the target PET image are input into a target model to obtain a second target PET image at a third dose, wherein the second target PET image is an attenuation-corrected PET image, the third dose is greater than the fourth dose, and the third dose is greater than the fifth dose, and the target model is obtained based on the model training method described in any one of claims 1 to 5.

7. A model training device, characterized in that: The device comprises: A first acquisition module is used to acquire a plurality of first training samples, each of which includes a CT image obtained by photographing a first target object at a first dose, a first PET image obtained by photographing the first target object at a second dose, and a second PET image obtained by photographing the first target object at a third dose, wherein the third dose is greater than the first dose, the third dose is greater than the second dose, the first PET image is a PET image without attenuation correction, and the second PET image is a PET image with attenuation correction; A training module is used to use multiple of the first training samples to train the image processing model to obtain a target model, and the target model is used to correct the input PET image that has not been attenuation corrected according to the input CT image to obtain an attenuation corrected PET image.

8. An image correction device, characterized in that: The device comprises: a second acquisition module, configured to acquire a target CT image obtained by photographing a second target object at a fourth dose, and a first target PET image obtained by photographing the second target object at a fifth dose, wherein the first target PET image is a PET image that has not been attenuated corrected; A correction module is used to input the target CT image and the target PET image into a target model to obtain a second target PET image at a third dose, wherein the second target PET image is a PET image corrected for attenuation, the third dose is greater than the fourth dose, and the third dose is greater than the fifth dose, and the target model is obtained based on the model training method described in any one of claims 1 to 5.

9. An electronic device, characterized in that: include: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the model training method as described in any one of claims 1 to 5, or the image correction method as described in claim 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, they implement the model training method as described in any one of claims 1 to 5, or the image correction method as described in claim 6.

Images