A personalized CT image reconstruction system and federated learning method based on double physical driving

Abstract

The application belongs to the technical field of image processing, and discloses a personalized CT image reconstruction system based on double physical driving, which comprises an encoder, an anatomical information capturing module, a scanning information capturing module, a personalized modulation module and a decoder; the encoder is used for feature extraction of a scanned image to obtain imaging features; the anatomical information capturing module is used for extracting anatomical modulation parameters containing anatomical information according to a radiology report; the scanning information capturing module is used for capturing potential relationships between scanning protocols and noise distributions to obtain scanning modulation parameters containing physical information; the personalized modulation module is used for modulating the imaging features obtained by the encoder according to the anatomical modulation parameters and the scanning modulation parameters to obtain personalized imaging features; and the decoder is used for generating a personalized CT image according to the personalized imaging features. The application also discloses a federated learning method suitable for personalized CT image reconstruction. Through double physical driving based on scanning parameters and anatomical information, the application can effectively realize personalized CT imaging.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology and relates to a CT image reconstruction method, and in particular to a personalized CT image reconstruction system and a federated learning method based on dual physical drive. Background Art

[0002] Computed tomography (CT) is an important clinical examination method because it can non-invasively present anatomical information within the body. Despite its clinical advantages, CT scanning carries the risk of radiation exposure, which can lead to diseases such as gene mutations and cancer. Reducing CT dose (i.e., reducing the number of incident photons and the sampling angle) has become an effective approach to reduce radiation risk and speed up scanning. However, under these conditions, the quality of reconstructed images is severely degraded, limiting its clinical applicability.

[0003] In recent years, deep learning (DL)-based methods have demonstrated great potential in low-dose computed tomography (LDCT) imaging. However, most DL-based LDCT reconstruction methods have specific conditional limitations and are designed without privacy considerations. In recent years, researchers have attempted to introduce federated learning (FL), a privacy-preserving distributed learning paradigm, into LDCT reconstruction to alleviate privacy concerns. However, directly using existing methods still struggles to cope with the noise variability introduced by the complexity and heterogeneity of the CT imaging process.

[0004] Therefore, to promote the further development of FL in LDCT reconstruction, a new CT image reconstruction method is urgently needed that can utilize the comprehensive physical information in the scan to break through the bottleneck of the existing framework and obtain better results. Summary of the Invention

[0005] The purpose of the present invention is to address the above-mentioned technical problems existing in the prior art and provide a personalized CT image reconstruction system based on dual physical drive, which integrates scanning parameters and radiological reports into imaging features to achieve personalized CT image reconstruction.

[0006] The present invention also provides a federated learning method for a personalized CT image reconstruction system based on dual physical drive. By training the system through the federated learning method, it is possible to protect personal privacy while achieving personalized image reconstruction.

[0007] The present invention adopts a two-layer strategy to alleviate the problem of data heterogeneity. First, a pre-trained large language model is used to generate detailed radiology reports. Then, a hypernetwork based on patient-level anatomical information is designed to generate anatomical modulation parameters of imaging parameters through radiology reports, so that they are seamlessly integrated into the imaging features to match the unique anatomical structure of each patient. At the same time, a hyperparameter network based on scan information is proposed to integrate scan features and personalize imaging features according to the scan protocol. This dual modulation strategy realizes personalized CT imaging at both the patient and scan levels, and is guided by the principles of the physical imaging process.

[0008] Furthermore, in existing work on personalized FL CT imaging, no research has focused on clients with unknown scanning protocols. This paper also proposes a protocol vector-quantization strategy (PVQS) that quantizes the unknown scanning protocol by matching its code vector with the closest known protocol and selecting the corresponding client-specific decoder for decoding.

[0009] Based on the above analysis, the present invention provides a personalized CT image reconstruction system based on dual physical drive, which includes:

[0010] An encoder, used for extracting features from the scanned image to obtain imaging features;

[0011] an anatomical information capture module, used to extract anatomical modulation parameters containing anatomical information based on radiology reports;

[0012] A scanning information capture module is used to capture the potential relationship between the scanning protocol and the noise distribution according to the scanning protocol, and obtain the scanning modulation parameters containing physical information;

[0013] A personalized modulation module is used to modulate the imaging features obtained by the encoder according to the anatomical modulation parameters and the scanning modulation parameters to obtain personalized imaging features;

[0014] The decoder is used to generate personalized CT images based on personalized imaging features.

[0015] In one implementation, the encoder and decoder are the same or different and can be any imaging network, such as RED-CNN, Uformer, U-Net, V-Net, HighResNet, etc.

[0016] In one implementation, the patient's unique anatomical structure has a significant impact on the noise distribution during imaging, and therefore becomes a key factor that must be considered. To address this issue, the present invention utilizes the help of MLLM, which has demonstrated strong performance in generating radiology reports when trained on large datasets. Specifically, the present invention uses a large language model MLLM to generate radiology reports from CT images, and uses the text feature output f of its text encoder to generate radiology reports. t ∈R 1×dAs a prompt to drive the anatomical information capture module hypernetwork, where d represents the feature dimension; the large language model can be miniGPT-Med (see Asma Alkhaldi, Raneem Alnajim, Layan Alabdullatef, Rawan Alyahya, Jun Chen, Deyao Zhu, Ahmed Alsinan, and Mohamed Elhoseiny. Minigpt-med: Large language model as a general interface for radiology diagnosis.arXiv preprint, arXiv:2407.04106, 2024), Hua Tuo GPT-Vision (see Chen J, Gui C, Ouyang R, et al. Huatuo GPT-vision, toward injecting medical visual knowledge into multimodal llms at scale[J].arXiv preprint arXiv:2406.19280, 2024), Llava-med (see Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. ArXiv, abs / 2306.00890, 2023) or MedKLIP (see Wu C, Zhang X, Zhang Y, et al. Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision.2023:21372-21383), etc. due to f t The dimension of is high, so the present invention first uses the average pooling layer and the linear layer to reduce the dimension, and then uses the multi-head attention layer and the linear layer to build the global connection of the features to obtain f an, i.e., the anatomical modulation parameters containing anatomical information. Therefore, the anatomical information capture module includes an average pooling layer and a multi-head attention layer, and a linear layer is provided after the average pooling layer and the multi-head attention layer.

[0017] In one implementation, in addition to anatomical differences affecting noise distribution, the physical imaging process shows that the noise distribution is largely determined by the scanning protocol, which mainly includes seven key physical parameters in the CT scanner, including the number of projection angles, the number of detector units, pixel length, detector unit length (here refers to the length of a single detector unit), the distance between the ray source and the rotation center, the distance between the detector unit and the rotation center, and the number of incident X-ray photons.

[0018] The present invention uses these seven scan protocol parameter vectors g to construct scan-level hints to capture the potential relationship between the scan protocol and the noise distribution, thereby hinting the imaging network. Since the parameter value ranges in g are different, all parameters in g are first normalized:

[0019]

[0020] in represents the jth element in the i-th scanning protocol g, and max(g j ) and min(g j ) represent the maximum and minimum values ​​of the j-th element of all scan protocols g, respectively.

[0021] Given the low-dimensional nature of physical parameters, the scanning information capture module used in the present invention includes a multi-layer perceptron (MLP) and two parallel linear layers arranged after the MLP. The MLP is composed of several linear layers arranged in sequence to capture and characterize the underlying physical information from the normalized scanning protocol parameter vector. Subsequently, two modulation parameter vectors are extracted through two parallel linear layers to form the scanning modulation parameters, thereby encapsulating the scanning-level personalized knowledge. The scanning information capture module can be expressed as:

[0022] α,β=H s (g;θ s );

[0023] Where α and β represent modulation parameters, which are used to modulate imaging features, g represents the scanning protocol parameter vector, and θ s Indicates the parameters of the scanning information capture module.

[0024] In one implementation, the anatomical modulation parameter f anContains anatomical specific information for different CT data. Given that this feature reflects anatomical specific details, the personalized modulation module first uses the anatomical modulation parameters to spatially modulate the imaging feature f , For personalization, the process can be expressed as:

[0025] f ana =f , ⊙Reshape(f an );

[0026] Among them, ⊙ represents the Hadamard product, Reshape() represents the feature dimension change function, f ana Represents the modulation imaging features modulated by anatomical information.

[0027] Then, the personalized modulation module modulates the modulation imaging feature modulated by the anatomical information using the scanning modulation parameters. This process can be expressed as:

[0028]

[0029] Among them, f 345 Represents personalized imaging features, Indicates multiplication operation along the number of channels.

[0030] It is worth noting that the personalized module designed in the present invention has no requirements on the architecture of the imaging network, so the dimensions of the personalized parameters should be fine-tuned according to the imaging network.

[0031] In one implementation, to overcome the performance degradation caused by the inability of existing federated CT imaging methods to handle unknown protocol data (new scanning protocols that have never been seen before) and to ensure robust performance in unknown protocols, the present invention introduces a new protocol vector-quantization strategy (PVQS). Although the patient's anatomical information remains largely consistent, the main noise difference between the unknown client and the existing client stems from the low-dose strategy. Therefore, PVQS is intended to enable the dual-physics-driven personalized CT image reconstruction system provided by the present invention to maintain its performance in diverse, unknown low-dose protocols.

[0032] Specifically, the scanning information capture module provided by the present invention also includes a protocol coding quantization component based on PVQS, which includes:

[0033] Scanning protocol code book, used to store the protocol coding vectors generated by the multi-layer perceptron for different existing scanning protocols;

[0034] A generation unit, used for generating an unknown protocol encoding vector by a multi-layer perceptron from an unknown protocol;

[0035] The matching unit is used to match the unknown protocol coding vector with the protocol coding vector in the scanned protocol coding book to obtain the protocol coding vector closest to the unknown protocol coding vector, and replace the unknown protocol coding vector with the closest protocol coding vector as the input of the two parallel linear layers.

[0036] The protocol coding quantization component quantizes an unknown protocol into a coding vector in the protocol codebook. The calculation process of the matching unit is:

[0037]

[0038] Among them, c >n and c i Represents the unknown code vector and the i-th point code vector in the scan protocol code book, i * Indicates the index of the closest protocol encoding vector.

[0039] After this, the unknown protocol is no longer used as the input of the scanning information driving the hypernetwork, but is directly used Replace unknown protocol encoding to generate personalized imaging features and use the corresponding decoder By projecting personalized imaging features back to the image domain, PVQS can effectively avoid potential distribution shifts in unknown data and maintain a consistent feature space across domains, thus achieving reliable and robust model performance in unknown protocols.

[0040] The present invention also provides a federated learning method suitable for personalized CT image reconstruction. Due to the differences in noise distribution between different protocols, it is challenging to use a single shared imaging network to adapt to all noise changes. In the LDCT imaging problem in the context of a federated school, the optimization problem can be divided into two main parts: imaging feature extraction and personalized projection. Specifically, for the feature extraction part, the present invention proposes an aggregated client encoder to capture robust and comprehensive imaging features. In order to make full use of the diverse data of each client, the present invention globally shares two super networks, the anatomical information capture module and the scanning information capture module, so that all clients can use them together. However, when using a single shared decoder to project imaging features from different domains into a consistent and clear representation, it is very difficult. Therefore, the present invention proposes a decoder corresponding to the client scanning protocol by locking the client decoder to effectively handle the differences in scanning protocols between clients, that is, the decoders of each client are not shared.

[0041] Based on the above analysis, the present invention provides a federated learning method for personalized CT image reconstruction, which includes the following steps:

[0042] S1. Each client trains the personalized CT image reconstruction system based on dual physical drive, and sends the trained encoder, anatomical information capture module and scanning information capture module parameters to the server;

[0043] S2: The server aggregates the parameters of the encoder, anatomical information capture module, and scan information capture module of each client, and feeds the aggregated result of the encoder, anatomical information capture module, and scan information capture module parameters back to each client;

[0044] Repeat the above steps S1-S2 until the number of iterations reaches the set upper limit.

[0045] In the above step S1, the unknown protocol code will be quantized into the scanned protocol codebook. Therefore, it is important to ensure that the protocol code is discriminative and supports comprehensive personalized information. To this end, the present invention introduces the orthogonal loss L C5tD , for i clients, it is defined as follows:

[0046]

[0047] Where K is the number of clients.

[0048] Meanwhile, the present invention adopts Mean Squared Error (MSE) as the imaging loss.

[0049] Therefore, the total loss used in the training of the personalized CT image reconstruction system based on dual physics drive in the i-th client can be expressed as:

[0050]

[0051] Where τ represents the temperature coefficient, represents the orthogonal loss of the i-th client, represents the imaging loss of the i-th client.

[0052] The parameters of the encoder, anatomical information capture module, scanning information capture module and decoder are optimized using the Adam algorithm based on the total loss.

[0053] In the above step S2, the server aggregates the parameters of the encoder, anatomical information capture module, and scan information capture module of each client according to the parameter mean of each client; in a specific implementation, the aggregation is performed according to the following formula:

[0054]

[0055] in, represents the parameters obtained by training the i-th client in the t-th round, x = E, H s 、Ha , corresponding to the encoder, scanning information capture module and anatomical information capture module respectively; |D i | represents the number of samples in the i-th client data set; |D| represents the sum of the number of samples in all client data sets; K represents the number of clients.

[0056] Therefore, in order to solve the heterogeneity problem, the present invention introduces the physical information of scanning and dissection as hints into the hyperparameter network to capture the information related to scanning and dissection. These hints come from the scanning protocol and the radiology report generated by the medical large language model (MLLM), respectively. Subsequently, these dual personalized scanning and anatomical imaging features are projected back to the image domain using personalized decoders for different clients. In addition, to meet the challenges of unseen data, the present invention proposes a new protocol vector-quantization strategy (PVQS) that quantizes the unseen scanning protocol into one of the codes in the scanning protocol codebook to ensure that the method of the present invention can maintain good performance on unseen scanning protocols. Moreover, the present invention also extracts robust imaging representations through the shared encoder of all clients of federated learning.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1) The personalized CT image reconstruction system based on dual physical drive provided by the present invention introduces a hypernetwork based on anatomical information and a hypernetwork based on scanning protocols into the system, realizing dual physical drive based on scanning parameters and anatomical information, thereby enabling personalized CT imaging at the scan level and patient level;

[0059] 2) This paper is the first to construct anatomical information prompts based on radiology reports generated by a large language model (MLLM) and apply them to CT images, thereby facilitating personalized CT image reconstruction;

[0060] 3) The present invention also proposes a protocol coding quantization strategy that can solve the encoding problem of unknown scanning protocols and ensure the stability of personalized CT image reconstruction;

[0061] 4) The present invention also proposes a federated learning method suitable for personalized CT image reconstruction, which improves the robust imaging performance by sharing and aggregating the parameters of the encoder, anatomical information capture module, and scanning information capture module, and improves the personalized imaging effect by locking the decoder. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1A schematic diagram of the structure of a personalized CT image reconstruction system based on dual physical drive provided in Example 1 of the present invention;

[0063] Figure 2 This is a schematic diagram of the anatomical information capture module structure;

[0064] Figure 3 This is a schematic diagram of the CT scanning principle;

[0065] Figure 4 This is a schematic diagram of the scanning information capture module structure;

[0066] Figure 5 Schematic diagram of the quantization strategy for protocol encoding;

[0067] Figure 6 Schematic diagram of the principle of federated learning method suitable for personalized CT image reconstruction;

[0068] Figure 7 The following are the CT image reconstruction results of different methods on different clients. The first to fifth rows correspond to clients Client#2, Client#3, Client#5, Client#6, and Client#7, respectively. The display window width of the first row is [-1024, 200] HU, and the bed width of other rows is [-160, 240] HU. DETAILED DESCRIPTION

[0069] The following will clearly and completely describe the technical solutions of various embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0070] Example 1

[0071] This embodiment provides a personalized CT image reconstruction system based on dual physical drive, such as Figure 1 As shown, it includes an encoder, an anatomical information capture module, a scanning information capture module, a personalized modulation module and a decoder.

[0072] In this embodiment, two physics-driven hypernetworks (anatomical information capture module and scan information capture module) are designed to capture scan-specific and anatomy-specific information and generate personalized imaging features in a dedicated personalized modulation module. The reconstruction process can be expressed as follows:

[0073]

[0074] Among them, f ,represents the imaging features of X-rays extracted by encoder E; is the corresponding normal dose CT image; H s and H a They are the hypernetworks driven by scanning information and anatomical information, namely the scanning information capture module and the anatomical information capture module, whose parameters are θ s and θ a ; g and f t denote the scan protocol vector and the radiology report features generated by MLLM, respectively; P denotes the personalized modulation module, and D denotes the decoder, whose parameters are θ W .

[0075] (1) Encoder and Decoder

[0076] The encoder (E) is used to extract features from the scanned image to obtain imaging features. The decoder (D) is used to generate a personalized CT image Y based on the personalized imaging features.

[0077] In this embodiment, both the encoder and the decoder use RED-CNN.

[0078] (2) Anatomical information capture module

[0079] The anatomical information capture module is used to extract anatomical modulation parameters containing anatomical information based on radiology reports.

[0080] In this embodiment, a large language model MLLM is used to generate radiology reports from CT images, and the text features of its text encoder are used to output f t ∈R 1×f As a prompt to drive the hypernetwork of the anatomical information capture module, d represents the feature dimension; the large language model is the pre-trained miniGPT-Med. For the model structure and training, please refer to Asma Alkhaldi, Raneem Alnajim, Layan Alabdullatef, Rawan Alyahya, Jun Chen, Deyao Zhu, Ahmed Alsinan, and Mohamed Elhoseiny. Minigpt-med: Large language model as a general interface for radiology diagnosis.arXiv preprint, arXiv:2407.04106, 2024.

[0081] In this embodiment, Figure 2As shown, the anatomical information capture module includes an average pooling layer and a multi-head attention layer, each of which is followed by a linear layer. The average pooling layer, the multi-head attention layer, and the linear layer all use conventional settings in the field.

[0082] Because f t The dimension is high. In this embodiment, the average pooling layer and the linear layer are first used to reduce the dimension, and then the multi-head attention layer and the linear layer are used to build the global connection of the features to obtain f an , that is, the anatomical modulation parameters containing anatomical information.

[0083] (3) Scanning information capture module

[0084] The scanning information capture module is used to capture the potential relationship between the scanning protocol and the noise distribution according to the scanning protocol, and obtain the scanning modulation parameters containing physical information.

[0085] In addition to anatomical differences affecting noise distribution, the physical imaging process shows that the noise distribution is largely determined by the scanning protocol, which mainly includes seven key physical parameters in the CT scanner, such as Figure 3 As shown, it includes the number of projection angles, the number of detector units, the length of the detector unit, the distance between the ray source and the rotation center, the distance between the detector unit and the rotation center, the number of incident X-ray photons, and the pixel length.

[0086] This embodiment uses these seven scan protocol parameter vectors g to construct scan-level hints to capture the potential relationship between the scan protocol and the noise distribution, thereby hinting the imaging network. Since the parameter value ranges in g are different, all parameters in g are first normalized:

[0087]

[0088] in, represents the jth element in the i-th scanning protocol g, max(g j ) and min(g j ) represent the maximum and minimum values ​​of the j-th element of all scan protocols g, respectively.

[0089] Given the low-dimensional nature of physical parameters, such as Figure 4As shown, the scanning information capture module used in this embodiment includes a multi-layer perceptron (MLP) and two parallel linear layers arranged after the MLP. The MLP is composed of three linear layers arranged in sequence, which are used to capture and represent the underlying physical information from the normalized scanning protocol parameter vector. Subsequently, two modulation parameter vectors are extracted through two parallel linear layers to form the scanning modulation parameters to encapsulate the scanning-level personalized knowledge. The scanning information capture module can be expressed as:

[0090] α,β=H s (g;θ s );

[0091] Where α and β represent modulation parameters, which are used to modulate the imaging characteristics.

[0092] (4) Personalized modulation module

[0093] The personalized modulation module is used to modulate the imaging features obtained by the encoder according to the anatomical modulation parameters and the scanning modulation parameters to obtain personalized imaging features.

[0094] Anatomical modulation parameter f an Contains anatomical specific information for different CT data. Given that this feature reflects anatomical specific details, the personalized modulation module first uses the anatomical modulation parameters to spatially modulate the imaging feature f , For personalization, the process can be expressed as:

[0095] f ana =f x ⊙Reshape(f an );

[0096] Among them, ⊙ represents the Hadamard product, Reshape() represents the feature dimension change function, f ana Represents the modulation imaging features modulated by anatomical information.

[0097] Then, the personalized modulation module uses the scanning modulation parameters to modulate the modulation imaging features modulated by the anatomical information. The process can be expressed as:

[0098]

[0099] Among them, f 345 Represents personalized imaging features, Indicates multiplication operation along the number of channels.

[0100] Example 2

[0101] This embodiment further improves the dual-physics-driven personalized CT image reconstruction system provided in Example 1. The primary difference from the dual-physics-driven personalized CT image reconstruction system provided in Example 1 lies in the scan information capture module, which also includes a PVQS-based protocol coding and quantization component. This PVQS-based protocol coding and quantization component is used to process unknown scan protocols from the client.

[0102] The PVQS-based protocol encodes quantization components, such as Figure 5 As shown, it includes a scanning protocol code book, a generation unit and a matching unit. The scanning protocol code book is used to store the protocol coding vectors generated by the multi-layer perceptron for different existing scanning protocols. The generation unit is used to generate an unknown protocol coding vector from the unknown protocol through the multi-layer perceptron. The matching unit is used to match the unknown protocol coding vector with the protocol coding vector in the scanning protocol code book to obtain the protocol coding vector that is closest to the unknown protocol coding vector, and replace the unknown protocol coding vector with the closest protocol coding vector as the input of two parallel linear layers; the corresponding scanning modulation parameters α and β are output through the two linear layers. For clients with unknown scanning protocols, the encoder, anatomical information capture module, and personalized modulation module of any client can be used; and for the decoder, the decoder of the client corresponding to the closest protocol coding vector can be used for decoding.

[0103] The calculation process of the above matching unit is:

[0104]

[0105] Among them, c un and c i Represents the unknown code vector and the i-th point code vector in the scan protocol code book, i * Indicates the index of the closest protocol encoding vector.

[0106] Example 3

[0107] This embodiment provides a federated learning method suitable for personalized CT image reconstruction, which can be used for training the personalized CT image reconstruction system based on dual physical drive in Example 1 or Example 2.

[0108] This embodiment provides a federated learning method suitable for personalized CT image reconstruction, which is as follows: Figure 6 As shown, the following steps are included:

[0109] S1 Each client trains the personalized CT image reconstruction system based on dual physical drive, and sends the trained encoder, anatomical information capture module and scanning information capture module parameters to the server.

[0110] Initially, the server first performs the encoder E and the anatomical information capture module H a and scanning information capture module H s The parameter θ R 、 and And the decoder of each client {D1,…,D K} Initialize and send these parameters to each client.

[0111] For the i-th client, according to the dataset D on the client i Training it includes the following steps:

[0112] (1) The encoder extracts features from each image sample to obtain imaging features f , .

[0113] (2) Generate text features f from image samples based on the large language model t The radiology report and the anatomical modulation parameter f are obtained through the anatomical information capture module an .

[0114] (3) Based on the normalized scanning protocol parameters, the scanning protocol encoding vector c is obtained through the multi-layer perceptron of the scanning information capture module i , and then two parallel linear layers are used to obtain the scanning modulation parameters α and β.

[0115] (4) The imaging features are modulated according to the anatomical modulation parameters and the scanning modulation parameters through the personalized modulation module. The process is as follows:

[0116] f ana =f x ⊙Reshape(f an );

[0117]

[0118] Among them, ⊙ represents the Hadamard product, Reshape() represents the feature dimension change function, f ana represents the modulation imaging feature modulated by anatomical information, f 345 Represents personalized imaging features, Indicates multiplication operation along the number of channels.

[0119] (5) The personalized imaging features are input into the decoder to generate a personalized CT image Y.

[0120] (6) Obtain the total loss.

[0121] According to the scanning protocol encoding vector c of the i-th client iThe orthogonal loss is obtained by scanning the protocol encoding vector of other clients and the calculation formula is as follows:

[0122]

[0123] Where K is the number of clients.

[0124] Mean Squared Error (MSE) is used as the imaging loss;

[0125] Therefore, the total loss used in the training of the i-th client can be expressed as:

[0126]

[0127] Wherein, τ represents the temperature coefficient, which is set to 0.00001 in this embodiment; represents the orthogonal loss of the i-th client; It represents the imaging loss of the i-th client, which is obtained by calculating the mean square error between the personalized CT image Y and the corresponding normal dose CT image.

[0128] (7) According to the total loss, the parameters (θ R 、 and ), optimized by Adam algorithm.

[0129] Repeat the above steps (1)-(7) until the set upper limit of the number of iterations is reached, completing the local training of the i-th client, and then send the trained encoder, anatomical information capture module and scanning information capture module parameters to the server.

[0130] The S2 server aggregates the parameters of the encoder, anatomical information capture module and scanning information capture module of each client, and feeds back the aggregated results of the encoder, anatomical information capture module and scanning information capture module parameters to each client.

[0131] In a specific implementation, the parameters of the current round encoder, anatomical information capture module, and scan information capture module are obtained by aggregating the parameters according to the following formula:

[0132]

[0133] in, Represents the aggregation result of all client parameters in round t; represents the parameters obtained by training the i-th client in the t-th round, x = E, H s 、H a , corresponding to the encoder, scanning information capture module and anatomical information capture module respectively; |Di | represents the number of samples in the i-th client data set; |D| represents the sum of the number of samples in all client data sets; K represents the number of clients.

[0134] The aggregated parameters of the current round are then sent to each client for the next round of training.

[0135] Repeat the above steps S1-S2 until the number of iterations reaches the set upper limit.

[0136] After the training is completed, each client can send the parameters of the system encoder, anatomical information capture module, scanning information capture module and decoder, as well as the scanning protocol coding vector to the server for storage. The scanning protocol coding vectors of all clients constitute the scanning protocol coding book, and each scanning protocol vector is associated with the corresponding encoder.

[0137] For clients with unknown scanning protocols, the parameters of the shared encoder, anatomical information capture module, and scanning information capture module can be first obtained through the server, and then the unknown protocol coding vector can be generated through the scanning information capture module generation unit according to the unknown scanning protocol, and the closest protocol coding vector can be obtained through the matching unit. The closest protocol coding vector is used to replace the unknown protocol coding vector, and the decoder corresponding to the closest protocol coding vector is used as the decoder of the client, thereby obtaining the personalized CT image reconstruction system of the client and performing personalized reconstruction of the input CT image.

[0138] Application Examples

[0139] The method proposed in this application example is implemented in PyTorch. The Adam algorithm (see Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014) is used for optimization, with a learning rate of 0.001. The number of communication rounds is set to 200, and the batch size is set to 20. The experimental environment includes an AMD Ryzen 7 5800X CPU and an NVIDIA GTX 3080Ti GPU.

[0140] The "2016 NIH-AAPM-Mayo Clinic Low Dose CT Challenge" dataset (see C McCollough. Tu-fg-207a-04: overview of the low dose ct grand challenge. Med. Phys., 43(6): 3759-3760, 2016), which contains 5,936 full-dose CT images from 10 patients, is used to evaluate the proposed method. In this application example, data from 8 patients are used for training and data from 2 patients are used for testing. To simulate real-world scenarios, this application example ensures that the training patients between different clients do not overlap. Two test patients are simulated using different protocols to form the test set.

[0141] First, the personalized CT image reconstruction systems of 8 clients are trained according to the federated learning method (SCAN-PhysFed) provided in Example 3, and then tested using the test set.

[0142] The scanning protocol parameters of the eight clients are shown in Table 1, where NV represents the number of projection angles, NDB represents the number of detector units, PL represents the pixel size, DBL represents the detector unit length, DSR represents the distance from the ray source to the rotation center, DDR represents the distance from the detector unit to the rotation center, and PN represents the number of incident X-ray photons.

[0143] Table 1 Scanning protocol parameters of different clients

[0144]

[0145] This application example compares the proposed SCAN-PhysFed with general and personalized FL methods. For global FL methods, FedAvg, FedProx, FedNova, MOON, FedDG, and FedKD are included. For personalized FL methods, FedPer, FedBN, FedMRI, HyperFed, and FedFDD are evaluated. To ensure fairness, the training settings for all methods are consistent.

[0146] Table 2 Performance comparison of different FL methods on multiple clients

[0147]

[0148] The present invention SCAN-PhysFed is compared with other FL methods. To ensure fairness, the same imaging network is used, namely RED-CNN, one of the most powerful models in CT imaging (see Hu Chen, Yi Zhang, MannudeepKKalra, et al. Low-dosect with a residual encoder-decoder convolutional neural network. IEEE Trans. Med. Imaging, 36(12): 2524-2535, 2017.524). The quantitative results are shown in Table 2. The best performance is achieved in most clients and outperforms other general and personalized FL methods in terms of average PSNR and SSIM. In addition, unlike other personalized methods that show instability in different clients (such as HyperFed in Client#2 / #7 and FedFDD in Client#4), the present invention maintains consistent performance in different protocols. It can also be seen that the sparse view protocols (Client#2, #4 and #6) pose a challenge to both general and personalized FL methods, and the performance of these methods in these protocols is significantly different from that of other protocols. However, by incorporating dual-level physical prior knowledge, the present invention is able to maintain robust performance.

[0149] Representative denoised slices generated by different methods are shown in Figure 2. Figure 7 As shown. The top three algorithms in general and personalized federated methods are compared. Representative clients with different radiation reduction strategies are selected: sparse view (clients #2 and #6) and low dose (clients #3, #5 and #7). It can be observed that other methods show obvious noise or artifacts in the sparse view protocol, and the details are not well preserved under the low dose strategy. However, the present invention effectively reduces noise and artifacts in the sparse view protocol, while preserving details in the low dose strategy. SCAN-PhysFed shows strong performance on slices at different anatomical locations by introducing anatomical physical information. Both quantitative and qualitative results demonstrate the effectiveness of combining physical information.

[0150] To demonstrate the effectiveness of our method for personalized reconstruction of CT images with unknown scanning protocols, this application example also tests on four clients that did not participate in training. For general FL methods, the shared model is directly evaluated on the unparticipating clients. However, most personalized methods ignore how they operate in unknown domains, so the performance of all local models is averaged here as a basic baseline.

[0151] The scanning protocol parameters of the clients that did not participate in the training are shown in Table 3.

[0152] Table 3 Scanning protocol parameters of clients not participating in training

[0153]

[0154] Table 4 Performance comparison of different FL methods on clients that did not participate in training

[0155]

[0156] The test results of different methods on clients that did not participate in training are shown in Table 4. It can be seen that the performance of the general FL method on clients that did not participate in training is only slightly lower than that of the clients that participated in training, while the personalized FL method shows a significant decline. This is because the local optimization in the personalized FL method mainly focuses on the specific data distribution of each client and cannot adapt well to other potential distributions, resulting in a significant performance drop on clients that did not participate in training. In PVQS, the present invention avoids the network directly mapping unknown protocols to modulate the imaging network, because if the unknown protocol differs significantly from the known protocol, it may cause the imaging network to collapse. Instead, the present invention uses a protocol codebook to quantize the unknown protocol, allowing SCAN-PhysFed to use known knowledge to better adapt to unknown distributions.

[0157] Therefore, to protect privacy and address heterogeneity in CT imaging, the present invention proposes a two-layer, physics-driven approach that leverages scan-level and anatomical-level cues to personalize CT imaging with the assistance of MLLM. This approach achieves both patient-level and scan-level personalization, and extensive results demonstrate that incorporating prior physical knowledge can significantly improve performance and effectively mitigate heterogeneity. Furthermore, the present invention proposes a new strategy, PVQS, for quantifying protocol encodings to maintain robust performance on unknown protocols. PVQS leverages known protocol encodings to prevent potential network crashes when unknown protocols differ significantly from known ones.

[0158] Those skilled in the art will appreciate that the embodiments herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art may make various other specific variations and combinations based on the technical teachings disclosed herein without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A personalized CT image reconstruction system based on dual physical drive, characterized in that: include: An encoder, used for extracting features from the scanned image to obtain imaging features; an anatomical information capture module, used to extract anatomical modulation parameters containing anatomical information based on radiology reports; A scanning information capture module is used to capture the potential relationship between the scanning protocol and the noise distribution according to the scanning protocol, and obtain the scanning modulation parameters containing physical information; A personalized modulation module is used to modulate the imaging features obtained by the encoder according to the anatomical modulation parameters and the scanning modulation parameters to obtain personalized imaging features; The decoder is used to generate personalized CT images based on personalized imaging features.

2. The personalized CT image reconstruction system based on dual physical drive according to claim 1, characterized in that: Generate radiology reports from CT images using a large language model (MLLM).

3. The personalized CT image reconstruction system based on dual physical drive according to claim 2, characterized in that: The anatomical information capture module includes an average pooling layer and a multi-head attention layer, and a linear layer is set after the average pooling layer and the multi-head attention layer.

4. The personalized CT image reconstruction system based on dual physical drive according to claim 1, characterized in that: The scanning protocol mainly includes the number of projection angles, the number of detector units, the pixel length, the detector unit length, the distance between the ray source and the rotation center, the distance between the detector unit and the rotation center, and the number of photons of the incident X-rays; the normalized results of these seven scanning protocol parameter vectors are used to construct the scanning level prompt.

5. The personalized CT image reconstruction system based on dual physical drive according to claim 4, characterized in that: The scanning information capture module includes a multi-layer perceptron and two parallel linear layers arranged after the multi-layer perceptron; the multi-layer perceptron is composed of a plurality of linear layers arranged in sequence.

6. The personalized CT image reconstruction system based on dual physical drive according to any one of claims 1 to 5, characterized in that: The personalized modulation module first uses the anatomical modulation parameters to spatially modulate the imaging feature f X For personalization, the process can be expressed as: f ana =f X ⊙Reshape(f an ); Among them, f X Represents imaging characteristics; f an represents the anatomical modulation parameter; ⊙ represents the Hadamard product; Reshape() represents the feature dimension change function; f ana represents the modulation imaging feature modulated by anatomical information; Then, the personalized modulation module modulates the modulation imaging feature modulated by the anatomical information using the scanning modulation parameters. The process is expressed as follows: Among them, f per Indicates personalized imaging features; Indicates the multiplication operation along the channel number; α and β represent the scanning modulation parameters.

7. The personalized CT image reconstruction system based on dual physical drive according to claim 6, characterized in that: The scanning information capture module also includes a protocol coding quantization component, which includes: Scanning protocol code book, used to store the protocol coding vectors generated by the multi-layer perceptron for different existing scanning protocols; A generation unit, used for generating an unknown protocol encoding vector by a multi-layer perceptron from an unknown protocol; The matching unit is used to match the unknown protocol coding vector with the protocol coding vector in the scanned protocol coding book to obtain the protocol coding vector closest to the unknown protocol coding vector, and replace the unknown protocol coding vector with the closest protocol coding vector as the input of the two parallel linear layers.

8. A federated learning method for personalized CT image reconstruction, characterized in that: The following steps are involved: S1. Each client trains the personalized CT image reconstruction system based on dual physical drive according to any one of claims 1 to 7, and sends the trained encoder, anatomical information capture module, and scanning information capture module parameters to the server; S2: The server aggregates the parameters of the encoder, anatomical information capture module, and scan information capture module of each client, and feeds the aggregated result of the encoder, anatomical information capture module, and scan information capture module parameters back to each client; Repeat the above steps S1-S2 until the number of iterations reaches the set upper limit.

9. The federated learning method for personalized CT image reconstruction according to claim 8, characterized in that: The total loss used in the training of the personalized CT image reconstruction system based on dual physics drive in the i-th client is expressed as: Where τ represents the temperature coefficient, represents the orthogonal loss of the i-th client, represents the imaging loss of the i-th client, and K represents the amount of client data.

10. The federated learning method for personalized CT image reconstruction according to claim 8, characterized in that: In step S2, the server aggregates the parameters of each client's encoder, anatomical information capture module and scanning information capture module according to the parameter mean of each client.

Images