Training methods and apparatus for image reconstruction models; image reconstruction methods and apparatus

By introducing modality invariance and specific feature extraction modules into the image reconstruction model, and combining them with federated learning techniques, the problem of poor training results caused by modal differences in image data among medical institutions is solved, thereby improving the quality of image reconstruction.

CN117012341BActive Publication Date: 2026-11-17腾讯医疗健康(深圳)有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211180487.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-09-26
Publication Date
2026-11-17
Estimated Expiration
2042-09-26

AI Technical Summary

Technical Problem

In the medical field, the difference in modalities of image data collected by different medical institutions leads to poor training results of image reconstruction models. Existing technologies are unable to effectively solve the problem of insufficient sample data, which affects the training effect of the models.

Method used

By introducing a first coding module to extract mode-invariant features and a second coding module to extract mode-specific features in the image reconstruction model, and performing feature fusion, the loss function is calculated to adjust the model parameters. The model is then trained using federated learning techniques until the parameters converge.

Benefits of technology

It improves the quality of image reconstruction models, solves the domain shift problem caused by modal differences between training samples from different terminals, and enhances the quality of image reconstruction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117012341B_ABST
    Figure CN117012341B_ABST
Patent Text Reader

Abstract

This application discloses a training method and apparatus for an image reconstruction model, as well as an image reconstruction method and apparatus. The method extracts modality-invariant features and modality-specific features from training sample images; fuses the modality-invariant and modality-specific features to obtain fused features; calculates a first loss based on the difference between the decoding result of the fused features and the training sample images; adjusts the model parameters of the model to be trained based on the first loss, and sends the adjusted model parameters to a server; receives the average model parameters returned by the server, and updates the model parameters of the model to be trained based on the average model parameters; and repeats the training process until the parameters of the model to be trained converge. This method can improve the model quality of the trained image reconstruction model, thereby improving the image reconstruction quality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a training method and apparatus for an image reconstruction model, and an image reconstruction method and apparatus. Background Technology

[0002] In recent years, image reconstruction technology has been widely used in the medical field. By training image reconstruction models based on artificial intelligence technology, the efficiency and quality of image reconstruction can be greatly improved.

[0003] In the medical field, patient medical image data is considered private and generally cannot be shared, often leading to insufficient sample data when training image reconstruction models. This results in suboptimal training performance. Federated learning technology, however, allows multiple medical institutions to jointly train image reconstruction models without sharing data, effectively addressing the dependence of image reconstruction models on large datasets and improving training efficiency.

[0004] However, different medical institutions may collect image data of different modalities when acquiring images, and the modal differences in training sample data from different terminals lead to poor training results for image reconstruction models. Summary of the Invention

[0005] This application provides a training method and apparatus for an image reconstruction model, and an image reconstruction method and apparatus. This method can improve the quality of the trained model, thereby improving the reconstruction quality of the image.

[0006] The first aspect of this application provides a method for training an image reconstruction model, the method comprising:

[0007] The training sample image is extracted based on the first encoding module to extract modality invariant features, and the training sample image is extracted based on the second encoding module to extract modality specific features. The model to be trained includes the first encoding module and the decoding module.

[0008] The modality-invariant features and the modality-specific features are fused to obtain fused features, and a first loss is calculated based on the difference between the decoding result and the training sample image. The decoding result is the result obtained by the decoding module from decoding the fused features.

[0009] The model parameters of the model to be trained are adjusted based on the first loss, and the adjusted model parameters are sent to the server;

[0010] Receive the average model parameters returned by the server, and update the model parameters of the model to be trained based on the average model parameters;

[0011] The process of repeatedly acquiring training sample images and updating the model parameters of the model to be trained based on the acquired training sample images is repeated until the parameters of the model to be trained converge.

[0012] Accordingly, a second aspect of this application provides a training apparatus for an image reconstruction model, the apparatus comprising:

[0013] An extraction unit is used to extract modality invariant features of training sample images based on a first encoding module, and to extract modality specific features of the training sample images based on a second encoding module. The model to be trained includes the first encoding module and a decoding module.

[0014] The computation unit is used to perform feature fusion on the mode invariance features and the mode specific features to obtain fused features, and to calculate a first loss based on the difference between the decoding result and the training sample image, wherein the decoding result is the result obtained by the decoding module from decoding the fused features;

[0015] An adjustment unit is used to adjust the model parameters of the model to be trained based on the first loss, and send the adjusted model parameters to the server;

[0016] An update unit is used to receive the average model parameters returned by the server and update the model parameters of the model to be trained based on the average model parameters.

[0017] The training unit is used to repeatedly execute the process of reacquiring training sample images and updating the model parameters of the model to be trained based on the acquired training sample images, until the parameters of the model to be trained converge.

[0018] In some embodiments, the extraction unit includes:

[0019] The extraction subunit is used to extract first image features of the training sample image based on the first encoding module in the model to be trained, and to extract second image features of the training sample image based on the second encoding module.

[0020] The first sending subunit is used to send the second image features and the modality label data corresponding to the training sample image to the server, and to receive the classification gradient returned by the server based on the second image features and the modality label data;

[0021] A subunit is defined for determining a second loss based on the similarity relationship between the first image features and the second image features;

[0022] The adjustment unit can also be used for:

[0023] The model parameters of the model to be trained are adjusted based on the first loss, the second loss, and the classification gradient.

[0024] In some embodiments, the model training apparatus provided in this application further includes:

[0025] A detection subunit is used to perform encrypted instance alignment detection on the training sample images;

[0026] The second sending unit is used to send the first image features to the server when the training sample image is detected to be a vertical sample, and to receive the consistency gradient returned by the server.

[0027] The adjustment unit can also be used for:

[0028] The model parameters of the model to be trained are adjusted based on the first loss, the second loss, the classification gradient, and the consistency gradient.

[0029] In some embodiments, the model training apparatus provided in this application further includes:

[0030] A sub-unit is used to obtain the first loss, the second loss, and the weight coefficients of the classification gradient when the training sample image is detected to be a parallel sample;

[0031] The adjustment subunit is used to adjust the model parameters of the model to be trained based on the weight coefficients, the first loss, the second loss, and the classification gradient.

[0032] In some embodiments, the computing unit includes:

[0033] The first computational subunit is used to add the mode-invariant feature and the mode-specific feature to obtain the fused feature;

[0034] A decoding subunit is used to decode the fused features according to the decoding module to obtain a decoding result;

[0035] The second computational subunit is used to calculate a first loss based on the difference between the decoding result and the training sample image.

[0036] A third aspect of this application also provides an image reconstruction method, the method comprising:

[0037] Obtain the target image to be reconstructed;

[0038] The target image to be reconstructed is reconstructed based on a preset image reconstruction model to obtain a reconstructed image. The preset image reconstruction model is a model trained based on the image reconstruction model training method provided in the first aspect.

[0039] Accordingly, a fourth aspect of this application also provides an image reconstruction apparatus, the apparatus comprising:

[0040] The acquisition unit is used to acquire the target image to be reconstructed;

[0041] The reconstruction unit is used to reconstruct the target image to be reconstructed based on a preset image reconstruction model to obtain the reconstructed image. The preset image reconstruction model is a model trained based on the image reconstruction model training method provided in the first aspect.

[0042] The fifth aspect of this application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in the training method of the image reconstruction model provided in the first aspect of this application or the image reconstruction method provided in the third aspect of this application.

[0043] The sixth aspect of this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the training method for the image reconstruction model provided in the first aspect of this application or the image reconstruction method provided in the third aspect of this application.

[0044] A seventh aspect of this application provides a computer program product or computer program, the computer program product or computer program including computer instructions stored in a storage medium. A processor of a computer device reads the computer instructions from the storage medium and executes the computer instructions, causing the computer device to perform steps in the training method of the image reconstruction model provided in the first aspect or the image reconstruction method provided in the third aspect of this application.

[0045] The image reconstruction model training method provided in this application extracts modality-invariant features from training sample images based on a first encoding module and modality-specific features from training sample images based on a second encoding module. The model to be trained includes a first encoding module and a decoding module. The modality-invariant and modality-specific features are fused to obtain fused features. A first loss is calculated based on the difference between the decoding result and the training sample images. The decoding result is the result obtained by the decoding module from decoding the fused features. The model parameters of the model to be trained are adjusted based on the first loss, and the adjusted model parameters are sent to a server. The average model parameters returned by the server are received, and the model parameters of the model to be trained are updated based on the average model parameters. The process of repeatedly acquiring training sample images and updating the model parameters of the model to be trained based on the acquired training sample images is repeated until the model parameters converge. Therefore, the image reconstruction model training method provided in this application, by decoupling the modality-invariant features of the training sample images, enables training samples from different terminals to obtain a consistent feature distribution. This improves the model quality of the trained model and, consequently, the reconstruction quality of the image. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This application provides a schematic diagram of the model training scenario;

[0048] Figure 2 This is a flowchart illustrating the training method for the image reconstruction model provided in this application;

[0049] Figure 3 This is another flowchart illustrating the training method for the image reconstruction model provided in this application;

[0050] Figure 4 This is another flowchart illustrating the training method for the image reconstruction model provided in this application;

[0051] Figure 5 This is a schematic diagram of the model training device provided in this application;

[0052] Figure 6 This is a flowchart illustrating the image reconstruction method provided in this application;

[0053] Figure 7This is a schematic diagram of the image reconstruction apparatus provided in this application;

[0054] Figure 8 This is a schematic diagram of the structure of the computer device provided in this application. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0056] This invention provides a training method and apparatus for an image reconstruction model, as well as an image reconstruction method and apparatus. The training method for the image reconstruction model can be used in a model training apparatus. This model training apparatus can be integrated into a computer device, which can be a terminal. The terminal can be a mobile phone, tablet computer, laptop computer, smart TV, wearable smart device, personal computer (PC), etc. The server mentioned in this application can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN) services, and big data and artificial intelligence platforms.

[0057] Please see Figure 1This is a schematic diagram of the model training scenario provided in this application. As shown in the figure, the model training system includes multiple distributed terminals B (only two are shown in the figure) and a server A. The distributed terminals B only acquire local training sample data, and the sample data acquired by multiple distributed terminals B are not shared. Each distributed terminal B deploys a model to be trained, which includes a first encoding module and a decoding module. In addition, a second encoding module is additionally set in the distributed terminal B. During model training, the distributed terminals B extract modality-invariant features from the acquired training sample images based on the first encoding module, and modality-specific features from the training sample images based on the second encoding module. The distributed terminals B perform feature fusion on the modality-invariant features and modality-specific features to obtain fused features, and calculate a first loss based on the difference between the decoding result and the training sample images. The decoding result is the result obtained by the decoding module decoding the fused features. Terminal B adjusts the model parameters of the model to be trained based on the first loss and sends the adjusted model parameters to server A. Terminal B receives the average model parameters returned by server A and updates the model parameters of the model to be trained based on the average model parameters. Furthermore, terminal B repeatedly executes the process of reacquiring training sample images and updating the model parameters of the model to be trained based on the acquired training sample images, until the model parameters of the model to be trained converge, thus completing the training of the model to be trained.

[0058] It should be noted that, Figure 1 The illustrated model training scenario is merely an example. The model training scenario described in this application is intended to more clearly illustrate the technical solution of this application and does not constitute a limitation on the technical solution provided in this application. Those skilled in the art will understand that, with the evolution of model training and the emergence of new business scenarios, the technical solution provided in this application is equally applicable to similar technical problems.

[0059] The implementation scenarios described above will be explained in detail below.

[0060] In related technologies, federated learning-based magnetic resonance imaging (MRI) reconstruction model training generally assumes that different clients possess MRI data of the same modality. However, in real-world medical applications, hospitals typically need to collect MRI data of multiple modalities for diagnostic accuracy. Specifically, these include T1-weighted imaging (T1WI) and T2-weighted imaging (T2WI) modalities. T1WI modality emphasizes the longitudinal relaxation (physical property, the process of returning from one state to an equilibrium state) differences in tissue, while minimizing the influence of other tissue characteristics such as lateral relaxation on the image. MRI images that primarily reflect differences in T1 values ​​between tissues are T1-weighted images; those that primarily reflect differences in T2 values ​​are T2-weighted images. When training image reconstruction models based on images of different modalities, domain shift issues arise, leading to poor model training results.

[0061] To address the problem of modal differences in training sample images from different terminals leading to domain shift and consequently poor model training performance in existing image reconstruction models, this application provides a training method for image reconstruction models to improve the quality of the trained model and thus enhance image reconstruction quality. The training method for the image reconstruction model provided in this application is described in detail below.

[0062] This application will describe the embodiments from the perspective of a model training device, which can be integrated into a computer device. The computer device can be a terminal, and the terminal can be any one of multiple distributed terminals. For example... Figure 2 The diagram shown is a flowchart illustrating the training method for the image reconstruction model provided in this application. The method includes:

[0063] Step 101: Extract modality invariant features of training sample images based on the first encoding module, and extract modality specific features of training sample images based on the second encoding module.

[0064] In this embodiment, the image reconstruction model training method provided can specifically be a federated learning-based model training method. The model to be trained in this embodiment can be an image reconstruction model deployed in each distributed terminal, and the image reconstruction model specifically includes an encoding module and a decoding module. Image reconstruction models deployed in different distributed terminals can have different initial model parameters or the same initial model parameters.

[0065] As mentioned earlier, during model training using federated learning techniques, the training sample data obtained from different terminals exhibit modal differences, leading to inconsistent feature distributions and thus affecting the model's training performance. To improve model training effectiveness, it is necessary to address the issue of inconsistent feature distributions caused by modal differences between training samples. This can be achieved by designing the encoding module of the training model to extract only modality-invariant features from the training sample data; that is, decoupling modality-invariant features from the image features of the training samples for model training. This solves the domain shift problem caused by modal differences between training sample images from different terminals, thereby improving the image reconstruction capabilities of the trained model.

[0066] Therefore, an additional encoding module can be designed in the terminal. To distinguish it from the encoding module in the model under training, the encoding module in the model under training can be designated as the first encoding module, and the additional encoding module can be designated as the second encoding module. During model training, the parameters of the second encoding module are adjusted accordingly. However, during model usage, this additional second encoding module is not needed. Only the first encoding module and decoding module inherent in the model are used for image reconstruction.

[0067] During model training, each distributed terminal can acquire local training sample data, which includes multiple training sample images and modality label data for each image. Then, the terminal can extract modality-invariant features from the training sample images based on the first encoding module of the model to be trained, and extract modality-specific features based on an additionally configured second encoding module. Specifically, modality-invariant features refer to common image features among training sample images of different modalities; modality-specific features refer to the different image features among training sample images of different modalities.

[0068] In some embodiments, extracting mode-invariant features of training sample images based on a first encoding module and extracting mode-specific features of training sample images based on a second encoding module includes:

[0069] 1. Extract the first image features of the training sample images based on the first encoding module in the model to be trained, and extract the second image features of the training sample images based on the second encoding module;

[0070] 2. Send the second image features and the modality label data corresponding to the training sample images to the server, and receive the classification gradient returned by the server based on the second image features and modality label data.

[0071] 3. Determine the second loss based on the similarity relationship between the first image features and the second image features.

[0072] The ability of the first encoding module to extract modality-invariant features from training sample images can be assigned through a preset feature extraction logic or learned. Specifically, features can be extracted from the training sample images based on the first and second encoding modules respectively to obtain first image features and second image features.

[0073] Then, the terminal can send the second image features extracted by the second encoding module and the modality label data corresponding to the training sample images to the server. The server can be configured with a classifier, specifically a multilayer perceptron (MLP). The server inputs the received second image features into this classifier for classification, using the received modality label data as the classification label. Further, the server can construct a classification loss based on the cross-entropy between the classifier's output and the classification label, then determine the classification gradient based on this loss and return it to the corresponding terminal. Upon receiving the classification gradient, the terminal adjusts the parameters of the second encoding module based on the gradient, enabling the second encoding module to learn the ability to extract modality-specific features from images.

[0074] Furthermore, a loss function can be constructed based on the similarity relationship between the first and second image features. The training objective of this loss function is to minimize the similarity between the first and second image features. This loss function can be referred to as the second loss. Since the second image feature is a modality-specific feature extracted by the second encoding module, when the similarity between the first and second image features is sufficiently small, the first image feature can be determined to be an extracted modality-invariant feature. This allows the first encoding module to learn the ability to extract modality-invariant features from an image.

[0075] In this embodiment, an additional encoding module is set in the terminal, and an additional classification module is set in the server. The modality label data of the training sample images are used to train the additional encoding module to classify the modality of the images, that is, to train the additional encoding module to extract the modality-specific features of the images. Further, the image invariant feature extraction ability of the encoding module (first encoding module) in the training model (image reconstruction model) is trained with reference to the encoding module (second encoding module) with the ability to extract image modality-specific features. Specifically, the first encoding module can learn to extract image invariant features by maximizing the difference between the feature extraction capabilities of the first encoding module and the second encoding module, that is, by maximizing the difference between the first image features and the second image features. When the encoding module of the image reconstruction model learns to extract image invariant features, the problem of inconsistent feature distribution caused by image modality differences, which reduces the image reconstruction ability of the model, can be avoided, thereby improving the image reconstruction quality of the model.

[0076] Step 102: Perform feature fusion on modality invariant features and modality specific features to obtain fused features, and calculate the first loss based on the difference between the decoding result and the training sample image.

[0077] After extracting the modality-invariant and modality-specific features of the training sample images, these features can be further fused. It can be understood that both the modality-invariant and modality-specific features of the training sample images are image features of the training sample images, and they are complementary features. Therefore, the modality-invariant and modality-specific features of the training sample images can be fused first, and then the fused image features can be decoded. The difference between the decoded result and the training sample images is calculated to construct a loss function, which can be called the first loss. The optimization objective of the first loss is to minimize the difference between the decoded result and the training sample images, so that the decoding module learns the ability to decode accurate images based on complete image features, i.e., the decoding module learns the ability to reconstruct images based on image features.

[0078] In some embodiments, modality-invariant features and modality-specific features are fused to obtain fused features, and a first loss is calculated based on the difference between the decoding result of the fused features by the decoding module and the training sample image, including:

[0079] 1. Add the modality-invariant features and the modality-specific features to obtain the fused features;

[0080] 2. The fused features are decoded using the decoding module to obtain the decoding result;

[0081] 3. Calculate the first loss based on the difference between the decoding result and the training sample image.

[0082] In this embodiment, the fusion operation of modality-invariant features and modality-specific features of the training sample image can specifically be achieved by adding the modality-invariant features and the modality-specific features together. Since the modality-invariant features and modality-specific features of the training sample image are complementary image features, adding the two by their feature dimensions yields the complete image features of the training sample image.

[0083] Step 103: Adjust the model parameters of the model to be trained based on the first loss, and send the adjusted model parameters to the server.

[0084] In this embodiment, the model parameters of the model to be trained (image reconstruction model) can be adjusted based on local sample data, i.e., the model parameters of the model to be trained can be adjusted based on the first loss. This enables the encoding module of the image reconstruction model to learn the ability to extract modality-invariant features of the image, and enables the decoding module of the image reconstruction model to learn the ability to reconstruct the image based on the image features.

[0085] The premise for adjusting the model parameters of the model to be trained based on the first loss is that the second encoding module has learned the ability to extract image-specific features and the first encoding module has learned the ability to extract image-invariant features. However, in some embodiments, the capabilities of each module in the model need to be trained simultaneously. That is, after receiving the classification gradient returned by the server, determining the second loss based on the similarity relationship between the first and second image features, and determining the first loss based on the difference between the decoding result and the training sample image, the model parameters of the second encoding module and the model to be trained (including the parameters of the first encoding module and the decoding module) can be adjusted simultaneously based on the first loss, the second loss, and the classification gradient.

[0086] In some embodiments, the training method for the image reconstruction model provided in this application further includes:

[0087] 1. Perform encrypted instance alignment detection on the training sample images;

[0088] 2. When a training sample image is detected to be a vertical sample, the first image feature is sent to the server, and the consistency gradient returned by the server is received.

[0089] 3. Adjust the model parameters of the model to be trained based on the first loss, the second loss, and the classification gradient, including:

[0090] The model parameters of the model to be trained are adjusted based on the first loss, the second loss, the classification gradient, and the consistency gradient.

[0091] In this embodiment, a consistent feature distribution can be obtained not only through feature decoupling within the terminal, but also by further constraining the clients to unify the modality invariant representation of vertical samples. Specifically, encrypted instance alignment detection can be performed on training sample images acquired by different terminals. The encrypted instance alignment detection method is a technique in vertical federated learning used to find common training sample IDs among different participants. This method not only finds common training sample IDs among different participants but also ensures that other sample IDs are not leaked to each other, thus guaranteeing data security. Vertical federated learning refers to a federated learning method where some datasets come from the same sample space but have different feature spaces. In contrast to vertical federated learning is horizontal federated learning, which refers to data from multiple clients coming from different sample spaces but sharing the same feature space.

[0092] In this embodiment, an encrypted instance alignment detection method is used to detect vertical samples in the terminal. Vertical samples refer to samples from the same object; when the training sample images are MRI images, vertical samples specifically refer to samples from the same patient. Parallel samples, in contrast to vertical samples, specifically refer to samples from different objects.

[0093] When a training sample is detected as a vertical sample, the first image feature of the training sample image can be sent to the server. Here, the first image feature is the mode-invariant feature obtained by the first encoder from feature extraction of the image. Upon receiving the mode-invariant features for vertical samples from each terminal, the server can calculate the consistency loss of each first image feature based on the difference between each first image feature and other first image features. For example, when terminal A1 sends its corresponding first image feature a1 to the server, terminal A2 sends its corresponding first image feature a2 to the server, terminal A3 sends its corresponding first image feature a3 to the server, and so on, terminal An sends its corresponding first image feature an to the server, the server can then calculate the consistency loss of the first image feature a1 based on the difference between it and other first image features ai (i = 2, 3...n). Further, a consistency gradient can be determined based on the consistency loss and backpropagated to terminal A1. After receiving the consistency gradient returned by the server, terminal A1 can adjust the model parameters of the model to be trained based on the aforementioned first loss, second loss, classification gradient, and consistency gradient. Therefore, by maximizing the similarity of modality-invariant features among clients, the consistency of modality-invariant features can be maintained, thereby further improving the quality of the trained model and thus further improving the quality of image reconstruction.

[0094] In some embodiments, the training method for the image reconstruction model provided in this application further includes:

[0095] A. When a training sample image is detected to be a parallel sample, obtain the weight coefficients of the first loss, the second loss, and the classification gradient.

[0096] B. Adjust the model parameters of the model to be trained based on the weight coefficients, the first loss, the second loss, and the classification gradient.

[0097] In this embodiment of the application, when the training sample image is determined to be a parallel sample (i.e., a vertical sample) by performing encrypted instance alignment detection, the weight coefficients of the first loss, the second loss, and the classification gradient can be further obtained. Then, the backpropagation gradient during model training can be adjusted based on the corresponding weight coefficients to control the impact of different losses on model training, thereby improving the model quality of the trained model.

[0098] In the federated learning-based model training process, after a terminal updates the model parameters based on local training sample data, the updated model parameters can be uploaded to the server for federated learning. Specifically, after receiving updated model parameters from multiple terminals, the server can average the model parameters sent by multiple terminals to obtain the global model parameters.

[0099] Step 104: Receive the average model parameters returned by the server, and update the model parameters of the model to be trained based on the average model parameters.

[0100] In this process, after the server performs averaging on the model parameters sent by the terminals to obtain the model parameters of the global model, it can further send the average model parameters to each terminal. Then, each terminal updates the model parameters of its local model based on the global model parameters (i.e., the average model parameters) returned by the server.

[0101] Step 105: Repeatedly execute the process of reacquiring training sample images and updating the model parameters of the model to be trained based on the acquired training sample images until the model parameters of the model to be trained converge.

[0102] Steps 101 to 104 constitute a loop for model training, executed based on one of multiple training sample images. Further, the above process can be repeated: acquiring training sample images, extracting modality-invariant and modality-specific features from them, fusing these features into a fused feature, calculating the loss based on the difference between the decoding result of the fused feature and the training sample image, adjusting the model parameters based on the calculated loss, sending the adjusted model parameters to the server for averaging, and then updating the local model parameters based on the global model parameters returned by the server. This loop is executed until the model parameters converge, completing the training of the model.

[0103] As described above, the image reconstruction model training method provided in this application embodiment extracts modality-invariant features from training sample images based on a first encoding module and modality-specific features from training sample images based on a second encoding module. The model to be trained includes a first encoding module and a decoding module. The modality-invariant features and modality-specific features are fused to obtain fused features, and a first loss is calculated based on the difference between the decoding result and the training sample images. The decoding result is the result obtained by the decoding module from decoding the fused features. The model parameters of the model to be trained are adjusted based on the first loss, and the adjusted model parameters are sent to the server. The average model parameters returned by the server are received, and the model parameters of the model to be trained are updated based on the average model parameters. The process of repeatedly acquiring training sample images and updating the model parameters of the model to be trained based on the acquired training sample images is executed until the model parameters converge.

[0104] Therefore, the image reconstruction model training method provided in this application decouples the modality-invariant features of the training sample images, enabling training samples from different terminals to obtain a consistent feature distribution. This improves the quality of the trained model, and consequently, the reconstruction quality of the image.

[0105] Accordingly, the embodiments of this application will further describe in detail the training method of the image reconstruction model provided in this application. For example... Figure 3 The diagram shown is another flowchart illustrating the training method for the image reconstruction model provided in this application. The method includes:

[0106] Step 201: The terminal acquires training sample images and extracts the first image features of the training sample images from the first encoder in the image reconstruction model to be trained.

[0107] First, before training the image reconstruction model, training sample data can be acquired. Specifically, this application employs a federated learning-based image reconstruction model training method, thus requiring multiple distributed terminals to acquire training sample data locally. The training sample images used in this application can be various types of images with multiple modalities. In this embodiment, MRI-generated MRI images are used as an example to illustrate the training method of the image reconstruction model provided in this application. As mentioned earlier, MRI images have two modalities: T1WI and T2WI. That is, each training sample image in the acquired training sample data has a corresponding modality identifier. The image reconstruction model to be trained in this application specifically includes an encoder and a decoder. In addition to the model to be trained, this application also sets up an encoder to assist in modal similarity feature extraction. To avoid confusion between the two, the encoder in the image reconstruction model can be named the first encoder, and the additional encoder can be called the second encoder.

[0108] In this embodiment, "terminal" refers to each distributed terminal. Since the execution steps and interaction steps with the server are the same for different distributed terminals, a single terminal is used as an example for description. It is understood that the operations performed in other distributed terminals and the interaction process with the server are consistent with those of the example terminal and will not be elaborated upon.

[0109] After the terminal acquires training sample data containing multiple training sample images and corresponding modality identifiers for each training sample image, it can first arbitrarily select one training sample image from the training sample data. Then, based on the first encoder, it performs image feature extraction on the selected training sample image and projects the extracted features into the latent space to obtain the first image features. To avoid inconsistencies in feature distribution caused by inter-modal domain shifts, which could reduce the overall model performance, the first encoder needs to learn the ability to extract modality-invariant features from the training sample images, thus making the first image features approximate modality-invariant features.

[0110] Step 202: The terminal extracts the second image features of the training sample images based on the second encoder.

[0111] To enable the first encoder to learn the ability to extract mode-invariant features from training sample images, this application additionally includes a second encoder to assist the first encoder in its learning process. This second encoder can first learn to extract mode-specific features from the training sample images, and then maximize the difference between the features extracted by the second encoder and those extracted by the first encoder. This allows the first encoder to learn the ability to extract mode-invariant features from the training sample images.

[0112] Specifically, a second encoder can be used to extract image features from the training sample images, and then the extracted features can be projected into the latent space to obtain the second image features. The details are as follows:

[0113]

[0114] Where x represents the training sample image, i.e., the input image; This represents the second encoder; Projection represents the operation of projecting features into the latent space, including global average pooling and unrolling. This represents the second image feature.

[0115] Thus, the first image feature mentioned above can be represented as:

[0116]

[0117] in, Indicates the first image feature; This indicates the first encoder.

[0118] Step 203: The terminal sends the second image features and the modality label data corresponding to the training sample images to the server.

[0119] To enable the second encoder to learn the ability to extract modality-specific features from training sample images, it can be trained in a supervised manner using the modality label data of each training sample image. Specifically, a classifier C can be introduced into the server to perform an auxiliary classification task; this classifier can be an MLP.

[0120] Then, the terminal can send the second image features and the modality label data corresponding to the training sample image to the server for classification training.

[0121] Step 204: The server uses a classifier to classify the second image features, calculates the first loss based on the classification result and modality label data, and returns the first loss to the terminal.

[0122] When the server receives the second image features and the corresponding modality label data, it can input the second image features into the classifier C for classification to obtain the classification result. Then, by minimizing the difference between the classification result and the modality label data, the second encoder can learn to extract image modality-specific features.

[0123] Specifically, the first loss, i.e., the classification loss, can be calculated based on the classification results and the modality label data:

[0124]

[0125] in, For the classification results, V k For modal label data, CE is the cross-entropy operator. This is the first loss.

[0126] The server can then determine the gradient based on the first loss and backpropagate the gradient to the terminal, allowing the terminal to adjust the parameters of the second encoder. Alternatively, the server can return the first loss to the terminal, enabling the terminal to determine the gradient based on the first loss and adjust the parameters of the second encoder.

[0127] Step 205: The terminal calculates the second loss based on the first image features and the second image features.

[0128] Furthermore, after the second encoder has learned the ability to extract modality-specific features of an image, it can further maximize the difference between the first image features and the second image features to enable the first encoder to learn the ability to extract modality-invariant features of the image. Maximizing the difference between the first image features and the second image features means minimizing the similarity between them. Specifically, the terminal can construct the following second loss based on the first image features and the second image features:

[0129]

[0130] Where ||·||1 represents the calculation of the l1 loss function. This indicates the second loss.

[0131] The terminal can then determine the gradient based on the second loss and adjust the parameters of the first encoder.

[0132] Step 206: The terminal fuses the first image features with the second image features to obtain fused features, and decodes the fused features based on the decoder in the image reconstruction model to obtain the reconstructed image.

[0133] Furthermore, the decoder in the image reconstruction model also needs to be trained. Since the first encoder and the second encoder extract the mode-invariant features and mode-specific features of the training sample images, respectively, the image features extracted by the first encoder and the image features extracted by the second encoder can be fused. Specifically, the first image features and the second image features can be added together to obtain fused features that can completely describe the training sample images.

[0134] Then, the fused features are decoded based on the decoder in the image reconstruction model to obtain the reconstructed image.

[0135] Step 207: The computer device calculates the third loss based on the reconstructed image and the training sample images.

[0136] Furthermore, by minimizing the difference between the reconstructed image and the ground truth image, i.e. the training sample image, the decoder can learn to reconstruct images based on image features.

[0137] Specifically, a third loss (reconstruction loss) can be constructed based on the reconstructed image and the training sample images:

[0138]

[0139] in, As the third loss, y represents the decoder and y represents the training sample image.

[0140] Then, the terminal can perform gradient backpropagation based on the third loss to adjust the parameters of the decoder.

[0141] like Figure 4 The diagram shows another flowchart of the training method for the image reconstruction model provided in this application. As shown, in terminal 10, when a training sample image 11 is acquired, the training sample image 11 can be processed by the first encoder 12 to extract features and project them into the latent space, resulting in first image features 13. Terminal 10 sends the first image features 13 to server 20. Server 20 inputs the first image features into classifier 21 for classification and determines a first loss 22 based on the classification result and modality label data. Simultaneously, the training sample image 11 can be processed by the second encoder 14 to extract features and project them into the latent space, resulting in second image features 15. Then, a second loss 16 can be calculated based on the first image features 13 and the second image features 15. Further, the first image features 13 and the second image features 15 can be fused to obtain fused features 17. The fused features 17 are input into decoder 18 to obtain an output image, and a third loss 19 is calculated based on the output image and the training sample image.

[0142] Step 208: When the training sample images are parallel samples, the terminal adjusts the parameters of the image reconstruction model based on the first loss, the second loss, and the third loss.

[0143] In this embodiment of the application, to further reduce the impact of pattern differences in training samples on the model training process, consistency regularization can be applied to the modal invariance features extracted by the first encoder for vertical samples in different terminals, thereby reducing the differences between the modal invariance features extracted by the first encoder for vertical samples in different terminals.

[0144] Therefore, in this embodiment of the application, the training sample data obtained by different terminals can be detected first by the encrypted instance alignment method to determine the vertical and parallel samples in the training sample data obtained by different terminals.

[0145] When the training sample images are determined to be parallel samples, the parameters of the image reconstruction model are adjusted based on the first loss, second loss, and third loss mentioned above. The image reconstruction model includes a first encoder and a decoder. It is understood that while adjusting the parameters of the image reconstruction model, the parameters of the second encoder and classifier are also adjusted. However, the second encoder and classifier only participate in the model training and do not affect the model's usage.

[0146] Specifically, when parallel samples are detected in the training sample images, the loss function for adjusting the model parameters can be determined first based on the first loss, the second loss, and the third loss:

[0147]

[0148] Here, μ1 and μ2 are weight hyperparameters used to control the degree of influence of different losses.

[0149] Once the loss function for adjusting the model parameters is determined, the model parameters can be adjusted based on that loss function.

[0150] Step 209: When the training sample image is a vertical sample, the terminal sends the first image features to the server.

[0151] When a training sample image is detected as a vertical sample, the terminal sends the mode invariant features (first image features) extracted by the first encoder to the server. After receiving multiple mode invariant features extracted from vertical samples by different terminals, the server makes the mode invariant feature extraction capability of the first encoder more accurate by minimizing the differences between the mode invariant features extracted by different terminals.

[0152] Step 210: The server calculates the fourth loss based on the first image features and the image features sent by other terminals, and returns the fourth loss to the terminal.

[0153] Specifically, the server can calculate the fourth loss (cross loss) for each terminal based on the multiple modality-invariant features extracted from the received vertical samples:

[0154]

[0155] Where K represents the number of terminals. This represents the cross-loss corresponding to the k-th terminal. This represents the mode invariance feature extracted from the k-th terminal. This represents the modality invariance feature extracted from the i-th terminal.

[0156] After determining the cross loss for each terminal, the server can further determine the corresponding gradient and backpropagate it to the terminal, or the server can return the cross loss to the corresponding terminal.

[0157] Step 211: The terminal adjusts the parameters of the image reconstruction model based on the first loss, the second loss, the third loss, and the fourth loss.

[0158] Furthermore, the terminal can determine the loss function corresponding to the vertical sample based on the first loss, the second loss, the third loss, and the fourth loss:

[0159]

[0160] Where μ3 is a hyperparameter.

[0161] Once the loss function corresponding to the vertical sample is determined, the parameters of the image reconstruction model can be adjusted based on this loss function.

[0162] Step 212: The terminal sends the adjusted model parameters to the server.

[0163] In this process, after the terminal adjusts the parameters of the image reconstruction model based on the loss function determined by the training sample images, it can further send the adjusted model parameters to the server to determine the global model parameters.

[0164] Step 213: The server calculates the mean of the received multiple sets of model parameters to obtain the global model parameters, and returns the global model parameters to the terminal.

[0165] After receiving the model parameters sent by each terminal, the server can calculate the global model parameters based on the multiple sets of received model parameters:

[0166]

[0167] Among them, w G These are global model parameters.

[0168] w k For the model parameters sent by the kth terminal, p k The weight coefficients of the model parameters sent to the k-th terminal.

[0169] in, That is, the model parameters of the kth terminal are the union of the parameters of the first encoder and the parameters of the decoder in that terminal.

[0170] in, n k This represents the number of training sample data obtained by the k-th terminal.

[0171] Then, the server can return the global model parameters to each terminal.

[0172] Step 214: The terminal updates the model parameters based on the received global model parameters and determines whether the model parameters have converged.

[0173] After obtaining the global model parameters, the terminal updates the model parameters based on the received global model parameters. Then, it can acquire the next training sample image and repeat the above training process until the model converges. Specifically, the terminal can determine in real time whether the model parameters of the image reconstruction model have converged; for example, when the number of iterations reaches a preset number, it can be determined that the model parameters have converged. If it detects that the model parameters have not converged, it can return to step 201 to continue iterative training.

[0174] Step 215: The terminal completes the training of the image reconstruction model.

[0175] When the convergence of the model parameters of the image reconstruction model is detected, it can be determined that the training of the image reconstruction model is complete.

[0176] As described above, the image reconstruction model training method provided in this application extracts modality-invariant features from training sample images based on a first encoding module and modality-specific features from training sample images based on a second encoding module. The model to be trained includes a first encoding module and a decoding module. The modality-invariant and modality-specific features are fused to obtain fused features. A first loss is calculated based on the difference between the decoding result and the training sample images. The decoding result is the result obtained by the decoding module from decoding the fused features. The model parameters of the model to be trained are adjusted based on the first loss, and the adjusted model parameters are sent to the server. The average model parameters returned by the server are received, and the model parameters of the model to be trained are updated based on the average model parameters. The process of repeatedly acquiring training sample images and updating the model parameters of the model to be trained based on the acquired training sample images is repeated until the model parameters converge. Therefore, the image reconstruction model training method provided in this application, by decoupling the modality-invariant features of the training sample images, enables training samples from different terminals to obtain a consistent feature distribution. This improves the model quality of the trained model, and consequently improves the reconstruction quality of the image.

[0177] To better implement the above methods, embodiments of the present invention also provide a training device for an image reconstruction model, which can be integrated into a server.

[0178] For example, such as Figure 5 The diagram shown is a structural schematic of a training device for an image reconstruction model provided in this application embodiment. The training device may include an extraction unit 301, a calculation unit 302, an adjustment unit 303, an update unit 304, and a training unit 305, as follows:

[0179] Extraction unit 301 is used to extract modality invariant features of training sample images based on the first encoding module and modality specific features of training sample images based on the second encoding module. The model to be trained includes the first encoding module and the decoding module.

[0180] The computation unit 302 is used to perform feature fusion on modality invariant features and modality specific features to obtain fused features, and to calculate the first loss based on the difference between the decoding result and the training sample image;

[0181] The adjustment unit 303 is used to adjust the model parameters of the model to be trained based on the first loss, and send the adjusted model parameters to the server;

[0182] The update unit 304 is used to receive the average model parameters returned by the server and update the model parameters of the model to be trained based on the average model parameters.

[0183] Training unit 305 is used to repeatedly execute the process of reacquiring training sample images and updating the model parameters of the model to be trained based on the acquired training sample images, until the parameters of the model to be trained converge.

[0184] In some embodiments, the extraction unit includes:

[0185] Extraction subunits are used to extract first image features of training sample images based on the first encoding module in the model to be trained, and to extract second image features of training sample images based on the second encoding module;

[0186] The first transmitting subunit is used to send the second image features and the modality label data corresponding to the training sample images to the server, and to receive the classification gradient returned by the server based on the second image features and modality label data;

[0187] A sub-unit is defined to determine a second loss based on the similarity relationship between the first image features and the second image features;

[0188] The adjustment unit can also be used for:

[0189] The model parameters of the model to be trained are adjusted based on the first loss, the second loss, and the classification gradient.

[0190] In some embodiments, the model training apparatus provided in this application further includes:

[0191] The detection subunit is used to perform encrypted instance alignment detection on the training sample images;

[0192] The second sending unit is used to send the first image features to the server when the training sample image is detected to be a vertical sample, and to receive the consistency gradient returned by the server.

[0193] The adjustment unit can also be used for:

[0194] The model parameters of the model to be trained are adjusted based on the first loss, the second loss, the classification gradient, and the consistency gradient.

[0195] In some embodiments, the model training apparatus provided in this application further includes:

[0196] The sub-unit is used to obtain the weight coefficients of the first loss, the second loss, and the classification gradient when the training sample image is detected as a parallel sample;

[0197] The adjustment sub-unit is used to adjust the model parameters of the model to be trained based on the weight coefficients, the first loss, the second loss, and the classification gradient.

[0198] In some embodiments, the computing unit includes:

[0199] The first computational subunit is used to add the modality-invariant features and the modality-specific features to obtain the fused features;

[0200] The decoding subunit is used to decode the fused features according to the decoding module to obtain the decoding result;

[0201] The second computational subunit is used to calculate the first loss based on the difference between the decoding result and the training sample image.

[0202] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0203] As described above, the model training apparatus provided in this embodiment extracts modality-invariant features of training sample images based on a first encoding module and modality-specific features based on a second encoding module using an extraction unit 301. The model to be trained includes a first encoding module and a decoding module. A calculation unit 302 performs feature fusion on the modality-invariant and modality-specific features to obtain fused features, and calculates a first loss based on the difference between the decoding result and the training sample images. The decoding result is the result obtained by the decoding module from decoding the fused features. An adjustment unit 303 adjusts the model parameters of the model to be trained based on the first loss and sends the adjusted model parameters to the server. An update unit 304 receives the average model parameters returned by the server and updates the model parameters of the model to be trained based on the average model parameters. A training unit 305 repeatedly executes the process of re-acquiring training sample images and updating the model parameters of the model to be trained based on the acquired training sample images until the model parameters converge. Thus, the image reconstruction model training method provided in this application decouples the modality-invariant features of the training sample images, enabling training samples from different terminals to obtain a consistent feature distribution. This can improve the quality of the trained model, and consequently improve the reconstruction quality of the image.

[0204] This application also provides an image reconstruction method, which can be specifically applied to an image reconstruction device, which can be integrated into a terminal or server. For example... Figure 6 The diagram shown is a flowchart illustrating the image reconstruction method provided in this application.

[0205] Step 401: Obtain the target image to be reconstructed.

[0206] In this embodiment of the application, the target image to be reconstructed can be any type of target image, specifically it can be a nuclear magnetic resonance image to be reconstructed.

[0207] Step 402: Reconstruct the target image to be reconstructed based on the preset image reconstruction model to obtain the reconstructed image.

[0208] In this embodiment, the preset image reconstruction model can be an image reconstruction model trained using the image reconstruction model training method provided in this application.

[0209] Accordingly, such as Figure 7 The diagram shown is a structural schematic of an image reconstruction apparatus provided in an embodiment of this application. The image reconstruction apparatus may include an acquisition unit 501 and a reconstruction unit 502. See below:

[0210] Acquisition unit 510 is used to acquire the target image to be reconstructed;

[0211] The reconstruction unit 502 is used to reconstruct the target image to be reconstructed based on a preset image reconstruction model to obtain the reconstructed image. The preset image reconstruction model is a model trained based on the image reconstruction model training method provided in this application.

[0212] This application also provides a computer device, which can be a terminal or a server, such as... Figure 8 The diagram shown is a structural schematic of the computer device provided in this application. Specifically:

[0213] The computer device may include components such as a processor 601 with one or more processing cores, a memory 602 with one or more storage media, a power supply 603, and an input unit 604. Those skilled in the art will understand that... Figure 8 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0214] The processor 601 is the control center of the computer device, connecting various parts of the computer device through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 602, and by calling data stored in the memory 602. Optionally, the processor 601 may include one or more processing cores; preferably, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 601.

[0215] The memory 602 can be used to store software programs and modules. The processor 601 executes various functional applications and model training by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, and web page access), etc.; the data storage area may store data created based on the use of the computer device. In addition, the memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.

[0216] The computer device also includes a power supply 603 that supplies power to the various components. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 603 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0217] The computer device may also include an input unit 604, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0218] Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 601 in the computer device loads the executable files corresponding to the processes of one or more application programs into the memory 602 according to the following instructions, and the processor 601 runs the application programs stored in the memory 602 to realize various functions, as follows:

[0219] The training model includes a first encoding module and a decoding module. The first encoding module extracts modality-invariant features from the training sample images, and the second encoding module extracts modality-specific features from the training sample images. The model then fuses the modality-invariant and modality-specific features to obtain fused features. A first loss is calculated based on the difference between the decoding result and the training sample images. The decoding result is obtained by the decoding module decoding the fused features. The model parameters of the training model are adjusted based on the first loss, and the adjusted model parameters are sent to the server. The average model parameters returned by the server are received, and the model parameters of the training model are updated based on the average model parameters. The process of repeatedly acquiring training sample images and updating the model parameters of the training model based on the acquired training sample images continues until the training model parameters converge.

[0220] Alternatively, the target image to be reconstructed can be obtained; the target image to be reconstructed can be reconstructed based on a preset image reconstruction model to obtain the reconstructed image. The preset image reconstruction model is a model trained based on the image reconstruction model training method provided in this application.

[0221] It should be noted that the computer device provided in this application embodiment and the image reconstruction model training method in the above embodiment belong to the same concept. The specific implementation of each of the above operations can be found in the previous embodiments, and will not be repeated here.

[0222] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0223] To this end, embodiments of the present invention provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in the training method of any image reconstruction model provided in the embodiments of the present invention. For example, the instructions can execute the following steps:

[0224] The training model includes a first encoding module and a decoding module. The first encoding module extracts modality-invariant features from the training sample images, and the second encoding module extracts modality-specific features from the training sample images. The model then fuses the modality-invariant and modality-specific features to obtain fused features. A first loss is calculated based on the difference between the decoding result and the training sample images. The decoding result is obtained by the decoding module decoding the fused features. The model parameters of the training model are adjusted based on the first loss, and the adjusted model parameters are sent to the server. The average model parameters returned by the server are received, and the model parameters of the training model are updated based on the average model parameters. The process of repeatedly acquiring training sample images and updating the model parameters of the training model based on the acquired training sample images continues until the training model parameters converge.

[0225] Alternatively, the target image to be reconstructed can be obtained; the target image to be reconstructed can be reconstructed based on a preset image reconstruction model to obtain the reconstructed image. The preset image reconstruction model is a model trained based on the image reconstruction model training method provided in this application.

[0226] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0227] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0228] Since the instructions stored in the computer-readable storage medium can execute the steps in the training method of any image reconstruction model provided in the embodiments of the present invention, the beneficial effects that the training method of any image reconstruction model provided in the embodiments of the present invention can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0229] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a storage medium. A processor of a computer device reads the computer instructions from the storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the training method of the image reconstruction model or various optional implementations of the image reconstruction method described above.

[0230] The training method and apparatus for an image reconstruction model, as well as the image reconstruction method and apparatus provided in the embodiments of the present invention, have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A training method for an image reconstruction model, characterized in that, The method includes: The training model includes a first encoding module and a decoding module. The steps involve: extracting first image features from the training sample images using the first encoding module and second image features from the training sample images using the second encoding module; sending the second image features and the corresponding modality label data of the training sample images to a server, and receiving the classification gradient returned by the server based on the second image features and the modality label data; and determining a second loss based on the similarity relationship between the first image features and the second image features. The modality-invariant features and the modality-specific features are fused to obtain fused features, and a first loss is calculated based on the difference between the decoding result and the training sample image. The decoding result is the result obtained by the decoding module from decoding the fused features. The model parameters of the model to be trained are adjusted based on the first loss, and the adjusted model parameters are sent to the server; the adjustment of the model parameters of the model to be trained based on the first loss includes: adjusting the model parameters of the model to be trained based on the first loss, the second loss, and the classification gradient; Receive the average model parameters returned by the server, and update the model parameters of the model to be trained based on the average model parameters; The process of repeatedly acquiring training sample images and updating the model parameters of the model to be trained based on the acquired training sample images is repeated until the model parameters of the model to be trained converge.

2. The method according to claim 1, characterized in that, The method further includes: Encryption instance alignment detection is performed on the training sample images; When the training sample image is detected to be a vertical sample, the first image feature is sent to the server, and the consistency gradient returned by the server is received; The adjustment of the model parameters of the model to be trained based on the first loss, the second loss, and the classification gradient includes: The model parameters of the model to be trained are adjusted based on the first loss, the second loss, the classification gradient, and the consistency gradient.

3. The method according to claim 2, characterized in that, The method further includes: When the training sample image is detected to be a parallel sample, the first loss, the second loss, and the weight coefficients of the classification gradient are obtained; The model parameters of the model to be trained are adjusted based on the weight coefficients, the first loss, the second loss, and the classification gradient.

4. The method according to claim 1, characterized in that, The process of fusing the modality-invariant features and the modality-specific features to obtain fused features, and calculating a first loss based on the difference between the decoding result and the training sample image, includes: The modality-invariant feature and the modality-specific feature are added together to obtain the fused feature; The decoding module decodes the fused features to obtain the decoding result; The first loss is calculated based on the difference between the decoding result and the training sample image.

5. An image reconstruction method, characterized in that, The method includes: Obtain the target image to be reconstructed; The target image to be reconstructed is reconstructed based on a preset image reconstruction model to obtain a reconstructed image. The preset image reconstruction model is a model trained based on the image reconstruction model training method of any one of claims 1 to 4.

6. A training device for an image reconstruction model, characterized in that, The training apparatus is an apparatus for performing the training method of the image reconstruction model according to any one of claims 1 to 4, the apparatus comprising: An extraction unit is configured to extract modality-invariant features of training sample images based on a first encoding module and modality-specific features of the training sample images based on a second encoding module. The model to be trained includes the first encoding module and a decoding module. Specifically, the extraction unit is configured to: extract first image features of the training sample images based on the first encoding module in the model to be trained, and extract second image features of the training sample images based on the second encoding module; send the second image features and the modality label data corresponding to the training sample images to a server, and receive the classification gradient returned by the server based on the second image features and the modality label data; and determine a second loss based on the similarity relationship between the first image features and the second image features. The computation unit is used to perform feature fusion on the mode invariance features and the mode specific features to obtain fused features, and to calculate a first loss based on the difference between the decoding result and the training sample image, wherein the decoding result is the result obtained by the decoding module from decoding the fused features; An adjustment unit is configured to adjust the model parameters of the model to be trained based on the first loss, and send the adjusted model parameters to the server; specifically, the adjustment unit is configured to: adjust the model parameters of the model to be trained based on the first loss, the second loss, and the classification gradient; An update unit is used to receive the average model parameters returned by the server and update the model parameters of the model to be trained based on the average model parameters. The training unit is used to repeatedly execute the process of reacquiring training sample images and updating the model parameters of the model to be trained based on the acquired training sample images, until the model parameters of the model to be trained converge.

7. An image reconstruction apparatus, characterized in that, The device includes: The acquisition unit is used to acquire the target image to be reconstructed; The reconstruction unit is used to reconstruct the target image to be reconstructed based on a preset image reconstruction model to obtain a reconstructed image. The preset image reconstruction model is a model trained based on the training method of the image reconstruction model according to any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the training method of the image reconstruction model of any one of claims 1 to 4 or the image reconstruction method of claim 5.

9. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the training method of the image reconstruction model according to any one of claims 1 to 4 or the steps of the image reconstruction method according to claim 5.