Image completion method, device, equipment and storage medium

Abstract

This application discloses an image completion method, apparatus, device, and storage medium, relating to the field of artificial intelligence. The method includes: acquiring a target image set of a target object, the target image set containing images of the target object in different modalities, and the images containing n missing images corresponding to missing modalities and m complete images corresponding to complete modalities, where n and m are positive integers; extracting target shared features from the complete images, the target shared features being features common to both the missing and complete images; and performing feature restoration on the target shared features to obtain the completed image corresponding to the missing image. Using the solution provided in this application, modal completion of images with missing modalities is achieved while ensuring the accuracy of the completion result, thereby guaranteeing the quality of image completion.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to an image completion method, apparatus, device, and storage medium. Background Technology

[0002] Image completion is the process of filling in missing areas of an image to be repaired based on information from the image itself or an image library, so that the repaired image looks very natural and is difficult to distinguish from the undamaged image.

[0003] A modality can be understood as multiple different manifestations of something. For example, in the process of magnetic resonance imaging (MRI), by changing the factors that affect the signal, we can obtain images of four modalities: T1, T2, FLAIR, and T1ce. However, due to different imaging methods, some images may lack necessary feature information. Such images are called missing images, and their corresponding modalities are called missing modalities.

[0004] Modal missingness can occur in various ways, and image completion methods in related technologies cannot guarantee the quality of image completion. Therefore, improving the quality of image completion has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides an image completion method, apparatus, device, and storage medium. The technical solution is as follows:

[0006] On one hand, embodiments of this application provide an image completion method, the method comprising:

[0007] Obtain a target image set of the target object, wherein the target image set contains images of the target object in different modalities, and the images contain n missing images corresponding to the missing modalities and m complete images corresponding to the complete modalities, where n and m are positive integers;

[0008] Extract target modality shared features from the complete image, wherein the target modality shared features are features shared by the missing image and the complete image;

[0009] The shared features of the target are restored to obtain the completed image corresponding to the missing image.

[0010] On the other hand, embodiments of this application provide an image completion device, the device comprising:

[0011] The acquisition module is used to acquire a set of target images of the target object. The set of target images contains images of the target object in different modalities, and the images contain n missing images corresponding to missing modalities and m complete images corresponding to complete modalities, where n and m are positive integers.

[0012] The feature extraction module is used to extract target modality shared features from the complete image, wherein the target modality shared features are features shared by the missing image and the complete image;

[0013] A feature restoration module is used to restore the shared features of the target modality to obtain the completed image corresponding to the missing image. On the other hand, embodiments of this application provide a computer device including a processor and a memory. The memory stores at least one program, which is loaded and executed by the processor to implement the image completion method as described above.

[0014] On the other hand, embodiments of this application provide a computer-readable storage medium storing at least one program that is loaded and executed by a processor to implement the image completion method as described above.

[0015] On the other hand, embodiments of this application provide a computer program product including computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform an image completion method as described above.

[0016] In this embodiment, after the computer device acquires the target image set of the target object, it extracts the paired modal shared features between the missing image and the complete image from the complete image, namely the target modal shared features, and then performs feature restoration on the target modal shared features to obtain the completed image corresponding to the missing image. By adopting the solution provided in this embodiment, while realizing modal completion of the image with missing modality, the accuracy of the completion result is guaranteed, thereby ensuring the quality of image completion. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1A schematic diagram of an implementation environment provided by an exemplary embodiment of this application is shown;

[0019] Figure 2 A flowchart illustrating an exemplary embodiment of the image completion method provided in this application is shown;

[0020] Figure 3 A flowchart of an image completion method provided in another exemplary embodiment of this application is shown;

[0021] Figure 4 This is a schematic diagram illustrating an exemplary embodiment of the image completion method of this application;

[0022] Figure 5 A flowchart illustrating a training method for an image completion model provided in an exemplary embodiment of this application is shown;

[0023] Figure 6 A schematic diagram illustrating a training method for an image completion model provided in an exemplary embodiment of this application is shown.

[0024] Figure 7 A flowchart illustrating a training method for an image completion model provided in another exemplary embodiment of this application is shown;

[0025] Figure 8 A schematic diagram of a training method for an image completion model provided in another exemplary embodiment of this application is shown;

[0026] Figure 9 This is an exemplary embodiment of this application illustrating a comparison of the supplementary effects of the embodiments of this application and related technologies;

[0027] Figure 10 This invention provides a structural block diagram of an image completion device according to an exemplary embodiment of the present application.

[0028] Figure 11 A schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0030] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0031] For ease of understanding, the terms used in the embodiments of this application will be explained below.

[0032] Generative Adversarial Networks (GANs) are a method of unsupervised learning that learns by having two neural networks compete against each other. A GAN consists of a generator and a discriminator. The core objective of a GAN is to train the generator. The generator aims to produce images that are as similar as possible to real sample images, while the discriminator aims to distinguish between real and generated images given a sample. These two objectives are contradictory, and they improve each other through continuous competition. Ultimately, even if the discriminator's ability is sufficiently reliable, it may still be unable to distinguish between real and generated images; that is, the images generated by the generator are indistinguishable from the sample images, and the discriminator cannot differentiate them.

[0033] Magnetic resonance imaging (MRI) is a medical imaging technique based on the principles of nuclear magnetic resonance (NMR). It uses magnetic fields and radio frequency waves to create images of human anatomy or physiological processes. An MRI sequence is a specific setup of radio frequency pulses and gradients that produces a specific image. MRI image modalities include T1, T2, FLAIR, and T1ce. T1 and T2 are physical quantities used to measure electromagnetic waves, and they can be used as imaging data. Imaging based on T1 is called "T1-weighted imaging," or simply "T1" in clinical practice; the same applies to T2. The overall appearance of T1 images is very close to the "customary color scheme" of "clinical images"—white matter is white, gray matter is gray, and cerebrospinal fluid is black—so T1 images can show various cross-sectional anatomical views. T2 signals are related to water content; many lesions have stronger T2 signals than surrounding normal tissue, often appearing as bright areas. Therefore, the location and size of lesions can be clearly seen from T2 images. FLAIR stands for Liquid Attenuation Inversion Magnetic Resonance Imaging, also known as Water Suppression Imaging. It suppresses high signal intensity in cerebrospinal fluid (making the cerebrospinal fluid darker) in T2-weighted imaging, thus making lesions near the cerebrospinal fluid clearer (brighter). T1-weighted imaging involves placing a contrast agent (pigment) in the blood before MRI. Bright areas have rich blood supply, and enhanced imaging indicates rich blood flow. Tumors are areas with very fast blood flow. T1-weighted imaging can further visualize the internal condition of tumors and differentiate between tumors and non-tumor lesions (i.e., gangrene).

[0034] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0035] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0036] Computer Vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to using cameras and computers to replace human eyes in recognizing and measuring targets, and then performing image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image segmentation, image semantic understanding, image retrieval, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0037] The image completion method involved in the embodiments of this application, namely the application of computer vision technology in the field of image processing, can improve the training effect of the image completion model and thus improve the accuracy of the completion result of the trained image completion model.

[0038] like Figure 1 The diagram illustrates an implementation environment provided by an exemplary embodiment of this application. This implementation environment includes a computer device 110 and a server 120. The computer device 110 and the server 120 communicate via a communication network. Optionally, the communication network can be a wired network or a wireless network, and the communication network can be at least one of a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN).

[0039] Computer device 110 is an electronic device with image completion requirements. The electronic device may be a smartphone, tablet computer, or personal computer, etc., and this embodiment does not limit it.

[0040] In some embodiments, the computer device 110 is equipped with an application program that has image completion functionality. When it is necessary to complete the image corresponding to the missing modality of a target object, the user inputs the image corresponding to the missing modality and the image corresponding to the complete modality of the target object into the application program in the form of an image set 121, thereby uploading the image set 121 to the server 120, which then performs image completion on the image corresponding to the missing modality of the target object and returns the image completion result.

[0041] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0042] In one possible implementation, computer device 110 uploads image set 121 to server 120, where server 120 performs image completion using image completion model 122 to obtain completed image 123. Image completion model 122 is an encoder-decoder network. Server 120 then sends the completed image 123 back to computer device 110 so that computer device 110 can display the image completion result.

[0043] Of course, in other possible implementations, the image completion model can also be deployed in the computer device 110, so that the computer device 110 can perform image completion locally, reducing the processing pressure on the server 120. This embodiment does not limit this.

[0044] Furthermore, the aforementioned image completion model can be trained by server 120, or it can be trained by other devices and then deployed on server 120. For ease of explanation, the following embodiments are illustrated by taking the application of the image completion method to a computer device and the image completion model training as an example.

[0045] It should be noted that the image completion method shown in the embodiments of this application can be applied to various image completion tasks. The embodiments of this application take the image completion of medical images as an example for illustration.

[0046] Please refer to Figure 2This document illustrates a flowchart of an image completion method provided in an exemplary embodiment of this application. This embodiment uses the method applied to a computer device as an example, and the method includes the following steps:

[0047] Step 201: Obtain the target image set of the target object. The target image set contains images of the target object in different modalities, and the images contain n missing images corresponding to the missing modalities and m complete images corresponding to the complete modalities, where n and m are positive integers.

[0048] In one possible implementation, the target object may be the central nervous system, brain, bones, spinal cord, or blood vessels, etc. The embodiments of this application do not limit the specific target object.

[0049] After the computer device obtains the target image set of the target object, it needs to perform image preprocessing operations on the images in the target image set to ensure that the input format of the images is consistent with the input format of the model training process.

[0050] Optionally, the preprocessing operation method is at least one of the preprocessing operations such as scaling, image normalization, image grayscale conversion, image enhancement, and image filtering. The embodiments of this application do not limit the specific preprocessing operation method.

[0051] Since MRI is the most commonly used and most important method for examining craniocerebral lesions, this application takes brain tumors as an example. In one possible implementation, when the image is a brain tumor image, the image modalities include T1 mode, T1ce mode, T2 mode, and FLAIR mode.

[0052] In this embodiment of the application, the computer device acquires a target image set of the target object and obtains images from the target image set. The images contain n missing images corresponding to the missing modalities of the target object and m complete images corresponding to the complete modalities, where n and m are positive integers. The missing images corresponding to the missing modalities are images that need to be image-completed, and the complete images corresponding to the complete modalities are reference images during the image-completed process.

[0053] Step 202: Extract target modality shared features from the complete image. Target modality shared features are features shared by the missing image and the complete image.

[0054] A feature is a corresponding (essential) characteristic or property that distinguishes one type of object from other types of objects, or a set of such characteristics or properties. In one possible implementation, a computer device can use a machine learning model to extract features from an image. The computer device extracts paired modal shared features between the complete image and the missing image from the complete image, that is, features common to both the complete image and the missing image, and uses these features as target modal shared features.

[0055] For each missing modality in the n missing modalities corresponding to the missing images, the computer device will extract the modality-shared features between the missing modalities and the m complete modalities corresponding to the complete images. That is, for each missing modality, the computer device can extract m modality-shared features from the m complete modalities corresponding to the complete images.

[0056] Step 203: Perform feature restoration on the target shared features to obtain the completed image corresponding to the missing image.

[0057] Image completion refers to the repair and reconstruction of damaged images. In one possible implementation, computer devices use machine learning models to restore the extracted features, thereby generating a completed image.

[0058] In summary, in the embodiments of this application, after the image completion model obtains the target image set of the target object, it extracts the target modality shared features from the complete image, and then performs feature restoration on the target modality shared features to obtain the completed image corresponding to the missing image. By adopting the scheme provided in the embodiments of this application, while realizing modality completion of the image with missing modality, the accuracy of the completion result is guaranteed, thereby ensuring the quality of image completion.

[0059] In this embodiment, the computer device pre-trains an image completion model using machine learning. This model consists of a feature encoder and a feature decoder. The feature encoder extracts shared feature information between the complete and missing images from the complete image. The feature decoder performs feature restoration on the modal-shared features extracted by the feature encoder to obtain the completed image. Please refer to... Figure 3 This illustration shows a flowchart of an image completion method provided in another exemplary embodiment of this application. This embodiment uses the method applied to a computer device as an example, and the method includes the following steps:

[0060] Step 301: Obtain the target image set of the target object. The target image set contains images of the target object in different modalities, and the images contain n missing images corresponding to the missing modalities and m complete images corresponding to the complete modalities, where n and m are positive integers.

[0061] The implementation method of this step can be referred to step 201, and will not be repeated here in this embodiment.

[0062] Step 302: Input the missing image and the complete image into the target feature encoder corresponding to the missing modality, where different modalities correspond to different feature encoders.

[0063] In one possible implementation, each modality of the image has a corresponding encoder, and the computer device inputs the missing image and the complete image into the target feature encoder corresponding to the missing modality.

[0064] In one possible implementation, the feature encoder is a hybrid expert network consisting of conditional convolutions, and the parameters of the conditional convolutions are determined based on the modalities corresponding to the feature encoder.

[0065] A Mixture of Experts (MOE) system is a type of neural network in which separate linear models are trained for local regions in the input dataset. These linear models are called experts, and a gating module is used to select which expert to use. The actual output of the model is a combination of the outputs of each model and the weights of the gating model. Each expert model can use different functions (various linear or nonlinear functions). A Mixture of Experts system integrates multiple models into a single task.

[0066] In this embodiment, the image completion model uses a feature encoder composed of conditional convolutions (CondConv), the parameters of which are determined by the input modality corresponding to the feature encoder, and uses s expert hybrid models. Where x represents the input image, i represents the input modality, σ(·) is the sigmoid activation function, # represents regular convolution, and {W1, …, W s} represents the network parameters associated with s experts. It is a hybrid weight for a specific modality.

[0067] In this embodiment of the application, the feature encoder consists of a downsampling module and a residual block. The downsampling module includes a 7×7 conditional convolutional block with a stride of 1, and two 4×4 conditional convolutional blocks with a stride of 2.

[0068] Indicative, such as Figure 4 As shown, the computer device acquires n missing images corresponding to missing modalities and m complete images corresponding to complete modalities. If the missing images are... The complete image is { j |j∈m}, since the image is an MRI multimodal brain tumor image, it includes four modalities: T1 modality, T1ce modality, T2 modality, and FLAIR modality. Feature encoders 1 to 4 each correspond to one of these modalities. If feature encoder 1 corresponds to the T1 modality, the image is missing. If the missing mode is T1, then the image completion model will complete the missing image. and the complete image { j |j∈m} is input into feature encoder 1. Similarly, if feature encoder 2 corresponds to the T2 mode, the missing image... If the missing mode is T2 mode, then the image completion model will complete the missing image. and full image The input is fed into feature encoder 2.

[0069] Step 303: Extract features from the missing and complete images using a target feature encoder to obtain target modality shared features.

[0070] In one possible implementation, the computer device extracts features from the missing image and the complete image using a target feature encoder. The extracted feature information is the feature information shared by the missing image and the complete image, which is extracted by the computer device from the complete image. That is, target modality shared features.

[0071] In some embodiments, the computer device extracts features from the missing image and the i-th complete image using a target feature encoder to obtain the i-th target modality shared features. The i-th complete image belongs to m complete images, and i is less than or equal to m.

[0072] Indicative, such as Figure 4 As shown, if the feature encoder 1 corresponds to the T1 mode, the missing image The missing mode is T1 mode. The image completion model uses feature encoder 1 to process the missing image. and the complete image {x j Feature extraction is performed on |j∈m} to extract the target modality shared features {s}. ij |j∈m}, because Figure 4 There is one missing image and three complete images. Let the missing image be labeled x1, and the three complete images be labeled x2, x3, and x4. Therefore, feature encoder 1 can extract three pairs of target modality shared features {s}. 12 ,s 13 ,s 14 The three pairs of target modalities share features extracted by feature encoder 1 from the complete images x2, x3, and x4, respectively. It's important to note that since the missing modality of the missing image is mode T1, only feature encoder 1 corresponding to mode T1 operates; the feature encoders for the other three complete modalities do not need to operate. Similarly, if the missing modality of the missing image is mode T2, only feature encoder 2 corresponding to mode T2 operates; the feature encoders for the other three complete modalities do not need to operate.

[0073] Step 304: Input the target modality shared features into the target feature decoder corresponding to the missing modality, wherein different modalities correspond to different feature decoders.

[0074] In one possible implementation, the computer device inputs the target modality shared features into the target feature decoder corresponding to the missing modality.

[0075] Since there are m complete images corresponding to complete modalities, the target feature encoder will obtain m pairs of target modal shared features. However, in practical applications, m is not fixed, so the number of target modal shared features is not fixed. The input of the feature decoder is of a fixed size. In order to meet the input requirements of the feature decoder, the computer equipment needs to perform some processing on the target modal shared features.

[0076] In one possible implementation, the computer device first performs feature fusion on the shared features of m target modalities to obtain fused shared features.

[0077] In this embodiment, the computer device first performs pooling on the target modality shared features, and then performs feature concatenation on the pooling results to achieve feature fusion and obtain fused shared features.

[0078] Pooling is a very common operation in convolutional neural networks. It mimics the human visual system to reduce the dimensionality of data. Pooling is also commonly called subsampling or downsampling. The significance of pooling lies in feature dimensionality reduction. Pooling technology greatly reduces the consumption of computational resources and also has the advantage of reducing model overfitting.

[0079] In one possible implementation, the computer device performs pooling processing on the i-th target modality shared feature using at least two pooling methods to obtain at least two pooled features corresponding to the i-th target modality shared feature. Then, the pooled features corresponding to each of the m target modality shared features are concatenated to obtain the fused shared feature.

[0080] Optionally, the pooling method can be general pooling, overlapping pooling, spatial pyramid pooling, center pooling, max-pooling, mean-mooling, min-pooling, stochastic-pooling, and global average pooling, etc. The embodiments of this application do not limit the specific pooling method.

[0081] Optionally, the computer device performs three pooling processes on the target modality shared features: max pooling, average pooling, and min pooling. The three pooled features obtained after pooling are then concatenated to obtain fused shared features while retaining as much feature information as possible.

[0082] Furthermore, the computer device will fuse the shared features input to the target feature decoder corresponding to the missing modality. Since it is impossible to determine at this time whether the number of channels of the fused shared features is consistent with the number of channels of the target feature decoder, in order to ensure that the number of channels is consistent, in one possible implementation, the image completion model performs channel dimensionality reduction or channel dimensionality increase processing on the fused shared features. After channel dimensionality reduction or channel dimensionality increase, the number of channels of the fused shared features is consistent with the number of channels of the output of the target feature encoder.

[0083] Optionally, the computer device can perform channel dimensionality reduction or channel dimensionality increase through methods such as interpolation, convolution, or principal component analysis; this embodiment does not limit this.

[0084] In this embodiment, the computer device performs channel dimensionality reduction or channel dimensionality increase on the fused shared features using 1×1 convolution to ensure that the number of channels in the fused shared features is consistent with the number of channels in the target feature decoder. Finally, the computer device inputs the channel-reduced or channel-increased fused shared features into the target feature decoder corresponding to the missing modality.

[0085] Indicative, such as Figure 4 As shown, the target modality shared features {s} generated by the feature encoder ij After multi-pooling feature fusion processing is performed on |j∈m}, fused shared features are obtained, and then the fused shared features are input into the corresponding feature decoder.

[0086] Step 305: The target modality shared features are restored by the target feature decoder to obtain the completed image.

[0087] In one possible implementation, the computer device uses a target feature decoder to restore the shared features of the target modality to obtain a completed image.

[0088] In this embodiment, the feature decoder includes four residual blocks, each containing two 3×3 conditional convolutional blocks with 256 filters and a stride of 1. It also includes two nearest neighbor upsamplers and a 5×5 conditional convolutional block with a stride of 1, used to upsample the fused shared features to the original image size. The number of filters is 64-128-256-128-64. Finally, a 7×7 conditional convolutional block with a stride of 1 and a filter output the completed image.

[0089] Optionally, the computer device uses a target feature decoder to restore the fused shared features to obtain a completed image.

[0090] Indicative, such as Figure 4 As shown, feature decoder 1 performs feature restoration on the fused shared feature 1 to obtain the completed image x1'.

[0091] In this embodiment, the computer device inputs the missing image and the complete image into the target feature encoder corresponding to the missing modality. The target feature encoder extracts features from the missing image and the complete image to obtain the target modality shared features. Then, the target modality shared features are fused to obtain fused shared features. Subsequently, the fused shared features are subjected to channel dimensionality reduction or channel dimensionality increase processing, and the processed fused shared features are input into the target feature decoder corresponding to the missing modality. Finally, the computer device restores the target modality shared features through the target feature decoder to obtain the completed image. The computer device improves the robustness of the extracted features, reduces information redundancy, and prevents overfitting by using multi-pooling feature fusion, thereby ensuring the accuracy of the image completion result.

[0092] The above embodiments illustrate the application process of the image completion model. The following exemplary embodiments illustrate the training process of the image completion model.

[0093] Please refer to Figure 5 The diagram illustrates a flowchart of a training method for an image completion model provided in an exemplary embodiment of this application.

[0094] Step 501: Obtain a set of sample images of the sample object. The set of sample images contains sample images of the sample object in different modalities, and the sample images contain at least one missing sample image corresponding to the missing modality and at least one complete sample image corresponding to the complete modality.

[0095] In one possible implementation, the computer device acquires a set of sample images of the sample object, and obtains the missing sample image corresponding to the missing modality and the complete sample image corresponding to the complete modality from the set of sample images.

[0096] Optionally, the sample object can be the central nervous system, brain, bones, spinal cord, or blood vessels, etc. The embodiments of this application do not limit the specific sample object.

[0097] Optionally, after the computer device obtains the target image set of the sample object, it needs to perform image preprocessing operations on the sample images in the sample image set. The preprocessing operation method can be at least one of the preprocessing operations such as scale transformation, image normalization, image grayscale conversion, image enhancement, and image filtering. The embodiments of this application do not limit the specific preprocessing operation method.

[0098] Optionally, the computer device trains feature encoders and feature decoders for various modalities based on a set of sample images.

[0099] Step 502: Extract features from the sample image using the feature encoder corresponding to the target modality to obtain the shared features of the first sample modality.

[0100] In one possible implementation, the computer device extracts features from the sample image using a feature encoder corresponding to the target modality to obtain a first sample modality shared feature. Wherein, when the target modality is a missing modality, the first sample modality shared feature is a feature shared by both the missing sample image and the complete sample image; when the target modality is a complete modality, the first sample modality shared feature is a feature shared by different complete sample images.

[0101] Unlike image completion models in the application phase that only extract features from missing and complete images, in the training phase, computer devices will also extract features from complete images of complete modal samples.

[0102] The computer device first extracts features from the sample image using the feature encoder corresponding to the target modality to obtain paired modality shared features. Similar to the application stage, in order to meet the input requirements of the feature decoder, the computer device performs multi-pooling fusion processing on the paired modality shared features obtained by the feature encoder to obtain fused shared features. Then, it performs 1×1 convolution processing on the fused shared features to ensure that the number of input channels of the feature decoder corresponding to the same modality is consistent with the number of output channels of the feature decoder. Finally, the processed fused shared features are used as the first sample modality shared features.

[0103] Indicative, such as Figure 6 As shown, there is a missing sample image x1, and complete sample images x2, x3, and x4. Feature encoder 1 is the feature encoder corresponding to the missing mode of the missing sample image x1. Feature encoder 1 will obtain the paired mode-shared features {s} that are common to the missing sample image x1 and the complete sample images x2, x3, and x4. 12 ,s 13 ,s 14 The computer device performs multi-pooling fusion processing on the paired shared modal features to obtain the first sample modal shared feature 1. Similarly, the feature encoder 2 obtains the paired modal shared features {s} common to the complete sample image x2 and the complete sample images x2, x3, and x4. 22 ,s 23 ,s 24 The computer device performs multi-pooling fusion processing on the paired shared modal features to obtain the first sample modal shared feature 2. Similarly, the feature encoder 3 will obtain the first sample modal shared feature 3 shared by the complete sample image x3 and the complete sample images x2, x3 and x4. The feature encoder 4 will obtain the first sample modal shared feature 4 shared by the complete sample image x4 and the complete sample images x2, x3 and x4.

[0104] Step 503: Use the feature decoder corresponding to the target modality to perform feature restoration on the shared features of the first sample modality to obtain the sample generated image.

[0105] The computer device inputs the shared features of the first sample mode into the feature decoder corresponding to the target mode. The feature decoder corresponding to the target mode then performs feature reconstruction on the shared features of the first sample mode to obtain the sample generated image.

[0106] Step 504: Based on the generated images and sample images, train the feature deencoders and feature encoders / decoders corresponding to each modality.

[0107] Since the feature decoder generates sample images based on the shared features of the first sample modality obtained by the feature decoder, if the sample generated image generated by the feature decoder is not similar enough to the sample image, the feature decoder and the feature encoder will continue to be trained together.

[0108] Optionally, this step may include the following sub-steps:

[0109] 1. Determine the image consistency loss based on the generated image and the sample image.

[0110] In one possible implementation, the feature decoder should generate an image similar to the input image. For this purpose, the image completion model employs an image consistency loss Li. img To characterize the similarity between the generated image and the input image, Where, x i For the input image, X i Image modality, c i Let E be the shared features of the first sample modality, G be the feature encoder, m be the total number of complete sample images, and G be the feature decoder. i (c i The image generated from the sample is obtained by feature decoding of the shared features of the first sample mode.

[0111] 2. Train the feature encoder and feature decoder for each modality based on image consistency loss.

[0112] In one possible implementation, if the image consistency loss is within a certain range, the sample generated image generated by the feature decoder is similar to the sample image, and the image completion model training is complete. Conversely, if the image consistency loss exceeds a certain range, the sample generated image generated by the feature decoder is not similar to the sample image, and the image completion model will continue to train the feature encoder and feature decoder corresponding to each modality.

[0113] In summary, in this embodiment, after the computer device acquires a set of sample images of the sample object, it extracts features from the sample images using the feature encoder corresponding to the target modality to obtain the first sample modality shared features. Then, it restores the first sample modality shared features using the feature decoder corresponding to the target modality to obtain the sample generated image. Based on the sample generated image and the sample image, it determines the image consistency loss and trains the feature deencoder and feature encoder / decoder corresponding to each modality based on the image consistency loss. While achieving image completion, the training can further ensure the accuracy of image completion.

[0114] To further improve the accuracy of training results, please refer to... Figure 7 The diagram illustrates a flowchart of a training method for an image completion model provided in another exemplary embodiment of this application.

[0115] Step 701: Obtain a set of sample images of the sample object. The set of sample images contains sample images of the sample object in different modalities, and the sample images contain at least one missing sample image corresponding to the missing modality and at least one complete sample image corresponding to the complete modality.

[0116] The implementation method of this step can be referred to step 501, and will not be repeated here in this embodiment.

[0117] Step 702: Extract features from the sample image using the feature encoder corresponding to the target modality to obtain the shared features of the first sample modality.

[0118] The implementation method of this step can be referred to step 502, and will not be repeated here in this embodiment.

[0119] Step 703: Use the feature decoder corresponding to the target modality to restore the shared features of the first sample modality to obtain the sample generated image.

[0120] The implementation method of this step can be referred to step 503, and will not be repeated here in this embodiment.

[0121] Step 704: Extract features from the sample generated image using the feature encoder corresponding to the target modality to obtain the shared features of the second sample modality.

[0122] In one possible implementation, the feature encoder corresponding to the target modality extracts features from the sample generated image to obtain shared features of the second sample modality.

[0123] Step 705: Based on the sample generated image, the sample image, the shared features of the first sample modality, and the shared features of the second sample modality, train the feature encoder and feature decoder corresponding to each modality.

[0124] In one possible implementation, the computer device trains feature encoders and feature decoders corresponding to various modalities based on sample generated images, sample images, first sample modality shared features, and second sample modality shared features.

[0125] Optionally, this step may include the following sub-steps:

[0126] 1. Determine the image consistency loss based on the generated image and the sample image.

[0127] In one possible implementation, the feature decoder should generate an image similar to the input image. For this purpose, the image completion model employs an image consistency loss Li. img To characterize the similarity between the generated image and the input image, Where, x i For the input image, X i Image modality, c i Let E be the shared features of the first sample modality, G be the feature encoder, m be the total number of complete sample images, and G be the feature decoder. i (c i The image generated from the sample is obtained by feature decoding of the shared features of the first sample mode.

[0128] 2. Determine the feature consistency loss based on the modal sharing features of the first sample and the modal sharing features of the second sample.

[0129] Feature consistency loss can also be called latent consistency loss L. latent This is used to characterize the similarity between the second sample modal shared features obtained by the feature encoder and the first sample modal shared features in the image generated by the feature decoder. Where, x i For the input image, X i Image modality, c i Let E be the shared features of the first sample modality, G be the feature encoder, m be the total number of complete sample images, and G be the feature decoder. i (c i E refers to the sample-generated image obtained by the feature decoder performing feature reconstruction on the shared features of the first sample mode. i (G i (c i i) represents the second sample modality shared feature obtained by the feature encoder corresponding to the target modality extracting features from the sample generated image.

[0130] 3. Input the generated image and the sample image into the discriminator to obtain the sample discrimination result. The discriminator is used to distinguish between the generated image and the real image, and determines the adversarial loss based on the sample discrimination result.

[0131] To make the generated images closer to real images, this application embodiment utilizes the generative adversarial approach. During the training process, a discriminator is used to distinguish between the sample image and the sample-generated image. Ultimately, if the discriminator's discrimination capability is sufficiently reliable but it still cannot distinguish whether a given image is a sample image or a sample-generated image, that is, when the sample-generated image generated by the feature decoder is close to the sample image and the discrimination model cannot distinguish it, the computer device completes the training.

[0132] In this embodiment, the discriminator includes four 4×4 conditional convolutional blocks with a span of 2, and the number of filters is 64-128-256-512. Furthermore, the discriminator uses a leaky ReLU activation function with a slope of 2. The adversarial loss L... adv It is used to characterize the distributional differences between generated and real images, and is defined as follows: Where, x i For the input image, X i The image modality to which the input image belongs, c i The first sample modality shares features, m is the total number of complete images in the sample, and G is the shared feature. i (c i The D refers to the sample-generated image obtained by the feature decoder performing feature reconstruction on the shared features of the first sample modality. i It is a discriminator for mode i, used to distinguish between the sample image and the generated image of mode i.

[0133] 4. Determine the symmetry loss based on the modal sharing features of the first sample. The symmetry loss is used to characterize the similarity of modal sharing features between paired modes.

[0134] Ideally, paired modal shared features should be symmetric. For example, the T1 modal shared features extracted from T2 should be similar to the T2 modal shared features extracted from T1. To achieve good decoupling of paired modal shared features, the image completion model introduces a symmetry loss Li. sym Its definition Where d(·,·) calculates the distance between two feature quantities, and s ij =E i (x j ;j) represents the shared features of mode i extracted from mode j, and α = 0.1 is preset in the image completion model.

[0135] 5. Determine the total loss based on image consistency loss, feature consistency loss, adversarial loss, and symmetry loss.

[0136] Finally, the total loss function of the image completion model is L, which is defined as L = λ img L img +λ latent Llatent +λ adv L adv +λ sym L sym In the image completion model, λ is pre-defined. img =10, λ latent =1,λ adv =1,λ sym =1.

[0137] 6. Train the corresponding feature encoders and feature decoders, as well as the discriminator, based on the total loss.

[0138] During training, the number and distribution of available modalities are random, and the computer device uses min... E,G max D L optimizes the total loss function L. When L reaches a certain target range, the discriminator cannot distinguish between the generated image and the sample image, and the computer device completes the training. Before L reaches a certain target range, that is, when the discriminator can distinguish between the generated image and the sample image, the computer device trains the corresponding feature encoder and feature decoder, as well as the discriminator, based on the total loss.

[0139] Indicative, such as Figure 8 As shown, there is a missing sample image x1, and complete sample images x2, x3, and x4. Feature encoder 1 is the feature encoder corresponding to the missing mode of the missing sample image x1. Feature encoder 1 obtains the paired mode-shared features common to the missing sample image x1 and the complete sample images x2, x3, and x4, and performs multi-pooling fusion processing on the paired shared mode features to obtain the first sample mode-shared feature 1. The computer device performs feature reconstruction on the first sample mode-shared feature 1 through the feature decoder 1 corresponding to the target mode to obtain the sample generated image x1'. Then, the feature encoder 1 corresponding to the target mode performs feature reconstruction on the sample generated image x1'. Extracting the second sample modality shared feature 1, the computer device determines the image consistency loss based on the sample generated image and the sample image, and determines the feature consistency loss based on the first and second sample modality shared features. The sample generated image and the sample image are input into the discriminator to obtain the sample discrimination result, and the adversarial loss is determined based on the sample discrimination result. The symmetry loss is determined based on the first sample modality shared feature. Finally, the computer device determines the total loss based on the image consistency loss, feature consistency loss, adversarial loss, and symmetry loss, and trains the corresponding feature encoder and feature decoder, as well as the discriminator, based on the total loss.

[0140] Two existing related technologies provide different image completion methods. However, because the image completion method in related technology 1 extracts invariant feature information between all modalities and completes the image based on this feature information, while the image completion method in related technology 2 only extracts invariant feature information between two modalities and completes the image based on this feature information, the completed images generated by both lose some image details and cannot achieve accurate image completion. Therefore, in order to improve the accuracy of image completion, this application embodiment extracts paired modal shared features shared between two or three modalities, i.e., target modal shared features, and performs modal completion on the missing image based on the target modal shared features, thereby obtaining the completed image corresponding to the missing image, such as... Figure 9 As shown, compared with the completed images of the two related technologies, the completed images of this scheme have more image details, ensuring the accuracy of image completion while achieving image completion.

[0141] As shown in Tables 1 and 2, compared to related techniques 1, which extracts invariant feature information between all modalities and completes the image based on this feature information, and related techniques 2, which extracts invariant feature information between only two modalities and completes the image based on this feature information, the image completion method provided in this application embodiment outperforms the two related techniques in terms of peak signal-to-noise ratio and structural similarity in most cases. This indicates that the image completion method provided in this application embodiment can generate more realistic completed images, that is, the completed images generated by this application embodiment have higher accuracy, and the image completion model has better performance.

[0142] Table 1

[0143]

[0144] Table 2

[0145]

[0146] Please refer to Figure 10 This illustrates a structural block diagram of an image completion device provided in an exemplary embodiment of this application, the device comprising:

[0147] The acquisition module 1001 is used to acquire a target image set of the target object. The target image set contains images of the target object in different modalities, and the images contain n missing images corresponding to missing modalities and m complete images corresponding to complete modalities, where n and m are positive integers.

[0148] The feature extraction module 1002 is used to extract target modality shared features from the complete image, wherein the target modality shared features are features shared by the missing image and the complete image;

[0149] The feature restoration module 1003 is used to restore the shared features of the target modality to obtain the completed image corresponding to the missing image. Optionally, the feature extraction module 1002 includes:

[0150] An image input unit is used to input the missing image and the complete image into the target feature encoder corresponding to the missing modality, wherein different modalities correspond to different feature encoders;

[0151] The feature extraction unit is used to extract features from the missing image and the complete image through the target feature encoder to obtain the target modality shared features;

[0152] The feature restoration module 1003 includes:

[0153] A feature input unit is used to input the shared features of the target modality into the target feature decoder corresponding to the missing modality, wherein different modalities correspond to different feature decoders;

[0154] The feature restoration unit is used to restore the target modality shared features through the target feature decoder to obtain the completed image.

[0155] Optionally, the feature extraction unit is used to extract features from the missing image and the i-th complete image through the target feature encoder to obtain the i-th target modality shared features, wherein the i-th complete image belongs to m of the complete images, and i is less than or equal to m;

[0156] The feature input unit is used for:

[0157] Feature fusion is performed on the shared features of the m target modalities to obtain fused shared features;

[0158] The fused shared features are input into the target feature decoder corresponding to the missing modality;

[0159] The feature restoration unit is used to restore the fused shared features through the target feature decoder to obtain the completed image.

[0160] Optionally, the feature input unit is further configured to:

[0161] The i-th target modality shared feature is pooled using at least two pooling methods to obtain at least two pooled features corresponding to the i-th target modality shared feature;

[0162] The pooling features corresponding to the shared features of each of the m target modalities are concatenated to obtain the fused shared features.

[0163] Optionally, the feature input unit is further configured to:

[0164] The fused shared features are subjected to channel dimensionality reduction or channel dimensionality increase processing, wherein the number of channels of the fused shared features after channel dimensionality reduction or channel dimensionality increase is consistent with the number of channels output by the target feature encoder;

[0165] The fused shared features, after channel dimensionality reduction or channel dimensionality increase, are input into the target feature decoder corresponding to the missing modality.

[0166] Optionally, the feature encoder is a hybrid expert network composed of conditional convolutions, and the parameters of the conditional convolutions are determined based on the modality corresponding to the feature encoder.

[0167] Optionally, the device further includes:

[0168] The training module is used to obtain a set of sample images of the sample object. The set of sample images contains sample images of the sample object in different modalities, and the sample images contain at least one missing sample image corresponding to the missing modality and at least one complete sample image corresponding to the complete modality.

[0169] Based on the sample image set, feature encoders and feature decoders corresponding to various modalities are trained.

[0170] Optionally, the training module is further configured to:

[0171] The sample images are feature extracted by a feature encoder corresponding to the target modality to obtain first sample modality shared features. Wherein, when the target modality is the missing modality, the first sample modality shared features are features common to both the missing sample image and the complete sample image; when the target modality is the complete modality, the first sample modality shared features are features common to different complete sample images.

[0172] The shared features of the first sample modality are restored by the feature decoder corresponding to the target modality to obtain the sample generated image;

[0173] Based on the sample images, generate images and train feature encoders and feature decoders for each modality.

[0174] Optionally, the training module is further configured to:

[0175] Based on the generated image from the sample and the sample image, determine the image consistency loss;

[0176] Based on the image consistency loss, train the feature encoder and feature decoder corresponding to each modality.

[0177] Optionally, the training module is further configured to:

[0178] The feature encoder corresponding to the target modality is used to extract features from the sample generated image to obtain the second sample modality shared features.

[0179] The process of generating images based on the samples and training feature encoders and feature decoders for various modalities includes:

[0180] Based on the generated image from the sample, the sample image, the first sample modality shared features, and the second sample modality shared features, the feature encoder and feature decoder corresponding to each modality are trained.

[0181] Optionally, the training module is further configured to:

[0182] Based on the generated image from the sample and the sample image, determine the image consistency loss;

[0183] The feature consistency loss is determined based on the first sample modal sharing features and the second sample modal sharing features;

[0184] The generated image and the sample image are input into a discriminator to obtain a sample discrimination result. The discriminator is used to distinguish between the generated image and the real image. The adversarial loss is determined based on the sample discrimination result.

[0185] The symmetry loss is determined based on the modal sharing features of the first sample, and the symmetry loss is used to characterize the similarity of modal sharing features between pairs of modes;

[0186] The total loss is determined based on the image consistency loss, the feature consistency loss, the adversarial loss, and the symmetry loss.

[0187] The corresponding feature encoder and feature decoder, as well as the discriminator, are trained based on the total loss.

[0188] Optionally, when the image is a brain tumor image, the image modality includes T1 mode, T1ce mode, T2 mode, and FLAIR mode.

[0189] Please refer to Figure 11This illustration shows a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Specifically, the computer device 1100 includes a Central Processing Unit (CPU) 1101, a system memory 1104 including a random access memory 1102 and a read-only memory 1103, and a system bus 1105 connecting the system memory 1104 and the CPU 1101. The computer device 1100 may also include a basic input / output system (I / O system) 1106 to facilitate the transfer of information between various devices within the computer, and a mass storage device 1107 for storing the operating system 1113, application programs 1114, and other program modules 1115.

[0190] In some embodiments, the basic input / output system 1106 may include a display 1208 for displaying information and an input device 1109 for user input, such as a mouse or keyboard. Both the display 1108 and the input device 1109 are connected to the central processing unit 1101 via an input / output controller 1110 connected to the system bus 1105. The basic input / output system 1106 may also include the input / output controller 1110 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1110 also provides output to a display screen, printer, or other types of output devices.

[0191] The mass storage device 1107 is connected to the central processing unit 1101 via a mass storage controller (not shown) connected to the system bus 1105. The mass storage device 1107 and its associated computer-readable media provide non-volatile storage for the computer device 1100. That is, the mass storage device 1207 may include computer-readable media (not shown) such as a hard disk or drive.

[0192] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 1104 and mass storage device 1107 described above can be collectively referred to as memory.

[0193] The memory stores one or more programs, which are configured to be executed by one or more central processing units 1101. The one or more programs contain instructions for implementing the methods described above, and the central processing unit 1101 executes the one or more programs to implement the methods provided in the various method embodiments described above.

[0194] According to various embodiments of this application, the computer device 1100 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1100 can be connected to the network 1112 via the network interface unit 1111 connected to the system bus 1105, or the network interface unit 1111 can be used to connect to other types of networks or remote computer systems (not shown).

[0195] The memory further includes one or more programs stored in the memory, and the one or more programs include steps performed by a computer device in the methods provided in the embodiments of this application.

[0196] This application also provides a computer-readable storage medium storing at least one program, which is loaded and executed by a processor to implement the image completion method as described in the above embodiments.

[0197] This application provides a computer program product containing computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image completion method as described in the above embodiments.

[0198] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0199] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image completion method, characterized in that, The method includes: Obtain a target image set of the target object, wherein the target image set contains images of the target object in different modalities, and the images contain n missing images corresponding to the missing modalities and m complete images corresponding to the complete modalities, where n and m are positive integers; The missing image and the complete image are input into the target feature encoder corresponding to the missing modality, wherein different modalities correspond to different feature encoders; the target feature encoder is used to extract features from the missing image and the complete image to obtain target modality shared features, which are features common to the missing image and the complete image; The target modality shared features are input into the target feature decoder corresponding to the missing modality, wherein different modalities correspond to different feature decoders; the target feature decoder is used to restore the target modality shared features to obtain the completed image corresponding to the missing image.

2. The method according to claim 1, characterized in that, The step of extracting features from the missing image and the complete image using the target feature encoder to obtain target modality shared features includes: The target feature encoder extracts features from the missing image and the i-th complete image to obtain the i-th target modality shared features. The i-th complete image belongs to m complete images, and i is less than or equal to m. The step of inputting the shared features of the target modality into the target feature decoder corresponding to the missing modality includes: Feature fusion is performed on the shared features of the m target modalities to obtain fused shared features; The fused shared features are input into the target feature decoder corresponding to the missing modality; The step of restoring the shared features of the target modality using the target feature decoder to obtain the completed image corresponding to the missing image includes: The target feature decoder performs feature restoration on the fused shared features to obtain the completed image.

3. The method according to claim 2, characterized in that, The feature fusion of the shared features of the m target modalities to obtain fused shared features includes: The i-th target modality shared feature is pooled using at least two pooling methods to obtain at least two pooled features corresponding to the i-th target modality shared feature; The pooling features corresponding to the shared features of each of the m target modalities are concatenated to obtain the fused shared features.

4. The method according to claim 2, characterized in that, The step of inputting the fused shared features into the target feature decoder corresponding to the missing modality includes: The fused shared features are subjected to channel dimensionality reduction or channel dimensionality increase processing, wherein the number of channels of the fused shared features after channel dimensionality reduction or channel dimensionality increase is consistent with the number of channels output by the target feature encoder; The fused shared features, after channel dimensionality reduction or channel dimensionality increase, are input into the target feature decoder corresponding to the missing modality.

5. The method according to claim 1, characterized in that, The feature encoder is a hybrid expert network composed of conditional convolutions, and the parameters of the conditional convolutions are determined based on the modality corresponding to the feature encoder.

6. The method according to claim 1, characterized in that, The method further includes: Obtain a set of sample images of the sample object, wherein the set of sample images contains sample images of the sample object in different modalities, and wherein the sample images contain at least one missing sample image corresponding to the missing modality and at least one complete sample image corresponding to the complete modality; Based on the sample image set, feature encoders and feature decoders corresponding to various modalities are trained.

7. The method according to claim 6, characterized in that, The training of feature encoders and feature decoders corresponding to various modalities based on the sample image set includes: The sample images are feature extracted by a feature encoder corresponding to the target modality to obtain first sample modality shared features. Wherein, when the target modality is the missing modality, the first sample modality shared features are features common to both the missing sample image and the complete sample image; when the target modality is the complete modality, the first sample modality shared features are features common to different complete sample images. The shared features of the first sample modality are restored by the feature decoder corresponding to the target modality to obtain the sample generated image; Based on the sample images, generate images and train feature encoders and feature decoders for each modality.

8. The method according to claim 7, characterized in that, The process of generating images based on the samples and training feature encoders and feature decoders for various modalities includes: Based on the generated image from the sample and the sample image, determine the image consistency loss; Based on the image consistency loss, train the feature encoder and feature decoder corresponding to each modality.

9. The method according to claim 7, characterized in that, The training of feature encoders and feature decoders corresponding to various modalities based on the sample image set also includes: The feature encoder corresponding to the target modality is used to extract features from the sample generated image to obtain the second sample modality shared features. The process of generating images based on the samples and training feature encoders and feature decoders for various modalities includes: Based on the generated image from the sample, the sample image, the first sample modality shared features, and the second sample modality shared features, the feature encoder and feature decoder corresponding to each modality are trained.

10. The method according to claim 9, characterized in that, The step of generating images based on the samples, the sample images, the first sample modality shared features, and the second sample modality shared features, and training feature encoders and feature decoders corresponding to each modality, includes: Based on the generated image from the sample and the sample image, determine the image consistency loss; The feature consistency loss is determined based on the first sample modal sharing features and the second sample modal sharing features; The generated image and the sample image are input into a discriminator to obtain a sample discrimination result. The discriminator is used to distinguish between the generated image and the real image. The adversarial loss is determined based on the sample discrimination result. The symmetry loss is determined based on the modal sharing features of the first sample, and the symmetry loss is used to characterize the similarity of modal sharing features between pairs of modes; The total loss is determined based on the image consistency loss, the feature consistency loss, the adversarial loss, and the symmetry loss. The corresponding feature encoder and feature decoder, as well as the discriminator, are trained based on the total loss.

11. The method according to any one of claims 1 to 10, characterized in that, In the case that the image is a brain tumor image, the modalities of the image include T1 mode, T1ce mode, T2 mode, and FLAIR mode.

12. An image completion device, characterized in that, The device includes: The acquisition module is used to acquire a set of target images of the target object. The set of target images contains images of the target object in different modalities, and the images contain n missing images corresponding to missing modalities and m complete images corresponding to complete modalities, where n and m are positive integers. The feature extraction module is used to input the missing image and the complete image into the target feature encoder corresponding to the missing modality, wherein different modalities correspond to different feature encoders; and to extract features from the missing image and the complete image through the target feature encoder to obtain target modality shared features, wherein the target modality shared features are features common to the missing image and the complete image; The feature restoration module is used to input the shared features of the target modality into the target feature decoder corresponding to the missing modality, wherein different modalities correspond to different feature decoders; and to restore the shared features of the target modality through the target feature decoder to obtain the completed image corresponding to the missing image.

13. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the image completion method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one program, which is loaded and executed by a processor to implement the image completion method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image completion method as described in any one of claims 1 to 11.

Images