A method for repairing occlusion artifacts of medical images

Abstract

The application discloses a kind of medical image's occlusion artifact repair method, first acquisition image sequence to be repaired, using occlusion artifact detection model to generate mask;Subsequently, image and mask are input into deep network comprising generation model, artifact image occlusion perception feature extraction and image repair module, under the guidance of artifact image occlusion perception feature, only replace artifact area, then output repair image.In training phase, large-scale artifact library is constructed by combining artificial labeling and simulation, artifact-containing images are synthesized, and generation model and repair module are jointly optimized with multiple losses such as reconstruction, adversarial, structural similarity, to realize high-quality, visually continuous repair.The application can accurately detect and restore occlusion artifacts in multiple modal medical images, and has higher precision, robustness and universality compared with traditional manual or interpolation methods, and has wide clinical application prospect.

Description

Technical Field

[0001] This invention relates to a deep learning-based image processing technology, specifically a technique for repairing occlusion artifacts in medical images. Background Technology

[0002] Medical imaging plays a crucial role in clinical diagnosis, especially in intravascular imaging (such as IVUS and intravascular OCT), medical ultrasound, and ophthalmic OCT. The clarity and accuracy of images directly affect doctors' judgments and treatment decisions. Optical coherence tomography (OCT) is a non-invasive medical imaging technique that typically uses near-infrared light with wavelengths of 800-1300 nm to illuminate tissue. By measuring the interference signal between the reflected or scattered light and the reference light, an image is generated, thereby obtaining a high-resolution cross-sectional image of the biological tissue.

[0003] However, during OCT (Optical Characteristic Telemetry) medical image acquisition, image information may be lost or distorted due to the influence of medical devices (such as catheters, metal sutures, etc.), other obstructions (such as metal implants), or external interference factors. This typically manifests as shadows or artifacts appearing in certain areas of the image. Artifacts are various forms of images that appear on the image even though the scanned object does not actually exist. These problems severely affect the measurement and assessment of critical tissues such as lesions, thus posing a significant challenge to physicians in performing accurate image analysis and making accurate diagnoses.

[0004] Currently, traditional medical image artifact restoration methods typically rely on manual or simple computational methods to process these artifacts, but these methods have significant limitations. Manual restoration methods are not only time-consuming and labor-intensive, but also prone to human error. Existing automated restoration methods often rely on relatively simple interpolation algorithms or traditional image processing techniques, making it difficult to fully consider the complex factors and high-dimensional data in images.

[0005] Occlusion in medical images differs significantly from occlusion in ordinary natural images. Occlusion in natural images is relatively concentrated, often affecting only a small area; for example, if a face is obscured, only that area needs to be repaired during restoration. However, occlusion in medical images is often caused by instruments blocking various lights or rays. The result is often artifacts covering the obscured object and a large area behind it, especially in OCT images. (See [link to OCT image documentation]). Figure 4 Before restoration, large areas of linear, strip-shaped, and ray / radial artifacts will appear in OCT images.

[0006] Existing methods for restoring occlusion in ordinary natural images are not suitable for occlusion artifacts in medical images. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a deep learning-based solution that accurately repairs occlusion artifacts in complex medical images by adding guidance information during the repair process.

[0008] The technical solution adopted by the present invention to solve the above-mentioned technical problems is a method for repairing occlusion artifacts in medical images, including a training process and an inference process;

[0009] The training process includes a pre-training phase and a training phase;

[0010] Pre-training phase:

[0011] Collect artifact-free images and generate artifact images corresponding to the artifact-free images;

[0012] The artifact-free image and the artifact image are input into the occlusion perception feature extraction module. The occlusion perception feature extraction module outputs the extracted occlusion perception features of the artifact image. The occlusion perception features of the artifact image are then used as the restoration guide for the image restoration module in the pre-training stage.

[0013] The artifact image and the artifact image occlusion perception features used as guidance for restoration are input into the image restoration module, and the image restoration module outputs the occluded restored image; the occluded restored image is close to the artifact-free image as the pre-training target;

[0014] After pre-training is completed, the parameters of the occlusion perception feature extraction module are frozen, and the training phase begins.

[0015] Training phase:

[0016] The artifact-free image and the artifact image are input into the occlusion perception feature extraction module, and the occlusion perception feature extraction module outputs the extracted occlusion perception features of the artifact image; at the same time, the artifact image is input into the generation module, and the generation module outputs the generated artifact image occlusion perception features as the restoration guide for the image restoration module in the training phase.

[0017] The artifact image and the artifact image occlusion perception features used as restoration guides are input into the image restoration module, and the image restoration module outputs the occlusion restored image.

[0018] The training targets are: the generated artifact image occlusion perception features are close to the extracted artifact image occlusion perception features, and the occlusion repaired image is close to the artifact-free image.

[0019] Reasoning process:

[0020] The medical image to be repaired is input into the occlusion artifact detection model, the occlusion artifact detection model outputs the predicted artifact region, and then the artifact region mask is generated.

[0021] The medical image to be repaired and the artifact region mask are fused and then input into the trained generation module and image inpainting module respectively; the generation module outputs the generated artifact image occlusion perception features as the repair guide for the image inpainting module in the inference process; the image inpainting module outputs the occluded repaired image.

[0022] Compared to manual methods, simple interpolation, and traditional image processing techniques, this invention proposes an automatic method for repairing occlusion artifacts in medical images, along with a corresponding network training method, enabling accurate detection and high-quality repair of artifact regions. This invention innovatively combines image + mask input with the use of artifact image occlusion perception features to guide the repair process and output the repaired result. Furthermore, when processing the frame to be repaired, it considers leveraging the contextual information of neighboring frames to generate repaired images with good visual continuity. Simultaneously, this invention constructs a large-scale multi-source artifact database, improving the generalization performance and clinical applicability of the method. Moreover, this method is applicable to various medical image modalities and compatible with multiple advanced generative models, demonstrating broad technical applicability and promising clinical application prospects.

[0023] The beneficial effects of this invention are: it can automatically detect and repair occlusion artifacts in medical images; by collecting a dataset of artifact-free medical images, a database of occlusion artifacts is constructed, and a generative model is used to guide the training of the repair network, thereby achieving efficient and accurate artifact repair. This significantly improves the quality of medical image restoration, providing doctors with clearer and more accurate image information, thereby improving diagnostic and treatment outcomes. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of an occlusion artifact repair framework proposed in the embodiment.

[0025] Figure 2 This is a schematic diagram of the pre-training of a perceptual feature extraction module and an image restoration module proposed in the embodiment.

[0026] Figure 3 This is a schematic diagram of a training artifact image occlusion perception feature generation and image restoration module proposed in the embodiment.

[0027] Figure 4 This is an example of the application of the occlusion artifact repair method of the present invention on intravascular OCT images. Detailed Implementation

[0028] The technical solutions in the embodiments of the present invention will now be clearly and completely described in conjunction with the accompanying drawings.

[0029] The following is combined with Figure 1 A method for repairing occlusion artifacts in medical images, the reasoning process of which includes:

[0030] S1. Obtain the medical image sequence to be repaired from the medical imaging process or database.

[0031] S2. Use the trained occlusion artifact detection model to detect artifact regions and post-process them to obtain artifact masks, where the artifact mask is a 0-1 matrix, where 1 represents the artifact region and 0 represents the non-artifact region.

[0032] Specifically, the occlusion artifact detection model described in S2 can directly detect artifact regions; it can also detect medical devices or other obstructions causing occlusion, and then include the artifact regions through post-processing based on the modal characteristics of medical imaging. The artifact detection model network is a pre-trained deep learning module / model, which is in a parameter-frozen state during inference.

[0033] S3. Transfer the medical image (sequence) to be processed (IMG) RAW and its corresponding artifact mask (sequence)

[0034] Mask Artifact Inputting the data into a trained occlusion artifact inpainting network, the predicted artifact-free image (sequence) is displayed in IMG format. Pred .

[0035] The occlusion artifact inpainting network comprises an artifact image occlusion perception feature extraction module, a generation module, and an image inpainting module. The artifact image occlusion perception feature extraction module is only used during the training process; its role is to help the generation module learn how to generate artifact image occlusion perception features.

[0036] Specifically, in the occlusion artifact inpainting network, guided by the generation conditions, the generation model outputs the occlusion perception features of the generated artifact image and inputs them into the image inpainting module; simultaneously, the medical image (sequence) IMG to be processed is... RAW and its corresponding artifact mask (sequence) Mask Artifact The image is input into the image inpainting module, where, guided by the artifact image occlusion perception features, the occlusion artifacts are repaired. The occlusion perception feature generation condition for single-frame image inpainting is the frame to be repaired and its adjacent frames; the occlusion perception feature generation condition for 3D medical image data inpainting is the 3D medical image data to be repaired.

[0037] The method of this invention can repair both single medical images and directly repair 3D medical image data (video / image sequences). In medical images, the positions of occlusion artifacts may not be the same in adjacent frames N and N+1. For example, the occlusion artifact in frame N may be at position 'a', while there may be no occlusion artifact at position 'a' in frame N+1. Considering the continuity of 3D information, the data at position 'a' in frame N+1 should contribute to the repair of the artifact at position 'a' in frame N. Therefore, when repairing frame N, its neighboring frames should be considered as part of the occlusion artifact perception features, thereby fully utilizing contextual information to generate a repaired image with good visual continuity.

[0038] In the inference process, the generative model and image inpainting module in the occlusion artifact inpainting network are both pre-trained deep learning modules / models with their parameters frozen.

[0039] S4. Based on the artifact mask information, Mask Artifact fused with the original image (sequence) containing artifacts (IMG). RAW And the predicted artifact-free image (sequence), i.e., the original image (sequence) IMG. RAW Regions unaffected by artifacts are preserved, and the predicted artifact-free image (sequence) IMG is used. Pred Content is replaced in artifact areas to generate the final restored image (sequence) IMG. Repaired IMG Repaired =(1-Mask) Artifact *IMG RAW +Mask Artifact *IMG Pred ;

[0040] S5. Complete the repair and save the artifact mask. Artifact and repair results (IMG) Reoaired This is for downstream tasks to retrieve and use.

[0041] Specifically, the training process of the occlusion artifact repair network is as follows:

[0042] Pre-training phase:

[0043] Step 1) Collect artifact-free medical image datasets and construct a database of occlusion artifact masks;

[0044] Step 2) Generate a medical image containing occlusion artifacts;

[0045] Step 3) Based on the artifact-free image (sequence) and the generated artifact image (sequence), train the artifact image occlusion perception feature extraction module, and use the artifact image occlusion perception feature extraction extracted during the training process to pre-train the image restoration module.

[0046] Training phase:

[0047] Step 4) Train a generative model to generate occlusion-aware features for artifact images;

[0048] Step 5) Guided by the occlusion perception features of the generated artifact image, the image inpainting module repairs the occluded image;

[0049] Step 6) Calculate the loss between the predicted artifact-free image and the real artifact-free image, and perform fine-tuning iterative training on the image inpainting module.

[0050] Specifically, step 1) involves constructing a database of occlusion artifact masks in two ways:

[0051] Method 1: Use manual annotation or a pre-trained occlusion artifact detection model to identify artifacts in real medical imaging scenarios.

[0052] Method 2: Based on the imaging method, use code to simulate occlusion artifacts that may exist during the imaging process. The code should have the same characteristics as the real occlusion artifacts, such as length, width, and positional distribution.

[0053] Large-scale artifact data is generated in the above manner, and each artifact is assigned a unique identifier to facilitate indexing.

[0054] Specifically, step 2) of the process for generating medical images containing occlusion artifacts includes:

[0055] 2-1) Extract an artifact-free image (sequence) sequentially from an artifact-free medical image dataset;

[0056] 2-2) Randomly select an artifact mask from the artifact database;

[0057] 2-3) Fusing the artifact-free image (sequence) and the artifact mask, i.e., setting the regions in the image corresponding to the artifact mask to 0, thereby generating an image (sequence) containing the artifact. Artifact :

[0058] Image Artifact =Image·copy()

[0059] Image Artifact [Mask==1]=0

[0060] Image.copy() represents copying the artifact image (sequence) Image. Artifact [Mask==1] represents Image Artifact The artifact mask region in the image.

[0061] Since the occluded area is usually a shadowy area with no signal or a noisy area with low brightness, the most common way is to set it to 0 based on the mask; during the inference process, the real artifact image will also be set to 0 based on the detected mask; the example is only one processing method, for example, it can also be set to 255 or random noise.

[0062] like Figure 2 As shown, step 3) includes the following steps:

[0063] 3-1) Image Artifact The corresponding Mask is merged along the channel dimension to generate the artifact image. Input ;

[0064] Image Input The corresponding artifact-free image (sequence) is input into the artifact image occlusion perception feature extraction module, and the artifact image occlusion perception feature extraction module outputs the extracted artifact image occlusion perception features.

[0065] Optionally, the extracted artifact image occlusion perception features are fusion features of the occluded image and the original artifact-free image.

[0066] 3-3) Image Input The input is fed into the image inpainting module, which, guided by the extracted artifact image occlusion perception features, generates a predicted occlusion-inpainted image (Image) for the artifact-free image (sequence). Pred ;

[0067] 3-4) Calculate Image Pred The reconstruction loss of the original artifact-free image (sequence) is used, and then iteratively updated in reverse to update the parameters of the artifact image occlusion perception feature extraction module and the image inpainting module; the occlusion-inpainted image is close to the artifact-free image as the pre-training target;

[0068] 3-5) Repeat steps 3-1) to 3-4) to train the artifact image occlusion perception feature extraction module and pre-train the image restoration module.

[0069] Specifically, such as Figure 3 As shown, the training phase includes the following steps:

[0070] 4-1) Freeze the parameters of the artifact image occlusion perception feature extraction module trained in step 3), and then freeze the Image. Input The corresponding artifact-free image (sequence) is input into the artifact image occlusion perception feature extraction module to obtain the extracted artifact image occlusion perception features; simultaneously, the Image... InputThe input is fed into the generation module, which outputs the generated artifact image occlusion perception features.

[0071] Optionally, the generative model used to generate occlusion-aware features of artifact images is a conditional generative model, specifically including, but not limited to, generative adversarial networks and their variants, diffusion models and their variants, and autoregressive models and their variants; Image Input Image is input as conditional information into the generative model. Input The occlusion perception feature can be the occluded image (sequence) or its neighboring frames (sequence), or a combination of the occluded image (sequence) and its neighboring frames (sequence). Here, neighboring frames refer to frames from the Nxth frame to the N+xth frame, where the value of x needs to be modified according to requirements. The occlusion perception feature generation condition for single-frame image inpainting is the frame to be inpainted and its neighboring frames; the occlusion perception feature generation condition for 3D medical image data inpainting is the 3D medical image data to be inpainted.

[0072] 4-2) Calculate the similarity loss between the extracted and generated artifact image occlusion perception features.

[0073] 5-1) Image Inpur The input is fed into the image inpainting module, which, guided by the occlusion-sensing features of the generated artifact images, generates a predicted image for the artifact-free image (sequence). Pted ;

[0074] 5-2) Calculate Image Pred The reconstruction loss of the original artifact-free image (sequence);

[0075] 6) Based on the loss calculated in steps 4-2) and 5-2), iterate backward to update the parameters of the generation module and the image inpainting module, and complete the training phase.

[0076] Optionally, the generated model can be trained independently or jointly with the image inpainting module. During training, a feature similarity function is calculated between the occlusion-aware features of the generated artifact image and the extracted artifact-aware image, including distance-based loss, contrast-based loss, distribution-matching loss, and combinations thereof. The loss between the predicted artifact-free image and the real artifact-free image includes, but is not limited to, pixel-by-pixel reconstruction loss, adversarial loss, structural similarity loss, and texture-aware loss.

[0077] Application example:

[0078] This application example demonstrates the repair of guidewire and stent obstruction artifacts on intravascular OCT images.

[0079] like Figure 4As shown in (a), the calcified plaques within the blood vessel are obscured by the guidewire in the original image. The restored image helps clinicians to more accurately assess parameters of the calcification, such as the angle of the calcification; Figure 4 As shown in (b), the proposed solution successfully repaired the area obscured by the guidewire and stent.

[0080] The embodiments described above are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

Claims

1. A method for repairing occlusion artifacts in medical images, comprising a training process and an inference process, characterized in that, The training process includes a pre-training phase and a training phase; Pre-training phase: Collect artifact-free images and generate artifact images corresponding to the artifact-free images; The artifact-free image and the artifact image are input into the occlusion perception feature extraction module. The occlusion perception feature extraction module outputs the extracted occlusion perception features of the artifact image. The occlusion perception features of the artifact image are then used as the restoration guide for the image restoration module in the pre-training stage. The artifact image and the artifact image occlusion perception features used as restoration guides are input into the image restoration module, and the image restoration module outputs the occlusion restoration image from the pre-training stage. The pre-training target is to obtain occlusion-repaired images that are close to artifact-free images during the pre-training phase. After pre-training is completed, the parameters of the occlusion perception feature extraction module are frozen, and the training phase begins. Training phase: The occlusion perception feature extraction module extracts both artifact-free and artifact-image data after freezing parameters. The occlusion perception feature extraction module outputs the extracted artifact image occlusion perception features. Simultaneously, the artifact image is used as conditional information input to the generation module, which is a conditional generation model. The generation module outputs the generated artifact image occlusion perception features as guidance for the image inpainting module during the training phase. The artifact image and the artifact image occlusion perception features used as restoration guides are input into the image restoration module, and the image restoration module outputs the occlusion restoration image from the training phase. The training targets are: the generated artifact image occlusion perception features are close to the extracted artifact image occlusion perception features, and the occlusion repaired image during the training phase is close to the artifact-free image. Reasoning process: The medical image to be repaired is input into the occlusion artifact detection model, the occlusion artifact detection model outputs the predicted artifact region, and then the artifact region mask is generated. The medical image to be repaired and the artifact region mask are fused together and then input into the trained generation module and image repair module respectively. The generation module outputs the occlusion perception features of the generated artifact image as a guide for the image inpainting module in the inference process; the image inpainting module outputs the final occlusion-repaired image.

2. The method as described in claim 1, characterized in that, During the inference process, after the image inpainting module outputs the occlusion repair image, it also includes determining the artifact region based on the artifact region mask, replacing the artifact region in the medical image to be repaired with the same region in the occlusion repair image, and the image after the replacement is the final artifact-free medical image.

3. The method as described in claim 1, characterized in that, Based on the constructed database of artifact masks, artifact images corresponding to artifact-free images are generated.

4. The method as described in claim 3, characterized in that, A database of artifact masks is built by identifying artifacts in real medical imaging scenarios using manual annotation or by using a pre-trained occlusion artifact detection model.

5. The method as described in claim 3, characterized in that, Based on the imaging method, code is used to simulate artifacts in the imaging process to build a database of artifact masks.

6. The method as described in claim 5, characterized in that, Code mimicry is accomplished by generating artifacts that have the same length, width, and positional distribution characteristics as real artifacts.

7. The method as described in claim 1, characterized in that, Conditional generative models include generative adversarial networks, diffusion models, or autoregressive models.

Images