Training method of generative adversarial network, c-arm image repairing method and device

Abstract

The application provides a training method of a generative adversarial network, a C-arm image repairing method and device. The method can include: constructing a generative adversarial network; obtaining a sample C-arm image and cropping the sample C-arm image into a plurality of sample sub-images; generating a training data set according to the sample sub-images, the training data set including at least one pair of matched sample sub-image and mask sub-image, the mask sub-image being obtained by superimposing the sample sub-image with a mask pattern used for simulating the circular shadow; training the generative adversarial network by using the training data set to obtain a trained generative adversarial network, the trained generative adversarial network being used for image repairing of a C-arm image to be repaired. Through the technical solution of the application, the required video memory space for training the generative adversarial network can be reduced, and the image repairing of the C-arm image can be implemented based on the trained generative adversarial network with a better effect.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, and in particular to training methods for generative adversarial networks, and methods and apparatus for C-arm image restoration. Background Technology

[0002] Due to the limitations of C-arm imaging technology, multiple circular shadows may appear in C-arm images. These shadows often prevent the complete display of structures such as bones and muscles in the images, severely impacting the medical diagnoses made by doctors based on C-arm images.

[0003] Currently, although generative adversarial networks (GANs) can be used to repair shadows in images, GAN-based image inpainting methods require a large amount of video memory and have high hardware requirements. For example, a GPU with 12GB of video memory can usually only support image inpainting of images with a resolution of 256*256, while C-arm images usually have a higher resolution, making it difficult to use GANs to inpaint C-arm images. Summary of the Invention

[0004] In view of this, this application provides a training method for generative adversarial networks, a method and apparatus for C-arm image restoration.

[0005] Specifically, this application achieves its purpose through the following technical solution:

[0006] According to a first aspect of this application, a method for training a generative adversarial network for C-arm machine image inpainting is proposed, comprising:

[0007] A generative adversarial network is constructed; the generative adversarial network is used to repair the input original image with circular shadows and output the repaired image with the circular shadows removed;

[0008] Acquire sample C-arm emulator images and crop the sample C-arm emulator images into several sample sub-images;

[0009] A training dataset is generated based on the sample sub-images. The training dataset includes at least one pair of matched sample sub-images and mask sub-images. The mask sub-images are obtained by superimposing the sample sub-images with a mask pattern used to simulate the circular shadow.

[0010] The generative adversarial network is trained using the training dataset to obtain a trained generative adversarial network, which is then used to perform image restoration on the C-arm machine image to be restored.

[0011] According to a second aspect of this application, a C-arm machine image inpainting method based on generative adversarial networks is proposed, comprising:

[0012] Obtain images of the C-arm machine to be repaired;

[0013] The C-arm machine image to be repaired is cropped to obtain several sub-images to be repaired;

[0014] The sub-image to be repaired is input into the generative adversarial network (GAN) to process the sub-image to be repaired and obtain the repaired sub-image, wherein the GAN is trained by the training method of the GAN described in any one of the first aspects.

[0015] The repaired sub-images are then stitched together to obtain the complete repaired image.

[0016] According to a third aspect of this application, a training device for a generative adversarial network for C-arm machine image inpainting is proposed, comprising:

[0017] A network construction unit is used to construct a generative adversarial network; the generative adversarial network is used to repair an input original image with circular shadows and output a repaired image with the circular shadows removed.

[0018] The sample acquisition unit is used to acquire sample C-arm emulator images and crop the sample C-arm emulator images into several sample sub-images;

[0019] A dataset generation unit is configured to generate a training dataset based on the sample sub-images. The training dataset includes at least one pair of matching sample sub-images and mask sub-images. The mask sub-images are obtained by superimposing the sample sub-images with a mask pattern used to simulate the circular shadow.

[0020] The network training unit is used to train the generative adversarial network using the training dataset to obtain the trained generative adversarial network, which is used to perform image restoration on the C-arm machine image to be restored.

[0021] According to a fourth aspect of this application, a C-arm machine image inpainting device based on generative adversarial networks is proposed, comprising:

[0022] Image acquisition unit, used to acquire images of the C-arm machine to be repaired;

[0023] The image cropping unit is used to crop the C-arm machine image to be repaired to obtain several sub-images to be repaired;

[0024] An image inpainting unit is configured to input the sub-image to be inpainted into the generative adversarial network (GAN) to process the sub-image to be inpainted through the GAN to obtain an inpainted sub-image, wherein the GAN is trained by the training method of the GAN described in any one of the first aspects.

[0025] The image stitching unit is used to stitch the repaired sub-images together to obtain a complete repaired image.

[0026] According to a fifth aspect of this application, an electronic device is provided, comprising:

[0027] processor;

[0028] Memory used to store processor-executable instructions;

[0029] The processor executes the executable instructions to implement the method described in the embodiments of the first and second aspects above.

[0030] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, having stored thereon computer instructions that, when executed by a processor, implement the steps of the methods described in the embodiments of the first and second aspects above.

[0031] As can be seen from the technical solution provided in this application, this application crops high-resolution sample C-arm android images to train a generative adversarial network (GAN) using the cropped sub-images. During image inpainting, the C-arm android image to be inpainted is cropped into multiple sub-images, each of which is then inpainted separately. The inpainted sub-images are then stitched together to obtain the inpainted image of the entire C-arm android image. This technical solution makes it possible to train a GAN for inpainting high-resolution C-arm android images and enables effective image inpainting of C-arm android images based on the trained GAN. Attached Figure Description

[0032] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0033] Figure 1 This is a flowchart illustrating a method for training a generative adversarial network for C-arm image inpainting according to an exemplary embodiment of this application;

[0034] Figures 2a-2b It is a pair of matched sample sub-images and mask sub-images shown according to an exemplary embodiment of this application;

[0035] Figure 3 This is a flowchart illustrating an image restoration method for a C-arm robot based on a generative adversarial network, according to an exemplary embodiment of this application.

[0036] Figure 4 This is a schematic diagram of image cropping of a sub-image to be repaired according to an exemplary embodiment of this application;

[0037] Figure 5 This is a schematic diagram of the portion used for image stitching in the repaired sub-image to be repaired, according to an exemplary embodiment of this application;

[0038] Figure 6 This is a schematic diagram of image stitching according to an exemplary embodiment of this application;

[0039] Figures 7a-7b This is an exemplary embodiment of the present application showing an image to be replaced and its corresponding sub-image to be repaired;

[0040] Figure 8 This is a schematic diagram of a training electronic device for a generative adversarial network for C-arm machine image inpainting, according to an exemplary embodiment of this application.

[0041] Figure 9 This is a block diagram illustrating a training apparatus for a generative adversarial network for C-arm image inpainting according to an exemplary embodiment of this application;

[0042] Figure 10 This is a schematic diagram of an electronic device for C-arm image restoration based on a generative adversarial network, according to an exemplary embodiment of this application;

[0043] Figure 11 This is a block diagram illustrating an image restoration apparatus for a C-arm machine based on a generative adversarial network, according to an exemplary embodiment of this application. Detailed Implementation

[0044] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0045] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0046] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0047] The embodiments of this application will now be described in detail.

[0048] Figure 1 This is a flowchart illustrating a method for training a generative adversarial network for C-arm image inpainting according to an exemplary embodiment of this application. Figure 1 As shown, the method may include the following steps:

[0049] Step 102: Construct a generative adversarial network; the generative adversarial network is used to repair the input original image with circular shadows and output a repaired image with the circular shadows removed.

[0050] Generative Adversarial Networks (GANs) are deep learning models consisting of two networks: a generator network and a discriminator network. The generator network produces random samples that resemble real samples and uses them as fake samples to deceive the discriminator network. The discriminator network judges the output of the generator network and learns to distinguish between real samples and fake samples generated by the generator network. GANs utilize the game between the generator and discriminator networks to continuously improve network performance, eventually reaching a balance where a high-quality generator network capable of generating relatively realistic samples and a discriminator network with strong judgment ability are obtained.

[0051] In the technical solution of this application, based on the inherent characteristic that C-arm emulator images usually contain many circular shadows, this application constructs a generative adversarial network to repair the input original image with circular shadows and outputs a repaired image with the circular shadows removed, so as to synthesize replacement content of circular shadows from the given C-arm emulator image to be repaired, making the repaired C-arm emulator image visually realistic and semantically reasonable. The specific construction method of the generative adversarial network can be referred to the records in related technologies, and will not be elaborated here.

[0052] In one embodiment, since generative adversarial networks (GANs) used for image inpainting in related technologies typically have three RGB input channels, and the C-arm image to be repaired in this application can be considered a grayscale image, the input of the GAN can be constructed as a single channel to input the grayscale C-arm image to be repaired. Similarly, this application can also directly modify the three-channel input GAN, adjusting its network parameters to change its input channel from three RGB channels to a single channel. By setting the input channel of the GAN to only the single channel required for the C-arm image, this application can reduce unnecessary data computation, improve the training speed of the GAN, and reduce the GPU memory requirement by two-thirds, thus lowering the hardware requirements for GAN training.

[0053] Step 104: Obtain sample C-arm images and crop the sample C-arm images into several sample sub-images.

[0054] The sample C-arm images are pre-acquired, intact, shadow-free, high-quality C-arm images. As mentioned earlier, due to the hardware limitations of generative adversarial networks (GANs), GPUs generally cannot train on the entire C-arm image. Therefore, each sample C-arm image needs to be cropped into smaller sub-images until it can be processed by the GPU. In one embodiment, the resolution of the sample sub-images to be cropped can be set based on the hardware attributes of the electronic device performing the GAN training. For example, if the maximum image resolution that the GAN can train on is 512*512, and the sample C-arm image resolution is 1024*1024, then the image can be cropped along the horizontal and vertical midlines, dividing it into four 512*512 sample sub-images. By cropping the sample C-arm machine images, on the one hand, the generative adversarial network (GAN) can support the training process of repairing high-resolution C-arm machine images; on the other hand, even if the GPU memory of the electronic device performing the GAN training is large enough to support processing the entire sample C-arm machine image, cropping the sample C-arm machine image can reduce the memory usage during subsequent training. For example, cropping a 1024*1024 resolution sample C-arm machine image into a 512*512 resolution sample sub-image can reduce memory usage by three-quarters during subsequent training.

[0055] Step 106: Generate a training dataset based on the sample sub-images. The training dataset includes at least one pair of matching sample sub-images and mask sub-images. The mask sub-images are obtained by superimposing the sample sub-images with a mask pattern used to simulate the circular shadow.

[0056] In generative adversarial networks (GANs), the training dataset includes not only the sample sub-images obtained in step 104 (which serve as real samples), but also shadow-containing images to be repaired, which the generative network can process to generate fake samples. In the technical solution of this application, a mask pattern can be used to replace the shadows in the C-arm image. A mask image simulating the shadows of the C-arm image is superimposed on the sample sub-image to obtain a mask sub-image. During training, the generative network can process this mask sub-image to use the repaired mask sub-image as a fake sample, while the discriminative network needs to distinguish between the repaired mask sub-image and the sample sub-image. Processing the sample sub-images yields the training dataset required for the GAN. This training dataset includes at least one set of image pairs, each pair consisting of one sample sub-image and one matching mask sub-image.

[0057] In one embodiment, since the shadows to be repaired on the C-arm image are circular and of roughly the same size, the mask patterns superimposed on the sample sub-images are not completely random. For any given sample sub-image, the multiple mask patterns superimposed on it are all circular and of the same size. However, since the size and position of the shadows to be repaired on the C-arm image are not fixed, the size and position of the mask patterns superimposed on different sample sub-images are random when generating mask sub-images. By generating mask patterns according to specific rules to replace the shadows in the image, the randomness of the shadows to be repaired is reduced, which can accelerate the convergence speed of the generative adversarial network.

[0058] Furthermore, since the circular shadows in the C-arm image to be repaired are not regularly distributed, the cropping path may pass through a circular shadow and cut it in half when cropping the C-arm image. Therefore, this application can superimpose several semi-circular mask patterns on the edge of the sample sub-image to simulate the above situation. Figure 2a and 2b For a pair of matched sample sub-images and mask sub-images, where, Figure 2a For a sample sub-image, Figure 2b To overlay the mask sub-image after the mask sub-pattern used to simulate a circle, Figure 2b The white dots in the image represent the mask patterns generated according to specific rules. By overlaying semi-circular mask patterns along the edges of sample sub-images to simulate the shadow states that may occur after cropping, the restoration effect of subsequent image inpainting of the C-arm image to be restored using this generative adversarial network can be improved.

[0059] Step 108: Train the generative adversarial network using the training dataset to obtain the trained generative adversarial network, which is used to perform image restoration on the C-arm machine image to be restored.

[0060] Similar to the training method of Generative Adversarial Networks (GANs) in related technologies, a GAN consists of a generator network and a discriminator network. The generator network receives a repaired mask sub-image as input, and then the repaired mask sub-image and its corresponding sample sub-image serve as the two inputs to the discriminator network. The discriminator network feeds back its judgment of the real data and the data generated by the generator network to the generator network. After receiving the judgment feedback, the generator network generates better fake data than before. The discriminator network, by learning from the previous real and fake data, judges the fake data generated by the generator network again. This process iterates and optimizes until the loss function converges, at which point training ends, resulting in a well-trained GAN.

[0061] In one embodiment, the generative adversarial network (GAN) can be pre-trained before formal training. A pair of sample sub-images and a mask sub-image are input into the generative network for pre-training, using the following loss function:

[0062]

[0063] In formula (1), X represents the unmasked raw data input into the model, and G(X) represents the output after the model is input with a mask. This represents pixel-wise multiplication, and MASK represents the mask layer. Based on this loss function, the generative adversarial network (GAN) can be pre-trained, and the trained model parameters can be saved as the initialization parameters of the GAN. This allows the GAN to be trained together with the pre-trained generative network and the randomly initialized discriminative network, thereby effectively improving the convergence rate of the GAN.

[0064] As can be seen from the technical solution provided in this application above, this application crops high-resolution sample C-arm machine images to train a generative adversarial network using the cropped sub-images, thereby making it possible to train a generative adversarial network for repairing high-resolution C-arm machine images and effectively reducing the video memory space required for training the generative adversarial network.

[0065] Figure 3 This is a flowchart illustrating an image inpainting method for a C-arm robot based on a generative adversarial network, according to an exemplary embodiment of this application. Figure 3 As shown, the method may include the following steps:

[0066] Step 302: Obtain the image of the C-arm machine to be repaired.

[0067] Step 304: Crop the C-arm machine image to be repaired to obtain several sub-images to be repaired.

[0068] As mentioned earlier, due to the limitations of the hardware environment for Generative Adversarial Networks (GANs), GPUs generally cannot perform image inpainting on the entire C-arm image. Furthermore, GANs typically achieve better inpainting results when working with images of the same size as their training dataset. Therefore, for the C-arm image to be inpainted, it needs to be cropped into smaller sub-images corresponding to the image resolution set by the GAN. For example, if the GAN's resolution is 512*512, and the C-arm image to be inpainted has a resolution of 1024*1024, it can be cropped along the horizontal and vertical midlines, dividing it into four equal 512*512 sub-images. By cropping the C-arm image into smaller sub-images, subsequent image inpainting using a pre-trained GAN becomes easier.

[0069] In one embodiment, considering that the circular shadow in the C-arm image to be repaired may be located exactly on the cropping path and cropped into two sub-images to be repaired, potentially causing discontinuity in the final stitched image, the C-arm image to be repaired can be cropped multiple times. First, based on the image resolution set by the adopted generative adversarial network, the C-arm image to be repaired is cropped into multiple first-type sub-images to be repaired. Second, based on the cropping path of the first cropping, the C-arm image to be repaired is re-cropped according to the image resolution set by the adopted generative adversarial network to obtain at least one second-type sub-image to be repaired, which covers the edge junction area of ​​multiple adjacent first-type sub-images to be repaired. Furthermore, the C-arm image to be repaired can be re-cropped according to the image resolution set by the adopted generative adversarial network to obtain several third-type sub-images to be repaired, which cover the vertex junction area of ​​multiple adjacent first-type sub-images to be repaired. For example, the generative adversarial network used in this application for C-arm image inpainting has an image resolution of 512*512. Figure 4 The diagram shows an image cropping schematic for a 1024*1024 resolution C-arm image to be repaired. The images within the dashed lines are the cropped sub-images to be repaired. Sub-images 401, 403, 407, and 409 are four first-type sub-images cropped along the horizontal and vertical center lines of the C-arm image to be repaired. Sub-images 402, 404, 406, and 408 are four second-type sub-images re-cropped along the first cropping path (the horizontal and vertical center lines), covering the edge intersection areas of each pair of first-type sub-images 401, 403, 407, and 409. Sub-image 405 is a third-type sub-image covering the vertex intersection area of ​​the first-type sub-images.

[0070] Step 306: Input the sub-image to be repaired into the generative adversarial network (GAN) to process the sub-image to be repaired and obtain the repaired sub-image.

[0071] The generative adversarial network (GAN) is trained using the GAN training method described in any of the above embodiments. In this application, the specific implementation of image inpainting using the trained GAN can be found in related art, and will not be repeated here.

[0072] Step 308: Stitch the repaired sub-images together to obtain the complete repaired image.

[0073] In one embodiment, the repaired first-type, second-type, and third-type sub-images to be repaired can be stitched together separately. First, the repaired first-type sub-images to be repaired are stitched together to obtain a first stitched image; then, the repaired second-type sub-images to be repaired can be stitched together and their corresponding content in the first stitched image can be replaced to obtain a second stitched image; finally, the repaired third-type sub-images to be repaired are used to replace their corresponding content in the second stitched image to obtain a complete repaired image.

[0074] Furthermore, for the repaired second and third type of sub-images to be repaired, only a portion of the images can be selected for splicing and replacement. The repaired second type of sub-image to be repaired can be cropped into a first type of replacement image with a width twice the diameter of the circular shadow in the C-arm crane image to be repaired, and the portion of the first spliced ​​image corresponding to the edge intersection area of ​​the first type of sub-image to be repaired can be replaced with the first type of replacement image; similarly, the repaired third type of sub-image to be repaired can be cropped into a second type of replacement image with both an image width and length twice the diameter of the circular shadow in the C-arm crane image to be repaired, and the portion of the first spliced ​​image corresponding to the vertex intersection area of ​​the first type of sub-image to be repaired can be replaced with the second type of replacement image. Figure 4 Taking the images to be repaired as an example, for the second type of sub-images 402, 404, 406, and 408 and the third type of sub-image 405, partial content from their repaired images can be selected for image stitching, such as... Figure 5 As shown, the shaded areas represent the parts of the repaired sub-images that will be used for image stitching. Images 501, 503, 507, and 509 are the first type of repaired sub-images; images 502, 504, 506, and 508 are the first type of images to be replaced; and image 505 is the second type of image to be replaced. Figure 5 The image shown is an example of the first type of repaired sub-image and the image to be replaced, used for image stitching. Figure 6This is a schematic diagram of image stitching. The repaired first-class sub-images 501, 503, 507, and 509 can be stitched together as the bottom layer to obtain a first stitched image with a resolution of 1024*1024. However, the stitched first image may have blurry and discontinuous areas at the stitching junctions. Therefore, it is necessary to supplement it with first-class replacement images and second-class replacement images. The first-class replacement images 502, 504, 506, and 508 are stitched over the corresponding positions on the first stitched image, and the covered content is replaced to obtain the second stitched image. Then, the second-class replacement image 505 is overlaid on the corresponding position on the second stitched image, and the covered content is replaced to obtain the final complete repaired image.

[0075] In one embodiment, although the cropping radius of the image to be replaced can be set by the user according to the size of the circular shadow in the C-arm image to be repaired, the edges of the cropped first and second types of images to be replaced are still obtained by repairing incomplete circular shadows. Therefore, the position information of the circular shadow in the C-arm image to be repaired can be determined by Hough transform or other methods. The method for determining the position information of the circular shadow can be referred to the relevant records in related technologies, which will not be repeated here. With the determined position information of the circular shadow, the images used for stitching can be processed in a targeted manner, so that the first type of image to be replaced contains only complete circular shadows in the corresponding area of ​​the second type of sub-image to be repaired, and the second type of image to be replaced contains only complete circular shadows in the corresponding area of ​​the third type of sub-image to be repaired. Figure 7a As shown in the figure, the sub-image to be repaired contains an image to be replaced. The edges of this sub-image include incomplete circular shadows. During image stitching, methods such as cropping the area of ​​these incomplete circular shadows can be used to allow the underlying image of the incomplete circular shadow area to replace the content of the image to be replaced. Figure 7b The image shown is the replacement image after the incomplete circular shadow portion has been cropped and removed. When image stitching is performed using this image, the content of the corresponding complete circular shadow repaired in the lower layer can be displayed at the position of the incomplete circular shadow, thereby improving the repair effect of the final complete repaired image.

[0076] As can be seen from the technical solution provided in this application above, this application achieves C-arm image restoration based on generative adversarial networks by cropping the C-arm image to be restored into multiple sub-images, performing image restoration on each sub-image separately through generative adversarial networks, and stitching the restored sub-images together to obtain the restored image of the entire C-arm image.

[0077] Corresponding to the above method embodiments, this specification also provides an embodiment of an apparatus.

[0078] Figure 8 This is a schematic diagram of the structure of a training electronic device for a generative adversarial network for C-arm image inpainting, according to an exemplary embodiment of this application. (Reference) Figure 8 At the hardware level, the electronic device includes a processor 802, an internal bus 804, a network interface 806, memory 808, and non-volatile memory 810, and may also include other hardware required for business operations. The processor 802 reads the corresponding computer program from the non-volatile memory 810 into the memory 808 and then runs it. Of course, in addition to software implementation, this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0079] Figure 9 This is a block diagram illustrating a training apparatus for a generative adversarial network for C-arm image inpainting according to an exemplary embodiment of this application. (Refer to...) Figure 9 The device includes a network construction unit 902, a sample acquisition unit 904, a dataset generation unit 906, and a network training unit 908, wherein:

[0080] The network construction unit 902 is configured to construct a generative adversarial network; the generative adversarial network is used to repair an input original image with circular shadows and output a repaired image with the circular shadows removed.

[0081] Optionally, the input to the generative adversarial network is a single channel, used to input the C-arm machine image to be repaired in grayscale mode.

[0082] The sample acquisition unit 904 is configured to acquire sample C-arm images and crop the sample C-arm images into several sample sub-images.

[0083] The dataset generation unit 906 is configured to generate a training dataset based on the sample sub-images, the training dataset including at least one pair of matching sample sub-images and mask sub-images, the mask sub-images being obtained by superimposing the sample sub-images with a mask pattern used to simulate the circular shadow.

[0084] Optionally, the mask pattern used in any mask sub-image includes at least one of the following: a circular mask, a semi-circular mask having the same radius as the circular mask and located at the edge of the any mask sub-image.

[0085] The network training unit 908 is configured to train the generative adversarial network using the training dataset to obtain a trained generative adversarial network, which is used to perform image inpainting on the C-arm machine image to be repaired.

[0086] Optionally, the above-mentioned device further includes:

[0087] The pre-training unit 910 is configured to input a pair of the sample sub-images and the mask sub-images into the generator network to pre-train the generator network.

[0088] The parameter initialization unit 912 is configured to use the pre-trained model parameters as the initialization parameters of the generator network.

[0089] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0090] Figure 10 This is a schematic diagram illustrating the structure of a C-arm image restoration electronic device based on a generative adversarial network according to an exemplary embodiment of this application. (Reference) Figure 10 At the hardware level, the electronic device includes a processor 1002, an internal bus 1004, a network interface 1006, memory 1008, and non-volatile memory 1010, and may also include other hardware required for business operations. The processor 1002 reads the corresponding computer program from the non-volatile memory 1010 into the memory 1008 and then runs it. Of course, in addition to software implementation, this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0091] Figure 11 This is a block diagram illustrating a training apparatus for a generative adversarial network for C-arm image inpainting according to an exemplary embodiment of this application. (Refer to...) Figure 11 The device includes an image acquisition unit 1102, an image cropping unit 1104, an image restoration unit 1106, and an image stitching unit 1108, wherein:

[0092] The image acquisition unit 1102 is configured to acquire images of the C-arm machine to be repaired.

[0093] The image cropping unit 1104 is configured to crop the C-arm machine image to be repaired to obtain several sub-images to be repaired.

[0094] The image inpainting unit 1106 is configured to input the sub-image to be repaired into the generative adversarial network (GAN) to process the sub-image to be repaired through the GAN to obtain the repaired sub-image, wherein the GAN is trained by the training device of the GAN described in any of the above embodiments.

[0095] The image stitching unit 1108 is configured to stitch the repaired sub-images together to obtain a complete repaired image.

[0096] Optionally, cropping the C-arm image to be repaired to obtain several sub-images to be repaired includes: cropping the C-arm image to be repaired into multiple first-type sub-images to be repaired according to the image resolution set by the generative adversarial network; re-cropping the C-arm image to be repaired according to the image resolution to obtain at least one second-type sub-image to be repaired, wherein the second-type sub-image to be repaired covers the edge junction area of ​​the multiple adjacent first-type sub-images to be repaired; and re-cropping the C-arm image to be repaired according to the image resolution to obtain several third-type sub-images to be repaired, wherein the third-type sub-images to be repaired cover the vertex junction area of ​​the multiple adjacent first-type sub-images to be repaired.

[0097] Optionally, the step of stitching the repaired sub-images together to obtain a complete repaired image includes: stitching the repaired first type of sub-images to be repaired together to obtain a first stitched image; replacing the corresponding content in the first stitched image with the repaired second type of sub-images to be repaired to obtain a second stitched image; and replacing the corresponding content in the second stitched image with the repaired third type of sub-images to be repaired to obtain a complete repaired image.

[0098] Optionally, replacing the corresponding content in the first stitched image with the repaired second type of sub-image to be repaired to obtain the second stitched image includes: cropping the repaired second type of sub-image to be repaired into a first type of image to be replaced, and replacing the part of the first stitched image corresponding to the edge intersection area with the first type of image to be replaced, wherein the image width of the first type of image to be replaced is twice the diameter of the circular shadow in the C-arm image to be repaired.

[0099] Optionally, replacing the corresponding content in the second stitched image with the repaired third type of sub-image to obtain a complete repaired image includes: cropping the repaired third type of sub-image to be repaired into a second type of image to be replaced, and replacing the portion of the first stitched image corresponding to the vertex intersection region with the second type of image to be replaced, wherein the image width and image length of the second type of image to be replaced are both twice the diameter of the circular shadow in the C-arm image to be repaired.

[0100] Optionally, the first type of image to be replaced contains only a complete circular shadow in the corresponding area of ​​the second type of sub-image to be repaired; the second type of image to be replaced contains only a complete circular shadow in the corresponding area of ​​the third type of sub-image to be repaired.

[0101] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0102] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including instructions, which can be executed by a processor of a C-arm image restoration apparatus based on a generative adversarial network to implement the method as described in any of the above embodiments. For example, the method may include:

[0103] Obtain images of the C-arm machine to be repaired;

[0104] The C-arm machine image to be repaired is cropped to obtain several sub-images to be repaired;

[0105] The sub-image to be repaired is input into the generative adversarial network (GAN) to process the sub-image to be repaired and obtain the repaired sub-image, wherein the GAN is trained by any of the above-described GAN training methods.

[0106] The repaired sub-images are then stitched together to obtain the complete repaired image.

[0107] The non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc., and this application does not limit it.

[0108] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A C-arm image inpainting method based on generative adversarial networks, characterized in that, The method includes: Obtain images of the C-arm machine to be repaired; The C-arm machine image to be repaired is cropped to obtain several sub-images to be repaired; The sub-image to be repaired is input into the generative adversarial network (GAN) to process it, thereby obtaining the repaired sub-image. The repaired sub-images are stitched together to obtain the complete repaired image; The process of cropping the C-arm image to be repaired to obtain several sub-images to be repaired includes: Based on the image resolution set by the generative adversarial network, the C-arm image to be repaired is cropped into multiple first-type sub-images to be repaired; The C-arm image to be repaired is re-cropped according to the image resolution to obtain at least one second type of sub-image to be repaired, and the second type of sub-image to be repaired covers the edge junction area of ​​multiple adjacent first type of sub-images to be repaired. The C-arm image to be repaired is re-cropped according to the image resolution to obtain several third-class sub-images to be repaired. The third-class sub-images to be repaired cover the vertex intersection area of ​​multiple adjacent first-class sub-images to be repaired. The training method for the generative adversarial network includes: A generative adversarial network is constructed; the generative adversarial network is used to repair the input original image with circular shadows and output the repaired image with the circular shadows removed; Acquire sample C-arm emulator images and crop the sample C-arm emulator images into several sample sub-images; A training dataset is generated based on the sample sub-images. The training dataset includes at least one pair of matched sample sub-images and mask sub-images. The mask sub-images are obtained by superimposing the sample sub-images with a mask pattern used to simulate the circular shadow. The generative adversarial network is trained using the training dataset to obtain the trained generative adversarial network.

2. The method according to claim 1, characterized in that, The step of stitching together the repaired sub-images to obtain the complete repaired image includes: The repaired first type of sub-images to be repaired are stitched together to obtain the first stitched image; The corresponding content in the first stitched image is replaced with the repaired second type of sub-image to be repaired to obtain the second stitched image; The corresponding content in the second stitched image is replaced with the repaired third type of sub-image to obtain a complete repaired image.

3. The method according to claim 2, characterized in that, The step of replacing the corresponding content in the first spliced ​​image with the repaired second type of sub-image to be repaired to obtain a second spliced ​​image includes: cropping the repaired second type of sub-image to be repaired into a first type of image to be replaced, and replacing the part of the first spliced ​​image corresponding to the edge intersection area with the first type of image to be replaced, wherein the image width of the first type of image to be replaced is twice the diameter of the circular shadow in the C-arm image to be repaired; The step of replacing the corresponding content in the second stitched image with the repaired third type of sub-image to obtain a complete repaired image includes: cropping the repaired third type of sub-image to be repaired into a second type of image to be replaced, and replacing the part of the first stitched image corresponding to the vertex intersection area with the second type of image to be replaced, wherein the image width and image length of the second type of image to be replaced are both twice the diameter of the circular shadow in the C-arm image to be repaired.

4. The method according to claim 3, characterized in that, The first type of image to be replaced contains only a complete circular shadow in the corresponding area of ​​the second type of sub-image to be repaired; The second type of image to be replaced contains only a complete circular shadow within the corresponding area of ​​the third type of sub-image to be repaired.

5. The method according to claim 1, characterized in that, The input to the generative adversarial network is a single channel, used to input the C-arm machine image to be repaired in grayscale mode.

6. The method according to claim 1, characterized in that, The mask pattern used for any mask sub-image includes at least one of the following: A circular mask, and a semi-circular mask having the same radius as the circular mask and located at the edge of any of the mask sub-images.

7. The method according to claim 1, characterized in that, The generative adversarial network includes a generator network and a discriminator network, and the method further includes: The sample sub-image and the mask sub-image are input into the generator network to pre-train the generator network; The pre-trained model parameters are used as the initialization parameters for the generator network.

8. A training device for a generative adversarial network for C-arm image inpainting, characterized in that, The apparatus is used to perform the method according to any one of claims 1-7, the apparatus comprising: A network construction unit is used to construct a generative adversarial network; the generative adversarial network is used to repair an input original image with circular shadows and output a repaired image with the circular shadows removed. The sample acquisition unit is used to acquire sample C-arm emulator images and crop the sample C-arm emulator images into several sample sub-images; A dataset generation unit is configured to generate a training dataset based on the sample sub-images. The training dataset includes at least one pair of matched sample sub-images and mask sub-images. The mask sub-images are obtained by superimposing the sample sub-images with a mask pattern used to simulate the circular shadow. The network training unit is used to train the generative adversarial network using the training dataset to obtain the trained generative adversarial network, which is used to perform image restoration on the C-arm machine image to be restored.

9. An image restoration device for a C-arm machine based on generative adversarial networks, characterized in that, The apparatus is used to perform the method according to any one of claims 1-7, the apparatus comprising: Image acquisition unit, used to acquire images of the C-arm machine to be repaired; The image cropping unit is used to crop the C-arm machine image to be repaired to obtain several sub-images to be repaired; An image inpainting unit is used to input the sub-image to be inpainted into the generative adversarial network, so that the sub-image to be inpainted is processed by the generative adversarial network to obtain the inpainted sub-image; The image stitching unit is used to stitch the repaired sub-images together to obtain a complete repaired image.

10. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-7 by executing the executable instructions.

11. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-7.

Images