Image shadow removing method and device for generating image shadow based on GAN model, storage medium and program product

Abstract

The invention discloses an image shadow removing method and device for generating an image shadow based on a GAN model, a storage medium and a program product. Two-stage models are adopted, the first-stage model is a shadow generation model, the second-stage model is an image shadow removal model, the shadow generation model adopts a GAN model architecture, and shadows very similar to real shadows are generated by using the GAN model, so that image pairs with shadows and without shadows can be prepared on a large scale; the technical route of removing the image shadow by adopting a deep learning method is implemented and developed, and many defects caused by removing the shadow by adopting a traditional image processing method are avoided.

Description

Technical Field

[0001] This invention belongs to the field of image processing, and in particular relates to microscopic imaging technology. Background Technology

[0002] Image stitching is frequently used in microscope imaging. However, the area captured by a single field of view of a microscope is limited, and the object to be photographed is often much larger than the microscope's field of view. In such cases, it is necessary to move the microscope and capture images of different positions on the object multiple times, then stitch these images together. During image stitching, a shadowing problem arises, where uneven lighting in the microscope creates a bright center and dark edges within a single field of view. When many such images are stitched together, the area where two images meet will be relatively dark. For example... Figure 1 As shown, after image stitching, vertical shadows appear on the image, with each shadow line located at the stitching point. This results in a large, stitched image with numerous "shadow grids." There are generally two approaches to removing shadows: traditional image processing methods and deep learning-based methods. Traditional image processing methods remove shadows through gain correction, but this can result in excessive or insufficient gain, failing to completely eliminate the shadows. Deep learning-based methods, on the other hand, input the image containing shadows into a deep learning model, which then outputs a shadow-free image.

[0003] Training deep learning models for shadow removal requires a large training set, which needs to include an image with shadows and an image without shadows. However, in real-world scenarios, it's difficult to obtain such one-to-one image pairs (the two images have the same content, differing only in that one has shadows and the other doesn't). Existing techniques for generating shadows often utilize known conditions, such as generating a shadow that spreads outwards from the center, gradually decreasing in size from the center outwards. Shadows generated by such methods are too smooth and simplistic, failing to consider the influence of reflections from the subject itself. Real shadows don't spread smoothly from the center outwards and are affected by reflections from different locations on the subject. Therefore, the key to current deep learning methods for removing image shadows is how to conveniently and accurately generate image pairs for training deep learning models. Summary of the Invention

[0004] To address the technical problems mentioned in the background section, this invention proposes an image shadow removal method, device, storage medium, and program product based on GAN model-generated image shadows.

[0005] To achieve the above-mentioned technical objectives, the technical solution of the present invention is as follows: A GAN-based image shadow removal method generates shadows in images by using a shadow generation model to obtain a shadow image that approximates a realistic shadow. The image without shadows is then combined with this approximate realistic shadow image to create a corresponding shadowed image. These two images form image pairs for training. The image removal model is trained based on this image pair training data. The image to be shadowed is then input into the trained image removal model to obtain the shadow-removed image. The shadow generation model adopts a GAN architecture, including a generator and a discriminator. Random noise is input into the generator to obtain a shadow image of the same size as the actual image. This shadow image is multiplied by the actual captured image without shadows to obtain the synthesized shadowed image. The synthesized shadowed image and the actual captured shadowed image are then input into the discriminator for binary classification training. This enables the discriminator to determine whether its input image is a real shadowed image or a synthesized shadowed image. The output of the discriminator is fed back to the generator, thus training the generator to generate shadow images that approximate realistic shadows.

[0006] Furthermore, the generator contains multiple deconvolutional layers.

[0007] Furthermore, the discriminator includes convolutional layers and downsampling layers.

[0008] Secondly, the present invention also provides a computer device, which includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the image shadow removal method based on the GAN model for generating image shadows.

[0009] Furthermore, the present invention also designs a computer storage medium storing a computer program, which, when executed on a processor, implements the image shadow removal method based on the GAN model for generating image shadows.

[0010] Finally, the present invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the image shadow removal method based on the GAN model for generating image shadows.

[0011] The beneficial effects of adopting the above technical solution are as follows: This invention uses a GAN model to generate shadows that are highly similar to real shadows, thereby enabling the preparation of large batches of image pairs with and without shadows for subsequent training of deep learning shadow removal models. This invention allows for the implementation and development of a deep learning-based approach to image shadow removal, avoiding many drawbacks of traditional image processing methods. Attached Figure Description

[0012] Figure 1 This is a diagram illustrating the shadows used in image stitching. Figure 2 This is a diagram of the GAN model architecture; Figure 3 This is a flowchart of the GAN model training process; Figure 4 This is a flowchart of shadow generation based on the GAN model. Detailed Implementation

[0013] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0014] The core idea of ​​this embodiment is to use a two-level model to remove shadows from images. The first-level model is a shadow generation model, which in this embodiment adopts a GAN model architecture. A GAN model includes a generator and a discriminator. The discriminator distinguishes between real and fake shadow images—that is, whether the shadow image comes from an actual photographed shadow image (real) or from a shadow image generated by the generator (fake). This "real" or "fake" information is fed back to the generator for optimization, prompting the generator to generate shadows that are more realistic. After continuous training in this way, the generator can generate shadows that are indistinguishable from reality. The other-level model is the image shadow removal model, which can use a conventional deep learning model architecture. In this embodiment, the first-level model (shadow generation model) is the focus.

[0015] Figure 2The GAN model architecture designed for this embodiment is as follows: The Generator is responsible for generating shadows from random noise; that is, after the random noise passes through the Generator, it outputs a shadow image of the same size as the actual image. This shadow image is multiplied with an actual captured image without shadows to obtain a synthesized image with shadows. Additionally, we have captured actual images with shadows, randomly select one of these images, and feed it along with the synthesized image with shadows into the Discriminator for binary classification ("true" and "false") training. This trains the Discriminator's ability to identify whether a shadow originates from a synthesized (real) image. The above process uses both images with and without shadows. It is important to note that the images with and without shadows here are not the final image pairs used to train the image shadow removal model. The images with and without shadows do not need to have identical content. The method for obtaining the image without shadows is to take images using a lens with minimal or no shadows, cropping the edges and retaining only the central image area to ensure the resulting image is shadow-free. This method is used to photograph numerous objects, obtaining a batch of shadow-free image data. Obviously, the dataset of shadowless images obtained in this way cannot be matched one-to-one with the images with shadows captured, because it cannot be guaranteed that the angle and position at the time of capture are completely consistent.

[0016] In this embodiment, the GAN model can be constructed using CNN as its base layer. The Generator can be viewed as taking one-dimensional noise as input and continuously enlarging the image size through multiple deconvolutional layers until it reaches the same size as the actual image. The Discriminator can be viewed as a binary classification model for images, constructed using convolutional layers and downsampling layers.

[0017] Training a GAN model involves two iterations per iteration. The first step trains the Discriminator, which is trained based on whether the input image contains real or synthetic shadows. The second step trains the Generator, which is trained using the "real" or "synthetic" information from the Discriminator's previous training. This allows the Generator to learn how to generate realistic shadows. Through this training, the Generator learns to generate indistinguishable shadows. This Generator is then used to generate shadows. In actual use, the Generator generates different shadows by continuously varying the input noise Z (randomly transforming it). These generated shadows are then sequentially synthesized with an image without shadows to obtain the final image with shadows, resulting in image pairs consisting of the image with and without shadows, all with identical content. The training and shadow generation processes described above are as follows: Figure 3 , Figure 4 As shown.

[0018] Furthermore, this embodiment also relates to a computer device, which includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the image shadow removal method based on the GAN model for generating image shadows.

[0019] Furthermore, this embodiment also relates to a computer storage medium storing a computer program, which, when executed on a processor, implements the image shadow removal method based on the GAN model for generating image shadows.

[0020] Finally, this embodiment also relates to a computer program product, including a computer program or instructions, which, when executed by a processor, implement the image shadow removal method based on the GAN model for generating image shadows.

[0021] It will be understood by those skilled in the art that this invention may relate to apparatus for performing one or more of the operations described in this application. The apparatus may be specifically designed and manufactured for the desired purpose, or may include known devices in general-purpose computers with programs stored therein that can be selectively activated or reconfigured. Such computer programs may be stored in a device (e.g., computer)-readable medium or in any type of medium suitable for storing electronic instructions and coupled to a bus, including but not limited to any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), random access memory (RAM), read-only memory (ROM), electrically programmable ROM, electrically erasable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic cards, or optical cards. A readable medium includes any mechanism for storing or transmitting information in a form readable by a device (e.g., computer). For example, readable media include random access memory (RAM), read-only memory (ROM), disk storage media, optical storage media, flash memory devices, signals propagated in electrical, optical, acoustic, or other forms (e.g., carrier waves, infrared signals, digital signals), etc.

[0022] Those skilled in the art will understand that each box in these structure diagrams and / or block diagrams and / or flow diagrams, as well as combinations of boxes in these structure diagrams and / or block diagrams and / or flow diagrams, can be implemented using computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing method to generate a machine, thereby creating, through execution by the processor of the computer or other programmable data processing method, methods specified in the boxes of the structure diagrams and / or block diagrams and / or flow diagrams.

[0023] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0024] The embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A method for removing image shadows based on GAN model-generated image shadows, characterized by: A shadow generation model is used to obtain a shadow image that approximates a realistic shadow. The image without shadow is then combined with this approximate shadow image to create a corresponding shadowed image. These two images constitute image pair training data. An image deshading model is trained based on this image pair training data. The image to be deshaded is input into the trained image deshading model to obtain the deshaded image. The shadow generation model adopts a GAN model architecture, including a generator and a discriminator. Random noise is input into the generator to obtain a shadow image of the same size as the actual image. This shadow image is multiplied by the actually acquired image without shadow to obtain a synthesized shadowed image. The synthesized shadowed image and the actually acquired shadowed image are respectively input into the discriminator for binary classification training. This enables the discriminator to determine whether its input image is a real shadowed image or a synthesized shadowed image. The output of the discriminator is fed back to the generator, thus training the generator to generate shadow images that approximate realistic shadows.

2. The image shadow removal method based on GAN model generation according to claim 1, characterized in that: The generator contains multiple deconvolutional layers.

3. The image shadow removal method based on GAN model generated image shadows according to claim 1, characterized in that: The discriminator comprises a convolutional layer and a downsampling layer.

4. A computer device, the computer device comprising a processor and a memory, the memory storing a computer program, characterized in that: The processor is used to execute the computer program to implement the image shadow removal method based on the GAN model for generating image shadows as described in any one of claims 1-3.

5. A computer storage medium storing a computer program, characterized in that: When the computer program is executed on the processor, it implements the image shadow removal method based on the GAN model for generating image shadows as described in any one of claims 1-3 above.

6. A computer program product, comprising a computer program or instructions, characterized in that: When the computer program or instructions are executed by the processor, they implement the image shadow removal method based on the GAN model for generating image shadows as described in any one of claims 1-3 above.

Images