A method for integrating image enhancement and training data generation using a generative recurrent neural network model, a computing device for the same, and a recording medium containing program code for performing the method.

Abstract

A method and apparatus for integrating image enhancement and training data generation using a generative recurrent neural network model is presented. [Solution] A method for integrating image enhancement and training data generation using a generative recurrent neural network model includes: (a) inputting a target image; and (b) applying the target image to a trained generative recurrent neural network model to selectively generate either a high-quality image or a low-quality image with improved image quality depending on the type of the target image.

Description

Technical Field

[0001] The present invention relates to a method and apparatus for integrating image enhancement and learning data generation using a generative recursive neural network model.

Background Art

[0002] Digital images have the disadvantage that their image quality deteriorates due to factors such as the shooting environment, the limitations of the optical system performance, and the limitations of the image sensor function. In order to improve such degraded image quality, an image enhancement model based on artificial intelligence is used.

[0003] The image enhancement technology based on artificial intelligence has developed and continuously improved its performance, such as methods based on noise removal and resolution improvement techniques using convolutional neural networks (CNNs), methods using adversarial generative neural networks, methods using vision transformers, methods using swin transformers, and methods using diffusion models.

[0004] Such an image enhancement model based on artificial intelligence requires learning using pairs of high-quality images and degraded images. In order to guarantee the performance of the image enhancement model based on artificial intelligence, a sufficient amount of diverse learning image data is required. However, it is very difficult or impossible to secure learning data in special environments other than the publicly available learning data.

[0005] Moreover, as the image enhancement model based on artificial intelligence develops and becomes more complex, the amount and types of data required for learning increase rapidly, the learning time also increases, and there are many many many constraints for effectively implementing the model in special environments.

[0006] For example, when trying to improve low illuminance in an autonomous driving vehicle, a general image enhancement model that does not consider the characteristics of images obtained on the road not only has difficulty providing optimal performance, but also has problems such as high implementation costs and processing times, making it difficult to implement. [Overview of the project] [Problems that the invention aims to solve]

[0007] The present invention provides a method and apparatus for integrating image enhancement and training data generation using a generative-recurrent neural network model. Furthermore, the present invention provides a method and apparatus for integrating image enhancement and training data generation using a generative recurrent neural network model, which not only generates various types of low-resolution images using only high-resolution image or video input, but also improves the low-resolution images thus generated by a trained model. [Means for solving the problem]

[0008] According to one aspect of the present invention, a method is provided for integrating image enhancement and training data generation using a generative-recurrent neural network model. One embodiment of the present invention provides a method for integrating image enhancement and training data generation using a generative recurrent neural network model, comprising the steps of (a) inputting a target image, and (b) applying the target image to a trained generative recurrent neural network model to selectively generate either a high-resolution image or a low-resolution image with improved image quality, depending on the type of target image.

[0009] The generative-recurrent neural network model comprises a trained image enhancement neural network module and a trained degraded image generation module. When the image quality of the target image is below the standard image quality, only the trained image enhancement neural network module operates, and the trained degraded image generation module does not operate. When the image quality of the target image exceeds the standard image quality, only the trained degraded image generation module operates, and the trained image enhancement neural network module may not operate.

[0010] Prior to step (a) above, the step further includes training the generative recurrent neural network model using an image dataset including a first image and a second image, the step of training the generative recurrent neural network model includes applying the first image and the second image to the generative recurrent neural network model to generate a high-resolution image with improved image quality from the low-resolution image among the first and second images, and training the model to generate a low-resolution image using the improved high-resolution image.

[0011] The first and second images are a pair of images of the same scene but with different image quality. The generative-recurrent neural network model comprises: an image enhancement neural network module that receives a low-resolution image from the first image and the second image as input and generates a high-resolution image with improved image quality; a degraded image neural network module located at the rear end of the image enhancement neural network module, which receives the high-resolution image with improved image quality as input and generates a low-resolution image; a first discrimination module that adjusts the weight values ​​of the image enhancement neural network module so as to minimize the first consistency loss for the high-resolution image from the first image and the second image and the high-resolution image with improved image quality; and a second discrimination module that calculates a second consistency loss using the low-resolution image from the first image and the second image and the low-resolution image output from the degraded image neural network module, and then adjusts the weight values ​​of the degraded image neural network module so as to minimize the second consistency loss.

[0012] The image enhancement neural network module and the degraded image neural network module each consist of an encoder with multiple transformer substrates and a decoder using multiple convolutional layers, wherein the multiple transformers and the multiple convolutional layers are configured to have a symmetrical structure with respect to each other.

[0013] Each transformer comprises a convolutional layer that extracts feature maps from an input image, a splitting layer that splits the extracted feature maps, a first swin transformer block that receives the split feature maps as input and extracts a first wide-range feature map, a first residual block located at the rear end of the first swin transformer block that receives the first wide-range feature map as input and extracts a first local feature map, a second residual block that receives the split feature maps as input and extracts a second local feature map for the split feature maps, a second swin transformer block located at the rear end of the second residual block that receives the second local feature map as input and extracts a second wide-range feature map, and can output a feature map after combining the first local feature map output from the first residual block and the second wide-range feature map output from the second swin transformer block and passing through the convolutional layer.

[0014] According to another aspect of the present invention, an apparatus is provided for integrating image enhancement and training data generation using a generative-recurrent neural network model. According to one embodiment of the present invention, a computing device is provided which includes a memory for storing at least one instruction word and a processor for executing the instruction word stored in the memory, wherein the instruction word executed by the processor is characterized in that it performs the steps of (a) receiving a target image as input and (b) applying the target image to a trained generative recurrent neural network model to selectively generate either a high-resolution image or a low-resolution image with improved image quality, depending on the type of the target image. [Effects of the Invention]

[0015] One embodiment of the present invention provides a method and apparatus for integrating image enhancement and training data generation using a generative recurrent neural network model. This method has the advantage of not only being able to generate various types of low-resolution images even with only high-resolution image or video input, but also being able to improve these generated low-resolution images using the trained model.

[0016] Furthermore, the present invention has the advantage of enabling the generation of image learning data using a generative recurrent neural network model, even in situations where access is difficult or data cannot be obtained, such as military facilities or dangerous areas. [Brief explanation of the drawing]

[0017] [Figure 1] This flowchart shows a method for integrating image enhancement and training data generation using a generative recurrent neural network model according to one embodiment of the present invention. [Figure 2] This is a diagram showing the overall configuration of a generative-recurrent neural network model according to one embodiment of the present invention. [Figure 3] This is a drawing showing the detailed structure of an image enhancement neural network module and a degraded image generation module according to one embodiment of the present invention. [Figure 4] This is a drawing showing the detailed structure of a transformer block according to one embodiment of the present invention. [Figure 5] This is a drawing showing the detailed structure of a swing transformer according to one embodiment of the present invention. [Figure 6] This diagram illustrates the operation of a generative-recurrent neural network model when a low-resolution image is input, according to one embodiment of the present invention. [Figure 7] This diagram illustrates the operation of a generative-recurrent neural network model when a high-resolution image is input, according to one embodiment of the present invention. [Figure 8] This is a block diagram schematically showing the internal configuration of a computing device according to one embodiment of the present invention. [Modes for carrying out the invention]

[0018] As used in this specification, the singular forms include the plural forms unless the context clearly dictates otherwise. In this specification, terms such as "composed of" or "including" shall not be construed as necessarily including all of the many components or many steps described in the specification, as some of the components or some of the steps may not be included, or it may have to be construed as further including additional components or steps. Also, terms such as "··· part" and "module" described in the specification mean units that process at least one function or operation, which may be implemented in hardware or software, or implemented by the combination of hardware and software.

[0019] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a flowchart showing a method of integrating image enhancement and learning data generation using a generative recursive neural network model according to an embodiment of the present invention, FIG. 2 is a drawing showing the overall configuration of a generative recursive neural network model according to an embodiment of the present invention, FIG. 3 is a drawing showing the detailed structure of an image enhancement neural network module and a degraded image generation module according to an embodiment of the present invention, FIG. 4 is a drawing showing the detailed structure of a transformer block according to an embodiment of the present invention, FIG. 5 is a drawing showing the detailed structure of a Swin Transformer according to an embodiment of the present invention, FIG. 6 is a drawing for explaining the operation of a generative recursive neural network model when a low-quality image is input according to an embodiment of the present invention, and FIG. 7 is a drawing for explaining the operation of a generative recursive neural network model when a high-quality image is input according to an embodiment of the present invention.

[0020] In step 110, the arithmetic unit 100 receives an input of an image data set. For the sake of convenience, the image data set is The format is shown as TIFF0007855259000001.tif14140. An image dataset can contain N pairs of images with different image quality from the same scene.

[0021] For the sake of understanding and explanation, the following will be used. TIFF0007855259000002.tif10140 is described as the high-resolution image of the i-th image pair. TIFF0007855259000003.tif10140 will be described as the low-resolution image of the i-th image pair.

[0022] In step 115, the arithmetic unit 100 processes the i-th image pair ( TIFF0007855259000004.tif10140, Applying TIFF0007855259000005.tif10140) to a generative recurrent neural network model, TIFF0007855259000006.tif10140: Improved high-resolution image. TIFF0007855259000007.tif11140 is generated, and the low-resolution image is used with the improved high-resolution image. Train a generative recurrent neural network model to generate TIFF0007855259000008.tif11140.

[0023] Figure 2 shows the detailed structure of the generative-recurrent neural network model. Refer to Figure 2 for further details. As shown in Figure 2, the generative-recurrent neural network model comprises an image enhancement neural network module 210, a degraded image generation module 220, a first identification module 230, and a second identification module 240.

[0024] For training generative recurrent neural network models, Apply TIFF0007855259000009.tif10140 to the image enhancement neural network module 210, and the image enhancement neural network module 210 will TIFF0007855259000010.tif10140: Improved high-resolution image. It is trained to generate TIFF0007855259000011.tif11140.

[0025] The output of the image enhancement neural network module 210 is a high-resolution image with improved image quality. TIFF0007855259000012.tif11140 is transmitted to the degraded image generation module 220 located at the rear end of the image enhancement neural network module 210. The degraded image generation module 220 then generates a high-resolution image with improved image quality. Enter TIFF0007855259000013.tif11140 and send a low-resolution image. It is trained to generate TIFF0007855259000014.tif11140.

[0026] The first identification module 230 is the output of the image enhancement neural network module 210, which is an improved high-resolution image. TIFF0007855259000015.tif11140 and The identification probability and consistency loss (referred to as the first consistency loss for convenience) can be calculated using TIFF0007855259000016.tif10140. The first identification module 230 can adjust the parameter weights of the image enhancement neural network module 210 so that the identification probability becomes 0.5 and the first consistency loss is minimized.

[0027] Furthermore, the second identification module 240 is the output of the degraded image generation module 220. TIFF0007855259000017.tif11140 and The identification probability and consistency loss (referred to as the second consistency loss for convenience) can be calculated using TIFF0007855259000018.tif10140. The second identification module 240 can adjust the parameter weighting of the degraded image generation module 220 so that the identification probability becomes 0.5 and the second consistency loss is minimized.

[0028] Using image pairs included in an image dataset, a generative recurrent neural network model can be repeatedly trained to generate improved quality images from low-quality images, and then use those improved images to generate low-quality images.

[0029] The image enhancement neural network module 210 and the degraded image generation module 220 included in the generative recurrent neural network model are composed of a decoder structure using multiple transformer-based encoders and multiple convolutional layers, and the decoder using multiple transformer-based encoders and multiple convolutional layers is composed of a UNet structure having a symmetrical structure with respect to each other.

[0030] Figure 3 shows in detail the UNet structures of the image enhancement neural network module 210 and the degraded image generation module 220. As shown in Figure 3, the image enhancement neural network module 210 and the degraded image generation module 220 are similarly composed of UNet structures having encoders with multiple transformer bases and decoders with multiple convolutional layers, but during the learning process, the parameters can be adjusted with different weighting values. As a result, the image enhancement neural network module 210 and the degraded image generation module 220 are learned to generate images of different quality levels by adjusting the parameter weighting values ​​differently.

[0031] Furthermore, the transformer blocks that constitute the encoders of the image enhancement neural network module 210 and the degraded image generation module 220 are configured, as shown in Figure 4, to extract a full-range feature map for the input image (or feature map), then extract a local feature map, then extract a full-range feature map, and finally integrate these.

[0032] Figure 4 shows the configuration of the transformer block. The operation of the transformer block will be briefly explained with reference to Figure 4. As shown in Figure 4, a feature map is extracted from the input image (or input feature map) via a convolutional layer, and the feature map extracted by the convolutional layer is separated into two branches via a separation layer. The separated feature maps are then used to extract a wide-area feature map (referred to as the first wide-area feature map for convenience) by the first Swin Transformer, and then transmitted to the first residual block located at the rear end of the first Swin Transformer to estimate the first local feature map. Furthermore, a second local feature map is extracted from the separated feature map via the second residual block, and then the second wide-area feature map is estimated by the second Swin Transformer. The detailed structures of the first and second Swin Transformers are shown in Figure 5.

[0033] After combining the estimated first local feature map and the second global feature map, a convolution operation can be applied to output the final feature map of the transformer block.

[0034] In one embodiment of the present invention, each transformer block can, as shown in Figure 4, apply a swin transformer via one branch to extract a global feature map, then transmit it to a residual block located at the rear end to reconstruct a local feature map, and then, via another branch and the residual block, extract a local feature map, pass it through a swin transformer located at the rear end to reconstruct a global feature map.

[0035] As mentioned above, by combining an image enhancement neural network module and a degraded image generation module with a generative-recurrent neural network model and training it, it is possible to generate image training data of high-resolution or low-resolution images for specific environments (in-vehicle cameras, military security sites), as well as low-resolution or improved-resolution image data, and then use this data to train the model again.

[0036] The learning process of a generative-recurrent neural network model is explained with reference to Figures 2 through 4.

[0037] This paper describes the operation of a generative-recurrent neural network model after it has been trained, using an image dataset.

[0038] In step 120, the arithmetic unit 100 receives a target image as input. Here, the target image may be a low-resolution image or a high-resolution image. In 125 steps, the computing unit 100 applies the target image to a trained generative recurrent neural network model and selectively generates either a high-resolution image or a low-resolution image with improved image quality, depending on the type of target image.

[0039] A pre-trained generative-recurrent neural network model can selectively operate either the trained image enhancement neural network module 210 or the trained degraded image generation module 220, depending on the type of target image.

[0040] For example, suppose the target image is a low-resolution image, below the standard resolution. The trained generative recurrent neural network model can operate only the image enhancement neural network module 210 and the degraded image generation module 220, while deactivating the degraded image generation module 220. As a result, when the target image is a low-resolution image, the trained generative recurrent neural network model behaves like an image enhancement neural network model that generates an improved image from a low-resolution image, as shown in Figure 6.

[0041] To give another example, suppose we assume that the image quality of the target image exceeds the standard image quality, meaning it is a high-quality image. The trained generative recurrent neural network model can operate only the degraded image generation module 220 of the trained image enhancement neural network module 210 and the trained degraded image generation module 220, while deactivating the image enhancement neural network module 210. In this way, when the target image is a high-quality image, the trained generative recurrent neural network model behaves like a degraded image generation model, generating a low-quality image with reduced quality from the high-quality image, as shown in Figure 7.

[0042] Figure 8 is a block diagram schematically showing the internal configuration of a computing device according to one embodiment of the present invention. Referring to Figure 8, the arithmetic unit 100 according to one embodiment of the present invention is configured to include a memory 810 and a processor 820.

[0043] Memory 810 stores instructions for performing a method for generating training data using a generative recurrent neural network model according to one embodiment of the present invention. The processor 820 works in conjunction with the memory 810 to execute the instruction words stored in the memory 810.

[0044] The instructions executed by the processor 820 can take a target image as input and apply the target image to a trained generative recurrent neural network model to perform a series of processes for selectively generating either a high-resolution image or a low-resolution image with improved image quality, depending on the type of target image.

[0045] In addition, the instructions executed by the processor 820 may perform a series of processes to train the generative recurrent neural network model using an image dataset including the first and second images. The training process is the same as that described with reference to Figures 1 to 4, so any overlapping explanation will be omitted.

[0046] The apparatus and methods according to embodiments of the present invention are embodied in the form of program instructions performed by various computer means and recorded on a computer-readable medium. The computer-readable medium includes program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the computer-readable medium may be specifically designed and configured for the present invention, or may be publicly known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floppy disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine code generated by compilers, but also high-level language code that can be executed by a computer using an interpreter or the like.

[0047] The aforementioned hardware device can be configured to operate with at least one software module in order to perform the operation of the present invention, and vice versa. To date, the present invention has been described primarily in terms of its embodiments. Those skilled in the art will understand that the present invention can be embodied in modified forms without departing from its essential characteristics. Therefore, the disclosed embodiments should be considered in an explanatory rather than restrictive manner. The scope of the present invention is defined in the claims, not in the foregoing description, and all differences within an equivalent scope should be interpreted as being included in the present invention. [Explanation of Symbols]

[0048] 210 Image Enhancement Neural Network Module 220 Degraded Image Generation Module 230 First Identification Module 240 Second Identification Module 810 memory 820 processor

Claims

1. (a) A step in which a target image is input, (b) The step of applying the target image to a trained generative recurrent neural network model to selectively generate either a high-resolution image or a low-resolution image with improved image quality, depending on the type of target image, The aforementioned generative-recurrent neural network model is An image enhancement neural network module that, after receiving a low-resolution image from among the first and second images, generates a high-resolution image with improved image quality, Located at the rear end of the aforementioned image enhancement neural network module, the degraded image neural network module generates a low-resolution image after receiving the improved high-resolution image as input. A first discriminant module that adjusts the weight values ​​of the image enhancement neural network module so as to minimize the first consistency loss for the first image and the high-resolution image with improved image quality, The system comprises a first image, a lower-resolution image from the second image, and a second discrimination module that calculates a second consistency loss using the lower-resolution image output from the degraded image neural network module, and then adjusts the weight values ​​of the degraded image neural network module so that the second consistency loss is minimized. The aforementioned image enhancement neural network module and the aforementioned degraded image neural network module are each composed of an encoder with multiple transformer substrates and a decoder using multiple convolutional layers, The multiple transformers and the multiple convolutional layers are configured to have a symmetrical structure with respect to each other. Each of the aforementioned multiple transformers is: A convolutional layer that extracts feature maps from the input image, A separation layer for splitting the extracted feature map, A first Swin transformer block that takes the separated feature maps as input and extracts a first global feature map, A first residual block located at the rear end of the first swin transformer block, which receives the first global feature map as input and extracts the first local feature map, A second residual block that takes the separated feature map as input and extracts a second local feature map for the separated feature map, A second swin transformer block located at the rear end of the second residual block, which receives the second local feature map as input and extracts a second global feature map, A method for integrating image enhancement and training data generation using a generative recurrent neural network model, executed by a computing device, characterized in that the first local feature map output from the first residual block and the second global feature map output from the second Swin transformer block are combined, and then the feature map is output by passing it through the convolutional layer.

2. The aforementioned generative-recurrent neural network model is It comprises a trained image enhancement neural network module and a trained degraded image generation module, If the image quality of the target image is below the standard image quality, only the trained image enhancement neural network module operates, and the trained degraded image generation module does not operate. A method for integrating image enhancement and training data generation using a generative-recurrent neural network model according to claim 1, characterized in that, when the image quality of the target image exceeds the reference image quality, only the trained degraded image generation module operates and the trained image enhancement neural network module does not operate.

3. Prior to step (a), The process further includes the step of training the generative recurrent neural network model using an image dataset containing the first image and the second image, The step of training the generative recurrent neural network model is as follows: A method for integrating image enhancement and training data generation using a generative recurrent neural network model according to claim 1, characterized by comprising the steps of applying the first image and the second image to the generative recurrent neural network model to generate a high-resolution image with improved image quality from the low-resolution image among the first image and the second image, and training the model to generate a low-resolution image using the improved high-resolution image.

4. A method for integrating image enhancement and training data generation using the generative recurrent neural network model according to claim 3, characterized in that the first image and the second image are a pair of images with different image quality in the same scene.

5. A computer-readable recording medium characterized by having a program code for performing the method described in claim 1 recorded on it.

6. Memory to store at least one instruction word, The system comprises a processor that executes instruction words stored in the aforementioned memory, The instruction words executed by the aforementioned processor are, respectively, (a) A step in which a target image is input, (b) The step of applying the target image to a trained generative recurrent neural network model to selectively generate either a high-resolution image or a low-resolution image with improved image quality, depending on the type of target image, The aforementioned generative-recurrent neural network model is An image enhancement neural network module that, after receiving a low-resolution image from among the first and second images, generates a high-resolution image with improved image quality, Located at the rear end of the aforementioned image enhancement neural network module, the degraded image neural network module generates a low-resolution image after receiving the improved high-resolution image as input. A first discriminant module that adjusts the weight values ​​of the image enhancement neural network module so as to minimize the first consistency loss for the first image and the high-resolution image with improved image quality, The system comprises a first image, a lower-resolution image from the second image, and a second discrimination module that calculates a second consistency loss using the lower-resolution image output from the degraded image neural network module, and then adjusts the weight values ​​of the degraded image neural network module so that the second consistency loss is minimized. The image enhancement neural network module and the degraded image neural network module each consist of an encoder with multiple transformer substrates and a decoder using multiple convolutional layers. The multiple transformers and the multiple convolutional layers are configured to have a symmetrical structure with respect to each other. Each of the aforementioned multiple transformers is: A convolutional layer that extracts feature maps from the input image, A separation layer for splitting the extracted feature map, A first Swin transformer block that takes the separated feature maps as input and extracts a first global feature map, A first residual block located at the rear end of the first swin transformer block, which receives the first global feature map as input and extracts the first local feature map, A second residual block that takes the separated feature map as input and extracts a second local feature map for the separated feature map, A second swin transformer block located at the rear end of the second residual block, which receives the second local feature map as input and extracts a second global feature map, A computing device characterized by combining the first local feature map output from the first residual block and the second global feature map output from the second swin transformer block, and then passing the result through a convolutional layer to output a feature map.

7. The aforementioned generative-recurrent neural network model is It comprises a trained image enhancement neural network module and a trained degraded image generation module, If the image quality of the target image is below the standard image quality, only the trained image enhancement neural network module operates, and the trained degraded image generation module does not operate. The computing device according to claim 6, characterized in that, when the image quality of the target image exceeds the reference image quality, only the trained degraded image generation module operates and the trained image enhancement neural network module does not operate.

8. Prior to step (a), The step of training the generative recurrent neural network model using an image dataset including the first image and the second image is further performed. The step of training the generative recurrent neural network model is as follows: The computing device according to claim 6, comprising the steps of applying the first image and the second image to the generative recurrent neural network model to generate a high-resolution image with improved image quality from the low-resolution image among the first image and the second image, and training the model to generate a low-resolution image using the improved high-resolution image.

Images