Information processing device, information processing method, and program

Abstract

This aims to improve the accuracy of image enhancement in scenes where it is difficult to prepare training data for a model that enhances image quality based on the characteristics of the subject. [Solution] First, a noise-free image taken during the daytime that includes a moving object is acquired as the first base image. Then, a style transfer model is acquired by learning to transform an image that does not include a moving object but was taken during the daytime and is noise-free into an image taken at night. Next, a pseudo-inference image that includes a moving object and was taken at night and is noise-free is generated using the first base image and the style transfer model. Finally, the generated pseudo-inference image is used as the ground truth image to train a denoising model.

Description

Technical Field

[0001] The present invention particularly relates to an information processing apparatus, an information processing method, and a program suitable for use in learning a model for high image quality.

Background Art

[0002] In recent years, machine learning has been applied to various image processing for high image quality. As an example of image processing for high image quality using a neural network model, there is, for example, noise removal processing, which is applied to a camera system. Generally, in order to shoot while removing noise with a camera, it is necessary to shoot under the setting conditions of long-time exposure. Therefore, by using the image taken under such setting conditions as a correct image to learn a noise removal model, a highly accurate noise removal model is generated. By performing image processing using the noise removal model learned in this way, for example, noise in an image of a landscape scene where noise is likely to occur when shooting at night can be removed.

[0003] On the other hand, when shooting a moving object at night, it may be difficult to remove noise using a neural network model. When shooting a moving object, it is necessary to shoot with short-time exposure and high ISO sensitivity, so the condition of long-time exposure required for shooting while removing noise is not suitable for shooting a moving object. Thus, when shooting a moving object at night, it is difficult to obtain an image with noise removed, so the image taken of a moving object at night cannot be set as a correct image, and the noise removal model cannot be sufficiently learned. As a result, the accuracy of noise removal using a neural network model decreases. Therefore, an improvement in the learning accuracy in a scene where it is difficult to prepare a correct image is required.

[0004] Patent Document 1 discloses a technique for generating a pseudo image using a style conversion generation AI technique to generate learning data of unknown observation conditions in a machine learning-based object recognition task based on an ultrasonic image during ultrasonic exploration in water, and performing augmentation of the learning data.

Prior Art Documents

[0005] [Patent Document 1] Japanese Patent Publication No. 2023-74829 [Overview of the project] [Problems that the invention aims to solve]

[0006] The technology described in Patent Document 1, mentioned above, does not perform style conversion that takes into account the subject conditions in addition to the observation conditions. Therefore, in high-resolution tasks, depending on the subject conditions, it may not be possible to appropriately generate images that are difficult to prepare through style conversion.

[0007] In view of the aforementioned problems, the present invention aims to improve the accuracy of image enhancement in scenes where it is difficult to prepare training data for a model that enhances image quality due to the characteristics of the subject. [Means for solving the problem]

[0008] The information processing apparatus according to the present invention is characterized by comprising: a first image acquisition means for acquiring a first base image including a moving object; a model acquisition means for acquiring a conversion model for converting an image without the moving object into an image that does not include the moving object and reduces noise to create an environment in which it is difficult to photograph the moving object; a generation means for generating an inference image based on the first base image and the conversion model, in which the first base image has been converted to an environment in which it is difficult to photograph the moving object by reducing noise; and a learning means for learning a noise reduction model that removes noise from an image based on the inference image. [Effects of the Invention]

[0009] According to the present invention, it is possible to improve the accuracy of image enhancement in scenes where it is difficult to prepare training data for a model that enhances image quality due to the characteristics of the subject. [Brief explanation of the drawing]

[0010] [Figure 1] This diagram illustrates the process of improving image quality in scenes where noise reduction is difficult. [Figure 2] This is a block diagram showing an example of the functional configuration of a high-definition image enhancement system according to the first embodiment. [Figure 3] This is a block diagram showing an example of the hardware configuration of an information processing device. [Figure 4] This flowchart shows an example of the overall processing procedure performed by the first information processing device in the first embodiment. [Figure 5] This flowchart shows an example of a detailed processing procedure for obtaining a style conversion model in the second information processing device. [Figure 6] This flowchart shows an example of a detailed processing procedure for training the noise reduction model in the first embodiment. [Figure 7] This diagram illustrates the process of acquiring a style transfer model, generating pseudo-inference images, and training a denoising model. [Figure 8] This flowchart shows an example of a detailed processing procedure for learning the noise reduction model in the second embodiment. [Figure 9] This figure shows an example of the correspondence between camera manufacturers and noise reduction characteristics. [Figure 10] This flowchart shows an example of the overall processing procedure performed by the first information processing device in the third embodiment. [Figure 11] This figure shows an example of a screen displaying the processing results. [Figure 12] This is a block diagram showing an example of the functional configuration of a high-definition image enhancement system according to the third embodiment. [Figure 13] This is a block diagram showing an example of the functional configuration of a high-definition image enhancement system according to the fourth embodiment. [Modes for carrying out the invention]

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the present invention is not limited to the embodiments described below, and various forms within the scope not departing from the gist of this invention are also included in the present invention. Furthermore, each of the embodiments described below merely shows one embodiment of the present invention, and it is also possible to appropriately combine each embodiment.

[0012] (First Embodiment) First, an overview of the environment used in the first embodiment of the present invention will be described. In this embodiment, a case where the processing according to this embodiment is applied in a situation of enhancing the quality of an image obtained by imaging a moving object at night with a camera will be described. Hereinafter, in this embodiment, style conversion is a technique of an image generation AI related to image synthesis in which the style of an image (hereinafter, style image) having a specific painting style (hereinafter, style) is reflected on the original image (hereinafter, base image) while leaving the prototype. A specific method of this is, for example, a known technique called StyleGAN.

[0013] In this embodiment, first, a neural network model for style conversion (hereinafter, style conversion model) for converting a daytime scene into a nighttime scene using an image in which a moving object is not shown as the base image and the style image is obtained through learning. Then, using the learned style conversion model and the base image in which the moving object appears in the daytime scene, a pseudo-inference image in which the moving object appears in the nighttime scene is generated. Then, the generated pseudo-inference image is set as learning data, and a neural network model for noise removal (hereinafter, noise removal model) is learned.

[0014] Here, in order to generate night images through style conversion under the condition where no noise is introduced (hereinafter referred to as noise-free), noise-free images are required. Therefore, when obtaining the style conversion model through learning, images in which the moving object is not captured are used. Then, by performing style conversion on the images of the moving object taken during the day using the style conversion model that can be darkened under the noise-free condition, an image of the moving object at night that is pseudo-noise-free is obtained. By pseudo-generating images of difficult scenes (environments where it is difficult to reduce noise and capture images) that are actually difficult to capture without noise and setting them as learning data, learning in difficult scenes can be carried out. As a result, the process of enhancing the image quality using the noise removal model can be accurately implemented. The detailed processes of style conversion learning, generation of pseudo-inference images, and learning of the noise removal model will be described later.

[0015] <Usage form> FIG. 1 is a schematic diagram showing an example of a usage scenario in this embodiment. Scene 101 in FIG. 1 shows a situation of enhancing the image quality of an image of a difficult scene for noise removal of a moving object captured at night, and is a scene where user 102 is capturing moving object 104 with camera 103. Then, a high-quality image 105 enhanced using the noise removal model learned by a personal computer (PC) 106 is obtained so that user 102 can view it. Hereinafter, the processing of this embodiment will be described while referring to the scene in FIG. 1.

[0016] <Configuration> FIG. 2 is a block diagram showing an example of the functional configuration of an image quality enhancement system 201 according to this embodiment. In FIG. 2, the image quality enhancement system 201 is composed of a first information processing device 202, a second information processing device 203, and an image quality enhancement processing unit 211.

[0017] The first information processing device 202 generates a pseudo-inferred image and trains a noise reduction model, while the second information processing device 203 acquires a style transfer model through training and provides the style transfer model to the first information processing device 202. The first information processing device 202 includes a base image acquisition unit 204, a model acquisition unit 205, a pseudo-inferred image acquisition unit 206, and a high-resolution learning unit 207. The second information processing device 203 includes a base image acquisition unit 208, a style image acquisition unit 209, and a style transfer learning unit 210.

[0018] First, the configuration of the first information processing device 202 will be described. The base image acquisition unit 204, which is the first image acquisition means, acquires an image of a moving object as the first base image. This first base image is used as the input image when generating a pseudo-inference image. In this embodiment, an image of a moving object captured in daylight under short exposure shooting conditions without noise is acquired as the first base image.

[0019] The model acquisition unit 205 acquires a style conversion model that converts the style of the first base image acquired by the base image acquisition unit 204 into a style for a scene where it is difficult to capture moving objects without noise. In this embodiment, a style conversion model that converts from a daytime style to a nighttime style is acquired from the second information processing device 203.

[0020] The pseudo-inference image acquisition unit 206 acquires a noise-free pseudo-inference image in scenes where it is difficult to actually capture a moving object without noise. Specifically, it generates a pseudo-inference image based on the first base image acquired by the base image acquisition unit 204 and the style conversion model acquired by the model acquisition unit 205. In this embodiment, a noise-free pseudo-image of a night style, in which a moving object is photographed with a short exposure, is generated as the pseudo-inference image.

[0021] The image quality enhancement learning unit 207 learns a denoising model based on the pseudo-inferred images acquired by the pseudo-inferred image acquisition unit 206. The specific method for learning the denoising model using pseudo-inferred images will be described later.

[0022] Next, the configuration of the second information processing device 203 will be described. The base image acquisition unit 208, which is the second image acquisition means, acquires an image that does not contain a moving object as the second base image. In this embodiment, an image is acquired that is similar to the first base image in a daytime style, takes a picture with a short exposure, and captures a noise-free image of a scene similar to the first base image that does not contain a moving object.

[0023] The third image acquisition means, the style image acquisition unit 209, acquires a style image taken at night of a scene similar to the second base image, without any moving objects. Specifically, in this embodiment, a style image is acquired as a noise-free image of a night style taken with a long exposure, of a scene similar to the second base image, without any moving objects.

[0024] The style conversion learning unit 210 generates a style conversion model through learning for converting to the style of scenes where it is difficult to capture moving objects without noise. In this embodiment, a style conversion model that converts from a daytime style to a nighttime style is obtained based on the second base image obtained by the base image acquisition unit 208 and the style image obtained by the style image acquisition unit 209. The specific learning method for style conversion will be described later.

[0025] The image quality enhancement processing unit 211 enhances the image quality of images captured by the camera using a noise reduction model learned by the image quality enhancement learning unit 207. In this embodiment, the image quality enhancement processing unit 211 is configured to function in a different device from the first information processing device 202 and the second information processing device 203, but it may also be configured to be included in the first information processing device 202.

[0026] <Description of Hardware Configuration> Figure 3 is a block diagram showing an example of the hardware configuration of the first information processing device 202 according to this embodiment. The first information processing device 202 in Figure 2 has the following hardware configuration.

[0027] The CPU (Central Processing Unit) 301 controls various devices connected to the bus 302 and performs the overall processing in this embodiment. The ROM (Read Only Memory) 303 stores the BIOS program and boot program. The RAM (Random Access Memory) 304 is used as the main memory of the CPU 301.

[0028] External memory 305 stores programs and other necessary information for processing by the CPU 301 in the first information processing device 202. Input unit 306 is a keyboard or mouse and performs processing related to information input. Display unit 307 displays the calculation results of the first information processing device 202 according to instructions from the CPU 301. The display unit 307 can be any type, such as a liquid crystal display, projector, or LED (Light Emitting Diode) indicator. Communication unit 308 is an interface for connecting to external devices such as cameras wirelessly or via wired connection.

[0029] The hardware configuration of the second information processing device 203 is the same as that shown in Figure 3. Furthermore, although the first information processing device 202 and the second information processing device 203 are separate in this embodiment, they may be combined into a single information processing device. In this case, the functional configuration in Figure 2 will also be assumed to function as a single information processing device, with each part of the first information processing device 202 and each part of the second information processing device 203 functioning together.

[0030] <Processing procedure and detailed processing method> Next, the processing procedure and detailed processing method of the first information processing device 202 according to this embodiment will be explained with reference to Figures 4 to 7. Figure 4 is a flowchart showing an example of the overall processing procedure performed by the first information processing device 202. The processing in Figure 4 is assumed to begin after the image to be enhanced has been captured. Here, the image to be enhanced is an image from which the enhanced image processing unit 211 is scheduled to remove noise using a trained noise reduction model.

[0031] First, in step S401, the base image acquisition unit 204 acquires an image of a moving object as a first base image. In this embodiment, the base image acquisition unit 204 acquires a group of images captured by the camera, performs subject detection processing, and acquires the first base image by selecting an image in which a moving object is captured. The subject detection processing used here is, for example, a known machine learning-based segmentation technique that detects a moving object from among objects in an captured image.

[0032] Next, in step S402, the model acquisition unit 205 acquires a style conversion model for converting the style of the first base image from the second information processing device 203. The specific method for acquiring the style conversion model in the second information processing device 203 will be described later with reference to Figure 5. Next, in step S403, the pseudo-inference image acquisition unit 206 generates a noise-free pseudo-inference image including a moving object at night using the first base image and the style conversion model. In this embodiment, the first base image is input to the style conversion model as an input image, and a pseudo-inference image is output as the inference result.

[0033] Then, in step S404, the image quality enhancement learning unit 207 trains a noise reduction model based on the pseudo-inference images acquired by the pseudo-inference image acquisition unit 206. In this embodiment, the pseudo-inference images are used as training data (ground truth images) for training the noise reduction model, but the detailed processing will be described later with reference to Figure 6.

[0034] In this embodiment, the method for acquiring a style conversion model in the second information processing device 203 will be described below with reference to Figure 5. Figure 5 is a flowchart showing an example of a detailed processing procedure for acquiring a style conversion model in the second information processing device 203. In this embodiment, each process in Figure 5 is performed in the second information processing device 203 before the image to be enhanced is captured, and is started at a timing specified by the user.

[0035] First, in step S501, the base image acquisition unit 208 acquires an image in which no moving object is captured as a second base image. This second base image is used as the input image during style transfer training. In this embodiment, the camera captures a group of images and performs subject detection processing. Then, the second base image is acquired by selecting an image in which no moving object is detected and whose other segmentation detection results are the same as the first base image acquired by the first information processing device 202. The subject detection processing used here may be a known machine learning-based segmentation technique.

[0036] Next, in step S502, the style image acquisition unit 209 acquires a style image, which is an image taken at night in the same area as the second base image and does not contain any moving objects. This style image is used as the correct image during style transformation training. In this embodiment, the style image is acquired by determining an image taken at night in the same area as the second base image and does not contain any moving objects, using a scene recognition process that applies known machine learning-based object recognition from the image group acquired from the camera 103.

[0037] Next, in step S503, the style transfer learning unit 210 acquires a style transfer model through learning. In this embodiment, the second base image is used as the input image, and the style image is used as the ground truth image to learn style transfer and acquire a style transfer model.

[0038] The detailed learning method for the noise reduction model in this embodiment will be explained below with reference to Figure 6. Figure 6 is a flowchart showing an example of a detailed processing procedure for step S404 in Figure 4.

[0039] In step S601, the image quality enhancement learning unit 207 sets the pseudo-inference image acquired by the pseudo-inference image acquisition unit 206 as the ground truth image for training the noise reduction model. Next, in step S602, the image quality enhancement learning unit 207 generates an image with added noise based on the pseudo-inference image acquired by the pseudo-inference image acquisition unit 206, and sets it as the input image for training the noise reduction model. In this embodiment, an artificially noised image is generated by adding artificial noise to the pseudo-inference image acquired by the pseudo-inference image acquisition unit 206, and this is set as the input image.

[0040] Next, in step S603, the image quality improvement learning unit 207 trains a noise reduction model using the ground truth image and input image set in steps S601 and S602.

[0041] Figure 7 is a diagram illustrating the overview of the style transfer model acquisition process, the pseudo-inference image generation process, and the denoising model training process in this embodiment. Specifically, Figure 7(a) is a diagram illustrating the style transfer model acquisition process performed in step S503 of Figure 5. Figure 7(b) is a diagram illustrating the pseudo-inference image generation process performed in step S403 of Figure 4. And Figure 7(c) is a diagram illustrating the denoising model training process performed in step S603 of Figure 6.

[0042] As shown in Figure 7(a), style conversion is learned by using the second base image 701 as the input image and the style image 702 as the ground truth image, and a style conversion model 703 is generated as a result of the learning. Here, the second base image 701 is a daytime style image taken with a short exposure, and is a noise-free image of a scene without moving objects. Similarly, the style image 702 is a nighttime style image taken with a long exposure, and is a noise-free image of the same scene. The style conversion model 703 generated using these second base image 701 and style image 702 has a style conversion function that converts from daytime style to nighttime style.

[0043] As shown in Figure 7(b), by performing inference on the style transfer model 703 using the first base image 704, which is a scene similar to the second base image 701, as input images, a pseudo-inferred image 705 that does not actually exist is generated. Here, the first base image 704 is a noiseless image taken in daytime style with a moving object, taken with a short exposure. Similarly, the pseudo-inferred image 705 is a noiseless pseudo-image in nighttime style with a moving object, taken with a short exposure. The second base image 701 and style image 702 use images of scenes similar to the first base image 704, which contains a moving object, and the style transfer is learned with image data similar to the pseudo-inferred image 705 that we want to generate. Therefore, the accuracy of the darkening style transfer can be further improved.

[0044] Next, as shown in Figure 7(c), the pseudo-inference image 705 is used as the ground truth image, and the artificially noised image 706, which is the pseudo-inference image 705 with artificial noise added, is used as the input image. By training the noise reduction model, a noise reduction model 707 that improves image quality is obtained. In training this noise reduction model, pseudo-inference images, which are actually difficult to capture noise-free, are used as the ground truth image, allowing training to be performed in scenes where noise reduction is difficult. As a result, noise reduction can be sufficiently performed even in difficult scenes where noise-free shooting is not possible.

[0045] As described above, this embodiment allows for the pseudo-generation of images in scenes where it is difficult to photograph moving objects without noise, using a style transfer model. In this way, images of difficult scenes due to the characteristics of the subject can be used as training data, and the accuracy of image quality enhancement can be improved by training a noise reduction model using this training data.

[0046] <Example 1> In this embodiment, we have described an example in which a noise reduction model is trained by using a pseudo-inferred image as the ground truth image and an artificially noised image (a pseudo-inferred image with artificial noise added) as the input image. However, the artificially noised image may be obtained by other methods. Other style transformations may be used as long as it is possible to generate a pseudo-inferred image using an image that can actually be captured and that depicts a scene where the subject is difficult to capture. For example, a method of transforming a daytime noiseless style into a nighttime noisy style (hereinafter referred to as "darkening and noise" style transformation) may be used.

[0047] In this case, the style image acquisition unit 209 specifically acquires a style image by taking a picture with noise using a short exposure of a scene similar to the second base image, where there is no moving object in the night style, and the scene is similar. By training with this style image and the second base image described in this embodiment, a style conversion model that performs "darkening and noise" style conversion is acquired. This style conversion model is a noise-adding model that converts from a daytime, noiseless style to a nighttime, noised style. When a pseudo-inference image is generated using this style conversion model and the first base image described in this embodiment, a pseudo-inference image of a nighttime style with noise is obtained by taking a picture of a moving object with a short exposure. Then, the high-image-quality learning unit 207 can learn a noise reduction model by using this pseudo-inference image as the input image and the pseudo-inference image described in this embodiment as the ground truth image. In this way, by using style conversion other than "darkening" style conversion, it is possible to generate a pseudo-inference image similar to the artificial noise-added image described in this embodiment and learn a noise reduction model.

[0048] <Modification 2> In addition to the "darkening and noise" style conversion described in Modification Example 1, a style conversion from a noisy style to a noiseless style (hereinafter referred to as "denoising" style conversion) may also be used as a style conversion.

[0049] In this case, the base image acquisition unit 208 acquires a second base image, which is a night scene without any moving objects and similar to the first base image, but captured with noise using a short exposure. Then, by learning using this second base image and the style images described in this embodiment, a style conversion model that performs "denoising" style conversion is acquired. This style conversion model has the function of converting from a noisy style to a noiseless style.

[0050] On the other hand, the base image acquisition unit 204 acquires a noisy first base image taken with a short exposure in a night style. When a pseudo-inference image is generated using this first base image and a style conversion model that performs a "denoicing" style conversion, it becomes a noiseless pseudo-image in a night style taken with a short exposure of a moving object, similar to the pseudo-inference image described in this embodiment. Then, as in this embodiment, the high-image-quality learning unit 207 learns a noise reduction model by using the pseudo-inference image as the ground truth image and the artificially noised image (a pseudo-inference image with artificial noise added) as the input image. Note that the "denoicing" in Modification 2 is denoising for the purpose of generating a pseudo-image, and is therefore different from the noise reduction process that generates an image with actual noise removed by the noise reduction model. As described above, pseudo-inference images can be generated and learning for noise reduction can be performed using style conversions other than "darkening," similar to this embodiment.

[0051] <Variation 3> In this embodiment, an example was described in which a first base image is obtained by selecting an image containing a moving object from a group of images captured by a camera using subject detection processing. However, the first base image may be obtained by other methods. For example, an image selected by the user from a group of images captured by a camera may be obtained as the first base image. In addition to selecting from a group of images captured by a camera, an image containing a moving object may be obtained as the first base image from a group of images stored on a server or the like. In this way, a pseudo-inference image can be generated by the method described in this embodiment, regardless of the method used to obtain the first base image.

[0052] <Modification 4> In this embodiment, an example was described in which a subject detection process is performed on a group of images captured by a camera, and a second base image is obtained by selecting an image in which the segmentation detection result for non-moving objects is the same as that of the first base image. On the other hand, the second base image may be obtained by other methods. For example, an image selected by the user from a group of images captured by a camera or a group of images stored on a server, etc., may be obtained as the second base image. Alternatively, a second base image linked to the first base image may be obtained according to predetermined rules.

[0053] Alternatively, object recognition processing may be performed on a set of images captured by a camera or stored on a server, etc., and images that do not contain moving objects may be selected from the recognition results and acquired as the second base image. Alternatively, the similarity of the areas excluding moving objects may be calculated for a set of images captured by a camera or stored on a server, etc., and images with a high similarity to the first base image to be used later may be acquired as the second base image.

[0054] Furthermore, in this embodiment, an image of a scene similar to the first base image and without moving objects was acquired as the second base image. However, if it is possible to generate a pseudo-inference image of a difficult scene, an image of another scene may be acquired as the second base image. For example, when performing "darkening" and "darkening and noise" style transformations, a daytime image of a different landscape scene may be used. Also, when performing "denoicing" style transformations, a nighttime image of a different landscape scene may be used. In this way, regardless of the method of acquiring the second base image, a style transformation model can be acquired by the method described in this embodiment.

[0055] <Modification 5> In this embodiment, we have described an example in which a style image is selected from a group of images captured by the camera, capturing the same range as the second base image and matching the style of the disliked scene. However, style images may be acquired by other methods. For example, an image selected by the user from a group of images captured by the camera or from a group of images stored on a server may be acquired as a style image. Alternatively, a style image linked to the second base image may be acquired according to predetermined rules.

[0056] Furthermore, in this embodiment, an image of a scene that does not contain a moving object and is similar to the first and second base images was acquired as a style image. On the other hand, an image of a scene different from the first base image may be acquired as a style image if it is similar to the second base image and can generate a pseudo-inference image of a difficult scene. For example, a night image of a different landscape scene may be acquired as a style image. In this way, a style transformation model can be obtained by the method described in this embodiment, regardless of the method used to acquire the style image.

[0057] <Variation 6> In this embodiment, an example of obtaining a style transfer model trained based on a second base image and a style image has been described, but the style transfer model may be obtained by other methods. For example, an existing trained style transfer model may be obtained according to the first base image. In this case, the existing trained style transfer model may be obtained as a model that performs the style transfers of "darkening," "darkening and noise," and "denoicing" as described in this embodiment, modifications 1 and 2. In this way, regardless of the method of obtaining the style transfer model, it is possible to generate pseudo-inference images and perform training for noise reduction using the method described in this embodiment.

[0058] (Second embodiment) In the first embodiment, we described an example in which a style transfer model is generated by learning based on images that do not contain moving objects, and an image that is difficult to capture noiselessly in situations where moving objects are present is simulated and applied to the training of a noise reduction model. In this embodiment, we describe a method for selecting the optimal style transfer and training the noise reduction model when there are multiple variations of style transfers for simulating images that are difficult to capture noiselessly in situations where moving objects are present.

[0059] Specifically, multiple style transfer models are used to generate pseudo-analytical images, and each pseudo-analytical image is used to train a noise reduction model. The training results of the noise reduction models are then compared through image quality scores, and the noise reduction model with the superior image quality score is selected. In this embodiment, an example is described in which "darkening" and "denoicing" style transfer models are selected. Note that the configuration of the high-image-quality enhancement system according to this embodiment is the same as in the first embodiment, so a detailed explanation is omitted. Below, only the differences from the first embodiment will be described.

[0060] The base image acquisition unit 204 acquires multiple images containing a moving object as first base images. In this embodiment, first, a noise-free image of a moving object taken in daytime style with a short exposure is acquired as the first first base image. Then, a noisy image of a moving object taken in nighttime style with a short exposure is acquired as the second first base image.

[0061] The model acquisition unit 205 acquires multiple style conversion models for converting the style of the first base image. In this embodiment, two types of models are acquired: a "darkening" style conversion model and a "denoicing" style conversion model.

[0062] The pseudo-inference image acquisition unit 206 acquires multiple pseudo-inference images using multiple style conversion models. In this embodiment, two types of pseudo-inference images are acquired: one generated using the "darkening" style conversion model and another generated using the "denoicing" style conversion model. Both types of pseudo-inference images are noiseless pseudo-inference images in a night style, taken with a short exposure of a moving object.

[0063] The image quality enhancement learning unit 207 acquires a trained denoising model based on multiple pseudo-inference images acquired by the pseudo-inference image acquisition unit 206. In this embodiment, a denoising model is trained using each of the pseudo-inference images acquired by the pseudo-inference image acquisition unit 206, and an image quality score is calculated for each. Finally, the denoising model with the better image quality score is acquired. Details of this process will be described later.

[0064] The base image acquisition unit 208 acquires multiple images that do not contain moving objects as second base images. In this embodiment, first, as the first second base image, an image is acquired that is taken with a short exposure in a daytime style and is a noise-free image of a scene similar to the first base image that does not contain moving objects (hereinafter referred to as the "second base image for darkening"). Then, as the second second base image, an image is acquired that is taken with a short exposure in a nighttime style and is a noise-included image of a scene similar to the first base image that does not contain moving objects (hereinafter referred to as the "second base image for denoising").

[0065] The style transfer learning unit 210 acquires a style transfer model for each second base image based on the second base image and the style image. In this embodiment, first, a "darkening" style transfer model is acquired by training with the second base image for "darkening". Then, a "denoicing" style transfer model is acquired by training with the second base image for "denoicing".

[0066] <Processing procedure and detailed processing method> Next, the processing procedure and detailed processing method of the first information processing device 202 according to this embodiment will be described. The overall processing procedure performed by the first information processing device 202 is the same as the procedure in Figure 4, and the detailed processing procedure for obtaining the style conversion model in the second information processing device 203 is the same as the procedure in Figure 5. In this embodiment, as mentioned above, processing is basically performed on two types of style conversion models: the "darkening" style conversion model and the "denoicing" style conversion model. Note that only the detailed processing procedure of step S404 in Figure 4 differs from that of the first embodiment, and this procedure will be described below with reference to Figure 8.

[0067] Figure 8 is a flowchart showing an example of a detailed processing procedure for step S404 in Figure 4 in this embodiment. Steps S601 to S603 are the same as steps S601 to S603 in Figure 6. In this embodiment, artificially noised images are generated from pseudo-inference images acquired using two different style transfer models. Then, denoising models are trained based on their respective training datasets (ground truth images and input images).

[0068] Next, in step S801, the image quality improvement learning unit 207 calculates an image quality score for each evaluation image of the learned noise reduction model. In this embodiment, the image quality score is calculated as the value of the loss function learned for the noise reduction model.

[0069] Then, in step S802, the image quality improvement learning unit 207 determines the style conversion model with the better image quality score calculated in step S801, and ultimately decides to use that style conversion model as the output noise reduction model. For example, suppose the loss function value obtained by learning the noise reduction model when using the "darkening" style conversion model is 0.05, and the loss function value obtained by learning the noise reduction model when using the "denoicing" style conversion model is calculated to be 0.10. In this case, since the loss function value when using the "darkening" style conversion model is lower, the noise reduction model trained on the pseudo-inference image obtained using the "darkening" style conversion model is decided as the output model.

[0070] As described above, according to this embodiment, when there are multiple style conversion models for generating images of scenes where it is difficult to photograph moving objects without noise, an image quality score is calculated and the most appropriate style conversion model can be selected. This makes it possible to select a more favorable style conversion model when generating training data for images of difficult scenes due to the characteristics of the subject, thereby improving the accuracy of image quality enhancement.

[0071] <Example 7> In this embodiment, an example was described in which the value of the learned loss function was calculated as the image quality score. However, other values ​​may be used as long as they allow for comparison of multiple learned denoising models. For example, values ​​such as SNR (Signal to Noise Ratio), SSIM (Structural SIMilarity), and PSNR (Peak Signal to Noise Ratio), which are image quality evaluation indices, may be used as the image quality score. In addition, values ​​such as MTF (Modulation Transfer Function) and chromatic difference (Δe) may be used as the image quality score. When using SNR, SSIM, PSNR, or MTF values ​​as the image quality score, the denoising model with the higher image quality score is determined as the output model. When using Δe values ​​as the image quality score, the denoising model with the lower image quality score is determined as the output model. In this way, regardless of the form of the image quality score, multiple denoising models can be compared and a more appropriate denoising model can be selected.

[0072] (Third embodiment) In the first and second embodiments, based on information that the subject is a moving object, training images for acquiring a style transfer model were determined, and a pseudo-analogous image was generated using an appropriate style transfer model to train a noise reduction model. In this embodiment, a method for training a noise reduction model while also taking into account the noise reduction characteristics of the image to be denoised is described. Here, the noise reduction characteristics refer to the noise reduction processing policy for each camera manufacturer in this embodiment. The noise reduction characteristics for each camera manufacturer are described below.

[0073] Figure 9 shows an example of the correspondence between camera manufacturers and noise reduction characteristics. In the example shown in Figure 9, manufacturer A's noise reduction characteristics prioritize color reproduction. Manufacturer B's noise reduction characteristics prioritize resolution reproduction, and manufacturer C's noise reduction characteristics are specialized in reproducing the colors of the landscape. As such, there are differences in noise reduction characteristics among camera manufacturers, and other aspects of image processing are also tailored to match these noise reduction characteristics.

[0074] Therefore, in this embodiment, by learning style transformation while also taking these noise reduction characteristics into account, the style transformation is made possible to match the noise reduction characteristics of the camera manufacturer, even in scenes where it is difficult to shoot noise-free with moving objects. With this approach, when generating pseudo-inference images and training the noise reduction model, it can be done in a way that matches the noise reduction characteristics of the camera manufacturer. The following describes only the differences from the first embodiment.

[0075] <Structure> Figure 12 is a block diagram showing an example of the functional configuration of the high-image-quality system 201 according to this embodiment. The first information processing device 202 according to this embodiment further includes a characteristic determination unit 1201 compared to the configuration in Figure 2. Note that in Figure 12, the style conversion learning unit 210, model acquisition unit 205, pseudo-inference image acquisition unit 206, high-image-quality learning unit 207, and high-image-quality processing unit 211 are the same as in the first embodiment, so their description is omitted.

[0076] The characteristic determination unit 1201 acquires information on the camera manufacturer's name as noise reduction policy information and determines the noise reduction characteristics from that camera manufacturer's name. The specific method for acquiring the characteristics of the noise reduction policy information will be described later. The base image acquisition unit 204 acquires an image as the first base image that matches the noise reduction characteristics of the camera manufacturer related to the noise reduction policy information acquired by the characteristic determination unit 1201 and that also contains a moving object.

[0077] The base image acquisition unit 208 acquires an image as the second base image that matches the noise reduction characteristics of the first base image and does not contain a moving object. The style image acquisition unit 209 acquires a style image that matches the same noise reduction characteristics, does not contain the moving object 104, and represents a scene that would be difficult to capture noiselessly if the moving object 104 were included.

[0078] <Processing procedure and detailed processing method> The processing procedure and detailed processing method of the first information processing device 202 according to this embodiment will be described below. Figure 10 is a flowchart showing an example of the overall processing procedure performed by the first information processing device 202 in this embodiment. Before the processing shown in Figure 10, the first information processing device 202 communicates with the camera that takes the image to be enhanced, and directly obtains the camera manufacturer's name information from the camera's memory information via the communication unit 308 and stores it in the ROM 303 or external memory 305.

[0079] First, in step S1001, the characteristic determination unit 1201 obtains information on the camera manufacturer's name as noise reduction policy information from the ROM 303 or external memory 305, and determines the noise reduction characteristics from that camera manufacturer's name.

[0080] Next, in step S401, the base image acquisition unit 204 acquires an image that matches the noise reduction characteristics determined in step S1001 and captures a moving object, as the first base image. Steps S402 to S404 are the same as steps S402 to S404 in Figure 4. The detailed processing procedure for step S404 is the same as in Figure 6.

[0081] The detailed processing procedure for acquiring the style conversion model in the second information processing device 203 is basically the same as the procedure in Figure 5. However, in steps S501 and S502, as mentioned above, images with matching noise reduction characteristics (matching camera manufacturers of the cameras that took the images) are acquired. The camera manufacturer name information may be obtained directly from the camera that takes the images to be enhanced, or it may be obtained from the first information processing device 202.

[0082] As described above, in this embodiment, the first base image, second base image, and style image are all selected to have noise reduction characteristics that match those of the image to be enhanced. This allows the style transformation to be learned in accordance with the different noise reduction characteristics of different camera manufacturers, and the noise reduction model can be trained using pseudo-inference images that match the noise reduction characteristics. As a result, a noise reduction model that matches the noise reduction characteristics can be generated in the end.

[0083] <Differentiation Example 8> In this embodiment, an example was described in which the first information processing device 202 directly obtains the camera manufacturer name information from the camera's memory information, but it may be obtained by other methods. For example, the first information processing device 202 may obtain an image to be targeted for image quality enhancement and obtain the camera manufacturer name information from the Exif information of that image. The image targeted for image quality enhancement from which the camera manufacturer name information is obtained may be an image that has been previously denoised, and may be obtained from the camera that took the image or from a server that holds the image. In this way, regardless of the method of obtaining the noise reduction characteristics, it is possible to achieve style conversion learning that takes noise reduction characteristics into account and high-precision noise reduction for image quality enhancement.

[0084] (Fourth embodiment) This embodiment describes an example of displaying the processing results described in the first to third embodiments above. Only the differences from these embodiments will be described below.

[0085] <Structure> Figure 13 is a block diagram showing an example of the functional configuration of the high-definition image enhancement system 201 according to this embodiment. The first information processing device 202 according to this embodiment further includes a notification unit 1301 compared to the configuration in Figure 2. Note that the other configurations are the same as in other embodiments, so a detailed explanation is omitted.

[0086] The notification unit 1301 displays at least one processing result from the base image acquisition unit 204, the model acquisition unit 205, the pseudo-inference image acquisition unit 206, and the high-resolution learning unit 207 on the display unit 307. In the case of displaying the processing results in the third embodiment, the processing result from the characteristic determination unit 1201 in Figure 12 may be displayed. The notification unit 1301 may also acquire at least one processing result from the base image acquisition unit 208, the style image acquisition unit 209, and the style conversion learning unit 210 from the second information processing device 203 via the communication unit 308 and display it on the display unit 307.

[0087] Figure 11 shows an example of a display screen showing the processing results in the second embodiment. Figure 11(a) shows an example of a display screen 1101 showing the results of style transfer training and the generation of pseudo-inference images using the trained style transfer model. Figure 11(b) shows an example of a display screen 1102 comparing the training results of each denoising model using multiple pseudo-inference images.

[0088] In Figure 11(a), the display area 1103 of the display screen 1101 mainly displays the training results of the style transfer model. Image 1105 is the second base image with the image file name "aaa.jpg", and image 1106 is the style image with the image file name "bbb.jpg". Furthermore, the display area 1103 also displays information about the style transfer model (model name "Model_aaa.onnx") acquired using these images. In the example in Figure 11(a), the style transfer model named "Model_aaa.onnx" is shown, and the type of style transfer, "darken", is indicated.

[0089] Furthermore, the display area 1104 on the display screen 1101 primarily displays the results of the pseudo-inference image generation. Image 1107 shows that the first base image, named "ccc.jpg," was used to generate the pseudo-inference image 1108 using a style transfer model named "model_aaa.onnx." The user can view the pseudo-inference image 1108 displayed in the display area 1104 and confirm whether or not to add it to the training dataset as the ground truth image for the denoising model. If the user decides to add the confirmed pseudo-inference image to the training dataset, they can select the "Add to dataset" button 1109, which will add the pseudo-inference image to the training dataset.

[0090] Meanwhile, the display area 1112 of the display screen 1102 in Figure 11(b) mainly displays the training results of the denoising model using each pseudo-inference image. Image 1110, with the file name "ddd.jpg", is an evaluation image in which the denoising model was trained using a pseudo-inference image generated using the "darkening" style transfer model as the ground truth image. Similarly, image 1111, with the file name "eee.jpg", is an evaluation image in which the denoising model was trained using a pseudo-inference image generated using the "denoicing" style transfer model as the ground truth image. The display area 1112 also displays the image quality score calculated for each evaluation image.

[0091] Furthermore, the display area 1113 on the display screen 1102 displays information about the final output noise reduction model. In the example in Figure 11(b), the image quality score when using the "darkening" style conversion model is 0.05, and the image quality score when using the "denoicing" style conversion model is 0.1. Therefore, the display area 1113 indicates that the final output noise reduction model will be the one using the "darkening" style conversion model, which has a better image quality score.

[0092] As described above, this embodiment displays pseudo-inference images and evaluation images of the noise reduction model, allowing the user to visually confirm the processing results.

[0093] <Modification 9> In this embodiment, an example of notification by displaying the processing results in a GUI format was described. On the other hand, as long as the processing results can be notified to the user, other notification formats may be used for things other than the training results of the denoising model, for example, the content may be notified in text format. In this way, the process up to the training of the denoising model can be notified to the user regardless of the notification method.

[0094] <Variation 10> In this embodiment, an example of displaying processing results was described using the processing content described in the second embodiment as an example. The user may switch between displaying or hiding all or part of the processing results on the display unit 307 by operating the input unit 306. In this way, the processing results can be notified to the user regardless of the usage mode of the first information processing device 202 or the second information processing device 203.

[0095] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0096] This embodiment includes the following configurations, methods, and programs.

[0097] (Composition 1) A first image acquisition means for acquiring a first base image including a moving object, A model acquisition means for acquiring a conversion model that converts an image that does not include the moving object into an image that does not include the moving object and reduces noise, in an environment where it is difficult to photograph the moving object. A generation means that generates an inference image by transforming the first base image into an environment where it is difficult to photograph the moving object, based on the first base image and the transformation model, A learning means for training a noise reduction model that removes noise from an image based on the aforementioned inferred image, An information processing device characterized by having the following features.

[0098] (Configuration 2) The first image acquisition means acquires an image of the moving object taken during the daytime as the first base image, The model acquisition means acquires a model that converts from a daytime environment to a nighttime environment as the conversion model, The information processing apparatus according to configuration 1, characterized in that the generation means generates an image as the inference image obtained by converting the first base image to a night environment. (Composition 3) The model acquisition means further acquires a noise-adding model that converts an image of a daytime environment without the moving object into an image of a nighttime environment that is noisier than the image of the daytime environment without the moving object. The information processing apparatus according to configuration 1 or 2, wherein the generation means further generates an image obtained by transforming the first base image into a night environment and having more noise than the first base image, as an input image for learning the noise reduction model, based on the first base image and the noise addition model.

[0099] (Composition 4) The first image acquisition means acquires an image of the moving object taken at night as the first base image, The model acquisition means acquires a model that transforms to reduce noise as the transformation model, The information processing apparatus according to configuration 1, wherein the generation means generates an image with less noise than the first base image as the inference image. (Composition 5) The information processing apparatus according to any one of configurations 1 to 4, characterized in that the learning means performs training of the noise reduction model using the inferred image as the ground truth image. (Composition 6) A second image acquisition means for acquiring a second base image that does not include the moving object, A third image acquisition means for acquiring images that do not include the aforementioned moving object and for acquiring images of environments where it is difficult to photograph the aforementioned moving object while reducing noise, A transformation learning means that learns the transformation model by performing learning using the second base image acquired by the second image acquisition means and the image acquired by the third image acquisition means, It further possesses, The information processing apparatus according to any one of configurations 1 to 5, characterized in that the model acquisition means acquires the transformation model learned by the transformation learning means.

[0100] (Composition 7) The model acquisition means acquires multiple types of the conversion model, The generation means generates the inference images using the acquired multiple types of transformation models, The information processing apparatus according to any one of configurations 1 to 6, characterized in that the learning means evaluates the learning of the denoising model using the inference images generated by the generation means, and determines a transformation model to be used to generate the inference images based on the results of the evaluation. (Composition 8) The system further includes a characteristic acquisition means for acquiring information on noise reduction characteristics, The information processing apparatus according to any one of configurations 1 to 7, characterized in that the first image acquisition means and the model acquisition means acquire the first base image and the conversion model, respectively, based on the noise reduction characteristics. (Composition 9) An information processing apparatus according to any one of configurations 1 to 8, further comprising a notification means for notifying at least one processing result among the first image acquisition means, the model acquisition means, the generation means, and the learning means.

[0101] (method) Image acquisition process to acquire a base image including a moving object, A model acquisition step to acquire a conversion model that converts an image that does not include the moving object into an image that does not include the moving object and reduces noise, in an environment where it is difficult to photograph the moving object. A generation step of generating an inference image by reducing noise and transforming the base image to an environment where it is difficult to photograph the moving object, based on the base image and the transformation model, A learning process is performed to train a denoising model that removes noise from an image based on the aforementioned inferred image. An information processing method characterized by having the following features.

[0102] (program) Image acquisition process to acquire a base image including a moving object, A model acquisition step to acquire a conversion model that converts an image that does not include the moving object into an image that does not include the moving object and reduces noise, in an environment where it is difficult to photograph the moving object. A generation step of generating an inference image by reducing noise and transforming the base image to an environment where it is difficult to photograph the moving object, based on the base image and the transformation model, A learning process is performed to train a denoising model that removes noise from an image based on the aforementioned inferred image. A program that causes a computer to execute something. [Explanation of symbols]

[0103] 204 Base image acquisition unit, 205 Model acquisition unit, 206 Pseudo-inference image acquisition unit, 207 High-resolution learning unit

Claims

1. A first image acquisition means for acquiring a first base image including a moving object, A model acquisition means for acquiring a conversion model that converts an image that does not include the moving object into an image that does not include the moving object and reduces noise, in an environment where it is difficult to photograph the moving object. A generation means that generates an inference image by transforming the first base image into an environment where it is difficult to photograph the moving object, based on the first base image and the transformation model, A learning means for training a noise reduction model that removes noise from an image based on the aforementioned inferred image, An information processing device characterized by having the following features.

2. The first image acquisition means acquires an image of the moving object taken during the daytime as the first base image, The model acquisition means acquires a model that converts from a daytime environment to a nighttime environment as the conversion model, The information processing apparatus according to claim 1, characterized in that the generation means generates an image as the inference image obtained by converting the first base image to a night environment.

3. The model acquisition means further acquires a noise-adding model that converts an image of a daytime environment without the moving object into an image of a nighttime environment that is noisier than the image of the daytime environment without the moving object. The information processing apparatus according to claim 2, wherein the generation means further generates an image obtained by transforming the first base image into a night environment and having more noise than the first base image, as an input image for learning the noise reduction model, based on the first base image and the noise addition model.

4. The first image acquisition means acquires an image of the moving object taken at night as the first base image, The model acquisition means acquires a model that transforms to reduce noise as the transformation model, The information processing apparatus according to claim 1, characterized in that the generation means generates an image with less noise than the first base image as the inference image.

5. The information processing apparatus according to claim 1, characterized in that the learning means performs training of the noise reduction model using the inferred image as the ground truth image.

6. A second image acquisition means for acquiring a second base image that does not include the moving object, A third image acquisition means for acquiring images that do not include the aforementioned moving object and for acquiring images of environments where it is difficult to photograph the aforementioned moving object while reducing noise, A transformation learning means that learns the transformation model by performing learning using the second base image acquired by the second image acquisition means and the image acquired by the third image acquisition means, It further possesses, The information processing apparatus according to claim 1, characterized in that the model acquisition means acquires the transformation model learned by the transformation learning means.

7. The model acquisition means acquires multiple types of the conversion model, The generation means generates the inference images using the acquired multiple types of transformation models, The information processing apparatus according to claim 1, characterized in that the learning means evaluates the learning of the denoising model using the inference images generated by the generation means, and determines a transformation model to be used to generate the inference images based on the results of the evaluation.

8. The system further includes a characteristic acquisition means for acquiring information on noise reduction characteristics, The information processing apparatus according to claim 1, characterized in that the first image acquisition means and the model acquisition means acquire the first base image and the conversion model, respectively, based on the noise reduction characteristics.

9. The information processing apparatus according to claim 1, further comprising a notification means for notifying at least one processing result from among the first image acquisition means, the model acquisition means, the generation means, and the learning means.

10. Image acquisition process to acquire a base image including a moving object, A model acquisition step to acquire a conversion model that converts an image that does not include the moving object into an image that does not include the moving object and reduces noise, in an environment where it is difficult to photograph the moving object. A generation step of generating an inference image by reducing noise and transforming the base image to an environment where it is difficult to photograph the moving object, based on the base image and the transformation model, A learning process is performed to train a denoising model that removes noise from an image based on the aforementioned inferred image. An information processing method characterized by having the following features.

11. Image acquisition process to acquire a base image including a moving object, A model acquisition step to acquire a conversion model that converts an image that does not include the moving object into an image that does not include the moving object and reduces noise, in an environment where it is difficult to photograph the moving object. A generation step of generating an inference image by reducing noise and transforming the base image to an environment where it is difficult to photograph the moving object, based on the base image and the transformation model, A learning process is performed to train a denoising model that removes noise from an image based on the aforementioned inferred image. A program that causes a computer to execute something.

Images