Information processing device, training device, and program

Abstract

To suppress, in a suitable manner, manifestation of fluctuations due to the application of image processing to a series of images that are successive in a time series.SOLUTION: A first network unit 111 inputs each of a series of image data successive in a time series into a neural network that performs prescribed image processing on the inputted image data sequentially and thereby generates first image data after the prescribed image processing. A second network unit 112, taking first image data outputted from the first network unit 111 at a first time of day or a second time of day preceding the first time of day and second image data generated by the second network unit 112 for the first image data outputted from the first network unit 111 at the second time of day, as inputs to the neural network at the first time of day, generates second image data corresponding to the first time of day.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, a learning device, and a program. [Background technology]

[0002] In recent years, in image processing techniques (e.g., image processing techniques for improving image quality) for performing various processes on images such as still images and moving images, techniques using neural networks have been actively developed. One example of such a technique is a technique for realizing high-quality image processing such as noise removal, blur removal, and super-resolution using a neural network. In addition, a technique for applying a recurrent neural network to time-series processing of moving images, audio, and the like has been proposed. A recurrent neural network has a recurrent structure in which a feature generated in processing at a previous time in time-series processing is used as an input for processing at the next time, and it is possible to obtain a feature that is effective for time-series processing. Non-Patent Document 1 discloses a technique for improving image quality by using a network with a recurrent structure in noise removal processing for moving images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2023-35928 [Non-patent literature]

[0004] [Non-Patent Document 1] Efficient Multi-Stage Video Denoising with Recurrent Spatio-Temporal Fusion, Matteo Maggioni et al., 2021. [Non-Patent Document 2] TempFormer: Temporally Consistent Transformer for Video Denoising, Mingyang Song et al., 2022. [Non-Patent Document 3] Toward Convolutional Blind Denoising of Real Photographs, Shi Guo et al., 2019. Summary of the Invention [Problem to be solved by the invention]

[0005] In a technique for performing image processing on a moving image using a neural network, image processing is performed sequentially on each of a series of frame images (still images) constituting the moving image, and the frame images after image processing are connected in chronological order to generate a moving image after image processing. When image processing is performed on a series of images having continuity in a chronological order such as a moving image, a phenomenon may become apparent in the series of images, in which artifacts occurring in the structure of an object or in flat areas of the background appear to fluctuate. Patent Document 1 and Non-Patent Document 2 disclose techniques for suppressing such fluctuations that become apparent in moving images, but there is a demand for the realization of a technique that can more effectively suppress the manifestation of fluctuations.

[0006] In view of the above problems, the present invention aims to more suitably suppress the emergence of fluctuations in a series of images having continuity along a time series when image processing is applied to the series of images. [Means for solving the problem]

[0007] The information processing device of the present invention comprises a first image processing means that generates first image data after the predetermined image processing by inputting each of a series of image data having continuity along a time series into a neural network that sequentially performs a predetermined image processing on the input image data, and a second image processing means that generates second image data after the predetermined image processing based on the neural network, for the first image data sequentially output from the first image processing means along a time series, and is characterized in that the second image processing means generates the second image data corresponding to the first time by inputting the first image data output from the first image processing means at a first time or a second time prior to the first time, and the second image data generated by the second image processing means for the first image data output from the first image processing means at the second time into the neural network at the first time. Effect of the Invention

[0008] According to the present invention, it is possible to more suitably suppress the emergence of fluctuations in a series of images having continuity along a time series, which occurs when image processing is applied to the series of images. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. [Diagram 2] FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing device and a learning device. [Diagram 3] 11 is a flowchart showing an example of processing by an information processing device and a learning device. [Figure 4] FIG. 1 is a diagram showing an example of a configuration of a neural network. [Diagram 5] FIG. 2 is a diagram illustrating an example of a configuration of a first network unit. [Figure 6] FIG. 4 is a diagram illustrating an example of a configuration of a second network unit. [Figure 7] FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing device and a learning device. [Figure 8] FIG. 1 is a diagram showing an example of a configuration of a neural network. [Figure 9] FIG. 1 is a diagram showing an example of a configuration of a neural network. [Figure 10] FIG. 1 is a diagram showing an example of a configuration of a neural network. [Figure 11] FIG. 1 is a diagram showing an example of a configuration of a neural network. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals to avoid repetitive description. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown in the drawings.

[0011] <First embodiment> A first embodiment of the present disclosure will be described below. First, an example of a hardware configuration of an information processing device 100 according to this embodiment will be described with reference to Fig. 1. The information processing device 100 includes a CPU (Central Processing Unit) 101, a memory 102, an input unit 103, a storage unit 104, an output unit 105, and a communication unit 106. These components of the information processing device 100 are connected via a bus so as to be able to transmit and receive information to and from each other.

[0012] The CPU 101 controls the operation of the entire information processing device 100 and realizes functions provided by the information processing device 100 by expanding various programs stored in the storage unit 104 or the like into the memory 102 and executing them. The memory 102 is also used as a storage area for temporarily storing various data, such as a working area for the CPU 101. The memory 102 may be realized by, for example, a random access memory (RAM) or the like. The storage unit 104 is a storage area for storing various programs and various data. The storage unit 104 may be realized by, for example, an auxiliary storage device represented by a read only memory (ROM) or a hard disk drive (HDD).

[0013] The input unit 103 serves as an input interface for receiving instructions from a user, and may be realized by an input device such as a mouse, a keyboard, or a touch panel. The output unit 105 serves as an output interface for presenting various information to the user. The configuration of the output unit 105 may be appropriately changed depending on the information output method. For example, the output unit 105 may be realized by a display device such as a display, and may display the information to be presented as an image in a predetermined display area. As another example, the output unit 105 may be realized by an audio output device such as a speaker, and may output the information to be presented as sound such as voice or electronic sound. The communication unit 106 serves as a communication interface for connecting the information processing device 100 to a network such as the Internet or a LAN (Local Area Network), etc. The configuration of the communication unit 106 may be appropriately changed depending on the type of the network to be connected and the communication method to be applied.

[0014] It should be noted that the hardware configuration of the learning device 120 according to this embodiment is substantially similar to that of the above-described information processing device 100, and therefore a detailed description thereof will be omitted. In addition, the program stored in the storage unit 104 is expanded into the memory 102 and executed by the CPU 101, thereby realizing the functional configuration described below with reference to Figures 2 and 7, etc., and the processing described below with reference to Figure 3, etc.

[0015] An example of the functional configuration of the information processing device 100 and the learning device 120 according to this embodiment will be described with reference to FIG. The information processing device 100 receives image data of an image such as a moving image, performs various image processing on the image, and outputs image data of the image after the image processing. The image processing performed on the input image by the information processing device 100 may include, for example, high-quality image processing of the target image such as noise removal, blur removal, and super-resolution. Another example of the image processing performed on the input image by the information processing device 100 may include image processing such as converting the style of the input image to another style. In this way, the type of image processing performed on the input image by the information processing device 100 is not particularly limited as long as it is image processing that performs some kind of processing on the input image. Note that, in this embodiment, for convenience, various explanations will be given focusing on the case where the input image is subjected to high-quality image processing such as the above examples. The information processing device 100 includes an image acquisition unit 110, a first network unit 111, and a second network unit 112. The information processing device 100 is also connected to an imaging device 200 via a predetermined transmission path (e.g., a cable, a network, etc.) so as to be able to transmit and receive data to and from the imaging device 200. The imaging device 200 is an imaging device having an imaging optical system and an imaging element, and outputs image data of images (e.g., moving images, still images, etc.) corresponding to the imaging results of a subject to the image acquisition unit 110 of the information processing device 100. Note that, for the sake of convenience, in this embodiment, the imaging device 200 sequentially outputs image data of moving images corresponding to the imaging results of a subject to the image acquisition unit 110.

[0016] The image acquiring unit 110 corresponds to an input interface for inputting image data of an image to be subjected to image processing by the information processing device 100 as an input image to the information processing device 100. In this embodiment, the image acquiring unit 110 acquires image data of a moving image corresponding to an imaging result by the imaging device 200 from the imaging device 200. The first network unit 111 includes a neural network that, by inputting image data, performs a predetermined image processing related to the modification of the image to the image represented by the image data, and generates image data representing the image after the image processing. Specifically, the first network unit 111 inputs the image data acquired by the image acquisition unit 110 to the neural network, thereby causing the neural network to generate image data after the image processing. The second network unit 112 has a neural network that performs image processing on input image data similar to that of the neural network of the first network unit 111. With this configuration, the second network unit 112 has a recursive structure in which the output from the first network unit 111 and the output of its own neural network are input to the neural network. In this manner, the second network unit 112 inputs the image data output from the first network unit 111 to its own neural network, causing the neural network to generate image data after image processing. In this embodiment, the image data after image processing output from the first network unit 111 corresponds to an example of "first image data", and the image data after image processing output from the second network unit 112 corresponds to an example of "second image data". The learning device 120 performs learning on the neural networks included in the first network unit 111 and the second network unit 112. An example of the configuration of the learning device 120 will be described in detail later.

[0017] Next, referring to Fig. 2 and Fig. 3(a), an example of the processing of the information processing device 100 according to this embodiment will be described, focusing in particular on the processing in the case where a predetermined image processing related to image processing is performed on a target image using a neural network. In this embodiment, an example of the image processing will be described in which a noise removal processing is applied as the above-mentioned image processing, which removes noise from an image containing noise (hereinafter also referred to as a noise-added image) to generate an image in which the influence of noise is reduced (ideally an image without noise). The image processing that the information processing device 100 applies to the target image is not limited to noise removal, and other high-image-quality image processing such as image resolution and blur removal may be applied. In addition, as long as the image processing is related to image processing, other image processing other than the high-image-quality image processing may be applied.

[0018] In S301, the image acquisition unit 110 acquires image data of a noise-added image in which noise is manifested in the image from the imaging device 200 as input image data, and inputs the image data of the noise-added image to the first network unit 111. The image acquisition unit 110 sequentially acquires image data of images sequentially acquired in time series by the imaging device 200, and sets the image indicated by the image data as an input image to the first network unit 111. Note that, for convenience, in this embodiment, it is assumed that the image captured by the imaging device 200 contains noise. An example of the noise is noise generated in an imaging element that is amplified and manifested when the sensor sensitivity is increased.

[0019] In S302, the first network unit 111 performs image processing on the input noise-added image to generate a high-quality image. The high-quality image here corresponds to an image in which the influence of noise is reduced by removing noise from the noise-added image (hereinafter, also referred to as a noise-removed image). As an example of a neural network that realizes high-quality image processing, Non-Patent Document 3 proposes a Convolutional Neural Network (CNN) that realizes noise removal. A CNN is composed of a large number of convolution layers and activation functions. In particular, a network called U-Net, which has a U-shaped structure, is used as a neural network that realizes high-quality image processing such as noise removal and super-resolution. In the network disclosed in Non-Patent Document 3, noise removal is also performed using U-Net. In this embodiment, a structure based on U-Net is also used, like the neural network disclosed in Non-Patent Document 3.

[0020] Here, an example of the configuration of the neural network used in the present embodiment will be described with reference to FIG. 4. The series of neural networks shown in FIG. 4 includes two neural networks, which are applied to a first network unit 111 that performs noise removal on an input image and a second network unit 112 that similarly performs noise removal on the input image. In addition, the series of neural networks shown in FIG. 4 sequentially inputs images that are consecutive in time series (for example, a series of frame images that constitute a moving image), and performs a predetermined image processing (for example, image processing related to noise removal) on each of the input images. Here, an example will be described in which the time at which the image processing is performed is s, and image processing is sequentially performed for each time s=0, 1, ... along the time series. In addition, hereinafter, a variable t representing time will be used in addition to the variable s.

[0021] The image processing at s=0 corresponds to the first image processing in the image processing executed from moment to moment. First, in the image processing at s=0, a noise-added image 401 at t=0 is acquired. When the image data of the acquired noise-added image 401 is input to the first network unit 111, the first network unit 111 executes image processing on the noise-added image 401 to remove noise that appears in the image. Then, the first network unit 111 outputs a noise-removed image generated as a result of the image processing as image data of a high-quality image 402 at t=0. Next, the second network unit 112 combines the high quality image 402 (noise-removed image) indicated by the image data output from the first network unit 111 with the high quality image indicated by the image data output by the second network unit 112 at the previous time in the channel direction. After that, the second network unit 112 performs image processing on the combined image as an input. Note that in the image processing at s=0, there is no high quality image corresponding to the output from the second network unit 112 at the previous time. Therefore, in such a case, the second network unit 112 generates a dummy input (dummy high quality image 403) and combines it with the high quality image 402 output from the first network unit 111, and sets the combined image as an input to itself. The second network unit 112 generates a noise-removed image by performing image processing on the above-mentioned input (combined image), and outputs the image data of the high quality image by treating the noise-removed image as the high quality image at t=0.

[0022] Next, at s=1, image data of a noise-added image 405 at t=1 is acquired. The first network unit 111 generates a high-quality image 406 (noise-removed image) by removing noise from an image represented by the input image data, and outputs image data of the high-quality image 406. The second network unit 112 combines the high-quality image 406 represented by the image data output from the first network unit 111 and the high-quality image 404 represented by the image data that is its own output in the image processing at the previous time in the channel direction, and inputs the combined image to itself. In this disclosure, a channel corresponds to a color channel represented by RGB, etc. Also, the high-quality image 404 that is the output of the second network unit 112 in the image processing at the previous time corresponds to a noise-removed image that the second network unit 112 outputs as a high-quality image at t=0 as a result of the image processing at time s=0. The second network unit 112 generates a noise-removed image by performing image processing on the above-mentioned input, and outputs image data of the noise-removed image as a high-quality image of t=1.

[0023] 5, an example of the configuration of the first network unit 111 will be described in more detail, focusing particularly on the configuration of a neural network applied to the first network unit 111. The first network unit 111 uses a U-Net as a neural network that performs image processing on an input image, the U-Net being configured with an encoder that generates features while compressing the image, and a decoder that restores the image from the compressed features. The neural network first generates features (features extracted from an image) with different resolutions and numbers of channels from an input image 501 in an encoder. The neural network applies a process 511 of applying a convolution operation and a RELU function to the input image 501 multiple times to generate features 512. A pooling process 513 is applied to the generated features 512 to reduce the resolution, and the convolution operation and the RELU function are repeated again to obtain features with an increased number of channels. The generated features 512 are used in the image restoration process described below, and are skip-combined 514 with another feature generated while being upsampled. By repeating a series of processes, the compressed feature is deconvoluted 515 to reduce the number of channels and restore the feature to an image while increasing the resolution. At this time, the feature upsampled by the deconvolution process is skip-combined with the feature generated by the encoder, and multiple convolution operations, application of the Relu function, and deconvolution are repeated. Finally, image data of a high-quality image 502 having a desired resolution and number of channels is output. The high-quality image 502 output by the first network unit 111 corresponds to an image obtained by removing noise from the input image 501. In this embodiment, a neural network configured as described above is used, but the structure of the neural network to be applied is not particularly limited as long as it is possible to perform image processing related to image processing such as improving the image quality of the input image.

[0024] Here, reference is made again to Fig. 3. In S303, the second network unit 112 determines whether or not there is image data of a high quality image that is its own output at the previous time. If the second network unit 112 determines in S303 that image data of the high quality image that was its own output at the previous time exists, the process proceeds to S304. In this case, the second network unit 112 acquires image data of the high quality image that was its own output at the previous time in S304. On the other hand, if the second network unit 112 determines in S303 that there is no image data of the high quality image that is its output at the previous time, the process proceeds to S305. In this case, the second network unit 112 generates image data of a dummy high quality image in S305.

[0025] The second network unit 112 uses the image data of the high quality image that it output in the image processing executed at the previous time as input in its own image processing at the current time. The information processing device according to this embodiment executes image processing on the noise-added image from moment to moment while sequentially acquiring image data of the noise-added image that is to be input. However, in the image processing at the start time of the image processing executed from moment to moment, the image data of the high quality image output by the second network unit 112 as a result of the image processing at the previous time is not present. Therefore, the second network unit 112 generates image data of a dummy high quality image, and uses the image data of the dummy high quality image as input instead of the image data of the high quality image output as a result of the image processing at the previous time. Here, the processing of the second network unit 112 will be described with reference to Fig. 4. In the image processing at s=0, the second network unit 112 attempts to acquire, as its input, image data of the high quality image (noise removed image) at t=-1 that was output as a result of its own image processing at s=-1. However, there is no image data of the high quality image (noise removed image) at t=-1 that corresponds to the result of the image processing at s=-1. Therefore, the second network unit 112 generates image data of a dummy high quality image 403 to be input instead of the high quality image at t=-1. The generated dummy high-quality image may be, for example, an image having the same width, height, and number of channels as the high-quality image output by the second network unit 112, and each pixel having a value of 0. Note that, as long as the dummy high-quality image has the same width, height, and number of channels as the high-quality image used as the input, the value of each pixel is not necessarily limited to 0, and for example, a value determined during learning may be applied.

[0026] Here, reference is made again to Fig. 3. In S306, the second network unit 112 combines, in the channel direction, the high quality image indicated by the image data it output in the image processing at the previous time and the high quality image at the current time indicated by the image data output by the first network unit 111. The second network unit 112 uses the combined image as an input, performs image processing to generate a high quality image (for example, a noise-removed image), and outputs image data of the high quality image. The image data of the high quality image (noise-removed image) output at this time is used as an input to the second network unit 112 in the image processing to be executed at the next time.

[0027] With reference to FIG. 6, an example of the configuration of the second network unit 112 will be described in more detail, focusing in particular on the configuration of the neural network applied to the second network unit 112. The second network unit 112 uses U-Net as a neural network that performs image processing on an input image, similar to the first network unit 111. However, since the second network unit 112 uses the high-quality image that is its output at the previous time as an input, it is configured to perform a combination process 611 for making its output a recursive input. At this time, before the combination process 611, convolution + relu function is applied multiple times to the high-quality image 602, and a feature amount is generated from the high-quality image that is the input of the image processing at the next time, and the feature amount may be used as an input. In addition, in the image processing at the current time, the high-quality image 502 output by the first network unit 111 and the high-quality image 601 output by the second network unit 112 in the image processing at the previous time are combined in the channel direction. The image generated by this combination is applied as the input of the second network unit 112 at the current time. The second network unit 112 performs a process of compressing the feature amount and restoring the image to the above input in the same manner as the first network unit 111, thereby generating a high quality image 602, and outputs image data of the high quality image 602. The high quality image 602 indicated by the image data corresponding to the output of the second network unit 112 is an image that has been subjected to higher image quality improvement than the high quality image 502 indicated by the image data corresponding to the output of the first network unit 111.

[0028] In this embodiment, a case where a neural network configured as above is used has been described, but this does not necessarily limit the structure of the neural network used in the first network unit 111 and the second network unit 112. In other words, as long as image data of an image output as a result of image processing at a previous time can be used as input to the second network unit 112 and image processing related to image processing such as image quality improvement can be performed on the input image, the structure of the neural network is not limited.

[0029] In this manner, the information processing device of the present invention performs image processing on sequentially acquired time-series images from moment to moment, thereby acquiring images after image processing (for example, high-quality images).

[0030] Next, an example of the functional configuration of the learning device according to this embodiment will be described with reference to Fig. 2. The learning device 120 learns neural networks applied to a first network unit that performs image processing such as the image quality improvement described above and a second network unit that performs image processing similar to the first network unit. The learning device 120 includes a database unit 121, an image acquisition unit 110, a degraded image generation unit 122, a first network unit 111, a second network unit 112, a difference calculation unit 126, an error calculation unit 124, and a learning unit 125.

[0031] The database unit 121 stores a large amount of image data of moving images after changes that have become apparent in the image, such as noise, caused by the target image processing have been restored. As a specific example, when a noise removal process is applied as the target image processing, the database unit 121 stores image data of clean moving images that are free of noise. This image data of a series of moving images stored in the database unit 121 corresponds to learning data (teacher data) used for learning the neural networks applied to the first network unit and the second network unit. The image acquiring unit 110 extracts image data of a moving image used for learning from image data of a series of moving images stored in the database unit 121. Then, the image acquiring unit 110 acquires image data of a series of frame images (i.e., a series of still images that are consecutive in a case sequence) constituting the moving image indicated by the extracted image data in chronological order as image data of true-value images. The degraded image generating unit 122 generates a degraded image by degrading the image quality of the true-value image based on the image data of the true-value image acquired by the image acquiring unit 110. For example, when a noise removal process is applied as the target image processing, the degraded image generating unit 122 generates a noise-added image by adding noise to the true-value image as the degraded image. The image data of the degraded image corresponds to an example of "processed image data".

[0032] The difference calculation unit 126 generates a difference image between two successive true-value images based on two successive true-value images extracted from a moving image used for input. As a specific example, the difference calculation unit 126 may generate a difference image between these true-value images based on true-value images corresponding to the current time and a previous time. The difference calculation unit 126 also generates a difference image between two successive high-quality images in a time series, which are generated by performing image processing by the second network unit 112 on two degraded images generated from the two successive true-value images in a time series. As a specific example, the difference calculation unit 126 may generate a difference image between these high-quality images based on high-quality images corresponding to the current time and a previous time. The difference image between the two true-value images corresponds to an example of a "first difference", and the difference between the two high-quality images (in other words, the difference between the images after two image processing) corresponds to an example of a "second difference". The error calculation section 124 calculates an error value using a difference image between two chronologically consecutive true-value images as a true value, and a difference image between two high-quality images corresponding to the two chronologically consecutive true-value images. The learning unit 125 uses the error value generated by the error calculation unit 124 to perform neural network learning for each of the first network unit 111 and the second network unit 112. Specifically, the learning unit 125 updates the parameters of the neural networks applied to each of the first network unit 111 and the second network unit 112, based on the error value generated by the error calculation unit 124.

[0033] Next, referring to FIG. 2 as well as FIG. 3(b), an example of the processing of the learning device 120 according to this embodiment will be described, focusing in particular on the processing related to learning of the neural network applied to the information processing device according to this embodiment.

[0034] In FIG. 3(b), a series of processes indicated by loop end S311 indicates processes related to neural network learning applied to the first network unit 111 and the second network unit 112. This series of processes repeatedly updates parameters such as the weights and biases of the neural network for each image data of a moving image corresponding to the learning data. Specifically, at the start of learning, initial parameter values ​​are given to the target neural network, and the parameters of the neural network are updated by repeatedly learning thereafter. In addition, the series of processes related to the neural network learning indicated by loop end S311 are repeatedly executed until a termination condition is satisfied. As a specific example, when the number of times the neural network has been learned reaches a predetermined number of times, it may be determined that the termination condition is satisfied, and the execution of the series of processes related to the neural network learning indicated by loop end S311 may be terminated.

[0035] Next, the content of the series of processes indicated by loop end S311 will be described below. In S312, the degraded image generating unit 122 acquires one of the image data of a series of moving images stored in the database unit 121. As described above, the database unit 121 stores a large amount of image data of clean moving images without noise. In other words, the database unit 121 stores image data of time-series images (for example, a series of images that are continuous in time, such as a series of frame images that make up a moving image) that serve as true values ​​when learning a neural network. In S313, the degraded image generating unit 122 generates a degraded image corresponding to a time-series image acquired from image data of one moving image acquired in S312. In this embodiment, a noise-added image in which noise is added to a clean image is generated as the degraded image. As a specific example, the degraded image generating unit 122 may generate a noise-added image by adding modeled noise to a target image by analyzing noise characteristics of a sensor. For example, noise generated in the process of converting photons detected by a sensor into a digital signal can be assumed as the noise source. For example, photon shot noise, read noise, dark current noise, quantization error, etc. can be cited as noise sources, and they are assumed to be modeled. The degraded image generating unit 122 associates the generated degraded image with the clean image from which it was generated, and adds time information to make them into a set. The data set of these images is then stored in the database unit 121.

[0036] The series of processes indicated by loop end S314 indicates a series of processes related to neural network learning, which are executed for the image data of one video acquired in S312. That is, a series of pairs of clean images and degraded images generated from the image data of the target video in S313 are acquired in chronological order, and a series of processes related to neural network learning, which are indicated by loop end S314, are executed for each pair. The contents of the series of processes indicated by loop end S314 will be specifically described below.

[0037] In S315, the image acquisition unit 110 acquires two sets of the clean image and the degraded image generated in S313, which are consecutive in time series. At this time, the image acquisition unit 110 acquires, as the older set of the two acquired sets, the set that is the same as the latest set among the series of sets acquired when image processing was performed at the previous time. However, if there is no set corresponding to the time to be acquired, the image acquisition unit 110 excludes that set from the acquisition targets. In this way, the image acquisition unit 110 acquires sets for the times that can be acquired. Next, the image acquisition unit 110 regards a clean image included in the acquired set as a true image, and outputs the true image and the degraded image included in the set to each component of the learning device 120. Specifically, the image acquisition unit 110 acquires one degraded image from the most recent set, and outputs the degraded image to the first network unit 111. Next, the image acquisition unit 110 acquires two true images in order from the two sets, and outputs them to the difference calculation unit 123 and the error calculation unit 124. Note that when only one set can be acquired, for example, when image processing for the first time is executed, the image acquisition unit 110 treats the image at the time that could not be acquired as not existing. As a specific example, consider a case where image processing is sequentially performed on each time series image of t=0, 1, .... In this case, assuming that the time when image processing is performed is s, in image processing starting at time s=0, an attempt is made to obtain a degraded image at t=0 and a true image at t=-1, 0 from the set of time t=-1, 0. However, in the case of s=0, since a true image at t=-1 does not exist, only the true image at t=0 is obtained. In addition, in image processing at the next time s=1, a degraded image at t=1 and a true image at t=0, 1 are obtained from the set of time t=0, 1.

[0038] Next, a series of processes from S302 to S306 are performed on the degraded image input to the first network unit 111 in S315, and a noise-removed image is acquired as an image with improved image quality. Note that the series of processes from S302 to S306 are substantially similar to the series of processes from S302 to S306 shown in FIG. 3(a), and therefore detailed description thereof will be omitted. The second network unit 112 outputs the high quality image output as a result of image processing at the current time and the high quality image at the previous time used as input to the difference calculation unit 123 and the error calculation unit 124. The second network unit 112 also holds the high quality image corresponding to the current time, which will be input to itself in image processing at the next time. As described above, when there is no high-quality image from the previous time, the second network unit 112 uses a dummy image instead of the high-quality image to be input to itself. In this case, the second network unit 112 uses, as the dummy image, an image that has the same width, height, and number of channels as the corresponding high-quality image and has a value of 0 for each pixel. However, as long as the dummy image has the same width, height, and number of channels as the high-quality image used as the input, the value of each pixel does not necessarily have to be 0. Furthermore, when the above-mentioned information processing device 100 executes image processing, for example, when a dummy image is used for input to the second network unit 112, a dummy high-quality image determined at the time of learning may be used as the dummy image.

[0039] In S316, the difference calculation section 123 generates a true-value difference image using the true-value image at the previous time acquired from the image acquisition section 110 and the true-value image at the current time. The difference calculation section 123 also generates a true-value difference image using the high-image-quality image at the previous time and the high-image-quality image at the current time. Here, the process of generating each of the difference images by the difference calculation unit 123 will be described in more detail. The difference calculation unit 123 generates a difference image using two images that are consecutive in time series. t and the true image I t-1 The true difference image I calculated usingdiff is expressed by the relational expression shown below as equation (1).

[0040]

number

[0041] In addition, the high-resolution image I^ t-1 and the high-resolution image I^ at the current time. t The difference image I^ of the improved image is calculated using diff is expressed by the following relational expression shown as Equation (2). Note that in this disclosure, the notation "I^" refers to the letter "I" with a hat attached.

[0042]

number

[0043] In S317, the error calculation unit 124 calculates the error used to update the parameters of the neural network. Here, the process of calculating the error by the error calculation unit 124 will be described in more detail. The error calculation unit 124 calculates an error used for learning to enable the neural network to execute high quality image processing. Specifically, the error calculation unit 124 first calculates an error (hereinafter also referred to as an image reconstruction error) between a high quality image at the current time and a true value image at the current time, which is a true value corresponding to the high quality image. Next, the error calculation section 124 calculates a difference error between frames using the difference image of the true image generated in S316 and the difference image of the image-enhanced image. These errors are added together to obtain the final error.

[0044] Image reconstruction error L i is the high-resolution image I^ at the current time. t and the true image I t Using these, the L1 error is calculated based on the relational expression shown below as equation (3).

[0045]

number

[0046] Interframe differential error L f is the difference image I^ of the improved image. diff and the true difference image I diff Based on this, the L1 error is calculated as expressed by the relational expression shown below as equation (4). Note that M in equation (4) is expressed by the relational expression shown below as equation (5), and is a mask created according to the difference value of the true-value difference image. The mask M is created so that its value becomes 0 or 1 according to a predetermined threshold value th. As shown below as equation (4), the inter-frame difference error L f is calculated by multiplying the L1 error calculated based on the difference image by the elements of mask M. For example, when a small value such as th=0.01 is set and mask M is created, an error value is calculated for a portion where there is little change between chronologically consecutive images, that is, a stationary region with little movement. In other words, weighting is performed by mask M so that a larger weight is given to the error in a portion where there is less change between true-value images. In addition, from the characteristics of the mask M described above, it is also possible to control the calculation target of the error value by adjusting the element values ​​(in other words, weights) corresponding to each condition in the mask M. As a specific example, by inverting 0 and 1 shown as values ​​corresponding to the threshold value th in the following formula (5), it is also possible to make the parts with larger changes between true value images (i.e., parts with large movement) the calculation target of the error value.

[0047]

number

[0048] The image reconstruction error L i and the interframe differential error L fBased on these, an error L that is finally used to update the parameters of the neural network is calculated. For example, the error L is expressed by the relational expression shown below as Equation (6). Note that the coefficient α shown in Equation (6) is the inter-frame error L f This is a term for adjusting the value of

[0049]

number

[0050] In S318, the learning unit 125 updates the parameters of the neural networks of the first network unit 111 and the second network unit 112 by the error backpropagation method using the error calculated in S317.

[0051] In this manner, the series of processes from S311 to S318 are repeatedly executed, whereby learning is performed for the neural network that performs image processing such as image quality improvement on the image represented by the target image data.

[0052] As described above, according to this embodiment, when a neural network is trained to process a series of images successive in time series by sequentially inputting each of the images, it becomes easy to learn the correspondence between successive images in time series, that is, spatiotemporal information. This is because an image after image processing at a previous time (e.g., an image with high image quality) is input to the second stage of a two-stage neural network that executes the same image processing task. Here, the reason why the above-mentioned operational effect can be achieved will be explained by focusing on the case where a noise removal process is performed from a target image. In this embodiment, first, a noise-removed image in which noise has been removed to a certain extent from an input image is output by image processing using a first-stage neural network. In addition, a noise-removed image in which noise has been removed to a certain extent in the first stage and a noise-removed image which is the output of the second-stage neural network at the previous time are input to a second-stage neural network. By applying such a configuration, the second-stage neural network learns by inputting images with less noise that are consecutive in time series, making it easier to learn the correspondence between two images. In addition, in this embodiment, the inter-frame difference error is used to train a neural network that performs image processing such as image quality improvement on the input image. With this configuration, even in a situation where image processing is performed on a series of images that have continuity along a time series, such as a moving image, it is expected to have the effect of further suppressing the manifestation of the phenomenon in which artifacts generated in the images appear to fluctuate.

[0053] <Second embodiment> A second embodiment of the present disclosure will be described below. In the above-mentioned first embodiment, an example of a recurrent neural network was described, in which an image after image processing at a previous time (e.g., an image with high image quality) is input to the second stage of a two-stage neural network that performs similar image processing on an input image. In this embodiment, an example of a neural network that receives multiple images as input and outputs multiple images that have been subjected to image processing such as image quality improvement will be described.

[0054] An example of a functional configuration of an information processing device 700 according to this embodiment will be described with reference to Fig. 7(a). The information processing device 700 according to this embodiment differs from the information processing device 100 shown in Fig. 2(a) in that it has a storage unit 104. The storage unit 104 is a storage area for holding image data of an image (e.g., a high-quality image) after image processing that is generated by image processing executed every moment by the information processing device 700 according to this embodiment. Note that other components are substantially similar to those of the information processing device 100 shown in FIG. 2(a), and therefore detailed description thereof will be omitted. Also, for the sake of convenience, in the following, similar to the first embodiment described above, various explanations will be given focusing on the case where noise removal processing for removing noise that has become apparent in the target image is applied as image processing by the information processing device 700.

[0055] An example of the configuration of a neural network used in this embodiment is shown in Fig. 8. Fig. 8 shows a series of inputs and outputs of the neural network at processing time s of image processing executed moment by moment by the information processing device 700 according to this embodiment. For convenience, in this embodiment, the start time of image processing executed by the information processing device 700 is set to s=0, and the time when the next image processing is executed is set to s=1.

[0056] Here, referring to Figs. 3(a), 7(a), and 8, an example of the processing of the information processing device 700 according to this embodiment will be described, focusing in particular on the processing when a predetermined image processing related to image manipulation is performed on a target image using a neural network.

[0057] In S301, the image acquisition unit 110 acquires image data of a noise-added image in which noise is manifested in the image from the imaging device 200 as input image data, and inputs the image data of the noise-added image to the first network unit 111. At this time, the image acquisition unit 110 acquires image data of three chronologically consecutive images as input images, and sets the three images indicated by the series of image data as input images to the first network unit 111. The information processing device 700 according to the present embodiment is designed so that, among three chronologically consecutive images acquired by the image acquisition unit 110, the same image as the image at the last time in the image processing at the previous time is used as the image at the first time. Specifically, the three noise-added images indicated by the series of image data acquired by the image acquisition unit 110 in the image processing at s=0 are the image 800 at time t=0, the image 801 at t=1, and the image 802 at t=2. Also, the three noise-added images indicated by the series of image data acquired by the image acquisition unit 110 in the image processing at s=1 are the image 802 at time t=2, the image 811 at t=3, and the image 814 at t=4. Thus, in the example shown in FIG. 8, in each image processing at s=0 and 1, the image acquisition unit 110 acquires image data of each of the three noise-added images to be input to each image processing, so that the image 802 (noise-added image) at time t=2 becomes the input to the neural network.

[0058] In S302, the first network unit 111 executes image processing using the noise-added image as input, and outputs image data of a noise-removed image corresponding to the result of the image processing to the second network unit 112. At this time, the first network unit 111 also outputs image data of the noise-removed image corresponding to the result of the image processing to the storage unit 104. As a result, the image data of the noise-removed image output from the first network unit 111 is saved in the storage unit 104. As the neural network applied to the first network unit 111 according to this embodiment, a neural network having a structure similar to that of the example shown in Fig. 5 is applied. However, in this embodiment, three chronologically consecutive noise-added images are applied as input to the neural network, and image data of three noise-removed images (images with high image quality) corresponding to the input images is output from the neural network. Specifically, in image processing at s=0, an image 800 at time t=0, an image 801 at t=1, and an image 802 at t=2 are used as noise-added images to be input. At this time, the first network unit 111 outputs a noise-removed image 803 at time t=0, a noise-removed image 804 at t=1, and a noise-removed image 805 at t=2 as high-quality images corresponding to the input images, respectively. The same is true for the image processing at s=1, but in response to the input of the noise-added image at t=2 that overlaps in the image processing at each time, the first network unit 111 outputs noise-removed images 805 and 815 that are different from each other. In addition, in the image processing at s=1, the image data of the noise-removed image 804 at time t=1 output by the first network unit 111 at the previous time s=0 and the noise-removed image 816 at t=2 are stored in the memory unit 104 to be input to the second network unit 112.

[0059] In S303, the second network unit 112 determines whether or not there is image data of a noise-removed image (image with improved image quality) that is its own output at the previous time. If the second network unit 112 determines in S303 that image data of the noise-removed image, which is its own output at the previous time, exists, the process proceeds to S304. In this case, the second network unit 112 acquires image data of the noise-removed image, which is its own output at the previous time, in S304. On the other hand, if the second network unit 112 determines in S303 that there is no image data of the noise-removed image (high-image-quality image) that is its output at the previous time, the process proceeds to S305. In this case, the second network unit 112 generates image data of a dummy high-image-quality image in S305. The second network unit 112 determines whether or not image data exists corresponding to the output of the first network unit 111 at the previous time and the output of the second network unit 112 at the previous time as a high quality image (noise removed image) corresponding to the previous time. If image data of the high quality images corresponding to the previous time exists, it is stored in the storage unit 104. Specifically, in the image processing at s=0, the second network unit 112 attempts to acquire, as its input, image data of the noise-removed image (high-quality image) at t=-1 output from the first network unit 111 at s=-1 and the high-quality image at t=0. The second network unit 112 also attempts to acquire image data of the noise-removed image (high-quality image) at t=-2 output from itself at s=-1. However, there is no image data of the noise-removed image corresponding to the result of the image processing at s=-1. Therefore, the second network unit 112 generates dummy image data of the high-quality image corresponding to the output of each of the first network unit 111 and itself as image data of the dummy high-quality image to be input instead of image data of the high-quality image corresponding to the result of the image processing at s=-1. In addition, in the image processing of s=1, the second network unit 112 attempts to acquire image data of the high quality image at t=1 (noise removed image 804) and the high quality image at t=2 (noise removed image 805) output from the first network unit 111 in the image processing of s=0. In addition, the second network unit 112 also attempts to acquire image data of the high quality image at t=0 (noise removed image 811) output from itself in the image processing of s=0.

[0060] In S304, the second network unit 112 acquires from the storage unit 104 image data of a noise-removed image (high-image-quality image) at a time prior to the time of input to the second network unit 112 itself. Specifically, in image processing for s=1, the second network unit 112 acquires, as its own input, image data of the noise-removed image (high-quality image) for t=1 and the high-quality image for t=2 output from the first network unit 111 in image processing for s=0. In addition, the second network unit 112 acquires image data of the noise-removed image for t=0 output by itself in image processing for s=0.

[0061] In S305, the second network unit 112 generates image data of a dummy high quality image to be input to the second network unit 112 itself. Specifically, when there is no image data of a high-quality image (noise-removed image) at a time before the second network unit 112 inputs to itself, the second network unit 112 generates a dummy input. The dummy high-quality image generated at this time has the same width, height, and number of channels as the high-quality image output by the first network unit 111 and the high-quality image output by the second network unit 112, and each pixel has a value of 0. However, as long as the dummy high-quality image has the same width, height, and number of channels as the high-quality image used as the input, each pixel does not necessarily have to have a value of 0.

[0062] In S306, the second network unit 112 generates a feature quantity indicating the feature of the image, using the high quality image or a dummy high quality image of the previous time and the high quality image output by the first network unit 111. The feature quantity generated at this time will be described in detail later together with the structure of the neural network applied to the second network unit 112. The second network unit 112 executes image processing using the generated feature quantity as an input, generates a noise-removed image (high quality image) at the time corresponding to the input feature quantity, and outputs image data of the noise-removed image. The image data of the noise-removed image output at this time is used as an input to the second network unit 112 in the image processing executed by the information processing device 100 from moment to moment, and is therefore stored in the storage unit 104.

[0063] Next, a description will be given of an example of the structure of a neural network applied to the second network unit 112. The neural network applied to the second network unit 112 has the same structure as that shown in FIG. The second network unit 112 receives as input a feature quantity generated using four chronologically consecutive high-quality images. Specifically, when image processing at a certain time is executed, the second network unit 112 acquires two images in the order of oldest time among three high-quality images output from the first network unit 111 as a series of image data. The second network unit 112 also acquires two images in the order of newest time among three high-quality images output from the first network unit 111 as a series of image data in the image processing at the previous time. The second network unit 112 also acquires the image at the middle time among three high-quality images output by itself as a series of image data in the image processing at the previous time. Then, when there are images at the same time between the high-quality image at the current time and the high-quality image at the previous time, the second network unit 112 synthesizes these high-quality images. However, when there is no high-quality image at the previous time, the second network unit 112 uses a dummy high-quality image at the previous time. The second network unit 112 then combines the result of synthesizing the multiple target high quality images in the channel direction to generate a feature quantity to be input to itself, and performs image processing on the feature quantity to generate a high quality image to be output by itself. At this time, the second network unit 112 generates image data of the high quality images for three times, which are the output of the first network unit 111 used for the input, as an output according to the result of the image processing, and a noise-removed image for a time period in which the image quality has been further improved. At this time, the image data of the series of noise-removed images output from the second network unit 112 is stored in the storage unit 104 to be used in the image processing at the next time. As described above, the feature amount in this embodiment can be said to correspond to a composite image obtained by combining a plurality of images (a plurality of images that may include a dummy image in part). Therefore, when the term image data is described in this disclosure, unless otherwise specified, it can include both data indicating the image itself and data indicating the feature amount generated based on the plurality of images as described above (in other words, a composite image obtained by combining a plurality of images).

[0064] An example of the processing of the second network unit 112 will be described in more detail with a specific example with reference to Fig. 8. In the image processing of s=0, the second network unit 112 attempts to acquire each image data of the noise-removed image (high-quality image) at t=-1, the noise-removed image at t=0, and the noise-removed image at t=1 output from the first network unit 111. Specifically, the second network unit 112 first acquires image data of the high-quality image at t=0 (noise-removed image 803) which is the output of the first network unit 111 in the image processing at s=0, and the high-quality image at t=1 (noise-removed image 804). Next, the second network unit 112 attempts to acquire image data of the high-quality image at t=-1 which is the output of the first network unit 111 in the image processing at the previous time, that is, s=-1, and the high-quality image at t=0. However, the image processing at s=-1 has not been performed, and image data of the target high-quality image does not exist. Therefore, the second network unit 112 acquires image data of dummy images 807 and 808 instead of the image data of the target high-quality image. The second network unit 112 also attempts to acquire image data of the high-quality image at t=-2 which is its own output in the image processing at s=-1, but similarly, the image processing has not been performed, and image data of the target high-quality image does not exist. Therefore, the second network unit 112 acquires image data of the dummy 806 instead of the image data of the target image quality improved image.

[0065] The second network unit 112 synthesizes images corresponding to the same time among the images that are input to the second network unit 112 as described above. Specifically, the second network unit 112 synthesizes a dummy 808 at t=0 in the image processing at s=-1 and a noise-removed image 803 (high-image quality image) at t=0 that is the output of the first network unit in the image processing at s=0. The dummy 808 and the noise-removed image 803 are synthesized, for example, by adding up the pixel values ​​of each image for each channel and generating an image whose pixel value is the average value of the values ​​for each channel. For example, a synthesis method determined at the time of learning the neural network is applied as a method of synthesizing the above images. In addition, the above method of synthesizing the images is merely an example, and the method is not particularly limited as long as the result obtained by synthesis can maintain the same width, height, and number of channels as the high-image quality image from which the synthesis is performed.

[0066] The second network unit 112 generates a feature by combining the dummy image 806 at t=-2, the dummy image 807 at t=-1, the synthesis result 809 at t=0, and the noise-removed image 804 at t=1 in the channel direction, which are obtained as described above, and uses the feature as an input to itself. At this time, for the time corresponding to the output image of the first network unit used for input to the second network unit 112, the image output by the second network unit 112 is treated as the output of the second network unit 112 at that time. Specifically, the second network unit 112 performs image processing on the above-mentioned feature amounts that are input to itself, and outputs image data of a noise-removed image 810 (high-quality image) at t=-1, a noise-removed image 811 at t=0, and a noise-removed image 812 at t=1. However, for the noise-removed image 810 (high-quality image) at t=-1, since the input image is a dummy, the output result is also a dummy image.

[0067] In the image processing of s=1, the second network unit 112 acquires image data of the noise-removed image of the high quality images t=1, t=2, and t=3, which are outputs of the first network unit 111. Specifically, the second network unit 112 first acquires image data of a noise-removed image 815 (high-quality image) at t=2, which is the output of the first network unit 111 at s=1, and a noise-removed image 816 at t=3. Next, the second network unit 112 acquires image data of noise-removed images 804 and 805 (high-quality images) at t=1 and t=2, which are the output of the first network unit 111 in the image processing at the previous time, that is, s=0. The second network unit 112 also acquires image data of a noise-removed image 811 (high-quality image) at t=0, which is its own output in the image processing at s=0. Then, the second network unit 112 synthesizes images at the same time among the series of images to be input, which are indicated by the series of image data acquired. Specifically, the second network unit 112 synthesizes a noise-removed image 805 (high quality image) at t=2 in the image processing at s=0 and a noise-removed image 815 (high quality image) at t=2 which is the output of the first network unit 111 in the image processing at s=1. Then, the second network unit 112 combines the acquired noise-removed image 811 at t=0, the noise-removed image at t=1, the synthesis result 818 at t=2, and the noise-removed image 816 at t=3 in the channel direction to generate features, and inputs the features to itself. Then, the second network unit 112 performs image processing on the features that are input to itself as described above, and outputs image data of a noise-removed image 819 (high-quality image) at t=1, a noise-removed image 820 at t=2, and a noise-removed image 821 at t=3.

[0068] In this manner, the information processing device of this embodiment sequentially performs image processing on images (e.g., frame images constituting a moving image) acquired from moment to moment, thereby acquiring an image after image processing (e.g., a high-quality image such as a noise-removed image).

[0069] Next, a learning device that performs learning of the neural networks applied to the first network unit 111 and the second network unit 112 in this embodiment will be described.

[0070] An example of the functional configuration of a learning device 701 according to this embodiment will be described with reference to Fig. 7(b). The learning device 701 differs from the learning device 120 shown in Fig. 2(b) in that it includes a storage unit 104. The storage unit 104 is a storage area for holding image data of a processed image (e.g., a high-quality image) generated by image processing executed every moment by the learning device 701 according to this embodiment. Note that other components are substantially similar to those of the learning device 120 shown in FIG. 2(b), and therefore detailed description thereof will be omitted.

[0071] 3 and 7(b), an example of the processing of the learning device 701 according to this embodiment will be described, focusing in particular on the processing related to learning of the neural network applied to the information processing device 700 according to this embodiment. Note that in this embodiment, detailed description of the processing that is substantially the same as that in the first embodiment will be omitted.

[0072] 3(b), a series of processes indicated by loop end S311 indicates processes related to learning of neural networks applied to the first network unit 111 and the second network unit 112. Through this series of processes, parameters such as the weights and biases of the neural network are repeatedly updated for each image data of a moving image corresponding to learning data.

[0073] In S312, the degraded image generating unit 122 acquires one of the image data of the series of moving images stored in the database unit 121. In S313, the degraded image generating unit 122 generates a degraded image corresponding to a time-series image acquired from image data of one moving image acquired in S312. In this embodiment, it is assumed that a noise-added image in which noise is added to a clean image is generated as the degraded image. The degraded image generating unit 122 associates the generated degraded image with the original clean image, adds time information to make them into one set, and stores the set of image data in the database unit 121.

[0074] The series of processes indicated by loop end S314 indicates a series of processes related to neural network learning, which are executed for the image data of one video acquired in S312. That is, a series of pairs of clean images and degraded images generated from the image data of the target video in S313 are acquired in chronological order, and a series of processes related to neural network learning, which are indicated by loop end S314, are executed for each pair. The contents of the series of processes indicated by loop end S314 will be specifically described below.

[0075] In S315, the image acquisition unit 110 acquires five sets of clean images and degraded images generated in S313 in chronological order. At this time, the image acquisition unit 110 acquires two sets from the oldest time out of the five acquired sets, which are the same as the two sets from the latest time out of the series of sets acquired when image processing was performed at the previous time. However, if there is no set corresponding to the time to be acquired, the image acquisition unit 110 excludes that set from the acquisition target. In this way, the image acquisition unit 110 acquires sets for the times that can be acquired. Next, the image acquisition unit 110 regards a clean image included in the acquired set as a true image, and outputs the true image and the degraded image included in the set to each component of the learning device 120. Specifically, the image acquisition unit 110 acquires three degraded images in order starting from the most recent set, and outputs the three degraded images to the first network unit 111. Next, the image acquisition unit 110 acquires four true images in order starting from the second most recent set, and outputs them to the difference calculation unit 123 and the error calculation unit 124. Note that when only three sets can be acquired, for example when image processing for the first time is executed, the image acquisition unit 110 treats the images at the time that could not be acquired as not existing.

[0076] Here, consider an example in which image processing is performed sequentially on each time series image of t=0, 1, 2, 3, 4, 5, .... In this case, the time when image processing is performed is s, and in the image processing at the start time s=0, an attempt is made to acquire image data of the degraded images at t=0, 1, 2 and the true images at t=-2, -1, 0, 1 from the set of times t=-2, -1, 0, 1, 2. However, in the case of s=0, image data of the true images at t=-2, -1 does not exist, so only image data of the true images at t=0, 1 is acquired. In addition, in the image processing at the next time s=1, an attempt is made to acquire image data of the degraded images at t=2, 3, 4 and the true images at t=-2, -1, 0, 1 from the set of times t=1, 2, 3, 4, 5.

[0077] Next, a series of processes from S302 to S306 are performed on the degraded image input to the first network unit 111 in S315, and a noise-removed image is acquired as an image with improved image quality. Note that the series of processes from S302 to S306 are substantially similar to the series of processes from S302 to S306 shown in FIG. 3(a), and therefore detailed description thereof will be omitted. In addition, when a dummy image is used as the high-quality image at the previous time, for example, a dummy image having the same width, height, and number of channels as the corresponding high-quality image and a value of 0 for each pixel is used. However, as long as the width, height, and number of channels of the dummy image are the same as those of the high-quality image used as the input, the value of each pixel does not necessarily have to be 0. In addition, when there are multiple images corresponding to the same time in the image to be input to the second network unit 112, a composite of these images is used as the input to the second network unit 112. In this case, a method of combining multiple images may be used, for example, by adding up the pixel values ​​of each image for each channel and generating an image whose pixel value is the average value of the values ​​for each channel. Of course, the above-mentioned method of combining images is merely one example, and the method is not particularly limited as long as the result obtained by the combination can maintain the same width, height, and number of channels as the original high-quality image.

[0078] In S316, the difference calculation unit 123 generates a true-value difference image by using the chronologically consecutive true-value images acquired from the image acquisition unit 110. At this time, the difference calculation unit 123 generates the difference image by calculating the difference between one time period in the same manner as in the first embodiment described above. The difference calculation unit 123 according to this embodiment obtains four high-quality images successive in time series, using three high-quality images successive in time series output in image processing at a certain time and one of the three high-quality images output in image processing at the previous time. The difference calculation unit 123 uses these four high-quality images to calculate the difference between the images in the same manner as in the first embodiment, thereby generating three difference images. As a specific example, in image processing at time s=1, image data of three noise-removed images (high-quality images) corresponding to t=2, 3, and 4 is output. In addition, in image processing at the previous time s=0, image data of three noise-removed images (high-quality images) corresponding to t=-1, 0, and 1 is output. In this case, the difference calculation unit 123 uses the noise-removed images (high-quality images) at t=1, 2, 3, and 4 to calculate the difference between one time, thereby generating a difference image of three high-quality images. The difference calculation unit 123 calculates the true value image at the time corresponding to the difference image of the high quality image. The image data is acquired. Then, the difference calculation section 123 generates three true value difference images in the same manner as in the case of generating the difference image of the image quality improved image described above.

[0079] In S317, the error calculation unit 124 calculates the error used to update the parameters of the neural network. Image reconstruction error L i is the image quality improved image I^ which is the output of the second network unit 112. t and the true image I tUsing these, the L1 error is calculated based on the relational expression shown below as Equation (7). At this time, the error calculation unit 124 uses the three high-quality images output in one image processing and the true-value image corresponding to that time, and determines the sum of the L1 errors calculated between the images at each time as the final error value.

[0080]

number

[0081] The inter-frame difference error Lf is calculated by multiplying the difference image I^ of the high-quality image by the same method as in the first embodiment. diff and the true difference image I diff Based on this, an L1 error is calculated, which is expressed by the relational expression shown below as equation (8). Note that M shown in equation (8) is a mask created according to the difference value of the true-value difference image shown as equation (5) in the first embodiment. However, since there are three difference images, the error calculation unit 124 determines the sum of the L1 errors calculated between the difference images of the high quality images corresponding to each time interval and the true-value difference images as the final error value.

[0082]

number

[0083] The image reconstruction error L i and the interframe differential error L f Based on this, the error L that is finally used to update the parameters of the neural network is calculated based on the relational expression shown as equation (6) in the same manner as in the first embodiment described above.

[0084] In S318, the learning unit 125 updates the parameters of the neural networks of the first network unit 111 and the second network unit 112 by the error backpropagation method using the error calculated in S317.

[0085] In this manner, the series of processes from S311 to S318 are repeatedly executed, whereby learning is performed for the neural network that performs image processing such as image quality improvement on the image represented by the target image data.

[0086] As described above, according to this embodiment, when learning a neural network that sequentially inputs a series of images in a time series and performs image processing on the images, it is easy to learn the correspondence between the images in a time series, that is, the spatiotemporal information. Therefore, even in a situation where image processing is performed on a series of images having continuity along a time series, such as a moving image, it is expected to have an effect of suppressing the manifestation of the phenomenon in which artifacts generated in the images appear to fluctuate. Furthermore, according to this embodiment, since image processing is performed on multiple images collectively, it is expected to have an effect of improving the processing speed related to image processing compared to a case where a series of images (e.g., a moving image) after image processing is generated while performing image processing sequentially.

[0087] <Variation 1> A first modified example of each embodiment of the present disclosure will be described below. In this modified example, as a derivative of the second embodiment described above, an example of a configuration will be described in which, in image processing at successive times, high-quality images that are the output of the second network unit 112 do not include high-quality images at the same time.

[0088] FIG. 9 is a diagram showing an example of the configuration of a neural network applied to the first network unit 111 and the second network unit 112 according to this modified example. In the configuration of the neural network according to the second embodiment described above, a noise-removed image (high-quality image) at t=1, which is the output of the second network unit 112, is output in each of the image processes at successive times s=0 and s=1. In contrast, in the configuration shown in FIG. 9, a dummy noise-removed image 908 at t=-1, a noise-removed image 909 at t=0, and a noise-removed image 910 at t=1 are acquired in the image process at s=0 as high-quality images that are the output of the second network unit 112. In addition, in the image process at s=1, a noise-removed image 917 at t=2, a noise-removed image 918 at t=3, and a noise-removed image 919 at t=4 are acquired as high-quality images that are the output of the second network unit 112.

[0089] With this configuration, it is possible to eliminate duplication of high quality images output between successive image processing times. With this configuration, when a series of images (e.g., a moving image) is generated while executing image processing sequentially, it is possible to increase the number of images output in one image processing compared to the configuration according to the second embodiment. This is expected to improve the processing speed of image processing.

[0090] <Variation 2> A second modified example of each embodiment of the present disclosure will be described below. In this modified example, an example of a derived form of a neural network applied to the second network unit 112 described above in the first and second embodiments will be described. For example, Fig. 10 is a diagram showing an example of the configuration of a neural network applied to the second network unit 112 according to this modified example. 10, a high-quality image 1000 from the second network unit 112 at a previous time used as input is connected in the channel direction to a high-quality image that is the image processing result of the second network unit 112 at the current time by skip connection 1002. After that, the second network unit 112 applies convolution and a Relu function to the high-quality image after the connection, and outputs the result again as a high-quality image 1003. By applying such a configuration, the high-quality image 1000, which is the result of image processing at the previous time, is directly combined with the high-quality image, which is the result of image processing at the current time, which is expected to result in improved effectiveness in suppressing fluctuations.

[0091] <Modification 3> Modification 3 of each embodiment of the present disclosure will be described below. In this modification, a configuration in which a noise estimation network unit that estimates noise manifested in an input image is added will be described as an example of a derived form of the neural network described above in the first and second embodiments. Since the noise estimation network unit disclosed in Non-Patent Document 3 can be applied, detailed description will be omitted. For example, FIG. 11 is a diagram showing an example of the configuration of a neural network according to this modification. The neural network shown in FIG. 11 differs from the neural network described in Modification 1 in that a noise estimation network unit 1120 is added.

[0092] Image processing at time s=0 will be described. First, the noise estimation network unit 1120 executes noise estimation using as input time-series images, a noise-added image at t=0, a noise-added image at t=1 1101, and a noise-added image at t=2 1102. The noise estimation network unit 1120 outputs a noise map for each of the input time-series images as a result of noise estimation. In the example shown in FIG. 10, a noise map 1103 at t=0, a noise map 1104 at t=1, and a noise map 1105 at t=2 are output. Note that the noise map in this modification is a map in which the noise intensity of the input noise-added image is associated with a position in the noise-added image as a pixel value.

[0093] Thereafter, the first network unit 111 executes image processing using as input image data of feature amounts obtained by combining the noise-added image and the noise map in the channel direction for each time along the time series. Specifically, for t=0, the noise-added image 1100 and the noise map 1103 are combined in the channel direction. Similarly, for t=1, the noise-added image 1101 and the noise map 1104 are combined, and for t=2, the noise-added image 1102 and the noise map 1105 are combined. As described above, the feature amounts 1106, 1107, and 1108 obtained by combining the noise-added image and the noise map for each time along the time series are further combined in the channel direction to generate image data indicating the feature amounts, which becomes the input for image processing by the first network unit 111. Note that the subsequent processing up to when the second network unit 112 outputs image data of an image after image processing (for example, an image with high image quality) is substantially similar to the above-described embodiments and other modified examples.

[0094] By applying such a configuration, it becomes possible for the neural network to learn the characteristics of the image indicated by the noise map. Therefore, for example, it is possible to expect an improvement in the image quality of the image output as a result of image processing. In addition, by adjusting the strength of the noise map, it is also possible to adjust the strength of noise removal for the input image by the neural network.

[0095] <Modification 4> A fourth modified example of each embodiment of the present disclosure will be described below. In the first and second embodiments described above, a learning method was described in which a plurality of still images (e.g., frame images) are acquired from a moving image in time series, and are sequentially subjected to image processing by a neural network, and parameters of the neural network are updated. In this modification, an example of a learning method different from the above learning method will be described. Specifically, in the learning method according to this modification, first, sequential image processing and error calculation processing are completed for each of a series of time-series images obtained from a video. Then, the series of calculated errors are summed up, and the resulting calculated error may be used to train the neural network, i.e., to update the parameters of the neural network. By applying such a configuration, for example, it becomes possible to have a neural network learn information acquired over a longer period of time, which is expected to have the effect of further improving the image quality of the image output as a result of image processing.

[0096] <Variation 5> A fifth modified example of each embodiment of the present disclosure will be described below. In the first and second embodiments described above, an example of a case where high-quality image processing such as noise removal processing for removing noise manifested in an input image is applied as image processing applied to an input image by a neural network is described. On the other hand, the type of image processing applied to an input image by a neural network is not necessarily limited to noise removal processing, and any image processing that applies some kind of processing to a target image can be applied. As a specific example, high-quality image processing may be applied to degradation such as aberration, compression, low resolution, and loss, and contrast reduction due to weather influences such as fog, haze, snow, and rain at the time of image capture.

[0097] Here, an example of a case where a blur removal process for removing blurring that is manifested in a target image is applied as the image processing that the neural network applies to the input image will be described. In this case, the image data of the time-series image that is input to the neural network is image data of an image in which the subject or the like is blurred, that is, blurring is manifested. In addition, in this case, the neural network applies image processing for removing (or reducing) blurring that is manifested in the input image, and outputs image data of the image (high-quality image) that has been subjected to the image processing. Note that other configurations and processes can be realized by applying the same configurations and processes as those of the first and second embodiments described above. Moreover, the degraded image generating unit 122 of the learning device may be configured to generate a blurred image as a degraded image. Specifically, the degraded image generating unit 122 may be configured to impart a blur modeled after a blur caused by the movement of a subject or a blur caused outside the depth of field to a target image. Then, the image to which the blur is imparted by the degraded image generating unit 122 may be applied as a degraded image in the first and second embodiments described above, thereby learning the neural network. By learning the neural network in the above manner, it is possible to obtain a high-quality image in which the appearance of fluctuations is suppressed while achieving blur removal as a high-quality image processing.

[0098] As described above, according to the technology of each embodiment of the present disclosure, even in a situation where various image processing is performed, not limited to high-quality image processing such as noise removal processing, etc., it is possible for a neural network to learn the correspondence between chronologically consecutive images. Furthermore, due to such characteristics, according to the technology of each embodiment of the present disclosure, it is possible to suppress the manifestation of fluctuations even in a situation where various image processing is performed on chronologically consecutive images by a neural network.

[0099] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0100] The disclosure of this embodiment also includes the following configuration and program. (Configuration 1) An information processing device comprising: a first image processing means for generating first image data after the predetermined image processing by inputting each of a series of image data having continuity along a time series into a neural network which performs a predetermined image processing on the input image data; and a second image processing means for generating second image data after the predetermined image processing based on the neural network, for the first image data outputted sequentially from the first image processing means along a time series, wherein the second image processing means generates the second image data corresponding to the first time by inputting the first image data outputted from the first image processing means at a first time or a second time prior to the first time, and the second image data generated by the second image processing means at the first image data outputted from the first image processing means at the second time into the neural network at the first time. (Configuration 2) The information processing device according to configuration 1, characterized in that the neural network learns based on an error between a first difference between true value image data corresponding to a first time and a second time prior to the first time, among a series of true value image data having continuity along a time series, and a second difference between the second image data generated by applying the specified image processing to the true value image data corresponding to the first time and the second time, which are processed to be restored by the specified image processing. (Configuration 3) The information processing device according to configuration 2, wherein the second image data corresponding to the output of the second image processing means at the second time is used in calculating the error. (Configuration 4) The information processing device according to any one of configurations 1 to 3, wherein the first image processing means generates, as the first image data, the image or one or more image data indicating the features of the image that has been subjected to the specified image processing, by treating the image data as image data to be input to the neural network. (Configuration 5) The information processing device of claim 4, wherein the second image processing means generates the second image data corresponding to the first time by using, as input to the neural network at the first time, one or more image data indicating an image or features generated from the image, which are output as the first image data from the first image processing means at a first time, and the second image data generated by the second image processing means for the first image data output from the first image processing means at the second time. (Configuration 6) The information processing device described in any one of configurations 1 to 3, characterized in that the first image processing means generates, as the first image data, a series of image data indicating a plurality of images that are successive in time series or a plurality of feature amounts generated from each of the plurality of images, by using the plurality of images that have been subjected to the specified image processing or the plurality of image data indicating the plurality of feature amounts as image data to be input to the neural network. (Configuration 7) The information processing device described in Configuration 6, wherein the second image processing means generates the second image data corresponding to the first time by using, as input to the neural network at the first time, a series of image data indicating a plurality of chronologically consecutive images or a plurality of feature amounts generated from the plurality of images, which are output as the first image data from the first image processing means at a first time, and the second image data generated by the second image processing means based on the first image data output from the first image processing means at the second time. (Configuration 8) The information processing device according to any one of configurations 1 to 7, characterized in that the second image processing means applies the second image data generated by the second image processing means to the first image data output from the first image processing means at the second time as the second image data to be input to the neural network at the first time. (Configuration 9) An information processing device according to any one of configurations 1 to 8, characterized in that the second image processing means applies the first image data output from the first image processing means at the first time as the first image data to be input to the neural network at the first time. (Configuration 10) An information processing device according to any one of configurations 1 to 8, characterized in that the second image processing means applies the first image data output from the first image processing means at the second time as the first image data to be input to the neural network at the first time. (Configuration 11) The information processing device according to any one of configurations 1 to 10, characterized in that the second image processing means generates the second image data corresponding to the first time by combining the second image data generated by the second image processing means at the second time, which is input to the neural network at the first time, with the output of the neural network at the second time and performing convolution. (Configuration 12) The information processing device according to any one of configurations 1 to 11, wherein the predetermined image processing is processing for removing noise that appears in the target image. (Configuration 13) An information processing device as described in any one of configurations 1 to 12, characterized in that it has an estimation means for estimating noise that is manifested in an image represented by each of the series of image data, and the first image processing means generates the first image data by applying the noise estimation result by the estimation means as an input to the neural network. (Configuration 14) A processed image data generating means for generating a series of processed image data by performing a first image processing on each of a series of true value image data having continuity along a time series; a first image processing means for generating first image data after the second image processing by inputting each of the series of processed image data into a neural network which performs a second image processing on input data in sequence; a second image processing means for generating second image data after the second image processing based on the neural network for the first image data output from the first image processing means in sequence along a time series; a first difference between the true value image data corresponding to a first time and a second time earlier than the first time, and a second difference between the true value image data corresponding to the first time and the second time and a difference calculation means for calculating a second difference between the second image data generated by subjecting the first difference to the second difference, an error calculation means for calculating an error between the first difference and the second difference, and a learning means for learning the neural network with respect to each of the first image processing means and the second image processing means based on the error, wherein the second image processing means uses the first image data output from the first image processing means at the first time or the second time, and the second image data generated by the second image processing means with respect to the first image data output from the first image processing means at the second time, as inputs to the neural network at the first time, to generate the second image data corresponding to the first time. (Configuration 15) The learning device according to configuration 14, wherein the error calculation means calculates, as the error, an L1 error between the first difference and the second difference. (Configuration 16) The learning device according to configuration 14 or 15, wherein the learning means trains the neural network based on an error backpropagation method utilizing the error. (Configuration 17) The learning device described in any one of configurations 14 to 16, characterized in that the error calculation unit performs weighting so that a larger weight is assigned to the error in a portion with less change between the true value image data corresponding to the first time and the second time. (Configuration 18) The learning device described in any one of configurations 14 to 16, characterized in that the error calculation unit performs weighting so that a larger weight is assigned to the error in a portion with less change between the true value image data corresponding to the first time and the second time. (Program 1) A program for causing a computer to function as an information processing device having a first image processing means for generating first image data after the predetermined image processing by inputting each of a series of image data having continuity along a time series into a neural network which performs a predetermined image processing on the input image data, and a second image processing means for generating second image data after the predetermined image processing based on the neural network, for the first image data outputted sequentially from the first image processing means along a time series, wherein the second image processing means generates the second image data corresponding to the first time by inputting the first image data outputted from the first image processing means at a first time or a second time prior to the first time, and the second image data generated by the second image processing means at the first image data outputted from the first image processing means at the second time into the neural network at the first time. (Program 2) A computer includes a processed image data generating means for generating a series of processed image data by performing a first image processing on each of a series of true value image data having continuity along a time series, a first image processing means for generating first image data after the second image processing by inputting each of the series of processed image data into a neural network that performs a second image processing on input data in sequence, a second image processing means for generating second image data after the second image processing based on the neural network for the first image data output from the first image processing means in sequence along a time series, and a first difference between the true value image data corresponding to a first time and a second time earlier than the first time, and a second difference between the true value image data corresponding to the first time and the second time. A program for causing the device to function as a learning device, comprising: a difference calculation means for calculating a second difference between the second image data generated by applying a second image processing; an error calculation means for calculating an error between the first difference and the second difference; and a learning means for learning the neural network for each of the first image processing means and the second image processing means based on the error, wherein the second image processing means uses the first image data output from the first image processing means at the first time or the second time, and the second image data generated by the second image processing means for the first image data output from the first image processing means at the second time, as input to the neural network at the first time, to generate the second image data corresponding to the first time. [Explanation of symbols]

[0101] 100 Information processing device 111 First Network Department 112 Second Network Department 120 Learning Device

Claims

1. a first image processing means for inputting each of a series of image data having continuity along a time series into a neural network that performs a predetermined image processing on the input image data, thereby generating a first image data after the predetermined image processing; a second image processing means for generating second image data after the predetermined image processing based on the neural network, for the first image data sequentially output from the first image processing means in a time series; having The second image processing means the first image data output from the first image processing means at a first time or a second time prior to the first time; the second image data generated by the second image processing means based on the first image data output from the first image processing means at the second time; to the neural network at the first time to generate the second image data corresponding to the first time. An information processing device comprising:

2. The neural network comprises: a first difference between true-value image data corresponding to a first time and a second time prior to the first time, among a series of true-value image data having continuity along a time series; a second difference between the second image data generated by performing the predetermined image processing on the processed image data generated by performing the processing to restore the true value image data corresponding to the first time and the second time, respectively, by the predetermined image processing; and It is trained based on the error between 2. The information processing device according to claim 1.

3. 3. The information processing apparatus according to claim 2, wherein the second image data corresponding to an output of the second image processing means at the second time is used for calculating the error.

4. The information processing device described in claim 1, characterized in that the first image processing means generates, as the first image data, the image that has been subjected to the specified image processing or one or more image data indicating the feature amounts by using one or more image data indicating an image or a feature amount generated from the image as image data to be input to the neural network.

5. The second image processing means one or more image data indicating an image or a feature amount generated from the image, the image data being output as the first image data from the first image processing means at a first time; the second image data generated by the second image processing means based on the first image data output from the first image processing means at the second time; to the neural network at the first time to generate the second image data corresponding to the first time.

5. The information processing device according to claim 4.

6. The information processing device described in claim 1, characterized in that the first image processing means generates, as the first image data, a series of image data indicating a plurality of images that are successive in time series or a plurality of feature amounts generated from each of the plurality of images, by using as image data to be input to the neural network, the plurality of images that have been subjected to the specified image processing, or the series of image data indicating the plurality of feature amounts.

7. The second image processing means a series of image data indicating a plurality of images successive in time series or a plurality of feature amounts generated from the plurality of images, the series of image data being output as the first image data from the first image processing means at a first time; the second image data generated by the second image processing means based on the first image data output from the first image processing means at the second time; to the neural network at the first time to generate the second image data corresponding to the first time.

7. The information processing device according to claim 6.

8. The second image processing means The second image data to be input to the neural network at the first time is The second image data generated by the second image processing means is applied to the first image data output from the first image processing means at the second time.

2. The information processing device according to claim 1.

9. The second image processing means The first image data to be input to the neural network at the first time is Applying the first image data output from the first image processing means at the first time 2. The information processing device according to claim 1.

10. The second image processing means The first image data to be input to the neural network at the first time is Applying the first image data output from the first image processing means at the second time 2. The information processing device according to claim 1.

11. The second image processing means the second image data generated by the second image processing means at the second time, which is to be input to the neural network at the first time; By combining the output of the neural network at the second time and performing a convolution, generating the second image data corresponding to the first time; 2. The information processing device according to claim 1.

12. 2. The information processing apparatus according to claim 1, wherein the predetermined image processing is processing for removing noise that appears in the target image.

13. an estimation means for estimating noise manifested in an image represented by each of the series of image data; The first image processing means generates the first image data by applying the noise estimation result by the estimation means as an input to the neural network.

2. The information processing device according to claim 1.

14. a processed image data generating means for generating a series of processed image data by performing a first image processing on each of a series of true value image data having continuity along a time series; a first image processing means for inputting each of the series of processed image data to a neural network that performs a second image processing on the input data in sequence, thereby generating a first image data after the second image processing; a second image processing means for generating second image data after the second image processing based on the neural network, for the first image data output from the first image processing means in sequence along a time series; a difference calculation means for calculating a first difference between the true image data corresponding to a first time and a second time earlier than the first time, and a second difference between the second image data generated by applying the second image processing to the processed image data generated from the true image data corresponding to the first time and the second time; an error calculation means for calculating an error between the first difference and the second difference; a learning means for learning the neural network based on the error and for each of the first image processing means and the second image processing means; having The second image processing means the first image data output from the first image processing means at the first time or the second time; the second image data generated by the second image processing means based on the first image data output from the first image processing means at the second time; to the neural network at the first time to generate the second image data corresponding to the first time. A learning device comprising:

15. 15. The learning device according to claim 14, wherein the error calculation means calculates, as the error, an L1 error between the first difference and the second difference.

16. 15. The learning device according to claim 14, wherein the learning means performs learning of the neural network based on an error backpropagation method utilizing the error.

17. The learning device according to claim 14, characterized in that the error calculation unit performs weighting so that a greater weight is assigned to an error in a portion where there is less change between the true value image data corresponding to the first time and the second time.

18. The learning device according to claim 14, characterized in that the error calculation unit performs weighting so that a greater weight is assigned to an error in a portion where there is less change between the true value image data corresponding to the first time and the second time.

19. Computer, a first image processing means for inputting each of a series of image data having continuity along a time series into a neural network that performs a predetermined image processing on the input image data, thereby generating a first image data after the predetermined image processing; a second image processing means for generating second image data after the predetermined image processing based on the neural network, for the first image data sequentially output from the first image processing means in a time series; having The second image processing means the first image data output from the first image processing means at a first time or a second time prior to the first time; the second image data generated by the second image processing means based on the first image data output from the first image processing means at the second time; to the neural network at the first time to generate the second image data corresponding to the first time. A program for causing the information processing device to function as the information processing device.

20. Computer, a processed image data generating means for generating a series of processed image data by performing a first image processing on each of a series of true value image data having continuity along a time series; a first image processing means for inputting each of the series of processed image data to a neural network that performs a second image processing on the input data in sequence, thereby generating a first image data after the second image processing; a second image processing means for generating second image data after the second image processing based on the neural network, for the first image data output from the first image processing means in sequence along a time series; a difference calculation means for calculating a first difference between the true image data corresponding to a first time and a second time earlier than the first time, and a second difference between the second image data generated by applying the second image processing to the processed image data generated from the true image data corresponding to the first time and the second time; an error calculation means for calculating an error between the first difference and the second difference; a learning means for learning the neural network based on the error and for each of the first image processing means and the second image processing means; having The second image processing means the first image data output from the first image processing means at the first time or the second time; the second image data generated by the second image processing means based on the first image data output from the first image processing means at the second time; to the neural network at the first time to generate the second image data corresponding to the first time. A program for causing a learning device to function as a learning device.

Images