Information processing apparatus, information processing method, and program

Abstract

To enable restoration for a degraded image even when fewer than N images are available.SOLUTION: An information processing apparatus includes: input means which acquires an input image; processing means which processes the input image to generate a processed image; and restoration means which generates a restored image obtained by removing degradation of the input image based on the input image and the processed image. The processing means is configured to process the input image or the processed image to generate processed images until the number of the input images and the processed images reaches the number of images necessary for restoration.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] A method for restoring the degradation of an image captured by a camera and an image of one frame of a moving image is known. When restoring the degradation of a moving image, temporal consistency is an important factor in perceived quality. Therefore, it is necessary to use information of images adjacent in time series.

[0003] Generally, when an information processing apparatus restores the degradation of a moving image, it acquires a plurality of temporally consecutive images and outputs a restored image obtained by restoring the degradation of one image. Non-Patent Document 1 discloses a method in which, for N (N is a natural number) temporally consecutive images, noise reduction in the spatial direction is performed for each of them, and after aligning the results, noise reduction processing in the temporal direction is performed, and the noise reduction result of one image in the center of the N images is output. Non-Patent Document 2 discloses a method of omitting the alignment performed after the noise reduction in the spatial direction of Non-Patent Document 1 by incorporating a motion compensation mechanism for alignment into a deep neural network (DNN).

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, any restoration method requires obtaining N images, and when there are less than N images immediately after starting video shooting or immediately after starting video playback, it is not possible to perform the restoration process of image degradation.

[0006] The present invention has been made in view of such problems, and in a degradation restoration process that outputs a restored image obtained by restoring degradation with N images (N is an integer of 2 or more) as input, even when less than N images are obtained, the purpose is to output a restored image.

Means for Solving the Problems

[0007] To solve this problem, for example, the information processing apparatus of the present invention has the following configuration. That is, input means for obtaining an input image; processing means for processing the input image to generate a processed image; restoration means for generating a restored image obtained by restoring the degradation of the input image based on the input image and the processed image; and the processing means generates the processed image by processing either the input image or the processed image until the number of the input image and the processed image reaches the number required for the restoration. This is the gist of the invention.

Effects of the Invention

[0008] According to the present invention, in a degradation restoration process that outputs a restored image with N (N is an integer of 2 or more) images as input, even when fewer than N images are acquired, a restored image can be output.

Brief Description of Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0011] (First Embodiment) FIG. 1 is a diagram showing an example of the hardware configuration of the information processing system according to the present embodiment. The information processing system includes an information processing apparatus 100 and an imaging apparatus 112 as an imaging unit. In the information processing system, the information processing apparatus 100 restores the degradation of an image generated by the imaging apparatus 112 capturing a subject. In the present embodiment, noise is taken as an example of an image degradation factor, and an example in which the information processing apparatus 100 performs noise reduction processing as degradation restoration processing will be described. In FIG. 1, a case where the information processing system has three imaging apparatuses 112 is illustrated, but the number of imaging apparatuses 112 is not limited to three and may be changed as appropriate. The information processing apparatus 100 and the imaging apparatus 112 are connected to be able to transmit and receive information to and from each other via a network 111 such as the Internet, a WAN (Wide Area Network), and a LAN (Local Area Network). Note that the term “image” may be used as a concept including data such as still images, moving images, videos, images of one frame of a moving image, and image data.

[0012] The information processing apparatus 100 is, for example, a computer. The information processing apparatus 100 includes a CPU 101, a ROM 102, a RAM 103, an external storage device 104, an input I / F 105, an output I / F 106, a communication I / F 107, and a system bus 108. The system bus 108 is a transmission path that communicably connects each unit of the CPU 101, the ROM 102, the RAM 103, the external storage device 104, the input I / F 105, the output I / F 106, and the communication I / F 107.

[0013] The CPU 101 is the abbreviation of Central Processing Unit. The CPU 101 controls the entire information processing device 100. The CPU 101 reads out the programs stored in the ROM 102 and the external storage device 104, etc., expands them in the RAM 103, and executes them to realize various functions and processes. Note that the information processing device 100 may have processors such as a GPU (Graphics Processing Unit), an MPU (Micro Processing Unit), and a QPU (Quantum Processing Unit) instead of or in addition to the CPU 101.

[0014] The ROM 102 is the abbreviation of Read Only Memory. The ROM 102 stores programs and parameters that do not require modification.

[0015] The RAM 103 is the abbreviation of Random Access Memory. The RAM 103 temporarily stores programs and data supplied from external devices, etc. The RAM 103 functions as a work area when the CPU 101 executes a program.

[0016] The external storage device 104 is a non-volatile storage device such as a hard disk and a memory card fixedly installed in the information processing device 100. Note that the external storage device 104 may include a flexible disk (FD), an optical disk such as a Compact Disc (CD), a magnetic card, an optical card, an IC card, and a memory card that are detachable from the information processing device 100.

[0017] The input I / F 105 is an interface that realizes input and output with the input device 109. The input device 109 may be a pointing device and a keyboard through which a user can input data, etc. The input I / F 105 receives inputs such as operations, instructions, and data from the user via the input device 109 and transmits them to the CPU 101, etc.

[0018] The output I / F 106 is an interface that realizes the output of data to the monitor 110. The output I / F 106 transmits and displays on the monitor 110 the data held by the information processing apparatus 100, images supplied from the imaging apparatus 112, etc., and images processed by the CPU 101. The monitor 110 may be a device capable of displaying images such as a liquid crystal display device or an organic EL (Electro Luminescence) display.

[0019] The communication I / F 107 is an interface that connects the information processing apparatus 100 and a network 111 such as the Internet so that data can be transmitted and received. The communication I / F 107 realizes communication between the information processing apparatus 100 and an external device.

[0020] The imaging apparatus 112 may be a digital camera such as a surveillance camera. The imaging apparatus 112 captures a subject and generates an image. The imaging apparatus 112 is connected to the information processing apparatus 100 via the network 111 so that data such as the generated image can be transmitted and received.

[0021] FIG. 2 is a functional block diagram showing the software configuration of the information processing apparatus 100 according to the present embodiment. The information processing apparatus 100 includes an input unit 201, a processing unit 202, a storage unit 203, and a restoration unit 204. The information processing apparatus 100 may realize the functions of the input unit 201, the processing unit 202, the storage unit 203, and the restoration unit 204 by reading and executing a program stored in the external storage device 104. Note that the information processing apparatus 100 may realize part or all of the functions of the input unit 201, the processing unit 202, the storage unit 203, and the restoration unit 204 by a circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). Each function of the input unit 201, the processing unit 202, the storage unit 203, and the restoration unit 204 will be described later.

[0022] <Learning by CNN> Here, the learning of the degradation restoration model used in the degradation restoration process performed by the restoration unit 204 of the information processing apparatus 100 according to the present embodiment will be described. FIG. 3 is a diagram for explaining the structure of a CNN.

[0023] The degradation restoration model is generated using a Convolutional Neural Network (CNN), which is generally used in information processing technologies applying deep learning. CNN is a technology that repeatedly performs non-linear operations after convolving a filter generated by learning (training or learning) with image data. Convolution and non-linear operations are the processes performed by the rightward arrow 301 in FIG. 3. Further, in the present embodiment, a method is used in which after non-linear operation, the input image is reduced (pooled) to concentrate and leave important feature amounts, and then restored to the original dimension again (deconvolution). Pooling is the process performed by the downward arrow 302 in FIG. 3. Also, deconvolution is the process performed by the upward arrow 303 in FIG. 3. The image data obtained by convolving a filter with image data and then performing non-linear operations is called a feature map. Also, learning is performed using training data (training images or data sets) consisting of a pair of input image data and output image data. Briefly, learning is to generate the values of a filter that can convert input image data to corresponding output image data with high accuracy from the training data.

[0024] When the image data has RGB color channels and the feature map is composed of multiple pieces of image data, the filter used for convolution also has multiple channels accordingly. That is, the convolution filter is represented by a 4D array with the number of channels added in addition to the vertical and horizontal sizes and the number. After convolving the filter with the image data (or feature map), the non-linear operation process is represented in units of "layers", for example, the n-th layer feature map and the n-th layer filter, etc. are expressed. Also, for example, a CNN that repeats convolution, non-linear operation, and pooling three times has a three-layer network structure. Such non-linear operation processing can be formulated as shown in the following formula (1).

[0025]

Number

[0026] In formula (1), W n is the filter of the n-th layer, b n is the bias of the n-th layer, f is the non-linear operator, X n is the feature map of the n-th layer, and * is the convolution operator. The (l) on the upper right indicates that it is the l-th filter or feature map. The filter and bias are generated by the learning described later and are collectively also called "network parameters". The non-linear operation may be, for example, a sigmoid function and ReLU (Rectified Linear Unit). In the case of ReLU, it is given by the following formula (2).

[0027]

Number

[0028] As shown in formula (2), among the elements of the input vector X, the negative ones become zero, and the positive ones maintain the value of X.

[0029] The pooling in this embodiment adopts Max Pooling, which is the most commonly used. For example, in the pooling of this embodiment, among 2×2 pixels, the maximum value is selected as the representative value. In this case, the image is reduced (downsampled) to an image that is reduced to half in both the vertical and horizontal directions. Therefore, the downsampling rate is 1 / 2.

[0030] The degradation restoration process of this embodiment repeats the above-mentioned convolution of the filter, non-linear operation, and pooling three times, and then performs upsampling by transposed convolution to finally restore the dimension of the original image.

[0031] Next, the learning of the CNN using the above-described CNN configuration will be described. In this embodiment, a model that takes a plurality of images (N images, where N is an integer of 2 or more) as input images and outputs an image in which the degradation of the image is restored is learned. At this time, the N input images are related images such as time-series images acquired in the order of time series and are different images. The restoration unit 204 improves the accuracy of degradation restoration by using the differences between the plurality of images. The learning of the CNN is generally performed by minimizing the objective function represented by the following equation (3) with respect to the learning data consisting of a set of input learning image data as student images and corresponding output learning image data as teacher images.

[0032]

Equation

[0033] In Equation (3), L is a loss function that measures the error between the correct answer and its estimation. Y i is the i-th output learning image data, and X i is the i-th input learning image data. F is a function that collectively represents the operations (Equation (1)) performed in each layer of the CNN. θ is a network parameter (filter and bias). ||Z|| 2is the L2 norm, which is simply the square root of the sum of the squares of the elements of vector Z. n is the total number of learning data used for learning. Generally, since the number of learning data is large, Stochastic Gradient Descent (SGD) randomly selects a part of the learning image data and uses it for learning. This reduces the computational load in learning using a large amount of learning data. As methods for minimizing (optimizing) the objective function, various methods such as the momentum method, AdaGrad method, AdaDelta method, and Adam method are known. The Adam method is given by the following equation (4).

[0034]

Equation

[0035] In Equation (4), θ i t is the i-th network parameter at the t-th iteration, g is the gradient of the loss function L with respect to θ i t . m and v are momentum vectors, α is the base learning rate, β 1 , β 2 are hyperparameters, and ε is a small constant. Note that since there is no selection guideline for the optimization method in learning, basically anything can be used, but since there are differences in the convergence of each method, it is known that there are differences in learning time.

[0036] In each of the embodiments described below, information processing (image processing) for reducing video degradation on a per-image basis using the aforementioned CNN is performed. Examples of image degradation factors include degradation such as noise, blur, aberration, compression, low resolution, and missing data, as well as contrast reduction due to weather conditions such as fog, haze, snow, and rain during shooting. Image processing for reducing and restoring degradation in an image includes noise reduction, blur removal, aberration correction, missing data completion, correction processing for degradation due to compression, super-resolution processing for low-resolution images, and processing for correcting contrast reduction caused by weather conditions during shooting, etc. The restoration processing of image degradation in each of the embodiments described below is a process of generating or restoring an image without (or with very little) degradation from a degraded image, and is also referred to as degradation restoration processing in the following description. That is, the degradation restoration processing includes not only restoring an image that was originally an image without (little) degradation but was degraded by subsequent amplification and compression-expansion, other image processing, etc., but also reducing and restoring the degradation originally included in the original image itself.

[0037] FIG. 4 is a block diagram showing an example of the data flow of the degradation restoration processing performed by the information processing apparatus 100 according to the present embodiment. In the example of FIG. 4, the information processing apparatus 100 according to the present embodiment is the part shown within the frame. The information processing apparatus 100 acquires an input image 401 generated by an external imaging apparatus 112 capturing a subject via a network 111 and a communication I / F 107. Then, the information processing apparatus 100 outputs a restored image obtained by restoring the degradation of the input image 401 to a monitor 110 via an output I / F 106.

[0038] The input unit 201 acquires the input image 401. The input unit 201 acquires the input image 401, for example, in time series order, one frame of an image at a time, in a streaming manner, for a video captured by the imaging apparatus 112. The input image 401 may be an RGB three-channel image. The input unit 201 stores the input image 401, which is a frame image, in the storage unit 203.

[0039] When the processing unit 202 determines that the number of input images 401 stored in the storage unit 203 is less than N (where N is an integer of 2 or more), the processing unit 202 processes the acquired input image 401 to generate a processed image 402 and stores it in the storage unit 203. Here, the processing unit 202 performs processing to shift all the pixels of the input image 401 8 pixels in the upper right direction within the screen, that is, to translate it. The processing unit 202 cuts off the pixels that protrude from the image due to the shift of the image. On the other hand, the processing unit 202 sets the pixel values of the pixels in the portion where the pixel values become undefined due to the shift of the image to 0 (that is, black). If the processing unit 202 determines that the number of frame images stored is still less than N even after storing the processed image 402 in the storage unit 203, the processing unit 202 further performs processing to shift the pixels of the processed image 402 in the same direction, that is, 8 pixels in the upper right direction, to generate and store the processed image 402. Even if the processing unit 202 performs processing to shift the pixels of the acquired input image 401 16 pixels in the upper right direction, the same image is obtained. Therefore, the processing unit 202 may perform a plurality of processing operations on one input image by using, as the shift amount, the value obtained by integrating the shift amount (here, 8 pixels) used for one-time processing, which is the unit processing amount, to generate a plurality of processed images. Also, the processing unit 202 sets the time of the processed image 402 to the time one before and two before from the one with the smaller shift amount. That is, the processing unit 202 treats the processed image 402 as a past image of the input image 401, and by treating the image with the larger shift amount as an image with an earlier past time, it maintains the temporal continuity with the next input image.

[0040] The degradation restoration model used in this embodiment is a model trained to perform degradation restoration with N different images as inputs. Therefore, the degradation restoration model of this embodiment cannot restore image degradation immediately after the start of streaming when it has not received N images. Also, even if the degradation restoration model makes up the shortage of images to N by simply copying them, since there is no difference between the copied images and the original images, it cannot accurately restore image degradation. The processing unit 202 in this embodiment can generate N mutually different images by shifting the pixels of the image as described above, and can make it appear as if different images are input. When generating a plurality of processed images 402 obtained by shifting the pixels of the input image 401 of the time-series image, the processing unit 202 may generate a plurality of processed images 402 obtained by shifting the pixels in the same direction so that the subject shown in the input image 401 appears to be moving pseudo-statically.

[0041] The processing unit 202 sets the shift amount for shifting pixels, which is the amount of processing when processing an image, to 8 pixels. This may be determined based on parameters for restoring degradation. The parameters for restoring degradation mentioned here include, for example, the reduction rate of the image when restoring degradation, the size of the pixel range during pooling, the number of pooling times, the downsampling rate during pooling, and the like. Here, since the restoration unit 204 of this embodiment performs 2×2 pixel pooling three times, the image is in a state of being reduced to one-eighth. Here, when the pixel shift amount is smaller than 8 pixels, the shifted pixels will be grouped into the same pixel when reduced, and the difference between the images before and after processing will disappear despite the shift. Therefore, the processing unit 202 of this embodiment sets the shift amount (amount of processing) of the pixels to be shifted to 8 pixels. Here, the number of pixels to be shifted is set to 8, which is the denominator of the reduction rate. However, if it is 8 or more, the images before and after the shift will be different pixels even after pooling, so there is an effect of increasing the number of images. For example, the processing unit 202 may be set as the shift amount of the pixels to be shifted by 10 pixels, which is 25% more than the denominator of the reduction rate. The processing unit 202 may determine the optimal value through experiments or the like based on the reduction rate.

[0042] The restoration unit 204 acquires the N input images and processed images stored in the storage unit 203, restores the degradation using the degradation restoration model 403, and generates a restored image 404. Then, the restoration unit 204 deletes the image with the oldest time from the storage unit 203, and sets the input images and processed images to be stored as (N - 1) frame images.

[0043] Finally, the restoration unit 204 outputs the restored image 404 to the monitor 110. The monitor 110 displays the acquired restored image 404.

[0044] Also, when the imaging parameters of the imaging device 112 are changed, or when the parameters of the process executed before performing the degradation restoration process are changed, etc., the image quality may change significantly before and after the parameter change. If the restoration unit 204 performs the degradation restoration process on the images in a state where such images of different qualities are mixed, it may not be able to generate the intended restored image. Therefore, when the input unit 201 determines that the parameters acquired from the imaging device 112 or the like have been changed, it deletes all the images stored in the storage unit 203. In this case, when the input unit 201 acquires the next frame image as the input image and stores it in the storage unit 203, the processing unit 202 generates a processed image by shifting the pixels of the input input image and stores it in the storage unit 203. The restoration unit 204 performs degradation restoration based on the newly stored N input images and processed images.

[0045] <Degradation Restoration Process> FIG. 5 is a flowchart showing an example of the degradation restoration process performed by the information processing apparatus 100 according to the present embodiment.

[0046] First, in step S501, the input unit 201 sets the time t = 0.

[0047] Next, in step S502, the input unit 201 acquires, as an input image at the time T = t, an image of one frame of the moving image generated by the imaging device 112 photographing a subject, and stores it in the storage unit 203.

[0048] In step S503, the processing unit 202 checks whether the images at times from T = t-(N - 1) to T = t - 1 are stored in the storage unit 203. That is, the processing unit 202 checks whether N images are stored in the storage unit 203. If the processing unit 202 determines that N images are stored, it proceeds to step S505; if it determines that they are not stored, it proceeds to step S504. Immediately after the start of the degradation restoration process, only one image is stored in the storage unit 203, so the processing unit 202 will proceed to step S504.

[0049] In step S504, the processing unit 202 shifts the pixels of the input image and the processed image stored in the storage unit 203 to generate a processed image, and sets the number of stored images to N. For example, the processing unit 202 performs processing to shift the pixels of the image at T = t within the screen, thereby generating processed images from T = t-(N - 1) to T = t - 1. That is, the processing unit 202 performs processing to shift the pixels of the input image or the processed image at T = t eight pixels in the upper right direction to generate a processed image at T = t - 1. Further, the processing unit 202 performs processing to shift the pixels of the processed image at T = t - 1 eight pixels in the upper right direction to generate a processed image at T = t - 2. By repeating the same processing, the processing unit 202 generates processed images up to T = t-(N - 1). Thereby, the processing unit 202 can set the number of images stored in the storage unit 203 to N. The processing unit 202 stores the generated processed image in the storage unit 203 and proceeds to step S505.

[0050] In step S505, the restoration unit 204 performs a degradation image restoration process using the images from T = t-(N - 1) to T = t, that is, N images, stored in the storage unit 203.

[0051] In step S506, the restoration unit 204 outputs the image restored by the degradation image restoration process as the restored image at T = t.

[0052] In step S507, the restoration unit 204 deletes the oldest image, that is, the image with T = t - (N - 1) from the storage unit 203.

[0053] In step S508, the restoration unit 204 increments t (t = t + 1).

[0054] In step S509, the input unit 201 checks whether it is necessary to reset the stored image. If it is not necessary to reset, it proceeds to step S511. If it is necessary to reset, it proceeds to step S510. For example, the input unit 201 may compare the imaging parameters indicated by the image data stored in the storage unit 203 with the imaging parameters acquired from the imaging device 112 together with the image, and determine whether reset is required based on whether the imaging parameters have changed. Specifically, the input unit 201 may determine that it is necessary to reset the image when the two imaging parameters are different, that is, when the imaging parameters have changed.

[0055] In step S510, the input unit 201 deletes the images with T = t - (N - 1) to T = t - 1, that is, all the images stored in the storage unit 203 from the storage unit 203, and proceeds to step S511.

[0056] In step S511, the restoration unit 204 checks whether all the degradation restoration processes have ended, that is, whether the acquisition of images has ended. If the restoration unit 204 determines that all the degradation restoration processes have ended, it ends the degradation restoration process. If the restoration unit 204 determines that they have not ended, that is, if it continues, it returns to step S502.

[0057] Since the time t is incremented in step S508, in step S502, the input unit 201 acquires an input image at the time when the value of T has advanced by one (for example, t + 1) and stores it in the storage unit 203. In step S510, if not all the images in the storage unit 203 have been deleted by reset, when the input unit 201 acquires an input image, the number of images stored in the storage unit 203 becomes N. Therefore, without the processing unit 202 newly processing the input image, the restoration unit 204 performs the degradation restoration process in step S505 using the N images. That is, only at the first time or immediately after reset, the processing unit 202 shifts the pixels of the input image or the processed image to generate a processed image.

[0058] As described above, in the degradation restoration process in which the information processing apparatus 100 of the present embodiment outputs a restored image obtained by restoring degradation using N (N is an integer of 2 or more) images as input, when the number of input images, which are frame images of a time-series image, is less than N, a processed image obtained by processing the input image is used. Thereby, even when the number of acquired images is less than N immediately after the start of the restoration process or the like, the information processing apparatus 100 can execute the degradation restoration process based on N images, and thus can output a restored image.

[0059] The information processing apparatus 100 can restore degradation with N different images by generating a processed image different from the input image. Thereby, the information processing apparatus 100 can output a restored image with high accuracy.

[0060] Since the information processing apparatus 100 generates a processed image by, for example, parallel movement of pixels, the processed image can be easily generated, and an increase in the load due to the restoration process can be reduced.

[0061] Since the information processing apparatus 100 generates a processed image by moving the positions of all pixels, a processed image that is different from the image before processing in all pixels can be easily generated. Thereby, the information processing apparatus 100 can generate a restored image with higher accuracy.

[0062] Since the information processing apparatus 100 determines the shift amount, which is the amount of pixel processing for an image, based on parameters for restoring degradation, it is possible to suppress the pixels shifted by the restoration process from being grouped together and the pixel shift from becoming ineffective.

[0063] Since the information processing apparatus 100 generates a processed image based on the shift amount obtained by integrating the shift amounts, a plurality of processed images can be generated from a single input image.

[0064] By setting the generated processed image as an image older than the input image, the information processing apparatus 100 can maintain temporal continuity with the next input image.

[0065] By setting the processed image with a larger shift amount among the plurality of processed images as an older image, the information processing apparatus 100 can maintain temporal continuity among the plurality of processed images.

[0066] In the above-described embodiment, the processing unit 202 generates a processed image by shifting all the pixels of an image such as an input image by 8 pixels in the upper right direction. However, the present invention is not limited to this. Therefore, the processing unit 202 may set the pixel shift direction to any of the up, down, left, right, or inclined directions from the up, down, left, and right directions based on conditions such as the installation conditions of the imaging device 112. For example, the processing unit 202 may detect a subject such as a living body and an object shown in the image, and set the shift direction based on the position of the subject in the input image. Specifically, when the subject is located at the edge of the image and may move out of the image depending on the shift direction, the processing unit 202 may set the moving direction so that the subject moves in the direction opposite to the edge of the image, that is, in the direction of the center of the image.

[0067] The processing unit 202 may track a subject moving within an image captured in a plurality of images by a known method, and set the moving direction of pixels, that is, the shift direction, based on the moving direction of the subject. Specifically, when a horizontal line is captured and the subject moves left and right, the processing unit 202 may set the shift direction to the left - right direction; when a camera is installed directly above a passage and the subject moves up and down within the screen, the processing unit 202 may set the shift direction to the up - down direction. In this case, the processing unit 202 may set the shift direction so that the subject does not protrude outside the image. Thereby, the restoration unit 204 can accurately restore the degradation of the subject.

[0068] The processing unit 202 may process the image by moving the pixels of the image in a direction other than the linear shift direction. For example, when the subject moves while rotating within the screen, the processing unit 202 may set the rotation direction as the moving direction of the pixels, and process the image by rotating and moving all the pixels of the image. When the subject moves away or approaches, the processing unit 202 may process the image by enlarging and reducing the image.

[0069] The processing unit 202 may determine the processing amount, that is, the shift amount of the translational movement of the pixels, the rotation angle of the rotational movement of the pixels, the magnification of the image, and the reduction ratio of the image, which are the amounts for processing the image when processing the image, based on the parameters for restoring the degradation by the degradation restoration process. Also, at this time, if there are pixels whose coordinate positions do not change before and after processing, the accuracy of the degradation restoration process may be degraded only in that part. To prevent such a situation, the processing unit 202 may set the center when performing rotational movement, enlargement, and reduction outside the image.

[0070] In the above-described degradation restoration process, since the processing unit 202 pseudo-generates a time-series image and the restoration unit 204 performs the degradation restoration process, the sharpness of the restored image may be lost. Therefore, when the restoration unit 204 executes the degradation restoration process using the processed image, the restoration unit 204 may additionally perform a sharpening process of sharpening (emphasizing sharpness) the restored image after restoration. Thereby, the restoration unit 204 can restore the sharpness of the restored image whose sharpness has been reduced by the processed image. At that time, the restoration unit 204 may change the intensity of the sharpening process according to the number of processed images used. Thereby, the restoration unit 204 can restore the sharpness of the restored image whose sharpness reduction becomes larger according to the number of processed images.

[0071] (Second Embodiment) In the first embodiment, in the degradation restoration process of outputting a restored image with N (N is an integer of 2 or more) images as input, when acquiring a time-series image, when only less than N images such as immediately after the start are input, an image obtained by processing the input image is used. In this embodiment, in the degradation restoration process of outputting a degraded restored image with N (N is an integer of 2 or more) images as input, the degradation restoration process when 1 to N images are input will be described.

[0072] The hardware configuration of the information processing system in this embodiment is the same as that of FIG. 1 in the first embodiment. Also, the software configuration of the information processing apparatus 100 according to this embodiment is the same as that of FIG. 2 in the first embodiment.

[0073] The data flow of the degradation restoration process performed by the information processing apparatus 100 in this embodiment is basically the same as that of FIG. 4 in the first embodiment. However, in this embodiment, the information processing apparatus 100 may acquire the images stored in the external storage device 104 in the order of time series instead of acquiring the images captured by the imaging device 112.

[0074] <Degradation Restoration Process> FIG. 6 is a flowchart showing an example of the deterioration restoration process performed by the information processing apparatus 100 according to the present embodiment.

[0075] First, in step S601, the input unit 201 acquires one to N images input to the information processing apparatus 100 and stores them in the storage unit 203. Note that the input unit 201 may acquire images stored in the external storage device 104.

[0076] Next, in step S602, the processing unit 202 checks whether the number of images acquired by the input unit 201 and stored in the storage unit 203 is N or more. If the number of stored images is N or more, the processing unit 202 proceeds to step S604 and performs the deterioration restoration process as it is. On the other hand, if the number of stored images is less than N, the processing unit 202 proceeds to step S603.

[0077] In step S603, the processing unit 202 shifts the pixels of the stored image to generate a processed image. That is, the processing unit 202 generates the first processed image by performing a process of shifting the pixels of the image 8 pixels in the upper right direction and stores it in the storage unit 203. If the number of images in the storage unit 203 is still less than N even after generating one processed image, the processing unit 202 further generates the second processed image by performing a process of shifting the pixels of the once-processed image 8 pixels in the upper right direction. Similarly, the processing unit 202 generates (N - number of images) processed images until the number of stored images becomes N. When there are a plurality of input images before processing in the storage unit 203, the processing unit 202 may process any of the images to generate a processed image. Also, the processing unit 202 may generate a plurality of processed images from a plurality of input images. In any case, basically, the processing unit 202 processes and generates the N images stored in the storage unit 203 including the processed images so that they are different from each other and do not become the same image. The processing unit 202 stores the generated processed image in the storage unit 203 and proceeds to step S604.

[0078] In step S604, the restoration unit 204 performs restoration processing of the degraded image using N images, outputs the result to the monitor 110 etc. in step S605, and ends the processing.

[0079] As described above, in the degradation restoration process in which the information processing apparatus 100 of the second embodiment outputs a degraded restoration image with N images (N is an integer of 2 or more) as input, even when the number of accumulated images is less than N, the degradation restoration process is performed using the processed image obtained by processing the input image together with the input image. As a result, the information processing apparatus 100 can output a highly accurate restored image even when the number of images required for the degradation restoration process is not complete.

[0080] In the above, when generating the processed image, it is assumed that the N images including the processed image are not generated to be the same image. However, when the input images themselves acquired from the imaging device 112 etc. are the same, or when the processed image processed in the same way already exists as an input image or a processed image, the same image may exist in the N accumulated images. In this case, the processing unit 202 can compare the N images including the processed image that has been processed with each other, and when the same image exists, generate a processed image again so that the same image does not exist in the accumulated images. However, it is not desirable because the accuracy decreases when the same images are mixed, but even when the same images are mixed, the degradation restoration process can still be executed. Therefore, when there is a limitation in the processing speed, the processing unit 202 does not have to check for the same images.

[0081] (Other Embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and having one or more processors in the computer of the system or apparatus read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0082] The disclosure of this specification includes the following information processing apparatus, information processing method, and program. (Item 1) Input means for acquiring an input image, Processing means for processing the input image to generate a processed image, Restoring means for generating a restored image obtained by restoring the degradation of the input image based on the input image and the processed image, Comprising: The processing means generates the processed image by processing either the input image or the processed image until the number of the input image and the processed image reaches the number required for the restoration. An information processing apparatus characterized by the above. (Item 2) The processing means generates a processed image different from the input image. The information processing apparatus according to Item 1, characterized by the above. (Item 3) The processing means processes the input image by at least any one of translation of pixels of the input image, rotational movement of pixels of the input image, enlargement of the input image, and reduction of the input image. The information processing apparatus according to Item 1 or Item 2, characterized by the above. (Item 4) The processing means generates the processed image by moving the positions of all pixels of the input image. The information processing apparatus according to any one of Items 1 to 3, characterized by the above. (Item 5) The processing means generates the processed image with a processing amount set based on parameters for restoring the degradation. The information processing apparatus according to any one of Items 1 to 4, characterized by the above. (Item 6) When the processing means generates a plurality of processed images from the input image, the processing means generates the plurality of processed images based on a plurality of different processing amounts generated by integrating the processing amounts used for one-time processing. The information processing apparatus according to any one of Items 1 to 5, characterized by the above. (Item 7) The processing means generates the processed image with a processing amount set based on any one of the reduction ratios of the input image and the processed image used for restoring the deterioration, the downsampling ratio during pooling of the input image and the processed image, and the number of times of pooling of the input image and the processed image. The information processing apparatus according to any one of Items 1 to 6, characterized in that. (Item 8) When the input image is a plurality of time-series images, the processing means generates the generated processed image as an image older than the input image. The information processing apparatus according to any one of Items 1 to 7, characterized in that. (Item 9) When the processing means generates a plurality of processed images in time series based on different processing amounts, the processed image with a larger processing amount is generated as an older processed image. The information processing apparatus according to any one of Items 1 to 8, characterized in that. (Item 10) When the restoration means restores the deterioration using the processed image, the restoration means performs sharpening processing on the restored image. The information processing apparatus according to any one of Items 1 to 9, characterized in that. (Item 11) When the restoration means restores the deterioration based on a plurality of processed images, the restoration means sets the intensity of the sharpening processing according to the number of the processed images. The information processing apparatus according to Item 10, characterized in that. (Item 12) The processing means detects a subject included in the input image, and generates the processed image in a pixel movement direction set based on the position of the subject in the input image. The information processing apparatus according to any one of Items 1 to 11, characterized in that. (Item 13) The processing means detects the movement of the subject included in a plurality of input images, and generates the processed image according to the moving direction of pixels set based on the movement of the subject in the plurality of input images. The information processing apparatus according to any one of Items 1 to 12, characterized in that. (Item 14) The restoration means executes the restoration of the degradation by at least any one of noise reduction, blur removal, aberration correction, defect completion, correction processing for degradation due to compression, super-resolution processing for a low-resolution image, and correction processing for contrast reduction caused by the weather at the time of shooting. The information processing apparatus according to any one of Items 1 to 13, characterized in that. (Item 15) The processing means generates the processed image so that all of the input image and the processed image are different from each other. The information processing apparatus according to any one of Items 1 to 14, characterized in that. (Item 16) The processing means sets the moving direction so that the subject moves in the central direction of the input image. The information processing apparatus according to Item 12, characterized in that. (Item 17) The processing means sets the moving direction so that the subject does not protrude from the processed image. The information processing apparatus according to Item 13, characterized in that. (Item 18) An input means for acquiring an input image, A processing means for processing the input image to generate a processed image, A restoration means for generating a restored image obtained by restoring the degradation of the input image based on the input image and the processed image, Comprising The processing means processes either the input image or the processed image to generate the processed image until the number of the input image and the processed image becomes the number required for the restoration. An information processing method, characterized in that. (Item 19) A program for causing a computer to function as each means of the information processing apparatus according to any one of Items 1 to 17.

[0083] The invention is not limited to the above embodiments, and various changes and modifications are possible without departing from the spirit and scope of the invention. Therefore, the claims are attached to disclose the scope of the invention.

Explanation of Reference Numerals

[0084] 100 ··· Information processing apparatus, 112 ··· Imaging apparatus, 201 ··· Input unit, 202 ··· Processing unit, 203 ··· Storage unit, 204 ··· Restoration unit.

Claims

1. Input means for acquiring an input image, Processing means for processing the input image to generate a processed image, Restoration means for generating a restored image obtained by restoring the degradation of the input image based on the input image and the processed image, Comprising: The processing means generates the processed image by processing either the input image or the processed image until the number of the input image and the processed image reaches the number required for the restoration. An information processing apparatus characterized by the above.

2. The processing means generates a processed image different from the input image. The information processing apparatus according to claim 1, characterized by the above.

3. The processing means processes the input image by at least any one of translation of pixels of the input image, rotational movement of pixels of the input image, enlargement of the input image, and reduction of the input image. The information processing apparatus according to claim 1, characterized by the above.

4. The processing means generates the processed image by moving the positions of all pixels of the input image. The information processing apparatus according to claim 1, characterized by the above.

5. The processing means generates the processed image with a processing amount set based on parameters for restoring the degradation. The information processing apparatus according to claim 1, characterized by the above.

6. When the processing means generates a plurality of processed images from the input image, the processing means generates the plurality of processed images based on a plurality of different processing amounts generated by integrating the processing amounts used for one-time processing. The information processing apparatus according to claim 1, characterized by the above.

7. The processing means generates the processed image with a processing amount set based on any one of the reduction rate of the input image and the processed image used for restoring the degradation, the downsampling rate during pooling of the input image and the processed image, and the number of times of pooling of the input image and the processed image. The information processing apparatus according to claim 1, characterized by the above.

8. When the input image is a plurality of time-series images, the processing means generates the generated processed image as an image older than the input image. The information processing apparatus according to claim 1, characterized by the above.

9. When the processing means generates a plurality of processed images in time series based on different processing amounts, the processed image with a larger processing amount is generated as an older processed image. The information processing apparatus according to claim 1, characterized by the above.

10. When the restoration means restores the degradation using the processed image, the restoration means performs a sharpening process on the restored image. The information processing apparatus according to claim 1, characterized in that.

11. When the restoration means restores the degradation based on a plurality of processed images, the restoration means sets the intensity of the sharpening process according to the number of the processed images. The information processing apparatus according to claim 10, characterized in that.

12. The processing means detects a subject included in the input image, and generates the processed image in a moving direction of pixels set based on a position of the subject in the input image. The information processing apparatus according to claim 1, characterized in that.

13. The processing means detects movement of a subject included in a plurality of input images, and generates the processed image in a moving direction of pixels set based on the movement of the subject in the plurality of input images. The information processing apparatus according to claim 1, characterized in that.

14. The restoration means performs restoration of the degradation by at least any one of noise reduction, blur removal, aberration correction, defect completion, correction processing for degradation due to compression, super-resolution processing for a low-resolution image, and correction processing for contrast reduction caused by weather at the time of shooting. The information processing apparatus according to claim 1, characterized in that.

15. The processing means generates the processed image so that all of the input image and the processed image are different from each other. The information processing apparatus according to claim 1, characterized in that.

16. The processing means sets the moving direction so that the subject moves in a direction toward the center of the input image. The information processing apparatus according to claim 12, characterized in that.

17. The processing means sets the moving direction so that the subject does not protrude from the processed image. The information processing apparatus according to claim 13, characterized in that.

18. An input means for acquiring an input image, A processing means for processing the input image to generate a processed image, A restoration means for generating a restored image obtained by restoring the degradation of the input image based on the input image and the processed image, Comprising, The processing means processes either the input image or the processed image to generate the processed image until the number of the input image and the processed image reaches the number required for the restoration. An information processing method, characterized in that.

19. A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 17.

Images