Image processing device and image processing method

Abstract

To provide an image processing device, method, and program that suppress flicker caused by differences in characteristics between images obtained by image quality enhancement.SOLUTION: In a cloud server being an image processing device, a period estimation unit 301 estimates, as an error calculation period 351, a period in which fluctuation occurs in characteristics of images output from an image quality enhancement convolutional neural network (CNN), a generation unit 304 generates a degraded image 354 by degrading the image quality of a learning image acquired by an acquisition unit 302, an image quality enhancement unit 305 generates an image quality enhancement image 355 by inputting the degraded image 354 into the image quality enhancement CNN, and an error calculation unit 306 and an update unit 307 perform learning of the image quality enhancement CNN based on the image quality enhancement images 355 with a total number corresponding to the error calculation period 351.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a technology for improving image quality. [Background technology]

[0002] In recent years, convolutional neural networks (CNNs) have been applied to many image quality improvement applications. Image quality improvement processes specifically refer to processes for restoring an original image from a degraded image, such as noise removal, super-resolution, blur removal, and loss restoration. For example, Non-Patent Document 1 proposes a technology for performing highly accurate noise removal using a two-stage noise removal CNN.

[0003] When performing image quality improvement processing on multiple images that are highly correlated in the time series direction, differences in characteristics between the images after image quality improvement processing occur due to the structure of CNN, and in the case of moving images, flickering may occur. A technique for reducing the differences in characteristics between images is known, as disclosed in Patent Document 1. This method reduces the differences in characteristics between images by training the CNN using a loss function based on temporal changes in the image between adjacent frames and a loss function based on the difference between the image after image quality improvement processing and the correct image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6755395 [Non-patent literature]

[0005] [Non-Patent Document 1] FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation [Non-patent document 2] Learning Temporal Coherence via Self-Supervision for GAN-based Video Generation Summary of the Invention [Problem to be solved by the invention]

[0006] When the characteristics of images input and output to a CNN fluctuate over multiple frames, if the reference range used to calculate errors due to differences in the characteristics of the output image is shorter than the period of the fluctuations, the fluctuations in the characteristics cannot be taken into account, resulting in insufficient correction of the differences. The method disclosed in Patent Document 1 calculates the difference in the characteristics of the output image as an error, but the reference range is limited to images between adjacent frames and does not consider the differences in the characteristics of the input image. Non-Patent Document 2 discloses a method for two-stage CNNs in which errors are calculated for intermediate outputs and corrected before being used as subsequent inputs, but this method does not correct errors based on differences in the characteristics of images between frames. The present invention provides a technology for suppressing flicker caused by differences in the characteristics of images obtained by improving image quality. [Means for solving the problem]

[0007] One aspect of the present invention is characterized by comprising a first acquisition means for acquiring a period in which the characteristics of an image output from a learning model that improves image quality fluctuate, a second acquisition means for acquiring a first output image output from a learning model that receives a degraded image of an input image as input, and a learning means for learning the learning model based on a number of first output images corresponding to the period. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a technique for suppressing flicker caused by differences in characteristics between images obtained by improving image quality. [Brief explanation of the drawings]

[0009] [Figure 1] A diagram showing an example of the configuration of a high-image-quality CNN. [Figure 2] FIG. 2 is a block diagram showing an example of the hardware configuration of a cloud server 200. [Figure 3] FIG. 2 is a block diagram showing an example of the functional configuration of a cloud server 200. [Figure 4] 10 is a flowchart of degradation restoration learning by the cloud server 200. [Figure 5] A diagram showing an example of the configuration of a high-image-quality CNN. [Figure 6] FIG. 2 is a block diagram showing an example of the functional configuration of a cloud server 200. [Figure 7] 10 is a flowchart of degradation restoration learning by the cloud server 200. [Figure 8] 10 is a flowchart showing details of the process in step S707. [Figure 9] 10 is a flowchart showing details of the process in step S708. [Figure 10] A diagram showing an example of the configuration of a high-image-quality CNN. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] First, a description will be given of a convolutional neural network (CNN), which is an example of a learning model used in general image processing techniques that apply deep learning and which will appear in the following embodiments. CNN is a technology that repeatedly convolves an image with a filter generated through training, followed by nonlinear calculations. The filter is also called a local receptive field. The image obtained by convolving the image with the filter and then performing nonlinear calculations is called a feature map. Furthermore, learning is performed using training data (training images or data sets) consisting of pairs of input and output images. Simply put, learning is the process of generating, from training data, filter values ​​that can convert an input image into a corresponding output image with high accuracy. This will be described in more detail later.

[0012] If an image has RGB color channels or if a feature map is composed of multiple images, the filter used for convolution also has multiple channels accordingly. In other words, a convolution filter is expressed as a four-dimensional array, which includes the number of channels in addition to the vertical and horizontal size and number of images. The process of nonlinear computation after convolving a filter with an image (or feature map) is expressed in units called layers, such as the nth layer feature map or the nth layer filter. For example, a CNN that repeats filter convolution and nonlinear computation three times has a three-layer network structure. This type of nonlinear computation process can be formulated as shown in Equation (1) below.

[0013]

number

[0014] Here, Wn is the nth (n = 1 to N) layer filter, Xn is the nth layer feature map, bn is the nth layer bias, f is the nonlinear operator, and * is the convolution operator. Note that the (l) on the right indicates the lth filter, lth feature map, and lth bias, respectively. The filter and bias are updated through learning, which will be described later, and are collectively referred to as "network parameters." For example, a sigmoid function or ReLU (Rectified Linear Unit) is used as a nonlinear operation. In the case of ReLU, it is given by the following equation (2).

[0015]

number

[0016] In the nonlinear operation of equation (2), 0 is output as the output value corresponding to an element having a negative value among the elements of the input vector X, and the value of the element is output as the output value corresponding to an element having a positive value.

[0017] Well-known examples of networks using CNNs include ResNet, which is used in the field of image recognition, and its application RED-Net, which is used in the field of super-resolution. Both of these use multi-layered CNNs and perform multiple filter convolutions to achieve high processing accuracy. For example, ResNet features a network structure that provides paths that shortcut the convolutional layers, resulting in a multi-layered network with 152 layers, achieving high-precision recognition that approaches the recognition rate of humans. The reason multi-layered CNNs achieve high processing accuracy is, simply put, because they can express nonlinear relationships between input and output by repeating nonlinear operations multiple times.

[0018] Next, we will explain CNN learning. CNN learning is performed by minimizing an objective function, generally expressed by the following equation (3), for learning data consisting of a pair of input learning images (student images) and output learning images (teacher images) corresponding to the input learning images.

[0019]

number

[0020] Here, L is the loss function that measures the error between the correct answer and its estimation. Yi is the i-th output training image, and Xi is the i-th input training image. F is a function that collectively represents the calculations performed in each layer of the CNN (Equation (1)). θ is the network parameters (filter and bias). ||Z|| is the L2 norm, or simply the square root of the sum of the squares of the elements of vector Z. n is the total number of training data used for training. Since the total number of training data is generally large, stochastic gradient descent (SGD) randomly selects a portion of the training data for use in training. This reduces the computational load when training using a large amount of training data. Various methods are known for minimizing (optimizing) the objective function, including the momentum method, AdaGrad method, AdaDelta method, and Adam method. The Adam method is given by the following Equation (4).

[0021]

number

[0022] Here, θi t is the i-th network parameter at iteration t, and g is θi t is the gradient of the loss function L with respect to . Furthermore, m and v are moment vectors, α is the base learning rate, β1 and β2 are hyperparameters, and ε is a small constant. Note that there are no guidelines for selecting an optimization method in learning, so essentially any optimization method can be used, however, it is known that differences in convergence between methods will result in differences in learning time.

[0023] Next, we will use Figure 1 to explain the issues that arise in high-resolution CNNs, which are an example of CNNs used as learning models to perform image quality improvement processing to improve the quality of input images. The high-resolution CNN shown in Figure 1 performs image quality improvement processing by stacking two image quality improvement networks in two stages, which input three images, perform image quality improvement processing, and output three images. In the high-resolution CNN, images at times (t-5), (t-4), and (t-3) are first input to the first-stage image quality improvement network, image quality improvement network A-1, to obtain output images corresponding to times (t-5), (t-4), and (t-3). In the high-resolution CNN, images at times (t-2), (t-1), and t are input to the first-stage image quality improvement network, image quality improvement network A-2, to obtain output images corresponding to times (t-2), (t-1), and t. Next, the images at each time (frame) of (t-3), (t-2), and (t-1) are input into the second-stage image quality improvement network, image quality improvement network B, to obtain output images corresponding to each time (frame) of (t-3), (t-2), and (t-1) as the image quality improvement results.

[0024] When training the image quality improvement CNN, a degraded image obtained by adding degradation to an original image (e.g., an RGB image) is used as the input image, and the difference between the output image after image quality improvement processing of the input image and the original image is calculated as an error. The error is then back-propagated to the image quality improvement CNN, and the network parameters in the image quality improvement CNN are updated based on the results.

[0025] Here, in a high-resolution CNN, there is a time difference between the input images in the first-stage and second-stage high-resolution networks, so the output image of the first stage does not directly become the input image for the second stage. When training is performed on such a high-resolution CNN, during backpropagation, learning of network parameters from the first-stage output that are related to the input to the second stage progresses, but learning of unrelated network parameters progresses more slowly. In Figure 1, connections 1-3 and 2-3 are parameters that are not related to the input to the second stage, and therefore are parameters for which learning does not progress easily.

[0026] As shown in Figure 1, in the second stage, images output from both image quality enhancement network A-1 and image quality enhancement network B are used to output images at the respective times (t-3), (t-2), and (t-1). Here, the image output from image quality enhancement network A-1 is input to image quality enhancement network B via connection 1-3. The error described above is backpropagated from image quality enhancement network B to connections 2-1 and 2-2, but not to connection 1-3. This makes learning difficult, and the network parameters for connections 1-3, 2-1, and 2-2 are different. As a result, when a trained image quality enhancement CNN is used to infer image quality improvement for a degraded image, one in three images will be output with weak image quality enhancement processing because the network parameters for connections 1-3 and 2-3 have not yet been trained. For example, if the images are frames of a video sequence, this can cause flickering.

[0027] As such, there is a problem that the results of image quality improvement processing will have differences in characteristics between images due to the influence of network parameters that are difficult to learn and exist in the structure of high-quality image CNN.

[0028] [First embodiment] In this embodiment, we will take "noise removal processing, which is processing to remove noise in an image" as an example of image quality improvement processing, and explain a method for suppressing flickering caused by differences in characteristics between images output from an image quality improvement CNN. Note that in this embodiment, we will explain an example of an image quality improvement CNN using one image quality improvement network.

[0029] First, an example of the hardware configuration of the cloud server 200, which is an image processing device according to this embodiment, will be described using the block diagram of Fig. 2. Note that the hardware configuration shown in Fig. 2 is an example and can be modified / altered as appropriate. Furthermore, the cloud server 200 is an example of an image processing device, and may be a computer device such as a PC, a smartphone, or a tablet terminal device.

[0030] The cloud server 200 of this embodiment is, for example, a server device capable of providing cloud services over the Internet. More specifically, the cloud server 200 generates training data, estimates image quality degradation of an input image, and performs restoration learning (degradation restoration learning), and generates a training model including network parameters that are the results of the degradation restoration learning, information indicating the structure of the high-image-quality CNN (network structure), and the like.

[0031] The CPU 201 executes various processes using computer programs and data stored in the RAM 203. As a result, the CPU 201 controls the overall operation of the cloud server 200, and executes or controls various processes described as processes performed by the cloud server 200.

[0032] ROM 202 stores setting data for cloud server 200, computer programs and data related to the startup of cloud server 200, computer programs and data related to the basic operation of cloud server 200, and the like.

[0033] The RAM 203 has an area for storing computer programs and data loaded from the ROM 202 or the mass storage device 204, and an area for storing computer programs and data received from external devices via the network I / F 205. The RAM 203 also has a work area used by the CPU 201 when executing various processes. In this way, the RAM 203 can provide various areas as needed.

[0034] The mass storage device 204 is a large-capacity secondary storage device such as a hard disk drive, a solid state drive, etc. The mass storage device 204 stores an OS, computer programs and data for causing the CPU 201 to execute or control various processes described as processes performed by the cloud server 200.

[0035] Network I / F 205 functions as an interface for performing data communication with an external device via a network (which is the Internet in this embodiment, but may be another type of network such as a LAN). For example, cloud server 200 receives a request from a web browser running on an external device via network I / F 205, and transmits (provides) a response in response to the request to the external device via network I / F 205. This response may include, for example, the above-mentioned network parameters, the result of the image quality improvement processing, etc.

[0036] The CPU 201, ROM 202, RAM 203, mass storage device 204, and network I / F 205 are all connected to a system bus 206. Note that the cloud server 200 has other components in addition to those described above, but these are not the focus of the following explanation, so they will not be shown or described here.

[0037] In addition, in this embodiment, it is assumed that the generation of learning data and degradation restoration learning are performed by the cloud server 200. However, this configuration is merely an example, and the present invention is not limited to this. For example, the functions of the cloud server 200 may be subdivided, and the generation of learning data and degradation restoration learning may be performed by separate devices.

[0038] An example of the functional configuration of cloud server 200 is shown in the block diagram of FIG. 3. In the following, a case will be described in which each functional unit shown in FIG. 3 is implemented by software (computer program). In the following, each functional unit shown in FIG. 3 will be described as the subject of processing, but in reality, the function of that functional unit is realized by CPU 201 executing a computer program corresponding to that functional unit. Note that one or more of the functional units shown in FIG. 3 may be implemented by hardware.

[0039] The configuration shown in FIG. 3 can be modified or changed as appropriate. For example, one functional unit may be divided into multiple functional units, or two or more functional units may be integrated into one functional unit. The configuration shown in FIG. 3 may be realized by two or more devices. In this case, the devices are connected via circuits, wired or wireless networks, etc., and perform data communication with each other to perform cooperative operations, thereby realizing the processes described below. The degradation restoration learning by the cloud server 200 will be described with reference to the flowchart in FIG. 4.

[0040] In step S401, the period estimation unit 301 estimates the period in which the characteristics of the image output from the high image quality CNN fluctuate as the error calculation period 351. The error calculation period 351 is estimated based on the structure of the high image quality CNN.

[0041] For example, the period estimation unit 301 estimates the error calculation period 351 from the number of input and output images and the number of image quality improvement networks for the images. For example, in the image quality improvement CNN shown in Fig. 5, there is one image quality improvement network that inputs four input images and outputs three output images at time 2.

[0042] In the high-resolution CNN, high-resolution network B inputs four input images (images at each time (frame) of t, (t-1), (t-2), and (t-3)) and outputs three output images (images at each time (frame) of t, (t-1), and (t-2)).

[0043] In the high-resolution CNN, high-resolution network A inputs four input images (images at each of the times (frames) of (t-2), (t-3), (t-4), and (t-5)) and outputs three output images (images at each of the times (frames) of (t-2), (t-3), and (t-4)).

[0044] In such an image quality improvement CNN, for example, there are two output images at time (t-2): an output image at time (t-2) from image quality improvement network A and an output image at time (t-2) from image quality improvement network B. Here, the output image at time (t-2) of the image quality improvement CNN is assumed to be an average image of the output image at time (t-2) from image quality improvement network A and the output image at time (t-2) from image quality improvement network B. The average image of image A and image B is an image in which the pixel value at pixel position (x, y) in the average image is the average value of the pixel value at pixel position (x, y) in image A and the pixel value at pixel position (x, y) in image B. In this case, when the output image at time (t-2) is compared with the output images at time (t-1) and time (t-3), each of the compared images will exhibit different characteristics.

[0045] Furthermore, like the output image at time (t-2), the output images at time (t-4) and time t are also average images of two images, and therefore exhibit different characteristics from the images at the preceding and following times. In such cases, the error calculation period 351 can be estimated to be 2. Furthermore, if the structure of the high-quality CNN is unknown, multiple images that are highly correlated in the time series direction (correlation above a threshold) can be used as sample data, and the sample data can be input into the high-quality CNN to estimate the period from the characteristics of the output image. For example, frequency analysis can be performed on the output image, and the period of fluctuations in the image characteristics can be estimated from the peak frequency.

[0046] In this embodiment, the error calculation period 351 is assumed to be obtained by estimation by the period estimation unit 301, but the method for obtaining the error calculation period 351 is not limited to a specific method. For example, the period estimation unit 301 may obtain the error calculation period 351 input by a user operating a user interface such as a keyboard or mouse. Furthermore, the period estimation unit 301 may receive the error calculation period 351 input or estimated in a device other than the cloud server 200 from that device via the network I / F 205.

[0047] In step S402, the acquisition unit 302 acquires training images, which are a set of RGB images for training. The training images are a set of multiple RGB images that are highly correlated in the time series direction. Multiple sets are prepared in advance, and in step S402, one of the multiple sets is acquired.

[0048] The acquisition unit 302 may acquire the learning images by any method, for example, it may acquire learning images that are pre-stored in the mass storage device 204, or it may acquire learning images from an external device via the network I / F 205.

[0049] In step S403, the determining unit 303 sequentially obtains RGB images 352 from the learning images obtained by the obtaining unit 302, converts the RGB images 352 into grayscale images, and obtains the grayscale images as region determining images.

[0050] In step S404, the determining unit 303 performs filtering on the region determining image using an edge detection filter to detect edges from the region determining image. Filters such as a Prewitt filter or a Sobel filter can be used as the edge detection filter. Edge detection by filtering is performed in both the vertical and horizontal directions.

[0051] In step S405, the determining unit 303 identifies an area in the area determining image where the edge strength is less than the threshold, and determines a corresponding area in the RGB image 352 that corresponds to the identified area in terms of position as the error calculation area 353.

[0052] In step S406, the generation unit 304 sequentially acquires RGB images 352 from the learning images acquired by the acquisition unit 302, and generates degraded images 354 by degrading the image quality of the RGB images 302. The method for generating the degraded images 354 from the RGB images 352 is not limited to a specific method, and for example, the generation unit 304 may generate the degraded images 354 by adding pseudo noise to the RGB images 352. Note that a set of the RGB images 352 and the degraded images 354 corresponding to the RGB images 352 may be generated in advance.

[0053] In step S407, the image quality improving unit 305 performs noise removal processing on the degraded image 354 to generate a high-image-quality image 355. Specifically, the image quality improving unit 305 performs a convolution operation and a non-linear operation using the filters expressed by the above formulas (1) and (2) multiple times on the degraded image 354, and generates the result as the high-image-quality image 355. In other words, the image quality improving unit 305 inputs the degraded image 354 to a "high-image-quality CNN that performs a convolution operation and a non-linear operation using the filters expressed by formulas (1) and (2) multiple times" and performs the operation of the high-image-quality CNN to generate the high-image-quality image 355.

[0054] Through the above processing, an error calculation region 353, a degraded image 354, and a high-quality image 355 can be obtained for each of the multiple RGB images 352 in the learning images.

[0055] In step S408, error calculation unit 306 quantifies, in accordance with a loss function, the difference in characteristics between the number of high quality images 355 corresponding to the error calculation cycle 351, and calculates the result as error 356. More specifically, error calculation unit 306 acquires the number of high quality images 355 corresponding to the error calculation cycle 351 (if error calculation cycle 351=N (N is a natural number), then N or more images) as images for error calculation. Here, the number of high quality images 355 corresponding to the error calculation cycle 351 refers to consecutive frame images equivalent to "the number corresponding to the error calculation cycle 351."

[0056] Then, the error calculation unit 306 determines a corresponding area in the error calculation image that is positionally corresponding to the error calculation area 353 of the RGB image 352 that corresponds to the error calculation image as a target area, and determines a common area (common area) among the target areas of all the error calculation images. The method for determining the common area is not limited to a specific method. For example, the error calculation unit 306 may determine an area that overlaps among the target areas of all the error calculation images as the common area, or may determine a rectangular area that surrounds the target areas of all the error calculation images as the common area.

[0057] Then, the error calculation unit 306 calculates the error 356 based on pixel values ​​in the common region in each of the error calculation images. For example, the error calculation unit 306 calculates the error 356 using the following loss function:

[0058]

number

[0059] where L is the loss function φ(x i) may be a statistic of pixel values ​​in the common region in the i-th error calculation image, or may be a statistic of the difference between pixel values ​​in the common region in the i-th error calculation image and pixel values ​​in the common region in the RGB image 352 corresponding to the i-th error calculation image. In other words, the loss function of the above formula (5) is a function that calculates the total value of the differences in statistics between adjacent frames as error 356. In addition, examples of values ​​used as the statistic of pixel values ​​in the common region in the error calculation image are shown below. · Value obtained by performing global average pooling on the image Total variation value of the image A value that quantifies the similarity of the histograms of pixel values ​​between images using KL divergence, JS divergence, etc. Flicker rate: A value that indicates the magnitude of flicker, and is given by the following formula (6) using the maximum value A and minimum value B of the RGB values ​​of a pixel.

[0060]

number

[0061] In addition, examples of values ​​used as statistical amounts of the difference between pixel values ​​in the common area in the error calculation image and pixel values ​​in the common area in the RGB image 352 corresponding to the error calculation image are shown below. Image quality evaluation value such as SSIM The error may be calculated by adding an error based on the difference between the image quality improved image 355 and the RGB image 352 before noise is added. In step S409, the update unit 307 performs error backpropagation using the above formula (4) so ​​as to reduce the error 356 calculated by the error calculation unit 306, and calculates weight parameters 357, which are new network parameters of the image quality improved CNN. The update unit 307 then updates the current weight parameters of the image quality improved CNN with the calculated weight parameters 357.

[0062] The initial values ​​of the weight parameters of the high image quality CNN are set to arbitrary values, and the learning of the high image quality CNN progresses by updating them to newly calculated weight parameters 357. Note that the high image quality CNN inputs multiple images and outputs multiple images.

[0063] In step S410, the update unit 307 determines whether or not a termination condition for the learning (updating of weight parameters) of the high-image-quality CNN is satisfied. The termination condition can be, for example, "error 356 is less than a threshold," "the rate of change of error 356 is less than a threshold," "the elapsed time since the start of learning is equal to or greater than a specified time," or "the number of learning iterations is equal to or greater than a threshold."

[0064] If the end condition is met as a result of such determination, the processing according to the flowchart in FIG. 4 ends, and if the end condition is not met, the processing proceeds to step S402.

[0065] In this way, according to this embodiment, it is possible to obtain a high-quality image CNN that has been trained to reduce the difference in characteristics between output images. When multiple images that are highly correlated in the time series direction are input to such a high-quality image CNN and inference is performed, flickering caused by the difference in characteristics between the output images can be suppressed.

[0066] The input image is not limited to an RGB image, but may be an image of another format such as a monochrome image, a RAW image, etc. In addition, in this embodiment, noise removal processing is used as an example of image quality improvement processing, but the present invention is not limited to this and can also be applied to other types of processing for restoring the original image from a degraded image, such as blur removal, super-resolution, and defect completion.

[0067] [Second embodiment] In this embodiment, differences from the first embodiment will be described, and unless otherwise specified below, it will be assumed that the present embodiment is the same as the first embodiment. In the first embodiment, a high-image-quality CNN using a single high-image-quality network was used as an example, and a learning process for suppressing flickering caused by differences in characteristics between output images was described. In contrast, in this embodiment, the learning process of a high-image-quality CNN having a structure in which multiple high-image-quality networks are stacked will be described. In this embodiment, a case will be described in which a high-image-quality CNN having the configuration shown in FIG. 1 is used as the high-image-quality CNN.

[0068] In the image-enhancing CNN configuration shown in Figure 1, the output image of the first-stage image-enhancing network is used as the input image of the second-stage image-enhancing network. To reduce the difference in characteristics of the output image from the second-stage image-enhancing network, it is desirable to reduce the difference in image characteristics at the time of output from the first stage before inputting it to the second stage. Therefore, for the first-stage output image, an error based on the difference in characteristics between images is calculated, and the weight parameters are updated to reduce this error, thereby reducing the difference in characteristics between the images input to the second stage. For the second-stage output image, an error based on the difference in characteristics between images is also calculated, and the weight parameters are updated to reduce this error, thereby reducing the difference in characteristics of the output image. In this image-enhancing CNN, the first-stage image-enhancing network and the second-stage image-enhancing network have different phases. This is because a wider range of images is referenced over time during image enhancement processing, thereby reducing temporal changes in the output image. Here, when referencing errors from the output results of the first and second stages, the phase of the time series reference range when calculating errors differs between stages 1 and 2. For example, for the output of the first stage, errors are calculated from images at times (t-2), (t-1), and t, and for the output of the second stage, errors are calculated from images at times (t-5), (t-4), and (t-3).

[0069] An example of the functional configuration of the cloud server 200 according to this embodiment is shown in the block diagram of Fig. 6. In Fig. 6, functional units having the same functions as those shown in Fig. 3 are assigned the same reference numerals as those functional units, and descriptions of those functional units will be omitted.

[0070] The degradation restoration learning by the cloud server 200 will be described with reference to the flowchart of Fig. 7. In Fig. 7, the same processing steps as those shown in Fig. 4 are assigned the same step numbers as those of the processing steps, and the description of those processing steps will be omitted.

[0071] In step S701, the period estimation unit 601 estimates the period in which the characteristics of the image output from the high image quality CNN fluctuate as the error calculation period 651. The high image quality CNN configured as shown in Fig. 1 has one high image quality network for time 3, and the time difference between the image input to the high image quality net A-1 (A-2) and the image input to the high image quality net B is 1. For this reason, one in three images are output with different characteristics, and the period in which the image characteristics fluctuate (error calculation period 651) is estimated to be 3.

[0072] In step S707, the first image quality improvement network (corresponding to the image quality improvement network A-1 (A-2) in FIG. 1) generates / calculates a first image quality improvement image 655 and a first error 656 based on the degraded image 354. Details of the processing in step S707 will be described later.

[0073] In step S708, the second image quality improvement network (corresponding to image quality improvement network B in FIG. 1) generates / calculates a second image quality improvement image 657 and a second error 658 based on the first image quality improvement image 655. Details of the processing in step S708 will be described later.

[0074] In step S709, the update unit 609 uses the above equation (4) to perform error backpropagation on the image quality improvement net A-1 (A-2) and the image quality improvement net B so as to reduce the overall error based on the first error 656 and the second error 658, calculates weight parameters 659 that are new network parameters of the image quality improvement net A-1 (A-2) and the image quality improvement net B, and updates the current weight parameters of the image quality improvement net A-1 (A-2) and the image quality improvement net B with the calculated weight parameters 659. The overall error based on the first error 656 and the second error 658 is, for example, a linear sum of the first error 656 and the second error 658.

[0075] In step S710, the update unit 609 determines whether or not the termination condition for the learning (updating of weight parameters) of the high-image-quality CNN is satisfied. The termination condition can be, for example, "the total error is less than a threshold," "the rate of change of the total error is less than a threshold," "the elapsed time since the start of learning is equal to or greater than a specified time," or "the number of learning iterations is equal to or greater than a threshold."

[0076] If the end condition is met as a result of such determination, the processing according to the flowchart in FIG. 7 ends, and if the end condition is not met, the processing proceeds to step S402.

[0077] Next, details of the processing in step S707 above will be described with reference to the flowchart in Fig. 8. In step S801, the first image quality improvement unit 605 inputs the degraded image 354 to "an image quality improvement net A-2 that performs a calculation that repeats convolution calculations and nonlinear calculations using filters expressed by equations (1) and (2) multiple times," and performs calculations on the image quality improvement net A-2 to generate a first image quality improvement image 655.

[0078] Here, the initial values ​​of the weight parameters of the image quality improvement network A-2 are set to arbitrary values, and learning progresses by updating them to weight parameters 659 obtained from the update unit 609. Note that the image quality improvement network A-2 inputs multiple images and outputs multiple images. In the image quality improvement CNN shown in Figure 1, the image quality improvement network A-2 inputs three images at times (t-2), (t-1), and t, and outputs three images at the same time.

[0079] In step S802, the first error calculation unit 606 acquires, as images for error calculation, a number of first high quality images 655 corresponding to the error calculation period 651, as in the first embodiment. Then, the error calculation unit 606 determines a common region, as in the first embodiment, and calculates a first error 656 based on pixel values ​​in the common region in each image for error calculation. In the high quality CNN shown in FIG. 1, the first error calculation unit 606 calculates the first error 656 based on the difference in characteristics among the three images at times (t-2), (t-1), and t.

[0080] Next, details of the processing in step S708 above will be described with reference to the flowchart in Fig. 9. In step S901, the second image quality improvement unit 607 inputs the first image quality improvement image 655 to "image quality improvement net B, which performs calculations that repeat convolution calculations and non-linear calculations using filters expressed by equations (1) and (2) multiple times," and performs calculations on the image quality improvement net B to generate a second image quality improvement image 657.

[0081] Here, the initial values ​​of the weight parameters of image quality improvement net B are set to arbitrary values, and learning progresses by updating them to weight parameters 659 obtained from weight parameter update unit 609. Note that image quality improvement net B inputs multiple images and outputs multiple images. In the image quality improvement CNN shown in Figure 1, image quality improvement net B inputs three images at times (t-3), (t-2), and (t-1), and outputs three images at the same time.

[0082] In step S902, the second error calculation unit 608 acquires, as images for error calculation, a number of second high-quality images 655 corresponding to the error calculation cycle 651, in the same manner as in the first embodiment. Then, the error calculation unit 606 determines a common region in the same manner as in the first embodiment, and calculates a second error 658 based on pixel values ​​in the common region in each image for error calculation. In the high-quality CNN shown in FIG. 1, the second error calculation unit 608 calculates the second error 658 based on the difference in characteristics among the three images taken at times (t-3), (t-2), and (t-1).

[0083] As described above, according to this embodiment, the phases of the time-series reference ranges used to calculate errors are different between the first and second stages, resulting in overlapping of the reference ranges and suppressing the step-like difference in characteristics between images at the edges of the reference ranges. Furthermore, by reducing the difference in characteristics between images at the first-stage output stage, the difference in characteristics between images in the second-stage output results is also reduced, enabling more effective suppression of flicker. The above explanation is also applicable to a high-image-quality CNN having a structure in which three or more high-image-quality networks are stacked.

[0084] [Third embodiment] In this embodiment, differences from the second embodiment will be described, and unless otherwise specified below, it will be assumed that the present embodiment is the same as the second embodiment. In the second embodiment, a high-quality image CNN having the structure shown in FIG. 1 was used as an example to describe a mode of suppressing flickering caused by differences in characteristics between output images, but a high-quality image CNN having another structure may also be used. In this embodiment, a high-quality image CNN having the structure shown in FIG. 10 will be used as an example to describe a mode of suppressing flickering caused by differences in characteristics between output images.

[0085] The image quality improvement CNN in Figure 10 has a structure in which two image quality improvement networks are stacked, each inputting five images and outputting five images, with one image quality improvement network at time 4. There is a time difference between the image input to image quality improvement network A-1 (A-2) and the image input to image quality improvement network B-1 (B-2) by one unit. As a result, one in four images with different characteristics are output, and it is estimated that the period in which the image characteristics fluctuate is four.

[0086] In this embodiment, in step S801, the first image quality improvement unit 605 inputs five degraded images 354 at times (t-4), (t-3), (t-2), (t-1), and t into an image quality improvement network A-2 that performs operations that repeat convolution operations and nonlinear operations using filters represented by equations (1) and (2) multiple times, and performs operations on the image quality improvement network A-2 to generate a first image quality improvement image 655 at the corresponding time.

[0087] Here, there are two output images at time (t-4): one from the image quality improvement net A-1 and one from the image quality improvement net A-2. Therefore, the image obtained by taking the average value of the RGB values ​​for each pixel of the two images is used as the output image at time (t-4). Also, since there are two output images for the image at time t, similar processing is performed to output one image. Note that if this embodiment is not applied, the output image at time (t-4) from the image quality improvement net A-1 and the output image at time (t-4) from the image quality improvement net A-2 will not have sufficient image quality processing.

[0088] In this embodiment, in step S802, the first error calculation unit 606 acquires five first high-quality images 655 at times (t-4), (t-3), (t-2), (t-1), and t as images for error calculation. The error calculation unit 606 then determines a common area in the same manner as in the first embodiment, and calculates a first error 656 based on pixel values ​​within the common area in each of the images for error calculation.

[0089] Also, in this embodiment, in step S901, the second image quality improvement unit 607 inputs five first image quality improved images 655 at times (t-5), (t-4), (t-3), (t-2), and (t-1) to "an image quality improvement network B-1 that performs an operation of repeating convolution operation and nonlinear operation using filters expressed in equations (1) and (2) multiple times," and performs an operation of the image quality improvement network B-1 to generate a second image quality improved image 657 at the corresponding time. Here, since there are two output images at times (t-5) and (t-1), the same process as in step S701 is performed to output one image.

[0090] In step S902, the second error calculation unit 608 acquires five second high-quality images 655 taken at times (t-5), (t-4), (t-3), (t-2), and (t-1) as error calculation images. The error calculation unit 606 then determines a common area in the same manner as in the first embodiment, and calculates a second error 658 based on pixel values ​​within the common area in each of the error calculation images.

[0091] According to this embodiment, it is possible to reduce the difference in characteristics between output images even for an image quality improvement CNN having a structure in which one image quality improvement network with five inputs and five outputs exists for time 4. The same can be applied regardless of the number of input and output images or the number of image quality improvement networks over time.

[0092] The numerical values, processing timing, processing order, processing subject, data (information) configuration / acquisition method / sending destination / sending source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0093] In addition, some or all of the above-described embodiments may be used in appropriate combination, and some or all of the above-described embodiments may be selectively used.

[0094] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0095] The invention of this specification includes the following image processing device, image processing method, and computer program. (Item 1) a first acquisition means for acquiring a period in which characteristics of an image output from a learning model that performs image quality improvement vary; a second acquisition means for acquiring a first output image output from a learning model to which a degraded image of an input image is input; a learning means for learning a learning model based on the number of first output images corresponding to the period; An image processing device comprising: (Item 2) The image processing device according to item 1, characterized in that the learning means calculates an error based on a difference in statistics based on pixel values ​​of the number of first output images corresponding to the cycle, and performs the learning based on the error. (Item 3) The image processing device described in item 2 is characterized in that the learning means determines a common area in areas where the edge strength is less than a threshold for the number of first output images corresponding to the period, calculates an error based on the difference in statistics based on the pixel values ​​of the common area, and performs the learning based on the error. (Item 4) The image processing device described in item 1, characterized in that the learning means calculates an error based on a difference in statistics based on pixel values ​​between the number of first output images corresponding to the cycle and the corresponding input image, and performs the learning based on the error. (Item 5) Item 5. The image processing device according to item 4, wherein the learning means determines a common area in an area where the edge strength is less than a threshold for the number of first output images corresponding to the period, and calculates an error based on the difference in statistics of the differences between the pixel values ​​of each common area in the first output images and the pixel values ​​of the common area in the corresponding input image. (Item 6) the second acquisition means acquires a first output image output from a first image quality improvement network in the learning model to which the degraded image is input; The learning means learns the first image quality improvement network based on the number of first output images corresponding to the period. 6. The image processing device according to any one of items 1 to 5, (Item 7) the second acquisition means acquires a first output image output from a first image quality improvement network in the learning model to which the degraded image is input; The image processing device described in any one of items 1 to 5, characterized in that the learning means acquires second output images output from a second image quality improvement network in the learning model to which the first output images have been input, and learns the learning model based on the number of first output images corresponding to the period and the number of second output images corresponding to the period. (Item 8) 8. The image processing device according to item 7, wherein the phase of the reference range differs between the image input to the first image quality improvement network and the image input to the second image quality improvement network. (Item 9) 9. The image processing device according to any one of items 1 to 8, wherein the first acquisition means estimates the period based on a structure of a learning model. (Item 10) 9. The image processing device according to any one of items 1 to 8, wherein the first acquisition means uses a plurality of images having a correlation in a time series direction equal to or greater than a threshold as sample data, inputs the sample data into a learning model, and estimates the period from the characteristics of the output image. (Item 11) 11. The image processing device according to any one of items 1 to 10, wherein the second acquisition means generates the degraded image by adding noise to the input image. (Item 12) An image processing method performed by an image processing device, a first acquisition step in which a first acquisition means of the image processing device acquires a period in which characteristics of an image output from a learning model that performs image quality improvement fluctuates; a second acquisition step in which a second acquisition means of the image processing device acquires a first output image output from a learning model to which a degraded image of an input image is input; a learning step in which a learning means of the image processing device learns a learning model based on the number of first output images corresponding to the period; An image processing method comprising: (Item 13) 12. A computer program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 11.

[0096] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0097] 301: Period estimation unit 302: Acquisition unit 303: Determination unit 304: Generation unit 305: Image quality improvement unit 306: Error calculation unit 307: Update unit 351: Error calculation period 352: RGB image 353: Error calculation area 354: Degraded image 355: Image quality improvement image 356: Error 357: Weighting parameter

Claims

1. a first acquisition means for acquiring a period in which characteristics of an image output from a learning model that performs image quality improvement vary; a second acquisition means for acquiring a first output image output from a learning model to which a degraded image of an input image is input; a learning means for learning a learning model based on the number of first output images corresponding to the period; An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein the learning means calculates an error based on a difference between statistics based on pixel values ​​of the first output images of a number corresponding to the cycle, and performs the learning based on the error.

3. The image processing device described in claim 2, characterized in that the learning means determines a common area in areas where the edge strength is less than a threshold for the number of first output images corresponding to the period, calculates an error based on the difference in statistics based on the pixel values ​​of the common area, and performs the learning based on the error.

4. 2. The image processing device according to claim 1, wherein the learning means calculates an error based on a difference in a statistical quantity based on pixel values ​​between a number of first output images corresponding to the period and a corresponding input image, and performs the learning based on the error.

5. The image processing device described in claim 4, characterized in that the learning means determines a common area common to areas where edge strength is less than a threshold for a number of first output images corresponding to the period, and calculates an error based on the difference in statistics of the differences between the pixel values ​​of each common area of ​​the first output images and the pixel values ​​of the common area of ​​the corresponding input image.

6. the second acquisition means acquires a first output image output from a first image quality improvement network in the learning model to which the degraded image is input; The learning means performs learning of the first image quality improvement network based on the number of first output images corresponding to the period.

2. The image processing device according to claim 1, wherein:

7. the second acquisition means acquires a first output image output from a first image quality improvement network in the learning model to which the degraded image is input; The image processing device described in claim 1, characterized in that the learning means acquires a second output image output from a second image quality improvement network in the learning model to which the first output image is input, and learns the learning model based on a number of first output images corresponding to the period and a number of second output images corresponding to the period.

8. 8. The image processing apparatus according to claim 7, wherein the phase of the reference range differs between the image input to the first image quality improvement network and the image input to the second image quality improvement network.

9. The image processing apparatus according to claim 1 , wherein the first acquisition means estimates the period based on the structure of a learning model.

10. The image processing device according to claim 1, characterized in that the first acquisition means uses a plurality of images having a correlation in a time series direction equal to or greater than a threshold as sample data, inputs the sample data into a learning model, and estimates the period from the characteristics of the output image.

11. The image processing apparatus according to claim 1 , wherein the second acquisition means generates the degraded image by adding noise to the input image.

12. An image processing method performed by an image processing device, a first acquisition step in which a first acquisition means of the image processing device acquires a period in which characteristics of an image output from a learning model that performs image quality improvement vary; a second acquisition step in which a second acquisition means of the image processing device acquires a first output image output from a learning model to which a degraded image of an input image has been input; a learning step in which a learning means of the image processing device learns a learning model based on the number of first output images corresponding to the period; An image processing method comprising:

13. A computer program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 11.

Images