Image processing device, image processing method, photographing device and program

Abstract

To solve the problem in which output images whose image quality has been improved are likely to be generated with many artifacts.SOLUTION: An image processing device performs image quality improvement processing using a neural network having a plurality of layers, and comprises: image quality improvement means for applying the image quality improvement processing using a weight parameter to an input image and outputting an inference result; selection means for selecting a plurality of inference results calculated using the single weight parameter as a peer data set; statistic calculation means for calculating a statistic for the inference results of the peer data set; error calculation means for calculating an error in the statistic for each peer data set based on the statistic and a reference value of the statistic; and update means for updating the weight parameter based on the error.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, an imaging device, and a program. [Background technology]

[0002] In recent years, convolutional neural networks (CNNs) have been applied to many image quality improvement applications. Image quality improvement processing specifically refers to processing for restoring the original image from a degraded image, such as noise removal, super-resolution, blur removal, and loss completion. As an example, a proposal has been made to use CNNs to perform highly accurate noise removal from videos (Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Matias Tassano and 2 others, “FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation”, [online], April 29, 2020, [Retrieved August 14, 2023], Internet,<URL:https: / / arxiv.org / pdf / 1907.01361.pdf> [Non-Patent Document 2] Markus Nagel and 5 others, “A White Paper on Neural Network Quantization”, [online], June 15, 2021, [Retrieved August 14, 2023], Internet,<URL: https: / / arxiv.org / pdf / 2106.08295.pdf> Summary of the Invention [Problem to be solved by the invention]

[0004] In the video quality improvement method disclosed in Non-Patent Document 1, multiple images that are consecutive in time series are input, and a calculation is performed to output one image with improved quality. In this method, one calculation is required for each frame of an image with improved quality to be output, so the processing cost is high. In order to reduce the processing cost, it is conceivable to add a branching structure to the architecture of Non-Patent Document 1, in which one layer outputs multiple inference results. That is, the architecture is changed so that N degraded images are input, and N output images with improved quality are output. This makes it possible to output output images with improved quality for N frames at a time with one calculation, thereby reducing the processing cost.

[0005] However, in the method of Non-Patent Document 1, the distributions of the inference results are likely to be learned to be different from each other. As a result, due to differences in the accumulated inference results, the image quality, such as the color and smoothness of the image, differs from frame to frame, and a high-quality output image with many artifacts is likely to be generated.

[0006] Therefore, an object of the present invention is to suppress artifacts in an output image with improved image quality. [Means for solving the problem]

[0007] In order to solve this problem, for example, an image processing device of the present invention has the following configuration. That is, the image processing device is an image processing device that performs image quality improvement processing using a neural network having multiple layers, an image quality improvement means for applying the image quality improvement process using the weighting parameters to an input image and outputting an inference result; A selection means for selecting a plurality of inference results calculated using a single weight parameter as a peer data set; A statistics calculation means for calculating statistics of the inference result of the peer data set; an error calculation means for calculating an error of the statistical quantity for each of the peer data sets based on the statistical quantity and a reference value of the statistical quantity; updating means for updating the weight parameters based on the error; The present invention is characterized by comprising: Effect of the Invention

[0008] According to the present invention, artifacts in an output image can be suppressed in improving image quality. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an image processing system. [Diagram 2] FIG. 2 is a diagram showing an example of a hardware configuration of an apparatus in the image processing system. [Diagram 3] FIG. 1 is a functional block diagram of an image processing system according to a first embodiment. [Figure 4] 4 is a flowchart showing a processing flow in the image processing system according to the first embodiment. [Diagram 5] A diagram explaining the architecture and how it can be improved. [Figure 6] 11A and 11B are diagrams for explaining a shift in distribution of a residual map. [Figure 7] 11A and 11B are diagrams for explaining factors that worsen the deviation in distribution of a residual map. [Figure 8] FIG. 1 is a diagram for explaining elements of multiplexing. [Figure 9] FIG. 13 is a diagram showing an example of architecture information. [Figure 10] FIG. 2 is a diagram for explaining the connection relationship between layers. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] <CNNについて> First, a description will be given of a Convolutional Neural Network (CNN) that is used in general image processing techniques that apply deep learning and that will appear in each of the following embodiments. CNN is a technique that repeats convolution of a filter generated by training with image data, followed by nonlinear calculation. The filter is also called a local receptive field. Image data obtained by convolving the filter with image data and then performing nonlinear calculation is called a feature map. Furthermore, learning is performed using training data (training images or data sets) consisting of pairs of input image data and output image data. Simply put, learning is the generation of filter values ​​that can convert input image data into corresponding output image data with high accuracy from training data. This will be described in detail later.

[0012] When image data has RGB color channels or when a feature map is composed of multiple pieces of image data, the filter used for convolution has multiple channels according to the number of color channels or the number of pieces of image data. In other words, a convolution filter is expressed as a four-dimensional array that includes the number of channels in addition to the vertical and horizontal sizes and number of sheets. The process of nonlinear calculation after convolving a filter with image data (or feature map) is expressed in units of layers, such as the nth layer feature map or the nth layer filter. In addition, for example, a CNN that repeats filter convolution and nonlinear calculation three times has a three-layer network structure. Such nonlinear calculation processing can be formulated as the following equation (1).

[0013]

number

[0014] In the above formula (1), W n is the nth layer filter, b nThe bias of the n-th layer is \(b^{(n)}\), \(f\) is a non-linear operator, and \(X\) n is the feature map of the n-th layer, and \(*\) is a convolution operator. Note that the \((l)\) on the upper right indicates that it is the \(l\)-th filter or feature map. The filters and biases are generated by the learning described later and are collectively also called "network parameters". As the non-linear operation, for example, a sigmoid function or ReLU (Rectified Linear Unit) is used. In the case of ReLU, it is given by the following formula (2).

[0015]

Equation

[0016] As shown in the above formula (2), among the elements of the input vector \(X\), the negative ones become zero and the positive ones remain as they are.

[0017] As networks using CNN, ResNet in the field of image recognition and its application RED-Net in the field of super-resolution are well-known. In both cases, by stacking multiple layers of CNN and performing convolution of filters multiple times, high-precision image processing is achieved. For example, ResNet features a network structure with a path that shortcuts the convolutional layers, thereby realizing a 152-layer multi-layer network and achieving high-precision recognition approaching the human recognition rate. The reason why the processing becomes more accurate with multi-layer CNN is simply that by repeating the non-linear operation multiple times, the non-linear relationship between the input and output can be expressed.

[0018] <Learning of CNN> Next, the learning of CNN will be explained. The learning of CNN is generally performed by minimizing the objective function expressed by the following formula (3) for the training data consisting of a pair of input training image (student image) data and corresponding output training image (teacher image) data.

[0019]

Equation

[0020] In the above formula (3), L is a loss function that measures the error between the correct answer and its estimate. i is the i-th output training image data, X i is the i-th input training image data. F is a function that collectively represents the calculations (Equation 1) performed in each layer of the CNN. θ is a network parameter including the filter and bias. ||Z||2 is the L2 norm, or more simply, the square root of the sum of the squares of the elements of the vector Z. n is the total number of training data used for training. Since the total number of training data is generally large, in the Stochastic Gradient Descent (SGD) method, a portion of the training image data is randomly selected and used for training. This reduces the computational load in training using a large amount of training data. In addition, various methods are known for minimizing (optimizing) the objective function, such as the momentum method, the AdaGrad method, the AdaDelta method, and the Adam method. The Adam method is given by the following equation (4).

[0021]

number

[0022] In the above formula (4), θ i t is the i-th network parameter at the t-th iteration, and g is θ i t is the gradient of the loss function L with respect to . Additionally, m and v are moment vectors, α is the base learning rate, β1 and β2 are hyperparameters, and ε is a small constant. Note that since there are no guidelines for selecting an optimization method in learning, essentially any method can be used, but it is known that differences in convergence between methods will result in differences in learning time.

[0023] <Challenges for high-resolution CNN> The problems that arise in high-image-quality CNN, as typified by Non-Patent Document 1, will be described in detail with reference to Figs. 5, 6, and 7.

[0024] 5(a) shows the data flow in the CNN in Non-Patent Document 1. The CNN 501 in Non-Patent Document 1 has two noise removal networks, Step 1 and Step 2, inside to improve noise removal performance.

[0025] First, a degraded image 503 with noise superimposed thereon as shown in Fig. 5(c) is input to a noise removal network 504 in Step 1, which outputs a residual map 505, which is the noise components (degraded components) of the degraded image. At this time, noise removal is performed on each of the degraded images 503 containing degraded components at adjacent times t-2, t-1, and t. The obtained residual map 505 in Step 1 is added to the degraded image, and a high-quality image 506 in Step 1 is output. At this point, the noise in the degraded image has not been completely removed.

[0026] To remove the remaining noise, the Step 1 enhanced image 506 is input to a Step 2 noise removal network 507 to obtain a Step 2 residual map 508. By adding this to the Step 1 enhanced image 506, the Step 2 enhanced image 509 is output.

[0027] However, this process requires one inference calculation for the noise reduction network at each time, which results in high processing costs. Therefore, we can consider an improvement to the form of CNN502 shown in Figure 5(b), which reduces the number of inferences per time.

[0028] In this CNN 502, the noise reduction network 504 and the noise reduction network 507 each take N images (here, N=3) as input and output N residual maps. The final layer of the network is configured so that one layer outputs multiple output data, and such a configuration is hereinafter referred to as layer multiplexing. This process makes it possible to input images of adjacent N frames and obtain high-quality images of N frames by performing one inference, achieving a maximum N-times speedup effect.

[0029] The N degraded images 503 are based on images taken at similar times, and the appearance of the subject is similar. Therefore, the N residual maps 505 in layer 510 representing the noise components added thereto ideally follow a similar distribution. The same is true for the residual map 508 in layer 511 in Step 2.

[0030] However, in the method of Non-Patent Document 1, since no constraints are set regarding the similarity of each output, differences in the distribution of each output are likely to occur due to learning. An example is shown in FIG.

[0031] Figure 6 shows nine distributions for each of the three channels R, G, and B of the residual maps of three frames at times t-2, t-1, and t. It is desirable for the values ​​of the three residual maps to follow the same distribution if they are in the same channel. However, in the histogram of Figure 6, each of the nine distributions has a different displacement and shape.

[0032] When there is a difference in the displacement or shape of the distribution for each residual map, the difference may be aggravated by the influence of other operations in the CNN. For example, the quantization method of the neural network represented by Non-Patent Document 2 is a method of performing a process of reducing the computation bit depth of the learning result model in order to increase the execution speed of the neural network. An example of this process is shown in FIG. 7. When the computation bit depth is reduced, the expression range of the value 701 of each output is limited to a limited section 702. In an ideal distribution, the loss of information due to the range restriction is minor, but there is an area 703 where the value cannot be expressed in a histogram with differences. In this case, the value of the area 703 is clipped to the maximum value of the section 702, and the positive and negative symmetry of the histogram is lost. As a result, color shifts and changes in sharpness occur due to the average value of the histogram changing for each frame of the multiple output images, and artifacts occur in which the image quality of the multiple output images with high image quality changes for each frame.

[0033] As described above, if constraints on the similarity of the CNN output are not set, there is a problem that artifacts may occur in the high-quality output image due to the influence of other operations in the CNN.

[0034] First Embodiment In this embodiment, a method for reducing the above-mentioned artifacts will be described by taking as an example a case where noise removal is performed to improve image quality.

[0035] <Image processing system configuration example> Fig. 1 is a diagram showing an example of the configuration of an image processing system according to this embodiment. The image processing system shown in Fig. 1 includes a cloud server 200 that generates learning data and estimates image quality degradation and learns restoration (hereinafter, degradation restoration learning), and an edge device 100 that performs degradation restoration (hereinafter, degradation restoration inference). The cloud server 200 and the edge device 100 are connected via the Internet. The cloud server 200 and the edge device 100 are, for example, computers.

[0036] <Edge device hardware configuration> FIG. 2 is a diagram showing an example of a hardware configuration of an apparatus of an image processing system. The edge device 100 of this embodiment is a processing apparatus that performs degradation restoration inference by applying learned network parameters provided from a cloud server 200 to image data input from an imaging apparatus 10. A user applies super-resolution processing to image data using an image processing application installed in the edge device 100. The edge device 100 has a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, a large-capacity storage device 104, a general-purpose interface (hereinafter, general-purpose I / F) 105, and a network I / F 106. The CPU 101, the RAM 102, the ROM 103, the large-capacity storage device 104, the general-purpose I / F 105, and the network I / F 106 are mutually connected by a system bus 107. In addition, the edge device 100 is connected to the imaging apparatus 10, the input device 20, the external storage device 30, and the display device 40 via the general-purpose I / F 105.

[0037] The CPU 101 executes a program stored in the ROM 103 using the RAM 102 as a work memory, and performs overall control of each component of the edge device 100 via a system bus 107. The RAM 102 is used as a temporary storage area such as the main memory and work area of ​​the CPU 101. The ROM 103 stores the program executed by the CPU 101 and parameters required for executing the program. The mass storage device 104 is, for example, a hard disk drive (HDD) or a solid state drive (SSD), and stores various programs and data handled by the edge device 100. The CPU 101 writes data to the mass storage device 104 and reads data stored in the mass storage device 104 via the system bus 107. The general-purpose I / F 105 is, for example, a serial bus interface such as USB, IEEE1394, or HDMI (registered trademark).

[0038] The edge device 100 acquires data from the external storage device 30 (for example, various storage media such as a memory card, a CF card, an SD card, a USB memory, etc.) via the general-purpose I / F 105. The edge device 100 accepts user instructions from an input device 20 such as a mouse or a keyboard via the general-purpose I / F 105. The edge device 100 outputs image data processed by the CPU 101 to a display device 40 (for example, various image display devices such as a liquid crystal display, etc.) via the general-purpose I / F 105. The edge device 100 acquires data of a captured image (RAW image) to be subjected to super-resolution processing from the imaging device 10 via the general-purpose I / F 105. The network I / F 106 is an interface for connecting to the Internet. The edge device 100 accesses the cloud server 200 via an installed web browser or the like to acquire network parameters including weight parameters for degradation restoration inference.

[0039] <Cloud server hardware configuration> The cloud server 200 of this embodiment is a server device that provides cloud services on the Internet, and is also an image processing device. More specifically, the cloud server 200 generates learning data and performs degradation restoration learning, and provides a learned model that stores weight parameters and a network structure of the learning result in response to a request from the edge device 100. The cloud server 200 includes a CPU 201, a ROM 202, a RAM 203, a mass storage device 204, and a network I / F 205. The CPU 201, the ROM 202, the RAM 203, the mass storage device 204, and the network I / F 205 are connected to each other by a system bus 206.

[0040] The CPU 201 reads out a control program stored in the ROM 202, expands it in the RAM 203, and executes various processes to control the overall operation of the cloud server 200. The ROM 202 stores the programs executed by the CPU 201 and parameters necessary for executing the programs. The RAM 203 is used as a temporary storage area such as the main memory and work area of ​​the CPU 201. The mass storage device 204 is a mass storage secondary storage device such as an HDD or SSD that stores image data and various programs. The network I / F 205 is an interface for connecting to the Internet, and provides the above-mentioned network parameters in response to a request from a web browser of the edge device 100.

[0041] Although there are other components of the edge device 100 and the cloud server 200 than those described above, they are not the main focus of the present invention, and therefore the description will be omitted. In the present embodiment, it is assumed that the cloud server 200 generates learning data and performs degradation restoration learning, and the learned model, which is the learning result, is downloaded to the edge device 100, and degradation restoration inference is performed on the RAW image data. However, the above system configuration is an example, and is not limited thereto. For example, the functions of the cloud server 200 may be subdivided, and the generation of learning data and the degradation restoration learning may be performed by separate devices. Furthermore, the imaging device 10, which has both the functions of the edge device 100 and the cloud server 200, may be configured to perform all of the generation of learning data, degradation restoration learning, and degradation restoration inference.

[0042] <Processing flow of the entire system> Next, functional blocks and a processing flow of the image processing system according to this embodiment will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is a functional block diagram of the image processing system according to the first embodiment. Fig. 4 is a flowchart showing a processing flow in the image processing system according to the first embodiment. As shown in Fig. 3, the cloud server 200 has an architecture information acquisition unit 301, a multiplexing information acquisition unit 302, a degraded image acquisition unit 303, an image quality improvement unit 304, a peer data set selection unit 305, a statistics calculation unit 306, an error calculation unit 307, and a weight parameter update unit 308. The multiplexing information acquisition unit 302 and the peer data set selection unit 305 are examples of a selection means. Cloud server 200 reads programs and parameters stored in ROM 202, mass storage device 204, etc., and expands them in RAM 203, thereby realizing the functions of architecture information acquisition unit 301, multiplexing information acquisition unit 302, degraded image acquisition unit 303, image quality improvement unit 304, peer data set selection unit 305, statistics calculation unit 306, error calculation unit 307, and weight parameter update unit 308. Note that cloud server 200 may read programs and parameters from other servers connected to the Internet.

[0043] The edge device 100 has an image quality improvement inference unit 309. The edge device 100 may realize the function of the image quality improvement inference unit 309 by reading programs and parameters stored in the ROM 103, the mass storage device 104, the external storage device 30, etc., and expanding them in the RAM 102.

[0044] The configuration shown in Fig. 3 can be modified or changed as appropriate. For example, one functional unit may be divided into multiple functional units, or two or more functional units may be integrated into one functional unit. The cloud server 200 and edge device 100 shown in Fig. 2 may be realized by two or more devices. In this case, the devices are connected via a circuit or a wired or wireless network, and perform data communication with each other to perform cooperative operations, thereby realizing each process described below.

[0045] In step S401, the architecture information acquisition unit 301 reads architecture information, which is information for constructing an architecture, from a file on the mass storage device 104 of the edge device 100. The architecture information describes information such as layer index numbers, layer connection relationships, output destinations of output tensors that are inference results, and hyperparameters of each layer. Note that the architecture information may be read from the ROM 202, the mass storage device 204, the ROM 103 of the edge device 100, the external storage device 30, and the input device 20.

[0046] FIG. 9 is a diagram for explaining an example of architecture information. Information such as the type of operation 900 performed in the layer, the ID 901 of the layer, and the hyperparameters of the layer are listed for each layer. In addition, since each layer generally outputs data in tensor format (in the case of a convolutional layer, a four-dimensional tensor in the vertical, horizontal, channel, and batch directions), the ID 902 of the output destination layer is also described. The calculated tensor may be output to a loss function calculation process instead of to another layer. By referring to the above information, the connection relationship for each layer can be known, so that a neural network as shown in FIG. 10 can be constructed. Note that the architecture information is not limited to the format shown in FIG. 9, and may include information on the connection relationship between layers and outputs.

[0047] In step S402, the multiplexing information acquisition unit 302 acquires layer multiplexing information from the architecture information. Layer multiplexing is a structure in which multiple inference results are output for a single weight parameter in a certain layer. An example of this will be described with reference to FIG. 8.

[0048] As shown in FIG. 8(a), a layer 801, which is a representative convolutional layer in a non-multiplexed neural network, includes three elements. The first element is "receive input 802." The second element is "apply operation 803 based on given weight parameters." The third element is "calculate output 804." The factors of multiplexing are broadly divided into three for each of these elements. Below, we will explain the factors of multiplexing and a method of identifying multiplexed layers from architecture information, in the order of the third element, the second element, and the first element.

[0049] As in layer 805 shown in FIG. 8(b), in the third element, if the layer has a branched output structure, the layer is determined to be multiplexed. In the architecture information, a layer with multiple output destination layer IDs 903 is determined to be multiplexed. As shown in FIG. 5, each branch output tends to have a structure that outputs similar images, such as images at adjacent times. Therefore, by focusing on a branch structure in which multiple output data (= inference results) are output for a single input data (hereinafter, input image), it is possible to suppress discrepancies in image quality between similar images.

[0050] In the second element, a layer in which a single weight parameter is applied to multiple input images is determined to be multiplexed, as in layer 806 shown in FIG. 8(c). Specifically, a layer having a description 904 in which weights are shared with another layer in the architecture information is determined to be multiplexed. When weights are shared, calculations based on the same weight parameters are performed in multiple layers. Each input image group to which the same calculation can be applied is considered to be images or feature maps with similar attributes such as subject, image quality, and shooting time, and therefore each output tensor is considered to follow a similar distribution. In this way, by focusing on the relationship in which a calculation based on a single weight parameter is applied to multiple input images, it is possible to suppress the appearance from fluctuating between images with similar attributes.

[0051] As shown in FIG. 8(d) for layer 807, the first element determines that a layer in which an input image is input multiple times to one layer is multiplexed. Specifically, a layer having its own ID as the output destination ID 905 in the architecture information, which indicates that data is input again to a location where data has already been input, is determined to be multiplexed. When data is input multiple times to the same layer, it is considered that the input images are images or feature maps with similar subjects, image quality, and shooting times, and therefore the output tensors are also considered to follow similar distributions. Such a recursive input architecture is suitable for repeatedly inputting data at each time point. For this reason, by focusing on a structure in which an input image is input multiple times to a single layer, it is possible to suppress the difference in image quality between times.

[0052] The information on the multiplexed layers obtained above is called multiplexing information, and the multiplexing information is output to the same-level data set selection unit 305. Here, the multiplexing information is the ID of the multiplexed layer, which in the example of FIG. 10 is layer 2, layer 1, layer 7, and layer 6. Note that layer 2 in FIG. 10 corresponds to FIG. 8(b). Layer 1 and layer 7 in FIG. 10 correspond to FIG. 8(c). Layer 6 in FIG. 10 corresponds to FIG. 8(d).

[0053] Note that multiplexing is not limited to the first, second, and third elements described above, and any layer in which high image quality processing calculations are performed in which a single weight parameter is applied to multiple input images is determined to be a multiplexed layer.

[0054] In step S403, the degraded image acquisition unit 303 acquires a degraded image which is an image with noise superimposed thereon. The degraded image acquisition unit 303 acquires the degraded image by capturing the image with the imaging device 10, reading from the external storage device 30, or downloading via a network I / F. Note that the degraded image acquisition unit 303 may acquire a degraded image by adding noise after acquiring an image that does not contain noise.

[0055] In step S404, the image quality improving unit 304 inputs the degraded image to a CNN having a given weight parameter, and obtains an inference result. In the CNN of Non-Patent Document 1, the inference result is a residual map r1 in Step 1 and a residual map r2 in Step 2, and a high-image-quality image is obtained by adding the sum of the residual maps (r1+r2) to the degraded image. Note that, although an example in which the inference result is a residual map is shown in this embodiment, a network in which the inference result is the high-image-quality image itself may also be used.

[0056] In step S405, the same-level data set selection unit 305 sets the inference results that are considered to have similar distributions as same-level data sets based on the multiplexing information. For example, in the neural network of FIG. 10, the multiplexed output tensors 2-1, 2-2, and 2-3 of layer 2 are considered to ideally have similar distributions. Therefore, the same-level data set selection unit 305 extracts the three output tensors 2-1, 2-2, and 2-3 of layer 2 from the inference results shown in FIG. 10 and sets them as one set. This extracted set is hereinafter referred to as a same-level data set. The same-level data set selection unit 305 extracts and sets the inference result as one same-level data set for each instance of the multiplexed layer. Similarly, the same-level data set selection unit 305 extracts tensor 1 and tensor 7 as same-level data sets for the multiplexed layers 1 and 7. In addition, the same-level data set selection unit 305 extracts the output tensor 6-1 and the output tensor 6-2 as the same-level data set for the multiplexed layer 6. In this case, the targets to be included in the same-level data set may be some of the output tensors, not all of the output tensors of the multiplexed layer. For example, if it can be read from the architecture information that a small number of normal images are present in a group representing a residual map, the same-level data set selection unit 305 can prevent unauthorized error calculation by excluding the small number of normal images from the group.

[0057] In addition, the same-level data set selection unit 305 may extract the results of applying approximately identical processing to the output tensors of the multiplexed layers as the same-level data set. For example, tensor 9-1 and tensor 9-2 are outputs obtained by applying the same convolution processing (processing in layer 9) to the output tensor 6-1 and output tensor 6-2 of the multiplexed layer 6. If the output tensor 6-1 and the output tensor 6-2 are data similar in subject, shooting conditions, and shooting time, the output tensor 9-1 and the output tensor 9-2 are also highly similar. Therefore, the same-level data set selection unit 305 may determine the output tensor 9-1 and the output tensor 9-2 as the same-level data set. In this way, the layer 9 can be trained to maintain the similarity of the input and output data. The same-level data set selection unit 305 outputs the set same-level data set to the statistics calculation unit 306.

[0058] In step S406, the statistics calculation unit 306 calculates statistics of each output tensor included in the same-level data set. It is considered that the distribution of each channel of each tensor included in the same-level data set is consistent for each batch. Therefore, the statistics calculation unit 306 performs an operation of extracting a feature map of the c-th channel in the b-th batch for each output tensor and calculating the statistics of the feature map for all (b, c). The statistics is the n-th moment (n=1, 2, 3, 4) around the center. Note that the statistics are not limited to this, and the statistics calculation unit 306 may calculate values ​​such as the median, mode, and histogram as the statistics.

[0059] In step S407, the error calculation unit 307 calculates the error from the obtained statistics. stat is the sum of the absolute values ​​of the differences in the statistics for each channel direction in each batch, and is calculated as follows. stat The larger the difference in statistics, the larger the

[0060]

number

[0061] Where X i b,c,n is the nth moment around the center of the feature map of the cth channel of the bth batch of the i-th tensor in the peer dataset. When i=all, X i b,c,n is an example of a standard value of a statistic, and represents the average for all i. Note that the error L stat The method of finding is not limited to the above formula, and any formula that can evaluate the magnitude of deviation between statistics may be used.

[0062] Next, the error calculation unit 307 calculates a reconstruction error L, which represents the magnitude of the difference between the improved image and the correct image. recon Calculate L stat The weighted sum of these is calculated and the loss value is L. The calculation formula is: recon and weight λ stat It is expressed as follows using

[0063]

number

[0064] The error included in the loss value L is the reconstruction error L recon However, the error calculation unit 307 may include, for example, an adversarial loss that indicates the deviation of the image quality improvement result from a natural image, an error in image features, or the like in the loss value L. In this way, the error calculation unit 307 can eliminate unevenness in the distribution of each channel and suppress color shifts in the output image by calculating the difference for each channel as an error. Furthermore, the error calculation unit 307 can speed up the progress of learning by setting the loss value L to be larger as the error in the statistics becomes larger.

[0065] In step S408, the weight parameter update unit 308 performs error backpropagation to reduce the loss value L calculated by the error calculation unit 307, that is, to reduce the error. The weight parameters of the CNN are updated based on Equation 4 by the error backpropagation. According to this equation, the larger the error, the larger the update amount of the weight parameters. Then, using the updated weight parameters, the generation of learning data and image quality improvement from step S402 onwards are performed again. When this learning loop is completed a given number of times, or when a given performance is reached through learning, the obtained weight parameters are output to the image quality improvement inference unit 309.

[0066] In step S409, the image quality improvement inference unit 309 applies image quality improvement to an unknown input image including noise based on the learning results up to step S408. The image quality improvement inference unit 309 uses the weighting parameters obtained by the weighting parameter update unit 308, inputs the acquired input image to the CNN to improve the image quality, and outputs an output image with improved image quality.

[0067] The above is the overall flow of the processing carried out in the image processing system of this embodiment.

[0068] In the present embodiment, the weight parameter update unit 308 treats the error calculated by the error calculation unit 307 as a loss function value and uses it for error backpropagation. However, the method of using the error as an error-based weight parameter update is not limited to this. For example, the weight parameter update unit 308 may treat the smallness of the error as a performance evaluation value and use the error to determine that learning of the weight parameters is to be terminated when the performance evaluation value becomes equal to or greater than a given threshold. This allows the image processing system to more appropriately determine the timing of the end of learning and reduce artifacts in the output image. In addition, the cloud server 200 may create multiple models with different learning data and learning times, and use the error to select the best model by comparing the performance evaluation values ​​of the weight parameters. This makes it possible to select a model with fewer artifacts such as color shifts and residual noise.

[0069] In the present embodiment, the multiplexing information acquisition unit 302 and the same-level data set selection unit 305 automatically acquire the multiplexing information and the same-level data set from the architecture information, but the acquisition method is not limited to this. For example, the multiplexing information acquisition unit 302 and the same-level data set selection unit 305 may receive the multiplexing information and the same-level data set by reading them from a file or inputting them from a user.

[0070] Furthermore, in the present embodiment, an example has been shown in which the error is calculated from the statistics of the residual map, but the target of the error calculation may be another product of the neural network, such as a feature map or a feature vector.

[0071] In this embodiment, multiple inference results output by a single parameter are set as the inference results of the multiplexed layer, and the multiple similar inference results are set as a peer data set. Then, in this embodiment, the error in the statistics of the inference result (output tensor) is calculated for each peer data set, and the parameters are updated based on the error, thereby suppressing the deviation in the statistics of each inference result and suppressing the difference between inference results that should be similar. As a result, in this embodiment, artifacts that cause image quality to vary from frame to frame, such as color shifts and changes in smoothness of images after image quality improvement processing, can be reduced.

[0072] In this embodiment, since the peer data sets are set based on architecture information that indicates the connection relationships between multiple layers of a neural network, similar inference results can be easily and reliably set as peer data sets.

[0073] In this embodiment, a peer data set is set based on multiplexing information generated by determining the multiplexing of layers based on architecture information, so that similar inference results of multiplexed layers can be more easily and reliably set as peer data sets.

[0074] In this embodiment, since it is determined that a layer that outputs multiple inference results from one input image is multiplexed, similar inference results of the layer can be set more reliably as peer data sets.

[0075] In this embodiment, since multiple layers to which a single weight parameter is applied are determined to be multiplexed, similar inference results of the layer can be more reliably set as peer data sets.

[0076] In this embodiment, since a layer to which an input image is input multiple times is determined to be multiplexed, similar inference results of the layer can be set more reliably as peer data sets.

[0077] In this embodiment, since the inference result is set as a peer data set for each instance of a multiplexed layer, it is possible to set similar inference results for each instance as a peer data set.

[0078] In this embodiment, the error is calculated based on the difference between the statistics of similar inference results and the average value, which is the reference value of the statistics. Furthermore, in this embodiment, the error is calculated to be larger as the difference between the statistics of the inference results is larger, so that the loss value can be increased to speed up the progress of learning.

[0079] In this embodiment, by calculating an error based on the difference for each channel, it is possible to improve the color shift of an output image after image quality improvement processing.

[0080] In this embodiment, the weight parameters are updated so as to reduce the loss value, i.e., to reduce the error in the statistics of similar inference results, thereby making it possible to further reduce artifacts.

[0081] <Second embodiment> In the first embodiment, a case where noise removal is performed as an example of image quality improvement is taken as an example, and a mode of reducing artifacts in an output image with improved image quality is described, but the application of the present invention is not limited to noise removal. In the present embodiment, an example of applying the present invention to other image quality improvement tasks such as super-resolution, deblur, fog removal, rain removal, inpainting, and HDR (High Dynamic Range) will be described.

[0082] Note that a description of the contents common to the first embodiment, such as the basic configuration of the image processing system, will be omitted, and the following description will focus on the differences.

[0083] In step S403, the degraded image acquisition unit 303 acquires a degraded image specific to the task. For example, when performing super-resolution, which is a task for improving the resolution of an image, a reduction process that reduces the number of pixels of the image is applied to the image to generate a degraded image. When performing deblur, which is a task for removing moving object blur from an image, a smoothing process that adds moving object blur to the image is performed to acquire a degraded image. Note that the method of acquiring a degraded image is not limited to those based on artificial image processing, and for example, a degraded image may be obtained by changing the focal length of the lens in the imaging device 10 and photographing a subject at a low resolution, or a degraded image with moving object blur may be obtained by photographing while moving the imaging device 10.

[0084] In steps S403 to S408, learning is performed based on the obtained degraded image to obtain weighting parameters.

[0085] In step S409, the image quality improvement inference unit 309 acquires an input image containing the same degradation as in step S403. Then, the input image is input to the CNN to improve the image quality, and an output image is output as a result of the improved image quality.

[0086] According to this embodiment, even in image quality improvement tasks other than noise removal, it is possible to suppress the deviation of the statistics of each output tensor of multiple layers. As a result, it is possible to reduce artifacts that cause image quality to vary from frame to frame, such as color shifts and changes in sharpness.

[0087] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0088] The disclosure of this specification includes the following image processing device, image processing method, imaging device, and program. (Item 1) In an image processing device that performs image quality improvement processing using a neural network having multiple layers, an image quality improvement means for applying the image quality improvement process using the weighting parameters to an input image and outputting an inference result; A selection means for selecting a plurality of inference results calculated using a single weight parameter as a peer data set; A statistics calculation means for calculating statistics of the inference result of the peer data set; an error calculation means for calculating an error of the statistical quantity for each of the peer data sets based on the statistical quantity and a reference value of the statistical quantity; updating means for updating the weight parameters based on the error; An image processing device comprising: (Item 2) The selection means sets the peer data set based on architecture information indicating a connection relationship between the plurality of layers of the neural network. 2. The image processing device according to item 1, (Item 3) The selection means determines layer multiplexing based on the architecture information to generate multiplexing information, and sets the peer data set based on the multiplexing information. 3. The image processing device according to item 2, (Item 4) The selection means determines that the layer that outputs the plurality of inference results from one of the input images is multiplexed and generates the multiplexing information. 4. The image processing device according to item 3, (Item 5) The selection means generates the multiplexing information by determining that a single weight parameter is applied to a plurality of layers. 5. The image processing device according to item 3 or 4, (Item 6) The selection means determines that the layer in which the input image is input multiple times is multiplexed, and generates the multiplexing information. 6. The image processing device according to any one of items 3 to 5, (Item 7) The selection means applies the same operation to the plurality of inference results output from the layer determined to be multiplexed, and sets the plurality of inference results output as the same-level data set. 7. The image processing device according to any one of items 3 to 6, (Item 8) The selection means sets the inference result as the peer data set for each of the multiplexed instances of the layer. 8. The image processing device according to any one of items 3 to 7, (Item 9) The error calculation means calculates the error based on the difference between the statistics. 9. The image processing device according to any one of items 1 to 8, (Item 10) The error calculation means calculates the error so that the error is larger as the difference between the statistics is larger. 10. The image processing device according to item 9, (Item 11) The error calculation means calculates the error based on a difference in the statistics for each channel of the inference result. 11. The image processing device according to claim 1, wherein: (Item 12) The updating means updates the weighting parameters so as to reduce the error. 12. The image processing device according to claim 1, wherein: (Item 13) The updating means evaluates the performance of the neural network based on the error and determines the end of updating the weight parameters. 13. The image processing device according to any one of items 1 to 12, (Item 14) The image quality improving means outputs an image as the inference result. 14. The image processing device according to claim 1, wherein the image processing device comprises: (Item 15) 15. The image processing device according to claim 1, wherein the image quality improving means outputs an image indicating a degraded component of the input image as the inference result. (Item 16) an imaging means for imaging a subject and generating the input image; An image processing device according to any one of items 1 to 15, An imaging device comprising: (Item 17) In an image processing method using a neural network having multiple layers for image quality improvement, an image quality improvement step of applying the image quality improvement process using the weighting parameters to an input image and outputting an inference result; A selection step of selecting a plurality of inference results calculated using a single weight parameter as a peer data set; A statistics calculation step of calculating statistics of the inference results of the peer data set; an error calculation step of calculating an error of the statistical quantity for each of the peer data sets based on the statistical quantity and a reference value of the statistical quantity; updating the weight parameters based on the error; An image processing method comprising: (Item 18) A program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 15.

[0089] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0090] 501...CNN, 502...CNN, 200...cloud server, 301...architecture information acquisition unit, 302...multiplexing information acquisition unit, 303...degraded image acquisition unit, 304...image quality improvement unit, 305...peer data set selection unit, 306...statistics calculation unit, 307...error calculation unit, 308...weight parameter update unit, 309...image quality improvement inference unit.

Claims

1. In an image processing device that performs image quality improvement processing using a neural network having multiple layers, an image quality improvement means for applying the image quality improvement process using the weighting parameters to an input image and outputting an inference result; A selection means for selecting a plurality of inference results calculated using a single weight parameter as a peer data set; A statistics calculation means for calculating statistics of the inference result of the peer data set; an error calculation means for calculating an error of the statistical quantity for each of the peer data sets based on the statistical quantity and a reference value of the statistical quantity; updating means for updating the weight parameters based on the error; An image processing device comprising:

2. The selection means sets the peer data set based on architecture information indicating a connection relationship between the plurality of layers of the neural network.

2. The image processing device according to claim 1,

3. The selection means determines layer multiplexing based on the architecture information to generate multiplexing information, and sets the peer data set based on the multiplexing information.

3. The image processing device according to claim 2.

4. The selection means determines that the layer that outputs the plurality of inference results from one of the input images is multiplexed and generates the multiplexing information.

4. The image processing device according to claim 3.

5. The selection means generates the multiplexing information by determining that a single weight parameter is applied to a plurality of layers.

4. The image processing device according to claim 3.

6. The selection means determines that the layer in which the input image is input multiple times is multiplexed, and generates the multiplexing information.

4. The image processing device according to claim 3.

7. The selection means applies the same operation to the plurality of inference results output from the layer determined to be multiplexed, and sets the plurality of inference results output as the same-level data set.

4. The image processing device according to claim 3.

8. The selection means sets the inference result as the peer data set for each of the multiplexed instances of the layer.

4. The image processing device according to claim 3.

9. The error calculation means calculates the error based on the difference between the statistics.

2. The image processing device according to claim 1,

10. The error calculation means calculates the error so that the error is larger as the difference between the statistics is larger.

10. The image processing device according to claim 9,

11. The error calculation means calculates the error based on a difference in the statistics for each channel of the inference result.

2. The image processing device according to claim 1,

12. The updating means updates the weighting parameters so as to reduce the error.

2. The image processing device according to claim 1,

13. The updating means evaluates the performance of the neural network based on the error and determines the end of updating the weight parameters.

2. The image processing device according to claim 1,

14. The image quality improving means outputs an image as the inference result.

2. The image processing device according to claim 1,

15. 2. The image processing apparatus according to claim 1, wherein the image quality improving means outputs, as the inference result, an image showing a degraded component of the input image.

16. an imaging means for imaging a subject and generating the input image; The image processing device according to claim 1 ; An imaging device comprising:

17. In an image processing method using a neural network having multiple layers for image quality improvement, an image quality improvement step of applying the image quality improvement process using the weighting parameters to an input image and outputting an inference result; A selection step of selecting a plurality of inference results calculated using a single weight parameter as a peer data set; A statistics calculation step of calculating statistics of the inference results of the peer data set; an error calculation step of calculating an error of the statistical quantity for each of the peer data sets based on the statistical quantity and a reference value of the statistical quantity; updating the weight parameters based on the error; An image processing method comprising:

18. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 15.

Images