Image processing apparatus, image processing method, and program

Abstract

To reduce the cost of acquiring weighting parameters for each imaging apparatus.SOLUTION: An image processing apparatus 150 according to the present disclosure performs learning of a neural network to generate data of a restored image in which image degradation included in an input image is reduced. The image processing apparatus 150 is configured to: determine a weight parameter corresponding to a first type among a plurality of weight parameters in the neural network; and determine a weight parameter, corresponding to a second type different from the first type among the plurality of weight parameters, which is more susceptible to the influence of differences in models of apparatuses that generate the data of the input image or individual differences of the apparatus than the weight parameters corresponding to the first type.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a technique for generating an image with reduced degradation from an image with degraded image quality. [Background technology]

[0002] There is a technology that uses a deep neural network (hereinafter referred to as "DNN") to restore images with degraded image quality (hereinafter referred to as "degraded image"). A DNN is a neural network with two or more hidden layers, and performance can be improved by increasing the number of hidden layers. Image restoration (hereinafter referred to as "image restoration") refers to the process of generating an image with reduced degradation from a degraded image, such as noise removal, blur removal, super-resolution, or missing parts completion. In image restoration processing using a DNN, weight parameters suitable for image restoration are obtained by learning using data of an image that includes degradation (degraded image), and an image with reduced degradation (hereinafter referred to as "restored image") is generated using the obtained weight parameters.

[0003] Such image restoration processing is applied to data of an image (hereinafter referred to as a "captured image") obtained by capturing an image using an imaging device such as a digital camera. However, the captured image is subject to image degradation (hereinafter referred to as "image degradation") whose intensity varies depending on the model or individual imaging device. Therefore, it is necessary to perform image restoration processing according to the characteristics of the image degradation of each captured image. Patent Document 1 discloses a technique for adjusting the intensity of image restoration for each imaging condition, in which a feature map representing the intensity of image restoration is generated inside a DNN, and weight parameters for adjusting the value of the feature map are changed when a restored image is generated. According to the technique disclosed in Patent Document 1, the intensity of image restoration can be adjusted for each captured image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2023-47600 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the method disclosed in Patent Document 1 does not describe a method for changing weight parameters for each imaging device, and weight parameters for adjusting the value of the feature map are assumed to be given. It is difficult to manually adjust and provide these weight parameters for each imaging device. If an inappropriate value is provided, residual noise due to uncorrected noise or blur due to overcorrection may occur in the restored image generated as a result of the image restoration process, and an appropriate restored image may not be obtained. In addition, it is possible to obtain weight parameters corresponding to each imaging device by acquiring captured image data for each imaging device and learning DNN using the data, but such a method results in an enormous cost of acquiring weight parameters.

[0006] An object of the present disclosure is to provide a technique for reducing the cost of acquiring weighting parameters for each imaging device. [Means for solving the problem]

[0007] The image processing device according to the present disclosure is an image processing device that trains a neural network to generate restored image data that reduces image degradation contained in an input image, and has a first determination means that determines a weight parameter corresponding to a first type among a plurality of weight parameters in the neural network, and a second determination means that determines a weight parameter corresponding to a second type among the plurality of weight parameters that is different from the first type and that corresponds to the second type, the second type being more affected by differences in model of a device that generates the input image data or individual differences in the device, than the weight parameter corresponding to the first type. Effect of the Invention

[0008] The cost of acquiring weighting parameters for each imaging device can be reduced. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of a configuration of an image processing system according to a first embodiment. [Diagram 2] 1 is a block diagram showing an example of a functional configuration of an image processing device according to a first embodiment. [Diagram 3] 4 is a flowchart showing an example of a processing flow of the image processing device 150 according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a profile 401 indicating the relationship between a second weighting parameter and an evaluation value of image quality according to the first embodiment. [Diagram 5] FIG. 11 is a block diagram showing an example of a functional configuration of an image processing device according to a second embodiment. [Figure 6] 11 is a flowchart showing an example of a processing flow of an image processing device 150 according to the second embodiment. [Figure 7] FIG. 11 is a diagram showing an example of a user interface for selecting a second weighting parameter according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present disclosure is not limited to the configurations shown in the drawings.

[0011] [Embodiment 1] In this embodiment, a method of classifying weight parameters of a DNN into a first weight parameter that is less influenced by the model and individual of an imaging device, and a second weight parameter that is more influenced by the model or individual of the imaging device will be described. In addition, in this embodiment, a method of determining a first weight parameter using image data corresponding to a reference imaging device, and determining a second weight parameter using image data corresponding to each model or individual different from the reference imaging device will be described. In addition, in this embodiment, a method of generating an appropriate restored image by using the determined first weight parameter and second weight parameter will be described. Note that in this embodiment, a case of performing a noise removal process to reduce noise contained in an image will be described as an example of an image restoration process.

[0012] <Image processing system configuration> 1 is a block diagram showing an example of the configuration of an image processing system 1 according to the first embodiment. The image processing system 1 includes an information processing device 100, an imaging device 11, an input device 12, an external storage device 13, a display device 14, and an image processing device 150, and the information processing device 100 and the image processing device 150 are connected to each other via the Internet so that they can communicate with each other. The image processing device 150 generates learning data, estimates image quality degradation, and learns image restoration processing. The information processing device 100 performs inference processing on the restored image and generates the restored image.

[0013] <Hardware configuration of information processing device> The information processing device 100 is configured with a personal computer or the like, acquires captured image data output by the imaging device 11, and generates a restored image corresponding to the captured image. Here, the captured image data is, for example, RAW image data in a Bayer array or the like. Specifically, the information processing device 100 applies weight parameters of a trained neural network provided by the image processing device 150 to the captured image data, and generates a restored image by performing an inference process for degradation restoration. A user can generate data of a restored image corresponding to the captured image data by using image processing application software installed in the information processing device 100.

[0014] The information processing device 100 has a CPU 101, a RAM 102, a ROM 103, a large-capacity storage device 104, a general-purpose I / F 105, and a network I / F 106, and each component is connected to each other so as to be able to communicate with each other via a system bus 107. The information processing device 100 is also connected to an imaging device 11, an input device 12, an external storage device 13, and a display device 14 so as to be able to communicate with each other via the general-purpose I / F 105.

[0015] The CPU 101 uses the RAM 102 as a work memory, executes programs stored in the ROM 103, and performs overall control of each component of the information processing device 100 via a system bus 107. The mass storage device 104 is configured, for example, by a hard disk drive or a solid state drive, and stores various data and the like handled in the information processing device 100. The CPU 101 writes data to the mass storage device 104 and reads data stored in the mass storage device 104 via the system bus 107. The general-purpose I / F 105 is a serial bus interface, for example, a Universal Serial Bus (USB), IEEE1394, or a High-Definition Multimedia Interface (HDMI (registered trademark)).

[0016] The information processing device 100 acquires data from an external storage device 13 constituted by various storage media such as a memory card, a CF card, an SD card, or a USB memory via a general-purpose I / F 105. The information processing device 100 also accepts user instructions from an input device 12 such as a mouse or a keyboard via the general-purpose I / F 105. The information processing device 100 also outputs image data processed by the CPU 101 to a display device 14 constituted by various image display devices such as a liquid crystal display via the general-purpose I / F 105. The information processing device 100 also acquires captured image data to be subjected to inference processing of degradation restoration from the imaging device 11 via the general-purpose I / F 105. The network I / F 106 is an interface for connecting to the Internet. The information processing device 100 accesses the image processing device 150 via an installed web browser or the like, and acquires weight parameters of a trained neural network for inference processing of degradation restoration.

[0017] <Hardware configuration of image processing device> The image processing device 150 is configured with a server or the like. The image processing device 150 may be configured with a cloud server in which a plurality of servers work together. The image processing device 150 has a CPU 151, a ROM 152, a RAM 153, a large-capacity storage device 154, and a network I / F 155, and each component is connected to be able to communicate with each other via a system bus 156. The CPU 151 executes a program stored in the ROM 152 using the RAM 153 as a work memory, and comprehensively controls each component of the image processing device 150 via the system bus 156.

[0018] The mass storage device 154 is configured by, for example, a hard disk drive or the like, and stores various data handled in the image processing device 150. The CPU 151 writes data to the mass storage device 154 and reads data stored in the mass storage device 154 via the system bus 107. The network I / F 155 is an interface for connecting to the Internet. In response to a request from the image processing device 150, the image processing device 150 outputs to the image processing device 150 weight parameters of a trained neural network for inference processing of degradation restoration.

[0019] <CNNについて> A neural network according to the present embodiment will be described. The neural network according to the present embodiment will be described as being configured by a convolutional neural network (hereinafter, referred to as "CNN" (Convolutional Neural Network)) as an example. CNN is a neural network used in general image processing technology that applies deep learning. CNN convolves a filter generated as a result of learning with image data, and then performs nonlinear arithmetic processing, and repeats this convolution processing and nonlinear arithmetic processing. The filter is also called a local receptive field. An image obtained by convolving the filter with image data and then performing nonlinear arithmetic processing is also called a feature map. CNN learning is performed using learning data consisting of a set of input image data and correct image data as output image data (hereinafter, referred to as "teacher image data"). Simply put, CNN learning is to generate, from learning data, filter values ​​that can be converted from input image data to output image data corresponding to the input image data with high accuracy. Details of CNN learning will be described later.

[0020] When the input image data has multiple color channels such as R (Red), G (Green), and B (Blue), or when the feature map is composed of multiple images, the filter used in the convolution process also has multiple channels according to the number of color channels. That is, the filter used in the convolution process is expressed using a four-dimensional array that includes the vertical and horizontal sizes of the image, the number of feature maps, and the number of channels. The nonlinear arithmetic process after convolution of the image data including the feature map is expressed in units of layers, such as the nth layer feature map (n is an integer equal to or greater than 1) and the nth layer filter. For example, a CNN that repeats the convolution process and the nonlinear arithmetic process by the filter three times has a three-layer network structure. Such nonlinear arithmetic process can be formulated, for example, using the following equation (1). TIFF2025022143000002.tif7150

[0021] Here, W n is the nth layer filter, b n is the bias of the nth layer, f is the nonlinear operator, X n is the n-th layer feature map, and * is the convolution operator. The (l) on the right indicates that it is the l-th filter or feature map. The filters and biases are generated by learning, which will be described later, and are collectively called "network parameters." For nonlinear arithmetic processing, a sigmoid function or ReLU (Rectified Linear Unit), etc., are used. When ReLU is used as the nonlinear arithmetic processing, the nonlinear operator f is given, for example, by the following formula (2). TIFF2025022143000003.tif8150

[0022] As shown in Equation (2), for the elements of vector X input to the non-linear operator f, those that are negative output 0, and those that are 0 or greater output their original values. As neural networks using CNNs, ResNet in the field of image recognition and RED-Net which applies ResNet in the field of super-resolution are known. Any of these neural networks can improve the processing accuracy by stacking multiple layers of CNNs and performing convolution of filters multiple times. For example, ResNet features a network structure with a path that omits the processing of one or more convolutional layers. As a result, a multi-layer network of 152 layers can be realized, and highly accurate recognition approaching the human recognition rate can be achieved. The reason why the processing accuracy is improved by making the CNN a multi-layer network is simply that by repeating the non-linear operation process multiple times, the non-linear relationship between the input and output can be expressed.

[0023] <Learning of CNN> The learning of CNN will be described. The learning of CNN is generally performed by minimizing the objective variable of the loss function represented by the following Equation (3) for the training data consisting of a pair of input image (also referred to as "student image") data and teacher image data corresponding to the student image data. TIFF2025022143000004.tif7150

[0024] Here, L is a loss function that measures the error between the inference result of the CNN for the input and the correct answer corresponding to the input. Y i is the i-th teacher image data, and X i is the i-th input image data. F is a function that collectively represents the operations shown as an example in Equation (1) performed in each layer of the CNN. θ is the network parameter (filter and bias). ||Z||2 is the L2 norm, which is simply the square root of the sum of the squares of the elements of vector Z. n is the total number of training data used for the learning of the CNN. Generally, the total amount of training data is huge, so in the stochastic gradient descent method, a part of the training data is randomly selected, and training is performed using the selected training data. This can reduce the computational load in training using a huge amount of training data. In addition, various methods such as the momentum method, the AdaGrad method, the AdaDelta method, and the Adam method are known as methods for minimizing, i.e. optimizing, the objective variable in the loss function. For example, the Adam method is given using a series of equations shown in the following equations (4) to (7). TIFF2025022143000005.tif30170

[0025] Here, θ i t is the i-th network parameter at iteration t, and g is the i t is the gradient of the loss function L with respect to m and v are moment vectors, α is the basic learning rate, β1 and β2 are hyperparameters, and ε is a small constant. Note that since there are no guidelines for selecting an optimization method in learning, any method can be used in principle, but it is known that the convergence of each method differs, resulting in differences in learning time.

[0026] <Characteristics of neural networks used in image restoration processing> The neural network (hereinafter, referred to as "image restoration NN") used in the image restoration process disclosed in Patent Document 1 is composed of the following three blocks. The first is a block that extracts features from input image data, and in this block, a feature map indicating image restoration strength is generated. The value of each pixel in the feature map indicates the amount of noise to be removed at that pixel. The second is a block that determines the image restoration strength based on the features indicated by the feature map, and in this block, the feature map is modified so that the image restoration strength becomes a value specific to the imaging conditions, and in this embodiment, a value specific to the imaging device. The third is a block that reconstructs an image based on the features and image restoration strength indicated by the modified feature map, and outputs data of a restored image as a result of the image restoration process. In this embodiment, the block of the image restoration NN that extracts features from an input image is referred to as a feature extraction unit, and the block that determines the image restoration strength based on the features is referred to as a correction amount determination unit. In addition, the block that reconstructs an image based on the features and image restoration strength, and generates data of a restored image as a result of the image restoration process is referred to as an image reconstruction unit.

[0027] The weight parameters used in the calculation of each block include those that tend to be common among imaging devices and those that tend to differ among imaging devices. In this embodiment, the value indicating the influence of the model or individual of such weight parameters on the imaging device is defined as the device dependency and will be described. For example, the feature extraction unit calculates the feature amount by the same convolution process for the data of the input image corresponding to any imaging device, so the weight parameters in the feature extraction unit tend to be common among imaging devices. Similarly, the image reconstruction unit has a tendency to have the weight parameters in common among imaging devices because the image quality of the image to be reconstructed does not change significantly even if the imaging device that outputs the input image data input to the image restoration NN is changed. In contrast, the noise amount in the captured image obtained when the same scene is captured varies greatly among the model or individual of the imaging device, and the desired noise removal amount differs for each model or individual of the imaging device. Therefore, the weight parameters of the correction amount determination unit can be changed so that the correction strength increases for imaging devices with a larger noise amount. In other words, the weight parameters of the correction amount determination unit tend to have a larger device dependency than the weight parameters of the feature extraction unit and the image reconstruction unit.

[0028] Considering the above characteristics, the weight parameters are classified into those with low device dependency and those with high device dependency, the former are calculated as weight parameters common to all imaging devices, and then the latter weight parameters are calculated individually for each imaging device. This can reduce the learning cost required for each imaging device. Here, the process in the correction amount determination unit is a process of uniformly enlarging or reducing the feature map indicating the feature amount extracted by the feature extraction unit. The weight parameters of the correction amount determination unit are expressed using a scalar value called a one-dimensional magnification r as a parameter indicating the amplification degree of the value of each pixel of the feature map generated by the feature extraction unit. Note that the weight parameters of the correction amount determination unit are not limited to one-dimensional values, and may be multidimensional values. The correction amount determination unit may be configured by CNN, and in this case, the correction amount determination unit may correct the feature map generated by the feature extraction unit by convolution processing.

[0029] <Functional configuration of image processing device> The functional configuration of the image processing device 150 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the functional configuration of the image processing device 150 according to the first embodiment. The image processing device 150 has a parameter classification unit 211, a first weighting parameter determination unit 212, and a second weighting parameter determination unit 218. The first weighting parameter determination unit 212 has a first image acquisition unit 213, a first feature extraction unit 214, a first correction amount determination unit 215, a first image reconstruction unit 216, and a first weighting parameter update unit 217. The second weighting parameter determination unit 218 has a second image acquisition unit 219, a second feature extraction unit 220, a second correction amount determination unit 221, a second image reconstruction unit 222, an image quality evaluation unit 223, a variation value calculation unit 224, and a second weighting parameter update unit 225.

[0030] The functional configuration shown in Fig. 2 can be modified or changed as appropriate. For example, one functional unit may be divided into multiple functional units, or two or more functional units may be integrated into one functional unit. The configuration shown in Fig. 2 may be realized by two or more devices. In this case, the devices are communicatively connected to each other via a circuit or a wired or wireless network, and perform data communication with each other to operate in cooperation, thereby realizing each process described below.

[0031] In this embodiment, first, a reference imaging device is used to obtain first learning image data, which is learning data common to all imaging devices, and a first weighting parameter, which is a weighting parameter that is low in device dependency and common between imaging devices, is determined. Then, second learning image data, which is learning data for each model or individual imaging device, is obtained, and a second weighting parameter, which is a weighting parameter that is highly device dependent and likely to differ between imaging devices, is determined. Here, the second learning image data may include learning data common between imaging devices in addition to learning data specific to each imaging device.

[0032] <Processing flow of image processing device> The operation of the image processing device 150 will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of a processing flow of the image processing device 150 according to the first embodiment. The CPU 151 reads a program for executing the processing of the flowchart shown in Fig. 3 from the mass storage device 154 or the like, and executes the program to realize the functions of the above-mentioned respective units that the image processing device 150 has as its functional configuration. In the following description, the symbol "S" means a step (process).

[0033] First, in S301, the parameter classification unit 211 classifies the weight parameters of the image restoration NN into a first weight parameter and a second weight parameter based on the device dependency. Hereinafter, as an example, a form in which the device dependency is set to two values, large and small, and the weight parameters are classified into two will be described. For example, in S301, first, the parameter classification unit 211 performs learning of the image restoration NN using a captured image obtained by imaging with a first imaging device, and obtains a weight parameter corresponding to the first imaging device. Subsequently in S301, the parameter classification unit 211 performs learning of the image restoration NN using a captured image obtained by imaging with a second imaging device that is different in model or individual from the first imaging device, and obtains a weight parameter corresponding to the second imaging device. Note that the image restoration NN before learning using the captured image obtained by imaging with the first imaging device and the image restoration NN before learning using the captured image obtained by imaging with the second imaging device are the same.

[0034] After S301, the parameter classification unit 211 performs image restoration processing on the captured image obtained by imaging using the first imaging device, using the weight parameters corresponding to the first imaging device, and acquires data of the first restored image. After S301, the parameter classification unit 211 changes a part of the weight parameters corresponding to the first imaging device to weight parameters corresponding to the second imaging device. After S301, the parameter classification unit 211 performs image restoration processing on the captured image obtained by imaging using the first imaging device, using the changed weight parameters, and acquires data of the second restored image. After S301, the parameter classification unit 211 evaluates the image quality of the acquired first and second restored images using a predetermined image quality evaluation index such as SSIM (Structural Similarity) or PSNR (Peak Signal to Noise Ratio). After S301, the parameter classification unit 211 compares the evaluation value of the image quality with a predetermined reference value to determine the magnitude of the device dependency of the changed weight parameters.

[0035] The parameter classification unit 211 repeatedly executes the above-mentioned weight parameter change process, the process of evaluating the image quality of the restored image, and the process of determining the device dependency of the changed weight parameter for all weight parameters possessed by the image restoration NN. Hereinafter, the parameter classification unit 211 will be described as classifying the weight parameters of the feature extraction unit and the image reconstruction unit, which have a small device dependency, into first weight parameters, and classifying the weight parameters of the correction amount determination unit, which have a large device dependency, into second weight parameters. Note that, in the present embodiment, as an example, a form in which the weight parameters are classified into two weight parameters based on the device dependency has been shown, but the classification method by the parameter classification unit 211 is not limited to this. For example, the parameter classification unit 211 may obtain the device dependency for each layer of the image restoration NN and classify the weight parameters into an arbitrary number of weight parameters.

[0036] Specifically, in this case, for example, first, the parameter classification unit 211 performs learning of the image restoration NN using the captured image obtained by imaging with the first imaging device. Next, the parameter classification unit 211 fixes the weight parameters of the first imaging device except for one layer, and performs learning of the image restoration NN again using the captured image obtained by imaging with the second imaging device. Next, the parameter classification unit 211 evaluates the image quality of the restored image output by the image restoration NN that has been trained using the captured image obtained by imaging with the second imaging device. Next, the parameter classification unit 211 determines the device dependency of the weight parameters of the layers that are not fixed in the image restoration NN based on the evaluation result of the image quality of the restored image. Specifically, for example, the parameter classification unit 211 determines that the greater the fluctuation of the image quality evaluation index in the evaluation result of the image quality, the greater the device dependency of the weight parameters of the layers that are not fixed. The parameter classification unit 211 repeatedly executes the above-mentioned layer fixing process, image restoration NN learning process, image quality evaluation process of the restored image, and device dependency determination process of the weight parameters of the layers that are not fixed for all layers that the image restoration NN has.

[0037] After S301, in S302, the first image acquisition unit 213 acquires data of a first image without image degradation as data of a learning image for determining a first weighting parameter. Hereinafter, the first image data acquired by the first image acquisition unit 213 will be described as data of an image in RGB format. Next, in S303, the first image acquisition unit 213 generates data of a first degraded image 226 to which image degradation that can be expected in a reference imaging device is added to the first image, and data of a first correct answer image 233 to which no image degradation is added. The data of the first degraded image 226 generated in S303 is input to the image restoration NN, and the data of the first correct answer image 233 is input to the first weighting parameter update unit 217.

[0038] Here, the imparting of image degradation means imparting degradation to an image to be restored in an image restoration process. For example, when the image restoration process is a process for removing noise contained in an image, the first image acquisition unit 213 generates data of a first degraded image 226 in which noise has been imparted to the first image by adding a random number that follows a normal distribution whose variance depends on luminance to each pixel value of the first image. The value of this variance is determined by a quadratic expression related to luminance, and the coefficient of the quadratic expression is a value specific to the imaging device.

[0039] The first image acquisition unit 213 may impart the following to the first image as image degradation other than noise. For example, when the image restoration process is a super-resolution process, the first image acquisition unit 213 imparts image degradation by reducing the first image. Also, for example, when the image restoration process is an inpainting process, the first image acquisition unit 213 imparts image degradation by masking a part of the first image by a mask process on the first image. Also, the first image acquisition unit 213 may generate data of the first deteriorated image 226 by imparting multiple types of image degradation. Note that multiple reference imaging devices are prepared, and in S303, the first image acquisition unit 213 may generate multiple pieces of data of the first deteriorated image 226 to which image degradation corresponding to each imaging device is imparted. Also, the first image acquisition unit 213 may acquire data of a captured image obtained by performing imaging at a high gain in the reference imaging device as data of the first deteriorated image 226 including noise.

[0040] After S303, in S304, the first feature extraction unit 214 extracts features using the input data of the first degraded image 226 and the current first weighting parameter 227, and generates a first feature map 228. The first feature extraction unit 214 here corresponds to a block from which features are extracted from an input image in the image restoration NN. Next, in S305, the first correction amount determination unit 215 determines the strength of the image restoration process based on the features using the first feature map 228 and the current second weighting parameter 229, and corrects the first feature map 228 based on the determined strength of the image restoration process. Through this correction, a first corrected feature map 230 corresponding to the first feature map 228 is generated. The second weighting parameter 229 is adjusted so that the magnification r indicating the weighting parameter of the correction amount determination unit is fixed to 1, and the first corrected feature map 230 and the first feature map 228 are identical. Moreover, the first correction amount determination unit 215 referred to here corresponds to a block in the image restoration NN that determines the image restoration strength based on the feature amount.

[0041] Next, in S306, the first image reconstruction unit 216 generates data of the first restored image 232 using the first corrected feature map 230 and the first weighting parameter 231. The first image reconstruction unit 216 corresponds to a block that reconstructs an image based on the feature amount and the image restoration strength in the image restoration NN, that is, based on the corrected feature map, and generates data of the restored image as a result of the image restoration process. Next, in S307, the first weighting parameter update unit 217 calculates the difference between the first restored image 232 generated in S306 and the first correct image 233 generated in S302 as a loss value, and updates the first weighting parameter so that the loss value is minimized.

[0042] After the image processing device 150 repeats the processes from S302 to S307 a predetermined number of times, in S310, the first weighting parameter update unit 217 determines the first weighting parameter using the finally updated first weighting parameter. The first weighting parameter determined in S310 is output to the second weighting parameter determination unit 218. The output first weighting parameter is input to the second feature extraction unit 220 and the second image reconstruction unit 222 of the second weighting parameter determination unit 218. Specifically, of the output first weighting parameters, the first weighting parameter 235 of the first feature extraction unit 214 is input to the second feature extraction unit 220, and the first weighting parameter 240 of the first image reconstruction unit 216 is input to the second image reconstruction unit 222.

[0043] After S310, the image processing device 150 executes the processes from S312 to S330. First, in S312, the second image acquisition unit 219 acquires data of a second image without image degradation as data of a learning image for determining a second weighting parameter. Hereinafter, the second image data acquired by the second image acquisition unit 219 will be described as data of an image in RGB format. Next, in S313, the second image acquisition unit 219 generates data of a second degraded image 234 to which image degradation expected in an imaging device of a model or individual different from the reference imaging device (hereinafter referred to as a "target imaging device") is added to the second image, and data of a second correct answer image 242 to which no image degradation is added. The data of the second degraded image 234 generated in S303 is input to the image restoration NN after the first weighting parameter output from the first weighting parameter update unit 217 is input, and the data of the second correct answer image 242 is input to the second weighting parameter update unit 225. The image degradation imparting process in the second image acquisition unit 219 is performed in the same manner as in S303. Furthermore, captured image data obtained by performing imaging at high gain in the target imaging device may be acquired as noise-containing second degraded image 234 data.

[0044] Next, in S314, the second feature extraction unit 220 extracts features using the input data of the second degraded image 234 and the first weighting parameter 235, and generates a second feature map 236. In the feature extraction process in the second feature extraction unit 220, the first weighting parameter 235 is fixed to the first weighting parameter 235 determined and input by the first weighting parameter determination unit 212. The second feature extraction unit 220 here corresponds to a block in the image restoration NN from which features are extracted from the input image.

[0045] Next, in S315, the second correction amount determination unit 221 determines the strength of the image restoration process based on the feature amount using the second feature map 236 and the second weighting parameter 237 or 238, and corrects the second feature map 236 based on the determined strength of the image restoration process. A second corrected feature map 239 is generated by the correction of the second feature map 236 by the second correction amount determination unit 221. Note that the second correction amount determination unit 221 here corresponds to a block in the image restoration NN that determines the image restoration strength based on the feature amount. First, the second correction amount determination unit 221 generates the second corrected feature map 239 using the second weighting parameter 237 to which a random value is assigned as an initial value. Thereafter, the second correction amount determination unit 221 generates the second corrected feature map 239 using the second weighting parameter 238 updated by the second weighting parameter update unit 225 described later. The second weighting parameter 237 input to the second correction amount determination unit 221 may be the same as the second weighting parameter 229 input to the first correction amount determination unit 215, or may be a different value.

[0046] Next, in S316, the second image reconstruction unit 222 generates data of a second restored image 241 using the second corrected feature map 239 generated in S315 and the first weighting parameter 240. The first weighting parameter 240 is fixed to a value determined by the first weighting parameter determination unit 212. The second image reconstruction unit 222 here corresponds to a block that reconstructs an image based on the feature amount and image restoration strength in the image restoration NN, i.e., based on the corrected feature map, and generates data of a restored image as a result of the image restoration process.

[0047] Next, in S317, the image quality evaluation unit 223 evaluates the image quality of the second restored image 241 generated in S316. Specifically, the image quality evaluation unit 223 evaluates the image quality of the second restored image 241 by calculating the difference between the data of the second restored image 241 and the data of the second correct answer image 242. The image quality evaluation unit 223 outputs the evaluation result of the image quality of the second restored image 241, that is, the value of the difference between the data of the second restored image 241 and the data of the second correct answer image 242, as an evaluation value 243 of the image quality of the second restored image 241. Note that an image quality evaluation index such as SSIM or PSNR is used to calculate the difference between the images.

[0048] Next, in S318, the variation value calculation unit 224 acquires the image quality evaluation value 243 output from the image quality evaluation unit 223, and calculates the variation value 244 of the image quality evaluation value 243 from the relationship between the second weighting parameter and the image quality evaluation value 243. In this embodiment, the variation value 244 of the image quality evaluation value 243 is a value obtained by second-order differentiation of the image quality evaluation value with the second weighting parameter. Next, in S320, the second weighting parameter update unit 225 updates the second weighting parameter 237 or 238 based on the variation value 244. Here, the second weighting parameter that maximizes the image quality evaluation value does not necessarily provide the result of the image restoration process with the best image quality. This is because, although the amount of noise removed increases when the second weighting parameter is increased, blurring due to overcorrection is likely to occur in the restored image if the second weighting parameter is increased too much.

[0049] The relationship between the second weighting parameter and the evaluation value of image quality will be described with reference to FIG. 4. FIG. 4 is a diagram showing an example of a profile 401 showing the relationship between the second weighting parameter and the evaluation value of image quality according to the first embodiment. When the second weighting parameter is increased from 0, the evaluation value of image quality improves gradually. However, when the second weighting parameter exceeds a certain value, the amount of noise removal increases abruptly, and the evaluation value of image quality improves abruptly. After that, when the second weighting parameter is further increased, blurring occurs in parallel with the removal of noise, and the improvement of the evaluation value of image quality becomes gradual again. The image quality of the restored image that is desirable as a result of the image restoration process is a state in which noise is removed but blurring does not occur. Therefore, in the vicinity of the second weighting parameter at which the rapid improvement of the evaluation value of image quality ends, it is considered that the effect of the image restoration process can be obtained while suppressing the deterioration of image quality due to blurring.

[0050] Here, the rapid improvement of the evaluation value of image quality starts when the second-order differential value is maximized, and ends when the second-order differential value is minimized. For this reason, the second weighting parameter update unit 225 updates the second weighting parameter to a value of the second weighting parameter that minimizes the second-order differential value of the evaluation value of image quality. Specifically, the second weighting parameter update unit 225 updates the second weighting parameter based on the above-mentioned formulas (4) to (7). Here, the loss function L is the second-order differential value of the evaluation value of image quality. Note that the loss function L may be the sum of the mean absolute error or mean square error between the data of the second restored image 241 and the data of the second correct answer image 242 and the second-order differential value of the evaluation value of image quality.

[0051] Next, in S330, the second weighting parameter update unit 225 judges whether the second weighting parameter has been updated a given number of times. If it is judged in S330 that the second weighting parameter has been updated a given number of times, the second weighting parameter update unit 225 ends a series of processes related to updating the second weighting parameter, and determines the second weighting parameter to be the finally updated second weighting parameter 238. After determining the second weighting parameter 238, the image processing device 150 ends the process of the flowchart shown in FIG. 3. If it is judged in S330 that the second weighting parameter has not been updated a given number of times, the image processing device 150 returns to the process of S315 and repeatedly executes the processes from S315 to S330 until it is judged in S330 that the second weighting parameter has been updated a given number of times.

[0052] In this repeated process, the second correction amount determination unit 221 executes the process of S315 using the second weighting parameter 238 updated by the second weighting parameter update unit 225. In addition, the condition for ending the update of the second weighting parameter is not limited to the fact that the second weighting parameter has been updated a given number of times, and the end of the update of the second weighting parameter may be determined based on the value of the loss function L, for example.

[0053] The first weighting parameter determined by the first weighting parameter determination unit 212 and the second weighting parameter determined by the second weighting parameter determination unit 218 are output to the information processing device 100 and used for image restoration processing in the information processing device 100. Specifically, the information processing device 100 can perform image restoration processing on a captured image including image degradation obtained by imaging using an imaging device, using the first weighting parameter and the second weighting parameter determined by the image processing device 150. More specifically, the information processing device 100 can perform image restoration processing on a captured image including image degradation obtained by imaging using a target imaging device when generating data of the second degraded image 234 in the second image acquisition unit 219.

[0054] As described above, the image processing device 150 is configured so that most of the weighting parameters (first weighting parameters) of the image restoration NN are determined based on the characteristics of the reference imaging device. According to such an image processing device 150, the weighting parameters to be determined for each model or individual imaging device can be limited to only a portion (second weighting parameters), so that weighting parameters suitable for each model or individual imaging device can be obtained at low cost.

[0055] The image processing device 150 reduces the cost of acquiring weight parameters for each model or individual imaging device by reducing the processing of the second weight parameter determination unit 218 through the processing of the first weight parameter determination unit 212. Here, the first weight parameter determination unit 212 uses first image data common to imaging devices acquired using a reference imaging device as learning image data. Also, the second weight parameter determination unit 218 uses second image data specific to the imaging device acquired for each target imaging device as learning image data.

[0056] In addition, in the image processing device 150, the weight parameters determined by the first weight parameter determination unit 212 and the second weight parameter determination unit 218 are classified in advance by the parameter classification unit 211 in order to reduce the processing of the second weight parameter determination unit 218. However, as long as the processing of the second weight parameter determination unit 218 can be reduced, the method is not limited to the method of classifying the weight parameters in advance by the parameter classification unit 211. For example, the second weight parameter determination unit 218 may determine the update rate of the weight parameters based on the device dependency. This is because the second weight parameters are updated to a larger extent and the update amount of the first weight parameters is reduced by increasing the update rate of the weight parameters having a larger device dependency. In addition, the second weight parameter determination unit 218 may decrease the loss value for learning image data common to imaging devices and increase it for learning image data specific to the target imaging device. This is because the weight parameters are updated to a larger extent when learning data specific to the target imaging device is used.

[0057] The image data acquired by the image processing device 150 is not limited to RGB format image data, and may be grayscale, monochrome, or other format data such as RAW format. In addition, in this embodiment, the noise removal process is described as an example of the image restoration process, but the application of the present disclosure is not limited to the noise removal process. For example, the present disclosure can also be applied to an image restoration NN that performs image restoration processes such as blur removal, super-resolution, or loss completion from data of a deteriorated image including image deterioration such as blur, low resolution, or loss, to generate a restored image with reduced image deterioration.

[0058] [Embodiment 2] In the first embodiment, a configuration in which the second weighting parameter is determined based on the variation value of the evaluation value of image quality has been described. When an image restoration process is performed using the second weighting parameter determined in this manner, the restored image generated may contain residual noise due to uncorrected noise or blur due to overcorrection of noise. Therefore, in the present embodiment, a configuration in which the range of values ​​that the second weighting parameter can take is determined in advance, the upper limit of the range of the second weighting parameter is determined from the variation value of the evaluation value of image quality, and the user selects the second weighting parameter based on his or her own preference will be described.

[0059] <Functional configuration of image processing device> FIG. 5 is a block diagram showing an example of a functional configuration of an image processing device 150 according to the second embodiment (hereinafter, simply referred to as "image processing device 150"). In addition, in the functional configuration of the image processing device 150, the same components as those of the image processing device 150 according to the first embodiment are denoted by the same reference numerals as those of the image processing device 150 according to the first embodiment, and the description thereof will be omitted. Hereinafter, the difference between the image processing device 150 and the image processing device 150 according to the first embodiment will be mainly described. The image processing device 150 has a parameter classification unit 211, a second weighting parameter input unit 537, a first weighting parameter determination unit 212, and a second weighting parameter determination unit 518. The second weighting parameter determination unit 518 has a second image acquisition unit 219, a second feature extraction unit 220, a second correction amount determination unit 521, a second image reconstruction unit 522, an image quality evaluation unit 523, a variation value calculation unit 524, and a second weighting parameter selection unit 525.

[0060] <Processing flow of image processing device> The operation of the image processing device 150 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of a processing flow of the image processing device 150 according to the second embodiment. The CPU 151 reads a program for executing the processing of the flowchart shown in Fig. 6 from the mass storage device 154 or the like, and executes the program to realize the functions of the above-mentioned respective units that the image processing device 150 has as its functional configuration.

[0061] First, the image processing device 150 executes the processes from S301 to S314. After S314, in S615, the second weighting parameter input unit 537 inputs the second weighting parameters 538 to the second correction amount determination unit 521, and the second correction amount determination unit 521 corrects the second feature map 236 based on the input second weighting parameters 538. Here, it is assumed that the second weighting parameter input unit 537 inputs a plurality of second weighting parameters 538 different from each other to the second correction amount determination unit 521, and that the values ​​of the second weighting parameters 538 input by the second weighting parameter input unit 537 are determined in advance.

[0062] The second weighting parameter is a parameter indicating the degree of amplification of the value of each pixel of the feature map generated by the second feature extraction unit 220. Therefore, for example, when the image restoration process is a noise removal process, the maximum value that the second weighting parameter can take is the ratio value between the amount of noise of the captured image obtained by capturing an image using a reference imaging device and the amount of noise of the imaging device that can have the maximum amount of noise. The second weighting parameter input unit 537 inputs a plurality of values ​​that divide the range from 0 to the value of the ratio at equal intervals to the second correction amount determination unit 521 as a plurality of second weighting parameters. For example, when the value of the above-mentioned ratio is 5.0 and the range from 0 to 5.0 is divided at intervals of 0.1, the second weighting parameter string composed of a plurality of second weighting parameters is {0.1, 0.2, ..., 4.9, 5.0}. The second correction amount determination unit 521 corrects the second feature map 236 based on the inputted second weighting parameters 538, thereby generating second corrected feature maps 539 corresponding to each of the inputted second weighting parameters 538. That is, the second correction amount determination unit 521 generates the second corrected feature maps 539 corresponding to the inputted second weighting parameters 538.

[0063] After S615, in S616, the second image reconstruction unit 522 generates data of a plurality of second restored images 541 using the plurality of second corrected feature maps 539 generated in S615 and the first weighting parameter 240. Specifically, the second image reconstruction unit 522 generates data of the second restored images 541 corresponding to each of the plurality of second corrected feature maps 539 generated in S615. Note that the first weighting parameter 240 is fixed to a value determined by the first weighting parameter determination unit 212. Next, in S617, the image quality evaluation unit 523 evaluates the image quality of each of the plurality of second restored images 541 generated in S516. The method of evaluating the image quality of the second restored image 541 is the same as the method of evaluating the image quality of the second restored image 241 in the image quality evaluation unit 223 according to the first embodiment, and therefore a detailed description thereof will be omitted.

[0064] Next, in S618, the variation value calculation unit 524 obtains the image quality evaluation value 543 of each of the multiple second restored images 541 output from the image quality evaluation unit 523. Also, in S618, the variation value calculation unit 524 generates a profile 401, as shown as an example in FIG. 4, from the second weighting parameters 538 and the image quality evaluation values ​​543 corresponding to each of the multiple second restored images 541, and calculates a variation value 544 of the image quality evaluation value. The profile 401 indicates the relationship between the image quality evaluation value 543 and the second weighting parameters 538 corresponding to each of the multiple second restored images 541. The variation value calculation unit 524 determines the value obtained by first-order or second-order differentiation of the image quality evaluation value with the second weighting parameter as the variation value 544 of the image quality evaluation value. The profile 401 includes the image quality evaluation values ​​543 for the number of second weighting parameters 538 input from the second weighting parameter input unit 537.

[0065] Next, in S619, the second weighting parameter selection unit 525 determines the range of the second weighting parameter. Specifically, the second weighting parameter selection unit 525 determines the range of the second weighting parameter by limiting the range of the second weighting parameter to a range in which artifacts can be suppressed in the restored image.

[0066] For example, according to the profile 401, it can be seen that as the second weighting parameter is increased from 0, the evaluation value of the image quality improves, eventually reaches a peak, and then decreases. The peak value of the evaluation value of the image quality occurs when the first-order differential value becomes 0. In order to suppress deterioration of image quality due to blurring caused by increasing the second weighting parameter, it is desirable to set the second weighting parameter to a value equal to or less than the second weighting parameter when the first-order differential value becomes 0, that is, when the evaluation value of the image quality reaches a peak. For this reason, the second weighting parameter selection unit 525 sets the upper limit of the range of the second weighting parameter to a value of the second weighting parameter that is approximately equal to the value at which the first-order differential value of the evaluation value of the image quality becomes 0. Note that the method of determining the upper limit of the range of the second weighting parameter is not limited to a method of determining based on a variation value of the evaluation value of the image quality, and may be a method of determining by learning in which the second weighting parameter is updated by error backpropagation, similar to the first weighting parameter.

[0067] After S619, in S620, the second weighting parameter selection unit 525 determines a second weighting parameter based on the user's preference from within the range of the second weighting parameter determined in S619. After S620, the image processing device 150 ends the process of the flowchart shown in Fig. 6. Note that the first weighting parameter determined by the first weighting parameter determination unit 212 and the second weighting parameter determined by the second weighting parameter determination unit 518 are output to the information processing device 100 as in the first embodiment, and are used in the image restoration process in the information processing device 100.

[0068] FIG. 7 is a diagram showing an example of a user interface (hereinafter referred to as a "selection screen") 700 for selecting a second weighting parameter according to the second embodiment. The selection screen 700 shown as an example in FIG. 7 is displayed on the display device 14 by the information processing device 100 rendering data showing the selection screen 700 in HTML format or the like output via the Internet. When the user inputs a provisional selection value 701 of the second weighting parameter to be displayed on the display device 14 via the input device 12, a preview 702 of a restored image based on the provisional selection value 701 is displayed on the selection screen 700. The user can input and select a value of the second weighting parameter that provides a preferred preview 702 via the input device 12. The second weighting parameter selection unit 525 acquires the value of the second weighting parameter selected by the user from the information processing device 100 via the Internet, and determines the second weighting parameter based on the acquired value. Note that the options of the value of the second weighting parameter and the upper limit value of the second weighting parameter on the selection screen 700 may be values ​​at which the first-order differential value or the second-order differential value of the evaluation value of the image quality is singular.

[0069] As described above, the image processing device 150 is configured so that the user determines the second weighting parameter based on his / her preference. According to such an image processing device 150, the user can select the second weighting parameter based on his / her preference. Therefore, the user can obtain restored image data that matches his / her preference. For example, when the input image includes characters as objects and the user wants to clearly view the structure of the characters, the user may select a second weighting parameter that allows residual noise and outputs a restored image in which blurring is suppressed. Conversely, when the user wants to suppress flickering of the moving image, the user may select a second weighting parameter that allows blurring and outputs a restored image in which residual noise is reduced.

[0070] [Other embodiments] The present disclosure can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0071] It should be noted that, within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0072] [Configuration of this disclosure] The present disclosure includes the following configurations, methods, and programs.

[0073] <Configuration 1> An image processing device that performs learning of a neural network to generate restored image data in which image degradation included in an input image is reduced, a first determination means for determining a weight parameter corresponding to a first type among a plurality of weight parameters in the neural network; a second determination means for determining a weight parameter corresponding to a second type different from the first type among the plurality of weight parameters, the weight parameter corresponding to the second type being more affected by a difference in model of a device generating data of the input image or an individual difference of the device than the weight parameter corresponding to the first type; 13. An image processing device comprising:

[0074] <Configuration 2> the weight parameters corresponding to the first type include weight parameters of a layer of the neural network corresponding to an extraction unit that extracts a feature from the input image; 2. The image processing device according to claim 1,

[0075] <Configuration 3> the weight parameters corresponding to the second type include weight parameters of a layer corresponding to a strength determination means for determining an image restoration strength based on the feature amount, among the plurality of weight parameters in the neural network; 3. The image processing device according to configuration 2,

[0076] <Configuration 4> the weight parameters corresponding to the first type include weight parameters of a layer of the neural network corresponding to a generation unit that generates the restored image based on the feature amount and the image restoration strength; 4. The image processing device according to configuration 3,

[0077] <Component 5> a classification means for classifying the plurality of weight parameters into weight parameters corresponding to the first type and weight parameters corresponding to the second type based on a degree of dependency which is an index indicating the influence; 5. The image processing device according to any one of configurations 1 to 4,

[0078] <Component 6> A method of determining a weight parameter in the first determination means and a method of determining a weight parameter in the second determination means are different from each other in at least one of the learning data used to determine the weight parameters, the amount of calculation in the process of determining the weight parameters, the update rate of the weight parameters, and the loss value used when updating the weight parameters; 6. The image processing device according to any one of configurations 1 to 5,

[0079] <Component 7> The first determination means determines a weight parameter corresponding to the first type by using learning data common to models or individuals of the device; the second determination means determines a weight parameter corresponding to the second type by using learning data specific to a model or an individual of the device; 7. The image processing device according to any one of configurations 1 to 6,

[0080] <Component 8> The second determination means has a smaller amount of calculation than the first determination means; 8. The image processing device according to any one of configurations 1 to 7,

[0081] <Component 9> the first determination means fixes a weight parameter corresponding to the second type and determines a weight parameter corresponding to the first type; 9. The image processing device according to any one of configurations 1 to 8,

[0082] <Component 10> the first determination means determines a weight parameter corresponding to the first type by repeatedly updating the weight parameter corresponding to the first type; the second determination means determines the weight parameters corresponding to the second type by increasing an update rate of the weight parameters corresponding to the second type for a weight parameter having a higher degree of dependency, which is an index indicating the influence; 10. The image processing device according to any one of configurations 1 to 9,

[0083] <Component 11> the second determination means, when learning data common to models or individuals of the device is input to the neural network, reduces an amount of change in the weight parameter corresponding to the second type, and when learning data specific to the model or individual of the device is input to the neural network, increases an amount of change in the weight parameter corresponding to the second type; 11. The image processing device according to any one of configurations 1 to 10,

[0084] <Component 12> the second determination means determines a weighting parameter corresponding to the second type based on a variation value of an evaluation value of image quality obtained by evaluating an image quality of the restored image; 12. The image processing device according to any one of configurations 1 to 11,

[0085] <Component 13> the second determination means calculates the variation value by first-order or second-order differentiation of the image quality evaluation value with a weighting parameter corresponding to the second type; 13. The image processing device according to claim 12,

[0086] <Component 14> said second determination means determines, as a weighting parameter corresponding to said second type, a value of the weighting parameter corresponding to said second type that minimizes a value obtained when said image quality evaluation value is second-order differentiated with said weighting parameter corresponding to said second type; 14. The image processing device according to configuration 12 or 13,

[0087] <Component 15> an input means for inputting weight parameters corresponding to the second types whose values ​​are determined to the second determination means; an acquisition means for acquiring a value of a weight parameter corresponding to the second type selected by a user; Further comprising: the second determination means determines a weighting parameter corresponding to the second type based on a value of a weighting parameter corresponding to the second type selected by the user; 15. The image processing device according to any one of configurations 1 to 14,

[0088] <Component 16> the second determination means sets an upper limit of a range of the weighting parameter corresponding to the second type selected by the user to a weighting parameter value corresponding to the second type at which a first-order differential value of an evaluation value of image quality obtained by evaluating image quality of the restored image is 0; 16. The image processing device according to claim 15,

[0089] <Component 17> the first determination means determines a weight parameter corresponding to the first type by performing learning of a first neural network having a layer corresponding to a first extraction means for extracting a first feature from a first input image, a layer corresponding to a first intensity determination means for determining a first image restoration intensity based on the first feature, and a layer corresponding to a first generation means for generating a first restored image based on the first feature and the first image restoration intensity; the second determination means determines a weight parameter corresponding to the second type by performing learning of a second neural network having a layer corresponding to a second extraction means for extracting a second feature from a second input image, a layer corresponding to a second intensity determination means for determining a second image restoration intensity based on the second feature, and a layer corresponding to a second generation means for generating a second restored image based on the second feature and the second image restoration intensity; 17. The image processing device according to any one of configurations 1 to 16,

[0090] <Component 18> The first strength determining means corresponds a weight parameter indicating an amplification degree of the first image restoration strength to a layer in the first neural network; the second strength determining means assigns a weight parameter indicating an amplification degree of the second image restoration strength to a layer in the second neural network; 18. The image processing device according to configuration 17,

[0091] <Method> 1. An image processing method for performing training on a neural network to generate restored image data in which image degradation contained in an input image is reduced, comprising the steps of: a first determination step of determining a weight parameter corresponding to a first type among a plurality of weight parameters in the neural network; a second determination step of determining a weight parameter corresponding to a second type different from the first type among the plurality of weight parameters, the weight parameter corresponding to the second type being more affected by a difference in model of a device generating data of the input image or an individual difference of the device than the weight parameter corresponding to the first type; 13. An image processing method comprising:

[0092] <Program> A program for causing a computer to function as the image processing device according to any one of configurations 1 to 18. [Explanation of symbols]

[0093] 150 Image Processing Device 212 First weight parameter determination unit 218 Second weight parameter determination unit

Claims

1. An image processing device that performs learning of a neural network to generate restored image data in which image degradation included in an input image is reduced, a first determination means for determining a weight parameter corresponding to a first type among a plurality of weight parameters in the neural network; a second determination means for determining a weight parameter corresponding to a second type different from the first type among the plurality of weight parameters, the weight parameter corresponding to the second type being more affected by a difference in model of a device generating data of the input image or an individual difference of the device than the weight parameter corresponding to the first type; 13. An image processing device comprising:

2. the weight parameters corresponding to the first type include weight parameters of a layer of the neural network corresponding to an extraction unit that extracts a feature from the input image; 2. The image processing device according to claim 1,

3. the weight parameters corresponding to the second type include weight parameters of a layer corresponding to a strength determination unit that determines an image restoration strength based on the feature amount, among the plurality of weight parameters in the neural network; 3. The image processing device according to claim 2,

4. the weight parameters corresponding to the first type include weight parameters of a layer of the neural network corresponding to a generation unit that generates the restored image based on the feature amount and the image restoration strength; 4. The image processing device according to claim 3,

5. a classification means for classifying the plurality of weight parameters into weight parameters corresponding to the first type and weight parameters corresponding to the second type based on a degree of dependency which is an index indicating the influence; 2. The image processing device according to claim 1,

6. a method for determining a weight parameter in the first determination means and a method for determining a weight parameter in the second determination means differ from each other in at least one of learning data used for determining the weight parameters, an amount of calculation in a process for determining the weight parameters, an update rate of the weight parameters, and a loss value used when updating the weight parameters; 2. The image processing device according to claim 1,

7. The first determination means determines a weight parameter corresponding to the first type by using learning data common to models or individuals of the device; the second determination means determines a weight parameter corresponding to the second type by using learning data specific to a model or an individual of the device; 2. The image processing device according to claim 1,

8. The second determination means has a smaller amount of calculation than the first determination means; 2. The image processing device according to claim 1,

9. the first determination means fixes a weight parameter corresponding to the second type and determines a weight parameter corresponding to the first type; 2. The image processing device according to claim 1,

10. the first determination means determines a weight parameter corresponding to the first type by repeatedly updating the weight parameter corresponding to the first type; the second determination means determines the weight parameter corresponding to the second type by increasing an update rate of the weight parameter corresponding to the second type for a weight parameter corresponding to the second type having a higher degree of dependency, which is an index indicating the influence; 2. The image processing device according to claim 1,

11. the second determination means, when learning data common to models or individuals of the device is input to the neural network, reduces an amount of change in the weight parameter corresponding to the second type, and when learning data specific to the model or individual of the device is input to the neural network, increases an amount of change in the weight parameter corresponding to the second type; 2. The image processing device according to claim 1,

12. the second determination means determines a weighting parameter corresponding to the second type based on a variation value of an evaluation value of image quality obtained by evaluating an image quality of the restored image; 2. The image processing device according to claim 1,

13. the second determination means calculates the variation value by first-order or second-order differentiation of the image quality evaluation value with a weighting parameter corresponding to the second type; The image processing device according to claim 12 .

14. the second determination means determines, as the weight parameter corresponding to the second type, a value of the weight parameter corresponding to the second type that minimizes a value obtained when the image quality evaluation value is second-order differentiated with the weight parameter corresponding to the second type; The image processing device according to claim 12 .

15. an input means for inputting weight parameters corresponding to the second types whose values ​​are determined to the second determination means; an acquisition means for acquiring a value of a weight parameter corresponding to the second type selected by a user; Further comprising: the second determination means determines a weighting parameter corresponding to the second type based on a value of a weighting parameter corresponding to the second type selected by the user; 2. The image processing device according to claim 1,

16. the second determination means sets an upper limit of a range of the weighting parameter corresponding to the second type selected by the user to a value of the weighting parameter corresponding to the second type such that a first-order differential value of an evaluation value of image quality obtained by evaluating image quality of the restored image is zero; The image processing device according to claim 15 .

17. the first determination means determines a weight parameter corresponding to the first type by performing learning of a first neural network having a layer corresponding to a first extraction means for extracting a first feature from a first input image, a layer corresponding to a first intensity determination means for determining a first image restoration intensity based on the first feature, and a layer corresponding to a first generation means for generating a first restored image based on the first feature and the first image restoration intensity; the second determination means determines a weight parameter corresponding to the second type by performing learning of a second neural network having a layer corresponding to a second extraction means for extracting a second feature from a second input image, a layer corresponding to a second intensity determination means for determining a second image restoration intensity based on the second feature, and a layer corresponding to a second generation means for generating a second restored image based on the second feature and the second image restoration intensity; 2. The image processing device according to claim 1,

18. The first strength determining means corresponds to a layer of a weight parameter in the first neural network, the weight parameter indicating an amplification degree of the first image restoration strength; the second strength determining means assigns a weight parameter indicating an amplification degree of the second image restoration strength to a layer in the second neural network; The image processing device according to claim 17 ,

19. 1. An image processing method for performing training on a neural network to generate restored image data in which image degradation contained in an input image is reduced, comprising the steps of: a first determination step of determining a weight parameter corresponding to a first type among a plurality of weight parameters in the neural network; a second determination step of determining a weight parameter corresponding to a second type different from the first type among the plurality of weight parameters, the weight parameter corresponding to the second type being more affected by a difference in model of a device generating data of the input image or an individual difference of the device than the weight parameter corresponding to the first type; 13. An image processing method comprising:

20. A program for causing a computer to function as the image processing device according to any one of claims 1 to 18.

Images