Image processing apparatus, image processing system, and image processing method

Abstract

To provide a technique to acquire a result of higher image quality as an image quality improvement result.SOLUTION: An image processing apparatus acquires, as a correction object, output of a learning model to which an image created on the basis of developing processing is input, or the difference between the image and the output. The image processing apparatus updates a parameter of the learning model on the basis of the difference between a first statistic that is the statistic of a correction result of correcting the correction object on the basis of information used in the developing processing and a second statistic acquired in advance.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an image processing technique, and more particularly to an image processing technique for improving image quality. [Background technology]

[0002] In recent years, convolutional neural networks (CNNs) have been applied to many image quality improvement applications. Image quality improvement processes specifically refer to processes that restore the original image from a degraded image, such as noise removal, super-resolution, blur removal, and missing image interpolation. For example, a technology that uses CNNs to perform high-precision noise removal has been proposed (Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation [Non-Patent Document 2] A White Paper on Neural Network Quantization [Non-Patent Document 3] SoDaCam: Software-defined Cameras via Single-Photon Imaging Summary of the Invention [Problem to be solved by the invention]

[0004] In the method disclosed in Non-Patent Document 1, learning is performed under constraint conditions so that the output image of the CNN matches the correct image. On the other hand, no constraint conditions are set for the output of each intermediate layer, such as the convolution layer. Therefore, the distribution of the output of the intermediate layer is likely to be learned to deviate from the ideal shape. As a result, the accumulated deviation is likely to cause the generation of an image as a high-image-quality result that includes artifacts such as residual noise and color shift. The present invention provides a technology for obtaining a higher image quality result as a high-image-quality result. [Means for solving the problem]

[0005] One aspect of the present invention is characterized in that it comprises an acquisition means for acquiring the output of a learning model to which an image generated based on a development process is input, or the difference between the image and the output, as a correction target, and an update means for updating parameters of the learning model based on the error between a first statistic, which is a statistic of the correction result obtained by correcting the correction target based on information used in the development process, and a second statistic acquired in advance. Effect of the Invention

[0006] According to the present invention, it is possible to obtain a higher image quality as a result of image quality improvement. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an image processing system. [Diagram 2] FIG. 2 is a block diagram showing an example of the hardware configuration of each of the cloud server 200 and the edge device 100. [Diagram 3] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing system. [Figure 4] 4 is a flowchart showing the operation of the image processing system. [Diagram 5] A diagram showing the data flow within a CNN. [Figure 6] 11A and 11B are diagrams for explaining issues with the distribution of a residual map, which is an intermediate output. [Figure 7]FIG. 4 is a diagram showing a flow of calculation of a degraded image. [Figure 8] FIG. 13 is a diagram for explaining the purpose of performing correction. [Figure 9] FIG. 3 is a block diagram showing an example of the functional configuration of an acquisition unit 301. [Figure 10] FIG. 11 is a diagram for explaining the process of step S401 according to the second embodiment. [Figure 11] FIG. 11 is a functional block diagram of an entire image processing system according to a third embodiment. [Figure 12] FIG. 11 is a functional block diagram of an entire image processing system according to a third embodiment. [Figure 13] 10 is a flowchart showing a processing flow in an image processing system according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0009] <CNNについて> First, a description will be given of a Convolutional Neural Network (CNN) that is used in general image processing techniques that apply deep learning and that will appear in each of the following embodiments. CNN is a technique that repeats convolution of a filter generated by training with an image and then nonlinear operation. The filter is also called a local receptive field. An image obtained by convolving a filter with an image and then nonlinear operation is called a feature map. Furthermore, learning is performed using training data (training images or data sets) consisting of pairs of input images and output images. Simply put, learning is the generation of filter values ​​that can convert an input image into a corresponding output image with high accuracy from training data. This will be described in detail later.

[0010] When an image has RGB color channels or when a feature map is composed of multiple images, the filter used for convolution also has multiple channels accordingly. In other words, a convolution filter is expressed as a four-dimensional array that includes the number of channels in addition to the vertical and horizontal sizes and number of images. The process of performing nonlinear operations after convolving a filter with an image (or feature map) is expressed in units of layers, such as the nth layer feature map or the nth layer filter. For example, a CNN that repeats filter convolution and nonlinear operations three times has a three-layer network structure. Such nonlinear operation processing can be formulated as the following equation (1).

[0011]

number

[0012] Here, \(W_n\) is the filter of the \(n\)th layer (\(n = 1\) to \(N\)), \(X_n\) is the feature map of the \(n\)th layer, \(b_n\) is the bias of the \(n\)th layer, \(f\) is a non-linear operator, and \(*\) is a convolution operator. Note that the superscript \((l)\) on the right indicates the \(l\)th filter, the \(l\)th feature map, and the \(l\)th bias. The filters and biases are updated by the learning described later and are collectively also called "network parameters". As non-linear operations, for example, the sigmoid function or ReLU (Rectified Linear Unit) is used. In the case of ReLU, it is given by the following equation (2).

[0013]

Equation

[0014] In the non-linear operation of Equation (2), 0 is output as the output value corresponding to the element having a negative value among the elements of the input vector \(X\), and the value of the element is output as the output value corresponding to the element having a positive value.

[0015] As networks using CNN, ResNet in the field of image recognition and its application RED-Net in the field of super-resolution are well-known. In both cases, by stacking multiple layers of CNN and performing convolution of filters multiple times, high-precision processing is achieved. For example, ResNet features a network structure with a path that shortcuts the convolutional layer, thereby realizing a multi-layer network of up to 152 layers and achieving high-precision recognition approaching the human recognition rate. The reason why the processing is made more accurate by multi-layer CNN is simply that by repeating non-linear operations multiple times, a non-linear relationship between the input and output can be expressed.

[0016] <Learning of CNN> Next, the learning of CNN will be explained. The learning of CNN is generally performed by minimizing the objective function represented by the following equation (3) for the training data consisting of a pair of an input training image (student image) and an output training image (teacher image) corresponding to the input training image.

[0017]

number

[0018] Here, L is a loss function that measures the error between the correct answer and its estimate. Yi is the i-th output training image, and Xi is the i-th input training image. F is a function that collectively represents the calculations (equation (1)) performed in each layer of CNN. θ is the network parameters (filter and bias). ||Z|| is the L2 norm, which is simply the square root of the sum of the squares of the elements of vector Z. n is the total number of training data used for learning. Since the total number of training data is generally large, in the Stochastic Gradient Descent (SGD) method, a portion of the training data is randomly selected and used for learning. This reduces the computational load in learning using a large amount of training data. In addition, various methods are known for minimizing (optimizing) the objective function, such as the momentum method, the AdaGrad method, the AdaDelta method, and the Adam method. The Adam method is given by the following equation (4).

[0019]

number

[0020] Here, θi t is the i-th network parameter at iteration t, and g is θi t is the gradient of the loss function L with respect to . Additionally, m and v are moment vectors, α is the base learning rate, β1 and β2 are hyperparameters, and ε is a small constant. Note that since there are no guidelines for selecting an optimization method in learning, essentially any optimization method can be used, but it is known that differences in convergence between methods result in differences in learning time.

[0021] <Challenges for high-resolution CNN> The issues that arise in the high-image-quality CNN typified by Non-Patent Document 1 will be described in detail with reference to Fig. 5 and Fig. 6. The data flow within the CNN in Non-Patent Document 1 is shown in Fig. 5. The CNN in Non-Patent Document 1 aims to improve noise removal performance by providing an internal noise removal network for step 1 and a noise removal network for step 2.

[0022] First, a degraded image 501 with noise superimposed thereon is input to a noise removal network (NRstep1) in step1, which outputs a residual map (residual map in step1) 502, which is the noise component of the degraded image 501. The residual map 502 is added to the degraded image 501 to output a "step1 image quality improvement result 503." At this point, the noise in the degraded image 501 has not been completely removed.

[0023] To remove the remaining noise, the "image quality improvement result of step 1 503" is input to a noise removal network of step 2 (NRstep2), and a "residual map of step 2 504" that is the noise component of the "image quality improvement result of step 1 503" is obtained from the NRstep2. Then, the "image quality improvement result of step 2 505" is output by adding the "residual map of step 2 504" to the "image quality improvement result of step 1 503".

[0024] The issue with the distribution of the residual map, which is an intermediate output, will be described with reference to Fig. 6. Generally, additive shot noise with positive and negative symmetric values ​​with an average of 0 is applied to an image captured by a CMOS sensor. Therefore, the distribution of the residual map ideally has a positive and negative symmetric shape with an average of 0, like a histogram 601 of the residual map in step 1 or a histogram 602 of the residual map in step 2. Here, the horizontal axis of the histogram represents the pixel value of the residual map, and the vertical axis represents the frequency.

[0025] On the other hand, in histogram 603 of the residual map in step 1 obtained by actually performing learning, the average value tends to deviate from 0, and local peaks such as in region 603b tend to occur. When such a phenomenon occurs, in the residual map in step 2, in order to correct the deviation in step 1, the average value tends to deviate in the opposite direction, as in histogram 604, and local peaks with opposite polarities tend to occur, as in region 604b.

[0026] In addition, the above deviation may worsen due to the influence of other operations in the CNN. For example, the neural network quantization method represented by Non-Patent Document 2 is a method of performing a process of reducing the calculation bit depth of the learning result model in order to increase the execution speed of the neural network. When the calculation bit depth is reduced, the expression range of the value of each intermediate output is limited to the range 601a to 604a. In an ideal distribution, the loss of information due to the range restriction is minor, but there is an area 604b where the value cannot be expressed in the histogram 604 with deviation. In this case, the value in the area 604b is clipped to the maximum value in the range 604a, and the positive and negative symmetry of the histogram is lost. As a result, the positive noise cannot be completely removed, and residual noise occurs in the high image quality result.

[0027] As described above, if constraints are not set for the intermediate output of the CNN, there is a problem that artifacts may occur in the image quality improvement results due to the influence of other calculations in the CNN.

[0028] [First embodiment] In this embodiment, a method for reducing the above-mentioned artifacts will be described by taking an example of a case where noise removal is performed to improve image quality. First, a configuration example of an image processing system according to this embodiment will be described with reference to FIG.

[0029] 1, the image processing system according to this embodiment includes a cloud server 200 and an edge device 100, and is configured so that the cloud server 200 and the edge device 100 can communicate data with each other via the Internet. The network between the cloud server 200 and the edge device 100 is not limited to the Internet, and may be other types of networks such as a LAN. The network between the cloud server 200 and the edge device 100 may be a wired network, a wireless network, or a combination of these networks.

[0030] The cloud server 200 is a server device that functions as an image processing device that provides cloud services on the Internet. More specifically, the cloud server 200 generates learning data and performs learning of a process for restoring an original image from a degraded image whose image quality has been degraded (degradation restoration learning). The cloud server 200 then transmits (provides) network parameters, which are the result of the learning, to the edge device 100 in response to a request from the edge device 100.

[0031] The edge device 100 generates (acquires) a captured image of higher image quality by performing degradation restoration inference (inference of the restoration result of the image before degradation) on a captured image input from the imaging device 10 described later by applying a learning model in which network parameters provided by the cloud server 200 are set. For example, a user applies super-resolution processing to an image by using an image processing application installed in the edge device 100.

[0032] Next, an example of the hardware configuration of each of the cloud server 200 and the edge device 100 included in the image processing system according to this embodiment will be described with reference to the block diagram of Fig. 2. Fig. 2 shows a case in which the cloud server 200 is configured with one device, but the cloud server 200 may be configured with two or more devices.

[0033] First, an example of the hardware configuration of the cloud server 200 will be described.

[0034] CPU 201 executes various processes using computer programs and data stored in RAM 203. As a result, CPU 201 controls the overall operation of cloud server 200, and executes or controls various processes that will be described as processes performed by cloud server 200.

[0035] ROM 202 stores setting data for cloud server 200, computer programs and data related to the startup of cloud server 200, computer programs and data related to the basic operation of cloud server 200, and the like.

[0036] The RAM 203 has an area for storing computer programs and data loaded from the ROM 202 or the mass storage device 204. The RAM 203 also has an area for storing computer programs and data received from the edge device 100 via the network I / F 205. The RAM 203 also has a work area used when the CPU 201 executes various processes. In this way, the RAM 203 can provide various areas as appropriate.

[0037] The mass storage device 204 is a non-volatile memory such as a hard disk drive for storing a large amount of computer programs and data. The mass storage device 204 stores an OS (operating system), computer programs and data for causing the CPU 201 to execute or control various processes described as processes performed by the cloud server 200, and the like. The computer programs and data stored in the mass storage device 204 are loaded into the RAM 203 as appropriate under the control of the CPU 201, and become targets for processing by the CPU 201.

[0038] The network I / F 205 is a communication interface for connecting the cloud server 200 to the Internet, and the cloud server 200 performs data communication with the edge device 100 via the network I / F 205. For example, the cloud server 200 receives a request from a web browser of the edge device 100 via the network I / F 205. Then, in response to the received request, the cloud server 200 transmits the above-mentioned network parameters to the edge device 100 on the Internet via the network I / F 205. The CPU 201, the ROM 202, the RAM 203, the mass storage device 204, and the network I / F 205 are all connected to a system bus 206.

[0039] Next, a description will be given of an example of a hardware configuration of the edge device 100. The CPU 101 executes various processes using computer programs and data stored in the RAM 102. In this way, the CPU 101 controls the operation of the entire edge device 100, and executes or controls various processes that will be described as processes performed by the edge device 100.

[0040] The RAM 102 has an area for storing computer programs and data loaded from the ROM 103 or the mass storage device 104, and an area for storing computer programs and data read from a storage medium by the external storage device 30. The RAM 102 also has an area for storing captured images received from the imaging device 10 via the general-purpose I / F 105, and an area for storing computer programs and data received from the cloud server 200 via the network I / F 106. The RAM 102 also has a work area used when the CPU 101 executes various processes. In this way, the RAM 102 can provide various areas as appropriate.

[0041] The ROM 103 stores setting data for the edge device 100, computer programs and data related to the startup of the edge device 100, computer programs and data related to the basic operation of the edge device 100, and the like.

[0042] The mass storage device 104 is a non-volatile memory for storing a large amount of computer programs and data, such as a hard disk drive device. The mass storage device 104 stores an OS (operating system), computer programs and data for causing the CPU 101 to execute or control various processes described as processes performed by the edge device 100, and the like. The computer programs and data stored in the mass storage device 104 are loaded into the RAM 102 as appropriate under the control of the CPU 101, and become targets for processing by the CPU 101.

[0043] The network I / F 106 is a communication interface for connecting the edge device 100 to the Internet, and the edge device 100 performs data communication with the cloud server 200 via the network I / F 106. The edge device 100 accesses the cloud server 200 via the network I / F 106 using an installed web browser, and acquires network parameters for degradation restoration inference.

[0044] The general-purpose I / F 105 is a serial bus interface such as USB, IEEE1394, HDMI (registered trademark), etc. The CPU 101, RAM 102, ROM 103, mass storage device 104, network I / F 106, and general-purpose I / F 105 are all connected to a system bus 107.

[0045] The imaging device 10 has an imaging sensor such as a CMOS, and acquires an original image in RGB format obtained by the imaging sensor. The original image is an image in a color space that has not been subjected to development processing, more specifically, an image in a color space that has not been subjected to white balance correction, gamma correction, or color matrix correction. The current image acquired by the imaging device 10 becomes a captured image (RAW image) that is to be subjected to super-resolution processing.

[0046] The input device 20 is a user interface such as a keyboard, a mouse, or a touch panel screen, and the user can input various instructions to the edge device 100 by operating it.

[0047] The external storage device 30 reads and writes computer programs and data from and to storage media such as a memory card, a CF card, an SD card, a USB memory, etc. Some or all of the computer programs and data described as the computer programs and data stored in the mass storage device 104 may be stored in the storage medium.

[0048] Display device 40 has a liquid crystal screen or a touch panel screen, and can display the results of processing by CPU 101 as images, characters, etc. Note that display device 40 may be a projection device such as a projector that projects images and characters.

[0049] Although there are other components of the edge device 100 and the cloud server 200 in addition to those described above, they are not the main focus of the description and will not be described here. In this embodiment, it is assumed that the cloud server 200 generates learning data and performs degradation restoration learning, and downloads network parameters that are the learning results to the edge device 100, and performs degradation restoration inference on a RAW image. However, this system configuration is merely an example and is not limited to this.

[0050] For example, the functions of the cloud server 200 may be subdivided, and the generation of learning data and the degradation restoration learning may be performed by separate devices. That is, the cloud server 200 may be realized by two or more devices. In this case, the devices are connected via a circuit or a wired or wireless network, and can realize the processing of the cloud server 200 by performing data communication with each other and operating cooperatively. The same applies to the edge device 100.

[0051] Furthermore, the imaging device 10 having both the functions of the edge device 100 and the cloud server 200 may be configured to perform all of the generation of learning data, degradation restoration learning, and degradation restoration inference.

[0052] <Overall image processing system processing flow> An example of the functional configuration of the image processing system according to this embodiment is shown in the block diagram of FIG. 3. Among the functional units shown in FIG. 3, the functional units with reference numbers 301 to 308 are functional units of the cloud server 200, and the functional units with reference numbers 309 to 311 are functional units of the edge device 100. In the following, it is assumed that all the functional units shown in FIG. 3 are implemented by computer programs. In the following, the functional units shown in FIG. 3 will be described as the subject of processing. However, in reality, the functions corresponding to the functional units with reference numbers 301 to 308 are realized by the CPU 201 executing the computer programs corresponding to the functional units. Also, the functions corresponding to the functional units with reference numbers 309 to 311 are realized by the CPU 101 executing the computer programs corresponding to the functional units.

[0053] In addition, the configuration shown in Fig. 3 can be modified or changed as appropriate. For example, one functional unit may be divided into multiple functional units, or two or more functional units may be integrated into one functional unit.

[0054] The operation of the image processing system according to this embodiment will be described with reference to the flowchart of FIG. 4. In step S401, the acquisition unit 301 acquires an ideal statistic (ideal statistic) that the residual map follows. In a CMOS sensor, the shot noise superimposed on each pixel can be modeled as being determined based on random numbers that follow a standard normal distribution. In this embodiment, the imaging device 10 is assumed to have a CMOS sensor, and the nth moment around the mean value of the standard normal distribution is assumed to be the ideal statistic. In this embodiment, n=4, and the ideal statistic includes four elements of the distribution: the mean, variance, kurtosis, and skewness. Note that the ideal statistic is not limited to the nth moment, and may include elements such as the median, mode, and histogram of the distribution. Note that the sensor included in the imaging device 10 is not limited to a CMOS sensor, and may include, for example, a SPAD sensor. In the case of a SPAD sensor, shot noise that follows a Poisson distribution is superimposed, so that an ideal statistic that follows it is calculated.

[0055] In step S402, the acquisition unit 302 acquires an original image in RGB format captured by the imaging device 10. When the acquisition unit 302 acquires an image after development processing, the acquisition unit 302 acquires the original image by applying the inverse correction of each correction.

[0056] In step S403, the generating unit 303 regards the original image acquired in step S402 as a real scene, virtually captures the real scene, and calculates a degraded image obtained by capturing the real scene. The flow of this calculation is shown in FIG.

[0057] First, the generation unit 303 executes an image capture process simulation 701 on the original image, which reproduces the process in which the light of the scene is received by the image sensor of the image capture device 10 and a RAW image is obtained.

[0058] In the imaging process simulation 701, first, the generation unit 303 applies color thinning 713 to the original image. The color thinning 713 is a process for reproducing the process of light passing through a color filter for an image in RGB format 710, and acquiring a one-channel image in which only pixel values ​​of a specific color component are recorded in each pixel, as in the Bayer format 709. Next, the generation unit 303 applies noise addition 704 to the image in the Bayer format to generate a RAW image after noise addition. In the CMOS sensor, additive noise dependent on luminance is added. The addition process is expressed by the following equation (5).

[0059]

number

[0060] Here, I is the Bayer format image, I0 is the offset of the original image (a value that is uniformly added to the shooting luminance value), α is a random number map that follows a standard normal distribution for the image size, and σ is a noise intensity parameter map for the image size. The noise intensity parameter is obtained by applying the quadratic equation f specific to the sensor to the luminance value acquired by the sensor (I-I0). Each coefficient of the quadratic equation is obtained by measuring the noise characteristics of the sensor in advance. In this way, a RAW image with noise added is obtained.

[0061] Next, the generation unit 303 executes a development processing simulation 702, which is a conversion process to an RGB image for human viewing, on the RAW image obtained in the imaging processing simulation 701.

[0062] In the development process simulation 702, first, the generation unit 303 applies demosaic 705 to the RAW image. The demosaic 705 is a process for estimating missing color channel information of the Bayer format image and obtaining an RGB format image. Next, the generation unit 303 applies white balance correction 706 to the RGB format image obtained by applying demosaic 705 to the RAW image. The white balance correction 706 is a process for multiplying a gain (white balance coefficient) W for each channel in order to correct the sensitivity difference for each color of the sensor. Next, the generation unit 303 applies color matrix correction 707 to the RGB format image obtained by the white balance correction 706. The color matrix correction 707 is a process for multiplying each pixel value by a conversion matrix M to convert to a color space suitable for human viewing. Next, the generation unit 303 applies gamma correction 708 to the RGB format image obtained by the color matrix correction 707 to generate a deteriorated image. Gamma correction 708 is an image enhancement process that obtains a correction result of p1 / γ for an image p with a given gamma coefficient γ. A degraded image is expressed by the following equation (6) for an image I in Bayer format.

[0063]

number

[0064] Here, x is a degraded image and y is a teacher image. The generation unit 303 generates a teacher image y by applying the same white balance correction 706, color matrix correction 707, and gamma correction 708 as the degraded image to the original image. The teacher image y is an image obtained by applying the remaining processes after removing the degradation factors, color thinning 703, noise addition 704, and demosaic 705, from the process group applied to generate the degraded image x. Note that the process for generating a degraded image / teacher image from the current image is not limited to the above process. For example, a development simulation that performs processes such as digital gain and YUV compression may be applied.

[0065] In step S404, the holding unit 304 stores the information used in the above-mentioned development process simulation (development information) in a memory device such as the RAM 203 or the mass storage device 204 for later reuse. The development information includes one or more of a Bayer format image I, a noise intensity parameter map σ, a gamma coefficient γ, a white balance coefficient W, a color matrix (transformation matrix) M, and a color filter array. Note that the development information may further include other development information such as gain information and YUV compression information.

[0066] In step S405, the image quality improving unit 305 obtains an inference result (image quality improvement inference result) as a correction target by performing CNN arithmetic processing with the degraded image generated in step S403 as input. The inference result obtained by the CNN in Non-Patent Document 1 is the residual map r1 in step 1 and the residual map r2 in step 2, and an image as an image quality improvement result is obtained by adding the residual map r1 and the residual map r2 to the degraded image. In this embodiment, a case will be described in which the residual map r1 and the residual map r2 are obtained as inference results by performing CNN arithmetic processing with the degraded image generated in step S403 as input, as in Non-Patent Document 1. However, a network in which the inference result is the image quality improvement result itself may be used.

[0067] In step S406, the correction unit 306 uses the development time information to correct the inference result (correction target) acquired by the image quality improvement unit 305. The purpose of performing the correction will be described with reference to FIG.

[0068] In the residual map obtained as a result of the inference, the noise components of each pixel follow a normal distribution whose variance depends on the luminance captured by the sensor, as shown on the left side of Figure 8. Because the distributions that each pixel follows are not the same, statistics cannot be obtained from the residual map. Therefore, a correction is performed using the information at the time of development to remove the components that depend on the luminance captured from the residual map. As a result, the values ​​of each pixel follow a standard normal distribution, as shown on the right side of Figure 8, making it possible to analyze the statistics.

[0069] As a method of correction, the equation r = yx holds for the residual map r, where y and x are defined in equation (6). By substituting x and y in equation (6) into r = yx and solving the equation for α, the following equation (7) is obtained.

[0070]

number

[0071] In addition, the formula is transformed to linearly approximate the nonlinear function. When correcting the residual map r1, the correction unit 306 obtains a value rr1 corresponding to each pixel of the residual map r1 by substituting the pixel value of the pixel into r on the right side of the formula (7) and calculating the right side, and generates α indicating the value rr1 corresponding to each pixel in the residual map r1 as the correction result map R1. When correcting the residual map r2, the correction unit 306 obtains a value rr2 corresponding to each pixel of the residual map r2 by substituting the pixel value of the pixel into r on the right side of the formula (7) and calculating the right side, and generates α indicating the value rr2 corresponding to each pixel in the residual map r2 as the correction result map R2. α is a map that follows a standard normal distribution. In this way, based on the inference result and the development time information, it is possible to analyze statistics in a state where the value of each pixel of the residual map follows the same distribution.

[0072] In step S407, the error calculation unit 307 acquires an actual measurement statistic ST1 measured from the correction result map R1 and an actual measurement statistic ST2 measured from the correction result map R2. The actual measurement statistic ST1 is an n-th moment around the mean value of the distribution corresponding to the correction result map R1, and includes the mean, variance, kurtosis, and skewness of the distribution. The actual measurement statistic ST2 is an n-th moment around the mean value of the distribution corresponding to the correction result map R2, and includes the mean, variance, kurtosis, and skewness of the distribution.

[0073] Then, the error calculation unit 307 calculates the error as the sum of error 1 between the measured statistic ST1 and the ideal statistic ST obtained in step S401, and error 2 between the measured statistic ST2 and the ideal statistic ST obtained in step S401.

[0074] More specifically, the mean, variance, kurtosis, and skewness included in the measured statistics ST1 are x11, x12, x13, and x14, respectively, and the mean, variance, kurtosis, and skewness included in the measured statistics ST2 are x21, x22, x23, and x24, respectively. Also, the mean, variance, kurtosis, and skewness included in the ideal statistics ST are y1, y2, y3, and y4, respectively. At this time, the error calculation unit 307 calculates the weighted sum of |x11-y1|, |x12-y2|, |x13-y3|, |x14-y4|, |x21-y1|, |x22-y2|, |x23-y3|, and |x24-y4| as the error.

[0075] The method of calculating the error between the actual statistics measured from each of the correction result maps R1 and R2 and the ideal statistics obtained in step S401 is not limited to the above method, and various methods are possible. In other words, the method of calculating the statistics from the set of absolute errors may be other methods, such as the difference between the most frequent values ​​of each or the distance of a histogram. In addition, although an example of obtaining two correction result maps has been shown in this embodiment, processing may be performed on other numbers of correction result maps depending on the architecture of the network. In this way, it is possible to evaluate the distribution deviation for each intermediate output of the network.

[0076] In step S408, the update unit 308 sets the error calculated in step S407 as a loss function value, and performs learning of the error backpropagation of the CNN so as to reduce the loss function value, thereby updating the network parameters of the CNN based on formula (4). According to formula (4), the larger the error, the larger the update amount of the network parameters.

[0077] In step S408a, the update unit 308 determines whether or not a learning end condition has been satisfied. There are various possible conditions for the learning end condition, and it is not limited to a specific condition. For example, possible learning end conditions are "the number of repetitions of the processes (learning process) from step S402 to step S408 is equal to or greater than a threshold", "the elapsed time since the start of the learning process is equal to or greater than a threshold", and "the loss function value is less than a threshold". In addition to these, another possible learning end condition is "the absolute value of the difference (or the square of the difference) between the previous loss function value and the current loss function value is less than a threshold".

[0078] If it is determined that the learning end condition is satisfied, the process proceeds to step S409; if the learning end condition is not satisfied, the process proceeds to step S402.

[0079] When the learning end condition is satisfied, the update unit 308 stores the updated (learned) network parameters in a memory device such as the RAM 203 or the mass storage device 204. Then, the update unit 308 transmits the network parameters to the edge device 100 in response to a request from the edge device 100.

[0080] According to this type of CNN learning process, the greater the difference between the measured statistics and the ideal statistics, the larger the loss function value becomes and the larger the update range of the network parameters becomes, so that learning tends to progress so as to reduce the deviation in the statistics of the residual map.

[0081] The processes of steps S409 to S411 described below are processes in which the edge device 100 applies image quality improvement based on the results of the learning processes up to step S408a to a newly input image.

[0082] In step S409, the acquisition unit 309 acquires the current image in RGB format captured by the imaging unit 10 (the RAW image to which image quality improvement is applied) as an original image for inference. In step S410, the generation unit 310 applies development processing to the original image for inference acquired in step S409 to generate an input image of three RGB channels. The development processing is processing that is substantially the same as the development processing simulation 702 in FIG. 7.

[0083] In step S411, the inference unit 311 transmits an acquisition request for the learned network parameters to the cloud server 200, and receives the network parameters transmitted from the cloud server 200 in response to the acquisition request. The inference unit 311 then inputs the input image generated in step S410 to the CNN in which the received network parameters are set, performs arithmetic processing of the CNN, and performs image quality improvement of the input image, and outputs the image obtained by the image quality improvement. The output destination of the image obtained by the image quality improvement is not limited to a specific output destination. For example, the inference unit 311 may display the image obtained by the image quality improvement on the display device 40, may store it in the mass storage device 104, or may control the external storage device 30 to write it to a storage medium.

[0084] In this embodiment, an example of using deep learning as a method for obtaining high image quality results has been shown, but this is merely one example, and it is also possible to use, for example, a rule-based image quality improvement method or a method using classical learning as a method for obtaining high image quality results.

[0085] In addition, in this embodiment, an example has been shown in which the error calculated by the error calculation unit 307 is treated as a loss function value and used for error backpropagation, but the method of using the error is not limited to this. For example, the error calculation unit 307 may use the smallness of the error as a "performance evaluation value that is a value for evaluating the performance of the CNN" and use it to determine that learning is to be terminated when the performance evaluation value becomes equal to or greater than a given threshold. In addition, multiple CNNs with different learning data and learning times may be created, and the best CNN may be selected and used by comparing the respective performance evaluation values.

[0086] Furthermore, according to this embodiment, the inference result is corrected based on the information at the time of development, and learning is performed to reduce the error between the statistics of the correction result and the ideal, thereby suppressing the deviation of the statistics of the inference result. As a result, artifacts such as residual noise and color shifts in the image quality improvement result can be reduced. In addition, by setting constraint conditions for the residual map, which is the intermediate output of the CNN, the deviation of the distribution of the intermediate output can be suppressed, and artifacts in the image quality improvement result can be reduced.

[0087] Although the learning according to the present embodiment does not use learning data, learning may be performed using further learning data. In this case, the update unit 308 obtains, as the second error, the difference between an image obtained by inputting the degraded image to the CNN and a teacher image generated from the current image from which the degraded image is generated, and the degraded image is obtained as a student image. For example, the update unit 308 obtains, as the second error, the sum (or weighted sum) of pixel values ​​of all pixels in the difference image between the image obtained by inputting the student image to the CNN and the teacher image. The update unit 308 then obtains, as the loss function value, the weighted sum of the error obtained in step S407 and the second error, and performs learning of the error backpropagation of the CNN so as to reduce the loss function value.

[0088] [Second embodiment] The following describes the differences from the first embodiment, and unless otherwise specified below, it is assumed that the present embodiment is the same as the first embodiment. In the first embodiment, a case where noise removal is performed as image quality improvement is taken as an example, and a case where artifacts in the image quality improvement result are reduced is described. However, image quality improvement is not limited to noise removal. In this embodiment, a case where other tasks are performed as image quality improvement is described.

[0089] An example of the functional configuration of the acquisition unit 301 according to this embodiment is shown in the block diagram of Fig. 9. The system according to this embodiment basically executes the process according to the flowchart of Fig. 4, but differs from the first embodiment in the following points.

[0090] In step S401, the acquisition unit 301 calculates a distribution to which the inference result follows in the image quality improvement task. As an example of the image quality improvement task, as shown in Fig. 9, super-resolution, which is a task for improving the resolution of a degraded image, is taken.

[0091] First, the setting unit 901 sets an original signal 1001 of a hard edge as shown in Fig. 10(a) as high-frequency information that should exist before resolution degradation. The original signal 1001 is a line profile of an edge, with spatial information on the horizontal axis and pixel values ​​on the vertical axis.

[0092] The degradation adding unit 902 applies resolution degradation, which is a process for degrading the resolution of an original signal 1001, to obtain a line profile 1002 after edge smoothing, as shown in FIG. 10(a).

[0093] A difference calculation unit 903 calculates a difference signal 1003 which is a signal indicating the difference between the smoothed line profile 1002 and the original signal 1001. A calculation unit 904 obtains statistics including the n-th moment around the average value as ideal statistics for the difference signal 1003. A histogram of the differences is shown as a histogram 1004 in FIG. 10(b).

[0094] In step S403, the generation unit 303 generates a reduced current image (a reduced image of the current image) by reducing the original image acquired in step S402, and performs a simulation of a development process similar to that in the first embodiment on the reduced current image to generate a degraded image.

[0095] In step S405, the image quality improving unit 305 obtains a super-resolution result by performing CNN arithmetic processing to which the degraded image generated in step S403 is input. Furthermore, the image quality improving unit 305 generates an enlarged degraded image by enlarging the degraded image generated in step S403 to the same size as the super-resolution result (output of CNN), which is the CNN arithmetic result. Then, the image quality improving unit 305 obtains a difference image between the super-resolution result and the enlarged degraded image as an inference result.

[0096] The application of this embodiment is not limited to super-resolution tasks. This embodiment can also be applied to other image quality enhancement tasks by applying a target degradation to an original signal and calculating the statistics of the difference from the original signal.

[0097] According to the present embodiment, it is possible to suppress deviations in the statistics of inference results even in image quality improvement tasks other than noise removal. For example, in a super-resolution task, it is possible to reduce blurring and ringing that occur due to deviations in the distribution of difference images.

[0098] [Third embodiment] The following describes the differences from the first embodiment, and unless otherwise specified below, it is assumed that the present embodiment is the same as the first embodiment. In the first embodiment, an example was described in which an error is calculated based on the result of correcting an image based on information used in the development process. In this example, if the RAW image before development has linear luminance characteristics in which the subject luminance and pixel value have a linear relationship, the profile of the correction result and the ideal statistics that follow it can be obtained well.

[0099] On the other hand, if the RAW image before development does not have a linear luminance characteristic, image processing with linear calculations cannot be applied without error. An example of a sensor whose characteristics are not linear with luminance is a SPAD sensor, which counts the number of occurrences of the physical phenomenon of incoming photons and obtains the result as a pixel value. Therefore, in such sensors, sensor characteristic conversion is applied to the RAW image so that it becomes equivalent to a linear luminance sensor, and then image processing is applied in the subsequent stage (Figure 11, Non-Patent Document 3).

[0100] In such processing, the characteristics of the image profile also change due to the sensor characteristic conversion. If the method of the first embodiment is applied to the changed profile, the statistical error cannot be calculated correctly, and learning fails. Therefore, in this embodiment, an example of correcting an image based on information at the time of sensor characteristic conversion will be described. Note that, although this embodiment shows an example of using a SPAD sensor as a sensor that is not linear in luminance, other sensors may be used.

[0101] An example of the functional configuration of an image processing system according to this embodiment is shown in the block diagram of Fig. 12. In Fig. 12, the same reference numerals are given to the same functional units as those shown in Fig. 3, and the description of these functional units will be omitted.

[0102] The operation of the image processing system according to this embodiment will be described with reference to the flowchart in Fig. 13. Note that in Fig. 13, the same processing steps as those shown in Fig. 4 are given the same step numbers, and descriptions of those processing steps will be omitted.

[0103] In step S1301, the generation unit 303 applies imaging simulation to the acquired original image to obtain a degraded image. At this time, the imaging processing simulation 701 obtains a RAW image that is not linear in luminance equivalent to a SPAD sensor. In this embodiment, the generation unit 303 has a conversion unit 1201 having a configuration shown in Fig. 12(c). The conversion unit 1201 sequentially performs three processes: defective pixel correction, linearization processing, and saturation normalization processing.

[0104] First, the converter 1201 convolves a smoothing filter H with the RAW image to reduce the effect of defective pixels on the sensor that deteriorate linearity. Next, the converter 1201 applies the linearization process of the following equation (9). In a SPAD sensor, it is generally determined whether or not a photon is incident on the sensor during a certain very short exposure time, and the number of incidence determinations is counted in a counting interval consisting of a given k exposures to obtain a pixel value v. In this sensor, when the expected value c of the number of incident photons in the counting interval is given, the expected value of the pixel value v is as follows (Non-Patent Document 3): TIFF2025009784000009.tif1450…(8)

[0105] The obtained RAW image corresponding to the SPAD sensor is input to the conversion unit 1201. Here, by applying the following equation (9) obtained by solving equation (8) for c, a response value c that is linear with respect to luminance is obtained from the sensor response value v. TIFF2025009784000010.tif1657…(9)

[0106] Next, in order to prevent the response value c from diverging when k≈v in the above formula, the conversion unit 1201 performs saturation normalization processing using a given upper limit m, clipping / scaling as shown in the following formula (10), to obtain the response value c' of the sensor characteristic conversion result, where b is the number of output bits. TIFF2025009784000011.tif1863…(10)

[0107] The generation unit 303 then applies the development process simulation 702 to the obtained sensor characteristic conversion result to obtain a degraded image. In step S1302, the storage unit 1202 stores sensor characteristic conversion information that is information used during the sensor characteristic conversion. Specifically, the storage unit 1202 stores the number of bits b, the maximum value m, the number of exposures k, and the smoothing filter H as the sensor characteristic conversion information.

[0108] In step S1303, the correction unit 306 performs correction based on the development information on the inference result (correction target) acquired by the image quality improvement unit 305, as in the first embodiment. The result is further corrected using the sensor characteristic conversion information. The correction content is the inverse correction of equations (10) and (9), and is described as the following equation (11) with r as the input and r' as the output. TIFF2025009784000012.tif1475…(11)

[0109] Here, H -1 is an inverse filter. The obtained correction map r' is output to the error calculation unit 307. In step S1304, the error calculation unit 307 calculates the error between the actual measurement statistics extracted from the correction map and the ideal statistics obtained in step S401. Since whether or not a photon arrives in each short exposure time is modeled as a Bernoulli trial with a certain probability, the number of photon counts follows a Bernoulli distribution (Non-Patent Document 3). Therefore, in this embodiment, the error calculation unit 307 calculates the error from the ideal statistics following the Bernoulli distribution using the method of the first embodiment.

[0110] In this embodiment, an example is shown in which the sensor characteristics conversion process is applied to the RAW image, and then the development process is applied, but the development process is not essential. For example, the RAW image after the sensor characteristics conversion may be directly output to the CNN to obtain a RAW image or an RGB image as a result of image quality improvement. In this case, the correction unit 306 performs only the correction using the sensor characteristics conversion information.

[0111] As described above, according to this embodiment, even when an input from a sensor that is not linear in luminance is assumed, the error in the statistics can be correctly calculated, so that artifacts in the inference result image can be reduced in the model of the learning result.

[0112] The numerical values, processing timing, processing order, processing subject, data (information) acquisition method / destination / source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0113] In addition, a part or all of the embodiments described above may be used in appropriate combination. In addition, a part or all of the embodiments described above may be used selectively.

[0114] (Other embodiments) The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0115] The invention of this specification includes the following image processing device, image processing method, and computer program. (Item 1) An acquisition means for acquiring an output of a learning model to which an image generated based on a development process is input, or a difference between the image and the output, as a correction target; an update means for updating parameters of the learning model based on an error between a first statistic, which is a statistic of a correction result obtained by correcting the correction target based on information used in the development process, and a second statistic obtained in advance; An image processing device comprising: (Item 2) 2. The image processing device according to item 1, wherein the acquisition means acquires a residual map indicating noise components of the image as the correction target. (Item 3) moreover, 3. The image processing device according to item 1 or 2, further comprising: a means for acquiring an ideal statistic to which the correction target conforms as the second statistic. (Item 4) The image processing device according to item 1, characterized in that the acquisition means acquires a difference based on the output of the learning model to which an image generated by performing the development process on a reduced image of a RAW image is input and the image as the correction target. (Item 5) moreover, A means for obtaining a raw signal of hard edges; a means for acquiring a line profile after smoothing edges by applying resolution degradation, which is a process for degrading the resolution of the original signal; a means for acquiring a difference signal which is a signal representing a difference between the smoothed line profile and the original signal; means for obtaining the second statistic for the difference signal; 5. The image processing device according to item 4, comprising: (Item 6) 6. The image processing device according to any one of items 1 to 5, wherein the update means updates the parameters so that the error becomes smaller. (Item 7) moreover, 7. The image processing device according to any one of items 1 to 6, further comprising a means for selecting a learning model to be used from among a plurality of learning models based on an evaluation value that evaluates the performance of the learning model based on the error. (Item 8) The image processing device described in any one of items 1 to 7, characterized in that the information used in the development process includes one or more of a Bayer format image, a noise intensity parameter map, a gamma coefficient, a white balance coefficient, a color matrix, and a color filter array. (Item 9) the acquiring means further includes a sensor characteristic conversion means for converting an image so that the RAW image substantially matches the characteristics of a sensor assumed by the development processing; The sensor characteristic conversion method further includes: The update means calculates the first statistics, which are statistics of a correction result obtained by correcting the correction target based on information used in a conversion process when the sensor characteristic conversion is executed. 9. The image processing device according to any one of items 1 to 8, (Item 10) The update means acquires the first statistics, which are statistics of correction results obtained by applying at least one of inverse correction of defective pixel correction, inverse correction of linearization processing, and inverse correction of saturation normalization processing to the correction target. 9. The image processing device according to any one of items 1 to 8, (Item 11) A generating means for generating a degraded image in which the image quality of the original image is degraded; A processing means for performing a process for improving the image quality of the degraded image and acquiring an inference result of the improvement of the image quality; an updating means for updating the image quality improvement parameters based on an error between the statistics of the inference result and statistics given as ideal statistics; Equipped with The inference result is a difference image between the result of the image quality improvement and the degraded image. 13. An image processing device comprising: (Item 12) The current image is an output image of a sensor that obtains a count value of the occurrence of a physical phenomenon, The ideal statistic follows a Bernoulli distribution. 12. The image processing device according to item 11, (Item 13) An acquisition means for acquiring an output of a learning model to which an image generated based on a sensor characteristic conversion process is input, or a difference between the image and the output, as a correction target; an update means for updating a parameter of the learning model based on an error between a first statistic, which is a statistic of a correction result obtained by correcting the correction target based on information used in a sensor characteristic conversion process, and a second statistic obtained in advance; An image processing device comprising: (Item 14) An image processing device according to any one of items 1 to 13, an apparatus for obtaining a high-quality image of an image before development by inputting the image before development into a learning model based on the parameters updated by the update means and performing a calculation of the learning model; An image processing system comprising: (Item 15) An image processing method performed by an image processing device, comprising: an acquisition step in which an acquisition means of the image processing device acquires, as a correction target, an output of a learning model to which an image generated based on a development process is input, or a difference between the image and the output; an updating step in which an updating means of the image processing device updates parameters of the learning model based on an error between a first statistic, which is a statistic of a correction result obtained by correcting the correction target based on information used in the development process, and a second statistic obtained in advance; An image processing method comprising: (Item 16) An image processing method performed by an image processing device, comprising: a generation step in which a generation means of the image processing device generates a degraded image in which the image quality of a current image is degraded; a processing step in which a processing means of the image processing device performs processing for improving the image quality of the degraded image, thereby acquiring an inference result of the image quality improvement; an updating step in which an updating means of the image processing device updates the image quality improvement parameters based on an error between a statistic of the inference result and a statistic given as an ideal statistic; Equipped with The inference result is a difference image between the result of the image quality improvement and the degraded image. 13. An image processing method comprising: (Item 17) An image processing method performed by an image processing device, comprising: an acquisition step in which an acquisition means of the image processing device acquires, as a correction target, an output of a learning model to which an image generated based on a sensor characteristic conversion process is input, or a difference between the image and the output; an updating step in which an updating means of the image processing device updates parameters of the learning model based on an error between a first statistic, which is a statistic of a correction result obtained by correcting the correction target based on information used in a sensor characteristic conversion process, and a second statistic obtained in advance; An image processing method comprising: (Item 18) A computer program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 13.

[0116] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0117] 301: Acquisition unit 302: Acquisition unit 303: Generation unit 304: Storage unit 305: Image quality improvement unit 306: Correction unit 307: Error calculation unit 308: Update unit 309: Acquisition unit 310: Generation unit 311: Inference unit

Claims

1. An acquisition means for acquiring an output of a learning model to which an image generated based on a development process is input, or a difference between the image and the output, as a correction target; an update means for updating parameters of the learning model based on an error between a first statistic, which is a statistic of a correction result obtained by correcting the correction target based on information used in the development processing, and a second statistic obtained in advance; An image processing device comprising:

2. The image processing apparatus according to claim 1 , wherein the acquisition means acquires a residual map indicating a noise component of the image as the correction target.

3. moreover, 2. The image processing apparatus according to claim 1, further comprising: means for acquiring an ideal statistic to which the correction target conforms as the second statistic.

4. The image processing device according to claim 1, characterized in that the acquisition means acquires, as the correction target, a difference based on the output of the learning model to which an image generated by performing the development process on a reduced image of a RAW image is input and the image itself.

5. moreover, A means for obtaining a raw signal of hard edges; a means for acquiring a line profile after smoothing edges by applying resolution degradation, which is a process for degrading the resolution of the original signal; a means for acquiring a difference signal which is a signal representing a difference between the smoothed line profile and the original signal; means for obtaining the second statistic for the difference signal; The image processing device according to claim 4 , further comprising:

6. 2. The image processing apparatus according to claim 1, wherein said updating means updates said parameters so that said error becomes smaller.

7. moreover, 2. The image processing apparatus according to claim 1, further comprising: means for selecting a learning model to be used from a plurality of learning models on the basis of an evaluation value that evaluates the performance of the learning model based on the error.

8. 2. The image processing apparatus according to claim 1, wherein the information used in the development process includes one or more of a Bayer format image, a noise intensity parameter map, a gamma coefficient, a white balance coefficient, a color matrix, and a color filter array.

9. the acquiring means further includes a sensor characteristic conversion means for converting an image so that the RAW image substantially matches the characteristics of a sensor assumed by the development processing; The sensor characteristic conversion method further includes: The update means calculates the first statistics, which are statistics of a correction result obtained by correcting the correction target based on information used in a conversion process when the sensor characteristic conversion is executed.

2. The image processing device according to claim 1,

10. The update means acquires the first statistics, which are statistics of correction results obtained by applying at least one of inverse correction of defective pixel correction, inverse correction of linearization processing, and inverse correction of saturation normalization processing to the correction target.

2. The image processing device according to claim 1,

11. A generating means for generating a degraded image in which the image quality of the original image is degraded; A processing means for performing a process for improving the image quality of the degraded image and acquiring an inference result of the improvement of the image quality; an updating means for updating the image quality improvement parameters based on an error between the statistics of the inference result and statistics given as ideal statistics; Equipped with The inference result is a difference image between the result of the image quality improvement and the degraded image.

13. An image processing device comprising:

12. The current image is an output image of a sensor that obtains a count value of the occurrence of a physical phenomenon, The ideal statistic follows a Bernoulli distribution. The image processing device according to claim 11 .

13. An acquisition means for acquiring an output of a learning model to which an image generated based on a sensor characteristic conversion process is input, or a difference between the image and the output, as a correction target; an update means for updating a parameter of the learning model based on an error between a first statistic, which is a statistic of a correction result obtained by correcting the correction target based on information used in a sensor characteristic conversion process, and a second statistic obtained in advance; An image processing device comprising:

14. An image processing device according to any one of claims 1 to 13, an apparatus for obtaining a high-quality image of an image before development by inputting the image before development into a learning model based on the parameters updated by the update means and performing a calculation of the learning model; An image processing system comprising:

15. An image processing method performed by an image processing device, comprising: an acquisition step in which an acquisition means of the image processing device acquires, as a correction target, an output of a learning model to which an image generated based on a development process is input, or a difference between the image and the output; an updating step in which an updating means of the image processing device updates parameters of the learning model based on an error between a first statistic, which is a statistic of a correction result obtained by correcting the correction target based on information used in the development process, and a second statistic obtained in advance; An image processing method comprising:

16. An image processing method performed by an image processing device, comprising: a generation step in which a generation means of the image processing device generates a degraded image in which the image quality of a current image is degraded; a processing step in which a processing means of the image processing device performs processing for improving the image quality of the degraded image, thereby acquiring an inference result of the image quality improvement; an updating step in which an updating means of the image processing device updates the image quality improvement parameters based on an error between a statistic of the inference result and a statistic given as an ideal statistic; Equipped with The inference result is a difference image between the result of the image quality improvement and the degraded image.

13. An image processing method comprising:

17. An image processing method performed by an image processing device, comprising: an acquisition step in which an acquisition means of the image processing device acquires, as a correction target, an output of a learning model to which an image generated based on a sensor characteristic conversion process is input, or a difference between the image and the output; an updating step in which an updating means of the image processing device updates parameters of the learning model based on an error between a first statistic, which is a statistic of a correction result obtained by correcting the correction target based on information used in a sensor characteristic conversion process, and a second statistic obtained in advance; An image processing method comprising:

18. A computer program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 13.

Images