Image processing apparatus and image processing method

Abstract

To provide a technique for performing appropriate degradation restoration as degradation restoration of an image even when adverse effects such as floating black, white sinking, and floating magenta occur.SOLUTION: The amount of noise in an input image and the amount of fluctuation in luminance of the input image caused by noise or development are estimated, and degradation of the input image is restored based on the amount of noise and the amount of fluctuation.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a technology for restoring deteriorated images. [Background technology]

[0002] One method for achieving different levels of sensitivity in an image sensor is to apply analog and digital gains to the image signal output from the image sensor. It is generally known that applying these gains tends to increase the effect of noise.

[0003] In particular, in dark areas of RAW images captured at high sensitivity, the effects of noise can cause a phenomenon known as "floating black," in which what should be black becomes brighter and closer to gray, or a phenomenon known as "sinking white," in which what should be white becomes dark. Furthermore, when a RAW image with floating black is developed to generate an RGB image, the R and B components become relatively larger than the G component due to the effects of white balance correction, destroying the RGB color balance and causing a color shift known as floating magenta. The degree of floating black, sinking white, and floating magenta varies depending on the amount of noise caused by temperature changes and individual differences in the image sensor.

[0004] Patent Document 1 discloses detecting pixels having pixel values ​​equal to or less than a preset threshold as black pixels for each frame before applying digital gain. Patent Document 1 also discloses a method of suppressing variation in the amount of floating black by adjusting an offset from the number of black pixels in the current frame and the previous frame. Patent Document 2 also discloses a method of reducing noise while suppressing color changes in an image using deep learning (DNN). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-006785 A [Patent Document 2] JP 2021-114180 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the method disclosed in Patent Document 1 is robust to temperature changes of the same sensor, but it is difficult to cover individual differences of sensors. Therefore, the method disclosed in Patent Document 1 has a problem that when noise is reduced using DNN, it is specialized for the learned sensor.

[0007] Furthermore, in the method disclosed in Patent Document 2, for an image in which magenta floating has already occurred, magenta is judged to be an original color and noise is reduced, so there is a problem that magenta floating remains. The present invention provides a technology for performing appropriate degradation restoration as image degradation restoration even when adverse effects such as black floating, white sinking, and magenta floating occur. [Means for solving the problem]

[0008] One aspect of the present invention is characterized by comprising an estimation means for estimating an amount of noise in an input image and an amount of fluctuation in luminance of the input image caused by noise or development, and a restoration means for restoring deterioration of the input image based on the amount of noise and the amount of fluctuation. Effect of the Invention

[0009] According to the present invention, even when problems such as floating black, sunken white, or floating magenta occur, appropriate degradation restoration can be performed as image degradation restoration. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing system. [Diagram 2] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing system. [Diagram 3]A diagram explaining the flow of inference and learning. [Figure 4] A diagram explaining the structure of CNN and the learning process. [Diagram 5] 5A and 5B are diagrams for explaining causes of floating black, sinking white, and floating magenta. [Figure 6] FIG. 11 is a diagram for explaining degradation adding processing. [Figure 7] 4 is a flowchart of various processes performed in the image processing system. [Figure 8] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing system. [Figure 9] 4 is a flowchart of various processes performed in the image processing system. [Figure 10] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing system. [Figure 11] 5 is a diagram illustrating suppression of magenta floating. [Figure 12] 4 is a flowchart of various processes performed in the image processing system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] <CNNについて> First, a description will be given of a Convolutional Neural Network (CNN) used in the general information processing technology to which deep learning is applied, which is used in each of the following embodiments. CNN is a technology that repeats convolution of a filter generated by training with an image and then nonlinear operation. The filter is also called a local receptive field. An image obtained by convolving a filter with an image and then performing nonlinear operation is called a feature map. Furthermore, learning is performed using training data (training images or data sets) consisting of a pair of an input image and an output image. Simply put, learning is to generate, from training data, filter values ​​that can convert an input image into a corresponding output image with high accuracy. This will be described in detail later.

[0013] When an image has RGB color channels or when a feature map is composed of multiple images, the filter used for convolution also has multiple channels accordingly. In other words, a convolution filter is expressed as a four-dimensional array that includes the number of channels in addition to the vertical and horizontal sizes and number of images. The process of nonlinear calculation after convolving a filter with an image (or feature map) is expressed in units of layers, such as the nth layer feature map or the nth layer filter. For example, a CNN that repeats filter convolution and nonlinear calculation three times has a three-layer network structure. This type of nonlinear calculation process can be formulated as shown in the following equation (1).

[0014]

number

[0015] In the above formula (1), Wn represents the n-th layer filter, bn represents the n-th layer bias, f represents a non-linear operator, Xn represents the n-th layer feature map, and * represents the convolution operator. Note that the (l) on the right shoulder of each variable indicates that it is the l-th filter / feature map. k represents the identifier of the neurons within the same layer, and K represents the total number of neurons within the same layer. The filters and biases are generated by the learning described later and are collectively also called "network parameters". As the non-linear operation, for example, the sigmoid function or ReLU (Rectified Linear Unit) is used. In the case of ReLU, it is given by the following formula (2).

[0016] [Number]

[0017] As shown in the above formula (2), among the elements of the input vector X, the negative ones become zero and the positive ones remain as they are. As networks using CNN, ResNet in the field of image recognition and SRCNN, its application in the field of super-resolution, are well-known. In both cases, by stacking multiple layers of CNN and performing convolution of the filters multiple times, the accuracy of the processing is improved. For example, ResNet features a network structure with a path that shortcuts the convolution layer, thereby realizing a multi-layer network of up to 152 layers and achieving high-precision recognition approaching the human recognition rate. Note that the reason why the processing is made more accurate by multi-layer CNN is simply that by repeating the non-linear operation multiple times, the non-linear relationship between the input and output can be expressed.

[0018] <Learning of CNN>[ Next, the learning of CNN will be explained. The learning of CNN is generally performed by minimizing the objective function represented by the following formula (3) with respect to the learning data consisting of a pair of an input learning image (student image) and an output learning image (teacher image) corresponding to the input learning image.

[0019] [Number]

[0020] In the above formula (3), L indicates a loss function that measures the error between the correct answer and its estimation. Furthermore, Yi indicates the i-th output training image, and Xi indicates the i-th input training image. Furthermore, F indicates a function that collectively expresses formula (1), which is the calculation performed in each layer of CNN. Furthermore, θ indicates the network parameters (filter and bias). Furthermore, ||Z|| 2 is the L2 norm, and more simply, it is the square root of the sum of the squares of the elements of vector Z. Furthermore, n is the total number of training data used for learning. Since the total number of training data is generally large, in the Stochastic Gradient Descent method (SGD), a portion of the training data is randomly selected and used for learning. This reduces the computational load in learning using a large amount of training data. In addition, various methods are known for minimizing (optimizing) objective functions, such as the momentum method, AdaGrad method, AdaDelta method, and Adam method. The Adam method is given by the following equation (4).

[0021]

number

[0022] In the above formula (4), θ i t is the i-th network parameter at the t-th iteration, and g is θ i t is the gradient of the loss function L with respect to . Additionally, m and v are moment vectors, α is the base learning rate, β1 and β2 are hyperparameters, and ε is a small constant. Note that since there are no guidelines for selecting an optimization method in learning, essentially any method can be used, but it is known that differences in convergence between methods will result in differences in learning time.

[0023] In this embodiment, information processing (image processing) is performed to reduce image degradation using the above-mentioned CNN. Image degradation factors include floating black, white, and floating magenta due to the influence of noise. The image degradation restoration process in this embodiment is a process for generating or restoring an image with no degradation (or very little degradation) from an image with degradation, and will be referred to as image restoration process in the following description.

[0024] [First embodiment] In this embodiment, a method will be described in which the amount of noise in an input image and the amount of magenta floating and whitening caused by the influence of noise are each estimated independently, and degradation of the input image is restored based on the results of each estimation.

[0025] <Image processing system configuration example> First, a hardware configuration example of an image processing system according to this embodiment will be described with reference to the block diagram of Fig. 1. As shown in Fig. 1, the image processing system according to this embodiment has a cloud server 200 and an edge device 100, and the cloud server 200 and the edge device 100 are configured to be able to communicate data with each other via the Internet. In the image processing system according to this embodiment, the cloud server 200 is responsible for generating learning data and learning CNN that restores image quality degradation (hereinafter referred to as degradation restoration learning), and the edge device 100 is responsible for image degradation restoration (hereinafter referred to as degradation restoration inference).

[0026] The network configuration between the cloud server 200 and the edge device 100 is not limited to a specific network configuration. For example, the cloud server 200 and the edge device 100 may be connected by a LAN, or the cloud server 200 and the edge device 100 may be connected via two or more types of networks. The network may be wireless, wired, or a combination of these.

[0027] <Example of hardware configuration for edge devices> The edge device 100 of this embodiment acquires an RGB image input directly or indirectly from the imaging device 10 as an input image to be subjected to image restoration processing. Then, the edge device 100 performs degradation restoration inference on the input image by applying a trained model provided by the cloud server 200. That is, the edge device 100 is an image processing device that reduces noise in an RGB image by using the trained model provided by the cloud server 200 and executing an information processing application program pre-installed in the edge device 100.

[0028] The CPU 101 executes various processes using computer programs and data stored in the RAM 102. As a result, the CPU 101 controls the overall operation of the edge device 100, and executes or controls various processes that will be described as processes performed by the edge device 100.

[0029] The RAM 102 has an area for storing computer programs and data loaded from the ROM 103, the mass storage device 104, or the external storage device 30. The RAM 102 further has an area for storing a trained model and the like received from the cloud server 200 via the network I / F 106, and an area for storing an input image output from the imaging device 10. The RAM 102 further has a work area used when the CPU 101 executes various processes. In this way, the RAM 102 can provide various areas as appropriate.

[0030] The ROM 103 stores setting data for the edge device 100, computer programs and data related to the startup of the edge device 100, computer programs and data related to the basic operation of the edge device 100, and the like.

[0031] The mass storage device 104 is a large-capacity secondary storage device such as an HDD or SSD. The mass storage device 104 stores an OS (operating system), computer programs and data for causing the CPU 101 to execute or control various processes described as processes performed by the edge device 100, and the like. The computer programs stored in the mass storage device 104 may include the above-mentioned information processing application programs. The computer programs and data stored in the mass storage device 104 are loaded into the RAM 102 as appropriate under the control of the CPU 101, and become targets for processing by the CPU 101.

[0032] The network I / F 106 is a communication interface for performing data communication with the cloud server 200 via the Internet. For example, the edge device 100 accesses the cloud server 200 via a web browser pre-installed in the edge device 100 to obtain a trained model for degradation restoration inference.

[0033] The general-purpose I / F 105 is a serial bus interface such as USB, IEEE1394, or HDMI (registered trademark). The edge device 100 acquires computer programs and data from the external storage device 30 (for example, various storage media such as a memory card, a CF card, an SD card, and a USB memory) via the general-purpose I / F 105. The edge device 100 also receives user instructions and information from the input device 20 via the general-purpose I / F 105. The input device 20 is a user interface such as a keyboard, a mouse, and a touch panel screen, and various user instructions and information can be input to the edge device 100 by a user's operation. The edge device 100 also outputs a processing result (image, text, etc.) by the CPU 101 to the display device 40 via the general-purpose I / F 105. As a result, the display device 40 can display the processing result by the edge device 100 with images, text, etc. The display device 40 may be a device having a liquid crystal screen or a touch panel screen, or may be a projection device such as a projector that projects images and text. In addition, the edge device 100 acquires an input image from the imaging device 10 via the general-purpose I / F 105.

[0034] The imaging device 10 may be an imaging device that captures moving images, or may be an imaging device that captures still images periodically or irregularly. When the imaging device 10 is an imaging device that captures moving images, the imaging device 10 outputs images of each frame in the moving images, and the edge device 100 acquires the images as input images. On the other hand, when the imaging device 10 is an imaging device that captures still images periodically or irregularly, the imaging device 10 outputs the still images, and the edge device 100 acquires the still images as input images. The CPU 101, the RAM 102, the ROM 103, the mass storage device 104, the network I / F 106, and the general-purpose I / F 105 are all connected to a system bus 107.

[0035] <Example of cloud server hardware configuration> The cloud server 200 of this embodiment is an image processing device that provides cloud services over the Internet. More specifically, the cloud server 200 generates training data and performs degradation restoration training, and generates a trained model that stores network parameters that are the results of the degradation restoration training. Then, the cloud server 200 provides the trained model to the edge device 100 in response to a request from the edge device 100.

[0036] CPU 201 executes various processes using computer programs and data stored in RAM 203. As a result, CPU 201 controls the overall operation of cloud server 200, and executes or controls various processes that will be described as processes performed by cloud server 200.

[0037] ROM 202 stores setting data for cloud server 200, computer programs and data related to the startup of cloud server 200, computer programs and data related to the basic operation of cloud server 200, and the like.

[0038] The RAM 203 has an area for storing computer programs and data loaded from the ROM 202 or the mass storage device 204, and a work area used when the CPU 201 executes various processes. The RAM 203 also has an area for storing computer programs and data transmitted from the edge device 100 via the network I / F 205. In this way, the RAM 203 can provide various areas as appropriate.

[0039] The mass storage device 204 is a large-capacity secondary storage device such as an HDD or SSD. The mass storage device 204 stores an OS (operating system), computer programs and data for causing the CPU 201 to execute or control various processes described as processes performed by the cloud server 200, and the like. The computer programs and data stored in the mass storage device 204 are loaded into the RAM 203 as appropriate under the control of the CPU 201, and become targets for processing by the CPU 201.

[0040] The network I / F 205 is a communication interface for performing data communication with the edge device 100 via the Internet. For example, the cloud server 200 provides a trained model to the edge device 100 in response to a request from a web browser running on the edge device 100. The CPU 201, the ROM 202, the RAM 203, the mass storage device 204, and the network I / F 205 are all connected to a system bus 206.

[0041] Note that the hardware configuration of the edge device 100 shown in Fig. 1 is an example and can be modified / changed as appropriate. Similarly, the hardware configuration of the cloud server 200 shown in Fig. 1 is an example and can be modified / changed as appropriate.

[0042] For example, cloud server 200 may be implemented by one computer device, or may be implemented by two or more computer devices. When cloud server 200 is implemented by two or more computer devices, learning data may be generated by one computer device, and degradation restoration learning may be performed by the other computer device.

[0043] In addition, in FIG. 1, the number of edge devices 100 connected to the cloud server 200 is one, but two or more edge devices 100 may be connected to the cloud server 200.

[0044] 1, the imaging device 10 and the edge device 100 are separate devices, but the imaging device 10 and the edge device 100 may be integrated to configure a single imaging device. In this case, the single imaging device has the functions of the edge device 100 in addition to the imaging function.

[0045] 1, the cloud server 200 and the edge device 100 are separate devices, but the cloud server 200 and the edge device 100 may be integrated into one device. In this case, this one device can generate learning data, learn degradation restoration, and infer degradation restoration.

[0046] 1, the cloud server 200, the edge device 100, and the imaging device 10 are separate devices, but the cloud server 200, the edge device 100, and the imaging device 10 may be integrated into one device. In this case, this one device can perform imaging, generation of learning data, degradation restoration learning, and degradation restoration inference.

[0047] <Example of functional configuration of the entire image processing system> Next, an example of the functional configuration of the image processing system according to this embodiment will be described with reference to the block diagram of FIG. 2. In this embodiment, a case will be described in which the functional units of the cloud server 200 and the edge device 100 shown in FIG. 2 are both implemented by computer programs. In this embodiment, the functional units of the cloud server 200 and the edge device 100 shown in FIG. 2 will be described as the subject of processing. However, in reality, the functions of the functional units are realized by the CPU 201 executing a computer program corresponding to the functional units of the cloud server 200. Similarly, the functions of the functional units are realized by the CPU 101 executing a computer program corresponding to the functional units of the edge device 100. Note that one or more of the functional units shown in FIG. 2 may be implemented by hardware. First, the functional units of the edge device 100 will be described.

[0048] The inference unit 111 estimates the amount of noise and the amount of brightness fluctuation of the input image 115 acquired from the imaging device 10 using the trained model 222 acquired from the cloud server 200, and performs degradation restoration inference based on the result of the estimation. Here, in this embodiment, an RGB image in which each pixel has an RGB pixel value is used as the input image 115. Also, the trained model 222 is a CNN that has been trained to estimate the amount of noise, estimate the amount of brightness fluctuation, and perform degradation restoration inference.

[0049] The estimation unit 112 estimates the amount of noise in the input image 115 by using the trained model 222. Fig. 3(A) is a diagram showing a processing flow in the inference unit 111. The trained model 222 includes CNN301, CNN303, and CNN305. As shown in Fig. 3(A), the estimation unit 112 inputs the input image 115 to the CNN301, and performs calculations of the CNN301 by repeating convolution calculations and nonlinear calculations using filters represented by equations (1) and (2) multiple times, thereby outputting a noise amount estimation result 302 which is an estimation result of the amount of noise in the input image 115.

[0050] 4(A) and 4(B) are diagrams for explaining the structure of CNN and the flow of inference and learning. First, the processing of CNN 301 will be explained with reference to FIG. 3(A) and FIG. 4(A). CNN 301 has a plurality of filters 401 that perform the calculation of the above-mentioned formula (1). First, the estimation unit 112 inputs an input image 115 to CNN 301, and sequentially applies the filters 401 to the input image 115 to calculate a feature map (not shown). Then, the estimation unit 112 calculates the result of applying the last filter 401 as a noise amount estimation result 302. The noise amount estimation result 302 has the same channel as the input image 115.

[0051] The estimation unit 113 estimates the amount of luminance fluctuation of the input image 115 caused by floating black, white sinking, floating magenta, etc., using the trained model 222. As shown in FIG. 3A, the estimation unit 113 inputs the input image 115 to the CNN 303, and performs the calculation of the CNN 303 by repeating the convolution calculation and nonlinear calculation using the filters represented by the formulas (1) and (2) multiple times, thereby outputting the luminance fluctuation amount estimation result 304, which is the estimation result of the luminance fluctuation amount of the input image 115. The processing of the CNN 303 will be described with reference to FIG. 3A and FIG. 4A. The CNN 303 has multiple filters 401 that perform the calculation of the above-mentioned formula (1). First, the estimation unit 113 inputs the input image 115 to the CNN 303, and sequentially applies the filters 401 to the input image 115 to calculate a feature map (not shown). Then, the estimation unit 113 calculates the result of applying the last filter 401 as the luminance fluctuation amount estimation result 304. The luminance fluctuation amount estimation result 304 has the same channel as the input image 115. Here, the luminance fluctuation of the input image 115 occurs due to floating black, white sinking, and floating magenta. The causes of floating black, white sinking, and floating magenta will be described below in order with reference to FIG. 5.

[0052] First, let us explain about black floating. Figure 5(A) shows the distribution of pixel values ​​in the optical black area (OB area) when an image is captured under conditions where digital gain is not applied, and the rectangular image is an example of an OB area. An OB area is an area that is shielded from light by a metal film or the like on the image sensor. The horizontal axis represents the pixel value, and the vertical axis represents the frequency of each pixel value. Also, due to the effects of dark current and readout noise, the distribution tends to accumulate near the offset.

[0053] Figure 5(B) shows the distribution of pixel values ​​in the OB region when digital gain is applied to the conditions of Figure 5(A). In reality, due to hardware limitations, negative values ​​are not possible, so negative values ​​are clipped to 0. This causes the average value, which was originally near the offset, to shift in the positive direction. As a result, "black" cannot be displayed correctly, and the rectangular image in Figure 5(B) appears brighter than the rectangular image in Figure 5(A). This is called "floating black."

[0054] Next, we will explain whitening. Figure 5(C) shows the distribution of pixel values ​​just before the maximum possible value, and an example is the white area in a rectangular image. Figure 5(D) shows the distribution of pixel values ​​when digital gain is applied to the conditions of Figure 5(C), and since it is actually not possible to have values ​​that exceed the maximum value due to hardware limitations, they are clipped to the maximum value. This causes the average value, which was originally just before the maximum value, to shift in the negative direction. As a result, the image becomes darker than its original brightness. This is whitening.

[0055] Finally, we will explain magenta floating. Figure 5(E) shows the distribution of pixel values ​​in the OB region when development processing is applied under the conditions of Figure 5(B). Due to the influence of white balance correction applied during development, the R and B components become relatively larger than the G component, disrupting the RGB color balance. This is magenta floating.

[0056] The above is a detailed description of floating black, white sinking, and floating magenta. The estimation unit 113 estimates the amount of luminance fluctuation caused by these phenomena. Note that the amount of noise and the amount of luminance fluctuation are in a proportional relationship, and for example, the amount of luminance fluctuation caused by processing to fit the amount of noise within the range of an unsigned integer that can be stored at a predetermined bit depth is estimated.

[0057] Although it is difficult to obtain the offset and white balance values ​​from an input image, the offset is generally a preset value, and the white balance is generally a value calculated within the imaging device 10 at the time of imaging. Therefore, by acquiring and utilizing these values ​​in advance, it is possible to improve the estimation accuracy of the amount of variation in luminance.

[0058] The restoration unit 114 acquires the noise amount (noise amount estimation result 302) of the input image 115 estimated by the estimation unit 112 and the luminance fluctuation amount (luminance fluctuation amount estimation result 304) of the input image 115 estimated by the estimation unit 113. Then, the restoration unit 114 performs restoration processing for the degradation of the input image 115 based on the noise amount (noise amount estimation result 302) of the input image 115 and the luminance fluctuation amount (luminance fluctuation amount estimation result 304) of the input image 115. Specifically, as shown in FIG. 3A, the restoration unit 114 inputs the input image 115, the noise amount estimation result 302, and the luminance fluctuation amount estimation result 304 to the CNN 305. Then, the restoration unit 114 performs the calculation of the CNN 305 by repeating the convolution calculation and the nonlinear calculation by the filters represented by the formulas (1) and (2) a plurality of times, thereby outputting the output image 116 after the restoration processing.

[0059] Next, the processing of the CNN 305 will be described with reference to FIG. 3(A) and FIG. 4(B). As shown in FIG. 4(B), the CNN 305 has a plurality of filters 401 and a concatenation layer 402. First, the restoration unit 114 inputs the input image 115, the noise amount estimation result 302, and the brightness variation amount estimation result 304 concatenated or added in the channel direction to the CNN 305 as input data. Next, the restoration unit 114 sequentially applies the filters 401 to the input data to calculate a feature map. Next, the restoration unit 114 concatenates the feature map and the input data in the channel direction by the concatenation layer 402. Furthermore, the restoration unit 114 sequentially applies the filters 401 to the concatenation result, and outputs the output image 116 having the same number of channels as the input image 115 from the final filter.

[0060] Next, each functional unit of the cloud server 200 will be described. The degradation adding unit 211 generates a student image by adding noise to a teacher image extracted from a teacher image group, which is a collection of images without degradation. In the case of this embodiment, the degradation adding unit 211 analyzes the physical characteristics of the imaging device 10, and generates a student image by adding noise corresponding to a wider range of degradation amount than the degradation amount that can occur in the imaging device 10 as a degradation element to the teacher image. The reason for adding noise corresponding to a wider range of degradation amount than the analysis result of the physical characteristics of the imaging device 10 (degradation amount that can occur in the imaging device 10) is to provide a margin and increase robustness since the range of degradation amount differs depending on the individual difference of the imaging device 10. That is, as shown in FIG. 6, the degradation adding unit 211 generates a student image 603 by adding 602 noise based on the physical characteristic analysis result 220, which is the analysis result of the physical characteristics of the imaging device 10, as a degradation element to a teacher image 601 extracted from the teacher image group 219. The degradation adding unit 211 then uses the pair of teacher image 601 and student image 603 as learning data. The degradation adding unit 211 generates a student image corresponding to each teacher image by adding a degradation element 220 to each teacher image included in the teacher image group 219, and generates the pair of the teacher image and the student image as learning data. In this way, the degradation adding unit 211 generates a learning data group 306, which is a collection of learning data (teacher image group and student image group).

[0061] Here, the teacher image group 219 includes various types of images, such as images of nature photos including landscapes and animals, images of people such as portraits and sports photos, and images of man-made objects such as architecture and products. In the case of this embodiment, the teacher image is a RAW image in which each pixel has a pixel value corresponding to one of the RGB colors (therefore, the student image is also a RAW image in which each pixel has a pixel value corresponding to one of the RGB colors). In addition, the physical property analysis result 220 of the imaging device 10 includes the amount of noise for each sensitivity generated by the imaging sensor built into the camera (imaging device). By using these, it is possible to estimate the degree of image quality degradation that occurs for each imaging condition. In other words, by adding the degradation estimated under a certain imaging condition to the teacher image, it is possible to generate an image equivalent to the image obtained during imaging.

[0062] The development unit 212 performs development processing on the teacher images and student images included in the learning data set 306 generated by the degradation adding unit 211, and converts each of the teacher images and student images included in the learning data set 306 from a RAW image to an RGB image. The development processing includes demosaic processing to interpolate pixels, white balance correction to make white appear white, edge emphasis to increase sharpness, gamma correction to correct brightness, color correction to increase vividness, and the like.

[0063] The learning unit 213 acquires network parameters 221 to be applied to the CNN for degradation restoration learning, initializes the weights of the CNN using the acquired network parameters 221, and then performs degradation restoration learning using the learning data generated by the degradation assignment unit 211. The network parameters 221 include initial values ​​of the parameters of the CNN, and hyperparameters indicating the structure and optimization method of the CNN. Fig. 3(B) is a diagram showing the flow of processing in the learning unit 213.

[0064] The estimation unit 214 acquires the learning data set 306 from the degradation adding unit 211, and estimates the amount of noise from the noise 307 added to the student image 308 included in the learning data set 306. Specifically, the estimation unit 214 inputs the student image 308 to the CNN 301, performs the calculations of the CNN 301 by repeating the convolution calculation and the nonlinear calculation using the filters represented by the formulas (1) and (2) a plurality of times, and outputs the noise amount estimation result 310 which is the estimation result of the amount of noise in the student image 308.

[0065] The estimation unit 215 acquires the learning data set 306 from the degradation adding unit 211, and estimates the luminance fluctuation amount of the student image 308 included in the learning data set 306. Specifically, the estimation unit 215 first inputs the student image 308 to the CNN 303, and performs the calculation of the CNN 303 by repeating the convolution calculation and nonlinear calculation using the filters represented by the formulas (1) and (2) multiple times, and outputs the luminance fluctuation amount estimation result 313 which is the estimation result of the luminance fluctuation amount of the student image 308.

[0066] The restoration unit 216 performs restoration processing on the student image 308. Specifically, the restoration unit 216 first inputs the student image 308, the noise amount estimation result 310, and the brightness variation amount estimation result 313 to the CNN 305, and performs the calculations of the CNN 305 by repeating the convolution calculation and nonlinear calculation using the filters represented by equations (1) and (2) multiple times, and outputs the restoration result 316 which is the restoration result of the degradation of the student image 308.

[0067] The error calculation unit 217 inputs the noise 307 and the noise amount estimation result 310 to a loss process 311 which is a loss function calculation, and calculates an error (first error) between the noise 307 and the noise amount estimation result 310. The model update unit 218 inputs the first error calculated by the error calculation unit 217 to an update process 312, and updates the network parameters related to the CNN 301 so that the first error becomes smaller (minimum).

[0068] Furthermore, error calculation unit 217 inputs teacher image 309 and brightness variation amount estimation result 313 to loss processing 314, which is a loss function calculation, and calculates an error (second error) between teacher image 309 and brightness variation amount estimation result 313. Model update unit 218 inputs the second error calculated by error calculation unit 217 to update processing 315, and updates network parameters related to CNN 303 so as to reduce the second error.

[0069] Moreover, the error calculation unit 217 inputs the teacher image 309 and the restoration result 316 to a loss process 317 which is a loss function calculation, and calculates an error (third error) between the teacher image 309 and the restoration result 316. The model update unit 218 inputs the third error calculated by the error calculation unit 217 to an update process 318, and updates the network parameters related to the CNN 305 so as to reduce the third error.

[0070] Here, noise 307, student image 308, noise amount estimation result 310, and brightness variation amount estimation result 313 are all pixel arrays (images) of the same size and with the same number of pixels. Note that although the timing at which the errors are calculated differs, the timing at which the network parameters are updated is the same. Also, CNN301, CNN303, and CNN305 used in the learning unit 213 are the same neural networks as CNN301, CNN303, and CNN305 used in the inference unit 111, respectively.

[0071] The configuration shown in Fig. 2 can be modified / altered as appropriate. For example, one functional unit may be divided into multiple functional units, or two or more functional units may be integrated into one functional unit. Each device shown in Fig. 2 may be realized by two or more devices. In this case, each device is connected via a circuit or a wired or wireless network, and performs data communication with each other to perform cooperative operations, thereby realizing each process according to this embodiment.

[0072] <Processing flow of the entire image processing system> Various processes performed in the image processing system of this embodiment will be described with reference to the flowchart of Fig. 7. First, the flow of an example of degradation restoration learning performed in cloud server 200 will be described with reference to the flowchart of Fig. 7(A).

[0073] In step S701, the degradation adding unit 211 acquires the teacher image group 219 and the physical property analysis result 220 which is the analysis result of the physical properties of the imaging device 10, such as the characteristics of the imaging sensor of the imaging device 10 and the sensitivity during imaging.

[0074] The teacher image is a RAW image of a Bayer array, and is obtained, for example, by capturing an image using the imaging device 10. However, the method of acquiring the teacher image is not limited to a specific method, and for example, an image captured by the imaging device 10 may be uploaded directly to the cloud server 200, or images captured in advance may be stored in a HDD or the like and then uploaded.

[0075] In step S702, the degradation adding unit 211 adds noise based on the physical property analysis result 220 to each teacher image included in the teacher image group 219 acquired in step S701, to generate a student image corresponding to the teacher image. The degradation adding unit 211 adds a noise amount measured in advance based on the physical property analysis result 220 in a preset order or in a random order. In this way, the degradation adding unit 211 generates learning data that is a pair of a teacher image and a student image. Next, the development unit 212 performs development processing on the learning data that is a RAW image, converts it into an RGB image, and sets them as the final learning data.

[0076] In step S703, the estimation unit 214 and the estimation unit 215 acquire the network parameters 221 to be applied to the CNN for degradation restoration learning. The network parameters 221 here include the initial values ​​of the parameters of the CNN, and the hyperparameters indicating the structure and optimization method of the CNN, as described above.

[0077] In step S704, the estimation unit 214 initializes the weights of CNN 301 using the network parameters 221, and then estimates the amount of noise in the student image generated in step S702 using CNN 301. In addition, the estimation unit 215 initializes the weights of CNN 303 using the network parameters 221, and then estimates the amount of luminance fluctuation in the student image generated in step S702 using CNN 303. Then, the restoration unit 216 inputs the student image, the noise amount estimation result, and the luminance fluctuation amount estimation result to CNN 305 and performs calculations on the CNN 305, thereby outputting a restoration result of the degradation of the student image.

[0078] In step S705, the error calculation unit 217 calculates the first error, the second error, and the third error as described above according to the loss function shown in Equation (3). In step S706, the update unit 218 updates the network parameters of CNN301, CNN303, and CNN305 based on the first error, the second error, and the third error calculated in step S705.

[0079] In step S707, the learning unit 213 determines whether or not a learning end condition has been satisfied. The learning end condition is not limited to a specific end condition. For example, the learning end condition may be "the number of times the network parameters have been updated reaches a specified number of times," "the error falls below a threshold," "the change in error (the difference between the previous error and the current error) falls below a threshold," or "the elapsed time since the start of learning exceeds a specified time." In addition, the learning end condition may be configured by combining two or more of such conditions.

[0080] If it is determined that the learning termination condition has been satisfied as a result of such a determination, the processing according to the flowchart in FIG. 7(A) is terminated, and if it is determined that the learning termination condition has not been satisfied, the processing proceeds to step S704.

[0081] Next, an example of degradation restoration inference performed in the edge device 100 will be described with reference to the flowchart in Fig. 7(B). In step S708, the inference unit 111 acquires the trained model 222 generated by the cloud server 200 and the input image 115 to be subjected to degradation restoration processing. The input image 115 may be, for example, an image captured by the imaging device 10, which is directly input, or an image captured in advance and stored in the large-capacity storage device 104 may be read out.

[0082] In step S709, the estimation unit 112 estimates the amount of noise in the input image 115 by using the CNN 301 included in the trained model 222 generated by the learning unit 213. In this manner, the estimation unit 112 inputs the input image 115 to the CNN 301 to which the network parameters updated by the learning unit 213 are applied, and estimates the amount of noise in the same manner as the learning unit 213.

[0083] In step S710, the estimation unit 113 estimates the amount of luminance variation of the input image 115 by using the CNN 303 included in the trained model 222 generated by the learning unit 213. In this manner, the estimation unit 113 inputs the input image 115 to the CNN 303 to which the network parameters updated by the learning unit 213 are applied, and estimates the amount of luminance variation in the same manner as the learning unit 213.

[0084] In step S711, the restoration unit 114 inputs the input image 115, the amount of noise estimated in step S709, and the amount of brightness fluctuation estimated in step S710 to the CNN 305 included in the trained model 222 generated by the learning unit 213. Then, the restoration unit 114 performs degradation restoration of the input image 115 by performing calculations of the CNN 305, and outputs the output image 116 after the restoration process. In this way, the restoration unit 114 performs degradation restoration in the same manner as the learning unit 213, using the CNN 305 to which the network parameters updated by the learning unit 213 are applied.

[0085] In this embodiment, a method has been described in which the amount of noise in an input image and the amount of fluctuation in luminance caused by the influence of noise are estimated independently, and degradation of the input image is restored based on the results of each estimation.

[0086] Until now, measures have often been taken to prevent magenta floating from occurring, and for images in which magenta floating has occurred, while it has been possible to reduce noise, it has not been possible to suppress the magenta floating.

[0087] On the other hand, as in the present embodiment, by estimating the amount of noise and the amount of fluctuation in luminance of an input image using CNN and restoring degradation of the input image based on the results of each estimation, it is possible to achieve both suppression of magenta floating and noise reduction.

[0088] In this embodiment, the learning data is generated in step S702, but it may be generated in a later step. Specifically, it may be configured to generate a student image corresponding to a teacher image in the subsequent degradation restoration learning.

[0089] In this embodiment, learning is performed from unlearned network parameters using a group of teacher images prepared in advance, but the processing of this embodiment may be performed based on learned network parameters.

[0090] In this embodiment, the learning process has been described using RAW images captured with color filters in a Bayer array, but RAW images captured with color filters in other color filter arrays may also be used.

[0091] In this embodiment, an example has been described in which the inference unit 111 outputs only the output image 116 after the restoration process of the input image 115. However, this is not limited to this, and for example, the degradation estimation result output by the estimation unit 114 may also be output together with the output image 116.

[0092] In addition, although the present embodiment does not refer to the output destination of the output image 116, the output destination of the output image 116 is not limited to a specific output destination. For example, the edge device 100 may display the output image 116 on the display device 40, may store the output image 116 in a memory device such as the mass storage device 104, or may transmit the output image 116 to another device such as the cloud server 200 via the network I / F 106.

[0093] In the present embodiment, the edge device 100 uses a trained model to perform restoration from an input image. However, a parameter that assists degradation restoration may be used. For example, the edge device 100 holds a lookup table that estimates in advance the degree of image quality degradation that occurs depending on the imaging conditions such as the size and exposure of the imaging sensor of the imaging device 10. The edge device 100 may adjust the amount of restoration (strength of restoration of degradation) by referring to the lookup table during degradation restoration. That is, the inference unit 111 of the edge device 100 acquires the amount of restoration corresponding to the imaging conditions when the input image was captured from the lookup table. For example, the inference unit 111 may reduce the pixel value of a pixel in a dark area that is higher than the reference value of black according to the amount of restoration, or increase the pixel value of a pixel in a bright area that is lower than the maximum value of the bit depth according to the amount of restoration. In the present embodiment, the order of estimating the amount of noise in the input image and the amount of luminance fluctuation in the input image is not limited, and it is sufficient that both estimated values ​​are calculated when performing degradation restoration.

[0094] In this embodiment, the suppression of magenta floating has been described using an RGB image as an example, but it is also possible to suppress black floating that occurs in a RAW image. In this case, the development unit 212 and the development process in step S702 are unnecessary, and the subsequent processes are performed on the RAW image as is.

[0095] [Second embodiment] In the following embodiments including this embodiment, the difference from the first embodiment will be described, and it is assumed that the embodiment is the same as the first embodiment unless otherwise specified below. In the first embodiment, an example in which the noise amount and the brightness fluctuation amount of an input image are estimated independently, and the degradation of the input image is restored using the result of the estimation will be described. In contrast, in this embodiment, an example in which the noise amount and the brightness fluctuation amount of an input image are simultaneously estimated, and the degradation of the input image is restored using the result of the estimation will be described. In addition, in this embodiment, an example in which degradation estimation and degradation restoration are performed on a RAW image will be described. An example of the functional configuration of the image processing system according to this embodiment will be described using the block diagram of FIG. 8. First, each functional unit of the edge device 800 will be described.

[0096] The estimation unit 802 acquires an input image 804, and estimates a degradation amount that indicates the degree of degradation of the input image 804 using the trained model 820. In this embodiment, a RAW image in a Bayer array is used as the input image 804. The degradation amount to be estimated is the amount of noise and the amount of brightness fluctuation of the input image 804, and a CNN is used to estimate the degradation amount. The estimation unit 802 inputs the input image 804 to the CNN, and performs the CNN operation by repeating the convolution operation and nonlinear operation using the filters represented by the formulas (1) and (2) multiple times, thereby outputting a degradation estimation result that is an estimation result of image degradation of the input image 804.

[0097] The restoration unit 803 performs restoration processing for the degradation of the input image 804 based on the degradation estimation result output by the estimation unit 802. A CNN is used for the restoration processing for the degradation of the input image 804. The restoration unit 803 combines the degradation estimation result output by the estimation unit 802 and the input image 804, inputs them to the CNN, and performs the CNN calculation by repeating the convolution calculation and nonlinear calculation by the filters represented by equations (1) and (2) multiple times, thereby outputting an output image 805 in which the degradation of the input image 804 has been restored.

[0098] Next, a description will be given of each functional unit of the cloud server 810. The degradation assigning unit 811 performs the same process as the degradation assigning unit 211. The learning unit 812 acquires network parameters 819 to be applied to the CNN for degradation restoration learning, initializes the weights of the CNN using the network parameters 819, and then performs degradation restoration learning using a group of learning data generated by the degradation assigning unit 811.

[0099] The estimation unit 813 estimates the amount of degradation contained in the student image included in the learning data acquired from the degradation adding unit 811, and outputs the degradation estimation result. The restoration unit 814 performs a restoration process of the degradation of the student image based on the student image and the degradation estimation result estimated by the estimation unit 813. The error calculation unit 815 has the same function as the error calculation unit 217, and the update unit 816 has the same function as the update unit 218. As described above, the difference between the second embodiment and the first embodiment is that the degradation to be estimated (noise amount and brightness fluctuation amount) is estimated simultaneously.

[0100] <Processing flow of the entire image processing system> Next, various processes performed in the image processing system according to this embodiment will be described with reference to the flowchart in Fig. 9. First, an example of the flow of degradation restoration learning performed in the cloud server 810 will be described with reference to the flowchart in Fig. 9(A). In step S901, the degradation adding unit 811 acquires a teacher image group 817 and a physical property analysis result 818 which is an analysis result of the physical properties of the imaging device 10, such as the characteristics of the imaging sensor of the imaging device 10 and the sensitivity during imaging.

[0101] In step S902, similarly to step S702, the degradation adding unit 811 adds noise based on the physical property analysis result 818 to each teacher image included in the teacher image group 817 acquired in step S901 to generate a student image corresponding to the teacher image. In this way, the degradation adding unit 811 generates learning data that is a pair of a teacher image and a student image.

[0102] In step S903, the estimation unit 813 acquires network parameters 819 to be applied to the CNN for degradation restoration learning, similar to step S703 above. In step S904, the estimation unit 813 initializes the weights of the CNN that estimates the amount of degradation using the network parameters 819, similar to step S704 above. Thereafter, the estimation unit 813 uses the CNN to estimate the amount of noise and the amount of brightness fluctuation of the student image generated in step S902 as the amount of degradation of the student image (degradation estimation result). Then, the restoration unit 814 performs restoration processing of the degradation of the student image based on the student image generated in step S902 and the degradation estimation result estimated by the estimation unit 813, similar to step S704 above.

[0103] In step S905, the error calculation unit 815 calculates the error between the result of the degradation restoration process by the restoration unit 814 and the teacher image according to the loss function shown in equation (3). In step S906, the update unit 816 updates the network parameters of the CNN used in step S904 based on the error calculated in step S905.

[0104] In step S907, the learning unit 812 judges whether or not the learning end condition is satisfied. If it is judged that the learning end condition is satisfied as a result of such judgment, the process according to the flowchart in Fig. 9(A) is terminated, and if it is judged that the learning end condition is not satisfied, the process proceeds to step S904.

[0105] Next, an example of degradation restoration inference performed in the edge device 100 will be described with reference to the flowchart in Fig. 9(B). In step S908, the inference unit 801 acquires the trained model 820 (CNN to which the network parameters updated by the training unit 812 are applied) generated by the cloud server 810 and the input image 804 to be subjected to degradation restoration processing.

[0106] In step S909, the estimation unit 802 estimates the amount of degradation (amount of noise and amount of luminance fluctuation) of the input image 804 using the trained model 820 generated by the training unit 812.

[0107] In step S910, the restoration unit 803 inputs the input image 804 and the amount of noise and the amount of brightness fluctuation estimated in step S909 to the trained model 820. Then, the restoration unit 803 performs a calculation of the trained model 820 to perform degradation restoration of the input image 804, and outputs an output image 805 after the restoration process.

[0108] In this manner, in this embodiment, an example has been described in which a CNN for estimating the amount of noise and the amount of brightness fluctuation of an input image is acquired by learning, the amount of noise and the amount of brightness fluctuation of an input image are simultaneously estimated using the CNN, and the result of the estimation is used to restore degradation of the input image.

[0109] As a result, even if an input image has magenta floating, the noise amount and brightness fluctuation amount of the input image can be simultaneously estimated, and based on the estimation result, it is possible to simultaneously suppress magenta floating and reduce noise. In this way, in this embodiment, since the noise amount and brightness fluctuation amount are simultaneously estimated, the processing can be simplified compared to the first embodiment.

[0110] [Third embodiment] In the first and second embodiments, an example was described in which degradation of an input image was restored based on the estimated results of the amount of noise and brightness fluctuation of the input image. This is a method for outputting an image in which degradation has been restored by suppressing both magenta floating and noise reduction for an input image in which magenta floating and whitening have occurred.

[0111] In this embodiment, the occurrence of black floating, magenta floating, or white sunkenness is determined from the estimated noise amount and brightness fluctuation amount, and if one or more of them are occurring, an example is described in which an image in which magenta floating is suppressed is output by switching to low sensitivity imaging and performing a separate process. As the separate process, in this embodiment, noise reduction is performed on a low sensitivity image in which magenta floating or the like does not occur, and a digital gain is applied to return to the sensitivity before switching, thereby outputting an image in which magenta floating or the like is suppressed.

[0112] An example of the functional configuration of an image processing system according to this embodiment will be described with reference to the block diagram of Fig. 10. First, each functional unit of an edge device 1000 will be described. An estimation unit 1001 acquires an input image 1005, and estimates a degradation amount representing the degree of degradation of the input image 1005 using a trained model 1019 provided by the cloud server 200 in the same manner as the estimation unit 802. Here, in this embodiment, an RGB image is used as the input image 1005.

[0113] The change unit 1002 determines whether or not black floating, magenta floating, or white floating occurs based on the degradation estimation result by the estimation unit 1001, and if one or more of them occur, changes the imaging conditions of the imaging device 10 to reduce exposure. As a guideline, the sensitivity is reduced to the level at which digital gain begins to be applied. This makes it possible to capture an image without black floating, white floating, or magenta floating (for example, an image with the pixel value distribution exemplified in FIG. 11(A)).

[0114] Generally, there are two types of gains for imaging devices when capturing high-sensitivity images: analog gain and digital gain. The former amplifies the charge during the process in which the light exposed to the image sensor is converted into an electric charge, while the latter amplifies the pixel value by software calculation after A / D conversion. Digital gain is applied after the analog gain is applied.

[0115] The reduction unit 1003 acquires an input image 1006 which is an image (subsequent RGB image) subsequent to the input image 1005, and reduces the amount of noise in the input image 1006. The input image 1006 may be an image of a subsequent frame adjacent to the input image 1005, or may be an image of a subsequent frame separated from the input image 1005 by one or more frames.

[0116] Any method may be used to reduce the amount of noise in an image, for example, a trained model that has learned noise reduction processing for low-sensitivity images may be used, or a simple average filter or an edge-preserving filter such as a bilateral filter may be used. Figure 11(B) shows the distribution of pixel values ​​in an image resulting from noise reduction performed on the image with the pixel value distribution in Figure 11(A).

[0117] A gain adjustment unit 1004 applies gain to the image whose noise amount has been reduced by the reduction unit 1003 so that the exposure becomes the same as that of the input image 1005 to generate an output image 1007 , and outputs the output image 1007 .

[0118] Fig. 11(C) shows the pixel value distribution of an image obtained by applying a gain to the image with the pixel value distribution of Fig. 11(B). By reducing noise before the gain is applied, it is possible to suppress floating blacks and floating magenta even if a gain is applied to restore the original brightness.

[0119] Next, a description will be given of each functional unit of the cloud server 1010. The degradation adding unit 1011 acquires a teacher image group 1017 and a physical property analysis result 1018 which is an analysis result of the physical property of the imaging device 10, and performs the same processing as the degradation adding unit 211 and the developing unit 212 to generate a learning data group.

[0120] The learning unit 1013 acquires network parameters 1019, initializes the weights of the CNN using the acquired network parameters 1019, and then performs learning to estimate degradation using the learning data generated by the degradation adding unit 1011 and the developing unit 1012. The estimation unit 1014, the error calculation unit 1015, and the update unit 1016 perform the same operations as the estimation unit 813, the error calculation unit 815, and the update unit 816, respectively.

[0121] <Processing flow of the entire image processing system> Various processes performed in the image processing system of this embodiment will be described with reference to the flowchart of Fig. 12. First, the flow of an example of degradation restoration learning performed in the cloud server 1010 will be described with reference to the flowchart of Fig. 12(A).

[0122] In step S1201, the degradation adding unit 1011 acquires the teacher image group 1016 and the physical property analysis result 1017 which is the analysis result of the physical properties of the imaging device 10, such as the characteristics of the imaging sensor of the imaging device 10 and the sensitivity during imaging.

[0123] In step S1202, the degradation adding unit 1011 adds noise based on the physical property analysis result 1017 to each teacher image included in the teacher image group 1016 acquired in step S1201, as in step S702 above, to generate a student image corresponding to the teacher image. In this way, the degradation adding unit 1011 generates learning data that is a pair of a teacher image and a student image. Next, the development unit 1012 performs development processing on the learning data that is a RAW image, converts it into an RGB image, and sets them as the final learning data.

[0124] In step S1203, the learning unit 1013 acquires network parameters 1019 to be applied to the CNN for degradation restoration learning, similarly to step S703 above. In step S1204, the estimation unit 1014 initializes the weights of the CNN that estimates the amount of degradation using the network parameters 1019, and then estimates the amount of noise and the amount of brightness fluctuation of the student image generated in step S1202 as the amount of degradation of the student image (degradation estimation result).

[0125] In step S1205, the error calculation unit 1015 calculates the error between the degradation estimation result by the estimation unit 1014 and the teacher image according to the loss function shown in equation (3). In step S1206, the update unit 1016 updates the network parameters of the CNN used in step S1204 based on the error calculated in step S1205.

[0126] In step S1207, the learning unit 1013 judges whether or not the learning end condition is satisfied. If it is judged that the learning end condition is satisfied as a result of such judgment, the process according to the flowchart in Fig. 12(A) is terminated, and if it is judged that the learning end condition is not satisfied, the process proceeds to step S1204.

[0127] Next, an example of degradation restoration inference performed in the edge device 100 will be described with reference to the flowchart in Fig. 12(B). In step S1208, the inference unit 1001 acquires the trained model 1019 (CNN to which the network parameters updated by the training unit 1013 are applied) generated by the cloud server 1010 and the input image 1005 to be subjected to degradation restoration processing.

[0128] In step S1209, the inference unit 1001 uses the trained model 1019 to estimate the amount of degradation (amount of noise and amount of brightness fluctuation) of the input image 1005. In step S1210, the change unit 1002 determines whether or not black floating, magenta floating, or white sinking has occurred based on the degradation estimation result by the estimation unit 1001, and if any one or more of them has occurred, changes the imaging conditions of the imaging device 10 to reduce exposure.

[0129] In step S1211, the reduction unit 1003 acquires the input image 1006. In step S1212, the reduction unit 1003 reduces the amount of noise in the input image 1006 acquired in step S1211.

[0130] In step S1213, the gain adjustment unit 1004 generates an output image 1007 by applying gain to the input image 1006 whose noise amount has been reduced in step S1212 so that the exposure is the same as that of the input image 1005, and outputs the output image 1007.

[0131] In this embodiment, an example has been described in which the occurrence of black floating, magenta floating, or white sunken is determined from the estimated noise amount and brightness fluctuation amount, and if one or more of these occur, the image is switched to low sensitivity imaging, noise is reduced, and then a gain is applied to restore the original brightness.

[0132] For example, when shooting video at a high sensitivity such as ISO 102400, if magenta floating occurs, change to a low sensitivity such as ISO 1600, and apply gain (6 steps) to the low sensitivity image to restore the original brightness after noise reduction.

[0133] As a result, since the difficulty of performing degradation restoration without causing magenta floating in the first place is lower than that of degraded restoration of an image that already has magenta floating, it is possible to obtain a more accurate degradation restoration result. Therefore, although the need to switch the imaging conditions can be time-consuming, it is an effective method for obtaining a better degradation restoration result.

[0134] In the present embodiment, an example has been described in which, when magenta floating occurs, gain is applied to the result of switching to low sensitivity shooting and performing noise reduction, but the present invention is not limited to this. For example, processing may be performed using a neural network that simultaneously performs noise reduction and gain application.

[0135] In this embodiment, the suppression of magenta floating has been described using an RGB image as an example, but it is also possible to suppress black floating that occurs in a RAW image. In this case, the development unit 1012 and the development process in step S1202 are unnecessary, and the subsequent processes are performed on the RAW image as is.

[0136] The configuration of the device described in each of the above embodiments may be modified or changed as appropriate depending on the specifications and various conditions (such as the conditions and environment in which the device is used) of the device to be actually used.

[0137] The numerical values, processing timing, processing order, processing subject, data (information) acquisition method / destination / source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0138] In addition, a part or all of the embodiments described above may be used in appropriate combination. In addition, a part or all of the embodiments described above may be used selectively.

[0139] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0140] The invention of this specification includes the following image processing device, image processing method, and computer program. (Item 1) An estimation means for estimating a noise amount of an input image and a fluctuation amount of luminance of the input image caused by noise or development; a restoration means for restoring deterioration of the input image based on the amount of noise and the amount of fluctuation; An image processing device comprising: (Item 2) 2. The image processing device according to item 1, wherein the estimation means estimates the amount of noise using a trained CNN for estimating the amount of noise. (Item 3) 3. The image processing device according to item 1 or 2, wherein the estimation means estimates the amount of variation using a trained CNN for estimating the amount of variation. (Item 4) 2. The image processing device according to item 1, wherein the estimation means estimates the amount of noise and the amount of fluctuation using a trained CNN for estimating the amount of noise and the amount of fluctuation. (Item 5) 2. The image processing device according to item 1, characterized in that the amount of noise and the amount of fluctuation are in a proportional relationship, and the estimation means estimates an amount of fluctuation in luminance caused by a process of fitting the amount of noise within a range of an unsigned integer that can be stored at a predetermined bit depth. (Item 6) 6. The image processing device according to any one of items 1 to 5, wherein the estimation means estimates an amount of fluctuation in luminance caused by floating black, sunken white, or floating magenta. (Item 7) An estimation means for estimating a noise amount of an input image captured by an imaging device and a fluctuation amount of luminance of the input image caused by noise or development; a control means for controlling exposure of the imaging device based on the amount of noise and the amount of fluctuation; An image processing device comprising: (Item 8) 8. The image processing device according to item 7, wherein the control means determines whether or not black floating, magenta floating, or white sunken has occurred based on the amount of noise and the amount of fluctuation, and if one or more of these has occurred, changes the imaging conditions of the imaging device to reduce exposure. (Item 9) moreover, Item 9. The image processing device according to item 7 or 8, further comprising a means for reducing the amount of noise in an image subsequent to the input image, and then generating and outputting an image to which a gain has been applied so that the exposure is the same as that of the input image as an output image. (Item 10) An image processing method performed by an image processing device, comprising: an estimation step in which an estimation means of the image processing device estimates a noise amount of an input image and a fluctuation amount of luminance of the input image caused by noise or development; a restoration step of restoring deterioration of the input image based on the amount of noise and the amount of fluctuation by a restoration means of the image processing device; An image processing method comprising: (Item 11) An image processing method performed by an image processing device, comprising: an estimation step in which an estimation means of the image processing device estimates a noise amount of an input image captured by an imaging device and a fluctuation amount of luminance of the input image caused by noise or development; a control step in which a control means of the image processing device controls exposure of the imaging device based on the amount of noise and the amount of fluctuation; An image processing method comprising: (Item 12) A computer program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 9.

[0141] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0142] 100: Edge device 111: Inference unit 112: Estimation unit 113: Estimation unit 114: Restoration unit 115: Input image 116: Output image 200: Cloud server 211: Degradation unit 212: Development unit 213: Learning unit 214: Estimation unit 215: Estimation unit 216: Restoration unit 217: Error calculation unit 218: Update unit 219: Teacher image group 220: Physical property analysis result 221: Network parameters 222: Learned model

Claims

1. An estimation means for estimating a noise amount of an input image and a fluctuation amount of luminance of the input image caused by noise or development; a restoration means for restoring deterioration of the input image based on the amount of noise and the amount of fluctuation; An image processing device comprising:

2. The image processing apparatus according to claim 1 , wherein the estimation means estimates the amount of noise using a trained CNN for estimating the amount of noise.

3. The image processing device according to claim 1 , wherein the estimation means estimates the amount of variation using a trained CNN for estimating the amount of variation.

4. The image processing device according to claim 1 , wherein the estimation means estimates the amount of noise and the amount of fluctuation using a trained CNN for estimating the amount of noise and the amount of fluctuation.

5. 2. The image processing device according to claim 1, wherein the amount of noise and the amount of fluctuation are in a proportional relationship, and the estimation means estimates the amount of fluctuation in luminance caused by a process of fitting the amount of noise within a range of an unsigned integer that can be stored at a predetermined bit depth.

6. 2. The image processing apparatus according to claim 1, wherein the estimation means estimates an amount of luminance fluctuation caused by floating black, floating white, or floating magenta.

7. An estimation means for estimating a noise amount of an input image captured by an imaging device and a fluctuation amount of luminance of the input image caused by noise or development; a control means for controlling exposure of the imaging device based on the amount of noise and the amount of fluctuation; An image processing device comprising:

8. 8. The image processing device according to claim 7, wherein the control means determines whether or not a black float, a magenta float, or a white sink occurs based on the amount of noise and the amount of fluctuation, and when one or more of them are occurring, changes an imaging condition of the imaging device so as to reduce exposure.

9. moreover, 8. The image processing device according to claim 7, further comprising a means for reducing the amount of noise in an image subsequent to the input image, and then generating and outputting an image to which a gain has been applied so that the exposure is the same as that of the input image as an output image.

10. An image processing method performed by an image processing device, comprising: an estimation step in which an estimation means of the image processing device estimates a noise amount of an input image and a fluctuation amount of luminance of the input image caused by noise or development; a restoration step of restoring deterioration of the input image based on the amount of noise and the amount of fluctuation by a restoration means of the image processing device; An image processing method comprising:

11. An image processing method performed by an image processing device, comprising: an estimation step in which an estimation means of the image processing device estimates a noise amount of an input image captured by an imaging device and a fluctuation amount of luminance of the input image caused by noise or development; a control step in which a control means of the image processing device controls exposure of the imaging device based on the amount of noise and the amount of fluctuation; An image processing method comprising:

12. A computer program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 9.

Images