Image processing device, image processing system, and image processing method

Abstract

To provide an image processing device, an image processing system, an image processing method, and a program that suppress overestimation or underestimation of noise intensity in an image.SOLUTION: An image processing system includes: an imaging apparatus having an acquisition unit 302 that acquires a first image generated by applying development processing to a RAW image; and an edge device having a color space conversion unit 305 that converts the first image into a second image that is an image in a color space independent of a development method, and an estimation unit 306 that estimates noise intensity on the basis of the second image. The image processing is controlled according to the estimated noise intensity.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a technique for estimating the intensity of noise in an image. [Background technology]

[0002] In recent years, there has been an increasing demand for real-time image processing, such as object detection, image quality improvement, and region segmentation, for images and videos captured by cameras. Many of these image processing methods have achieved remarkable performance improvements through the use of neural networks. Modern neural networks have many layers, which requires a large amount of computing resources, but it is difficult to provide all of this within the camera. For this reason, a device configuration is often used in which images from a camera are sent to an external edge device with more abundant computing resources, and image processing is performed within the edge device using a neural network.

[0003] In addition, when the time of day or conditions during which images are taken decrease the ambient illumination or the shutter speed becomes faster, the number of photons incident on each pixel decreases, resulting in an image with a lower S / N ratio and strong noise components. Under conditions where the image noise intensity is high, it is desirable to switch the image processing algorithm or neural network model to one that is robust to noise, or to apply high-quality processing to remove noise. Conversely, under conditions where the noise intensity is low, it is desirable to switch the neural network model to a lightweight one dedicated to low noise, or to turn off noise removal processing, in order to improve throughput and reduce the load on the hardware.

[0004] To realize the above-mentioned switching process, it is necessary to estimate the intensity of noise contained in an image based on the image received from a camera in the edge device. For example, Non-Patent Document 1 discloses a method of calculating the noise intensity by calculating the variation of pixel values. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent 05653098 [Non-patent literature]

[0006] [Non-Patent Document 1] FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation Summary of the Invention [Problem to be solved by the invention]

[0007] In general, a camera converts a RAW image, which is an array of pixel values ​​obtained by receiving light with a sensor, into an image suitable for human viewing through a development process and outputs the converted image. In the development process, the amount of variation in pixel values ​​may be amplified or attenuated for each color and texture. Furthermore, the range of this variation may vary depending on the development method. Therefore, when an edge device receives such a developed image and applies the method disclosed in Patent Document 1, overestimation or underestimation of noise intensity may occur. The present invention provides a technology for suppressing overestimation or underestimation of noise intensity in an image. [Means for solving the problem]

[0008] One aspect of the present invention is characterized by comprising an acquisition means for acquiring a first image generated by applying a development process to a RAW image, and an estimation means for converting the first image into a second image which is an image in a color space independent of the development method, and estimating noise intensity based on the second image. Effect of the Invention

[0009] According to the present invention, it is possible to prevent overestimation or underestimation of the intensity of noise in an image. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an image processing system. [Diagram 2] FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing system. [Diagram 3] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing system. [Figure 4] 4 is a flowchart showing the operation of the image processing system. [Diagram 5] 1 is a diagram showing an example of a RAW image acquired by an image sensor included in an image capturing apparatus; [Figure 6] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing system. [Figure 7] 4 is a flowchart showing the operation of the image processing system. [Figure 8] FIG. 11 is a functional block diagram of an entire image processing system according to a third embodiment. [Figure 9] 10 is a flowchart showing a processing flow in an image processing system according to a third embodiment. [Figure 10] FIG. 4 is a graph for explaining the relationship between noise intensity and image processing intensity. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] <CNNについて> First, a description will be given of a Convolutional Neural Network (CNN) used in the general information processing technology to which deep learning is applied, which is used in each of the following embodiments. CNN is a technology that repeats convolution of a filter generated by training with an image and then nonlinear operation. The filter is also called a local receptive field. An image obtained by convolving a filter with an image and then performing nonlinear operation is called a feature map. Furthermore, learning is performed using training data (training images or data sets) consisting of a pair of an input image and an output image. Simply put, learning is to generate, from training data, filter values ​​that can convert an input image into a corresponding output image with high accuracy. This will be described in detail later.

[0013] If an image has RGB color channels, or if a feature map is composed of multiple images, the filter used for convolution also has multiple channels accordingly. In other words, a convolution filter is expressed as a four-dimensional array, which includes the number of channels in addition to the vertical and horizontal sizes and number of filters.

[0014] The process of convolving a filter with an image (or feature map) and then performing nonlinear calculations is expressed in units of layers, such as the nth layer feature map or the nth layer filter. For example, a CNN that repeats filter convolution and nonlinear calculations three times has a three-layer network structure. This type of nonlinear calculation process can be formulated as shown in the following equation (1).

[0015]

number

[0016] In the above formula (1), Wn represents the filter of the n-th layer, bn represents the bias of the n-th layer, f represents a non-linear operator, Xn represents the feature map of the n-th layer, and * represents a convolution operator. Note that the (l) on the right shoulder of each variable indicates that it is the l-th filter / feature map. The filters and biases are generated by the learning described later and are collectively also called "network parameters". For non-linear operations, for example, the sigmoid function or ReLU (Rectified Linear Unit) is used. In the case of ReLU, it is given by the following formula (2).

[0017]

Number

[0018] As shown in the above formula (2), among the elements of the input vector X, the negative ones become zero and the positive ones remain as they are. As networks using CNN, ResNet in the field of image recognition and its application RED-Net in the field of noise removal are well-known. In both cases, by stacking multiple layers of CNN and performing convolution of filters multiple times, high-precision processing is achieved. For example, ResNet features a network structure with a path that shortcuts the convolutional layers, thereby realizing a 152-layer multi-layer network and achieving high-precision recognition approaching the human recognition rate. The reason why the processing is made more accurate by multi-layer CNN is simply that by repeating non-linear operations multiple times, a non-linear relationship between the input and output can be expressed.

[0019] <Learning of CNN> Next, the learning of CNN will be explained. The learning of CNN is generally performed by minimizing the objective function represented by the following formula (3) for the training data consisting of a pair of an input training image (student image) and an output training image (teacher image) corresponding to the input training image.

[0020]

Number

[0021] In the above formula (3), L indicates a loss function that measures the error between the correct answer and its estimate. Yi indicates the i-th output training image, and Xi indicates the i-th input training image. F indicates a function that compiles the formula (1), which is the calculation performed in each layer of the CNN. θ indicates the network parameters (filter and bias). ||Z||2 is the L2 norm, which simply indicates the square root of the sum of the squares of the elements of the vector Z. n indicates the total number of training data used for learning. Since the total number of training data is generally large, in the Stochastic Gradient Descent (SGD) method, a portion of the training data is randomly selected and used for learning. This reduces the computational load in learning using a large amount of training data. Various methods are known for minimizing (optimizing) the objective function, such as the momentum method, the AdaGrad method, the AdaDelta method, and the Adam method. The Adam method is given by the following formula (4).

[0022]

number

[0023] In the above formula (4), θ i t is the i-th network parameter at the t-th iteration, and g is θ i t is the gradient of the loss function L with respect to . Additionally, m and v are moment vectors, α is the base learning rate, β1 and β2 are hyperparameters, and ε is a small constant. Note that since there are no guidelines for selecting an optimization method in learning, essentially any method can be used, but it is known that differences in convergence between methods will result in differences in learning time.

[0024] [First embodiment] In this embodiment, a case where noise removal processing is performed as image processing using CNN will be taken as an example, and an example of estimating noise intensity that serves as an index for switching between execution / non-execution of noise removal will be described.

[0025] First, a configuration example of an image processing system according to this embodiment will be described with reference to the block diagram of Fig. 1. As shown in Fig. 1, the image processing system according to this embodiment includes a cloud server 200, an edge device 100, and an imaging device 10. The cloud server 200 and the edge device 100 are configured to be able to communicate data with each other via the Internet. The edge device 100 and the imaging device 10 are also configured to be able to communicate data with each other via a wired and / or wireless network.

[0026] The network configuration between the cloud server 200 and the edge device 100 is not limited to a specific network configuration. For example, the cloud server 200 and the edge device 100 may be connected by a LAN, or the cloud server 200 and the edge device 100 may be connected via two or more types of networks. The network may be wireless, wired, or a combination of these. Similarly, the network configuration between the edge device 100 and the imaging device 10 is not limited to a specific network configuration.

[0027] First, the cloud server 200 will be described. The cloud server 200 generates learning data for "noise removal processing, which is processing for removing noise in an input image." The cloud server 200 then performs a learning process for a CNN using the generated learning data to update the network parameters of the CNN, thereby generating a CNN having the updated network parameters as a trained model.

[0028] Next, the edge device 100 will be described. The edge device 100 acquires an "image generated by applying a development process to a RAW image" from the imaging device 10 as an input image. The edge device 100 generates a conversion result image by performing an inverse conversion of the development process on the input image, and estimates the intensity of noise in the conversion result image. If the noise intensity is equal to or greater than a threshold, the edge device 100 performs a noise removal process on the conversion result image using a trained model acquired from the cloud server 200, and outputs an image obtained by the noise removal process as an output image. For example, a user applies a noise removal process to an image using an image processing application installed in the edge device 100. On the other hand, if the noise intensity is less than the threshold, the edge device 100 does not perform the noise removal process, and outputs the conversion result image as an output image.

[0029] Next, a description will be given of the imaging device 10. The imaging device 10 may be an imaging device that captures moving images, or an imaging device that captures still images periodically or irregularly.

[0030] When the imaging device 10 is an imaging device that captures moving images, the imaging device 10 applies a development process to the RAW image of each frame in the moving image, and outputs the image of the frame to which the development process has been applied as an input image to the edge device 100.

[0031] In addition, when the imaging device 10 is an imaging device that captures still images periodically or irregularly, the imaging device 10 applies a development process to the still images (RAW images) and outputs the image obtained by the development process to the edge device 100 as an input image.

[0032] Next, an example of the hardware configuration of the image processing system according to this embodiment will be described with reference to the block diagram of Fig. 2. First, an example of the hardware configuration of the edge device 100 will be described.

[0033] The CPU 101 executes various processes using computer programs and data stored in the RAM 102. As a result, the CPU 101 controls the overall operation of the edge device 100, and executes or controls various processes that will be described as processes performed by the edge device 100.

[0034] The RAM 102 has an area for storing computer programs and data loaded from the ROM 103, the mass storage device 104, or the external storage device 30. The RAM 102 also has an area for storing various data received from the cloud server 200 via the network I / F 106, and an area for storing images output from the imaging device 10. The RAM 102 also has a work area used when the CPU 101 executes various processes. In this way, the RAM 102 can provide various areas as appropriate.

[0035] The ROM 103 stores setting data for the edge device 100, computer programs and data related to the startup of the edge device 100, computer programs and data related to the basic operation of the edge device 100, and the like.

[0036] The mass storage device 104 is a large-capacity secondary storage device such as an HDD or SSD. The mass storage device 104 stores an OS (operating system), computer programs and data for causing the CPU 101 to execute or control various processes described as processes performed by the edge device 100, and the like. The computer programs stored in the mass storage device 104 may include computer programs for the image processing applications described above. The computer programs and data stored in the mass storage device 104 are loaded into the RAM 102 as appropriate under the control of the CPU 101, and become targets for processing by the CPU 101.

[0037] The network I / F 106 is a communication interface for performing data communication with the cloud server 200 via the Internet. For example, the edge device 100 accesses the cloud server 200 via a web browser pre-installed in the edge device 100 to obtain a trained model.

[0038] The general-purpose I / F 105 is a serial bus interface such as USB, IEEE1394, or HDMI (registered trademark). The edge device 100 acquires computer programs and data from the external storage device 30 (for example, various storage media such as a memory card, a CF card, an SD card, and a USB memory) via the general-purpose I / F 105. The edge device 100 also receives user instructions and information from the input device 20 via the general-purpose I / F 105. The input device 20 is a user interface such as a keyboard, a mouse, and a touch panel screen, and various user instructions and information can be input to the edge device 100 by a user's operation. The edge device 100 also outputs a processing result (image, text, etc.) by the CPU 101 to the display device 40 via the general-purpose I / F 105. As a result, the display device 40 can display the processing result by the edge device 100 with images, text, etc. The display device 40 may be a device having a liquid crystal screen or a touch panel screen, or may be a projection device such as a projector that projects images and text. In addition, the edge device 100 acquires an image from the imaging device 10 via the general-purpose I / F 105. The CPU 101, the RAM 102, the ROM 103, the mass storage device 104, the network I / F 106, and the general-purpose I / F 105 are all connected to a system bus 107.

[0039] Next, a description will be given of an example of a hardware configuration of the cloud server 200. The cloud server 200 is an image processing device that provides cloud services over the Internet. The cloud server 200 generates and stores a trained model as described above, and provides the trained model to the edge device 100 in response to a request from the edge device 100.

[0040] CPU 201 executes various processes using computer programs and data stored in RAM 203. As a result, CPU 201 controls the overall operation of cloud server 200, and executes or controls various processes that will be described as processes performed by cloud server 200.

[0041] ROM 202 stores setting data for cloud server 200, computer programs and data related to the startup of cloud server 200, computer programs and data related to the basic operation of cloud server 200, and the like.

[0042] The RAM 203 has an area for storing computer programs and data loaded from the ROM 202 or the mass storage device 204, and a work area used when the CPU 201 executes various processes. The RAM 203 also has an area for storing computer programs and data transmitted from the edge device 100 via the network I / F 205. In this way, the RAM 203 can provide various areas as appropriate.

[0043] The mass storage device 204 is a large-capacity secondary storage device such as an HDD or SSD. The mass storage device 204 stores an OS (operating system), computer programs and data for causing the CPU 201 to execute or control various processes described as processes performed by the cloud server 200, and the like. The computer programs and data stored in the mass storage device 204 are loaded into the RAM 203 as appropriate under the control of the CPU 201, and become targets for processing by the CPU 201.

[0044] The network I / F 205 is a communication interface for performing data communication with the edge device 100 via the Internet. For example, the cloud server 200 provides a trained model to the edge device 100 in response to a request from a web browser running on the edge device 100. The CPU 201, the ROM 202, the RAM 203, the mass storage device 204, and the network I / F 205 are all connected to a system bus 206.

[0045] Note that the hardware configuration of the edge device 100 shown in Fig. 1 is an example and can be modified / changed as appropriate. Similarly, the hardware configuration of the cloud server 200 shown in Fig. 1 is an example and can be modified / changed as appropriate.

[0046] For example, cloud server 200 may be implemented by one computer device, or may be implemented by two or more computer devices. When cloud server 200 is implemented by two or more computer devices, generation of training data may be performed by one computer device, and generation (training) of a trained model may be performed by the other computer device.

[0047] In addition, in FIG. 1, the number of edge devices 100 connected to the cloud server 200 is one, but two or more edge devices 100 may be connected to the cloud server 200.

[0048] 1, the imaging device 10 and the edge device 100 are separate devices, but the imaging device 10 and the edge device 100 may be integrated to configure a single imaging device. In this case, the single imaging device has the functions of the edge device 100 in addition to the imaging function.

[0049] Also, for example, while the cloud server 200 and the edge device 100 are separate devices in FIG. 1, the cloud server 200 and the edge device 100 may be integrated into a single device.

[0050] Also, for example, in FIG. 1, the cloud server 200, the edge device 100, and the imaging device 10 are separate devices, but the cloud server 200, the edge device 100, and the imaging device 10 may be integrated into one device.

[0051] Next, we will explain the issues that arise when applying noise intensity estimation using the technology described in Patent Document 1 to a developed image, which is an image obtained by applying a development process to a RAW image captured by an imaging device, and approaches to addressing these issues.

[0052] FIG. 5 shows an example of a RAW image acquired by an image sensor of an imaging device. Patch 501 is an achromatic part, which is an achromatic image area in the RAW image, and patch 504 is a chromatic part, which is a chromatic image area in the RAW image. A RAW image is a color filter array image with linear luminance, and in order to convert it into an image suitable for human viewing, a development process that interpolates missing channel information and converts luminance is applied in the imaging device. In this luminance conversion, as shown in graph 507, a nonlinear conversion is performed on pixel values ​​(input pixel values) in an input image to obtain pixel values ​​(output pixel values) in an output image. As a result, in curves 508 and 509 that represent the "relationship between input pixel values ​​and output pixel values" when nonlinear conversion is performed, pixel values ​​in dark areas are large, compared to a straight line 510 that represents the "relationship between input pixel values ​​and output pixel values" when nonlinear conversion is not performed. This curve is called a gamma curve. Other typical processes that constitute the development process include OB correction, WB (white balance) correction, color matrix application, gain application, etc.

[0053] In addition, since the desired development method differs depending on the scene, imaging devices are often designed so that the user can select and switch between development methods from multiple candidates. For example, if you want to improve the visibility of low-luminance areas, a steep gamma curve like curve 509 is suitable. Patches 503 and 506 are obtained by developing patches 501 and 504 using development method B that uses this gamma curve. On the other hand, if you want to maintain the gradation expression of all luminance, a gentle gamma curve like curve 508 is suitable. Patches 502 and 505 are obtained by developing patches 501 and 504 using development method A that uses this gamma curve.

[0054] With development method A, pixel variations can be seen in both achromatic and chromatic areas, just like with RAW images. On the other hand, with development method B, although pixel value variations can be seen in achromatic areas, the variations are extremely small in chromatic areas due to the influence of a gamma curve that compresses high-brightness areas. In such cases, the method of Patent Document 1 may underestimate the noise intensity for the image due to the influence of patch 506.

[0055] To address this issue, in this embodiment, the image in the color space after development is converted to the color space of a RAW image, and the noise intensity is estimated while eliminating the influence of development. The color space conversion is realized by performing an inverse conversion of the development process based on development parameters such as the gamma curve during development processing, the OB value representing the zero point of the RAW image, the color filter array representing the arrangement of the color filters of each pixel, the WB coefficient representing the ratio of the transmittance of each color of the color filter, the color matrix which is a matrix for performing color conversion to improve the viewing quality, and the gain value which is a coefficient for uniformly multiplying the pixel value of the received light.

[0056] A functional configuration example of the image processing system according to this embodiment is shown in the block diagram of FIG. 3. In this embodiment, the case will be described in which the functional units of the cloud server 200 and the edge device 100 shown in FIG. 3 are both implemented by computer programs. In this embodiment, the functional units of the cloud server 200 and the edge device 100 shown in FIG. 3 will be described as the subject of processing. However, in reality, the functions of the functional units are realized by the CPU 201 executing a computer program corresponding to the functional units of the cloud server 200. Similarly, the functions of the functional units are realized by the CPU 101 executing a computer program corresponding to the functional units of the edge device 100. Note that one or more of the functional units shown in FIG. 3 may be implemented by hardware. In addition, the acquisition unit 302, which is a functional unit of the imaging device 10, is implemented by hardware having an optical lens, a circuit for driving and controlling the optical lens, an imaging sensor, a circuit for performing various image processing including development processing, and the like.

[0057] The configuration shown in FIG. 3 can be modified or changed as appropriate. For example, one functional unit may be divided into multiple functional units, or two or more functional units may be integrated into one functional unit. The configuration shown in FIG. 2 may be realized by two or more devices. In this case, each device is connected via a circuit or a wired or wireless network, and can realize each process described later by performing data communication with each other and performing cooperative operations. The operation of the image processing system according to this embodiment will be described with reference to the flowchart of FIG. 4.

[0058] In step S401, the model acquisition unit 301 acquires a trained model that has been trained for noise removal processing. For example, the architecture described in Non-Patent Document 1 can be used for the trained model. The weight parameters included in the network parameters of the trained model are acquired by learning using the update method shown in Equation (4). Note that the method of acquiring the trained model by the model acquisition unit 301 is not limited to a specific acquisition method. For example, the model acquisition unit 30 may acquire the trained model stored in advance in the external storage device 30 from the edge device 100 via the Internet. Then, the model acquisition unit 301 transmits the acquired trained model to the edge device 100.

[0059] In step S402, the acquisition unit 302 captures a RAW image and applies a development process to the captured RAW image to generate an input image. The development method in the development process can be selected by, for example, the user operating a user interface such as a button of the imaging device 10. In this case, the acquisition unit 302 generates an input image by applying the development process of the development method selected by the user to the RAW image. Note that the method of selecting the development method is not limited to a specific selection method. For example, the imaging device 10 may automatically select the development method based on the average luminance or texture amount of the scene. In the following, the color space before development (i.e., the color space of the RAW image) is referred to as the first color space, and the color space after development (i.e., the color space of the input image) is referred to as the second color space.

[0060] In addition, some neural networks may perform better when they use an image in the first color space as an input. Therefore, depending on the processing of the neural network performed by the forward propagation unit 307, an image in the first color space may be transmitted to the edge device 100 as an input image.

[0061] In this embodiment, the acquisition unit 302 transmits to the edge device 100 a RAW image acquired by imaging and an input image generated by applying a development process to the RAW image.

[0062] In step S403, color space determination unit 303 receives the RAW image and the input image transmitted from imaging device 10. Then, color space determination unit 303 sets pixel values ​​in the RAW image as input pixel values, sets pixel values ​​in the input image as output pixel values, and determines whether the relationship between the input pixel values ​​and the output pixel values ​​(the relationship between pixel values ​​before development processing and pixel values ​​after development processing) is closer to line 510 or curve 508 / 509 in FIG.

[0063] Then, if the color space determination unit 303 determines that the relationship between the pixel values ​​in the RAW image and the pixel values ​​in the input image is closer to the straight line 510 than to the curve 508 (or the curve 509) (i.e., if it determines that no nonlinear pixel value conversion is performed in the development process of the RAW image), it determines that the color space of the input image is the first color space.

[0064] On the other hand, if the color space determination unit 303 determines that the relationship between the pixel values ​​in the RAW image and the pixel values ​​in the input image is closer to the curve 508 (or the curve 509) than to the straight line 510 (i.e., if it determines that a nonlinear pixel value conversion is performed in the development process of the RAW image), it determines that the color space of the input image is the second color space.

[0065] In addition, the relationship between pixel values ​​may be used as well as the relationship between textures, the number of channels, or other elements. Also, the color space of the input image may be determined based on the development method selected by the user operating the imaging device. In this way, the method for determining whether or not nonlinear pixel value conversion is performed in the development process of the RAW image is not limited to a specific method.

[0066] In step S404, if it is determined that the color space of the input image is the second color space, the acquisition unit 304 acquires development parameters corresponding to the development method used by the acquisition unit 302. The development parameters are parameters that determine the content of the processing applied in the development method used, such as a gamma curve, an OB value, a WB coefficient, a color matrix, a color filter arrangement, and a gain value.

[0067] In this embodiment, the edge device 100 holds representative development parameters for each development method. The acquisition unit 304 acquires development parameters corresponding to the development method selected for development processing by the acquisition unit 302 from the table. Note that the method of acquiring the development parameters corresponding to the development method selected for development processing to generate an input image is not limited to a specific acquisition method, and the development parameters may be estimated based on statistics such as the luminance distribution and local variance of the input image, for example.

[0068] It should be noted that if the acquisition unit 304 determines that the color space of the input image is the first color space, the acquisition unit 304 does not perform the process of acquiring development parameters.

[0069] In step S405, if the color space of the input image is determined to be the second color space (developed color space), the color space conversion unit 305 applies a reverse development process to the input image using the development parameters acquired in step S404 to generate a conversion result image. The reverse development process for an input image is a process of applying the reverse conversion of the development process to the input image based on the development parameters. Since the equation for obtaining an input image (developed image) y from a RAW image x is as shown in equation (5), to perform color space conversion to obtain a RAW image x^, which is the reverse development result (conversion result image) from the input image (developed image) y, the color space conversion unit 305 performs the calculation of equation (6).

[0070]

number

[0071]

number

[0072] Here, the coefficient d is the OB value, the matrix W is the WB coefficient, the coefficient g is the gain value, and the matrix M is the color matrix. The function γ() is a table lookup that performs luminance conversion according to the gamma curve, and the inverse function γ -1() is a reverse lookup in the table. The function D() is an interpolation function for converting a one-channel color filter array image into a three-channel RGB image, and the inverse function D -1 () is a function for color thinning processing to convert an RGB image into a color filter array image. The noise intensity of the channel information added by the development process becomes indefinite due to the interpolation process, so if this value is referenced to estimate the noise intensity, the error will increase. Therefore, by excluding the information of the added channel from the noise intensity estimation target according to the color filter array information, it is possible to reduce the error in the noise intensity estimation in the subsequent stage.

[0073] Then, color space conversion unit 305 outputs the generated conversion result image to estimation unit 306. Note that, when color space conversion unit 305 determines that the color space of the input image is the first color space, it outputs the input image (the result of identity conversion of the input image) to estimation unit 306 as the conversion result image.

[0074] In step S406, the estimation unit 306 estimates the noise intensity in the conversion result image. For example, the estimation unit 306 obtains the variation of pixel values ​​in each local region in the conversion result image. For example, the estimation unit 306 obtains the standard deviation of pixel values ​​of pixels in a tap, which is a local region having a size of r (r is an integer equal to or greater than 2) pixels x r pixels centered on a pixel at pixel position (i, j) in the conversion result image. The estimation unit 306 performs such processing for each pixel position in the conversion result image. At this time, flat portion determination is performed to specify a flat portion where there is no texture in the image and the standard deviation can be measured satisfactorily. In addition, a determination is performed based on pixel values ​​to specify a blown-out highlight portion or a crushed-black portion where pixel values ​​are saturated and measurement of the standard deviation is likely to fail. Then, in order to remove outliers from the standard deviations calculated for all local areas (for each pixel position in the converted image) that are flat areas that are not blown out highlights or crushed shadows, the estimation unit 306 determines the median standard deviation among the standard deviations calculated for all local areas as the noise intensity in the converted image.

[0075] The method for estimating the noise intensity in the converted image is not limited to a specific method. For example, the estimation unit 306 may include a process for improving robustness, such as measuring the standard deviation by referring to pixel values ​​of images before and after the time, or excluding from the measurement a portion where the standard deviation is larger than the surroundings as a non-flat area such as an edge. Also, the variation in pixel values ​​may be measured by another method such as application of a filter or frequency analysis, instead of calculating the standard deviation.

[0076] In this way, regardless of whether the color space of the input image is the first color space or the second color space, overestimation or underestimation of noise intensity can be suppressed, and it is possible to pass on an image in a color space suitable for improving the performance of the downstream neural network.

[0077] In step S407, the forward propagation unit 307 applies noise removal processing to the conversion result image. For example, if the noise intensity estimated in step S406 is equal to or greater than a threshold, the forward propagation unit 307 inputs the conversion result image into the trained model received from the cloud server 200, performs calculations on the trained model (forward propagation processing), generates an image that has been subjected to noise removal processing as an output of the trained model, and outputs the generated image as an output image. On the other hand, if the noise intensity estimated in step S406 is less than the threshold, the forward propagation unit 307 outputs the conversion result image as an output image.

[0078] Note that the execution control of the noise removal process according to the noise intensity is not limited to the above-mentioned execution control. For example, the forward propagation unit 307 may perform the noise removal process of a method according to the noise intensity. For example, the forward propagation unit 307 may select one of four types of candidates, "noise removal process using a trained model with high calculation cost but high image quality improvement performance" (first candidate), "noise removal process using a trained model with low calculation cost but low image quality improvement performance" (second candidate), "rule-based noise removal process with lower calculation cost" (third candidate), and "no noise removal" (fourth candidate), according to the noise intensity. For example, the forward propagation unit 307 may select the first candidate if the noise intensity is equal to or greater than the first threshold, select the second candidate if the noise intensity is equal to or greater than the second threshold and less than the first threshold, select the third candidate if the noise intensity is equal to or greater than the third threshold and less than the second threshold, and select the fourth candidate if the noise intensity is less than the third threshold. Then, when the forward propagation unit 307 selects any one of the first to third candidates, it performs noise removal processing on the selected candidate, and when the forward propagation unit 307 selects the fourth candidate, it does not perform noise removal processing.

[0079] The noise removal process does not necessarily have to include a method using a neural network. Alternatively, a noise removal model that can input noise strength from outside as an argument may be adopted, and the noise strength may be given as the argument. In this case, the noise removal model performs noise removal process using network parameters with higher image quality improvement performance as the noise strength given as the argument increases.

[0080] As described above, in the present embodiment, the case where the estimated noise intensity is applied to the noise removal process has been described, but the application of the estimated noise intensity is not limited to the noise removal process. For example, the noise intensity may be applied to other image quality improvement processes such as super-resolution and blur removal, or image analysis processes such as object recognition and object detection, and in this case, the model or method to be applied is switched according to the noise intensity.

[0081] In step S407a, the forward propagation unit 307 judges whether or not to perform CNN learning. The criterion for judging whether or not to perform CNN learning is not limited to a specific criterion. For example, the forward propagation unit 307 judges to perform CNN learning if one or more of the following conditions are satisfied: the error described below is equal to or greater than a threshold, the difference between the previous error and the current error is equal to or greater than a threshold, the number of learning times is less than a threshold, and the elapsed time since the start of learning is less than a threshold. Also, for example, the forward propagation unit 307 judges to perform CNN learning if it is set in advance to perform CNN learning, and judges not to perform CNN learning if it is set in advance not to perform CNN learning.

[0082] As a result of such a determination, if it is determined that CNN learning is to be performed, the process proceeds to step S408, and if it is determined that CNN learning is not to be performed, the process according to the flowchart in FIG. 4 ends.

[0083] In step S408, the update unit 308 obtains an error between the output image (image quality improvement result) output from the forward propagation unit 307 and the correct answer data (true value) corresponding to the output image. Then, the update unit 308 performs a learning process of the trained model by updating the network parameters of the trained model according to the above formula (4) so ​​as to reduce the error.

[0084] In step S409, the update unit 308 determines whether or not a CNN learning termination condition has been satisfied. The CNN learning termination condition is not limited to a specific termination condition. For example, the CNN learning termination condition may be one or more of the following conditions: "error is less than a threshold value," "difference between previous error and current error is less than a threshold value," "number of learning times is greater than or equal to a threshold value," and "elapsed time since learning started is greater than or equal to a threshold value."

[0085] If it is determined that the CNN learning termination condition is not satisfied as a result of such a determination, the process proceeds to step S402, and if it is determined that the CNN learning termination condition is satisfied, the process according to the flowchart in Fig. 4 ends. The output image that is the result of the forward propagation process is output as the final result of the noise removal process.

[0086] The output destination of the output image is not limited to a specific output destination. For example, the forward propagation unit 307 may display the output image on the display device 40, or may store the output image in a memory device such as the mass storage device 104 or the external storage device 30. Also, for example, the forward propagation unit 307 may transmit the output image to another device such as the cloud server 200 via the network I / F 106.

[0087] In this way, according to this embodiment, the input image in the second color space received by the edge device is converted into an image in the first color space that approximately matches the color space before development. In other words, a conversion is performed to realize a state in which "pixel values ​​at the same coordinates in a RAW image and a converted image in the first color space obtained by converting the RAW image approximately match." By applying noise intensity estimation to this converted image, overestimation or underestimation of noise intensity can be suppressed.

[0088] In this embodiment, an example has been described in which a conversion result image in the same color space as the RAW image is obtained, but the color space of the conversion result image is not limited to this, and may be any color space that does not depend on the development method. For example, when performing the inverse development process of equation (6), if it is known in advance that only the gamma conversion γ() differs significantly between development methods, the inverse gamma conversion γ -1It is also possible to perform only (). In this way, the color space after the inverse gamma conversion becomes a color space in which it is possible to calculate noise intensity independent of the development method. In other words, it is sufficient to calculate noise intensity in a color space in which "pixel values ​​at the same coordinates are approximately the same between the conversion result image obtained in this embodiment and an image obtained by developing the original RAW image using a different development method and then color space converting the image." By allowing this conversion, it is possible to simplify the calculation of equation (6) and reduce the calculation cost.

[0089] In addition, in the present embodiment, an example of color space conversion by reverse development processing has been shown, but the image obtained as a result of the reverse development processing is likely to be an image with low brightness and low saturation, and has low visibility. Therefore, the edge device 100 may estimate the noise intensity in a color space that has undergone a given color conversion for the result of the reverse development processing. For example, the edge device 100 may convert the converted image, which is the result of the reverse development processing, into an image in the L*a*b* color space using a uniform conversion coefficient, and estimate the noise intensity in the image. In this way, the visibility of the image can be improved while maintaining the advantage of being able to estimate the noise intensity independent of the development method, making it easier for humans to interpret the result of the noise intensity estimation.

[0090] [Second embodiment] In this embodiment, the difference from the first embodiment will be described, and unless otherwise specified below, it is assumed that the present embodiment is the same as the first embodiment. In the first embodiment, a mode in which the edge device 100 applies color space conversion to an input image so that noise intensity estimation independent of the development method is possible has been described. However, when the size of the input image is large, the calculation cost of the color space conversion in the edge device 100 increases in proportion to the size. Therefore, in this embodiment, the imaging device 10 applies noise intensity estimation to a RAW image and transmits the result of the noise intensity estimation to the edge device 100, thereby reducing the calculation cost of the color space conversion in the edge device 100.

[0091] An example of the functional configuration of an image processing system according to this embodiment is shown in the block diagram of Fig. 6. In Fig. 6, the same reference numerals are given to the same functional units as those shown in Fig. 3, and the description of these functional units will be omitted.

[0092] The operation of the image processing system according to this embodiment will be described with reference to the flowchart in Fig. 7. Note that in Fig. 7, the same processing steps as those shown in Fig. 4 are given the same step numbers, and descriptions of those processing steps will be omitted.

[0093] In step S701, the acquisition unit 601 acquires a RAW image in the same manner as in step S402. In step S702, the estimation unit 602 estimates the noise intensity in the RAW image acquired in step S701 by the same process as in step S406.

[0094] In step S703, the development unit 603 applies development processing to the RAW image acquired in step S701 in the same manner as in step S402, thereby generating a developed image.

[0095] In step S704, the transmission unit 604 transmits to the edge device 100 the developed image generated in step S703 and the noise intensity estimated in step S702 (noise intensity estimation result).

[0096] In step S705, the receiving unit 605 receives the developed image and noise intensity transmitted from the cloud server 200. Note that in step S407, the forward propagation unit 307 treats the developed image as a converted image and applies noise removal processing to the converted image in the same manner as in the first embodiment, but the noise intensity referred to at that time is the noise intensity received in step S705.

[0097] Thus, according to this embodiment, the need for reverse development processing is eliminated compared to the first embodiment, and the calculation cost of the entire image processing system can be reduced. Note that the image processing executed in the edge device 100 is not limited to noise removal processing, and may be any image processing such as image quality improvement processing, recognition processing, object detection, and area division.

[0098] [Third embodiment] In this embodiment, the difference from the first embodiment will be described, and unless otherwise specified below, it is assumed that this embodiment is the same as the first embodiment. In the first embodiment, the estimation unit 306 calculates the variation for an image that has been subjected to development processing by the color space conversion unit 305. However, in the development processing, image processing that has a high-frequency component reduction effect, such as noise removal, demosaic, and defective pixel removal, is generally applied, and the noise component may be smoothed in the spatial direction. As a result, noise generated in a single pixel may spread across multiple pixels, and the calculation required to calculate the variation in pixel values ​​may become complicated. For example, as shown in the image 1001 of FIG. 10, when the noise intensity is weak, the intensity of the image processing applied can be low without any problem, so that the high-frequency components of the noise are preserved, and the tap size (size of the white-framed rectangle) required to cover the noise grains is small. Conversely, as shown in the image 1003, when the noise intensity is strong, it is necessary to apply high-intensity image processing, so that the high-frequency components are reduced, and a larger tap size (size of the white-framed rectangle) is required to cover the noise grains. Furthermore, as shown in image 1002, when the noise intensity is medium, the tap size required to cover the noise grains is an intermediate tap size between the above two tap sizes (the size of the white rectangle).

[0099] The larger the tap size, the greater the computational cost required to calculate the variance. This computational cost for noise intensity estimation is called computational complexity. If this characteristic is not taken into consideration, it will lead to increased error in the noise intensity estimation and increased computational cost.

[0100] The image processing described above during the development process is often performed at a higher intensity in order to suppress the higher noise intensity. Therefore, even if the image processing intensity is unknown, it can be estimated based on the noise intensity.

[0101] Therefore, in this embodiment, the image processing strength is calculated based on the noise strength, and the noise strength is calculated using the calculation complexity derived from the image processing strength. By alternately performing the procedure, the two pieces of information are estimated while being refined successively. This makes it possible to reduce the amount of calculation required for noise strength estimation while achieving a small estimation error.

[0102] An example of the functional configuration of an image processing system according to this embodiment is shown in the block diagram of Fig. 8. In Fig. 8, the same reference numerals are given to the same functional units as those shown in Fig. 3, and the description of these functional units will be omitted.

[0103] The operation of the image processing system according to this embodiment will be described with reference to the flowchart in Fig. 9. Note that in Fig. 9, the same processing steps as those shown in Fig. 4 are given the same step numbers, and descriptions of those processing steps will be omitted.

[0104] In step S901, the intensity estimation unit 801 receives an estimate of a given noise intensity. The noise intensity is expressed as a standard deviation, and the maximum value σmax and minimum value σmin of the estimated standard deviation are received as initial values. A unit impulse response when a development process is applied with that standard deviation is calculated in advance, and the number of pixels in which the impulse spreads is stored in a table. A value corresponding to (σmin, σmax) is extracted from the table to obtain the image processing intensity. As an example, it is assumed here that when the initial value (σmin, σmax)=(15, 70), the minimum / maximum corresponding number of pixels is (4, 32).

[0105] In step S902, the determination unit 802 determines the tap size r from the image processing strength. The tap size is set to a size that covers the impulse spread when it is assumed to be at a minimum. In the above example, r=4. Note that the method of determination is not limited to the above example, and may be determined, for example, as an average size or a maximum size. Also, the computational complexity is not limited to the tap size, and may be, for example, the sample size extracted from the image when measuring the variation, or the number of channels and frames used in measuring the variation.

[0106] In step S406, the estimation unit 306 performs standard deviation measurement with a determined computational complexity to estimate noise intensity. The noise intensity is calculated as the standard deviation of pixel values ​​in a tap, as in the first embodiment. The standard deviation is calculated in a given p% confidence interval. For example, it is assumed that for p=95, an interval (σmin, σmax)=(40, 70) is obtained.

[0107] The intensity estimation unit 801 uses the updated (σmin, σmax) to obtain the image processing intensity again. For example, if a value of (8, 32) is obtained, this is output to the determination unit 802, and the calculation complexity is updated. The noise intensity is estimated based on this value, and for example, an interval of (σmin, σmax)=(40, 55) is obtained. In this manner, a loop process is performed in which the image processing intensity and the noise intensity are sequentially refined.

[0108] In step S903, the storage unit 803 stores intermediate output data of the noise intensity estimation. The intermediate output data is the determination result of a blown-out highlight portion, a crushed-shadow portion, and a flat portion. This intermediate output data is read in the next noise intensity estimation, and is used to calculate the standard deviation of the flat portion that is not a blown-out highlight portion or a crushed-shadow portion. Note that information other than the above may be used as the intermediate output data. For example, information on the variation in pixel values ​​may be saved as intermediate output data, and the standard deviation may be measured based on that value in the next noise intensity estimation.

[0109] In step S904, the end determination unit 804 determines whether to end the loop processing. If the value of (σmax-σmin) is less than a given threshold, it is determined that the possible range of noise intensity has converged to be sufficiently small, and the loop is determined to be ended. On the other hand, if the value of (σmax-σmin) is equal to or greater than the given threshold, it is not determined that the possible range of noise intensity has converged to be sufficiently small, and the process proceeds to step S901. The end determination unit 804 outputs the average value (σmin+σmax) / 2 to the forward propagation unit 307 as the final noise intensity.

[0110] In this embodiment, an example of loop processing in which the range between the maximum and minimum values ​​of the noise intensity is successively reduced is shown, but the method of calculating the noise intensity is not limited to this. For example, the expected value and variance of the noise intensity may be calculated, and the variance may be reduced while updating the expected value for each loop.

[0111] In addition, in this embodiment, an example has been shown in which a reverse developed image is used as input to perform noise intensity estimation and image processing intensity estimation, but the format of the input image is not limited to this and may be other formats, such as a general developed image or an image obtained by image conversion of a RAW image.

[0112] As described above, in this embodiment, the estimated values ​​of the image processing intensity and the noise intensity are successively refined. This makes it possible to reduce the amount of calculation required for estimating the noise intensity and achieve a small estimation error even for an image that has undergone image processing in which noise generated in a single pixel spreads across multiple pixels.

[0113] The numerical values, processing timing, processing order, processing subject, data (information) acquisition method / destination / source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0114] In addition, a part or all of the embodiments described above may be used in appropriate combination. In addition, a part or all of the embodiments described above may be used selectively.

[0115] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0116] The invention of this specification includes the following image processing device, image processing system, image processing method, and computer program. (Item 1) An acquisition means for acquiring a first image generated by applying a development process to a RAW image; an estimation means for converting the first image into a second image which is an image in a color space independent of a development method, and estimating a noise intensity based on the second image; An image processing device comprising: (Item 2) 2. The image processing device according to item 1, wherein the acquisition means acquires the first image generated by applying a development process to a RAW image in an imaging device. (Item 3) The image processing device according to item 1 or 2, characterized in that the estimation means acquires parameters of the development processing, and generates the second image by applying an inverse transformation of the development processing to the first image using the parameters. (Item 4) The image processing device described in item 1 or 2, characterized in that the estimation means acquires parameters of the development process, applies an inverse transformation of the development process to the first image using the parameters, and then performs color space conversion to generate the second image. (Item 5) 5. The image processing device according to item 3 or 4, wherein the estimation means acquires parameters corresponding to a development method of the development process. (Item 6) 6. The image processing device according to any one of items 3 to 5, wherein the parameters include one or more of a gamma curve, an OB value, a WB coefficient, a color matrix, a color filter arrangement, and a gain value in a development process. (Item 7) The estimation means includes: determining whether or not a nonlinear pixel value conversion is performed in the development process; When a nonlinear pixel value conversion is performed in the development process, the first image is converted into the second image, and a noise intensity is estimated based on the second image; When the development process does not include nonlinear pixel value conversion, the noise intensity is estimated based on the first image. 7. The image processing device according to any one of items 1 to 6, (Item 8) moreover, 8. The image processing device according to any one of items 1 to 7, further comprising a control unit that controls execution of image processing in accordance with the noise intensity estimated by the estimation unit. (Item 9) 9. The image processing device according to item 8, wherein the control means executes image processing if the noise intensity is equal to or greater than a threshold, and does not execute image processing if the noise intensity is less than the threshold. (Item 10) 9. The image processing device according to item 8, wherein the control means performs image processing using a method according to the noise intensity. (Item 11) moreover, 11. The image processing device according to any one of items 8 to 10, further comprising a learning unit that learns a model for performing the image processing based on a result of the image processing. (Item 12) moreover, a means for determining a calculation complexity based on an image processing intensity in the development process; means for calculating the image processing strength based on the noise strength; Equipped with The estimation means estimates the noise intensity based on the computational complexity. 2. The image processing device according to item 1, (Item 13) The estimation means performs an update process for refining the estimated noise intensity, moreover, Item 13. The image processing device according to item 12, further comprising a means for determining whether or not updating of the noise strength has been terminated based on the update information of the noise strength. (Item 14) moreover, A means for storing intermediate output data of the noise amount estimation, The estimation means estimates the amount of noise based on the intermediate output data. 13. The image processing device according to item 12, (Item 15) Item 15. The image processing device according to item 14, characterized in that in estimating the amount of noise, the intermediate output data including information regarding the variation in pixel values ​​or the availability of the data for estimating the amount of noise for one or more regions is output. (Item 16) An acquisition means for acquiring a first image generated by applying image conversion to a RAW image; an estimation means for determining a calculation complexity based on an image processing intensity in the image conversion, estimating a noise intensity based on the calculation complexity, and calculating the image processing intensity based on the noise intensity; An image processing device comprising: (Item 17) An image processing system having a first device and a second device, The first device is An acquisition means for acquiring a development result image generated by applying a development process to a RAW image; An estimation means for estimating noise intensity in the RAW image; a transmitting means for transmitting the developed image and the noise intensity to the second device; Equipped with The second device is a receiving means for receiving the developed image and the noise intensity transmitted by the transmitting means; a control means for controlling execution of image processing based on the developed image received by the receiving means in accordance with the noise intensity received by the receiving means; Equipped 1. An image processing system comprising: (Item 18) An image processing method performed by an image processing device, comprising: an acquisition step in which an acquisition means of the image processing device acquires a first image generated by applying a development process to a RAW image; an estimation step in which an estimation means of the image processing device converts the first image into a second image which is an image in a color space independent of a development method, and estimates a noise intensity based on the second image; An image processing method comprising: (Item 19) An image processing method performed by an image processing device, comprising: an acquisition step in which an acquisition means of the image processing device acquires a first image generated by applying image conversion to a RAW image; an estimation step in which an estimation means of the image processing device determines a calculation complexity based on an image processing intensity in the image conversion, estimates a noise intensity based on the calculation complexity, and calculates the image processing intensity based on the noise intensity; An image processing method comprising: (Item 20) 12. A computer program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 11.

[0117] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0118] 10: Imaging device 10 100: Edge device 200: Cloud server 301: Model acquisition unit 302: Acquisition unit 303: Color space determination unit 304: Acquisition unit 305: Color space conversion unit 306: Estimation unit 307: Forward propagation unit 308: Update unit

Claims

1. An acquisition means for acquiring a first image generated by applying a development process to a RAW image; an estimation means for converting the first image into a second image which is an image in a color space independent of a development method, and estimating a noise intensity based on the second image; An image processing device comprising:

2. The image processing apparatus according to claim 1 , wherein the acquisition unit acquires the first image generated by applying a development process to a RAW image in an image capture device.

3. The image processing apparatus according to claim 1 , wherein the estimation means acquires parameters of the development processing, and generates the second image by applying an inverse transformation of the development processing to the first image using the parameters.

4. The image processing device according to claim 1 , characterized in that the estimation means acquires parameters of the development process, applies an inverse transformation of the development process to the first image using the parameters, and then performs a color space conversion to generate the second image.

5. 4. The image processing apparatus according to claim 3, wherein the estimation means acquires a parameter corresponding to a development method of the development process.

6. 4. The image processing apparatus according to claim 3, wherein the parameters include one or more of a gamma curve, an OB value, a WB coefficient, a color matrix, a color filter arrangement, and a gain value in a development process.

7. The estimation means includes: determining whether or not a nonlinear pixel value conversion is performed in the development process; When a nonlinear pixel value conversion is performed in the development process, the first image is converted into the second image, and a noise intensity is estimated based on the second image; When the development process does not include nonlinear pixel value conversion, the noise intensity is estimated based on the first image.

2. The image processing device according to claim 1,

8. moreover, 2. The image processing apparatus according to claim 1, further comprising a control unit that controls execution of image processing in accordance with the noise intensity estimated by the estimation unit.

9. 9. The image processing apparatus according to claim 8, wherein the control means executes image processing if the noise intensity is equal to or greater than a threshold value, and does not execute image processing if the noise intensity is less than the threshold value.

10. 9. The image processing apparatus according to claim 8, wherein the control means performs image processing using a method according to the noise intensity.

11. moreover, 9. The image processing apparatus according to claim 8, further comprising a learning unit for learning a model for performing the image processing based on a result of the image processing.

12. moreover, a means for determining a calculation complexity based on an image processing intensity in the development process; means for calculating the image processing strength based on the noise strength; Equipped with The estimation means estimates the noise intensity based on the computational complexity.

2. The image processing device according to claim 1,

13. The estimation means performs an update process for refining the estimated noise intensity, moreover, 13. The image processing apparatus according to claim 12, further comprising: a means for determining whether or not updating of the noise intensity has been completed based on the update information of the noise intensity.

14. moreover, A means for storing intermediate output data of the noise amount estimation, The estimation means estimates the amount of noise based on the intermediate output data. The image processing device according to claim 12 .

15. 15 . The image processing apparatus according to claim 14 , wherein, in estimating the amount of noise, the intermediate output data includes information regarding one or more regions as to whether the pixel values ​​can be used for estimating the amount of noise or a variation in pixel values.

16. An acquisition means for acquiring a first image generated by applying image transformation to a RAW image; an estimation means for determining a calculation complexity based on an image processing intensity in the image conversion, estimating a noise intensity based on the calculation complexity, and calculating the image processing intensity based on the noise intensity; An image processing device comprising:

17. An image processing system having a first device and a second device, The first device is An acquisition means for acquiring a development result image generated by applying a development process to a RAW image; An estimation means for estimating noise intensity in the RAW image; a transmitting means for transmitting the developed image and the noise intensity to the second device; Equipped with The second device is a receiving means for receiving the developed image and the noise intensity transmitted by the transmitting means; a control means for controlling execution of image processing based on the developed image received by the receiving means in accordance with the noise intensity received by the receiving means; Equipped 1. An image processing system comprising:

18. An image processing method performed by an image processing device, comprising: an acquisition step in which an acquisition unit of the image processing device acquires a first image generated by applying a development process to a RAW image; an estimation step in which an estimation means of the image processing device converts the first image into a second image which is an image in a color space independent of a development method, and estimates a noise intensity based on the second image; An image processing method comprising:

19. An image processing method performed by an image processing device, comprising: an acquisition step in which an acquisition unit of the image processing device acquires a first image generated by applying image conversion to a RAW image; an estimation step in which an estimation means of the image processing device determines a calculation complexity based on an image processing intensity in the image conversion, estimates a noise intensity based on the calculation complexity, and calculates the image processing intensity based on the noise intensity; An image processing method comprising:

20. A computer program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 11.

Images