Image processing device, image processing method and program

Abstract

To solve a problem in which in noise removal, noise cannot be removed properly.SOLUTION: An image processing device comprises: first setting means for setting a reduction region in an image; second setting means for setting an estimated region in the image on the basis of the reduction region; estimation means for estimating noise characteristics of the estimated region; and reduction means for reducing noise of the reduction region in the image by a parameter based on the noise characteristics. The second setting means sets the estimated region so that an area of the estimated region becomes a threshold area or more, if the area of the reduction region is smaller than the threshold area.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In the surveillance market, in extremely low light environments, there is a demand for improved visibility of the target subject, and extremely high gain is applied when shooting with a camera. Applying high gain brightens the captured image, but it also increases noise. However, in surveillance applications, the visibility of the subject is important, even if it means sacrificing image quality for viewing purposes.

[0003] Noise reduction (hereinafter referred to as NR) has been commonly used to reduce noise contained in captured images. NR may be performed inside the camera, or may be performed by an external image processing device that acquires images from the camera. In recent years, noise reduction processing based on deep learning artificial intelligence technology (hereinafter referred to as DLNR) has also been used. DLNR has been confirmed to be more effective than conventional NR, but the large-scale processing required makes the processing time an issue.

[0004] By the way, when image processing that requires a processing time is used, a technique is generally known in which a part of the entire image is designated as a region of interest (hereinafter referred to as ROI) and the target area of ​​image processing is limited to the ROI to reduce the processing time and improve the overall throughput. The ROI function may be provided to the photographer.

[0005] Patent Document 1 discloses a technique for switching whether or not to perform image restoration processing depending on the degree of degradation for each block in an input image. According to Patent Document 1, an area where NR is required as image restoration processing is determined as an ROI and applied, so that NR is applied only to the minimum necessary area, thereby reducing processing time.

[0006] Patent Document 2 discloses an example in which the amount of noise in an ROI and the edges of the area near the ROI are calculated, weighting is performed according to the amount of noise and the edges, and noise reduction processing is performed according to the weighting. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent Publication No. 2021-118403 [Patent Document 2] JP 2007-110338 A Summary of the Invention [Problem to be solved by the invention]

[0008] However, in the conventional technology, noise is estimated based only on the ROI, so when the ROI is narrow, the noise estimation accuracy decreases, and noise may not be appropriately removed during noise removal.

[0009] Therefore, an object of the present invention is to provide an image processing device that can appropriately remove noise even when the area from which noise is to be removed, such as an ROI, is small. [Means for solving the problem]

[0010] In order to solve this problem, for example, an image processing device of the present invention has the following arrangement: a first setting means for setting a reduction area in an image; a second setting means for setting an estimated area in the image based on the reduced area; An estimation means for estimating a noise characteristic of the estimation region; a reduction means for reducing noise in the reduction region of the image using a parameter based on the noise characteristics; When the area of ​​the reduced region is smaller than a threshold area, the second setting means sets the estimated region to be equal to or larger than the threshold area. 13. An image processing device comprising: Effect of the Invention

[0011] According to the present invention, even if the area for removing noise set by a user or the like is small, it is possible to appropriately estimate noise and remove the noise. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing a hardware configuration of an image processing apparatus. [Diagram 2] 4 is a flowchart of image processing executed by the image processing device. [Diagram 3] 11 is a flowchart of a subroutine of the initialization process of S100. [Figure 4] An explanatory diagram of a neural network. [Diagram 5] 13 is a flowchart of a subroutine for image processing setting in S120. [Figure 6] FIG. 13 is a diagram showing an example of setting an ROI. [Figure 7] 13 is a flowchart of a subroutine for calculating a noise estimated region in S130. [Figure 8] FIG. 13 is a diagram showing an example of ROI boundary setting. [Figure 9] An example of noise estimation region boundary setting. [Figure 10] 13 is a flowchart of a subroutine of the noise estimation process in S140. [Figure 11] FIG. 13 is a diagram showing an example of flat area extraction. [Figure 12] 11A and 11B are diagrams showing examples of noise dispersion characteristics for different camera settings. [Figure 13] 13 is a flowchart of a subroutine for the noise reduction process of S150. [Figure 14] FIG. 13 is a diagram showing an example of noise reduction processing for an ROI. [Figure 15] 11A to 11C are diagrams showing examples of noise estimation regions for different camera settings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0014] Hereinafter, an embodiment of the present invention will be described. The image processing device of this embodiment estimates noise from an input image acquired from a connected camera, infers a desired output image for a target area of ​​image processing set as a region of interest (hereinafter, ROI) using a neural network (hereinafter, NN), which is one of deep learning models, and outputs the inferred image. Note that the concept of an image includes still images, videos, and images of one frame of a video. The ROI is an example of a reduction area. In the following description, the concept of an image includes video, still images, and videos. The image may also indicate image data. The image processing device executes learning of an NN, such as bringing the feature distribution of multiple prepared student images closer to the feature distribution of the corresponding teacher image, and optimizes the parameters of the trained neural network, such as weights and biases. This enables the image processing device to perform accurate inference even for input images that have not been trained. The image processing device can perform inference on an input image and generate an inferred image with reduced noise by holding the parameters of the trained neural network obtained by performing multiple learnings according to the characteristics of the camera.

[0015] Fig. 1 is a block diagram showing a hardware configuration of an image processing device 100. The image processing device 100 is, for example, a computer such as a personal computer. Note that, although an image processing device will be described in this embodiment, the processing described later can also be performed in a lens-interchangeable camera such as a lens-interchangeable video camera, a single-lens reflex camera, or a mirrorless single-lens camera. For example, the camera 200 in Fig. 1 may perform processing such as the image processing described later.

[0016] Camera 200 captures a light beam incident from a subject field to generate an image. Camera 200 is equipped with or has a built-in lens (not shown), and includes a zoom lens group, a focus lens group, an iris mechanism, and the like. Camera 200 can change the accumulation time for exposure, and applies gain to the captured image when capturing images in dark places using an automatic exposure function. Camera 200 outputs the captured image.

[0017] 1, the image processing device 100 includes an image input unit 110, a CPU 130, an operation input unit 150, a memory 140, an image processing unit 160, an image output unit 170, a storage unit 180, and a bus 120. The image input unit 110, the CPU 130, the operation input unit 150, the memory 140, the image processing unit 160, the image output unit 170, and the storage unit 180 are connected to each other via the bus 120 so as to be able to transmit and receive data therebetween. The CPU 130 and the image processing unit 160 are examples of a first setting means, a second setting means, an estimation means, and a reduction means.

[0018] The image input unit 110 acquires the image output by the camera 200 and stores it in the memory 140 .

[0019] CPU 130 is an abbreviation for Central Processing Unit. In addition to CPU 130, image processing device 100 may have an MPU (Micro Processing Unit), a QPU (Quantum Processing Unit), a GPU (Graphics Processing Unit), etc. CPU 130 executes each process executed by image processing device 100, which will be described later. For example, CPU 130 executes a program stored in storage unit 180 to realize each function and execute each process.

[0020] The memory 140 is, for example, a random access memory (RAM), and temporarily stores image data, programs, and data required for executing the programs.

[0021] The operation input unit 150 acquires an operation signal of the external controller 300 input by a user or the like. The controller 300 includes a keyboard, a mouse, a touch panel, and a switch. The operation signal acquired by the operation input unit 150 is processed by the CPU 130, and setting operations required for various image processing executed by the image processing unit 160 are performed.

[0022] The image processing unit 160 reads and writes images held in the memory 140, and executes noise estimation processing, ROI processing, NR processing, and UI image generation processing for displaying on a user interface (hereinafter, UI), which will be described later. The image processing unit 160 stores images and the like after processing in the memory 140. The image processing unit 160 may include an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), a GPU, and the like. The image processing unit 160 may also be realized by a program executed by the CPU 130.

[0023] The image output unit 170 outputs the image processed by the image processing unit 160 and stored in the memory 140 to an external monitor 400. The image output unit 170 outputs images from an HDMI (registered trademark) (High Definition Multimedia Interface) terminal and an SDI (Serial Digital Interface) terminal of the image processing device 100.

[0024] The storage unit 180 includes a hard disk drive (HDD), a solid state drive (SSD), etc. The storage unit 180 stores programs, data such as parameters necessary for executing the programs, and data such as images.

[0025] The monitor 400 receives and displays the image output from the image output unit 170 of the image processing device 100. The photographer and the viewer can check, on the monitor 400, the image captured by the camera 200, the menu screen of the image processing device 100, and the image after image processing.

[0026] 1, each function is shown as a separate component, but these may be realized by hardware such as one or more ASICs or programmable logic arrays (PLAs). Also, a programmable processor such as a CPU or MPU may realize the above configuration by executing software.

[0027] Next, the image processing performed by the CPU 130 in the image processing device 100 will be described with reference to the flowchart in Fig. 2. Fig. 2 is a flowchart of the image processing performed by the image processing device 100. When the image processing device 100 is powered on, the CPU 130 loads a computer program stored in the storage unit 180. As a result, the CPU 130 executes the processes in order starting from S100 in Fig. 2. Note that after the CPU 130 executes the processes up to S160 in Fig. 2, it will repeatedly execute the process of S110 again. In the following description, it will be described on the assumption that the CPU 130 executes the processes from S110 to S160 for an input image from the camera 200 for each frame.

[0028] In S100 of Fig. 2, the CPU 130 performs an initialization process of the image processing device 100. Fig. 3 shows a subroutine of the initialization process of S100.

[0029] 3, the CPU 130 executes initialization processing for image input / output. The CPU 130 executes initialization of the image input unit 110 and the image output unit 170 so that image input / output processing can be performed.

[0030] In S102, the CPU 130 executes an initialization process of the ROI. The CPU 130 sets the initial value of the ROI so that all pixels of the input image are the processing target. For example, when the size of the image input to the image input unit 110 is 1920x1080, the CPU 130 sets the start coordinate (0,0) and the end coordinate (1919,1079) as the initial value of the ROI.

[0031] In S103, the CPU 130 executes NR initialization processing. The CPU 130 loads pre-trained NN parameters from the storage unit 180 for the NR processing executed by the image processing unit 160. The storage unit 180 holds trained NN parameters optimized for each model and image quality setting of the camera 200. The CPU 130 sets the NN parameters to be loaded by the image processing unit 160 in correspondence with various models and image quality of the cameras 200 connected to the image processing device 100.

[0032] Here, the NN executed by the image processing unit 160 and the NN parameters to be loaded will be described with reference to Fig. 4. Fig. 4 is a diagram for explaining the NN for an input image. Fig. 4(a) is a diagram for explaining an example of outputting a feature map. Fig. 4(b) is a diagram for explaining an example of outputting an attention layer.

[0033] In the following description, an example will be given in which the NN is a Convolutional Neural Network (CNN), but the present embodiment is not limited to this. The NN may be, for example, a Generative Adversarial Network (GAN), or may have a skip connection. The NN may also be a recurrent type, such as a Recurrent Neural Network (RNN).

[0034] In FIG. 4, an input image 501 represents an image to be input to a NN or a feature map described later. A convolution matrix 503 is a filter that performs a convolution operation on the input image 501. A bias 504 is a value that is added to a result output by a convolution operation 502 of the input image 501 and the convolution matrix 503. A feature map 505 is a convolution operation result after the bias 504 is added to the result output by the convolution operation 502. In FIG. 4, the number of neurons, intermediate layers, and channels is small for simplicity, but the number of neurons and layers, and the number and weights of connections between neurons are not limited to this example. In addition, when the NN shown in FIG. 4 is implemented in an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit), the connections and weights between neurons may be reduced.

[0035] CNN generates a feature map of an input image by performing a convolution operation on the input image using a certain filter. The size of the filter of the CNN is arbitrary. In the next layer, the CNN generates a different feature map by performing a convolution operation on the feature map of the previous layer using a different filter. In addition, in each layer, the CNN multiplies an input signal by a filter and calculates the sum with a bias. Then, the CNN applies an activation function to the calculated result and outputs an output signal in each neuron. The weights and biases in each layer are called NN parameters, and a process of updating the NN parameters is performed in learning. In addition, examples of activation functions include a sigmoid function and a ReLU function, and the CNN of this embodiment uses the Leaky ReLU function shown in the following formula (1), but is not limited to this. In addition, in formula (1), max represents a function that outputs the maximum value of the arguments. f(x)=max(x,x×0.2) Equation (1) In the pre-learning for obtaining the NN parameters, the image processing unit 160 of the present embodiment uses an image having the noise variance characteristic of the camera 200 as a student image, and uses a corresponding noise-free image of the student image as a teacher image. The noise variance characteristic is an example of a noise characteristic. The image processing unit 160 realizes NR by performing learning in which the student image and the teacher image are paired and setting the NN parameters.

[0036] The feature map 505 focuses on noise in the input image, and the image processing unit 160 can learn a region of interest in which the noise components are emphasized by applying another parameter to the feature map 505. The image processing unit 160 calculates an average in the channel direction for an input image 601 in which the input image 501 is divided into channels for each color, for example, to generate an intermediate layer 602. The image processing unit 160 performs convolution on the intermediate layer 602 multiple times using equation (1) to obtain an intermediate layer 603. The image processing unit 160 performs convolution on the intermediate layer 603 so that the output channel becomes 1, to obtain an attention layer 604. The attention layer 604 is an intermediate layer in which features appear in noise regions excluding high-frequency components of the subject. The image processing unit 160 multiplies the attention layer 604 by a noise intensity parameter 605 that specifies the strength of the noise reduction process, thereby obtaining an attention layer 606 in which noise is emphasized. The image processing unit 160 convolves the attention layer 606 together with the above-mentioned input image 501. This allows the image processing unit 160 to adjust, that is, emphasize, the degree of attention paid to noise regions included in the input images 501 and 601, thereby generating an NN capable of performing NR processing.

[0037] Returning to the description of Fig. 3, in S104, the CPU 130 executes operation input initialization. The CPU 130 executes initialization processing for acquiring an operation signal from the controller 300. After executing S104, the CPU 130 ends the subroutine S100 and proceeds to S110 in Fig. 2.

[0038] 2, the CPU 130 stores the input image acquired by the image input unit 110 from the camera 200 in the memory 140. In S120, the CPU 130 performs image processing settings. Fig. 5 shows a flowchart of the image processing subroutine of S120.

[0039] In S121, CPU 130 acquires information about the model of camera 200 selected by a user or the like using controller 300 via operation input unit 150. CPU 130 may acquire information about camera 200 selected by a user from a setting menu of image processing device 100 or the like, or may acquire information about camera 200 automatically detected by image input unit 110 from information embedded in an input image signal.

[0040] In S122, CPU 130 acquires information on the gamma applied to camera 200 selected by a user or the like using controller 300 via operation input unit 150. CPU 130 may acquire information on the gamma selected from a setting menu of image processing device 100 or may acquire information on the gamma automatically detected by image input unit 110 from information embedded in an image signal. Information on the model of camera 200 and information on the gamma of camera 200 are examples of camera information.

[0041] In S123, the CPU 130 determines whether the information of the camera 200 has been changed in S121, or whether the gamma information has been changed in S122. If the information has been changed, the CPU 130 proceeds to S124, and if the information has not been changed, the CPU 130 proceeds to S125.

[0042] In S124, the CPU 130 loads and sets the trained NN parameters corresponding to the model and gamma of the camera 200 selected in S121 and S122 into the image processing unit 160. This results in a state in which the appropriate NN parameters trained according to the connected camera 200 and its gamma setting are applied to the NN loaded into the image processing unit 160. In S124, the image processing unit 160 is in a state in which it can perform inference with the NN, that is, load the input image from the memory 140 and perform NR processing.

[0043] 5, the CPU 130 executes ROI setting. The CPU 130 acquires an operation signal input by a user or the like from the controller 300 via the operation input unit 150, and sets the ROI as a target region to which image processing is applied.

[0044] Fig. 6 is a diagram showing an example of setting an ROI. As shown in Fig. 6, if the size of the input image input to the image input unit 110 is 1920x1080, the CPU 130 can define the entire area of ​​the input image as Area 100. Furthermore, if Area 101 is the ROI, the CPU 130 sets start coordinates (X1, Y1) and end coordinates (X2, Y2) as examples of setting values.

[0045] In S126 of Fig. 5, the CPU 130 calculates the area α of the ROI. According to the example of the ROI setting shown in Fig. 6, the area α can be obtained by the following formula (2). α=|X2-X1|×|Y2-Y1| Formula (2) In S127, CPU 130 loads from storage unit 180 threshold area β associated with the model and gamma of camera 200 selected in S121 and S122. Threshold area β is the minimum area for obtaining accuracy in noise estimation processing in S140, which will be described later. For example, CPU 130 may select and load threshold area β from a table stored in storage unit 180, in which camera 200 model and gamma are associated with threshold area β. After executing S127, CPU 130 ends the image processing setting subroutine of S120 and proceeds to step S130 in FIG. 2.

[0046] In S130 of Fig. 2, the CPU 130 calculates a noise estimated region. Fig. 7 is a flowchart of the noise estimated region calculation subroutine in S130.

[0047] In S131, the CPU 130 compares the area α calculated in S126, ie, using equation (2), with the threshold area β acquired in S127. If the area α is smaller than the threshold area β, the CPU 130 proceeds to S132.

[0048] In S132, the CPU 130 calculates the ratio γ. Here, if the area α of the set ROI is smaller than the threshold area β and the ROI is set as a noise estimation area for noise estimation processing, the accuracy of the noise estimation processing described below may be reduced. Therefore, in this embodiment, the CPU 130 calculates the ratio γ shown in formula (3) to ensure a noise estimation area that satisfies the threshold area β for the area α of the ROI. The CPU 130 calculates the ratio γ as the square root of the threshold area β divided by the area α. γ=√(β / α) Equation (3) In S133 of FIG. 7, the CPU 130 judges whether the width and height of the provisional noise estimation region are outside the region of the input image. In order to obtain a noise estimation region that satisfies the above-mentioned threshold area β, the CPU 130 calculates the width W1 and height H1 of the provisional noise estimation region based on the width and height of the ROI and the ratio γ using the following formulas (4) and (5). The CPU 130 compares the width W0 and height H0 of the input image with the calculated width W1 and height H1, respectively. If the logical sum of formulas (4) and (5) is true, the CPU 130 proceeds to S134. On the other hand, if the logical sum is false, that is, if both the width W1 and height H1 of the provisional noise estimation region are equal to or smaller than the width W0 and height H0 of the input image, the CPU 130 proceeds to S136. It should be noted that both formulas (4) and (5) do not become true. W1=|X2-X1|×γ>W0 Equation (4) H1=|Y2-Y1|×γ>H0 Equation (5) In S134, the CPU 130 calculates new ratios γx and γy. Here, if the logical sum of formulas (4) and (5) is true, that is, the width W1 and height H1 of the noise estimation region calculated by the CPU 130 to satisfy the threshold area β may exceed the width W0 and height H0 of the input image. In this case, the CPU 130 calculates new ratios γx and γy so that the width W0 or height H0 of the input image is set as the upper limit.

[0049] The CPU 130 calculates the ratio γx and the ratio γy using one of the formulas (6) to (9) associated with the following conditions. If the formula (4) is true in S133, the CPU 130 calculates the ratio γx and the ratio γy using the formulas (6) and (7). If the formula (5) is true in S133, the CPU 130 calculates the ratio γy and the ratio γx using the formulas (8) and (9).

[0050] If equation (4) is true (W1>W0), γx=W0 / |X2-X1| Equation (6) γy=γ×γ / γx Equation (7) If equation (5) is true (H1>H0), γy=H0 / |Y2-Y1| Equation (8) γx=γ×γ / γy Equation (9) In S135, the CPU 130 calculates the width W2 and height H2 of the noise estimation range as the range of the noise estimation region based on the ratio γx and ratio γy newly calculated in S134, using the following formulas (10) and (11). The area of ​​the region indicated by the width W2 and height H2 obtained by formulas (10) and (11) is equal to or larger than the threshold area β. W2=γx×|X2-X1| (W2≦W0) Equation (10) H2=γy×|Y2-Y1| (where H2≦H0) Equation (11) In S136, the CPU 130 converts the width W2 and height H2 into the start coordinates (X3, Y3) and end coordinates (X4, Y4) of the noise estimated region to perform outside-region determination. In other words, the CPU 130 determines whether or not at least a part of the noise estimated region is outside the input image. If step S135 is not executed, that is, if step S133 is No, the CPU 130 may set W2=W1 and H2=H1. The CPU 130 determines whether or not it is outside the region of the image based on the following formulas (12) to (15). X3=X1-(W2―|X2-X1|) / 2<0 Equation (12) Y3=Y1-(H2―|Y2-Y1|) / 2<0 Equation (13) X4=X2+(W2―|X2-X1|) / 2>(W0-1) Equation (14) Y4 = Y2 + (H2 - |Y2 - Y1|) / 2>(H0 - 1) Equation (15) If the logical sum of equations (12) to (15) is true, the CPU 130 proceeds to S137, and if it is false, the CPU 130 ends the subroutine of S130 in FIG. 7 and proceeds to S140 in FIG.

[0051] In S137, the CPU 130 shifts the start coordinate (X3, Y3) and end coordinate (X4, Y4) of the noise estimated region so that it fits within the region of the input image based on the determination result of S136. The CPU 130 calculates the start coordinate (X5, Y5) and end coordinate (X6, Y6) of the shifted noise estimated region according to the following conditions and formulas (16) to (23).

[0052] X5 and X6 setup requirements When W2=W0, X5=0, X6=W0-1 Equation (16) If equation (12) is true (X3<0), then X5=0, X6=W2-1 Equation (17) If equation (14) is true (X4>W0-1), then X5=W0-W2, X6=W0-1 Equation (18) Other conditions, that is, if formula (12) is false (X3≧0) and formula (14) is false (X4≦W0-1), then X5=X3, X6=X4 Formula (19) Setting conditions for Y5 and Y6 If H2=H0, Y5=0, Y6=H0-1 Equation (20) If equation (13) is true (Y3<0), then Y5=0, Y6=H2-1 Equation (21) If equation (15) is true (Y4>H0-1), then Y5=H0-H2, Y6=H0-1 Equation (22) Other conditions, that is, if equation (13) is false (Y3≧0) and equation (14) is false (Y4≦H0-1), Y5=Y3, Y6=Y4 Equation (23) If S131 is false, that is, if the area α of the ROI is equal to or larger than the threshold area β, the CPU 130 proceeds to S138. Since the condition that the area α of the ROI is equal to or larger than the threshold area is satisfied in S138, the CPU 130 sets the coordinates of the noise estimation region using the following formulas (24) to (27). X5=X1 formula (24) Y5 = Y1 Equation (25) X6=X2 formula (26) Y6=Y2 formula (27) Here, FIG. 8(a) and FIG. 8(b) show examples of ROI boundary setting. If the input image is Area100, i.e., 1920×1080, the ROI boundary conditions are Area102 to Area109. Assume that the condition that the areas α of Area102 to Area109 are equal to or larger than the threshold area β is not satisfied. In this case, in order to satisfy the condition, the CPU 130 determines the coordinates X5, Y5, X6, and Y6 based on the above-mentioned formulas (16) to (18) and formulas (20) to (22) as in Area112 to Area119 shown in FIG. 9(a) to FIG. 9(i). Note that, if Area111, whose area is equal to or larger than the threshold area β with respect to Area101 of the ROI, does not reach the boundary of the input image as shown in FIG. 9(e), the CPU 130 determines the coordinates X5, Y5, X6, and Y6 based on formulas (19) and (23).

[0053] When the CPU 130 executes S137 or S138 and determines the coordinates of the noise estimation region, it ends the subroutine of S130 in FIG. 7 and proceeds to S140 in FIG.

[0054] In S140 of Fig. 2, the image processing unit 160 performs noise estimation processing using the noise estimation region determined in S130. Fig. 10 is a flowchart of a subroutine of the noise estimation processing for estimating noise distribution characteristics in S140.

[0055] 10, the image processing unit 160 divides the noise estimation region into blocks. For example, the image processing unit 160 divides the noise estimation region into blocks each including 3×3 pixels.

[0056] In S142, the image processing unit 160 calculates the luminance and noise variance of each block.

[0057] In S143, the image processing unit 160 extracts a flat area in the noise estimation area. Specifically, the image processing unit 160 extracts the flat area by determining whether the noise variance of each block is equal to or less than a predetermined threshold. For example, as shown in FIG. 11, when the noise variance of a block is equal to or less than the threshold, the image processing unit 160 determines that the block is a flat area and sets the block to 1. On the other hand, when the noise variance of a block is greater than the threshold, the image processing unit 160 determines that the block is not a flat area and sets the block to 0. When the above-mentioned block unit includes a difference in luminance such as texture information of the subject, the difference in luminance is calculated as the noise variance, so that it is difficult to estimate noise correctly. Therefore, the image processing unit 160 uses the above-mentioned threshold to determine in advance the blocks of the flat area.

[0058] In S144, the image processing unit 160 collects statistics of the luminance vs. noise variance limited to the blocks determined to be flat parts, and plots the luminance vs. noise variance characteristics.

[0059] In S145, the image processing unit 160 loads noise variance characteristic data corresponding to the currently set gamma characteristic from the storage unit 180. The noise variance characteristic data is stored in the storage unit 180 by measuring the luminance vs. noise variance characteristic for each gain value in advance for the connected camera 200 and the set gamma. For example, table data of the luminance vs. noise variance characteristic as shown in Fig. 12(a) and Fig. 12(b) is stored in the storage unit 180.

[0060] In S146, the image processing unit 160 selects the noise distribution characteristic at each gain contained in the loaded noise distribution characteristic table.

[0061] In S147, the image processing unit 160 measures the distance between the variance value at each luminance plotted in S144 and the noise variance value indicated by the noise variance characteristic data corresponding to the gain selected in S146, and judges whether the characteristics match. The image processing unit 160 may judge that the characteristics match when the distance is equal to or less than a predetermined threshold. If the noise variance characteristics corresponding to the gain do not match in S147, that is, if the result is false, the image processing unit 160 refers to the variance value of the noise variance characteristic data in the next gain and judges whether the noise variance characteristics corresponding to the gain match. If the image processing unit 160 judges that the noise variance values ​​match, that is, if the result is true, the process proceeds to S148. It can be estimated that the gain selected in S146 and judged in S147 to have a matching noise variance characteristic is input to the image processing device 100.

[0062] Various scenes are captured by the camera 200 and input to the image processing device 100. In S130 of Fig. 2, the CPU 130 calculates a noise estimated region that satisfies the threshold area β. By the CPU 130 setting the noise estimated region to an area equal to or larger than the threshold area β, the image processing unit 160 can improve the probability of detecting blocks in flat parts of noise variance, and increase the number of detected blocks. As a result, the image processing device 100 can increase the amount of information on variance characteristics to be collected in S144 of Fig. 11.

[0063] Returning to the description of Fig. 10, when S148 is executed, the subroutine of S140 in Fig. 10 is terminated, and the process proceeds to S150 in Fig. 2. In S150 in Fig. 2, noise reduction processing is executed to reduce noise from the ROI of the input image. Fig. 13 is a flowchart of the subroutine of the noise reduction processing in S150.

[0064] In S151 of FIG. 13, the image processing unit 160 sets an NR intensity parameter according to the camera 200 selected in S121 and S122 of FIG. 5, the gamma, and the gain estimated in S148 of FIG. 11. The NR intensity parameter is applied to the noise intensity parameter 605 described in FIG. 4, and is a parameter for specifying the intensity of the noise reduction process, and is an intensity parameter applied to an attention map focusing on noise. The amount of noise input by the camera 200 and its gamma setting value may increase or decrease depending on not only the gain but also the set gamma. If it is estimated with a high gain, the intensity parameter is set to a high value, and the sensitivity to focus on noise can be increased so that a higher NR effect can be obtained. On the other hand, if it is estimated with a low gain, the emphasis parameter is set low to suppress the application of NR more than necessary, and as a result, the possibility of suppressing adverse effects due to NR, such as lack of resolution and texture loss, can be increased. The NR intensity parameter may be stored in the storage unit 180 as table information corresponding to the camera 200, the gamma, and the estimated gain.

[0065] In S152, the image processing unit 160 loads and acquires the input image acquired in S110 of FIG. 2 from the memory 140 in order to perform NR inference. FIG. 14(a) shows an example of an image of a dark area captured by the camera 200. The image in FIG. 14(a) includes an image of a person, although the person is buried in the background. The image processing device 100 acquires an input image including noise, shown in FIG. 14(b), in which gain has been applied to the image in FIG. 14(a). The image of a person can be seen in the input image in FIG. 14(b).

[0066] In S153, the image processing unit 160 reads out information such as the coordinates of the set ROI.

[0067] In S154, the image processing unit 160 executes the NR inference process using the NN limited to the read ROI information.

[0068] In S155, the image processing unit 160 generates an image as a result of the inference performed in the NR inference process.

[0069] In S156, the image processing unit 160 synthesizes the image resulting from the NR inference obtained in S155 with the input image used for inference, i.e., the input image in Fig. 14(b). As a result, the image processing unit 160 applies the NR process only to the selected ROI as shown in Fig. 14(c) to generate an image resulting from reducing noise in the ROI. After executing S156, the image processing unit 160 ends the subroutine in Fig. 13 and proceeds to S160 in Fig. 2.

[0070] 2, CPU 130 outputs the image obtained in S156 from image processing device 100 via image output unit 170, and displays it on monitor 400. After executing S160, CPU 130 returns to S110, and repeatedly executes the processes from S110 onwards.

[0071] As described above, this embodiment has shown an example of estimating the noise variance characteristics corresponding to the gain set in the camera 200 by evaluating the NR application range limited by an ROI setting by a user or the like against the threshold area β and setting a noise estimation region having a threshold area β or more. Then, this embodiment can appropriately select the NR strength parameter to be given to the NN based on the estimated noise variance characteristics according to the gain of the camera 200, and can obtain an appropriate NR processing result even for a noisy input image in a low-illumination environment or the like.

[0072] In this way, in this embodiment, by appropriately setting the noise estimation region, it is possible to prevent the noise amount estimation result from being estimated lower than the desired noise amount, improve the NR effect, and reduce the noise remaining in the input image. Also, in this embodiment, by appropriately setting the noise estimation region, it is possible to prevent the noise amount estimation result from being estimated higher than the desired noise amount, and suppress loss of resolution and texture loss due to an excessively strong NR effect, thereby improving image quality.

[0073] In this embodiment, the noise estimation region for estimating the noise distribution characteristics is set to include at least a portion of the ROI for reducing noise, so that appropriate noise distribution characteristics corresponding to the noise in the ROI can be estimated.

[0074] In this embodiment, when the ROI has an area equal to or larger than the threshold area β, the ROI is set as an estimation area, so that the noise distribution characteristic corresponding to the noise in the ROI can be estimated.

[0075] In this embodiment, since the noise estimation region is set using the threshold area β corresponding to the model and gain information of the camera 200, the noise estimation region can be set according to the state of the camera 200.

[0076] In this embodiment, the noise estimation region is set according to the width and height of the tentative noise estimation region, so that the noise estimation region can be set while maintaining the aspect ratio of the ROI.

[0077] In this embodiment, when the noise estimation region is outside the input image, the noise estimation region is shifted, so that the noise estimation region can be set within the input image to estimate appropriate noise distribution characteristics.

[0078] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0079] In the above embodiment, the noise estimation area satisfying the threshold area β is calculated as shown in FIG. 7, but when the On Screen Display (hereinafter, OSD) superimposed image by the camera 200 is known in advance as shown in FIG. 15, the coordinate information as shown in FIG. 15(a) may be provided in advance for each connected camera. In this case, as shown in FIG. 15(b) and FIG. 15(c), when there is a superimposed area where the OSD superimposed image is displayed, Area 201 and Area 202, which are set in advance as the noise estimation area, are set not to overlap the superimposed area where the OSD superimposed image by the camera 200 is displayed. As described above, in order to estimate appropriate noise, it is necessary to calculate the noise variance in the flat area. On the other hand, if the noise is estimated including the OSD display area, it may be calculated with a low gain, so it is preferable not to overlap with the OSD display area.

[0080] The present invention also includes cases where a software program that realizes the functions of the above-described embodiments is supplied to a system or device having a computer capable of executing the program directly from a recording medium or via wired / wireless communication, and the program is executed.

[0081] Therefore, the program code itself that is supplied to and installed in a computer to realize the functional processing of the present invention also realizes the present invention. In other words, the computer program itself for realizing the functional processing of the present invention is also included in the present invention.

[0082] In this case, so long as the program has the functions of a program, the form of the program is not important, such as object code, a program executed by an interpreter, or script data supplied to an OS.

[0083] The recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as a magnetic tape, an optical / magneto-optical storage medium, or a non-volatile semiconductor memory.

[0084] As a method for supplying the program, a method is also conceivable in which the computer program forming the present invention is stored in a server on a computer network, and a connected client computer downloads the computer program.

[0085] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0086] The disclosure of this specification includes the following image processing device, image processing method, and program. (Item 1) A first setting means for setting a reduction region in an image; a second setting means for setting an estimated area in the image based on the reduced area; An estimation means for estimating a noise characteristic of the estimation region; a noise reduction unit that performs a noise reduction process to reduce noise in the reduction region of the image using a parameter based on the noise characteristics; When the area of ​​the reduced region is smaller than a threshold area, the second setting means sets the estimated region to be equal to or larger than the threshold area. 13. An image processing device comprising: (Item 2) The reduction means is a first parameter of a trained neural network associated with information of a camera that generated the image; A second parameter that specifies the strength of the noise reduction process based on the noise characteristics; and The noise is reduced based on 2. The image processing device according to item 1, (Item 3) The second setting means sets the estimation region so as to include at least a part of the reduction region. 3. The image processing device according to item 1 or 2, (Item 4) The second setting means sets the reduced area as the estimated area when the reduced area is equal to or larger than the threshold area. 4. The image processing device according to claim 1, wherein the image processing device is a digital camera. (Item 5) The second setting means sets the estimated area at least partially different from the reduced area when the area of ​​the reduced area is smaller than the threshold area. 5. The image processing device according to any one of items 1 to 4, (Item 6) The second setting means sets the estimated area based on the threshold area associated with information of the camera that generated the image. 6. The image processing device according to any one of items 1 to 5, (Item 7) The second setting means is If both the width and height of the provisional region calculated based on the ratio of the square root of the reduced area and the threshold area divided by the area of ​​the reduced area are equal to or less than the width and height of the image, set the width and height of the provisional region as the width and height of the estimated region; When at least one of the width and height of the provisional region is larger than the width and height of the image, the width and height of the estimated region are set based on at least one of the ratio of the width of the reduced region to the width of the image and the ratio of the height of the reduced region to the height of the image. 7. The image processing device according to any one of items 1 to 6, (Item 8) The second setting means shifts the estimated region into the image when a part of the estimated region is outside the image. 8. The image processing device according to any one of items 1 to 7, (Item 9) The second setting means sets the estimated area to an area different from the overlapping area when an overlapping area in which another image is overlapped on the image exists. 9. The image processing device according to any one of items 1 to 8, (Item 10) A setting means for setting a reduction region in an image; a second setting means for setting an estimated area in the image based on the reduced area; An estimation means for estimating a noise characteristic of the estimation region; a noise reduction unit that performs a noise reduction process to reduce noise in the reduction region using a parameter based on the noise characteristics; The second setting means sets the estimated area to an area different from the overlapping area when an overlapping area in which another image is overlapped on the image exists. 13. An image processing device comprising: (Item 11) The reduction means is a first parameter of a trained neural network associated with information of a camera that generated the image; A second parameter that specifies the strength of the noise reduction process based on the noise characteristics; and The noise is reduced based on 11. The image processing device according to item 10, (Item 12) a first setting step of setting a reduction region in an image; a second setting step of setting an estimation area in the image based on the reduced area; an estimation step of estimating noise characteristics of the estimation region; a noise reduction step of performing a noise reduction process to reduce noise in the reduction region of the image using a parameter based on the noise characteristics; In the second setting step, when the area of ​​the reduced region is smaller than a threshold area, the estimated region is set to be equal to or larger than the threshold area. 13. An image processing method comprising: (Item 13) 12. A program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 11.

[0087] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0088] 100 ··· image processing device, 200 ··· camera, 130 ··· CPU, 160 ··· image processing unit, 501 ··· input image, 601 ··· input image, 605 ··· noise intensity parameter.

Claims

1. A first setting means for setting a reduction region in an image; a second setting means for setting an estimated area in the image based on the reduced area; An estimation means for estimating a noise characteristic of the estimation region; a noise reduction unit that performs a noise reduction process to reduce noise in the reduction region of the image using a parameter based on the noise characteristics; The second setting means sets the estimated region to be equal to or larger than the threshold area when the area of ​​the reduced region is smaller than a threshold area.

13. An image processing device comprising:

2. The reduction means is a first parameter of a trained neural network associated with information of a camera that generated the image; A second parameter that specifies the strength of the noise reduction process based on the noise characteristics; and The noise is reduced based on 2. The image processing device according to claim 1,

3. The second setting means sets the estimation region so as to include at least a part of the reduction region.

2. The image processing device according to claim 1,

4. The second setting means sets the reduced area as the estimated area when the reduced area is equal to or larger than the threshold area.

2. The image processing device according to claim 1,

5. The second setting means sets the estimated area at least partially different from the reduced area when the area of ​​the reduced area is smaller than the threshold area.

2. The image processing device according to claim 1,

6. The second setting means sets the estimated area based on the threshold area associated with information of the camera that generated the image.

2. The image processing device according to claim 1,

7. The second setting means is If both the width and height of the provisional region calculated based on the ratio of the square root of the reduced area and the threshold area divided by the area of ​​the reduced area are equal to or less than the width and height of the image, set the width and height of the provisional region as the width and height of the estimated region; When at least one of the width and height of the provisional region is larger than the width and height of the image, the width and height of the estimated region are set based on at least one of the ratio of the width of the reduced region to the width of the image and the ratio of the height of the reduced region to the height of the image.

2. The image processing device according to claim 1,

8. The second setting means shifts the estimated region into the image when a part of the estimated region is outside the image.

2. The image processing device according to claim 1,

9. The second setting means sets the estimated area to an area different from the overlapping area when an overlapping area in which another image is overlapped on the image exists.

2. The image processing device according to claim 1,

10. A setting means for setting a reduction region in an image; a second setting means for setting an estimated area in the image based on the reduced area; An estimation means for estimating a noise characteristic of the estimation region; a noise reduction unit that performs a noise reduction process to reduce noise in the reduction region using a parameter based on the noise characteristics; The second setting means sets the estimated area to an area different from the overlapping area when an overlapping area in which another image is overlapped on the image exists.

13. An image processing device comprising:

11. The reduction means is a first parameter of a trained neural network associated with information of a camera that generated the image; A second parameter that specifies the strength of the noise reduction process based on the noise characteristics; and The noise is reduced based on The image processing device according to claim 10 .

12. a first setting step of setting a reduction region in an image; a second setting step of setting an estimation area in the image based on the reduced area; an estimation step of estimating noise characteristics of the estimation region; a noise reduction step of performing a noise reduction process to reduce noise in the reduction region of the image using a parameter based on the noise characteristics; In the second setting step, when the area of ​​the reduced region is smaller than a threshold area, the estimated region is set to be equal to or larger than the threshold area.

13. An image processing method comprising:

13. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 11.

Images