Image processing device, image processing method and program

Abstract

To solve a problem in which in noise removal, noise cannot be removed properly.SOLUTION: An image processing device comprises: first setting means for setting a reduction region in an image; second setting means for setting an estimated region in the image on the basis of the reduction region; estimation means for estimating noise characteristics of the estimated region; and reduction means for reducing noise of the reduction region in the image by a parameter based on the noise characteristics. The second setting means sets the estimated region so that an area of the estimated region becomes a threshold area or more, if the area of the reduction region is smaller than the threshold area.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In the surveillance market, extremely low light environments require improved visibility of the target subject, and so extremely high gain is applied when capturing images with cameras. While applying high gain brightens the captured image, it also increases noise, but in surveillance applications, subject visibility is important even at the expense of image quality for viewing purposes.

[0003] Noise reduction (hereinafter referred to as NR) has been commonly used to reduce noise contained in captured video. NR can be performed inside the camera, or it can be performed by an external image processing device that acquires images from the camera. In recent years, noise reduction processing based on deep learning artificial intelligence technology (hereinafter referred to as DLNR) has also been used. DLNR has been confirmed to be more effective than conventional NR, but the large-scale processing required poses a problem in terms of processing time.

[0004] When image processing that requires a long processing time is used, a commonly known technique is to reduce processing time and improve overall throughput by designating a part of the entire image as a region of interest (hereinafter referred to as ROI) and limiting the target area of ​​image processing to the ROI. The ROI function may be provided to the photographer.

[0005] Patent Document 1 discloses a technology for switching whether or not to perform image restoration processing depending on the degree of degradation for each block in an input image. According to Patent Document 1, an area where NR is required as image restoration processing is determined as a ROI and applied, so NR is applied only to the minimum necessary area, thereby reducing processing time.

[0006] Patent Document 2 discloses an example in which the amount of noise in an ROI and the edges of the region near the ROI are calculated, weighting is performed according to the amount of noise and the edges, and noise reduction processing is performed according to the weighting. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent Publication No. 2021-118403 [Patent Document 2] Japanese Patent Application Laid-Open No. 2007-110338 Summary of the Invention [Problem to be solved by the invention]

[0008] However, in the conventional technology, noise is estimated only from the ROI, so when the ROI is small, the noise estimation accuracy decreases, and noise may not be appropriately removed during noise removal.

[0009] Therefore, an object of the present invention is to provide an image processing device that can appropriately remove noise even when the area from which noise is to be removed is small, such as an ROI. [Means for solving the problem]

[0010] In order to solve this problem, for example, an image processing device of the present invention has the following arrangement: a first setting means for setting a reduction region in an image; a second setting means for setting an estimated area in the image based on the reduced area; an estimation means for estimating noise characteristics of the estimation region; a noise reduction unit that performs noise reduction processing to reduce noise in the reduction region of the image using parameters based on the noise characteristics, When the area of ​​the reduced area is smaller than a threshold area, the second setting means sets the estimated area so that the area is equal to or larger than the threshold area. If the reduced area is equal to or larger than the threshold area, the reduced area is set as the estimated area. 1. An image processing device comprising: [Effects of the Invention]

[0011] According to the present invention, even if the area for removing noise set by the user or the like is small, it is possible to appropriately estimate noise and remove it. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing a hardware configuration of an image processing apparatus. [Figure 2] 4 is a flowchart of image processing executed by the image processing device. [Figure 3] 10 is a flowchart of a subroutine of the initialization process of S100. [Figure 4] An explanatory diagram of a neural network. [Figure 5] 10 is a flowchart of a subroutine for image processing setting in S120. [Figure 6] FIG. 10 is a diagram showing an example of setting an ROI. [Figure 7] 10 is a flowchart of a subroutine for calculating a noise estimation region in S130. [Figure 8] FIG. 10 is a diagram showing an example of ROI boundary setting. [Figure 9] An example of boundary setting for noise estimation region. [Figure 10] 10 is a flowchart of a subroutine of noise estimation processing in S140. [Figure 11] FIG. 10 is a diagram showing an example of flat area extraction. [Figure 12] FIG. 10 is a diagram showing an example of noise dispersion characteristics for different camera settings. [Figure 13] 10 is a flowchart of a subroutine for NR processing in S150. [Figure 14] FIG. 10 is a diagram showing an example of NR processing on an ROI. [Figure 15] 10A to 10C are diagrams showing examples of noise estimation regions for different camera settings. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0014] An embodiment of the present invention will be described below. An image processing device according to this embodiment estimates noise from an input image acquired from a connected camera, infers a desired output image for a target region of image processing set as a region of interest (hereinafter referred to as ROI) using a neural network (hereinafter referred to as NN), which is a type of deep learning model, and outputs the inferred image. Note that the concept of an image includes still images, videos, and images of one frame of a video. An ROI is an example of a reduction region. In the following description, the concept of an image includes video, still images, and videos. The image may also refer to image data. The image processing device performs NN training, such as approximating the feature distribution of multiple prepared student images to the feature distribution of the corresponding teacher images, and optimizes trained neural network parameters such as weights and biases. This enables the image processing device to perform accurate inference even for untrained input images. The image processing device retains trained neural network parameters obtained by multiple learnings according to the characteristics of the camera, allowing it to perform inference on the input image and generate an inferred image with reduced noise.

[0015] Fig. 1 is a block diagram showing the hardware configuration of an image processing device 100. The image processing device 100 is, for example, a computer such as a personal computer. Note that, although an image processing device will be described in this embodiment, the processing described below can also be performed by a camera with interchangeable lenses, such as a video camera with interchangeable lenses, a single-lens reflex camera, or a mirrorless single-lens camera. For example, the camera 200 in Fig. 1 may perform the image processing and other processing described below.

[0016] The camera 200 captures a light beam incident from the subject field to generate an image. The camera 200 is equipped with or has built-in lenses (not shown), and includes a zoom lens group, a focus lens group, an iris mechanism, and the like. The camera 200 can change the exposure accumulation time, and applies gain to the captured image when capturing images in dark places using an auto-exposure function. The camera 200 outputs the captured image.

[0017] 1, image processing device 100 includes image input unit 110, CPU 130, operation input unit 150, memory 140, image processing unit 160, image output unit 170, storage unit 180, and bus 120. Image input unit 110, CPU 130, operation input unit 150, memory 140, image processing unit 160, image output unit 170, and storage unit 180 are connected via bus 120 so as to be able to transmit and receive data to and from each other. CPU 130 and image processing unit 160 are examples of first setting means, second setting means, estimation means, and reduction means.

[0018] The image input unit 110 acquires the image output by the camera 200 and stores it in the memory 140 .

[0019] CPU 130 is an abbreviation for Central Processing Unit. Note that the image processing device 100 may have an MPU (Micro Processing Unit), a QPU (Quantum Processing Unit), a GPU (Graphics Processing Unit), etc. in addition to the CPU 130. The CPU 130 executes each process to be described later that is executed in the image processing device 100. For example, the CPU 130 executes a program stored in the storage unit 180 to realize each function and execute each process.

[0020] The memory 140 is, for example, a random access memory (RAM), and temporarily stores image data, programs, and data required for executing the programs.

[0021] The operation input unit 150 acquires an operation signal for the external controller 300 input by a user or the like. The controller 300 includes a keyboard, a mouse, a touch panel, and switches. The operation signal acquired by the operation input unit 150 is processed by the CPU 130, and setting operations required for various image processing executed by the image processing unit 160 are performed.

[0022] The image processing unit 160 reads and writes images stored in the memory 140, and executes noise estimation processing, ROI processing, NR processing, and UI image generation processing for display on a user interface (hereinafter referred to as UI), which will be described later. The image processing unit 160 stores processed images and the like in the memory 140. The image processing unit 160 may include an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), a GPU, and the like. The image processing unit 160 may also be realized by a program executed by the CPU 130.

[0023] The image output unit 170 outputs the image processed by the image processing unit 160 and stored in the memory 140 to an external monitor 400. The image output unit 170 outputs the image from an HDMI (registered trademark) (High Definition Multimedia Interface) terminal, an SDI (Serial Digital Interface) terminal, or the like of the image processing device 100.

[0024] The storage unit 180 includes a hard disk drive (HDD), a solid state drive (SSD), etc. The storage unit 180 stores programs, data such as parameters required for executing the programs, and data such as images.

[0025] The monitor 400 receives and displays the image output from the image output unit 170 of the image processing device 100. The photographer and viewers can use the monitor 400 to check the image captured by the camera 200, the menu screen of the image processing device 100, and the image after image processing.

[0026] 1, each function is shown as a separate component, but these may be realized by hardware such as one or more ASICs or programmable logic arrays (PLAs).Also, the above configuration may be realized by a programmable processor such as a CPU or MPU executing software.

[0027] Next, the image processing performed by the CPU 130 in the image processing device 100 will be described using the flowchart in Fig. 2. Fig. 2 is a flowchart of the image processing performed by the image processing device 100. When the image processing device 100 is powered on, the CPU 130 loads a computer program stored in the storage unit 180. As a result, the CPU 130 executes the processing in order from S100 in Fig. 2. Note that after the CPU 130 executes the processing up to S160 in Fig. 2, it will repeatedly execute the processing of S110 again. In the following description, it will be described on the assumption that the CPU 130 executes the processing from S110 to S160 for each frame of the input image from the camera 200.

[0028] In S100 of Fig. 2, the CPU 130 performs initialization processing of the image processing device 100. Fig. 3 shows a subroutine of the initialization processing of S100.

[0029] 3, the CPU 130 executes initialization processing for image input / output. The CPU 130 executes initialization of the image input unit 110 and the image output unit 170 so that image input / output processing can be performed.

[0030] In S102, the CPU 130 executes initialization processing of the ROI. The CPU 130 sets the initial value of the ROI so that all pixels of the input image are the processing target. For example, when the size of the image input to the image input unit 110 is 1920 x 1080, the CPU 130 sets the start coordinates (0, 0) and end coordinates (1919, 1079) as the initial values ​​of the ROI.

[0031] In S103, the CPU 130 executes NR initialization processing. The CPU 130 loads pre-trained NN parameters from the storage unit 180 for the NR processing executed by the image processing unit 160. The storage unit 180 holds trained NN parameters optimized for each model and image quality setting of the camera 200. The CPU 130 sets the NN parameters to be loaded by the image processing unit 160 in accordance with various models and image qualities of the cameras 200 connected to the image processing device 100.

[0032] Here, the NN executed by the image processing unit 160 and the NN parameters to be loaded will be described with reference to Fig. 4. Fig. 4 is a diagram illustrating the NN for an input image. Fig. 4(a) is a diagram illustrating an example of outputting a feature map. Fig. 4(b) is a diagram illustrating an example of outputting an attention layer.

[0033] In the following description, an example will be given in which the NN is a Convolutional Neural Network (CNN), but the present embodiment is not limited to this. For example, a Generative Adversarial Network (GAN) or the like may be applied to the NN, and the NN may have a skip connection or the like. The NN may also be a recurrent type such as a Recurrent Neural Network (RNN).

[0034] In FIG. 4, an input image 501 represents an image to be input to a neural network (NN) or a feature map (described later). A convolution matrix 503 is a filter that performs a convolution operation on the input image 501. A bias 504 is a value added to the result output by a convolution operation 502 of the input image 501 and the convolution matrix 503. A feature map 505 is the result of the convolution operation after the bias 504 is added to the result output by the convolution operation 502. For simplicity, FIG. 4 illustrates a small number of neurons, hidden layers, and channels. However, the number of neurons and layers, as well as the number and weights of connections between neurons, are not limited to this example. Furthermore, when the NN illustrated in FIG. 4 is implemented in a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), the number of connections and weights between neurons may be reduced.

[0035] CNN generates a feature map of an input image by performing a convolution operation on the input image using a certain filter. The size of the CNN filter can be arbitrary. CNN generates a different feature map in the next layer by performing a convolution operation on the feature map of the previous layer using a different filter. In each layer, CNN multiplies an input signal by a filter and calculates the sum with a bias. CNN then applies an activation function to the calculated result to output an output signal for each neuron. The weights and biases in each layer are called NN parameters, and the NN parameters are updated during training. Examples of activation functions include the sigmoid function and the ReLU function. The CNN of this embodiment uses the Leaky ReLU function shown in the following equation (1), but is not limited to this. In equation (1), max represents a function that outputs the maximum value among its arguments. f(x)=max(x,x×0.2) Equation (1) In the pre-learning for obtaining the NN parameters, the image processing unit 160 of this embodiment uses an image having the noise variance characteristics of the camera 200 as a student image, and a corresponding noise-free image of the student image as a teacher image. The noise variance characteristics are an example of noise characteristics. The image processing unit 160 realizes NR by performing learning on pairs of student images and teacher images and setting the NN parameters.

[0036] The feature map 505 focuses on noise in the input image. The image processing unit 160 can learn a region of interest with enhanced noise components by applying a different parameter to the feature map 505. The image processing unit 160 calculates an average in the channel direction for an input image 601, which is obtained by dividing the input image 501 into channels, for example, by color, to generate an intermediate layer 602. The image processing unit 160 performs convolution on the intermediate layer 602 multiple times using equation (1) to obtain an intermediate layer 603. The image processing unit 160 performs convolution on the intermediate layer 603 so that the output channel is 1 to obtain an attention layer 604. The attention layer 604 is an intermediate layer in which features appear in noise regions excluding high-frequency components of the subject. The image processing unit 160 multiplies the attention layer 604 by a noise intensity parameter 605, which specifies the strength of the noise reduction process, to obtain an attention layer 606 with enhanced noise. The image processing unit 160 convolves the attention layer 606 with the above-mentioned input image 501. As a result, the image processing unit 160 can adjust, that is, emphasize, the degree of attention paid to noise regions included in the input images 501 and 601, and generate an NN capable of performing NR processing.

[0037] Returning to the description of Fig. 3, in S104, the CPU 130 executes operation input initialization. The CPU 130 executes initialization processing for acquiring an operation signal from the controller 300. After executing S104, the CPU 130 ends the subroutine S100 and proceeds to S110 in Fig. 2.

[0038] 2, the CPU 130 stores the input image acquired by the image input unit 110 from the camera 200 in the memory 140. In S120, the CPU 130 performs image processing settings. Fig. 5 shows a flowchart of the image processing subroutine of S120.

[0039] In S121, CPU 130 acquires information about the model of camera 200 selected by a user or the like using controller 300 via operation input unit 150. Note that CPU 130 may acquire information about camera 200 selected by a user from a setting menu of image processing device 100 or the like, or may acquire information about camera 200 automatically detected by image input unit 110 from information embedded in the signal of the input image.

[0040] In S122, CPU 130 acquires information about the gamma applied to camera 200 selected by a user or the like using controller 300 via operation input unit 150. Note that CPU 130 may acquire information about the gamma selected from a setting menu of image processing device 100 or the like, or may acquire information about the gamma automatically detected by image input unit 110 from information embedded in an image signal. Information about the model of camera 200 and information about the gamma of camera 200 are examples of camera information.

[0041] In S123, the CPU 130 determines whether the information of the camera 200 has been changed in S121 or whether the gamma information has been changed in S122. If the information has been changed, the CPU 130 proceeds to S124, and if the information has not been changed, the CPU 130 proceeds to S125.

[0042] In S124, the CPU 130 loads and sets the trained NN parameters corresponding to the model and gamma of the camera 200 selected in S121 and S122 into the image processing unit 160. As a result, the appropriate NN parameters trained according to the connected camera 200 and its gamma setting are applied to the NN loaded in the image processing unit 160. In S124, the image processing unit 160 is ready to perform inference with the NN, that is, to load the input image from the memory 140 and perform NR processing.

[0043] 5, the CPU 130 executes ROI setting. The CPU 130 acquires an operation signal input by a user or the like from the controller 300 via the operation input unit 150, and sets an ROI as a target region to which image processing is applied.

[0044] Fig. 6 is a diagram showing an example of setting an ROI. As shown in Fig. 6, if the size of the input image input to the image input unit 110 is 1920 x 1080, for example, the CPU 130 can define the entire area of ​​the input image as Area 100. Furthermore, if Area 101 is set as the ROI, the CPU 130 sets start coordinates (X1, Y1) and end coordinates (X2, Y2) as example setting values.

[0045] In S126 of Fig. 5, the CPU 130 calculates the area α of the ROI. According to the example of the ROI setting shown in Fig. 6, the area α can be obtained by the following formula (2). α=|X2-X1|×|Y2-Y1| Formula (2) In S127, CPU 130 loads from storage unit 180 the threshold area β associated with the model and gamma of camera 200 selected in S121 and S122. Threshold area β is the minimum area required to achieve accuracy in the noise estimation process of S140, which will be described later. For example, CPU 130 may select and load threshold area β from a table stored in storage unit 180 that associates the model and gamma of camera 200 with the threshold area β. After executing S127, CPU 130 ends the image processing setting subroutine of S120 and proceeds to step S130 in FIG. 2.

[0046] 2, the CPU 130 calculates the noise estimated region. Fig. 7 is a flowchart of the noise estimated region calculation subroutine of S130.

[0047] In S131, the CPU 130 compares the area α calculated in S126, that is, using equation (2), with the threshold area β acquired in S127. If the area α is smaller than the threshold area β, the CPU 130 proceeds to S132.

[0048] In S132, the CPU 130 calculates the ratio γ. Here, if the area α of the set ROI is smaller than the threshold area β and the ROI is set as a noise estimation area, which is an area for noise estimation processing, the accuracy of the noise estimation processing, which will be described later, may be reduced. Therefore, in this embodiment, the CPU 130 calculates the ratio γ shown in equation (3) to ensure a noise estimation area that satisfies the threshold area β for the area α of the ROI. The CPU 130 calculates the ratio γ as the square root of the threshold area β divided by the area α. γ=√(β / α) Equation (3) In S133 of FIG. 7, the CPU 130 determines whether the width and height of the provisional noise estimation region are outside the region of the input image. To obtain a noise estimation region that satisfies the aforementioned threshold area β, the CPU 130 calculates the width W1 and height H1 of the provisional noise estimation region based on the width and height of the ROI and the ratio γ using the following equations (4) and (5). The CPU 130 compares the width W0 and height H0 of the input image with the calculated width W1 and height H1, respectively. If the logical sum of equations (4) and (5) is true, the CPU 130 proceeds to S134. On the other hand, if the logical sum is false, i.e., if both the width W1 and height H1 of the provisional noise estimation region are equal to or smaller than the width W0 and height H0 of the input image, the CPU 130 proceeds to S136. It should be noted that both equations (4) and (5) will never be true. W1=|X2-X1|×γ>W0 Equation (4) H1=|Y2-Y1|×γ>H0 Equation (5) In S134, the CPU 130 calculates new ratios γx and γy. Here, if the logical sum of equations (4) and (5) is true, that is, if the width W1 and height H1 of the noise estimation region calculated by the CPU 130 so as to satisfy the threshold area β, may exceed the width W0 and height H0 of the input image. In this case, the CPU 130 calculates new ratios γx and γy so that the width W0 or height H0 of the input image is the upper limit.

[0049] The CPU 130 calculates the ratio γx and the ratio γy using one of equations (6) to (9) associated with the following conditions: If equation (4) is true in S133, the CPU 130 calculates the ratio γx and the ratio γy using equations (6) and (7). If equation (5) is true in S133, the CPU 130 calculates the ratio γy and the ratio γx using equations (8) and (9).

[0050] If equation (4) is true (W1>W0), γx=W0 / |X2-X1| Equation (6) γy=γ×γ / γx Equation (7) If equation (5) is true (H1>H0), γy=H0 / |Y2-Y1| Equation (8) γx=γ×γ / γy Equation (9) In S135, the CPU 130 calculates the width W2 and height H2 of the noise estimation range as the range of the noise estimation region based on the ratio γx and ratio γy newly calculated in S134, using the following equations (10) and (11). The area of ​​the region indicated by the width W2 and height H2 obtained by equations (10) and (11) is equal to or greater than the threshold area β. W2=γx×|X2-X1| (W2≦W0) Equation (10) H2=γy×|Y2-Y1| (where H2≦H0) Equation (11) In S136, the CPU 130 converts the width W2 and height H2 into the start coordinates (X3, Y3) and end coordinates (X4, Y4) of the noise estimated region and performs outside-region determination. In other words, the CPU 130 determines whether or not at least a part of the noise estimated region is outside the input image. Note that if step S135 is not executed, that is, if step S133 returns No, the CPU 130 may set W2=W1 and H2=H1. The CPU 130 determines whether or not it is outside the region of the image based on the following equations (12) to (15). X3=X1-(W2―|X2-X1|) / 2<0 Equation (12) Y3=Y1-(H2―|Y2-Y1|) / 2<0 Equation (13) X4=X2+(W2―|X2-X1|) / 2>(W0-1) Equation (14) Y4 = Y2 + (H2 - |Y2 - Y1|) / 2 > (H0 - 1) Equation (15) If the logical sum of equations (12) to (15) is true, the CPU 130 proceeds to S137, and if false, the CPU 130 ends the subroutine of S130 in FIG. 7 and proceeds to S140 in FIG.

[0051] In S137, the CPU 130 shifts the start coordinates (X3, Y3) and end coordinates (X4, Y4) of the noise estimation region so that it fits within the region of the input image based on the determination result of S136. The CPU 130 calculates the start coordinates (X5, Y5) and end coordinates (X6, Y6) of the shifted noise estimation region using the following conditions and equations (16) to (23):

[0052] X5 and X6 setup requirements When W2=W0, X5=0, X6=W0-1 Equation (16) If equation (12) is true (X3<0), then X5=0, X6=W2-1 Equation (17) If equation (14) is true (X4>W0-1), then X5=W0-W2, X6=W0-1. Equation (18) Other conditions, that is, if equation (12) is false (X3≧0) and equation (14) is false (X4≦W0−1), then X5=X3, X6=X4 Equation (19) Setting conditions for Y5 and Y6 When H2=H0, Y5=0, Y6=H0-1 Equation (20) If equation (13) is true (Y3<0), then Y5=0, Y6=H2-1 Equation (21) If equation (15) is true (Y4>H0-1), then Y5=H0-H2, Y6=H0-1 Equation (22) Other conditions, that is, if equation (13) is false (Y3≧0) and equation (14) is false (Y4≦H0−1), then Y5=Y3, Y6=Y4 Equation (23) If S131 is false, that is, if the area α of the ROI is equal to or larger than the threshold area β, the CPU 130 proceeds to S138. In S138, since the condition that the area α of the ROI is equal to or larger than the threshold area is satisfied, the CPU 130 sets the coordinates of the noise estimation region using the following equations (24) to (27). X5=X1 formula (24) Y5=Y1 Equation (25) X6=X2 formula (26) Y6=Y2 formula (27) Here, FIGS. 8(a) and 8(b) show examples of ROI boundary setting. If the input image is Area 100, i.e., 1920 × 1080, then Areas 102 to 109 are considered as boundary conditions for the ROI. Assume that the condition that the area α of each of Areas 102 to 109 is equal to or greater than the threshold area β is not satisfied. In this case, to satisfy this condition, the CPU 130 determines the coordinates X5, Y5, X6, and Y6 based on the above-described formulas (16) to (18) and (20) to (22) for Areas 112 to 119 shown in FIGS. 9(a) to 9(i). Note that, if Area 111, whose area is equal to or greater than the threshold area β relative to Area 101 of the ROI, does not reach the boundary of the input image, as shown in FIG. 9(e), the CPU 130 determines the coordinates X5, Y5, X6, and Y6 based on formulas (19) and (23).

[0053] When the CPU 130 executes S137 or S138 and determines the coordinates of the noise estimation region, it ends the subroutine of S130 in FIG. 7 and proceeds to S140 in FIG.

[0054] In S140 of Fig. 2, the image processing unit 160 performs noise estimation processing using the noise estimation region determined in S130. Fig. 10 is a flowchart of the noise estimation processing subroutine for estimating noise variance characteristics in S140.

[0055] 10, the image processing unit 160 divides the noise estimation region into blocks. For example, the image processing unit 160 divides the noise estimation region into blocks, each block including an area including 3×3 pixels.

[0056] In S142, the image processing unit 160 calculates the luminance and noise variance of each block.

[0057] In S143, the image processing unit 160 extracts flat areas from the noise estimation area. Specifically, the image processing unit 160 extracts flat areas by determining whether the noise variance of each block is equal to or less than a predetermined threshold. For example, as shown in FIG. 11, if the noise variance of a block is equal to or less than the threshold, the image processing unit 160 determines that the block is a flat area and sets 1 to the block. On the other hand, if the noise variance of a block is greater than the threshold, the image processing unit 160 determines that the block is not a flat area and sets 0 to the block. If the above-mentioned block-by-block difference in luminance, such as texture information of the subject, is included, the luminance difference is calculated as the noise variance, making it difficult to accurately estimate noise. Therefore, the image processing unit 160 uses the above-mentioned threshold to determine in advance which blocks are flat areas.

[0058] In S144, the image processing unit 160 collects statistics of brightness versus noise variance limited to the blocks determined to be flat portions, and plots the brightness versus noise variance characteristics.

[0059] In S145, the image processing unit 160 loads noise variance characteristic data corresponding to the currently set gamma characteristic from the storage unit 180. The noise variance characteristic data is obtained by measuring the luminance versus noise variance characteristic for each gain value in advance for the connected camera 200 and the set gamma, and is stored in the storage unit 180. For example, table data of the luminance versus noise variance characteristic as shown in FIGS. 12(a) and 12(b) is stored in the storage unit 180.

[0060] In S146, the image processing unit 160 selects the noise variance characteristics at each gain contained in the loaded noise variance characteristics table.

[0061] In S147, the image processing unit 160 measures the distance between the variance values ​​at each luminance plotted in S144 and the noise variance value indicated by the noise variance characteristic data corresponding to the gain selected in S146, and determines whether the characteristics match. The image processing unit 160 may determine that the characteristics match if the distance is equal to or less than a predetermined threshold. If the noise variance characteristics corresponding to the gain do not match in S147, i.e., if the result is false, the image processing unit 160 references the variance value of the noise variance characteristic data for the next gain and determines whether the noise variance characteristics corresponding to the gain match. If the image processing unit 160 determines that the noise variance values ​​match, i.e., if the result is true, the process proceeds to S148. It can be assumed that the gain selected in S146 and determined to have a matching noise variance characteristic in S147 has been input to the image processing device 100.

[0062] Various scenes are captured by the camera 200 and input to the image processing device 100. In S130 of FIG. 2, the CPU 130 calculates a noise estimation region that satisfies the threshold area β. By setting the noise estimation region to an area equal to or greater than the threshold area β, the image processing unit 160 can improve the probability of detecting blocks in flat areas of noise variance, thereby increasing the number of detected blocks. As a result, the image processing device 100 can increase the amount of variance characteristic information compiled in S144 of FIG. 11.

[0063] Returning to the description of Figure 10, when S148 is executed, the subroutine of S140 in Figure 10 is terminated and the process proceeds to S150 in Figure 2. In S150 in Figure 2, noise reduction processing is performed to reduce noise from the ROI of the input image. Figure 13 is a flowchart of the subroutine of the noise reduction processing in S150.

[0064] In S151 of FIG. 13, the image processing unit 160 sets an NR intensity parameter based on the camera 200 selected in S121 and S122 of FIG. 5, the gamma, and the gain estimated in S148 of FIG. 11. The NR intensity parameter is applied to the noise intensity parameter 605 described in FIG. 4 and is a parameter for specifying the intensity of the noise reduction process, and is an intensity parameter applied to an attention map focusing on noise. The amount of noise input by the camera 200 and its gamma setting value may fluctuate depending on not only the gain but also the set gamma. If a high gain is estimated, the intensity parameter is set to a high value, thereby increasing the sensitivity to focus on noise so as to obtain a higher NR effect. On the other hand, if a low gain is estimated, the enhancement parameter is set to a low value, thereby preventing NR from being applied more than necessary, and as a result, the possibility of suppressing adverse effects caused by NR, such as loss of resolution and texture loss, can be increased. The NR intensity parameter may be stored in the storage unit 180 as table information corresponding to the camera 200, gamma, and estimated gain.

[0065] In S152, the image processing unit 160 loads and acquires the input image acquired in S110 of FIG. 2 from the memory 140 in order to perform NR inference. FIG. 14(a) shows an example of an image of a dark area captured by the camera 200. The image in FIG. 14(a) includes an image of a person, although the person is buried in the background. The image processing device 100 acquires an input image containing noise, shown in FIG. 14(b), in which gain has been applied to the image in FIG. 14(a). The image of the person is visible in the input image in FIG. 14(b).

[0066] In S153, the image processing unit 160 reads information such as the coordinates of the set ROI.

[0067] In S154, the image processing unit 160 executes the NR inference process using the NN limited to the information of the read ROI.

[0068] In S155, the image processing unit 160 generates an image as a result of the inference performed in the NR inference process.

[0069] In S156, the image processing unit 160 synthesizes the image resulting from the NR inference obtained in S155 with the input image used for inference, i.e., the input image of FIG. 14(b). As a result, the image processing unit 160 applies NR processing only to the selected ROI as shown in FIG. 14(c), generating an image resulting from noise reduction in the ROI. After executing S156, the image processing unit 160 ends the subroutine of FIG. 13 and proceeds to S160 of FIG. 2.

[0070] 2, the CPU 130 outputs the image obtained in S156 from the image processing device 100 via the image output unit 170 and displays it on the monitor 400. After executing S160, the CPU 130 returns to S110 and repeatedly executes the processes from S110 onwards.

[0071] As described above, this embodiment has shown an example in which the NR application range limited by ROI setting by a user or the like is evaluated against the threshold area β, and a noise estimation region equal to or larger than the threshold area β is set, thereby estimating the noise variance characteristics corresponding to the gain set in the camera 200. Then, this embodiment can appropriately select the NR strength parameter to be given to the NN based on the estimated noise variance characteristics according to the gain of the camera 200, and can obtain appropriate NR processing results even for noisy input images in low-light environments, etc.

[0072] In this way, by appropriately setting the noise estimation region, this embodiment can prevent the estimated noise amount from being estimated lower than the desired noise amount, improve the NR effect, and reduce the noise remaining in the input image.Furthermore, by appropriately setting the noise estimation region, this embodiment can prevent the estimated noise amount from being estimated higher than the desired noise amount, prevent loss of resolution and texture loss due to an excessively strong NR effect, and improve image quality.

[0073] In this embodiment, the noise estimation region for estimating the noise variance characteristics is set to include at least a portion of the ROI for reducing noise, so that appropriate noise variance characteristics corresponding to the noise in the ROI can be estimated.

[0074] In this embodiment, if the ROI has an area equal to or larger than the threshold area β, the ROI is set as an estimation area, so that the noise variance characteristics corresponding to the noise in the ROI can be estimated.

[0075] In this embodiment, the noise estimation region is set using the threshold area β corresponding to the model and gain information of the camera 200, so that the noise estimation region can be set according to the state of the camera 200.

[0076] In this embodiment, the noise estimation region is set according to the width and height of the provisional noise estimation region, so that the noise estimation region can be set while maintaining the aspect ratio of the ROI.

[0077] In this embodiment, if the noise estimation region is outside the input image, the noise estimation region is shifted, so that the noise estimation region can be set within the input image and appropriate noise variance characteristics can be estimated.

[0078] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0079] In the above embodiment, a noise estimation area satisfying the threshold area β was calculated as shown in FIG. 7. However, if the on-screen display (hereinafter, OSD) superimposed image from camera 200 is known in advance as shown in FIG. 15, coordinate information such as that shown in FIG. 15(a) may be provided in advance for each connected camera. In this case, as shown in FIGS. 15(b) and 15(c), if there is an overlapping area where an OSD superimposed image is displayed, Area 201 and Area 202, which are preset as noise estimation areas, are set so as not to overlap the overlapping area where the OSD superimposed image from camera 200 is displayed. As described above, to estimate appropriate noise, it is necessary to calculate noise variance in a flat area. However, if noise is estimated including the OSD display area, the calculation may be performed with a low gain, so it is desirable not to overlap with the OSD display area.

[0080] The present invention also includes cases where a software program that realizes the functions of the above-described embodiments is supplied to a system or device having a computer that can execute the program directly from a recording medium or via wired / wireless communication, and the program is executed.

[0081] Therefore, the program code itself that is supplied to and installed on a computer to realize the functional processing of the present invention also realizes the present invention. In other words, the computer program itself for realizing the functional processing of the present invention is also included in the present invention.

[0082] In this case, as long as it has the functionality of a program, the form of the program does not matter, such as object code, a program executed by an interpreter, or script data supplied to an OS.

[0083] The recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as a magnetic tape, an optical / magneto-optical storage medium, or a non-volatile semiconductor memory.

[0084] Another method of supplying the program is to store the computer program forming the present invention in a server on a computer network, and have connected client computers download and program the computer program.

[0085] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0086] The disclosure of this specification includes the following image processing device, image processing method, and program. (Item 1) a first setting means for setting a reduction region in an image; a second setting means for setting an estimated area in the image based on the reduced area; an estimation means for estimating noise characteristics of the estimation region; a noise reduction unit that performs noise reduction processing to reduce noise in the reduction region of the image using parameters based on the noise characteristics, When the area of ​​the reduced area is smaller than a threshold area, the second setting means sets the estimated area so that the area is equal to or larger than the threshold area. 1. An image processing device comprising: (Item 2) The reduction means is a first parameter of a trained neural network associated with information about a camera that generated the image; a second parameter that specifies the strength of the noise reduction processing based on the noise characteristics; The noise is reduced based on 2. The image processing device according to item 1, (Item 3) The second setting means sets the estimation region to include at least a part of the reduction region. 3. The image processing device according to item 1 or 2, (Item 4) The second setting means sets the reduced area as the estimated area when the reduced area is equal to or larger than the threshold area. 4. The image processing device according to any one of items 1 to 3, wherein: (Item 5) When the area of ​​the reduced area is smaller than the threshold area, the second setting means sets the estimated area that is at least partially different from the reduced area. 5. The image processing device according to any one of items 1 to 4, wherein: (Item 6) The second setting means sets the estimated area based on the threshold area associated with information about the camera that generated the image. 6. The image processing device according to any one of items 1 to 5, wherein: (Item 7) The second setting means If both the width and height of the provisional region calculated based on the ratio of the square root of the reduced area and the threshold area divided by the area of ​​the reduced area are equal to or less than the width and height of the image, set the width and height of the provisional region as the width and height of the estimated region; If at least one of the width and height of the tentative region is larger than the width and height of the image, the width and height of the estimated region are set based on at least one of the ratio between the width of the reduced region and the width of the image and the ratio between the height of the reduced region and the height of the image. 7. The image processing device according to any one of items 1 to 6, wherein: (Item 8) The second setting means shifts the estimated area into the image when a part of the estimated area is outside the image. 8. The image processing device according to any one of items 1 to 7, wherein: (Item 9) When an overlapping area where another image is overlapped on the image exists, the second setting means sets the estimated area to an area different from the overlapping area. 9. The image processing device according to any one of items 1 to 8, wherein: (Item 10) a setting means for setting a reduction region in an image; a second setting means for setting an estimated area in the image based on the reduced area; an estimation means for estimating noise characteristics of the estimation region; a noise reduction unit that performs noise reduction processing to reduce noise in the reduction region using parameters based on the noise characteristics, When an overlapping area where another image is overlapped on the image exists, the second setting means sets the estimated area to an area different from the overlapping area. 1. An image processing device comprising: (Item 11) The reduction means is a first parameter of a trained neural network associated with information about a camera that generated the image; a second parameter that specifies the strength of the noise reduction processing based on the noise characteristics; The noise is reduced based on 11. The image processing device according to item 10. (Item 12) a first setting step of setting a reduction region in an image; a second setting step of setting an estimation area in the image based on the reduced area; an estimation step of estimating noise characteristics of the estimation region; a noise reduction step of performing a noise reduction process to reduce noise in the reduction region of the image using parameters based on the noise characteristics, In the second setting step, when the area of ​​the reduced area is smaller than a threshold area, the estimated area is set to be equal to or larger than the threshold area. An image processing method comprising: (Item 13) 12. A program for causing a computer to function as each means of the image processing device according to any one of items 1 to 11.

[0087] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0088] 100···Image processing device, 200···Camera, 130···CPU, 160···Image processing unit, 501···Input image, 601···Input image, 605···Noise intensity parameter.

Claims

1. a first setting means for setting a reduction region in an image; a second setting means for setting an estimated area in the image based on the reduced area; an estimation means for estimating noise characteristics of the estimation region; a noise reduction unit that performs noise reduction processing to reduce noise in the reduction region of the image using parameters based on the noise characteristics, The second setting means sets the estimated region so that the area of ​​the reduced region is equal to or larger than the threshold area when the area of ​​the reduced region is smaller than the threshold area, and sets the reduced region as the estimated region when the area of ​​the reduced region is equal to or larger than the threshold area.

1. An image processing device comprising:

2. The reduction means is a first parameter of a trained neural network associated with information about a camera that generated the image; a second parameter that is based on the noise characteristics and specifies the strength of the noise reduction process; The noise is reduced based on 2. The image processing device according to claim 1, wherein:

3. The second setting means sets the estimation region to include at least a part of the reduction region.

2. The image processing device according to claim 1, wherein:

4. When the area of ​​the reduced area is smaller than the threshold area, the second setting means sets the estimated area that is at least partially different from the reduced area.

2. The image processing device according to claim 1, wherein:

5. The second setting means sets the estimated area based on the threshold area associated with information about the camera that generated the image.

2. The image processing device according to claim 1, wherein:

6. When the area of ​​the reduction region is smaller than a threshold area, the second setting means: The width and height of the estimated region are set based on a comparison of the width and height of the image with the width and height of the hypothetical region calculated based on the ratio of the square root of the reduction region and the threshold area divided by the area of ​​the reduction region.

2. The image processing device according to claim 1, wherein:

7. When the area of ​​the reduced region is smaller than a threshold area, the second setting means If both the width and height of the provisional region calculated based on the ratio of the square root of the reduced area and the threshold area divided by the area of ​​the reduced area are equal to or less than the width and height of the image, set the width and height of the provisional region as the width and height of the estimated region; If at least one of the width and height of the tentative region is larger than the width and height of the image, the width and height of the estimated region are set based on at least one of the ratio between the width of the reduced region and the width of the image and the ratio between the height of the reduced region and the height of the image.

2. The image processing device according to claim 1, wherein:

8. The second setting means shifts the estimated area into the image when a part of the estimated area is outside the image.

2. The image processing device according to claim 1, wherein:

9. When an overlapping area where another image is overlapped on the image exists, the second setting means sets the estimated area to an area different from the overlapping area.

2. The image processing device according to claim 1, wherein:

10. a first setting step of setting a reduction region in an image; a second setting step of setting an estimation area in the image based on the reduced area; an estimation step of estimating noise characteristics of the estimation region; a noise reduction step of performing a noise reduction process to reduce noise in the reduction region of the image using parameters based on the noise characteristics, In the second setting step, when the area of ​​the reduced region is smaller than a threshold area, the estimated region is set to be equal to or larger than the threshold area, and when the area of ​​the reduced region is equal to or larger than the threshold area, the reduced region is set as the estimated region. An image processing method comprising:

11. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 9.

Images