Image processing device, image processing method, and program

Abstract

To provide a high quality image for which a smooth NR effect is applied.SOLUTION: An image processing device comprises: image processing means for performing image processing to an acquired input image, by a learned model which is acquired by learning a noise characteristic of an image and has a first parameter; noise reduction means for reducing noise in the input image which is subjected to the image processing by a second parameter; combining means for combining the input image and the input image from which the noise is reduced, on the basis of a third parameter; setting means for, when any of the first parameter, the second parameter, and the third parameter is changed, changing the parameter to be changed on the basis of any of a control amount and preset reference data over multiple input images which are sequentially acquired and setting the changed parameter.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In the surveillance market, extremely low light environments require improved visibility of the target subject, and so extremely high gain is applied when capturing images with cameras. While applying high gain brightens the captured image, it also increases noise, but in surveillance applications, subject visibility is important even at the expense of image quality for viewing purposes.

[0003] Noise reduction (hereinafter referred to as NR) has been commonly used to reduce noise contained in captured images. NR can be performed inside the camera or in an external image processing device that processes images input from the camera. In recent years, noise reduction processing based on deep learning artificial intelligence technology (hereinafter referred to as DLNR) has also been used. DLNR has been confirmed to be more effective than conventional NR.

[0004] Various image processing functions for captured images are configured by combining multiple different image processing methods, and the parameters used in each image processing method are also varied. In order to obtain optimal image processing effects for capturing various scenes, various combinations of image processing functions and parameters have been stored for each scene.

[0005] In Patent Document 1, a predicted image is generated from two interpolated images using a bidirectional gradient change process based on gradient change, making it possible to present to the viewer an image transition from a first state to a second state.

[0006] The technology of Patent Document 2 changes parameters in stages from parameter group A to parameter group B in order to adjust the image quality so that there is less sudden fluctuation in image quality and less sense of incongruity when switching scenes. Patent Document 2 lists brightness, sharpness, hue, saturation, etc. as examples of image processing functions and parameters. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Publication No. 2020-109919 [Patent Document 2] Japanese Patent Application Laid-Open No. 2007-288587 Summary of the Invention [Problem to be solved by the invention]

[0008] However, when switching parameters in DLNR, shocks can occur in the image. In Patent Document 1, when generating an interpolated image from two images, an image different from the actual image is predicted and generated, and if the prediction differs from the actual image, shock can occur in the image. Furthermore, in Patent Document 2, simply applying interpolation from pre-transition A to post-transition B can result in the image transition via parameters that make the image look unnatural. There was no disclosure regarding parameters that are difficult to change stepwise, such as those in a trained model used in DLNR.

[0009] Therefore, the present invention aims to provide high-quality images with a smooth NR effect by reducing the shock when switching parameters of a trained model such as DLNR or intensity parameters for reducing noise. [Means for solving the problem]

[0010] In order to solve this problem, for example, the image processing device of the present invention has the following configuration: The network parameters of the neural network arean image processing means for processing an acquired input image using a trained model having a first parameter; The noise of the image-processed input image is This is the strength parameter for emphasizing and reducing By the second parameter The noise a noise reduction means for reducing the noise; a synthesis means for synthesizing the input image and the noise-reduced input image based on a third parameter; When changing any one of the first parameter, the second parameter, and the third parameter, the parameter to be changed is changed over a plurality of input images that are sequentially acquired. , to approach the target value according to the user's selection. Control amount or a setting means for changing and setting the data based on preset reference data; Equipped with. [Effects of the Invention]

[0011] According to the present invention, it is possible to reduce the shock to the image when switching parameters such as the parameters of a trained model that performs image processing such as DLNR or parameters such as the intensity parameters for reducing noise, and to apply a smooth NR effect to provide a high-quality image. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing the configuration of an image processing apparatus. [Figure 2] 4 is a flowchart of image processing executed by the image processing device. [Figure 3] 10 is a flowchart of a subroutine of the initialization process of S100. [Figure 4] An explanatory diagram of a neural network that processes an input image. [Figure 5] 10 is a flowchart of a subroutine for image processing setting in S120. [Figure 6] 10 is a flowchart of a subroutine of noise estimation processing in S130. [Figure 7] FIG. 10 is a diagram showing an example of flat area extraction. [Figure 8] Examples of noise dispersion characteristics for different camera settings. [Figure 9] 10 is a flowchart of a subroutine for image processing change control in S140. [Figure 10] 10 is a flowchart of a subroutine for NR processing in S150. [Figure 11] A diagram showing images to be processed by NR in the order they are processed. [Figure 12] 10 is a flowchart of a subroutine of image output processing in S160. [Figure 13] 10 is a flowchart of a subroutine including NN redundancy switching processing in the image processing setting of S120. [Figure 14] 10 is a flowchart executed by the image processing apparatus according to the second embodiment. [Figure 15] 10 is a flowchart of a subroutine for image processing change control in S240. [Figure 16] An example of a reference table for setting values ​​to be referenced as the next setting value of S2417. [Figure 17] FIG. 10 is a block diagram showing the configuration of an image processing apparatus according to a third embodiment. [Figure 18] 10 is a flowchart executed by the image processing apparatus according to the third embodiment. [Figure 19] 10 is a flowchart of a subroutine of a camera parameter acquisition process in S320. [Figure 20] 10 is a flowchart of a subroutine for setting image processing in S330. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0014] The present embodiment will be described below. In this embodiment, an image processing device is exemplified that estimates noise from an input image from a connected camera, infers a desired output image using a neural network (NN), and outputs the inferred image. Images include video, still images, videos, and images of one frame of a video. In NN training, multiple student images and multiple corresponding teacher images are prepared. Next, training is performed to approximate the feature distribution of the student images to the feature distribution of the teacher images, and network parameters such as weights and biases are optimized. This enables accurate inference even for input images that have not been trained. By retaining trained parameters obtained by performing multiple training sessions according to the characteristics of the camera, inference can be performed on the input image to obtain an inferred image with reduced noise.

[0015] First Embodiment 1 is a block diagram showing the configuration of an image processing device 100. Note that, although an image processing device will be described in this embodiment, the processing described below can also be performed in a camera with interchangeable lenses, such as a video camera with interchangeable lenses, a single-lens reflex camera, or a mirrorless single-lens camera. For example, the processing described below can also be performed in a control device within the camera 200 shown in FIG. 1.

[0016] In FIG. 1, camera 200 captures a light beam incident from a subject field to generate an image. Camera 200 is equipped with or has built-in lenses (not shown), and includes a zoom lens group, a focus lens group, an iris mechanism, and the like. Camera 200 can change the exposure accumulation time, and an auto-exposure function can apply gain to the captured image when capturing images in dark places. Camera 200 sends the captured image to image processing device 100. Image processing device 100 acquires the image via image input unit 110.

[0017] The image processing device 100 includes an image input unit 110, a bus 120, a CPU 130, a memory 140, an operation input unit 150, an image processing unit 160, and an image output unit 170. The image input unit 110, the CPU 130, the memory 140, the operation input unit 150, the image processing unit 160, and the image output unit 170 are connected via the bus 120 so as to be able to transmit and receive data to and from each other. The CPU 130 or the image processing unit 160 executes a program to perform various processes, such as image processing to be described later, executed by the image processing device 100, and realize various functions. For example, the CPU 130 and the image processing unit 160 execute a program to realize functions such as a trained model, a noise reduction means, a synthesis means, a setting means, and an image synthesis means.

[0018] The image input unit 110 stores the input image acquired from the camera 200 in the memory 140 or the storage unit 180 via the bus 120 .

[0019] The operation input unit 150 acquires an operation signal input from a controller 300 external to the image processing device 100. The controller 300 has a switch, a keyboard, a mouse, a touch panel, etc. A user inputs an operation signal to the image processing device 100 via the controller 300.

[0020] The CPU 130 executes various processes such as image processing based on programs and parameters stored in the storage unit 180. The CPU 130 executes various processes based on operation signals acquired by the operation input unit 150. For example, the CPU 130 executes setting processes required for image processing executed by the image processing unit 160. In addition to the CPU 130, the image processing device 100 may have other processors such as an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), or a QPU (Quantum Processing Unit).

[0021] The memory 140 functions as a work area when the CPU 130 executes a program. The memory 140 also temporarily stores parameters and the like required when the CPU 130 executes a program. The memory 140 may be a RAM (Random Access Memory).

[0022] The image processing unit 160 reads out images stored in the memory 140 or the storage unit 180 and performs various image-related processes such as noise estimation processing, image signal synthesis processing, noise reduction processing, and UI image generation processing for display on a user interface (hereinafter referred to as UI), which will be described later. The image processing unit 160 stores the images that have undergone various processes in the memory 140 or the storage unit 180. The image processing unit 160 may include a processor such as a GPU (Graphics Processing Unit). The image processing unit 160 may also be realized as part of the functions of the CPU 130 that executes programs.

[0023] The image output unit 170 outputs the image processed by the image processing unit 160 and held in the memory 140 or the storage unit 180 to the outside of the image processing device 100. The image output unit 170 outputs an image signal for output from an HDMI (registered trademark) (High Definition Multimedia Interface) terminal or an SDI (Serial Digital Interface) terminal provided in the image processing device 100.

[0024] The storage unit 180 stores programs executed by the CPU 130, parameters required for the programs, etc. The storage unit 180 stores images acquired from the camera 200, images processed by the image processing unit 160, etc. The storage unit 180 has a non-volatile storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), and a ROM (Read Only Memory).

[0025] The monitor 400 receives and displays the image output by the image processing device 100. Users such as the photographer and viewers can check the image captured by the camera 200, the menu screen of the image processing device 100, and the image after image processing through the monitor 400. The monitor 400 may be, for example, a liquid crystal display device or an organic EL (Electro Luminescence) display device.

[0026] 1, each function is shown as a separate component, but these may be realized by hardware such as one or more ASICs (Application Specific Integrated Circuits), programmable logic arrays (PLAs), FPGAs (Field Programmable Gate Arrays), etc. Also, each of the above-described functions may be realized by a programmable processor such as a CPU or MPU executing software.

[0027] Next, the image processing performed by the CPU 130 in the image processing device 100 will be described with reference to Fig. 2. Fig. 2 is a flowchart of the image processing performed by the image processing device 100. When the image processing device 100 is powered on, the CPU 130 loads a computer program stored in the storage unit 180 and executes the processing in order from S100 in Fig. 2. After executing the processing up to S160 in Fig. 2, the CPU 130 starts the loop processing again from S110. The CPU 130 and the image processing unit 160 execute the processing from S110 to S160 for each frame or every few frames of the input image sequentially acquired from the camera 200. Either the CPU 130 or the image processing unit 160 executes each processing, but the entity that executes each processing may be changed as appropriate.

[0028] In S100 of Fig. 2, the CPU 130 performs initialization processing of the image processing device 100. Fig. 3 shows a flowchart of the subroutine of the initialization processing of S100.

[0029] 3, the CPU 130 executes initialization processing for image input / output. The CPU 130 initializes the image input unit 110 and the image output unit 170 so that the image processing device 100 can execute image input / output processing.

[0030] In S102, the CPU 130 performs initialization processing of the image processing function. The CPU 130 initializes the image processing unit 160 so that the image processing unit 160 can execute image processing. For example, the CPU 130 performs initial settings for performing luminance signal synthesis processing and color signal synthesis processing of the image after NR processing and the input image in the image synthesis processing described below. For example, when the image processing program is started, the CPU 130 performs initialization processing with the input image at an addition ratio of 100%. The addition ratio is a ratio used when synthesizing a luminance signal or color signal with an inference image.

[0031] In S103, the CPU 130 performs NR initialization processing. The CPU 130 loads trained NN parameters that have been pre-trained in advance for the NR processing to be executed by the image processing unit 160, and applies them to the trained model of the image processing unit 160. This embodiment assumes various models of cameras 200 connected to the image processing device 100 and setting states including the image quality of the cameras 200. Information about the various models of cameras 200 and the image quality settings of the cameras 200 is an example of information about the camera 200. The memory 140 or the storage unit 180 holds trained NN parameters optimized for each model of camera 200 and each image quality setting.

[0032] Here, the NN loaded into the image processing unit 160 will be described with reference to Fig. 4. Fig. 4 is an explanatory diagram of the NN that processes an input image. As an example, a convolutional NN (hereinafter referred to as CNN) is taken as an example, but the NN in the embodiment is not limited to a CNN. For example, the NN may be a generative adversarial network (GAN) or may have a skip connection. The NN may also be a recurrent type such as a recurrent neural network (RNN).

[0033] In FIG. 4(a), an input image 501 represents an image to be input to a neural network (NN) or a feature map (described later). A convolution matrix 503 in FIG. 4(a) is a filter that performs a convolution operation 502 on the input image 501. A bias 504 is a value added to the result output by the convolution operation 502 between the input image 501 and the convolution matrix 503. A feature map 505 is the result of the convolution operation after the bias 504 is added to the result output by the convolution operation 502. For simplicity, FIG. 4 depicts a small number of neurons, hidden layers, and channels. However, the number of neurons and layers, as well as the number and weights of connections between neurons, are not limited to this example. Furthermore, when the NN shown in FIG. 4 is implemented in an FPGA, ASIC, or the like, the number of connections and weights between neurons may be reduced.

[0034] CNN obtains a feature map of an input image by performing a convolution operation on the input image using a filter. The size of the filter can be arbitrary. In the next layer, a different feature map is obtained by performing a convolution operation on the feature map of the previous layer using a different filter. In each layer, an input signal is multiplied by a filter and summed with a bias. An activation function is then applied to the result to obtain an output signal for each neuron. The weights and biases in each layer are called NN parameters, and the NN parameters are updated during training. Examples of activation functions include the sigmoid function and the ReLU function. In this embodiment, the Leaky ReLU function shown in the following equation (1) is used, but this is not limited to this. In equation (1), max represents a function that outputs the maximum value among its arguments. f(x)=max(x,x×0.2) Equation (1) As described above, in the pre-learning to obtain the NN parameters, an image with the noise characteristics of the camera 200 is used as a student image, and a corresponding noise-free image of the student image is used as a teacher image. Learning is performed by pairing the student image and the teacher image, thereby realizing NR.

[0035] The aforementioned feature map 505 focuses on noise in the input image, and the CNN can learn a region of interest that emphasizes noise components by applying a different parameter to the feature map 505. As shown in Figure 4(b), the CNN generates an intermediate layer 602 by averaging the channel direction of an input image 601, which is obtained by dividing the input image 501 into channels, for example, by color. The CNN performs multiple convolutions on the intermediate layer 602 using equation (1) to obtain an intermediate layer 603. The CNN further performs convolutions so that the output channel is 1 to obtain an attention layer 604. The attention layer 604 is an intermediate layer in which features appear in noise regions excluding high-frequency components of the subject. The CNN multiplies the attention layer 604 by a noise intensity parameter 605 to obtain an attention layer 606 in which noise is emphasized. By convolving the attention layer 606 with the above-mentioned input image 501 together, the degree of attention paid to the noise areas contained in the input images 501 and 601 is adjusted, i.e., emphasized, thereby making the NN capable of NR processing.

[0036] Returning to the description of FIG. 3, in S104, the CPU 130 performs operation input initialization. The CPU 130 executes initialization processing for inputting an operation signal from the controller 300 external to the image processing device 100. The image processing device 100 also has a function capable of displaying a menu for the NR function. The CPU 130 can switch the NR function between enabled and disabled based on user input from the controller 300. The CPU 130 can also switch the NR mode setting between auto mode and manual mode, and in manual mode, can display an NR intensity menu setting screen for setting the NR intensity. As an example of this initialization processing, the CPU 130 sets NR enabled / disabled to enabled and sets the NR mode setting to auto mode. After the CPU 130 executes S104, it ends the subroutine S100 and proceeds to S110 in FIG. 2.

[0037] In S110 of FIG. 2, the CPU 130 stores the input image received by the image input unit 110 from the camera 200 in the memory 140.

[0038] In S120, the CPU 130 performs image processing settings. Fig. 5 shows a flowchart of the subroutine for image processing settings in S120.

[0039] In S121, the CPU 130 selects and sets the model of the camera 200 connected to the image processing device 100 based on information input via the controller 300 and the operation input unit 150. Note that the CPU 130 may select the model of the camera 200 based on information selected by the user from a setting menu of the image processing device 100, or may select the model of the camera 200 based on information automatically detected by the image input unit 110 from information embedded in the image signal.

[0040] In S122, similarly to S121, the CPU 130 selects the gamma applied to the camera 200 connected to the image processing device 100. Note that the CPU 130 may select the gamma based on information selected by the user from a setting menu of the image processing device 100 or the like, or may select the gamma based on information automatically detected by the image input unit 110 from information embedded in the image signal.

[0041] In S123, the CPU 130 determines whether the camera model was changed in S121 or whether the gamma selection was changed in S122. If the gamma selection was changed, the CPU 130 proceeds to S124, and if the gamma selection was not changed, the CPU 130 proceeds to S125.

[0042] In S124, the CPU 130 loads and sets the trained NN parameters corresponding to the model and gamma of the camera 200 selected in S121 and S122 into the image processing unit 160. As a result, the NN loaded into the image processing unit 160 becomes an NN having appropriate NN parameters trained according to the connected camera 200 and its gamma setting. In S124, the NN performs inference, i.e., the image processing unit 160 loads images for NR processing from the memory 140 and becomes ready for processing.

[0043] In S125 of Fig. 5, the CPU 130 selects the NR mode. The CPU 130 selects the NR mode based on an operation signal input from the controller 300 via the operation input unit 150 by the user or the like via a selection menu or the like. For example, the CPU 130 selects and sets either the auto mode or the manual mode as the NR mode based on the operation signal input by the user. Details of the NR mode processing will be described later in S140 and S150. After executing S125, the CPU 130 ends the subroutine S120 and proceeds to step S130 of Fig. 2.

[0044] In S130 of Fig. 2, the image processing unit 160 performs noise estimation processing on the input image acquired in S110. Fig. 6 shows a flowchart of the noise estimation processing subroutine in S130.

[0045] 6, the CPU 130 sets the input image as a noise-detection target region and divides the noise-detection target region into blocks. Note that in this embodiment, the entire input image is described as the noise-detection target region, but the CPU 130 may set a partial region of the input image as the noise-detection target region. For example, the CPU 130 may set 3×3 pixels of the input image as one block.

[0046] Next, in S132, the CPU 130 calculates the luminance and noise variance of each block as a divided region.

[0047] Next, in S133, the CPU 130 extracts a flat area of ​​noise variance in the noise detection area. FIG. 7 is a diagram illustrating an example of flat area extraction. Specifically, the CPU 130 extracts a flat area based on whether the noise variance value of each block is equal to or less than a threshold. For example, as shown in FIG. 7, if the noise variance of a block is equal to or less than the threshold, the CPU 130 determines that the block is a flat area and sets 1 to the block. On the other hand, if the noise variance of a block is equal to or greater than the threshold, the CPU 130 determines that the block is not a flat area and sets 0 to the block. If the above-mentioned block-by-block contains brightness differences such as texture information of the subject, the brightness differences will be calculated as noise variance values, making it difficult for the CPU 130 to accurately estimate noise.

[0048] In S134, the CPU 130 collects statistics of luminance versus noise variance for only the blocks determined to be flat areas and plots the luminance versus noise variance characteristics. As a result, even if luminance differences such as texture information of the subject are included in the above-mentioned block units, the CPU 130 can suppress the luminance differences from being calculated as noise variance values, thereby enabling accurate noise estimation.

[0049] In S135, the CPU 130 loads noise variance characteristic data corresponding to the currently set gamma characteristic. This noise variance characteristic data is stored in the storage unit 180 by measuring the luminance-to-noise variance characteristic for each gain in advance for the model of the connected camera 200 and the set gamma. Fig. 8 is a diagram showing examples of noise variance characteristics for each setting of the camera 200. For example, the storage unit 180 holds table data as noise variance characteristics, including noise variance characteristics associated with luminance and gain, as shown in Figs. 8(a) and 8(b).

[0050] In S136 of FIG. 6, the CPU 130 selects and refers to the noise dispersion characteristics at each gain contained in the noise dispersion characteristics table loaded in S135.

[0051] In S137, the CPU 130 measures the distance of the noise variance characteristic data corresponding to the gain selected in S136 relative to the variance value at each luminance plotted in S134, and performs a characteristic match determination. At this time, a match determination may be made if the distance is equal to or less than an arbitrary threshold. If the noise variance characteristics do not match in S137, i.e., if the result is false, the CPU 130 returns to step 136 and refers to the noise variance characteristic data for the next gain to perform a noise variance characteristic match determination. If the CPU 130 determines a match in S137, i.e., if the result is true, the CPU 130 proceeds to S138.

[0052] In S138, the CPU 130 estimates the gain for which it has been determined that the noise variance characteristics match in S136 and S137 as the gain input to the image processing device 100.

[0053] Various scenes are captured by the camera 200 and input to the image processing device 100. When estimating the noise variance characteristics shown in FIG. 6, it is desirable to secure as large an area as possible for the detection target region. The larger the detection target region, the higher the probability of detecting the number of flat blocks described above. As a result, it is possible to increase the amount of variance characteristic information compiled in S134 of FIG. 6.

[0054] Returning to the description of Fig. 6, after executing S138, the CPU 130 ends the subroutine of S130 in Fig. 6 and proceeds to S140 in Fig. 2. In S140 in Fig. 2, the CPU 130 executes image processing change control. Fig. 9 shows a flowchart of the subroutine of image processing change control of S140.

[0055] 9, CPU 130 selects whether NR is enabled or disabled. CPU 130 may select whether NR is enabled or disabled based on an operation signal input by a user or the like from controller 300 via operation input unit 150. CPU 130 may also select whether NR is enabled or disabled based on a signal or command input from outside via a network or the like. If S1401 is true, i.e., NR is enabled, CPU 130 proceeds to S1402, and if S1401 is false, i.e., NR is disabled, CPU 130 proceeds to S1409.

[0056] In S1402, the CPU 130 determines the NR mode selected in S125 of Fig. 5. If the determination in S1402 is true, that is, if the NR mode is auto mode, the CPU 130 proceeds to S1403, and if the determination is false, that is, if the NR mode is manual mode, the CPU 130 proceeds to S1405.

[0057] In step S1403, that is, when the NR mode is the auto mode, the CPU 130 acquires the gain of the input image estimated in step S138 of FIG.

[0058] In S1404, the CPU 130 sets a target value A when the NR mode is the auto mode. The target value A here refers to the target values ​​of the enhancement parameter, the luminance signal addition ratio, and the color signal addition ratio to be assigned to the noise intensity parameter 605 in FIG. 4. Since the higher the gain, the more noise is included, the CPU 130 sets a high value for the enhancement parameter target value A when the estimated gain of the camera 200 acquired in S1403 is high. On the other hand, the CPU 130 sets a low value for the enhancement parameter target value A when the estimated gain is low. As a result, the luminance signal addition ratio and the color signal addition ratio are set to low values ​​when the estimated gain is high, and set to high values ​​when the estimated gain is low. In particular, when the gain of the camera 200 is high, color noise may affect image quality. In this way, when the estimated gain is high, the CPU 130 may not set a color signal addition ratio.

[0059] The emphasis parameter will be described. The amount of noise input according to the camera 200 and the gamma setting value of the camera 200 may vary not only depending on the gain but also depending on the set gamma. If a high gain is estimated, the emphasis parameter is set to a high value, and the CPU 130 can increase the sensitivity to pay attention to noise so that a higher NR effect can be obtained. On the other hand, when a low gain is estimated, by setting the emphasis parameter low, it is possible to suppress the application of NR more than necessary, and as a result, the CPU 130 can suppress the adverse effects of NR such as loss of resolution and texture loss.

[0060] When S1405, that is, the NR mode is the manual mode, the CPU 130 sets the NR intensity in the same manner as S125 in FIG. 5. For example, the CPU 130 acquires the information of High, Middle, and Low selected by the user in the NR intensity menu and sets the NR intensity. That is, unlike the internal ones such as the aforementioned emphasis parameter, the NR intensity is selected externally by the user of the image processing apparatus 100 such as through the UI. When the CPU 130 sets High, it proceeds to S1406 and sets the target value B. When the CPU 130 sets Middle, it proceeds to S1407 and sets the target value C. When the CPU 130 sets Low, it proceeds to S1408 and sets the target value D. Similar to the target value A, the CPU 130 sets the target values B, C, and D for the emphasis parameter, the luminance signal addition ratio, and the color signal addition ratio, respectively. Since these processes are the operations when the NR mode is the manual mode, the image processing apparatus 100 stores predetermined parameters for each of the target values B, C, and D in the storage unit 180 or the like in advance. The target values of the emphasis parameter are maintained in the relationship of B > C > D. The target values of the luminance signal addition ratio and the color signal addition ratio are maintained in the relationship of B < C < D. Note that, similar to S1404 described above, at the target value B, when the estimated gain estimation is high, the CPU 130 may set the color signal addition ratio to be none.

[0061] In step S1409, that is, when NR is disabled, the CPU 130 sets the target value E. For example, the CPU 130 sets the emphasis parameter to disabled, and the luminance signal addition ratio and the color signal addition ratio to 100%, as the target value E.

[0062] When the CPU 130 executes any one of S1404, S1406, S1407, S1408, and S1409 to set the target values ​​A to E in the operating state of each NR, the process proceeds to S1410.

[0063] In S1410, the CPU 130 calculates the deviations of the current enhancement parameters, luminance signal addition ratio, and color signal addition ratio from the target values ​​described above.

[0064] In S1411, CPU 130 determines whether the deviation from the target value calculated in S1410 is greater than threshold value α. As described above, CPU 130 calculates the deviation from the target value for each of the emphasis parameter, luminance signal addition ratio, and color signal addition ratio, and therefore threshold value α may be set separately for each of the emphasis parameter deviation, luminance signal addition ratio deviation, and color signal addition ratio deviation. If S1411 is true, i.e., the deviation of the target value is greater than threshold value α, CPU 130 determines that each of the emphasis parameter, luminance signal addition ratio, and color signal addition ratio has not reached the target value, and proceeds to S1412; if S1411 is false, CPU 130 determines that each of the emphasis parameter, luminance signal addition ratio, and color signal addition ratio has sufficiently approached the target value, and proceeds to S1416.

[0065] In S1412, the CPU 130 sets the current control amount for each target value, i.e., the emphasis parameter, the luminance signal addition ratio, and the color signal addition ratio. In S1412, the control amount may be set using a method recognized in general control engineering. The control amount is a value added to the parameter to be changed, among the emphasis parameter, the luminance signal addition ratio, and the color signal addition ratio, in order to bring it closer to the target value.

[0066] In S1413, the CPU 130 determines whether the control amount calculated in S1412 exceeds a threshold value β. The value of the threshold value β may be any value. The threshold value β may be a constant value in the various NR states described above, or may be increased or decreased in conjunction with the threshold value α. The control amount calculated by the CPU 130 in S1412 may be applied as is, but in S1413, the CPU 130 limits the control amount for one input image to a constant amount, i.e., the threshold value β, with the aim of mitigating the amount of image change toward a set target value. If S1413 is true, i.e., the control amount exceeds the threshold value β, the CPU 130 proceeds to S1414. If false, i.e., the control amount is equal to or less than the threshold value β, the CPU 130 proceeds to S1415.

[0067] In S1414, the CPU 130 sets the control amount to the threshold value β. In S1416, the CPU 130 sets the control amount to 0.

[0068] In S1415, CPU 130 sets the next setting values ​​to the current setting values, i.e., the current emphasis parameter, luminance signal addition ratio, and color signal addition ratio, plus the control amount, and then terminates the subroutine of FIG. 9 and proceeds to S150 of FIG. 2. CPU 130 then repeats step S1415 multiple times for each input image acquired sequentially, thereby adding the control amount to the emphasis parameter, luminance signal addition ratio, and color signal addition ratio multiple times. In this way, CPU 130 brings the emphasis parameter, luminance signal addition ratio, and color signal addition ratio closer to their target values. FIG. 10 shows a flowchart of the NR processing subroutine of S150.

[0069] In S151 of Fig. 10, the CPU 130 sets the enhancement parameter set in S1415 of Fig. 9. The enhancement parameter is applied to the noise intensity parameter 605 described in Fig. 4, and acts as an enhancement parameter applied to an attention map focusing on noise.

[0070] In S152, the CPU 130 loads and acquires the image acquired in S110 of FIG. 2 from the memory 140 and inputs it to the image processing unit 160 for NR inference. FIG. 11 is a diagram showing images to be subjected to NR processing arranged in the order of processing. FIG. 11(a) is an example of an image of a dark area captured by the camera 200. The image in FIG. 11(a) includes an image of a person, although it is difficult to see. Because the camera 200 applies gain to the image in FIG. 11(a), the image processing device 100 acquires an image containing noise, as shown in FIG. 11(b). Therefore, the CPU 130 loads and acquires an image such as that shown in FIG. 11(b) from the memory 140 as an image for NR inference.

[0071] In S153, the image processing unit 160 executes the NR inference process.

[0072] In S154, the CPU 130 stores the image resulting from the inference performed by the image processing unit 160 in S153 (hereinafter referred to as the inference image) in the memory 140 or the like. The inference image obtained in S153 has had noise reduced by applying NR inference processing as shown in Fig. 11(c). When the CPU 130 executes S154, it ends the subroutine in Fig. 10 and proceeds to S160 in Fig. 2.

[0073] 2, the CPU 130 executes image output processing for outputting the inferred image that has been subjected to the NR processing up to S150. Fig. 12 shows a flowchart of the image output processing subroutine of S160.

[0074] In S161 of FIG. 12, the CPU 130 acquires the inference image saved in S154 of FIG.

[0075] In S162, the image processing unit 160 performs a luminance signal synthesis process on the inference image saved in S154. The image processing unit 160 synthesizes the luminance signal with the inference image based on the luminance signal addition ratio as the next setting value set in S1415 of FIG.

[0076] In S163, the image processing unit 160 performs a color signal synthesis process on the inference image with the luminance signal synthesized. The image processing unit 160 synthesizes the color signal with the inference image based on the color signal addition ratio set as the next setting value.

[0077] In S164, the CPU 130 outputs the image obtained by the synthesis process up to S163 by the image processing unit 160 from the image processing device 100 via the image output unit 170, and displays it on the monitor 400. For example, a part of the image in Fig. 11(b) in NR mode and in the intensity menu state is synthesized with the inference image shown in Fig. 11(c), resulting in the image shown in Fig. 11(d).

[0078] After executing S164, the CPU 130 ends the subroutine of the image output process in Fig. 12 and returns to the process in Fig. 2. After executing S160 in Fig. 2, the CPU 130 returns to S110 and repeats the image process again.

[0079] By repeatedly executing the image processing of FIG. 2 by CPU 130 when switching between NR enabled and disabled, when switching between NR auto mode and manual mode, and when setting the target value by changing the intensity setting from the menu, S1415 of FIG. 9 is updated, and the image output in S164 of FIG. 12 changes smoothly.

[0080] In the above-described embodiment, an example has been given in which target values ​​are set for the emphasis parameters, the luminance signal addition ratio, and the color signal addition ratio, and the control amounts are added to sequentially set the next setting values, but the targets to be sequentially set by adding the control amounts are not limited to the above-described example. For example, in this embodiment, target values ​​may be set for the NN parameters, and the NN parameters may be sequentially set by adding the control amounts to the NN parameters.

[0081] As described above, in this embodiment, target values ​​for the enhancement parameter, luminance signal addition ratio, and color signal addition ratio are set by switching NR between enabled and disabled, switching the NR mode between auto mode and manual mode, changing the intensity setting from a menu, etc. This embodiment has described a method in which the next setting values ​​for the enhancement parameter, luminance signal addition ratio, and color signal addition ratio are sequentially updated for each frame of image based on the control amount relative to the set target values. As a result, this embodiment suppresses sudden changes in NR parameters, including the enhancement parameter, luminance signal addition ratio, and color signal addition ratio, reduces image shock when parameters are changed, and provides a high-quality image to which a smooth NR effect is applied.

[0082] In this embodiment, the control amount is compared with a threshold value β, and if the control amount is greater than the threshold value β, the threshold value β is set as the control amount. This prevents the control amount from exceeding the threshold value β, and can mitigate changes in the emphasis parameter, luminance signal addition ratio, and color signal addition ratio to which the control amount is added.

[0083] In this embodiment, the target value is set based on the NR strength selected by the user, which allows the user to set a target value that meets the user's desire, and the image output after noise removal using the enhancement parameters, luminance signal addition ratio, and color signal addition ratio set based on the target value can be made to meet the user's desire.

[0084] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0085] In the above embodiment, as shown in FIG. 3, when there is a change in the model selection and gamma selection of the camera 200, the CPU 130 performs NR initialization in S124. At this time, it may take time for the CPU 130 to load the DLNR trained model, making it difficult to smoothly infer the NR effect. Therefore, the image processing device 100 may implement the image processing setting of S120 in FIG. 2 using a subroutine shown in FIG. 13 by making the NN processing redundant. FIG. 13 is a flowchart of a subroutine that includes NN redundancy switching processing in the image processing setting of S120. Note that in the description of FIG. 13, the description of steps similar to those in FIG. 5 will be omitted or simplified.

[0086] For example, the image processing device 100 may prepare two NN systems, NN-A and NN-B, as NN redundant processing systems, and perform NN processing on either NN-A or NN-B, and perform NR initialization on the other NN to change the NN parameters based on the control amount.

[0087] In S123, if the CPU 130 determines that the model or gamma of the camera 200 has been changed, it determines whether the NN in use is NN-A or NN-B as shown in S221, and if NN-A is in use, it proceeds to S222, and if NN-B is in use, it proceeds to S224.

[0088] In S222, the CPU 130 performs NR initialization on NN-B, which is exclusive to NN-A, and continues the NR processing by NN-A.

[0089] In S223, the CPU 130 sets the switching flag of NN-B.

[0090] In S224 and S225, the CPU 130 executes processing exclusive of S222 and S226. Specifically, in S224, the CPU 130 performs NR initialization on the NN-A, which is exclusive to the NN-B, and continues NN processing by the NN-B. In S225, the CPU 130 sets the switching in progress flag for the NN-A.

[0091] From the next time onward, in the process from S110 in FIG. 2, S123 in FIG. 13 is determined to be false, and the process proceeds to S226.

[0092] In S226, the CPU 130 refers to the switching flag set in S223 or S225, and if S226 is true, that is, if it determines that NN switching is in progress, the process proceeds to S227, and if it is false, that is, if it determines that NN switching has been completed, the process proceeds to S125.

[0093] In S227, CPU 130 updates the composition ratio, which is the ratio for composition of an image NN-A with an image NN-B, or for composition of an image NN-B with an image NN-A. Here, CPU 130 changes the NN parameters of the NN system to be switched to target values ​​by transitioning them across multiple images using a predetermined control amount, as shown in FIG. 9. Therefore, CPU 130 may transition and update the composition ratio in accordance with the progress of the transition of the NN parameters being changed. Note that image processing unit 160 composites the image output from NN-A and the image output from NN-B based on the composition ratio.

[0094] In S228, the CPU 130 determines whether the switching of NN has been completed. For example, when the composition ratio indicating the composition ratio of the image from either NN-A or NN-B to the other has transitioned to 100%, the CPU 130 determines that this is true, i.e., that the NN switching has been completed, and proceeds to S229, but if this is false, i.e., if the NN switching is in progress, proceeds to S125.

[0095] In S229, the CPU 130 clears the NN-A switching flag and the NN-B switching flag set in S223 or S225, and then the process proceeds to S125.

[0096] Smoother DLNR switching processing is possible by making the NN processing redundant and by the subroutine shown in Fig. 13. Furthermore, this may be implemented in combination with the switching processing shown in Figs. 9, 11, and 12.

[0097] Second Embodiment In the second embodiment, a method is shown in which the next set values ​​of the enhancement parameters, luminance signal addition ratio, and color signal addition ratio for the set target values ​​are stored in advance as a reference table, and are updated sequentially for each frame of image.

[0098] The configuration of the image processing device 100 in the second embodiment is the same as that in the first embodiment, ie, FIG. 1, and therefore description thereof will be omitted.

[0099] Next, the image processing performed by the CPU 130 in the image processing device 100 will be described with reference to Fig. 14. The same processing as in the first embodiment is executed from S100 to S130 in Fig. 14, and then the process proceeds to S240. Image processing change control is performed in S240 in Fig. 14. The subroutine of S240 is shown in Fig. 15.

[0100] In S1401 of Fig. 15, the CPU 130 selects whether NR is enabled or disabled, as in Fig. 9. If S1401 is true, the CPU 130 proceeds to S1402, and if it is false, the CPU 130 proceeds to S2409. In S1402, the CPU 130 determines the NR mode, as in Fig. 9. If S1402 is true, that is, the NR mode is auto mode, the CPU 130 proceeds to S2403, and if it is false, that is, the NR mode is manual mode, the CPU 130 proceeds to S1405. In S2403, the CPU 130 acquires an estimated gain value A of the input image, as in S138 and S1403 of the first embodiment, and proceeds to S2404.

[0101] In S2404, the CPU 130 stores the estimated gain value A when the NR mode is the auto mode in the variable TempGain, and proceeds to S2410.

[0102] In S1405, i.e., when the NR mode is manual mode, the user of the image processing device 100 switches the gain value by operating an external NR intensity menu on a UI or the like. If the NR intensity setting is set to High in accordance with the user's selection, the CPU 130 proceeds to S2406 and stores a gain value B in the variable TempGain. If the NR intensity setting is set to Middle, the CPU 130 proceeds to S2407 and stores a gain value C in the variable TempGain. If the NR intensity setting is set to Low, the CPU 130 proceeds to S2408 and stores a gain value D in the variable TempGain.

[0103] As in the first embodiment, the gain values ​​B to D are used in the case where the NR mode is manual mode. The magnitude relationship between the gain values ​​is B>C>D.

[0104] In step S2409, the CPU 130 stores the gain value E in the case where NR is disabled in the variable TempGain. For example, the CPU 130 sets the gain value E to, for example, 1x or 0 dB. Note that the gain value E may generally be set to a low gain, i.e., a gain value where NR is not required. Note that the relationship with the gain value D is D>E.

[0105] After S2404, S2406, S2407, S2408, and S2409 have been executed, the process proceeds to S2410.

[0106] In this embodiment, in order to smoothly transition the enhancement parameters, luminance signal addition ratio, and color signal addition ratio, the gain values ​​stored in the variable TempGain are stored in an array Arr. Note that in order to hold the gain values ​​for the most recent frames, the gain values ​​are stored in a so-called FIFO (First In First Out) format. Detailed processing will be described later from step S2410 onward. For example, if the array length of array Arr is set to 10, the TempGain values ​​updated for each frame will be held for the most recent 10 frames. By averaging the gain values ​​stored in array Arr, gain values ​​to which a time filter has been applied for the most recent 10 frames can be obtained. The gain values ​​stored in TempGain are subjected to a time filter in response to changes in settings or, if the NR mode is auto mode, changes in camera noise, resulting in a gradually changing average gain value. Note that the array length of array Arr is not limited and may be any length. The array length of array Arr is indicated by the variable length.

[0107] In S2410, the CPU 130 initializes a reference index variable i of the array Arr to 0. Furthermore, the CPU 130 initializes a variable Sum for calculating the sum of the gain values ​​stored in the array Arr to the value stored in TempGain, that is, the latest gain value.

[0108] At S2411, the CPU 130 stores Arr[i+1], the (i+1)th element of the array Arr, in Arr[i], the i-th element of the array Arr. At S2412, the CPU 130 increments the index variable i by 1. At S2413, the CPU 130 adds the gain value stored in the i-th element of the array Arr to the variable Sum. At S2414, the CPU 130 determines whether the index variable i is smaller than the array length variable length-1 of the array Arr. If S2414 is true, that is, if the CPU 130 determines that the index variable i is smaller than the array length variable length-1, the process returns to S2411 and repeats the process. If S2414 is false, that is, if the CPU 130 determines that the index variable i is not smaller than the array length variable length-1, the FIFO process for the array Arr is completed, and the sum Sum stored in the array Arr is calculated, and the process proceeds to S2415.

[0109] In S2415, the CPU 130 stores the value obtained by dividing the above-mentioned sum value Sum by the array length (length) of the array Arr, i.e., the average value of the gain values, in the variable AveGain. In S2416, the CPU 130 converts the variable AveGain into a gain index value for referencing each table of the enhancement parameters, luminance signal addition ratios, and color signal addition ratios. For example, if the array length (length) of the array Arr is 10, the CPU 130 calculates the average value of the gain values ​​for the most recent 10 frames and derives the corresponding gain index value.

[0110] FIG. 16 shows examples of reference tables for the setting values ​​to be referenced as the next setting values ​​for the enhancement parameters, luminance signal addition ratio, and color signal addition ratio. The reference tables are an example of reference data. FIG. 16(a) shows an enhancement parameter table. FIG. 16(b) shows a luminance signal addition ratio table. FIG. 16(c) shows a color signal addition ratio table. Each row stores parameters for each gain index value for the camera and its gamma. For example, in the enhancement parameter table of FIG. 16(a), the enhancement multiplier is set to 1.0 when the gain value is low, and the higher the gain, the higher the value. In the luminance signal addition ratio table of FIG. 16(b) and the color signal addition ratio table of FIG. 16(c), the ratio is increased when the gain value is low and decreased when the gain value is low. Therefore, each table may be based on the same concept as described in the first embodiment. It is recommended that each parameter be determined in advance to achieve optimal NR image quality based on the gain characteristics of the camera and gamma. The resolution of the gain index value may be, for example, 1 dB. A finer resolution will result in a smoother NR change. However, since a finer resolution may increase the memory size of each table, it is better to use 3 dB or 6 dB resolution. In S2417, you can simply implement a formula for converting gain index values ​​according to the table resolution.

[0111] Returning to the description of Fig. 15, in S2417, the CPU 130 refers to each reference table described in Fig. 16 based on the table reference index converted in S2416, acquires the enhancement parameter, the luminance signal addition ratio, and the color signal addition ratio, and sets them in the image processing unit 160.

[0112] By repeating step S2417 multiple times for sequentially acquired input images, CPU 130 can gradually change the above-mentioned average values ​​and gradually bring the referenced enhancement parameters, luminance signal addition ratio, and color signal addition ratio closer to the target values.

[0113] After the CPU 130 executes S2417, the process proceeds to S2418. In S2418, the CPU 130 stores the TempGain value in Arr[i+1], which is the (i+1)th element of the array Arr. If the array length is 10, the latest gain value is stored at i+1=9, that is, at the end of the array Arr.

[0114] When CPU 130 executes S2418, it ends the subroutine of Fig. 15 and proceeds to S150 of Fig. 14. S150 and S160 of Fig. 14 are the same processes as S150 and S160 of Fig. 2, and therefore description thereof will be omitted.

[0115] As described above, the second embodiment has shown an embodiment in which the next setting values ​​of the enhancement parameter, luminance signal addition ratio, and chrominance signal addition ratio are stored in advance as a reference table. Furthermore, the second embodiment makes it possible to smoothly change the next setting values ​​to be referenced by averaging the gain values ​​obtained for each frame over multiple frames, i.e., by applying a time filter.

[0116] Third Embodiment In the first and second embodiments, an example has been shown in which the gain value set in the camera 200 is estimated from the amount of noise contained in the video input from the camera 200 to the image processing device 100 by the noise estimation process shown in S130 in Fig. 2 and Fig. 14. However, if the camera 200 is equipped with a communication means, the image processing device 500 may be able to acquire the camera name, gamma setting, and gain setting of the camera 200. The image processing device 500, which can acquire the setting values ​​themselves, can accurately acquire gain value information of the camera 200, and can perform more optimal noise reduction processing.

[0117] FIG. 17 is a diagram showing the configuration of an image processing device 500 according to the third embodiment. The image processing device 500 has a configuration in which a communication unit 510 is added to the image processing device 100 shown in FIG. 1. The communication unit 510 is connected to the bus 120. The CPU 130 transmits commands corresponding to the protocol of the camera 200 via the communication unit 510 to control the camera 200 and acquire the setting values ​​of the camera 200. The physical communication format between the camera 200 and the communication unit 510 can be Ethernet, serial communication, or the like, but the communication format is not particularly limited. In the case of Ethernet, a protocol based on socket communication or a WebAPI using the HTTP protocol is generally used. In the case of serial communication, a protocol based on start-stop synchronous communication is used. These protocols are also not particularly limited.

[0118] Next, the image processing performed by the CPU 130 in the image processing device 500 will be described with reference to Fig. 18 to Fig. 20. The CPU 130 performs the same processing as in the first embodiment from S100 to S110 in Fig. 18, and then proceeds to S320. In S320 in Fig. 18, the CPU 130 performs a camera parameter acquisition process to acquire camera information. As described above, the CPU 130 acquires the setting values ​​and the like of the camera 200 through the communication unit 510 in accordance with the communication protocol of the camera 200.

[0119] 19 is a diagram showing a subroutine of the camera parameter acquisition process of S320. In S321 of FIG. 19, CPU 130 acquires model information of camera 200. In S322, CPU 130 acquires gamma information currently set in camera 200. In S323, CPU 130 acquires a gain value currently set in camera 200. The CPU 130 also sets the acquired gain value as an estimated gain value, which will be described later. In S324, CPU 130 acquires a shutter value, i.e., an exposure time setting value, currently set in camera 200. When CPU 130 has executed up to S324, it ends the subroutine of FIG. 19 and proceeds to S330 of FIG. 18.

[0120] In S330 of FIG. 18, the CPU 130 performs image processing settings.

[0121] FIG. 20 is a diagram of the image processing setting subroutine of S330 in FIG. 18. Among the steps of the image processing setting in FIG. 20, steps similar to those of S120 in FIG. 2 will be briefly described. In S331, the CPU 130 selects the camera model information acquired in S321 in FIG. 19. In S332, the CPU 130 selects the gamma value information selected for the camera 200 acquired in S322 in FIG. 19. In the first and second embodiments, examples were shown in which the user selected the camera and the camera's gamma value information via a menu on the image processing device 100, but in this embodiment, the CPU 130 selects the camera and the gamma value information acquired from the camera 200 via communication. Therefore, the NR setting value is automatically selected according to the gamma value setting value of the connected camera 200. After the CPU 130 executes S332, the process proceeds to S123. The same processing as in the first embodiment is executed from S123 to S125, and therefore description thereof will be omitted. In this subroutine, the CPU 130 automatically executes NR initialization when the connected camera 200 is changed or when the gamma value information of the camera 200 is changed. When the CPU 130 executes the NR mode selection process of S125, the subroutine of Fig. 20 ends and the process proceeds to S140 of Fig. 18.

[0122] 18, the CPU 130 executes the same process as in the first embodiment, and therefore the description thereof will be omitted. Note that even if S140 in this embodiment is replaced with S240 described in FIG. 14 of the second embodiment, the subsequent processes are executed in the same manner.

[0123] The processing in S150 and S160 in FIG. 18 is the same as that in the first embodiment, and therefore a description thereof will be omitted.

[0124] As described above, in the third embodiment, it has been shown that by obtaining setting value information directly from the camera 200, it is possible to perform processing similar to that of the above-described embodiments.

[0125] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0126] In this embodiment, an example is shown in which an image is input from a camera to the image processing device, but the present invention can also be implemented in a configuration in which an image is input from an image output device other than a camera.

[0127] The present invention also includes cases where a software program that realizes the functions of the above-described embodiments is supplied to a system or device having a computer that can execute the program directly from a recording medium or via wired / wireless communication, and the program is executed.

[0128] Therefore, the program code itself that is supplied to and installed on a computer to realize the functional processing of the present invention also realizes the present invention. In other words, the computer program itself for realizing the functional processing of the present invention is also included in the present invention.

[0129] In this case, as long as it has the functionality of a program, the form of the program does not matter, such as object code, a program executed by an interpreter, or script data supplied to an OS.

[0130] The recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as a magnetic tape, an optical / magneto-optical storage medium, or a non-volatile semiconductor memory.

[0131] Another method of supplying the program is to store the computer program forming the present invention in a server on a computer network, and have connected client computers download and program the computer program.

[0132] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0133] The disclosure of this specification includes the following image processing device, image processing method, and program. (Item 1) an image processing means for performing image processing on an acquired input image using a trained model having first parameters obtained by training noise characteristics of the image; a noise reduction means for reducing noise in the image-processed input image using a second parameter; a synthesis means for synthesizing the input image and the noise-reduced input image based on a third parameter; a setting means for changing and setting the parameter to be changed across a plurality of input images acquired sequentially based on a control amount or preset reference data when changing any of the first parameter, the second parameter, and the third parameter; An image processing device comprising: (Item 2) The first parameter is a parameter of a neural network resulting from learning noise characteristics based on information about the camera that captured the input image. 2. The image processing device according to item 1, (Item 3) The second parameter is a parameter indicating the strength of the noise reduction. 3. The image processing device according to item 1 or 2, (Item 4) The third parameter is a ratio for combining the noise-reduced input image with the input image. 4. The image processing device according to any one of items 1 to 3, wherein: (Item 5) The third parameter is a ratio for combining the luminance signal of the noise-reduced input image with the luminance signal of the input image. 5. The image processing device according to any one of items 1 to 4, wherein: (Item 6) The third parameter is a ratio for combining the color signal of the noise-reduced input image with the color signal of the input image. 6. The image processing device according to any one of items 1 to 5, wherein: (Item 7) The setting means sets a lower ratio for combining the color signals as the noise in the input image increases. 7. The image processing device according to item 6, (Item 8) the image processing means has two trained models based on the first parameters, and when one trained model is in use, the first parameters of the other trained model are changed across a plurality of input images using the control amount; an image synthesis means for synthesizing an input image that is image processed by the one trained model and output after noise reduction by the noise reduction means and an input image that is image processed by the other trained model and output after noise reduction by the noise reduction means based on a synthesis ratio; 8. The image processing device according to any one of items 1 to 7, further comprising: (Item 9) The synthesizing means synthesizes the images based on the synthesis ratio that changes in accordance with the progress of the change of the first parameter being changed. 9. The image processing device according to item 8, (Item 10) The setting means changes the second parameter and the third parameter by setting the control amount so that the second parameter and the third parameter approach the target value of the parameter to be changed. 10. The image processing device according to any one of items 1 to 9, wherein: (Item 11) The setting means changes the first parameter, the second parameter, and the third parameter based on the control amount when either the first parameter or the target value is changed. 11. The image processing device according to item 10. (Item 12) The setting means compares the control amount with a threshold value, and if the control amount is greater than the threshold value, sets the threshold value as the control amount. 11. The image processing device according to item 10. (Item 13) The setting means sets the target value in accordance with the intensity of the noise reduction selected by a user. Item 13. The image processing device according to item 12. (Item 14) The setting means acquires at least one of the second parameter and the third parameter from reference data stored in advance. 14. The image processing device according to any one of items 1 to 13, (Item 15) The setting means acquires at least one of the second parameter and the third parameter from the reference data based on an average value of gain values ​​of images of a plurality of frames. Item 15. The image processing device according to item 14. (Item 16) The setting means sets at least one of the second parameter and the third parameter based on information about a camera acquired from the camera that captured the input image. 16. The image processing device according to any one of items 1 to 15, (Item 17) an image processing step of processing an acquired input image using a trained model having first parameters obtained by training noise characteristics of the image; a noise reduction step of reducing noise in the image-processed input image using a second parameter; a combining step of combining the input image and the noise-reduced input image based on a third parameter; a setting step of changing and setting the parameter to be changed across a plurality of input images acquired sequentially based on a control amount or preset reference data when changing any of the first parameter, the second parameter, and the third parameter; An image processing method comprising: (Item 18) A program for causing a computer to function as each means of the image processing device according to any one of items 1 to 16.

[0134] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0135] 100···Image processing device, 200···Camera, 130···CPU, 160···Image processing unit, 170···Image output unit.

Claims

1. an image processing means for performing image processing on an acquired input image using a trained model having first parameters which are network parameters of a neural network obtained by training noise characteristics of the image; a noise reduction means for reducing the noise in the image-processed input image using a second parameter, which is an intensity parameter for emphasizing and reducing the noise; a synthesis means for synthesizing the input image and the noise-reduced input image based on a third parameter; a setting means for changing and setting the parameter to be changed across a plurality of input images acquired sequentially, when changing any of the first parameter, the second parameter, and the third parameter, based on a control amount for bringing the parameter closer to a target value selected by a user or preset reference data; An image processing device comprising:

2. The first parameter is a parameter of a neural network resulting from learning noise characteristics in information about the camera, which is information about the model or image quality settings of the camera that captured the input image.

2. The image processing device according to claim 1, wherein:

3. The second parameter is a parameter indicating the strength of the noise reduction.

2. The image processing device according to claim 1, wherein:

4. The third parameter is a ratio for combining the noise-reduced input image with the input image.

2. The image processing device according to claim 1, wherein:

5. The third parameter is a ratio for combining the luminance signal of the noise-reduced input image with the luminance signal of the input image.

2. The image processing device according to claim 1, wherein:

6. The third parameter is a ratio for combining the color signal of the noise-reduced input image with the color signal of the input image.

2. The image processing device according to claim 1, wherein:

7. The setting means sets a lower ratio for combining the color signals as the noise in the input image increases.

7. The image processing device according to claim 6,

8. the image processing means has two trained models based on the first parameters, and when one trained model is in use, the first parameters of the other trained model are changed across a plurality of input images using the control amount; an image synthesis means for synthesizing an input image that is image processed by the one trained model and output after noise reduction by the noise reduction means and an input image that is image processed by the other trained model and output after noise reduction by the noise reduction means based on a synthesis ratio; The image processing device according to claim 1 , further comprising:

9. The synthesizing means synthesizes the images based on the synthesis ratio that changes in accordance with the progress of the change of the first parameter that is being changed.

9. The image processing device according to claim 8,

10. The setting means changes the second parameter and the third parameter by setting the control amount so that the second parameter and the third parameter approach the target value of the parameter to be changed.

2. The image processing device according to claim 1, wherein:

11. The setting means changes the first parameter, the second parameter, and the third parameter based on the control amount when either the first parameter or the target value is changed.

11. The image processing device according to claim 10.

12. The setting means compares the control amount with a threshold value, and if the control amount is greater than the threshold value, sets the threshold value as the control amount.

11. The image processing device according to claim 10.

13. The setting means sets the target value in accordance with the intensity of the noise reduction selected by a user.

13. The image processing device according to claim 12.

14. The setting means acquires and sets at least one of the second parameter and the third parameter from reference data stored in advance.

2. The image processing device according to claim 1, wherein:

15. The setting means acquires at least one of the second parameter and the third parameter from the reference data based on an average value of gain values ​​of images of a plurality of frames.

15. The image processing device according to claim 14.

16. The setting means sets at least one of the second parameter and the third parameter based on information about a camera acquired from the camera that captured the input image.

2. The image processing device according to claim 1, wherein:

17. an image processing step of performing image processing on the acquired input image using a trained model having first parameters which are network parameters of a neural network obtained by training noise characteristics of the image; a noise reduction step of reducing the noise in the image-processed input image using a second parameter, which is an intensity parameter for enhancing and reducing the noise; a combining step of combining the input image and the noise-reduced input image based on a third parameter; a setting step of changing and setting the parameter to be changed across a plurality of input images acquired sequentially based on a control amount for bringing the parameter closer to a target value selected by a user or based on preset reference data, when changing any of the first parameter, the second parameter, and the third parameter; An image processing method comprising:

18. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 16.

Images