Image processing apparatus, image processing method, and program

Abstract

To generate a restored image in which image quality degradation is sufficiently reduced.SOLUTION: An image processing apparatus 100 according to the present disclosure for generating restored image data in which image quality degradation contained in an input image is reduced by inference using a neural network, is configured to estimate the degree of the image quality degradation contained in the input image, and determine adjustment parameters to be used in an image restoration process for reducing the image quality degradation based on the estimated degree of the image quality degradation.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a technique for generating an image with reduced degradation in image quality from an image with degraded image quality. [Background technology]

[0002] There is a technology that uses a deep neural network (hereinafter referred to as "DNN") to restore an image from an image with degraded image quality (hereinafter referred to as a "degraded image"). A DNN refers to a neural network with two or more hidden layers, and performance can be improved by increasing the number of hidden layers. Factors that cause image quality degradation include noise, blur, low resolution, and loss, and processes that reduce image quality degradation, i.e., processes that restore an image (hereinafter referred to as "image restoration processes") include noise removal, blur removal, super-resolution, and loss completion. One factor that can cause image quality degradation is the time of image capture or the image capture conditions. For example, due to changes in the surrounding brightness or the shutter speed, the number of photons entering the light receiving element of the image sensor decreases, resulting in a lower signal-to-noise ratio and an image with noise added (degraded image).

[0003] Patent Document 1 discloses a technique for adjusting the strength of image restoration processing for each input image, in which a feature map is generated in a DNN that indicates the strength of the image restoration processing corresponding to each input image, and each pixel value of the generated feature map is multiplied by a given coefficient. According to the technique disclosed in Patent Document 1, by multiplying each pixel value of the generated feature map by a given coefficient, it is possible to adjust the strength of the image restoration processing for each input image when generating an image with reduced degradation of image quality (hereinafter referred to as a "restored image"). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2023-47600 Summary of the Invention [Problem to be solved by the invention]

[0005] However, simply multiplying each pixel value of the generated feature map by a given coefficient may not be enough to generate a restored image with sufficiently reduced degradation in image quality. Therefore, the present disclosure aims to provide a technology capable of generating a restored image with sufficiently reduced degradation in image quality. [Means for solving the problem]

[0006] The image processing device of the present disclosure is an image processing device that generates data of a restored image in which image quality degradation contained in an input image is reduced by inference using a neural network, and has an estimation means that estimates the degree of image quality degradation contained in the input image, and a determination means that determines adjustment parameters to be used in an image restoration process to reduce the image quality degradation based on the estimated degree of image quality degradation. Effect of the Invention

[0007] According to the present disclosure, it is possible to generate a restored image in which degradation of image quality is sufficiently reduced. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of a configuration of an image processing system according to a first embodiment. [Diagram 2] 1 is a block diagram showing an example of each functional configuration of an image processing device and an information processing device according to a first embodiment. [Diagram 3] 4 is a flowchart showing an example of a processing flow in the information processing device according to the first embodiment. [Figure 4] 5 is a flowchart showing an example of a processing flow in the image processing device according to the first embodiment. [Diagram 5] 5 is a flowchart showing an example of the flow of noise amount estimation processing and determination processing according to the first embodiment. [Figure 6]FIG. 11 is a diagram showing an example of a UI screen according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present disclosure is not limited to the configurations shown in the drawings. Hereinafter, the same configurations will be described with the same reference numerals.

[0010] <CNNについて> Before describing each embodiment, a convolutional neural network (hereinafter referred to as "CNN") used in general image processing technology applying deep learning will be described. In CNN, a filter generated as a result of learning is convolved with image data, and then nonlinear arithmetic processing is performed, and this convolution processing and nonlinear arithmetic processing are repeated. The filter is also called a local receptive field. An image obtained by convolving the filter with image data and then nonlinear arithmetic processing is called a feature map. CNN learning is performed using learning data consisting of a set of input image data and correct image data as output image data (hereinafter referred to as "teacher image data"). Simply put, CNN learning is to generate, from learning data, filter values ​​that can convert input image data into output image data corresponding to the input image data with high accuracy. Details of CNN learning will be described later.

[0011] When the input image data has multiple color channels such as R (Red), G (Green), and B (Blue), or when the feature map is composed of multiple images, the filter used in the convolution process also has multiple channels according to the number of color channels. That is, the filter used in the convolution process is expressed using a four-dimensional array that includes the vertical and horizontal sizes of the image, the number of feature maps, and the number of channels. The nonlinear arithmetic process after convolution of the image data including the feature map is expressed in units of layers, such as the nth layer feature map (n is an integer equal to or greater than 1) and the nth layer filter. For example, a CNN that repeats the convolution by a filter and the nonlinear arithmetic process three times has a three-layer network structure. Such nonlinear arithmetic process can be formulated, for example, using the following equation (1). TIFF2025030735000002.tif7170

[0012] Here, W n is the nth layer filter, b n is the bias of the nth layer, f is the nonlinear operator, X n is the n-th layer feature map, and * is the convolution operator. The (l) on the right indicates that it is the l-th filter or feature map. The filters and biases are generated by learning, which will be described later, and are collectively called "network parameters." As the nonlinear arithmetic processing, a sigmoid function or ReLU (Rectified Linear Unit), etc., are used. When ReLU is used as the nonlinear arithmetic processing, the nonlinear operator f is given, for example, by the following formula (2). TIFF2025030735000003.tif8150

[0013] As shown in Equation (2), for the elements of the vector X input to the non-linear operator f, those that are negative output 0, and those that are 0 or greater output their original values. As neural networks using CNNs, ResNet in the field of image recognition and RED-Net which applies ResNet in the field of super-resolution are known. Any neural network can improve the processing accuracy by stacking multiple CNNs and performing convolution of filters multiple times. For example, ResNet features a network structure with a path that omits the processing of one or more convolutional layers. As a result, a multi-layer network of 152 layers can be realized, and high-precision recognition approaching the human recognition rate can be achieved. The reason why the processing accuracy is improved by making the CNN a multi-layer network is simply that the non-linear relationship between the input and output can be expressed by repeating the non-linear operation process multiple times.

[0014] <Learning of CNN> The learning of CNN will be described. The learning of CNN is generally performed by minimizing the objective variable of the loss function represented by the following Equation (3) for the training data consisting of a set of input image (also referred to as "student image") data and teacher image data corresponding to the student image data. TIFF2025030735000004.tif7150

[0015] Here, L is a loss function that measures the error between the inference result of the CNN for the input and the correct answer corresponding to the input. Also, Y i is the i-th teacher image data, and X i is the i-th input image (student image) data. F is a function that collectively represents the operations shown as an example in Equation (1) performed in each layer of the CNN. θ is the network parameter (filter and bias). ||Z||2 is the L2 norm, which is simply the square root of the sum of the squares of the elements of the vector Z. n is the total number of training data used for learning.

[0016] Generally, the total amount of training data is huge, so in the stochastic gradient descent method, a part of the training image data is randomly selected, and training is performed using the selected training data. This can reduce the computational load in training using a huge amount of training data. In addition, various methods such as the momentum method, the AdaGrad method, the AdaDelta method, and the Adam method are known as methods for minimizing, i.e. optimizing, the objective variable in the loss function. For example, the Adam method is given using a series of equations shown in the following equations (4) to (7). TIFF2025030735000005.tif30150

[0017] Here, θ i t is the i-th network parameter at iteration t, and g is the i t is the gradient of the loss function L with respect to m and v are moment vectors, α is the basic learning rate, β1 and β2 are hyperparameters, and ε is a small constant. Note that since there are no guidelines for selecting an optimization method in learning, any method can be used in principle, but it is known that the convergence of each method differs, resulting in differences in learning time.

[0018] [Embodiment 1] In this embodiment, a method is described in which the degree of degradation of image quality in an input image is estimated, and the degree of image restoration processing of the input image is adjusted based on the estimation result without changing the network configuration of the CNN. Note that in this embodiment, as an example, a case in which noise, which is one of the elements of image quality degradation, is included in the input image is described.

[0019] <Image processing system configuration example> FIG. 1 is a block diagram showing an example of the configuration of an image processing system 1 according to the first embodiment. The image processing system 1 includes an image processing device 100, an imaging device 10, an input device 20, an external storage device 30, a display device 40, and an information processing device 150, and the image processing device 100 and the information processing device 150 are connected to each other via the Internet so that they can communicate with each other. The information processing device 150 generates learning data and learns a learning model that realizes image restoration processing (hereinafter referred to as "degradation restoration learning"). The image processing device 100 performs inference processing of a restored image (hereinafter referred to as "degradation restoration inference") using a learned model obtained as a result of the degradation restoration learning, i.e., weight parameters of a learned neural network, to generate data of the restored image.

[0020] <Hardware configuration of image processing device> The image processing device 100 is configured with a personal computer or the like, acquires captured image data output by the imaging device 10, and generates restored image data corresponding to the captured image data. Here, the captured image data is, for example, RAW image data in a Bayer array or the like. Specifically, the image processing device 100 applies weight parameters of a trained neural network provided by the information processing device 150 to the captured image data, and generates a restored image by performing an inference process for degradation restoration. A user can generate restored image data corresponding to the captured image data by using image processing application software installed in the image processing device 100.

[0021] The image processing device 100 has a CPU 101, a RAM 102, a ROM 103, a large-capacity storage device 104, a general-purpose I / F 105, and a network I / F 106, and each component is connected to each other so as to be able to communicate with each other via a system bus 107. The image processing device 100 is also connected to the imaging device 10, the input device 20, the external storage device 30, and the display device 40 so as to be able to communicate with each other via the general-purpose I / F 105.

[0022] The CPU 101 uses the RAM 102 as a work memory, executes programs stored in the ROM 103, and performs overall control of each component of the image processing device 100 via a system bus 107. The mass storage device 104 is configured, for example, by a hard disk drive or a solid state drive, and stores various data and the like handled in the image processing device 100. The CPU 101 writes data to the mass storage device 104 and reads data stored in the mass storage device 104 via the system bus 107. The general-purpose I / F 105 is, for example, a serial bus interface such as USB (Universal Serial Bus), IEEE1394, or HDMI (registered trademark) (High-Definition Multimedia Interface).

[0023] The image processing device 100 acquires data from an external storage device 30 constituted by various storage media such as a memory card, a CF (CompactFlash (registered trademark)) card, an SD (registered trademark) card, or a USB memory via a general-purpose I / F 105. The image processing device 100 also accepts user instructions from an input device 20 such as a mouse or a keyboard via the general-purpose I / F 105. The image processing device 100 also outputs image data processed by the CPU 101 to a display device 40 constituted by various image display devices such as a liquid crystal display via the general-purpose I / F 105. The image processing device 100 also acquires captured image data to be subjected to degradation restoration inference from the imaging device 10 via the general-purpose I / F 105. The network I / F 106 is an interface for connecting to the Internet. The image processing device 100 accesses the information processing device 150 via an installed web browser or the like, and acquires weight parameters of a trained neural network for degradation restoration inference.

[0024] <Hardware configuration of information processing device> The information processing device 150 is composed of a server and the like. The information processing device 150 may be composed of a cloud server in which a plurality of servers cooperate with each other. The information processing device 150 has a CPU 151, a ROM 152, a RAM 153, a mass storage device 154, and a network I / F 155, and each component is connected to be able to communicate with each other via a system bus 156. The CPU 151 uses the RAM 153 as a work memory to read out a control program stored in the ROM 152 and execute various processes, thereby controlling the operation of the entire information processing device 150. The RAM 153 is used as a temporary storage area such as the main memory and work area of ​​the CPU 151. The mass storage device 154 is composed of, for example, a hard disk drive, and stores image data, various programs, and the like. The network I / F 155 is an interface for connecting to the Internet. The information processing device 150 outputs weight parameters of a trained neural network for inference processing of degradation restoration to the image processing device 100 in response to a request from the image processing device 100.

[0025] Note that the image processing device 100 and the information processing device 150 have components other than those described above, but since they are not the main focus of this disclosure, their description will be omitted. In addition, in the first embodiment, the information processing device 150 generates learning data and performs degradation restoration learning, and the learned model, which is the result of the learning, is output to the image processing device 100, and the image processing device 100 performs degradation restoration inference for the captured image data. However, this form is merely an example and is not limited to this. For example, the functions of the information processing device 150 may be subdivided, and the generation of learning data and the degradation restoration learning may be performed in separate devices. In addition, for example, a device such as an imaging device 10 that has both the functions of the image processing device 100 and the functions of the information processing device 150 may be configured to perform all of the generation of learning data, degradation restoration learning, and degradation restoration inference.

[0026] <Functional configuration of image processing system> FIG. 2 is a block diagram showing an example of each functional configuration of the image processing device 100 and the information processing device 150 included in the image processing system 1 according to the first embodiment. The image processing device 100 has, as its functional configuration, an image acquisition unit 211, a parameter acquisition unit 212, a degradation restoration unit 213, a noise amount estimation unit 214, a noise amount determination unit 215, and an adjustment parameter update unit 216. The information processing device 150 has, as its functional configuration, a degradation addition unit 251, a degradation restoration unit 252, an error calculation unit 253, and a learning model update unit 254. The functional configuration shown in FIG. 2 may be modified or changed as appropriate. For example, one functional unit may be divided into multiple functional units, or two or more functional units may be integrated into one functional unit. The functional configuration shown in FIG. 2 may be realized by two or more devices. In this case, the devices are connected via a circuit or a wired or wireless network, and perform data communication with each other to operate in cooperation, thereby achieving the processing described later in each functional unit.

[0027] <Application examples of image processing systems> Hereinafter, as an example, a case where the image processing system 1 is applied to real-time monitoring will be described. In real-time monitoring, captured image data obtained by imaging using the imaging device 10 is transmitted from the imaging device 10 to the image processing device 100 in chronological order. The degradation restoration unit 213 performs image restoration inference on the captured image data in chronological order received by the image processing device 100 using the adjustment parameters acquired by the parameter acquisition unit 212. The restored image 119 generated by the image restoration inference is displayed on the display device 40, allowing the user to perform real-time monitoring. Here, the adjustment parameters may be preset values, or may be values ​​updated at a timing desired by the user.

[0028] Due to differences in the imaging time or imaging conditions, for example, the amount of noise added to a captured image changes between imaging in a bright scene such as daytime or indoors with lights on and imaging in a dark scene such as nighttime or indoors with lights off. If adjustment parameters according to this changing amount of noise are not appropriately set, the amount of noise reduction becomes insufficient or excessive in the image restoration in the degradation restoration unit 113. As a result, there is a possibility that appropriate restored image data with sufficiently reduced degradation of image quality will not be generated. In addition, when a user manually updates the adjustment parameters according to changes in the amount of noise in a captured image, the user is required to take the trouble of setting the adjustment parameters. The image processing device 100 automatically updates the adjustment parameters according to changes in degradation of image quality in a captured image. The processing of each functional unit of the image processing device 100 and the processing flow in the image processing device 100 will be described later.

[0029] <Processing of each functional unit of the information processing device> The processing of each functional unit of the information processing device 150 will be described. The degradation adding unit 251 acquires data of the teacher image 255, and generates data of the student image by adding noise, which is one of the elements of image quality degradation, to the acquired data of the teacher image 255. An image data pair consisting of the data of the teacher image 255 and the student image data is used as learning data in learning of a learning model described later. The learning data is input to the degradation restoration unit 252 and the error calculation unit 253. The teacher image 255 may include various types of images such as natural photos with a landscape or a living thing as a subject, and portraits or photos of people such as sports events with a person as a subject. The teacher image 255 is not limited to the above, and may include images such as artificial object photos with an artificial object such as a building or a product as a subject. In addition, the teacher image 255 may include multiple images with different brightnesses. Hereinafter, the data of the teacher image 255 will be described as RAW image data in which each pixel has a pixel value corresponding to at least one color of R, G, and B, as an example.

[0030] The degradation restoration unit 252 performs degradation restoration learning using the learning data generated by the degradation adding unit 251. Specifically, first, the degradation restoration unit 252 acquires the network parameters 256, and initializes the weight parameters of the CNN that is the target of degradation restoration learning using the acquired network parameters 256. Next, the degradation restoration unit 252 receives the learning data from the degradation adding unit 251, inputs the student image data included in the learning data to the CNN, and repeats the convolution operation by the filter according to the formulas (1) and (2) and the nonlinear operation a number of times. The restored image data is generated by repeating these operations. The generated restored image data is input to the error calculation unit 253. The error calculation unit 253 acquires the data of the teacher image 255 and the restored image data included in the learning data, and calculates the error between the data of the teacher image 255 and the restored image data using formula (3). The learning model update unit 254 updates the weight parameters in the CNN so that the error calculated by the error calculation unit 253 becomes smaller. The details of the CNN according to this embodiment and the processing flow in the information processing device 150 will be described later.

[0031] <Processing of each functional unit of the image processing device> The processing of each functional unit of the image processing device 100 will be described. The image acquisition unit 211 sequentially acquires captured image data obtained by imaging using the imaging device 10 in chronological order as data of an input image 217. The data of the input image 217 acquired by the image acquisition unit 211 is input to a degradation restoration unit 213 and a noise amount estimation unit 214. As an example, the data of the input image 217 will be described below as RAW image data in which each pixel has a pixel value corresponding to at least one color of R, G, and B. In addition, the RAW image data will be described below on the premise that it is image data obtained by imaging using a Bayer array color filter in which each pixel has information for one color.

[0032] The parameter acquisition unit 212 acquires an adjustment parameter for adjusting the strength of the image restoration process (hereinafter referred to as "degradation restoration strength"). The parameter acquisition unit 212 holds the acquired adjustment parameter until a new adjustment parameter is acquired, and deletes the previously acquired adjustment parameter when the new adjustment parameter is acquired. For example, the parameter acquisition unit 212 acquires the adjustment parameter 118 as the initial value of the adjustment parameter by an external device not shown in Figs. 1 and 2 or an input operation by the user. Thereafter, the parameter acquisition unit 212 acquires the adjustment parameter output by the adjustment parameter update unit 216 described later as a new adjustment parameter. The parameter acquisition unit 212 may acquire various other parameters used by the degradation restoration unit 213 in addition to the adjustment parameter.

[0033] The degradation restoration unit 213 receives the trained model output from the information processing device 150, i.e., the trained CNN weight parameters, the captured image data acquired by the image acquisition unit 211, and the adjustment parameters acquired by the parameter acquisition unit 212. Using the input information, the degradation restoration unit 213 performs degradation restoration inference for the captured image data input as data of the input image 217. Specifically, the degradation restoration unit 213 inputs the captured image data acquired by the image acquisition unit 211 to the trained model 257, and repeats the convolution operation and nonlinear operation by the filter according to the formulas (1) and (2) a plurality of times. Then, the degradation restoration unit 213 generates data of the restored image 219 using the adjustment parameters held by the parameter acquisition unit 212.

[0034] More specifically, the adjustment parameter is a coefficient α used when processing each pixel, and the degradation restoration strength is adjusted by multiplying each pixel value in the feature map obtained as a result of the filter operation by the coefficient α. When the coefficient α is greater than 1, the strength of the image restoration process for the input image data is increased, and conversely, when the coefficient α is less than 1, the strength of the image restoration process for the data of the input image 217 is decreased. The data of the restored image 219 generated by the degradation restoration unit 213 is output to, for example, the external storage device 30 or the display device 40.

[0035] Here, the CNN used as the learning model according to the first embodiment will be described. The CNN according to the first embodiment includes one or more layers (hereinafter referred to as "map generation layers") for generating a feature map indicating the strength of the image restoration process for input image data. The CNN according to the first embodiment also includes one or more layers (hereinafter referred to as "strength adjustment layers") for adjusting the degradation restoration strength by multiplying each pixel value in the generated feature map by a coefficient α, i.e., an adjustment parameter. Furthermore, the CNN according to the first embodiment includes one or more layers (hereinafter referred to as "image generation layers") for generating data of the restored image 219 corresponding to the data of the input image 217 based on the feature map after each pixel value is multiplied by the coefficient α.

[0036] In degradation restoration learning in the information processing device 150, for example, the adjustment parameters, which are weight parameters in the intensity adjustment layer, are fixed to a predetermined positive real number such as "1.0". In this case, in degradation restoration learning in the information processing device 150, the adjustment parameters, which are weight parameters in the intensity adjustment layer, are fixed, and the weight parameters in the map generation layer and the image generation layer are updated. In degradation restoration inference in the image processing device 100, in a trained model obtained as a result of degradation restoration learning in the information processing device 150, the adjustment parameters held by the parameter acquisition unit 212 are substituted for the adjustment parameters, which are weight parameters in the intensity adjustment layer. By such substitution, it is possible to adjust the degradation restoration intensity without changing the network configuration of the CNN.

[0037] The noise amount estimation unit 214 sequentially receives captured image data (data of the input image 217) acquired by the image acquisition unit 211, and sequentially stores the received captured image data in the RAM 102 or the like. Hereinafter, the multiple captured image data that the noise amount estimation unit 214 sequentially stores in the RAM 102 or the like is referred to as a captured image data group. The noise amount estimation unit 214 estimates the amount of noise in the captured image data included in the captured image data group at a predetermined timing. Specifically, for example, the noise amount estimation unit 214 estimates the amount of noise for each captured image data included in the captured image data group. Note that a method for estimating the amount of noise in the captured image data will be described later. The noise amount estimation result for each of the multiple captured image data by the noise amount estimation unit 214 is output to the noise amount determination unit 215.

[0038] The noise amount determination unit 215 determines the amount of noise based on the result of estimating the amount of noise for each of the multiple captured image data by the noise amount estimation unit 214. Specifically, the noise amount determination unit 215 determines the amount of noise based on statistics of the estimation results of the amount of noise in the multiple captured image data in time series. Furthermore, the noise amount determination unit 215 determines adjustment parameters corresponding to the determined amount of noise, and outputs the determined adjustment parameters to the adjustment parameter update unit 216. Note that the method of determining the amount of noise and the method of determining the adjustment parameters corresponding to the determined amount of noise will be described later.

[0039] The adjustment parameter update unit 216 determines whether the current adjustment parameter held by the parameter acquisition unit 212 and the new adjustment parameter determined by the noise amount determination unit 215 have different values. If the current adjustment parameter and the new adjustment parameter have different values, the adjustment parameter update unit 216 outputs the new adjustment parameter to the parameter acquisition unit 212, and the parameter acquisition unit 212 acquires it. As a result, the adjustment parameter held by the parameter acquisition unit 212 is updated.

[0040] <Processing flow of information processing device> The operation of the information processing device 150 will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of a processing flow in the information processing device 150 according to the first embodiment. The CPU 151 reads a program for implementing the processing of the flowchart shown in Fig. 3 from the mass storage device 154 or the like, and executes the program to implement the functions of the above-mentioned respective units that the information processing device 150 has as its functional configuration. In the following description, the symbol "S" means a step (process).

[0041] First, in S301, the degradation restoration unit 252 acquires the network parameters 256, and initializes the weight parameters of the CNN that is the target of degradation restoration learning using the acquired network parameters 256. As described above, the CNN is a DNN including a map generation layer, an intensity adjustment layer, and an image generation layer. Next, in S302, the degradation addition unit 251 acquires data of the teacher image 255. Next, in S303, the degradation addition unit 251 generates data of a student image by adding noise to the data of the teacher image 255 acquired in S302. Next, in S304, the degradation restoration unit 252 inputs the student image data generated in S303 to the CNN and executes image restoration processing on the student image data. Next, in S305, the error calculation unit 253 calculates the error between the data of the teacher image 255 acquired in S302 and the restored image data output as a result of the image restoration processing in S304.

[0042] Next, in S306, the learning model update unit 254 updates the weight parameters in the CNN so that the error calculated in S305 becomes smaller. Here, the learning model update unit 254 fixes the weight parameters of the layer corresponding to the intensity adjustment layer among the weight parameters of each layer in the CNN, and updates only the weight parameters of the layers corresponding to the map generation layer and the image generation layer. Next, in S307, the learning model update unit 254 determines whether or not the weight parameters in the CNN have been updated a predetermined number of times, that is, whether or not the CNN has been learned a predetermined number of times. If it is determined in S307 that the update (learning) has not been performed a predetermined number of times, the information processing device 150 repeatedly executes the processes from S302 to S307 until it is determined in S307 that the update (learning) has been performed a predetermined number of times. If it is determined in S307 that the update (learning) has been performed a predetermined number of times, in S308, the learning model update unit 254 outputs the weight parameters of the learned CNN, which is the learned model 257, to the image processing device 100. After S308, the information processing device 150 ends the process of the flowchart shown in FIG.

[0043] <Processing flow of the image processing device 100> The operation of the image processing device 100 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of a processing flow in the image processing device 100 according to the first embodiment. The processing of the flowchart shown in Fig. 4 is started when the image processing device 100 is powered on and receives input of captured image data from the imaging device 10. The CPU 101 reads a program for implementing the processing of the flowchart shown in Fig. 4 from the mass storage device 104 or the like, and executes the program to implement the functions of the above-mentioned respective units that the image processing device 100 has as a functional configuration.

[0044] First, in S401, the degradation restoration unit 213 acquires weight parameters of a trained CNN, which is a trained model, from the information processing device 150, and expands them in the RAM 102. Also, in S401, the parameter acquisition unit 212 acquires adjustment parameters 218. After S401, in S402, the image acquisition unit 211 acquires captured image data (data of the input image 217) output from the imaging device 10. The captured image data acquired in S402 is transmitted to the degradation restoration unit 213 and the noise amount estimation unit 214, and the degradation restoration unit 213 and the noise amount estimation unit 214 receive it.

[0045] Here, the noise amount estimation unit 214 stores the captured image data received in S402 in the RAM 102 or the like in sequence and accumulates it. For example, the noise amount estimation unit 214 prepares an image buffer for sequentially accumulating the received captured image data in the RAM 102, and sequentially accumulates the received captured image data in the image buffer. Specifically, for example, when accumulating data of 100 captured images, the noise amount estimation unit 214 stores the captured image data from the 1st to the 100th captured images in chronological order in the image buffer, with the captured image data received at a certain time point being the data of the first captured image. When the noise amount estimation unit 214 receives the data of the 101st captured image, the noise amount estimation unit 214 deletes the data of the first captured image from the image buffer, and stores the data of the 101st captured image next to the data of the 100th captured image. By configuring the image buffer as a ring buffer in this way, the noise amount estimation unit 214 can temporarily accumulate data of a predetermined number of captured images in the image buffer.

[0046] After S402, in S403, the noise amount estimation unit 214 determines whether or not a predetermined timing set in advance has arrived. If it is determined in S403 that the predetermined timing has arrived, the image processing device 100 executes the process of S404, and if it is determined in S403 that the predetermined timing has not arrived, the image processing device 100 executes the process of S408. In S404, the noise amount estimation unit 214 estimates the amount of noise in the captured image data accumulated in the image buffer in S402. Details of the noise amount estimation method in S404 will be described later with reference to FIG. 5. After S404, in S405, the noise amount determination unit 215 determines the amount of noise at the above-mentioned predetermined timing based on the estimation result of the amount of noise in each captured image data estimated in S404. Furthermore, following S405, the noise amount determination unit 215 determines a new adjustment parameter corresponding to the amount of noise based on the determined amount of noise. Specifically, for example, the noise amount determination unit 215 holds a lookup table in which adjustment parameter values ​​corresponding to each noise amount are estimated in advance, and determines the adjustment parameter value corresponding to the determined noise amount by referring to the lookup table. Details of the method for determining the noise amount in S405 will be described later with reference to FIG.

[0047] After S405, in S406, the adjustment parameter update unit 216 determines whether or not to update the existing adjustment parameters held by the parameter acquisition unit 212. For example, the adjustment parameter update unit 216 compares the existing adjustment parameters held by the parameter acquisition unit 212 with the new adjustment parameters determined in S405, and determines whether or not the adjustment parameters have different values. If the adjustment parameters have different values, the adjustment parameter update unit 216 determines to update the existing adjustment parameters held by the parameter acquisition unit 212 using the new adjustment parameters determined in S405. If the adjustment parameters are not different values, that is, if the adjustment parameters have the same value, the adjustment parameter update unit 216 determines not to update the existing adjustment parameters held by the parameter acquisition unit 212.

[0048] Note that the same value here is not limited to a completely same value, and may include an approximately same value. In this case, the adjustment parameters being different from each other does not necessarily include the case where the difference between the adjustment parameters is within a predetermined range. If it is determined in S406 that the existing adjustment parameters held by the parameter acquisition unit 212 are to be updated, the image processing device 100 executes the process of S407. On the other hand, if it is determined in S406 that the existing adjustment parameters held by the parameter acquisition unit 212 are not to be updated, the image processing device 100 executes the process of S408.

[0049] If it is determined in S406 that the existing adjustment parameters held by the parameter acquisition unit 212 are to be updated, in S407 the adjustment parameter update unit 216 outputs the new adjustment parameters determined in S405 to the parameter acquisition unit 212. In S407, the parameter acquisition unit 212 acquires the new adjustment parameters output by the adjustment parameter update unit 216, and updates the existing adjustment parameters it holds to the acquired new adjustment parameters. The new adjustment parameters after the update are held as the existing adjustment parameters in the parameter acquisition unit 212. After S407, the image processing device 100 executes the process of S408.

[0050] In S408, the degradation restoration unit 213 executes image restoration processing by inputting the captured image data acquired in S402 to the trained model 257. Specifically, the degradation restoration unit 213 inputs the captured image data acquired in S402 to the CNN in which the adjustment parameters for adjusting the degradation restoration intensity held by the parameter acquisition unit 212 are input to the weight parameters in the intensity adjustment layer in the trained model 257. Data of the restored image 219 generated by the image restoration processing is transmitted to, for example, the display device 40 and displayed.

[0051] After S408, in S409, the image processing device 100 judges whether or not to end the series of processes from S402 to S408. Specifically, for example, if the image acquisition unit 211 acquires new captured image data from the imaging device 10, the image processing device 100 judges not to end the series of processes described above. Also, if the image acquisition unit 211 does not acquire new captured image data from the imaging device 10 within a predetermined period, the image processing device 100 judges to end the series of processes described above. If it is judged not to end the series of processes described above in S409, the image acquisition unit 211 returns to S402 and continues the series of processes described above until it is judged to end the series of processes described above in S409. On the other hand, if it is judged to end the series of processes described above in S409, the image acquisition unit 211 ends the process of the flowchart shown in FIG. 4. Note that the condition for judging whether or not to end the process in S409 is not limited to the presence or absence of input of new captured image data from the imaging device 10, and for example, the image processing device 100 may make a judgment based on an input of an instruction from a user.

[0052] <Noise amount estimation and calculation processing> The noise amount estimation process in S404 and the noise amount determination process in S405 will be described in detail with reference to Fig. 5. Fig. 5(a) is a flowchart showing an example of the process flow of S404 in the noise amount estimation unit 214 according to the first embodiment. First, in S501, the noise amount estimation unit 214 extracts captured image data to be subjected to noise amount estimation process from the captured image data group accumulated in the image buffer in S402. Specifically, for example, the noise amount estimation unit 214 determines the captured image data to be extracted based on a preset execution timing and a preset division number to be used.

[0053] Here, the information indicating the execution timing is information indicating at what timing the noise amount estimation unit 214 executes the noise amount estimation process. For example, in a case where 10 captured image data are output from the imaging device 10 per second and the image acquisition unit 211 acquires this, the noise amount estimation unit 214 is set to execute the noise amount estimation process when data of 10 captured images is newly accumulated in the image buffer. In this case, the interval of the execution timing is 1 second, and the noise amount estimation unit 214 executes the noise amount estimation process once per second. By setting the execution timing in this way, the image processing device 100 can adjust the deterioration restoration strength within the interval of the execution timing at the latest in response to changes in imaging conditions such as changes in brightness in the imaging range.

[0054] The interval of the execution timing is not limited to a fixed interval such as one second interval. For example, when the captured image data captured in a sufficiently bright environment such as during the day or indoors with the lights on is acquired, the amount of noise added to the captured image is smaller than that of the captured image data captured in a dark environment. Therefore, in such a case, the interval of the execution timing can be set longer than in other cases, assuming that the change in the amount of noise in the captured image data is small. On the other hand, in a period in which the brightness of the imaging range changes from moment to moment, such as from evening to nighttime, the amount of noise in the captured image data may change according to the change in brightness. In such a period in which the brightness changes from moment to moment, the interval of the execution timing of the noise amount estimation process can be shortened compared to other periods, so that the deterioration restoration strength can be adjusted more quickly following the change in the amount of noise.

[0055] The number of divisions used indicates the number of one set of captured images used by the noise amount estimation unit 214 in the noise amount estimation process. For example, when the number of divisions used is 5, the noise amount estimation unit 214 extracts one of two sets of captured image data by treating each set of five captured images in chronological order from the data of 10 captured images newly stored in the image buffer. The noise amount estimation unit 214 estimates the noise amount in a series of processes in S502 and S503 described later using the set of captured image data extracted based on the number of divisions used that is set in advance. By treating the data of a plurality of captured images as one set of captured image data in this way, it is possible to exclude image areas with movement in the captured images, that is, image areas that are not flat over time, during the flat area extraction process in S502 described later.

[0056] In this embodiment, for the sake of simplicity, the number of divisions to be used is set to 1. In other words, in the following description, the noise amount estimation unit 214 extracts data of one captured image from data of a plurality of captured images newly stored in the image buffer in the process of S501. The data set of one captured image or one captured image extracted in S501 becomes the captured image data to be processed in the series of processes of S503 and S504.

[0057] After S501, in S502, the noise amount estimation unit 214 analyzes the captured image data extracted in S501 to extract flat image regions (hereinafter referred to as "flat regions") in the captured image. A flat region is any image region in the captured image in which the spatial change in pixel values ​​is small. Specifically, for example, the noise amount estimation unit 214 analyzes pixel values ​​of an image region of s×t pixels surrounding a pixel of interest (i, j) in the captured image, and extracts the image region as a flat region when the change in pixel values ​​of the image region is small. Here, each of i, j, s, and t is an arbitrary integer equal to or greater than 0. Hereinafter, a region other than the flat region in the captured image is referred to as a non-flat region. A non-flat region is, for example, an image region including an edge in the captured image, and is an image region in which the difference between values ​​of pixels existing in close proximity to each other is large.

[0058] More specifically, for example, the noise amount estimation unit 214 calculates a value indicating the variation of pixel values ​​in an image region of s×t surrounding pixels including the coordinates (i, j) of the pixel of interest in the captured image. Hereinafter, the noise amount estimation unit 214 will be described as calculating a variance value as an example of a value indicating the variation of the pixel values. If the calculated variance value is equal to or less than a preset threshold, the noise amount estimation unit 214 extracts the image region of s×t surrounding pixels including the coordinates (i, j) of the pixel of interest as a flat region. On the other hand, if the calculated variance value is greater than the threshold, the noise amount estimation unit 214 does not extract the image region of s×t surrounding pixels including the coordinates (i, j) of the pixel of interest as a flat region. As a result of the flat region extraction, the noise amount estimation unit 214 does not use the image region that is not extracted as a flat region, i.e., a non-flat region, in subsequent processing. The noise amount estimation unit 214 can improve the accuracy of the noise amount estimation by estimating the noise amount in subsequent processing using only the extracted flat region.

[0059] In addition, when a set of captured image data consisting of a plurality of captured image data is extracted in S501, the noise amount estimation unit 214 executes the following process in S502. In this case, for example, the noise amount estimation unit 214 extracts flat regions from each of the plurality of captured image data extracted in S501, and sets the common regions in the flat regions of each captured image data as the flat region of the captured image data set. In addition, for example, the noise amount estimation unit 214 may calculate the variance value of pixel values ​​in an image region that combines image regions composed of pixels at the same positions in each of the plurality of captured image data, and extract the image region in which the calculated variance value is smaller than a predetermined value as the flat region. By determining the flat region in this way, it is possible to exclude image regions with movement in the captured image, that is, image regions that are not flat over time, from the flat region.

[0060] After S502, in S503, the noise amount estimation unit 214 calculates the amount of noise in the flat region extracted in S502. For example, the noise amount estimation unit 214 calculates the amount of noise in the flat region by calculating an arbitrary pixel coordinate (h i ,h j ) is calculated to show the variation of pixel values ​​in the surrounding image area of ​​r × q pixels. i , h j Each of a, r, and q is an arbitrary integer equal to or greater than 0. In the following description, the noise amount estimation unit 214 will be described as calculating a standard deviation as an example of a value indicating the variation in pixel values. Furthermore, the noise amount estimation unit 214 calculates the above-mentioned standard deviation for all pixels in the flat region extracted in S502, calculates a median of all the calculated standard deviations, and calculates the amount of noise by setting the calculated median as the amount of noise.

[0061] It should be noted that the method of calculating the amount of noise is not limited to the above-mentioned method. For example, the noise amount estimation unit 214 may take the average value of all the calculated standard deviations as the amount of noise, instead of the median value of the standard deviations calculated for all pixels in the flat region. Also, for example, the noise amount estimation unit 214 may calculate the amount of noise based on the luminance values ​​of pixels included in the flat region. Specifically, for example, the noise amount estimation unit 214 may calculate the amount of noise based on the luminance values ​​of pixels included in the flat region. i ,h j Alternatively, the standard deviation of the luminance values ​​in a surrounding image region of r×q pixels including the pixel value q may be calculated, and the standard deviation of the luminance values ​​included in the predetermined range may be used as the amount of noise.

[0062] After S503, in S504, the noise amount estimation unit 214 judges whether or not there remains captured image data that has not been extracted in S501 and that should be the target of the noise amount estimation process among the captured image data group stored in the image buffer in S402. If it is judged that there remains captured image data in S504, the noise amount estimation unit 214 returns to the process of S501 and repeatedly executes the processes from S501 to S504 until it is judged that there is no captured image data remaining in S504. Note that in this case, in S501, the noise amount estimation unit 214 extracts captured image data that has not been extracted in S501 and that should be the target of the noise amount estimation process among the captured image data group stored in the image buffer in S402. If it is judged that there is no captured image data remaining in S504, the noise amount estimation unit 214 ends the process of the flowchart shown in FIG. 5(a), that is, the process of S404.

[0063] FIG. 5B is a flowchart showing an example of the process flow of S405 in the noise amount determination unit 215 according to the first embodiment. First, in S511, the noise amount determination unit 215 acquires the estimation result of the noise amount estimated by the noise amount estimation unit 214 in S404, that is, the calculation results of the multiple noise amounts calculated by the noise amount estimation unit 214 in S503. Next, in S512, the noise amount determination unit 215 arranges the calculation results of the multiple noise amounts acquired in S511 in chronological order based on the acquisition time of the captured image data corresponding to each calculation result of the noise amount, and calculates a moving average of the noise amount in the time series. For example, the noise amount determination unit 215 calculates the moving average of the noise amount by calculating a simple arithmetic mean of the noise amount using the calculation results of the noise amounts adjacent in the chronological order. Note that the method of acquiring the moving average of the noise amount is not limited to the above-mentioned method. For example, the noise amount determination unit 215 may calculate the moving average of the noise amount by performing weighting based on the difference in the difference in the time interval between the noise amounts adjacent in the chronological order, and then calculating the weighted moving average of the noise amount.

[0064] Next, in S513, the noise amount determination unit 215 determines the value of the moving average of the noise amount calculated in S512 as the latest noise amount at the execution timing. Next, in S514, the noise amount determination unit 215 determines an adjustment parameter corresponding to the noise amount determined in S513, for example, by referring to the above-mentioned lookup table. Next, in S515, the noise amount determination unit 215 transmits the noise amount determined in S514 to the adjustment parameter update unit 216 as a new adjustment parameter. After S515, the noise amount determination unit 215 ends the process of the flowchart shown in FIG. 5(b), that is, the process of S405. Note that, in the above description, the calculated value of the moving average of the noise amount is determined as the latest noise amount at the execution timing, but the method of determining the noise amount is not limited to the method by calculating the moving average of the noise amount. For example, the noise amount determination unit 215 may calculate a statistical value indicating a trend of multiple noise amounts, such as an average value, a median value, a mode value, a minimum value, or a maximum value, in the calculation results of multiple noise amounts, and determine the calculated statistical value as the latest noise amount at the execution timing.

[0065] As described above, in this embodiment, the image processing device 100 is configured to calculate the amount of noise in a captured image data group acquired in chronological order, and to determine the amount of noise at the execution timing based on the calculated amount of noise. Furthermore, in this embodiment, the image processing device 100 is configured to determine new adjustment parameters at the execution timing based on the determined amount of noise, execute image restoration processing using the determined new adjustment parameters, and generate data of a restored image 219. According to the image processing device 100 configured as described above, it is possible to generate data of a restored image 219 in which degradation of image quality is sufficiently reduced, without changing the network configuration of a trained CNN used in the image restoration processing.

[0066] [Modification 1 of the first embodiment] A first modified example of the first embodiment will be described. In the first embodiment, the execution timing has been described as being set in advance, but the execution timing may be dynamically determined. For example, the image processing device 100 further includes an execution timing determination unit (not shown in FIG. 2) as a functional configuration, and the execution timing determination unit determines the execution timing. For example, the execution timing determination unit may sequentially analyze the time-series captured image data acquired by the image acquisition unit 211, and determine the execution timing for estimating the amount of noise based on the amount of change in brightness of the captured image.

[0067] Specifically, for example, the execution timing determination unit first sets one or more local regions (also called Regions of Interest (ROI)) for analyzing brightness in the captured image data acquired in S402. Then, the execution timing determination unit calculates the average luminance of the set local region, and holds the calculated average luminance value as a first average luminance. Then, the execution timing determination unit calculates the average luminance of the local region similarly set for the captured image data acquired next in S402, and holds the calculated average luminance value as a second average luminance. Then, the execution timing determination unit compares the first average luminance value with the second average luminance. For example, when the difference between the first average luminance value and the second average luminance is equal to or greater than a predetermined threshold, the execution timing determination unit determines that the time is the execution timing, and the image processing device 100 executes the noise amount estimation process in S404. On the other hand, if the difference between the first average brightness value and the second average brightness value is less than the threshold value, the execution timing determination unit determines that the current point in time is not the execution timing, and the image processing device 100 does not execute the noise amount estimation process in S404.

[0068] In addition, the local region is preferably a flat region in the captured image. Therefore, it is preferable that the execution timing determination unit sets an image region that can be a flat region in the captured image as the local region. The image region that can be a flat region may be set in advance by, for example, an instruction by a user.

[0069] According to the image processing device 100 configured as above, it is possible to more efficiently set (decide) the execution timing of the noise amount estimation process in S404 and the noise amount determination process in S405. As a result, it is possible to reduce the number of times the noise amount estimation process in S404 and the noise amount determination process in S405 are performed, and the total amount of calculation of the image processing device 100 can be reduced.

[0070] [Modification 2 of the first embodiment] Although the CNN performing the image restoration process according to the first embodiment has been described as including a map generation layer, an intensity adjustment layer, and an image generation layer, the CNN may be composed of a plurality of CNNs. For example, the CNN may be composed of a first CNN, a second CNN, and a third CNN, where the first CNN includes a map generation layer, the second CNN includes an intensity adjustment layer, and the third CNN includes an image generation layer. That is, in this case, the first CNN generates a feature map, the second CNN adjusts the degradation restoration intensity, and the third CNN generates a restored image.

[0071] For example, when the CNN is composed of a plurality of CNNs as described above, the degradation restoration learning in the information processing device 150 is configured to learn the first CNN and the third CNN. In such a case of learning, the feature map output by the first CNN may be configured to be input to the third CNN. The image processing device 100 acquires weight parameters of the trained first CNN and weight parameters of the trained third CNN from the information processing device 150. In addition, in the degradation restoration inference in the image processing device 100, the feature map output by the trained first CNN may be configured to be input to the second CNN. In addition, the feature map after each pixel value of the feature map is multiplied by a coefficient α in the second CNN may be configured to be input to the trained third CNN.

[0072] Note that the processing in the second CNN is simply a process of multiplying each pixel value of the feature map by the coefficient α, and therefore the second CNN does not necessarily need to be configured by a neural network.

[0073] [Embodiment 2] In the first embodiment, an example of automatically updating the adjustment parameters for adjusting the degradation restoration strength is described. In the second embodiment, a method in which a user manually sets and updates the adjustment parameters on a UI screen is described. In the following, a description of the contents common to the configuration of the first embodiment in the basic configuration of the image processing system 1 is omitted, and differences are mainly described.

[0074] Fig. 6 is a diagram showing an example of a UI (user interface) screen 600 displayed on the display device 40 according to the second embodiment. A user can set a correction intensity on the UI screen 600 shown as an example in Fig. 6. When the user sets the correction intensity, an adjustment parameter 218 corresponding to the set correction intensity is input to the image processing device 100, and the parameter acquisition unit 212 acquires this. The image processing device 100 executes an image restoration process on the captured image data using the adjustment parameter 218 acquired by the parameter acquisition unit 212, and generates data of a restored image 219 in which degradation in image quality has been restored.

[0075] The pull-down list box 601 is a UI component for selecting a mode for setting adjustment parameters in the image processing device 100. For example, when the user selects “manual” from the pull-down list displayed in the pull-down list box 601, the image processing device 100 operates in a mode in which the adjustment parameters are manually set by the user. When the user selects “auto” from the pull-down list displayed in the pull-down list box 601, the image processing device 100 operates in a mode in which the adjustment parameters are automatically updated, similar to the image processing device 100 according to the first embodiment. The items in the pull-down list displayed in the pull-down list box 601 are not limited to “manual” and “auto”. For example, the items in the pull-down list may include various items such as a mode in which a predetermined fixed adjustment parameter such as “default” is set. The UI screen 600 shown in FIG. 6 shows a state in which “manual” is selected from the pull-down list displayed in the pull-down list box 601.

[0076] The slider bar 602 is a UI component for the user to select a value of the correction strength for reducing image quality degradation. When "manual" is selected in the pull-down list box 601, the slider bar 602 is in an active state in which the user's operation can be accepted. In the active state, the correction strength can be set by the user's operation, and the setting can be adjusted. The area 603 is an area in which a preview image of the captured image data acquired by the image acquisition unit 211 is displayed. FIG. 6 shows, as an example, a state in which a preview image of a captured image in which a ship on the sea, distant land, and trees on the land are captured as subjects is displayed in the area 603. FIG. 6 also shows a state in which a preview image of a captured image in which noise is added to the entire captured image is displayed in the area 603. For example, the preview images of the captured image data acquired in time series by the image acquisition unit 211 are displayed in sequence in the area 603. The user can check the state of image quality degradation in the captured image by viewing the preview image displayed in the area 603.

[0077] The area 604 is an area where a preview image of the restored image 219 obtained as a result of the image restoration process on the captured image data acquired by the image acquisition unit 211 is displayed. The area 604 sequentially displays the time-series restored images 219 obtained as a result of the image restoration process on the captured image data acquired in time series by the image acquisition unit 211. By viewing the preview image displayed in the area 604, the user can check to what extent the degradation in image quality of the captured image confirmed by viewing the preview image displayed in the area 603 has been restored.

[0078] According to the image processing device 100 configured as above, the user can view the preview image displayed in the area 603 to check the state of image quality degradation in the captured image, and then set the correction strength of the image quality degradation according to the use case. For example, when the user selects "auto" or "default" as the operation mode of the image processing device 100, the user can reduce the effort of manually setting the correction strength. Also, when the user selects "manual" as the operation mode of the image processing device 100, the user can set the details of the image restoration process, and can set the correction strength of the image quality degradation while checking the degree of restoration of the image quality degradation. As a result, it is possible to generate a restored image in which image quality degradation is sufficiently reduced.

[0079] [Other embodiments] The present disclosure can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0080] It should be noted that, within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0081] [Configuration of this disclosure] The present disclosure includes the following configurations, methods, and programs.

[0082] [Configuration 1] An image processing device that generates restored image data in which image degradation included in an input image is reduced by inference using a neural network, an estimation means for estimating a degree of image quality degradation contained in the input image; a determination means for determining an adjustment parameter to be used in an image restoration process for reducing the image quality degradation based on the estimated degree of the image quality degradation; 13. An image processing device comprising:

[0083] [Configuration 2] the estimation means extracts a flat region having little variation in pixel values ​​from the input image based on a degree of variation in pixel values ​​of a partial region of the input image including a pixel of interest in the input image, and estimates the degree of image quality degradation included in the extracted flat region, thereby estimating the degree of image quality degradation included in the input image; 2. The image processing device according to claim 1,

[0084] [Configuration 3] an image acquisition means for acquiring time-series data of the input image obtained by imaging using an imaging device; an extraction means for extracting data of the input image for which the degree of image quality degradation is to be estimated from the time-series data of the input image; Further comprising: the estimation means estimates a degree of image quality degradation in the extracted input image data of the object; 3. The image processing device according to configuration 1 or 2,

[0085] [Configuration 4] The extraction means extracts, as the data of the target input image, an input image data group including data of one or more of the input images; the estimation means estimates a degree of image quality degradation for each of the input image data included in the extracted input image data group, thereby estimating a degree of image quality degradation in the input image data group; the determining means determines the adjustment parameter based on the estimated degree of image quality degradation in the input image data group; 4. The image processing device according to configuration 3,

[0086] [Configuration 5] The extraction means extracts a plurality of input image data groups, The estimation means estimates a degree of image quality degradation for each of the extracted plurality of input image data groups, the determining means determines the adjustment parameter based on the plurality of estimated degrees of image quality degradation; 5. The image processing device according to configuration 4,

[0087] [Configuration 6] the determining means calculates at least one of a moving average, a weighted moving average, a mean value, and a median value of the estimated degrees of image quality degradation, and determines the adjustment parameter based on the calculated statistical value; 6. The image processing device according to configuration 5,

[0088] [Configuration 7] an image acquisition means for acquiring time-series data of the input image obtained by imaging using an imaging device; A change amount acquisition means for acquiring a change amount of brightness of the input image in time series; Further comprising: the estimation means performs an estimation process of a degree of image quality deterioration included in the input image when the acquired change amount satisfies a predetermined condition; 7. The image processing device according to any one of configurations 1 to 6,

[0089] [Configuration 8] The neural network includes a map generation layer, which is composed of one or more hidden layers, for generating a feature map representing the strength of an image restoration process for the data of the input image, and an image generation layer, which is composed of one or more hidden layers, for generating data of the restored image corresponding to the data of the input image, the adjustment parameter corresponds to a coefficient by which each pixel value of the feature map generated in the map generation layer is multiplied; the image generation layer generates data of the restored image based on data of the input image and the feature map after each pixel value of the feature map has been multiplied by the coefficient; 8. The image processing device according to any one of configurations 1 to 7,

[0090] [Configuration 9] The neural network further includes an intensity adjustment layer, which is composed of one or more hidden layers, and which multiplies each pixel value in the feature map generated in the map generation layer by the coefficient based on the adjustment parameter; 9. The image processing device according to configuration 8,

[0091] [Configuration 10] The neural network is a trained model obtained as a result of training in a state in which weight parameters of the intensity adjustment layer are fixed; 10. The image processing device according to claim 9,

[0092] [Configuration 11] An image processing device that generates restored image data in which image degradation included in an input image is reduced by inference using a neural network, A setting means for setting a degree of reducing the image quality degradation included in the input image; a generating means for generating image data representing the input image in which the image quality degradation has been reduced in accordance with the set degree; 13. An image processing device comprising:

[0093] [Method 1] 1. An image processing method for generating restored image data in which image degradation contained in an input image is reduced by inference using a neural network, comprising: an estimation step of estimating a degree of the image quality degradation contained in the input image; determining an adjustment parameter to be used in an image restoration process for reducing the image quality degradation based on the estimated degree of the image quality degradation; 13. An image processing method comprising:

[0094] [Method 2] 1. An image processing method for generating restored image data in which image degradation contained in an input image is reduced by inference using a neural network, comprising: A setting step of setting a degree of reducing the image quality degradation included in the input image; a generating step of generating image data representing the input image in which the image quality degradation has been reduced in accordance with the set degree; 13. An image processing method comprising:

[0095] [program] A program for causing a computer to function as the image processing device according to any one of configurations 1 to 11. [Explanation of symbols]

[0096] 100 Image processing device 214 Noise amount estimation unit 215 Noise amount determination unit

Claims

1. An image processing device that generates restored image data in which image degradation included in an input image is reduced by inference using a neural network, an estimation means for estimating a degree of image quality degradation contained in the input image; a determination means for determining an adjustment parameter to be used in an image restoration process for reducing the image quality degradation based on the estimated degree of the image quality degradation; 13. An image processing device comprising:

2. the estimation means extracts a flat region having little variation in pixel values ​​from the input image based on a degree of variation in pixel values ​​of a partial region of the input image including a pixel of interest in the input image, and estimates the degree of image quality degradation included in the extracted flat region, thereby estimating the degree of image quality degradation included in the input image; 2. The image processing device according to claim 1,

3. an image acquisition means for acquiring time-series data of the input image obtained by imaging using an imaging device; an extraction means for extracting data of the input image for which the degree of image quality degradation is to be estimated from the time-series data of the input image; Further comprising: the estimation means estimates a degree of image quality degradation in the extracted input image data of the object; 2. The image processing device according to claim 1,

4. The extraction means extracts, as the data of the target input image, an input image data group including data of one or more of the input images; the estimation means estimates a degree of image quality degradation for each of the input image data included in the extracted input image data group, thereby estimating a degree of image quality degradation in the input image data group; the determining means determines the adjustment parameter based on the estimated degree of image quality degradation in the input image data group; 4. The image processing device according to claim 3,

5. The extraction means extracts a plurality of input image data groups, The estimation means estimates a degree of image quality degradation for each of the extracted plurality of input image data groups, the determining means determines the adjustment parameter based on the plurality of estimated degrees of image quality degradation; 5. The image processing device according to claim 4,

6. the determining means calculates at least one of a moving average, a weighted moving average, a mean value, and a median value of the estimated degrees of image quality degradation, and determines the adjustment parameter based on the calculated statistical value; 6. The image processing device according to claim 5,

7. an image acquisition means for acquiring time-series data of the input image obtained by imaging using an imaging device; A change amount acquisition means for acquiring a change amount of brightness of the input image in time series; Further comprising: the estimation means performs an estimation process of a degree of image quality deterioration included in the input image when the acquired change amount satisfies a predetermined condition; 2. The image processing device according to claim 1,

8. The neural network includes a map generation layer, which is composed of one or more hidden layers, for generating a feature map representing the strength of an image restoration process for the data of the input image, and an image generation layer, which is composed of one or more hidden layers, for generating data of the restored image corresponding to the data of the input image, the adjustment parameter corresponds to a coefficient by which each pixel value of the feature map generated in the map generation layer is multiplied; the image generation layer generates data of the restored image based on data of the input image and the feature map after each pixel value of the feature map has been multiplied by the coefficient; 2. The image processing device according to claim 1,

9. The neural network further includes an intensity adjustment layer, which is composed of one or more hidden layers, and which multiplies each pixel value in the feature map generated in the map generation layer by the coefficient based on the adjustment parameter; The image processing device according to claim 8 .

10. The neural network is a trained model obtained as a result of training in a state in which weight parameters of the intensity adjustment layer are fixed; The image processing device according to claim 9 .

11. An image processing device that generates restored image data in which image degradation included in an input image is reduced by inference using a neural network, A setting means for setting a degree of reducing the image quality degradation included in the input image; a generating means for generating image data representing the input image in which the image quality degradation has been reduced in accordance with the set degree; 13. An image processing device comprising:

12. 1. An image processing method for generating restored image data in which image degradation contained in an input image is reduced by inference using a neural network, comprising: an estimation step of estimating a degree of the image quality degradation contained in the input image; determining an adjustment parameter to be used in an image restoration process for reducing the image quality degradation based on the estimated degree of the image quality degradation; 13. An image processing method comprising:

13. 1. An image processing method for generating restored image data in which image degradation contained in an input image is reduced by inference using a neural network, comprising: A setting step of setting a degree of reducing the image quality degradation included in the input image; a generating step of generating image data representing the input image in which the image quality degradation has been reduced in accordance with the set degree; 13. An image processing method comprising:

14. A program for causing a computer to function as the image processing device according to any one of claims 1 to 11.

Images