Image processing method, image processing apparatus, image processing system, and program

Abstract

To provide an image processing method that can achieve both prevention of harmful effects and maintaining of a correction effect.SOLUTION: An image processing method includes: a step of acquiring correction information related to brightness in performing development processing or correction processing on a picked-up image (S112); a step of correcting the picked-up image on the basis of the correction information to acquire a first image (S113); and a step of inputting the first image to a machine learning model for sharpening to create a second image (S114). The correction information is constant with respect to a position in the image, or changes depending on the position in the image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing method, an image processing device, an image processing system, and a program. [Background technology]

[0002] Patent Document 1 discloses a method for sharpening blurred captured images using a convolutional neural network (CNN). In the sharpening process in Patent Document 1, a training data set is generated by blurring an image having a signal value equal to or greater than the brightness saturation value of the captured image, and a machine learning model is trained with the training data set. By using the machine learning model trained in this way, the captured image is sharpened while suppressing adverse effects even in the vicinity of the brightness saturated region, even for a captured image having brightness saturation. Patent Document 1 also discloses a method for adjusting the strength of sharpening by weighting the captured image and an estimated image (sharpened image) based on the brightness saturated region. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-166628 A Summary of the Invention [Problem to be solved by the invention]

[0004] When performing correction processing using a method using a machine learning model, if the training data and the correct answer data are appropriately set, correction is possible even if the above-mentioned information-missing area is included. However, in the case of a method using a machine learning model, there is a possibility that adverse effects may occur even in areas that do not occur in deconvolution processing. In particular, adverse effects such as black sinking and white floating may occur in bright areas close to the luminance saturation value and dark areas close to the black level. Black sinking is a phenomenon in which the luminance value is lower than the original value, and white floating is a phenomenon in which the luminance value is higher than the original value. Such adverse effects tend to occur when the subject is significantly blurred due to aberrations in the optical system, when the blur changes significantly with respect to the screen position, when a machine learning model that supports various aberrations is used, or when the task is difficult. In addition, adverse effects tend to occur not only near edges but also in bright and dark areas as described above. For this reason, even if they are not visible in the estimated image, when brightness correction such as peripheral illumination correction and exposure correction or development processing is performed, the above-mentioned adverse effects may be emphasized to a degree that they can be seen.

[0005] SUMMARY OF THE PRESENT DISCLOSURE An object of the present invention is to provide an image processing method that can suppress adverse effects while maintaining the correction effect. [Means for solving the problem]

[0006] An image processing method as one aspect of the present invention includes a step of acquiring correction information regarding brightness when performing a development process or a correction process on a captured image, a step of acquiring a first image by correcting the captured image based on the correction information, and a step of generating a second image by inputting the first image into a machine learning model and sharpening it, wherein the correction information is constant with respect to a position in the image or varies depending on the position in the image.

[0007] Other objects and features of the present invention are illustrated in the following examples. Effect of the Invention

[0008] According to the present invention, it is possible to provide an image processing method that can suppress adverse effects while maintaining the correction effect. [Brief description of the drawings]

[0009] [Figure 1] 4 is a flowchart showing a process of generating a corrected image in the first embodiment. [Diagram 2] 1 is a block diagram of an image processing system according to a first embodiment. [Diagram 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4] 4 is a flowchart showing a learning process in the first embodiment. [Diagram 5] FIG. 2 is an explanatory diagram of a learning flow of the neural network in the first embodiment. [Figure 6] 6A to 6C are diagrams illustrating the effect of the correction process in the first embodiment. [Figure 7] 5A to 5C are diagrams illustrating the advantages and disadvantages of the correction process in the first embodiment. [Figure 8] FIG. 4 is an explanatory diagram of gamma correction in the first embodiment. [Figure 9] FIG. 11 is an explanatory diagram of the demosaicing process for G in the second embodiment. [Figure 10] FIG. 11 is an explanatory diagram of an interpolation filter in the demosaicing process in the second embodiment. [Figure 11] FIG. 11 is an explanatory diagram of the demosaicing process for R in the second embodiment. [Figure 12] FIG. 11 is a block diagram of an image processing system according to a second embodiment. [Figure 13] FIG. 11 is an external view of an image processing system according to a second embodiment. [Figure 14] 13 is a flowchart showing a process of generating a corrected image in the second embodiment. [Figure 15] FIG. 11 is a block diagram of an image processing system according to a third embodiment. [Figure 16] 13 is a flowchart showing image processing in the third embodiment. [Figure 17]FIG. 13 is an explanatory diagram of a response function used when creating a gain map in the third embodiment. [Figure 18] FIG. 13 is an explanatory diagram of a gain map in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0011] First, before describing each embodiment in detail, the gist of each embodiment will be described. In each embodiment, a correction process is performed on a captured image captured using an optical system based on correction information regarding brightness. Then, an estimated image is generated by sharpening the captured image after the correction process regarding brightness by using a machine learning model to remove blur caused by the optical system.

[0012] Here, the blur caused by the optical system includes blur caused by aberration, diffraction, defocus, the action of an optical low-pass filter, deterioration of the pixel aperture of the image sensor, etc. The machine learning model includes, for example, a neural network, genetic programming, a Bayesian network, etc. The neural network includes a CNN (Convolutional Neural Network), a GAN (Generative Adversarial Network), an RNN (Recurrent Neural Network), etc.

[0013] Although the sharpening process is given as an example, each image processing described in the following embodiment can also be applied to image processing such as increasing contrast, improving brightness, defocus blur conversion, lighting conversion, etc. The above processing can suppress adverse effects such as highlighting to a visible level when an estimated image is generated first and then brightness correction is performed, and black sinking and whitening that occur in bright areas close to the brightness saturation value or dark areas close to the black level.

[0014] In the following, the stage in which the weights of the machine learning model are learned will be referred to as the learning phase, and the stage in which blur sharpening is performed using the machine learning model with the learned weights will be referred to as the estimation phase. EXAMPLES

[0015] First, an image processing system according to a first embodiment of the present invention will be described. In this embodiment, blur is sharpened (blur correction processing) for a captured image using a machine learning model. The blur to be sharpened is aberration and diffraction occurring in the optical system, and blur caused by an optical low-pass filter. However, the effects of the invention can be obtained in the same way when blur caused by pixel aperture, defocus, or shaking is sharpened. The invention can also be implemented and the effects obtained in the same way for tasks other than blur sharpening (blur correction processing).

[0016] Fig. 2 is a block diagram of an image processing system 100 in this embodiment. Fig. 3 is an external view of the image processing system 100. The image processing system 100 includes a learning device (image processing device) 101, an imaging device 102, an image estimation device (image processing device) 103, a display device 104, a recording medium 105, an output device 106, and a network 107.

[0017] The learning device 101 is an image processing device that executes a learning process, and includes a storage unit 101a, an acquisition unit 101b, a generation unit 101c, and an update unit 101d. The acquisition unit 101b acquires training images and correct images. The generation unit 101c inputs the training images into a multi-layered neural network to generate an output image. The update unit 101d updates the network parameters of the neural network based on the error between the output image and the correct image calculated by the generation unit 101c. Details of the learning process will be described later using a flowchart. The network parameters obtained by learning are stored in the storage unit 101a.

[0018] The imaging device 102 has an optical system 102a and an imaging element 102b. The optical system 102a collects light incident on the imaging device 102 from the subject space. The imaging element 102b receives (photoelectrically converts) an optical image (subject image) formed via the optical system 102a to obtain a captured image. The imaging element 102b is, for example, a charge coupled device (CCD) sensor or a complementary metal-oxide semiconductor (CMOS) sensor. The captured image obtained by the imaging device 102 contains blur due to aberration and diffraction of the optical system 102a and noise due to the imaging element 102b.

[0019] The image estimation device 103 is a device that executes the estimation process, and includes a storage unit 103a, an acquisition unit 103b, and a correction unit 103c. The image estimation device 103 sharpens the blur of the captured image acquired to generate an estimated image (first image). A multi-layered neural network is used for the blur sharpening, and information on the network parameters is read from the storage unit 103a. The multi-layered neural network is trained by the learning device 101, and the image estimation device 103 reads the network parameters from the storage unit 101a via the network 107 in advance and stores them in the storage unit 103a. The stored network parameters may be in the form of their numerical values ​​or in an encoded format. Details regarding the training of the multi-layered neural network and the blur sharpening process using the multi-layered neural network will be described later.

[0020] The output image is output to at least one of a display device 104, a recording medium 105, and an output device 106. The display device 104 is, for example, a liquid crystal display or a projector. A user can perform editing work while checking an image being processed through the display device 104. The recording medium 105 is, for example, a semiconductor memory, a hard disk, a server on a network, etc. The output device 106 is, for example, a printer. The image estimation device 103 has a function of performing development processing and other image processing as necessary. (How to learn) Next, a method for updating weights (weight information) (a method for manufacturing a trained model) executed by the learning device 101 in this embodiment will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a flowchart related to weight learning. Each step in Fig. 4 is mainly executed by the acquisition unit 101b, the generation unit 101c, or the update unit 101d of the learning device 101. Fig. 5 is a diagram showing the flow of updating the weights of a neural network (learning a machine learning model).

[0021] First, in step S101 of FIG. 4, the acquisition unit 101b acquires an original image (subject image). In this embodiment, the original image is a high-resolution (high-quality) image with little blurring due to aberration and diffraction of the optical system 102a. A plurality of original images are acquired, and are images of various subjects, that is, images having edges of various strengths and directions, textures, gradations, flat areas, etc. The original image may be a real image or an image generated by CG (Computer Graphics).

[0022] Preferably, the original image has a signal value higher than the brightness saturation value of the image sensor 102b. This is because, even in actual subjects, there are subjects that do not fall within the brightness saturation value when imaged by the image capture device 102 under specific exposure conditions. A high-resolution captured image is generated by reducing the original image and clipping the signal at the brightness saturation value of the image sensor 102b. In particular, when a real image is used as the original image, blurring has already occurred due to aberration and diffraction, so that by reducing the image, the effect of blurring can be reduced and a high-resolution (high-quality) image can be obtained. Note that if the original image contains sufficient high-frequency components, it is not necessary to reduce the image. Also, the original image may contain noise components. In this case, the noise contained in the original image can be considered to be the subject, so the noise in the original image is not particularly problematic.

[0023] Next, in step S102, the acquisition unit 101b acquires a blur used to perform an imaging simulation, which will be described later. First, an imaging condition corresponding to the lens state (zoom, aperture, and focal distance state) of the optical system 102a is acquired. Then, a blur determined according to the imaging condition and the screen position is acquired. Here, the blur is the PSF (point spread function) or OTF (optical transfer function) of the optical system 102a. The blur can be acquired by optical simulation or measurement in the optical system 102a. Note that the blur due to the lens state, image height, aberration of azimuth, and diffraction, which differ for each original image, is acquired. This makes it possible to perform imaging simulations corresponding to a plurality of imaging conditions, image heights, and azimuth. In addition, a component such as an optical low-pass filter included in the imaging device 102 may be added to the blur to be applied, if necessary.

[0024] Next, in step S103, the generating unit 101c generates a correct answer patch (correct answer image) and a training patch (training data). A plurality of correct answer patches and a plurality of training patches are generated, and one or more patches are generated corresponding to one original image. In this embodiment, the correct answer patch and the training patch are images of the same subject. In this embodiment, a collection of a plurality of combinations of the correct answer patch and the training patch is used as learning data. Note that a patch refers to an image having a predetermined number of pixels (for example, 64×64 pixels). Also, the number of pixels of the correct answer patch and the training patch does not necessarily have to be the same.

[0025] In this embodiment, mini-batch learning is used to learn the weights of the multi-layered neural network. Therefore, in step S103, multiple pairs of correct answer patches and training patches are generated. However, this embodiment is not limited to this, and online learning or batch learning may be used. In this embodiment, the original image is an undeveloped RAW image, and the correct answer patches and training patches are also RAW images. However, this embodiment is not limited to this, and the original image may be a developed image, or a feature map obtained by converting an image as described later. In addition, the position of a partial region refers to the center of the partial region.

[0026] Next, in step S104, the generation unit 101c inputs the training patches (training data) 212 in Fig. 5 to a multi-layer neural network to generate estimated patches (estimated images) 213. For mini-batch learning, estimated patches 213 corresponding to the multiple training patches 212 are generated. Fig. 5 is an explanatory diagram of the flow of neural network learning, and shows the flow from step S104 to step S105 in Fig. 4.

[0027] The estimated patch 213 has a higher sharpness than the training patch 212, and ideally coincides with the correct patch (correct image) 211. In this embodiment, the neural network configuration shown in FIG. 5 is used, but this embodiment is not limited to this. CN in FIG. 5 represents a convolution layer, and DC represents a deconvolution layer. In both CN and DC, the convolution of the input and the filter and the sum of the bias are calculated, and the result is nonlinearly transformed by the activation function. The initial values ​​of each component of the filter and the bias are arbitrary, and are determined by random numbers in this embodiment. For example, ReLU (Rectified Linear Unit) or a sigmoid function can be used as the activation function. The output of each layer except the final layer is called a feature map.

[0028] The skip connections 222 and 223 combine feature maps output from discontinuous layers. The combination of feature maps may be done by element-by-element summation or by channel-wise concatenation. In this embodiment, element-by-element summation is adopted. The skip connection 221 takes the sum of the residual estimated from the training patch 212 and the ground truth patch 211, and the training patch 212 to generate an estimated patch 213. An estimated patch 213 is generated for each of the multiple training patches 212. Next, in step S105, the update unit 101d updates the weights (weight information) of the neural network based on the error between the estimated patch 213 and the correct patch (correct image) 211. Here, the weights include the filter components and biases of each layer. Backpropagation is used to update the weights, but this embodiment is not limited to this. For mini-batch learning, the errors between the multiple correct patches 211 and the corresponding estimated patches 213 are obtained, and the weights are updated. For the loss function, for example, the L2 norm or the L1 norm may be used.

[0029] Next, in step S106, the update unit 101d determines whether the weight update is complete. Completion of the weight update can be determined by whether the number of iterations of learning (weight update) reaches a specified value, or whether the amount of change in weight at the time of update is smaller than a specified value. If it is determined that the weight update is not complete, the process returns to step S103, and multiple new answer patches and training patches are obtained. On the other hand, if it is determined that the weight update is complete, the learning device 101 (update unit 101d) ends learning, and stores the weight information in the storage unit 101a.

[0030] Although the learning method in this embodiment has been described above, data other than images may be added to the training data for learning. Examples of the data to be added include a map (image plane coordinate map) representing image plane coordinate information and a map (noise map) representing noise information. The image plane coordinate map is a map representing image plane coordinates corresponding to the blur acting on the blurred image. Since the image sensor is disposed on the image plane, the image plane coordinates are synonymous with the position on the image sensor. The noise map is a map representing the noise intensity of the blurred image. The noise refers to noise (shot noise, etc.) generated in the image sensor 102b (or other image sensor that can be combined with the optical system 102a). The multiple blurred images stored in the storage unit 101a include noise of various intensities that may occur when capturing an image using the optical system 102a. When the blurred image is captured by actual shooting, the noise intensity can be obtained from the image sensor used for capturing the image and the ISO sensitivity at the time of capturing the image. When the blurred image is generated by imaging simulation, the noise intensity can be obtained because the intensity of the added noise is known. The intensity of the noise can be expressed as the standard deviation of the noise for a particular luminance.

[0031] Alternatively, a second map may be generated for each channel, with each coefficient when the variance of noise with respect to luminance is approximated by the following formula, n (n is a natural number), as the noise intensity. Inputting the noise map into the CNN has the advantage that the CNN can easily distinguish between noise and the subject. The number of pixels in the noise map is determined based on the number of pixels in the blurred image. If the target image does not contain noise, the CNN is trained to perform denoising at the same time as sharpening. If the target image contains noise that is of the same intensity as and correlated with the noise in the blurred image, the CNN is trained to perform blur sharpening that suppresses noise fluctuations. (Estimation method) Next, the generation (blur correction process) of a blur-corrected image (estimated image) executed by the image estimation device 103 in this embodiment will be described. Before going into a detailed description of the correction process, the effect of this embodiment will be described. FIGS. 6(a) to 6(d) are explanatory diagrams of the blur correction process using a machine learning model, and each is a schematic diagram showing the brightness value and spatial domain of an image. FIG. 6(a) shows an original image in an ideal state without aberration (blur), FIG. 6(b) shows a captured image degraded by the aberration of the optical system, FIG. 6(c) shows an image after blur correction by deconvolution processing, and FIG. 6(d) shows an estimated image after blur correction processing by this method. In FIGS. 6(a) to 6(d), the dotted lines indicate the brightness saturation value, and object A has a brightness value below the brightness saturation value, and object B has a brightness value above the brightness saturation value. When object B, which has a brightness above the brightness saturation value in FIG. 6(a), is captured by an imaging device, the image in the captured image in FIG. 6(b) is degraded by the aberration of the optical system, and then the signal is clipped at the brightness saturation value. FIG. 6(c) shows an example of applying deconvolution processing to such a captured image.

[0032] Examples of filters using deconvolution include Wiener filters and inverse filters. Equivalent processing can be performed by multiplying the image and filter by Fourier transforming them in frequency space. In this type of processing, degradation due to aberration can be reduced compared to the captured image in FIG. 6(b) for degradation such as that of object A in FIG. 6(c). However, for areas including luminance saturation areas such as object B, there is a risk of serious problems such as overshoot and undershoot. The overshoot occurring near the edge of the luminance saturation area in FIG. 6(c) is actually clipped at the luminance saturation value and does not exist in the image, but the undershoot occurring in the dark area does exist in the image. The black level of the captured image is offset from zero, and when an undershoot occurs, it is clipped so that it does not go below zero, but it falls below this offset value (OB value). In this way, the area below the black level becomes darker than the surroundings, and is emphasized. This is due to the loss of information caused by the luminance value of the object not falling within the dynamic range of the image sensor and being clipped. In the case of convolution processing such as that shown in Figure 6(c), since it is a linear operation, it is difficult in principle to avoid these issues.

[0033] On the other hand, as shown in Figure 6(d), when a machine learning model is used, it is possible to reduce these adverse effects. For example, even in the case of a neural network, convolution calculations are used, but a nonlinear function can be used for the activation function. When a machine learning model is used, nonlinear processing is used, allowing for more flexible processing. However, when training, it is necessary to use learning data that includes luminance saturation parts. If training is performed using a data set that does not include luminance saturation parts, there is a possibility that serious adverse effects such as overshoot and undershoot may occur, as in Figure 6(c). On the other hand, by using captured images that include luminance saturation parts in the training data and images with little aberration (blur) and no adverse effects around the luminance saturation parts as correct images, it is possible to perform blur correction processing that reduces aberrations while suppressing adverse effects for both subjects A and B, as shown in Figure 6(d).

[0034] So far, we have discussed the advantages of using a machine learning model to correct aberrations, but this is not necessarily a method that can correct aberrations without causing any problems. Even in the case of captured images that contain brightness saturation areas, it can be considered that aberration correction can be performed while suppressing adverse effects by determining whether or not there is a brightness saturation area inside the machine learning model and changing the processing depending on the result. However, if there is an error in this determination, it will also affect the result.

[0035] With reference to Figs. 7(a)-(d), we will describe the problems that occur when making corrections using a machine learning model and how to deal with them. Figs. 7(a)-(d) are explanatory diagrams of the effects and problems of blur correction processing. Figs. 7(a) and (b) respectively show an original image in an ideal state without aberration (blur) and a captured image degraded by the aberration of the optical system. While Figs. 6(a)-(d) only show two subjects, A and B, Figs. 7(a)-(d) show three subjects: subject A with a low brightness value, subject B with a value about half the brightness saturation value, and subject C with a value close to the brightness saturation value.

[0036] Subject C in Fig. 7(a) shows an example where misjudgment is likely to occur, and it is difficult to judge that a subject that is flat near the brightness saturation value has a value above the brightness saturation value and is clipped at the brightness saturation value. Note that Figs. 6(a)-(d) and Figs. 7(a)-(d) are schematic diagrams and are evaluated without noise, but actual captured images are also affected by noise. When values ​​vary due to noise near the brightness saturation value as in the case of subject C in Fig. 7(a), the noise can cause the value to exceed or fall below the brightness saturation value. If a region that was not saturated is erroneously judged to be a brightness saturation region and corrected, a larger correction will be applied.

[0037] FIG. 7(c) shows this state, and the result of the estimation process using the machine learning model for the captured image of FIG. 7(b) is shown in FIG. 7(c). As shown in FIG. 7(a), the maximum luminance value of the object C coincides with the luminance saturation value in a state without aberration, and FIG. 7(c) shows the result of the machine learning model erroneously determining that the luminance is saturated, resulting in overcorrection for the object C. Basically, such a problem is caused by an attempt to reduce the signal more than necessary, so that the luminance value tends to be lower than it should be due to blackening. In this embodiment, the purpose is to suppress such a problem. In addition, such a problem does not necessarily occur only on the high luminance side, but also appears at other luminance values. The object A in FIG. 7(c) shows one of the patterns.

[0038] As described above, when learning is performed using a RAW image in the learning process, blur correction processing is also performed on the RAW image during estimation. A RAW image is undeveloped image data output from an image sensor, and the light amount and luminance value of each pixel have an approximately linear relationship. Normally, before a user views an image, a RAW image is developed and various correction processing is performed, and at that time, brightness correction such as exposure correction and peripheral illumination correction, and gradation conversion such as gamma correction and bit conversion are applied. Brightness correction processing such as exposure correction and peripheral illumination correction makes adverse effects in low brightness or high brightness areas more noticeable. Gamma correction is, for example, a process of raising the input luminance value to a power, and 1 / 2.2 is used as the exponent. FIG. 8 is an explanatory diagram of gamma correction, and shows an example of a function used in gamma correction. In FIG. 8, the horizontal axis indicates the luminance value before gamma correction, and the vertical axis indicates the luminance value after gamma correction. As shown in FIG. 8, gamma correction is a process in which a curve (gamma curve) showing the relationship between luminance values ​​before and after correction is at a position equal to or greater than a straight line with a slope of 1 (the dashed line in FIG. 7(a) to (d)). Since such a conversion process is performed during development, the signal value on the low luminance side is stretched more than that on the high luminance side. In other words, the adverse effects of the low luminance side in the RAW image state become more noticeable due to gamma correction. In FIG. 7(c), the amount of undershoot is shown to be about the same for subject C on the high luminance side and subject A on the low luminance side for ease of understanding, but even if the amount is small in the RAW image, the adverse effects on the low luminance side may become noticeable after development.

[0039] In this embodiment, in order to suppress these adverse effects, a correction process based on the correction information on brightness is first performed on the captured image, and an image (first image) after the correction process on brightness is generated. Then, an estimated image (second image) is generated by sharpening the image after the correction process on brightness using a machine learning model. This embodiment is characterized in that the correction process on brightness is performed first, and the reason why this order is better is thought to be the influence of error calculation in the learning process. Basically, in the learning process, the error between the correct image and the training image is calculated based on the difference between the images. In other words, whether the pixel value is large or small, if the difference is the same, it is considered to be the same error.

[0040] Here, when considering the ratio of this error to the pixel value, even if the error is the same, if the original pixel value is small, the amount of error for the pixel value will be large. In particular, when a process that changes the pixel value, such as exposure correction, is performed on an image after estimation processing using a machine learning model, the error will be enlarged by the correction process. On the other hand, as in this embodiment, by first performing a process such as exposure correction and then performing an estimation process, the aforementioned error can be prevented from being enlarged. In this way, by performing a sharpening process using a machine learning model after a correction process related to brightness, it is possible to maintain the correction effect and suppress adverse effects at the same time.

[0041] Next, the generation of a corrected image (correction process) executed by the image estimation device 103 in this embodiment will be described with reference to Fig. 1. Fig. 1 is a flowchart showing the generation process of a corrected image. Each step in Fig. 1 is mainly executed by the acquisition unit 103b and the correction unit 103c of the image estimation device 103.

[0042] First, in step S111, the acquisition unit 103b acquires information on a captured image and weights. The captured image is an undeveloped RAW image, similar to learning, and in this embodiment, is transmitted from the imaging device 102. The weight information is the weights of the machine learning model transmitted from the learning device 101 and stored in the storage unit 103a.

[0043] Next, in step S112, the acquisition unit (first means) 103b acquires correction information related to brightness. The correction information related to brightness is correction information applied to the blur-corrected image (correction information when performing development processing or correction processing on the captured image), and is a parameter (correction amount) such as exposure correction or peripheral illumination correction. A constant parameter that does not depend on the position on the screen (position in the image) such as exposure correction may be acquired, or a parameter that changes depending on the position on the screen (position in the image) such as peripheral illumination correction may be acquired. Alternatively, both may be acquired.

[0044] Furthermore, white balance correction parameters may be acquired as correction information related to brightness. The white balance correction parameters are constant parameters that do not depend on the position on the screen (the position in the image). White balance correction can be regarded as exposure correction that differs for each color of the image, and white balance correction parameters (correction amount) may be acquired for each color. In this case, the parameter (correction amount) with the largest absolute value among the white balance correction parameters (correction amount) for each color may be acquired as correction information related to brightness.

[0045] Furthermore, the correction information regarding brightness may include not only the absolute value of the correction amount but also positive / negative information as the correction direction of whether to brighten or darken. For example, a parameter expressed as +3 steps to brighten and -3 steps to darken may be used, or a parameter regarding a ratio expressed as +30% to brighten and -30% to darken. Even in the case of a parameter based on 100%, it can be considered as a positive parameter to brighten and a negative parameter to darken. Note that the peripheral illumination correction may include not only positive correction to brighten but also negative correction to add peripheral illumination (darken).

[0046] For parameters that depend on the position of the screen, such as peripheral illumination correction, parameters corresponding to the position of the entire screen may be obtained as a two-dimensional map. Alternatively, a coefficient obtained by converting a correction value according to the distance from the center of the screen (correction center) into a function may be obtained, and a parameter (correction amount) corresponding to the distance from the center of the screen (correction center) may be obtained. Furthermore, the correction information regarding brightness is performed on the blur-corrected image (second image) obtained by the processing of this embodiment, and may be a default value, or the correction information regarding brightness specified by the user may be obtained. Alternatively, information obtained by estimating correction information regarding preferred brightness from the captured image or the blur-corrected image may be obtained.

[0047] Next, in step S113, the correction unit (second means) 103c corrects the captured image based on the correction information regarding brightness acquired in step S112, and acquires a first image (an image after correction processing regarding brightness).

[0048] First, a case where the correction information regarding brightness is exposure correction will be described. If the information is exposure correction +N steps, each pixel value of RGB is increased by 2 to the power of N for the captured image acquired in step S111. If the black level is offset here, the offset is subtracted and then the pixel value is multiplied by 2 to the power of N. If the multiplication exceeds the upper limit value of the image, it is clipped at the upper limit value. If it is normalized from 0 to 1, it is set to 1, and if it is expressed in 16 bits, it is set to 65535. Also, the brightness correction may be applied to the luminance signal instead of directly applying it to the RGB pixel values. In that case, RGB may be converted to YUV and the luminance signal may be multiplied by the power of N. Also, exposure correction may be applied to the negative side instead of the positive side, in which case division may be used instead of multiplication.

[0049] Next, a case where a white balance coefficient is used as the correction information related to brightness will be described. In this embodiment, the white balance coefficient refers to a coefficient used for white balance correction. White balance correction is a process of adjusting the brightness level by performing gain processing for each color component on the RGB components output by the image sensor so that the achromatic parts of the subject become achromatic in the output image. If white balance correction is not performed, the color of the subject is not reproduced correctly due to the color characteristics of the image sensor, and an image with a color different from the actual subject is generated. In this embodiment, the white balance coefficient uses the setting at the time of capturing the captured image. Normally, there is a white balance coefficient for each RGB color component, so white balance correction can be performed by multiplying the coefficients by the color components of the captured image corresponding to the color components. Here, if the black level is offset, the offset is subtracted before white balance correction is performed. Also, if the white balance coefficients are divided into G1 and G2, the average of the white balance coefficients is used as the white balance coefficient for G.

[0050] In this embodiment, when the correction information related to brightness is a white balance coefficient, the white balance correction process is performed on the captured image using this coefficient. The white balance coefficient is not limited to the setting at the time of capture, and a different combination of coefficients may be used. For example, a typical digital camera has a function called auto white balance that automatically determines the type of light source and corrects it, and the coefficient data is also attached to the captured image and can be used for white balance correction.

[0051] Next, a case where peripheral illumination correction parameters (correction amount) are used as the correction information related to brightness will be described. Exposure correction and white balance correction are constant parameters independent of the position on the screen, but peripheral illumination correction is a parameter that changes depending on the position on the screen. When information on peripheral illumination correction of a two-dimensional map is acquired as the correction information related to brightness, in step S113, correction is performed by calculating the product for each pixel of the captured image and the two-dimensional map of peripheral illumination correction. Other processing such as clipping of upper and lower limits is similar to that of exposure correction and white balance correction.

[0052] Next, in step S114, the acquisition unit (third means) 103b executes a sharpening process on the captured image whose brightness has been corrected in step S113, based on the weight of the machine learning model acquired in step S111. That is, the acquisition unit 103b inputs the first image to the machine learning model, and generates an output image which is a sharpened second image.

[0053] In addition, in the machine learning model used in this embodiment, the estimated image is also a RAW image, so a development process is performed as necessary. In this embodiment, the development process is, for example, gamma correction, white balance correction, or demosaicing, but is not limited to these. Since gamma correction and white balance correction have already been described, their explanations are omitted. Note that, if the correction information regarding brightness acquired in step S112 is a white balance coefficient, white balance correction has already been applied in step S113, so it does not need to be performed in the development process in step S114.

[0054] In this embodiment, the method may further include a step of obtaining a third image by inversely correcting the second image based on the correction information after step S114. Note that the inverse correction in this embodiment refers to performing a process on an image that has been subjected to a predetermined correction process, to return the image to its original state after the predetermined correction process.

[0055] Next, the demosaicing process will be described with reference to Fig. 9(a) to (c). Fig. 9(a) to (c) are explanatory diagrams relating to color components of a captured image, and show the demosaicing process for G. In this embodiment, the captured image is an image of a Bayer array as shown in Fig. 9(a). Here, RGB represent red, green, and blue, respectively. There are two types of Bayer array for G only, G1 and G2, so the demosaicing process is different for G and RB.

[0056] Fig. 9(b) shows the state where only G1 and G2 are extracted from Fig. 9(a). When performing demosaicing processing on G in Fig. 9(a), linear interpolation processing is performed after extracting G as shown in Fig. 9(b). Fig. 10(a) shows a filter used for linear interpolation, and by convolving the filter in Fig. 10(a) with G in Fig. 9(b), it is possible to generate an interpolated G as shown in Fig. 9(c).

[0057] Next, the demosaicing process for RB will be described. Similar processes can be applied to both R and B with only one type, so in this embodiment, the process contents will be described only for R. Figs. 11(a) to (c) are explanatory diagrams of the demosaicing process method for R. Fig. 11(a) shows a state in which only R is extracted from the Bayer array in Fig. 9(a), and in this embodiment, color difference information is calculated by taking the difference of G with respect to R. Fig. 11(b) shows the color difference of R before demosaicing process, and the difference of G after demosaicing process in Fig. 9(c) is taken only for pixels where the value of R in Fig. 11(a) exists.

[0058] Next, an interpolation process is performed on the color difference in Fig. 11(b). Fig. 10(b) shows an example of a filter for performing the interpolation process, and color difference information corresponding to all pixels can be calculated by convolving such a filter. Fig. 11(c) shows R after demosaicing process, and can be calculated by adding G to the color difference after convolving the filter in Fig. 10(b).

[0059] Although the demosaicing process for R has been described above, the demosaicing process can be similarly performed for B. In this embodiment, the interpolation process is performed using color difference for R and B, but the interpolation process may be performed directly for R and B. Furthermore, the demosaicing process may be performed by incorporating not only simple interpolation processing but also adaptive processing using the gradient of RGB values, etc.

[0060] According to this embodiment, the brightness correction is performed first, and then the estimation process is performed using the machine learning model. This makes it possible to suppress adverse effects such as emphasis, darkening, or whitening that would occur if the brightness correction were performed after the estimated image was generated. EXAMPLES

[0061] Next, an image processing system according to a second embodiment of the present invention will be described with reference to Fig. 12. In this embodiment, a corrected image is generated by an image estimation unit 323 in an imaging device. Also, the flow of generating the corrected image is different from that of the first embodiment. In this embodiment, a noise map is further used to generate the corrected image in comparison with the first embodiment.

[0062] Fig. 12 is a block diagram of an image processing system 300 in this embodiment. Fig. 13 is an external view of the image processing system 300. The image processing system 300 includes a learning device (image processing device) 301 and an imaging device 302 connected via a network 303. The learning device 301 has a storage unit (storage means) 311, an acquisition unit (acquisition means) 312, a generation unit (generation means) 313, and an update unit (learning means) 314, and updates the weights (weight information) of the neural network to perform blur correction.

[0063] The imaging device 302 captures an image of a subject space to obtain a captured image, generates an estimated image from the captured image using the read weight information, and generates a corrected image by weighted addition of the captured image and the estimated image. The imaging device 302 has an optical system 321 and an image sensor 322. The image estimation unit 323 has an acquisition unit 323a and a correction unit 323b, and performs estimation processing on the captured image using weight information stored in a storage unit 324.

[0064] The weight information is learned in advance by the learning device 301 and stored in the storage unit 311. The imaging device 302 reads out the weight information from the storage unit 311 via the network 303 and stores it in the storage unit 324. The corrected image generated using the captured image and the estimated image is stored in the recording medium 325. When an instruction to display the corrected image is issued by the user, the stored corrected image is read out and displayed on the display unit 326. Note that a captured image already stored in the recording medium 325 may be read out and corrected by the image estimation unit 323. The above series of controls are performed by the system controller 327.

[0065] The learning method of the machine learning model executed by the learning device 301 of this embodiment is the same as that of the first embodiment, and therefore the description thereof will be omitted.

[0066] Next, generation of a corrected image executed by the image estimation unit 323 in this embodiment will be described with reference to Fig. 14. Fig. 14 is a flowchart related to generation of a corrected image. Each step in Fig. 14 is mainly executed by the acquisition unit 323a and correction unit 323b of the image estimation unit 323. Steps S211 to S213 are similar to steps S111 to S113 executed by the acquisition unit 103b and correction unit 103c in the first embodiment.

[0067] In step S214, the acquisition unit 323a acquires noise information (first noise information) based on the captured image or information related to the captured image. In this embodiment, the noise information is the variance of noise in the optical black area. The optical black area is an area outside the effective pixel area of ​​the captured image where light is blocked. In this embodiment, the noise variance in this area is used as the noise information, but the standard deviation may be used or coefficient data by function approximation may be used. The noise information is used as a noise map for CNN in the subsequent processing. Therefore, it is necessary to correspond to the noise information at the time of learning. In addition, the noise information may be acquired by acquiring the optical black area of ​​the captured image and calculating the variance, or information attached to the captured image may be acquired. If the acquired noise information is a scalar value, it is necessary to make it a two-dimensional noise map so that it can be input to the CNN. As described in the learning process, the number of pixels of the noise map is determined based on the number of pixels of the captured image. When the variance value is used as in this embodiment, it becomes a scalar value, so it is handled as a two-dimensional map by arranging the variance value so as to correspond to the size of the captured image. The noise map may be obtained for each of RGB, or a representative value may be used by converting the noise map into the maximum value or brightness of RGB and then calculating the noise map. As described above, the noise map may be generated from the noise information obtained in the process of step S214, or a noise map calculated in advance may be obtained as the noise information.

[0068] Next, in step S215, the correction unit 323b corrects the noise map (first noise information) acquired or generated in step S214 based on the correction information on brightness acquired in step S212 to acquire second noise information. Processing that changes the brightness of a captured image, such as exposure correction or peripheral illumination correction, also affects noise contained in the captured image. Therefore, by reflecting the influence of the change in the noise information, the correction accuracy by the machine learning model can be further improved.

[0069] In this embodiment, as an example, the correction information regarding brightness acquired in step S212 is an exposure correction parameter, and a case will be described in which the information is exposure correction +N steps. In this case, if a variance value is used for the noise map, it is desirable to multiply by 2 to the power of 2N. For example, if the exposure correction is +3 steps, the noise map should be a value that is about 64 times larger. Note that this is on the assumption that the noise is normally distributed, so this value is not necessarily required, and it is sufficient to take into account that the variance of the noise increases as the image becomes brighter. If the noise map is based on the standard deviation rather than the variance, the value should be 8 times the square root.

[0070] Next, in step S216, the acquisition unit (third means) 323a executes a sharpening process on the captured image (first image) whose brightness has been corrected in step S213, based on the weight of the machine learning model acquired in step S211. In this embodiment, not only the captured image but also the noise map (second noise information) corrected in step S215 is used as input data for the machine learning model. Note that the machine learning model used here is a model trained using the noise map as input data. As other contents are similar to the correction process in step S114 in the first embodiment, a description thereof will be omitted.

[0071] This embodiment differs from the first embodiment in that it uses a noise map, and is characterized in that the noise map is corrected based on correction information related to brightness. By using the noise map corrected in this way, it becomes possible to further suppress the amount of adverse effects that occur. EXAMPLES

[0072] Next, an image processing system according to a third embodiment of the present invention will be described. The image processing system of this embodiment differs from the first and second embodiments in that it includes a processing device (computer) that transmits a captured image to be subjected to image processing to an image estimation device and receives a processed output image (corrected image) from the image estimation device.

[0073] FIG. 15 is a block diagram of an image processing system 600 in this embodiment. The image processing system 600 includes a learning device 601, an imaging device 602, an image estimation device (second device) 603, and a computer (processing device, first device) 604. The learning device 601 and the image estimation device 603 are, for example, servers. The computer 604 is, for example, a user terminal (personal computer or smartphone). The computer 604 is connected to the image estimation device 603 via a network 605. The image estimation device 603 is connected to the learning device 601 via a network 606. That is, the computer 604 and the image estimation device 603 are configured to be able to communicate with each other, and the image estimation device 603 and the learning device 601 are configured to be able to communicate with each other.

[0074] The configuration of the learning device 601 is similar to that of the learning device 101 in the first embodiment, and therefore a description thereof will be omitted. The configuration of the imaging device 602 is similar to that of the imaging device 102 in the first embodiment, and therefore a description thereof will be omitted.

[0075] The image estimation device 603 has a storage unit 603a, an acquisition unit 603b, a correction unit 603c, and a communication unit (reception means) 603e. The storage unit 603a, the acquisition unit 603b, and the correction unit 603c are similar to the storage unit 103a, the acquisition unit 103b, and the correction unit 103c of the image estimation device 103 of the embodiment 1, respectively. The communication unit 603d has a function of receiving a request transmitted from the computer 604, and a function of transmitting an output image generated by the image estimation device 603 to the computer 604.

[0076] The computer 604 has a communication unit (transmission means) 604a, a display unit 604b, an image processing unit 604c, and a recording unit 604d. The communication unit 604a has a function of transmitting a request to the image estimation device 603 to cause the image estimation device 603 to execute processing on the captured image, and a function of receiving an output image processed by the image estimation device 603. The display unit 604b has a function of displaying various information. The information displayed by the display unit 604b includes, for example, the captured image to be transmitted to the image estimation device 603 and the output image received from the image estimation device 603. The image processing unit 604c has a function of further performing image processing on the output image received from the image estimation device 603.

[0077] The recording unit 604d records the captured image acquired from the imaging device 602, the output image received from the image estimation device 603, and the like.

[0078] Next, image processing in this embodiment will be described with reference to Fig. 16. Image processing in this embodiment includes processing equivalent to the correction processing (Fig. 1) described in embodiment 1. Fig. 16 is a flowchart showing image processing in this embodiment. The image processing shown in Fig. 16 is started when an instruction to start image processing is given by a user via computer 604. First, the operation of computer 604 will be described.

[0079] First, in step S701, the computer 604 transmits a request for processing a captured image to the image estimation device 603. Note that the method of transmitting the captured image to be processed to the image estimation device 603 is not important. For example, the captured image may be uploaded to the image estimation device 603 simultaneously with step S701, or may be uploaded to the image estimation device 603 before step S701. Furthermore, the captured image may be an image stored on a server different from the image estimation device 603. Note that in step S701, the computer 604 may transmit ID information for authenticating a user together with the request for processing the captured image.

[0080] In step S702, the computer 604 receives the output image generated in the image estimation device 603. The output image is a corrected image obtained by weighted addition of the captured image and the estimated image, similar to the first embodiment.

[0081] Next, a description will be given of the operation of the image estimation device 603. First, in step S801, the image estimation device 603 receives a request for processing a captured image transmitted from the computer 604. The image estimation device 603 determines that a correction process for the captured image has been instructed, and executes the processes from step S802 onward.

[0082] Next, in step S802, the image estimation device 603 acquires the captured image and weight information. The weight information is information (trained model) learned by the same method (FIG. 4) as in the first embodiment. The image estimation device 603 may acquire the weight information from the learning device 601, or may acquire weight information previously acquired from the learning device 601 and stored in the storage unit 603a. The following steps S803 to S807 are similar to steps S212 to S216 in the second embodiment, respectively.

[0083] Next, in step S808, the image estimation device 603 generates a gain map to be used when interpolating the captured image and the estimated image generated in step S807. As described above, when the captured image is corrected using a machine learning model, in low-luminance and high-luminance regions, problems such as black sinking and whitening become more noticeable due to the brightness correction process.

[0084] In this embodiment, these adverse effects are suppressed by inputting an image corrected based on the correction information on brightness to a machine learning model. In this embodiment, the adverse effects are further suppressed by taking a weighted average of the estimated image and the captured image. That is, in this embodiment, a gain map is generated based on the luminance value in the first image or the second image, and the first image and the second image are weighted averaged based on the gain map and the correction information. Preferably, the maximum color component is obtained for each pixel in the captured image and the second image, and the gain map is generated based on the maximum color component.

[0085] In this embodiment, a first gain map for suppressing the amount of adverse effects in low-luminance areas (dark areas) and a second gain map for suppressing the amount of adverse effects in high-luminance areas (bright areas) are calculated. Then, based on these two gain maps, a gain map used for interpolation between the captured image and the estimated image is calculated. In addition, the captured image and the estimated image are used to calculate the first gain map and the second gain map, but in this embodiment, two-dimensional data obtained by scaling these images is used. Therefore, the scaling process will be described first.

[0086] First, the maximum value of RGB is obtained for each pixel of the captured image of the three colors of RGB. In this embodiment, the maximum value is used, but this is not limited to this, and other indices such as the median, average, or minimum value may be used. In this manner, in this embodiment, one value is set for one pixel, so that the weights do not differ between RGB. This makes it possible to avoid the problem of the image breaking down when, for example, an estimated image is used for RB and a captured image is used for G.

[0087] Next, the two-dimensional data composed of the maximum values ​​of RGB is scaled so that the black level becomes 0 and the brightness saturation value becomes 1. Here, scaling is performed based on the black level and brightness saturation value, but scaling may be performed based on other values. For example, conversion may be performed by dividing by 2 bytes so that 2 bytes become 1. A gain map is then calculated based on the scaled two-dimensional data.

[0088] FIG. 17(a) shows a response function for obtaining the first gain map. In FIG. 17(a), the horizontal axis is the two-dimensional data after scaling that is the input, and the vertical axis is the weight used when interpolating the captured image and the estimated image. In this embodiment, the weight on the vertical axis in FIG. 17(a) is the weight applied to the estimated image. Therefore, in FIG. 17(a), when the pixel value of the two-dimensional data after scaling increases, the output value also increases, which means that the estimated image is adopted. Conversely, when the pixel value of the two-dimensional data after scaling decreases, the output value also decreases, which means that the captured image is adopted. In FIG. 17(a), the thresholds are set to 0.2 and 0.3, and 0.2 or less is the captured image, 0.3 or more is the estimated image, and the weight changes linearly between them. In this embodiment, two thresholds are set so that each section changes linearly, but a nonlinear function such as a sigmoid function may be used as the response function without setting a threshold. In addition, the two thresholds are not limited to these, and other values ​​may be adopted. When priority is given to the correction effect, the threshold value should be set smaller. Conversely, when priority is given to suppression of adverse effects, the threshold value should be set larger.

[0089] Fig. 18(a) is a simulation diagram of the first gain map calculated using the captured image of Fig. 7(b). In Fig. 18(a), the gain map for the areas of subjects B and C in Fig. 7(b) is also 1, and it is understood that the estimated image is adopted in these areas. On the other hand, for the low-luminance subject A, the gain map is 0, and it is understood that the estimated image is adopted.

[0090] Next, the second gain map will be described. As described above, the second gain map is a weight map used to suppress the amount of damage in high-luminance areas (bright areas). Here, the second gain map represents the gain applied to the estimated image. That is, the larger the second gain map is, the greater the weight of the estimated image is than that of the captured image when performing the interpolation process. The second gain map is generated based on the pixel values ​​of the estimated image. FIG. 17(b) shows a response function for determining the second gain map. In FIG. 17(b), the horizontal axis is the two-dimensional data after scaling generated based on the pixel values ​​of the estimated image as the input, and the vertical axis is the weight used when interpolating the captured image and the estimated image. In this embodiment, the weight on the vertical axis in FIG. 17(b) is the weight applied to the estimated image. Therefore, in FIG. 17(b), when the pixel values ​​of the two-dimensional data after scaling are large, the output value is small, which means that the captured image is adopted. Conversely, when the pixel values ​​of the two-dimensional data after scaling are small, the output value is large, so it becomes an estimated image. In FIG. 17(b), the thresholds are set to 0.8 and 0.9, with values ​​below 0.8 being the estimated image and values ​​above 0.9 being the captured image, and the weighting changes linearly between these values.

[0091] In this embodiment, two thresholds are set so that each section changes linearly, but a nonlinear function such as a sigmoid function may be used as the response function without setting a threshold. The two thresholds are not limited to these, and other values ​​may be used. If the correction effect is to be prioritized, the threshold can be set to a larger value, and conversely, if the suppression of adverse effects is to be prioritized, the threshold can be set to a smaller value.

[0092] Fig. 18(b) is a simulation diagram of a second gain map calculated using the estimated image of Fig. 7(c). In Fig. 18(b), the gain map for the areas of subjects A and B in Fig. 7(c) is also 1, and it is understood that the estimated image is adopted in this area. On the other hand, for the high-luminance subject C, the gain map is 0, and it is understood that the estimated image is adopted.

[0093] Next, the correction unit 603c generates one gain map using the first gain map and the second gain map. FIG. 18(c) is a schematic diagram of a gain map generated using the first gain map and the second gain map. In this embodiment, the two gain maps are compared and the smaller value is adopted to generate a combined gain map. In this embodiment, the minimum value is adopted, but the combined gain map may be calculated by multiplying or averaging the two gain maps.

[0094] Next, in step S809 of Fig. 16, the correction unit 603c performs an interpolation process using the captured image, the estimated image, and the composite gain map calculated in step S808. Fig. 7(d) is a simulation diagram of a corrected image after weighted addition using the composite gain map. It can be seen that, compared to the estimated image in Fig. 7(c), the undershoot that occurred in the low-luminance subject A and the high-luminance subject A has been reduced. It can also be seen that the sharpening effect is comparable to that of the estimated image in Fig. 7(c), and the correction effect can be maintained.

[0095] 16, the image estimation device 603 transmits the output image to the computer 604. When the correction process is performed within the image estimation device 603 as in this embodiment, the processing load due to the correction process can be borne within the image estimation device 603, and therefore the processing capacity required for the computer 604 can be reduced.

[0096] As in this embodiment, the image estimation device 603 may be configured to be controlled by a computer 604 communicably connected to the image estimation device 603 .

[0097] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0098] According to each embodiment, it is possible to provide an image processing method, an image processing device, an image processing system, and a program that are capable of suppressing adverse effects while maintaining the correction effect.

[0099] The disclosure of each embodiment includes the following configurations and methods. (Method 1) A step of acquiring correction information related to brightness when performing development processing or correction processing on a captured image; correcting the captured image based on the correction information to obtain a first image; and inputting the first image into a machine learning model and sharpening the first image to generate a second image. An image processing method, wherein the correction information is constant with respect to a position in an image or varies depending on a position in the image. (Method 2) The image processing method according to the first method, wherein the correction information is correction information relating to either exposure correction or peripheral light amount correction. (Method 3) 3. The image processing method according to method 1 or 2, wherein the correction information does not depend on the luminance value of the image. (Method 4) 4. The image processing method according to any one of Methods 1 to 3, further comprising the step of obtaining a third image by inversely correcting the second image based on the correction information. (Method 5) acquiring first noise information based on a captured image or information related to the captured image; and correcting the first noise information based on the correction information to obtain second noise information, An image processing method according to any one of methods 1 to 4, characterized in that in the step of generating the second image, the second image is generated by inputting the first image and the second noise information into the machine learning model. (Method 6) An image processing method described in any one of methods 1 to 5, characterized in that a gain map is generated based on brightness values ​​in a first image or a second image, and a weighted average of the first image and the second image is calculated based on the gain map and the correction information. (Method 7) The image processing method according to method 6, characterized in that the gain map weights the captured image more heavily as the pixel values ​​of the first image become smaller. (Method 8) The image processing method according to Method 6 or 7, wherein the gain map weights the captured image more heavily as the pixel value of the second image increases. (Method 9) The image processing method described in any one of Methods 6 to 8, characterized in that the maximum color component is obtained for each pixel of the captured image and the second image, and the gain map is generated based on the maximum color component. (Configuration 1) A first means for acquiring correction information related to brightness when performing development processing or correction processing on a captured image; a second means for correcting the captured image based on the correction information to obtain a first image; and third means for inputting the first image into a machine learning model to generate a sharpened second image. An image processing method, wherein the correction information is constant with respect to a position in an image or varies depending on a position in the image. (Configuration 2) An image processing system including the image processing device according to configuration 1 and a control device capable of communicating with the image processing device, The control device has a means for transmitting a request for executing a process on the image, 2. An image processing system according to claim 1, wherein the image processing device comprises means for executing processing on the image in response to the request. (Configuration 3) A program for causing a computer to execute the image processing method according to any one of Methods 1 to 9.

[0100] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0101] 103 Image Estimation Device (Image Processing Device) 103b Acquisition unit (first means, third means) 103c Correction unit (second means)

Claims

1. A step of acquiring correction information related to brightness when performing development processing or correction processing on a captured image; correcting the captured image based on the correction information to obtain a first image; and inputting the first image into a machine learning model and sharpening it to generate a second image. An image processing method, wherein the correction information is constant with respect to a position in an image or varies depending on the position in the image.

2. 2. The image processing method according to claim 1, wherein the correction information is correction information relating to either exposure correction or peripheral light amount correction.

3. 2. The image processing method according to claim 1, wherein the correction information does not depend on a luminance value of the image.

4. 2. The image processing method according to claim 1, further comprising the step of obtaining a third image by inversely correcting the second image based on the correction information.

5. acquiring first noise information based on a captured image or information related to the captured image; and correcting the first noise information based on the correction information to obtain second noise information, The image processing method according to claim 1 , wherein in the step of generating the second image, the second image is generated by inputting the first image and the second noise information into the machine learning model.

6. The image processing method according to claim 1, further comprising the steps of: generating a gain map based on brightness values ​​in a first image or a second image; and calculating a weighted average of the first image and the second image based on the gain map and the correction information.

7. 7. The image processing method according to claim 6, wherein the gain map weights the captured image so that the pixel values ​​of the first image are smaller.

8. 7. The image processing method according to claim 6, wherein the gain map weights the captured image so that the pixel value of the second image is larger.

9. The image processing method according to claim 6 , further comprising obtaining a maximum color component for each pixel of the captured image and the second image, and generating the gain map based on the maximum color component.

10. a first means for acquiring correction information relating to brightness when performing a development process or a correction process on a captured image; a second means for acquiring a first image by correcting the captured image based on the correction information; and a third means for generating a second image by inputting the first image into a machine learning model and sharpening the first image. An image processing method, wherein the correction information is constant with respect to a position in an image or varies depending on the position in the image.

11. 11. An image processing system including the image processing device according to claim 10 and a control device capable of communicating with the image processing device, The control device has a means for transmitting a request for executing a process on the image, 2. An image processing system according to claim 1, wherein the image processing device comprises means for executing processing on the image in response to the request.

12. A program for causing a computer to execute the image processing method according to any one of claims 1 to 9.

Images