Image processing method, image processing apparatus, and storage medium

Abstract

Provided is a method and the like capable of performing image processing using a machine learning model on various image data with high accuracy.SOLUTION: The image processing method includes a step S202 of acquiring an inputted image and range information about the inputted image, a step S203 of selecting at least one machine-learning model from a plurality of machine-learning models based on the range information, and a step S206 of generating an estimated image by inputting the inputted image to the machine-learning model selected in the step of selecting a machine-learning model. Alternatively, the method includes a step S202 of acquiring an inputted image and range information relating to the inputted image, and a step S206 of generating an estimated image by inputting the inputted image and the range information into a machine-learning model.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to a method for processing images using machine learning models. [Background technology]

[0002] As an example of image processing using a machine learning model, Patent Document 1 discloses a method for performing image restoration processing using a machine learning model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US2020 / 0389645 Summary of the Invention [Problem to be solved by the invention]

[0004] Image data is saved in various image formats, such as JPEG (Joint Photographic Experts Group) and HEIF (High Efficiency Image File Format). JPEG and HEIF have different ranges of values ​​that can be represented in an image. When image processing is performed using the method disclosed in Patent Document 1 on image data with different ranges, the range of the image data to be processed may differ from the range of the image data used to train the machine learning model. In such cases, there is a risk that the desired accuracy cannot be achieved in image processing. [Means for solving the problem]

[0005] An image processing method according to one aspect of the present invention is characterized by comprising the steps of: acquiring an input image and range information relating to the input image; selecting at least one machine learning model from a plurality of machine learning models based on the range information; and generating an estimated image by inputting the input image into the machine learning model selected in the machine learning model selection step.

[0006] Furthermore, an image processing method according to one aspect of the present invention is characterized by comprising a step of acquiring an input image and range information relating to the input image, and a step of generating an estimated image by inputting the input image and range information into a machine learning model. [Effects of the Invention]

[0007] According to the above-described image processing method, it is possible to provide an image processing method or the like that is capable of performing image processing using a machine learning model with high accuracy on various image data. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram of an image processing system according to a first embodiment. [Figure 2] 1 is an external view of an image processing system according to a first embodiment. [Figure 3] FIG. 1 is a diagram illustrating a flow of learning of a machine learning model in a first embodiment. [Figure 4] 1 is a flowchart relating to learning of a machine learning model in the first embodiment. [Figure 5] 1 is a flowchart relating to image processing using a machine learning model in the first embodiment. [Figure 6] FIG. 10 is a block diagram of an image processing system according to a second embodiment. [Figure 7] FIG. 10 is an external view of an image processing system according to a second embodiment. [Figure 8] FIG. 10 is a block diagram of an image processing system according to a third embodiment. [Figure 9]11 is a flowchart illustrating generation of an output image using a machine learning model in the third embodiment. [Figure 10] 1 is a diagram illustrating the gist of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and redundant explanations will be omitted.

[0010] In this embodiment, a method for processing images in various image formats using a machine learning model will be described.

[0011] In the image processing of this embodiment, as an example, a model constructed by a neural network is used as the machine learning model. The neural network uses a filter that is convolved with an image, a bias that is added, and an activation function that performs nonlinear transformation. The filter and bias are called weights, and are generated by learning using training images and correct images.

[0012] It should be noted that the machine learning model in this embodiment is not limited to a model structured by a neural network. The machine learning model in the embodiment may be any mathematical model obtained by deep learning. For example, a model structured by a Convolutional Neural Network (CNN) may be used as the machine learning model in this embodiment. Furthermore, a model structured by a Generative Adversarial Network (GAN), a Recurrent Neural Network (RNN), a Fully Connected Network (FCN), or a Transformer may also be used as the machine learning model. In the following, to avoid redundant expression, a model structured by a neural network may be simply referred to as a neural network.

[0013] In image processing using a machine learning model, the range of pixel values ​​in the input image is generally set to a predetermined range before being input into the machine learning model to generate a processed image, and if necessary, the range of the processed image is adjusted to the same range as the input image. The former process is called normalization, and the latter process is called denormalization, and the constants used in the process are called normalization constants.

[0014] Specifically, if the range of the input image is 0 to 255 and the default range is 0 to 1, 255 is used as the normalization constant. Normalization can be performed by dividing the pixel values ​​of the input image by the normalization constant 255. Also, if the range of the input image is 0 to 1023 and the default range is -1 to 1, 1023 is used as the normalization constant. In this case, normalization can be performed by dividing the pixel values ​​of the input image by the normalization constant 1023 and then subtracting 1. Note that inverse normalization is the inverse process of normalization, using the same normalization constant as normalization.

[0015] Next, image processing according to this embodiment will be described with reference to Fig. 10. In this embodiment, image processing is performed using range information. The range information in this embodiment includes information on the image format or information on the shooting mode.

[0016] The image format in this embodiment corresponds to the encoding format of the first image to be processed when it is stored in a storage medium or the like, such as JPEG, TIFF, HEIF, etc. Generally, the expressible range in JPEG and TIFF is 0 to 255, and 8 bits for gradation. Furthermore, the expressible range in HEIF is 0 to 1023, and 10 bits for gradation.

[0017] Furthermore, the shooting modes in this embodiment include a shooting mode (first shooting mode information) related to basic settings for shooting in a normal shooting mode (non-HDR shooting) and an HDR (high dynamic range) shooting mode. Note that the HDR shooting mode is a shooting mode that reduces blown-out highlights in high-brightness areas. Furthermore, in addition to the first shooting mode information (first range information), the shooting mode may also have information on a second shooting mode (second range information) that includes more detailed information. In this embodiment, the range used to express an image may also differ depending on the second shooting mode information. The second shooting mode will be described later.

[0018] Depending on the imaging device used to capture the first image, an image format corresponding to the shooting mode may be set. For example, images captured in normal shooting modes such as portrait, landscape, and sports may be stored in image formats such as JPEG and TIFF. Images captured in HDR shooting modes may also be stored in image formats such as HEIF. Furthermore, the HDR shooting mode may allow the user to select the degree of expansion of the dynamic range of the image. In this case, the degree of expansion of the dynamic range of the image corresponds to the second shooting mode information described above.

[0019] Generally, the range that can be expressed in HEIF is 0 to 1023, but the range used to express an image may be 0 to 600, 700, 800, etc. depending on the setting (second range information). In this case, the range used to express the first image can be determined by the second shooting mode.

[0020] 10 shows an example in which the image format is JPEG and the shooting mode is portrait (non-HDR shooting). In this case, the range of the captured image is 0 to 255, so 255 can be used as the normalization constant. In addition, the machine learning model is selected and used after learning using training images in JPEG image format.

[0021] However, this embodiment is not limited to this. For example, when the image format is HEIF and the shooting mode information is HDR shooting mode, the range of the captured image is 0 to 1023, so 1023 can be used as the normalization constant. Furthermore, it is preferable to select and use a machine learning model trained on training images in the HEIF image format. HEIF images acquired in HDR shooting mode have a different tone curve than JPEG images acquired in normal shooting mode, so the contrast is likely to be low. Due to this difference in image quality, there is a risk that the desired effect will not be obtained when a machine learning model trained on JPEG training images is applied to HEIF images.

[0022] On the other hand, machine learning models trained using HEIF training images tend to have a better correction effect (greater changes due to correction) on low-contrast subjects than machine learning models trained using JPEG training images.

[0023] In this manner, in this embodiment, by varying the processing depending on the range information of the image to be processed (first image), it is possible to perform image processing using a machine learning model with high accuracy.

[0024] 10 shows an example of upscaling (super-resolution) as image processing. Upscaling is image processing that estimates high-frequency components that cannot be expressed in a low-resolution image and generates a high-resolution image. Note that the image processing is not limited to this, and blur correction, contrast enhancement, brightness improvement, denoising, defocus blur conversion, lighting conversion, etc. may also be performed.

[0025] Furthermore, while the above discussion has focused on images in JPEG and HEIF formats, the same can be applied to videos. There are various shooting modes for videos, including those that save in 8-bit format during normal shooting, and those that save (store) in 10-bit format, such as HDR and LOG shooting modes, which allow for wide dynamic range shooting. Therefore, by changing the machine learning model and normalization constant according to the range information of the video, it is possible to accurately perform image processing using the machine learning model on each frame of the video. [Example]

[0026] First, an image processing system 100 according to a first embodiment of the present invention will be described with reference to Figures 1 and 2. In this embodiment, image processing is performed to generate a high-resolution image with high accuracy by upscaling a low-resolution JPEG image using a machine learning model.

[0027] Fig. 1 is a block diagram of an image processing system 100 in this embodiment. Fig. 2 is an external view of the image processing system 100. The image processing system 100 includes a learning device 101, an imaging device 102, an image estimation device 103, a display device 104, a recording medium 105, an input device 106, an output device 107, and a network 108.

[0028] The learning device 101 includes a storage unit (storage means) 101a, an image acquisition unit (acquisition means) 101b, a setting acquisition unit 101c, a determination unit 101d, a normalization unit 101e, and a learning unit (learning means) 101f.

[0029] The imaging device 102 has an optical system 102a and an imaging element 102b. The optical system 102a collects light incident on the imaging device 102 from the subject space. The imaging element 102b receives an optical image of the subject formed via the optical system 102a and acquires a captured image (low-resolution color image). The imaging element 102b is a CCD (Charge Coupled Device) sensor, a CMOS (Complementary Metal-Oxide Semiconductor) sensor, or the like.

[0030] Information about the shooting conditions of the captured image (shooting mode information, pixel pitch of the image sensor 102b, type of optical low-pass filter, ISO sensitivity, etc.) can be acquired together with the image. In addition, the development conditions of the captured image (image format, noise reduction strength, sharpness strength, image compression rate, etc.) can also be acquired together with the image. This information acquired together with the image can also be transmitted together with the image to an image acquisition unit 103b of the image estimation device 103, which will be described later. Also not shown are a storage unit that saves the acquired image, a display unit that displays it, a transmission unit that transmits it to the outside, and an output unit that saves it in an external storage medium. Also not shown is a control unit that controls each unit of the imaging device 102.

[0031] The image estimation device 103 includes a storage unit 103a, an image acquisition unit 103b, a setting acquisition unit 103c, a model selection unit 103d, a determination unit 103e, a normalization unit 103f, an image processing unit (estimation means) 103g, and a denormalization unit 103h. The image estimation device 103 acquires information (range information) on the image format or shooting mode from the low-resolution JPEG image (captured image) acquired by the image acquisition unit 103b using the setting acquisition unit 103c.

[0032] The model selection unit 103d selects a machine learning model based on the image format or the corresponding shooting mode information, and the determination unit 103e determines a normalization constant based on the image format or the corresponding shooting mode information.

[0033] The normalization unit 103f generates an image (input image) by normalizing pixel values ​​of the captured image using a normalization constant. The image processing unit 103g generates a high-resolution image (estimated image) by upscaling the normalized image using a machine learning model.

[0034] The inverse normalization unit 103h generates an image (output image) by inversely normalizing the pixel values ​​of the high-resolution image using a normalization constant. Note that the image processing unit 103g may perform upscaling using information (image information) related to the shooting conditions and development conditions acquired by the image acquisition unit 103b.

[0035] The low-resolution JPEG image (captured image) in this embodiment may be an image captured by the imaging device 102 or an image stored in the recording medium 105.

[0036] In the image processing of this embodiment, a neural network is used as a machine learning model. Furthermore, information about the weights of the neural network is read from the storage unit 103a. The weights are obtained through learning by the learning device 101, and the image estimation device 103 reads the weight information from the storage unit 101a via the network 108 in advance and stores it in the storage unit 103a. The stored weight information may be the weight's numerical value itself or in an encoded format. The numerical precision representing the weights is quantized, and the calculation precision of the image processing using the weights is also determined by this. Details regarding the learning of the machine learning model, the quantization of the weights, and the image processing using the weights will be described later.

[0037] The output image is output to at least one of a display device 104, a recording medium 105, and an output device 107. The display device 104 is, for example, a liquid crystal display or a projector. A user can check the image being processed via the display device 104 and perform image editing work via the input device 106. The recording medium 105 is, for example, a semiconductor memory, a hard disk, or a server on a network. The input device 106 is, for example, a keyboard or a mouse. The output device 107 is, for example, a printer.

[0038] Next, a weight learning method executed by the learning device 101 in this embodiment will be described with reference to Figures 3 and 4. Figure 3 is a diagram showing the flow of updating the weights of a neural network (learning a machine learning model). Figure 4 is a flowchart related to updating the weights of a neural network. The steps in Figure 4 are mainly executed by the image acquisition unit 101b, the setting acquisition unit 101c, the determination unit 101d, the normalization unit 101e, and the learning unit 101f.

[0039] First, in step S101, the image acquisition unit 101b acquires a low-resolution patch (first training image) 21 and a corresponding high-resolution patch (first gold standard image) 20 as training images. In this embodiment, a patch is an image having a predetermined number of pixels. For example, the low-resolution patch may have 128 x 128 pixels (number of vertical x horizontal pixels), and the corresponding high-resolution patch may have 256 x 256 pixels. In this case, since the vertical and horizontal sizes are doubled, the upscaling factor is doubled (the number of pixels is enlarged by four times).

[0040] The upscaling factor is not limited to 2x, and can be any factor as long as a low-resolution patch and a corresponding high-resolution patch can be obtained. Also, a low-resolution patch and a corresponding high-resolution patch can be obtained by photographing the same subject with optical systems having different focal lengths and cutting out corresponding portions of the two resulting images.

[0041] In addition, in this embodiment, the low-resolution patch and the corresponding high-resolution patch are generated by numerical calculation, but the present invention is not limited to this. The corresponding low-resolution patch may be generated by downsampling the high-resolution patch. Alternatively, an equivalent low-resolution patch acquired by the imaging device 102 and a corresponding high-resolution patch that is less affected (by aberration and diffraction) by the optical system 102a may be generated by numerical calculation. Furthermore, in this embodiment, the image formats of the low-resolution patch and the corresponding high-resolution patch are both JPEG, but the low-resolution patch and the corresponding high-resolution patch may be TIFF. In addition, in this embodiment, the shooting mode information of the low-resolution patch is portrait, but the present invention is not limited to this.

[0042] In step S102, the setting acquisition unit 101c acquires range information (image format and shooting mode information) from the first training image 21.

[0043] In step S103, the determination unit 101d determines a normalization constant from the shooting mode information. Since the shooting mode information (first imaging mode) in this embodiment is information indicating portrait and not HDR shooting, 255 is used as the normalization constant. Note that the normalization constant may also be determined from the image format, and since the image format is JPEG, it may be determined to use 255 as the normalization constant.

[0044] In step S104, the normalization unit 101e normalizes the range of values ​​of the first training image 21 and the corresponding first gold standard image 20 to a predetermined range using a normalization constant. In this embodiment, the range of the low-resolution patch and the high-resolution patch is 0 to 255, and the predetermined range is 0 to 1.

[0045] In step S105, the learning unit 101f uses a neural network to upscale the normalized low-resolution patch (second training image) 23 to generate a high-resolution patch (third training image) 24. Ideally, the third training image 24 and the high-resolution patch (second gold standard image) 22 obtained by normalizing the first gold standard image will match.

[0046] Note that upscaling may be performed taking into account image information by inputting image information into the neural network along with the second training image 23. For example, by using ISO sensitivity as information related to the shooting conditions, it is possible to perform upscaling when the ISO sensitivity is high without overemphasizing noise.

[0047] Furthermore, if the image format is used as information about the development conditions, upscaling can be performed taking the image format into consideration. In this case, the machine learning model can be trained using training images in various image formats.

[0048] With this configuration, the machine learning model obtained by learning can perform upscaling according to the image format by inputting captured images in various image formats and the image formats of the captured images. This is preferable because it allows high-precision image processing without having to select a machine learning model according to range information (image format) from multiple machine learning models.

[0049] One method for inputting image information into a neural network is to generate an image (map) with pixel values ​​related to the image information across the entire image, and then link this to the training image in the learning direction and to the captured image in the estimation direction. Specifically, an image with a pixel value of 0 across the entire image can be used if the image format is JPEG, or a pixel value of 1 across the entire image if the image format is HEIF. Alternatively, an image with corresponding shooting mode information as pixel values, rather than the image format, can be used as information related to development conditions.

[0050] In step S106, the learning unit 101f updates the weights of the machine learning model based on the error between the second ground truth image 22 and the third training image 24 corresponding to the second ground truth image 22. Here, the weights include the filter components and biases of each layer. Backpropagation is used to update the weights, but the present invention is not limited to this. In mini-batch learning, the errors between multiple normalized high-resolution patches 22 and the corresponding third training images 24 are calculated, and the weights are updated. For example, the L2 norm or the L1 norm may be used as the loss function. The weight update method (learning method) is not limited to mini-batch learning, and batch learning or online learning may also be used.

[0051] In step S107, the learning unit 101f determines whether weight learning is complete. Completion of learning can be determined by, for example, whether the number of iterations of weight update has reached a specified value, or whether the amount of change in weight during update is smaller than a specified value. If it is determined that learning is incomplete, the process returns to step S101, and multiple first training images 21 and corresponding first answer images 20 are obtained. On the other hand, if it is determined that learning is complete, the process proceeds to the next step.

[0052] In step S108, the learning unit 101f quantizes the weights for which learning has been completed. In this embodiment, since the image format of the training images is JPEG with 8-bit gradation (range 0 to 255), the weights are also quantized to 8 bits to match the numerical precision of the weights. However, the present invention is not limited to this.

[0053] Generally, if the numerical precision (gradation) representing the neural network weights is lower than the gradation representing the pixel values ​​of the input image, the precision of processing using the neural network will decrease. Therefore, for example, if the image format of the input image is HEIF, which has 10 bits of gradation, it is preferable to perform processing using a neural network based on weights quantized to 10 bits or more.

[0054] Furthermore, if the numerical precision used to express the weights in steps S105 and S106 is 8 bits and the learning calculation precision is also 8 bits, this step does not need to be executed. Also, if the neural network is trained using a mixture of various image formats as training images, it is preferable to use a neural network quantized with numerical precision equal to or greater than the number of bits of the image format with higher gradations. Information about the quantized weights is stored in the storage unit 101a.

[0055] In this embodiment, a neural network is trained to generate a high-resolution image by upscaling a low-resolution image. However, the present invention is not limited to this, and neural networks for various tasks may be trained and used. For example, when generating an image from which noise has been removed from a captured image, a noisy patch and a corresponding low-noise patch are acquired as training images, and a neural network is trained and used. Other tasks include blur correction, contrast enhancement, brightness enhancement, denoising, defocus blur conversion, and lighting conversion. By using training images appropriate for the task, a neural network capable of performing other tasks can be trained using the above method.

[0056] CN in Figure 3 represents a convolutional layer. In the convolutional layer CN, the sum of the input, the convolution of the filter, and the bias is calculated, and the result is nonlinearly transformed using an activation function. The initial values ​​of each filter component and the bias are arbitrary, and in this embodiment, they are determined by random numbers. The activation function can be, for example, a ReLU (Rectified Linear Unit) or a sigmoid function. The multidimensional array output from each layer except the final layer is a feature map.

[0057] A feature map is a four-dimensional array with dimensions of batch, length, width, and channel. In the skip connection 25, feature maps output from discontinuous layers are combined. In this case, the feature maps can be combined by adding each element together or by concatenation in the channel direction.

[0058] The elements (blocks or modules) within the frame in Figure 3 represent residual blocks. A network with multiple layers of residual blocks is called a residual network, and is widely used in DL image processing.

[0059] In this embodiment, the neural network configuration shown in Figure 3 is used, but the present invention is not limited to this. For example, an inception module may be used in which convolution layers with different convolution filter sizes are arranged side by side and the resulting feature maps are integrated to form a final feature map. Furthermore, a network may be configured by layering other elements, such as dense blocks with dense skip connections.

[0060] Alternatively, the processing load (~convolution operations) can be reduced by shrinking the feature map in the layer closest to the input and expanding it in the layer closest to the output, thereby reducing the size of the feature map in the intermediate layer. Here, pooling and stride can be used to shrink the feature map. Deconvolution (or transposed convolution), pixel shuffle, interpolation, etc. can be used to expand the feature map.

[0061] In addition, a low-resolution feature map is enlarged in a layer close to the output to create a high-resolution feature map. In this example, pixel shuffling (PS in the figure) is used as a method for upsampling the feature map, but the present invention is not limited to this.

[0062] Next, generation of an upscaled image from a captured image, which is executed by the image estimation device 103 in this embodiment, will be described with reference to Fig. 5. Fig. 5 is a flowchart related to generation of an upscaled image. The steps in Fig. 5 are mainly executed by the image acquisition unit 103b, setting acquisition unit 103c, model selection unit 103d, determination unit 103e, normalization unit 103f, image processing unit (estimation means) 103g, and inverse normalization unit 103h of the image estimation device 103.

[0063] First, in step S201, the image acquisition unit 103b acquires a captured image (first image). The captured image is a low-resolution JPEG image, similar to that used in learning. In this embodiment, the captured image is transmitted from the imaging device 102, but the present invention is not limited to this. Note that image information may also be acquired together with the captured image and used in the steps described below.

[0064] In step S202, the setting acquisition unit 103c acquires the image format or shooting mode information (range information) from the captured image. In this embodiment, the image format is JPEG, and the shooting mode information is portrait.

[0065] In step S203, the model selection unit 103d selects a neural network (machine learning model) to be used to generate an upscaled image from the range information of the captured image. In this embodiment, the image format of the captured image is JPEG, so a neural network trained using training images in JPEG format by the learning method shown in FIG. 3 is selected. Alternatively, a neural network may be selected from shooting mode information (first range information) corresponding to the image format. Note that the weight information of the selected neural network is transmitted from the learning device 101 and stored in the storage unit 103a. The numerical precision expressing the weights of the neural network is quantized to 8 bits.

[0066] In step S204, the determination unit 103e determines a normalization constant from the range information of the captured image. In this embodiment, the range information is information indicating that the shooting mode of the captured image is portrait mode. In this embodiment, if the range information is portrait mode, the normalization constant is determined to be 255. Alternatively, the normalization constant may be determined using the image format as range information. In this embodiment, the image format of the captured image is JPEG, so the normalization constant may also be determined to be 255.

[0067] In step S205, the normalization unit 103f normalizes the range of the captured image using a normalization constant so that it falls within a predetermined range. In this embodiment, the range of the captured image is 0 to 255, the predetermined range is 0 to 1, and the normalization constant is 255, so the pixel values ​​of the captured image are normalized by dividing them by the normalization constant. Note that the normalization constant may be determined using a predetermined value or a value specified by the user as range information instead of image format or shooting mode information.

[0068] In step S206, the image processing unit 103g generates an upscaled image from the normalized captured image using a machine learning model. Note that the numerical calculation precision when upscaling using the machine learning model is 8 bits.

[0069] In step S207, the inverse normalization unit 103h generates an image (output image) that has been inverse normalized using the normalization constant so that the range of the upscaled image matches the range of the original captured image. In this embodiment, the range of the upscaled image is 0 to 1, the range of the original captured image is 0 to 255, and the normalization constant is 255, so the pixel values ​​of the upscaled image are inverse normalized by multiplying them by the normalization constant.

[0070] In this embodiment, an example is shown in which an image in JPEG format is upscaled when it is captured in a shooting mode other than HDR shooting. This is not a limitation, and in step S203, the model selection unit 103d can select a neural network trained using HEIF training images according to the range information of the image to be processed. In this case, it is preferable that the numerical precision representing the weights of the neural network is quantized to 10 bits or more. In step S204, the determination unit 103e determines 1023 as the normalization constant based on the image format (HEIF) or shooting mode information (HDR shooting).

[0071] At this time, in step S205, the range of the captured image is normalized using a normalization constant so that it falls within a predetermined range, and in step S206, an upscaled image is generated from the normalized captured image using a machine learning model. Also, in step S207, an image (output image) is generated that is inversely normalized using the normalization constant so that the range of the upscaled image falls within the range of the original captured image.

[0072] Furthermore, when a machine learning model is trained using training images including images of various image formats, an upscaled image can be generated by inputting the normalized captured image and range information into the machine learning model.

[0073] Furthermore, in this embodiment, the learning device 101 and the image estimation device 103 are separate devices, but the present invention is not limited to this. The learning device 101 and the image estimation device 103 may be integrated. That is, the learning process (the process shown in FIG. 4) and the estimation process (the process shown in FIG. 5) may be performed within an integrated device.

[0074] With the above configuration, according to this embodiment, it is possible to generate a high-resolution image by upscaling a low-resolution image using a neural network with high accuracy. [Example]

[0075] Next, an image processing system 200 according to a second embodiment of the present invention will be described with reference to Figures 6 and 7. Figure 6 is a block diagram of the image processing system 200 according to this embodiment.

[0076] 7 is an external view of the image processing system 200. The image processing system 200 of this embodiment differs from the first embodiment in that the imaging device acquires a captured image (a blurred HEIF image) and performs image processing.

[0077] The image processing system 200 includes a learning device 201 and an imaging device 202 connected via a network 203. The learning device 201 corresponds to a first device, and the imaging device 202 corresponds to a second device. Furthermore, the learning device 201 and the imaging device 202 do not need to be constantly connected via the network 203.

[0078] The learning device 201 learns a machine learning model used in image processing to generate a blur-removed image from a captured image. The learning device 201 includes a memory unit (memory means) 211, an image acquisition unit (acquisition means) 212, a setting acquisition unit (setting means) 213, a determination unit (determination means) 214, a normalization unit (normalization means) 215, and a learning unit (learning means) 216.

[0079] The imaging device 202 captures an image of a subject space to obtain a captured image (a blurred HEIF image), and generates a blur-reduced image from the captured image. Details of the image processing performed by the imaging device 202 will be described later. The imaging device 202 has an optical system 221 and an image sensor 222. The image estimation unit 223 has an image acquisition unit 223a, a setting acquisition unit 223b, a model selection unit 223c, a determination unit 223d, a normalization unit 223e, an image processing unit (estimation means) 223f, and an inverse normalization unit 223g.

[0080] The learning of the neural network by the learning device 201 differs from that of the first embodiment in that sharp patches with less blur corresponding to blurred patches in the HEIF image format are acquired as training images.

[0081] The weight information of the neural network is generated by learning in the learning device 201 and stored in the storage unit 211. The imaging device 202 reads the weight information from the storage unit 211 via the network 203 and stores it in the storage unit 224.

[0082] The image estimation unit 223 generates a blur-reduced image (output image) from the captured image mainly in the image processing unit 223f, using information on the weights of the trained neural network stored in the storage unit 224, and the blurred image (captured image) and its image information acquired by the acquisition unit 223a. The generated blur-reduced image is stored in the recording medium 225a. When a user issues an instruction regarding the display of the blur-reduced image, the stored image is read out and displayed on the display unit 225b.

[0083] The captured image and its image information stored in the recording medium 225a may be read out, and the blur-reduced image may be generated by the image estimation unit 223. The above series of controls are performed by the system controller 227.

[0084] Next, the generation of a blur-reduced image executed by the image estimation unit 223 in this embodiment will be described.

[0085] First, in step S301, the image acquisition unit 223a acquires a captured image (first image). Note that in this embodiment, the captured image is a blurred HEIF image, similar to the learning. The captured image in this embodiment is acquired by the imaging device 202 and stored in the storage unit 224, but is not limited to this. Furthermore, image information may also be acquired together with the captured image and used in the steps described below.

[0086] In step S302, the setting acquisition unit 223b acquires range information from the captured image. In the following, the image format in this embodiment is HEIF. Also, an example will be described in which information is acquired that the first shooting mode information (first range information) in this embodiment is HDR shooting, and the second shooting mode information (second range information) is dynamic range +1. Note that, in the following, "dynamic range +1" is indicated as D+1.

[0087] In step S303, the model selection unit 223c selects a neural network to be used to generate a blur-reduced image based on the image format of the captured image. In this embodiment, the image format of the captured image is HEIF, so a trained neural network is selected using training images (sharp patches with less blur corresponding to blurred patches) in HEIF image format. Alternatively, a neural network may be selected based on shooting mode information corresponding to the image format. Note that the information on the weights of the neural network is transmitted from the learning device 201 and stored in the storage unit 224. The numerical precision expressing the weights of the neural network is quantized to 16 bits.

[0088] In step S304, the determination unit 223d determines a normalization constant from the shooting mode information of the captured image. In this embodiment, the shooting mode information of the captured image is D+1 for HDR shooting. D+1 is second shooting mode information (second range information) that indicates the degree of dynamic range expansion during HDR shooting, so the normalization constant can be determined according to the range used to express the captured image determined by D+1. For example, in D+1, one of the degrees of dynamic range expansion in HDR shooting, the range used to express the captured image is 0 to 600, so 600 is determined as the normalization constant. In the case of D+2, which allows shooting with a wider dynamic range than D+1, the range used to express the captured image is 0 to 700, so 700 can be determined as the normalization constant. Note that if shooting mode information is not available, a predetermined value or a value specified by the user may be used as range information and the normalization constant.

[0089] In step S305, the normalization unit 223e normalizes the range of the captured image using a normalization constant to set the range of the captured image to a predetermined range. In this embodiment, the range of the captured image is 0 to 600, and the normalization constant is 600. Furthermore, since the predetermined range is -1 to 1, the pixel value of the captured image can be normalized by dividing it by 300 and then subtracting 1.

[0090] In step S306, the image processing unit 223f generates a deblurred image by inputting the normalized captured image to a machine learning model. Note that the numerical calculation precision when deblurring using the machine learning model is 16 bits.

[0091] In step S307, the denormalization unit 223g generates an image (output image) that has been denormalized using the normalization constant so that the range is within the range of the original captured image. In this embodiment, the default range of the blur-reduced image is -1 to 1, the range of the original captured image is 0 to 600, and the normalization constant is 600, so the pixel values ​​of the blur-reduced image can be denormalized by adding 1 to them and then multiplying them by 200.

[0092] In this embodiment, an example of deblurring an image captured in a shooting mode with an HEIF image format and HDR shooting has been shown. When deblurring an image captured in a shooting mode with a JPEG image format and not HDR shooting, it is preferable to select a neural network trained using training images with a JPEG image format, as in the first embodiment.

[0093] With the above configuration, it is possible to generate an image with reduced blur from a blurred image with high accuracy using a neural network. [Example]

[0094] Next, an image processing system 300 according to a third embodiment of the present invention will be described with reference to Fig. 8 and Fig. 9. Fig. 8 is a block diagram of the image processing system 300 according to the third embodiment. Fig. 9 is a flowchart relating to generation of an estimated image using a machine learning model.

[0095] The image processing system 300 of this embodiment differs from the first and second embodiments in that it includes a processing device that transmits a captured image (low-resolution image) to be processed to the image estimation device and receives an estimated image or an output image from the image estimation device.

[0096] Image processing system 300 includes a learning device 301, an imaging device 302, an image estimation device 303, and a processing device (computer) 304. Learning device 301 and image estimation device 303 are, for example, servers. Computer 304 is, for example, a user terminal (a personal computer or a smartphone). Computer 304 is connected to image estimation device 303 via network 305. Image estimation device 303 is connected to learning device 301 via network 306. Computer 304 and image estimation device 303 are configured to be able to communicate with each other, and image estimation device 303 and learning device 301 are configured to be able to communicate with each other.

[0097] The learning of the neural network by the learning device 301 differs from that of the first embodiment in that when acquiring low-resolution patches and corresponding high-resolution patches as training images, training images in HEIF and JPEG image formats are mixed and acquired. Also, in this embodiment, learning is performed by inputting range information (image format or corresponding shooting mode information) along with the training images to the neural network as in step S105.

[0098] The configuration of the image capturing device 302 is the same as that of the image capturing device 102 of the first embodiment, and therefore a description thereof will be omitted.

[0099] The image estimation device 303 includes a storage unit 303a, an acquisition unit 303b, a setting acquisition unit 303c, a determination unit 303d, a normalization unit 303e, an image processing unit 303f, a denormalization unit 303g, and a communication unit 303h. The communication unit 303h has a function of receiving a request transmitted from the computer 304 and a function of transmitting an output image generated by the image estimation device 303 to the computer 304.

[0100] The computer 304 has a communication unit (transmission means) 304a, a display unit 304b, an input unit 304c, a processing unit 304d, and a recording unit 304e. The communication unit 304a has a function of transmitting a request to the image estimation device 303 to cause the image estimation device 303 to execute processing on a captured image (a low-resolution HEIF image), and a function of receiving an output image processed by the image estimation device 303.

[0101] The display unit 304b has a function of displaying various information, including, for example, a captured image to be transmitted to the image estimation device 303 and an output image received from the image estimation device 303.

[0102] The input unit 304c receives inputs such as instructions to start image processing from a user. The processing unit 304d has a function of performing image processing including noise removal and sharpening on the output image received from the image estimation device 303. The recording unit 304e saves the captured image acquired from the imaging device 302, the output image received from the image estimation device 303, etc.

[0103] Next, the image processing in this embodiment will be described.

[0104] 9 is started when a command to start image processing is given by the user via the computer 304. First, the operation of the computer 304 will be described.

[0105] First, in step S401, the computer 304 transmits a request for processing a captured image to the image estimation device 303. Note that the method for transmitting the captured image to be processed to the image estimation device 303 does not matter. For example, the captured image may be uploaded to the image estimation device 303 simultaneously with S401, or may be uploaded to the image estimation device 303 before S401. Furthermore, the captured image may be an image stored on a server different from the image estimation device 303. Furthermore, in S401, the computer 304 may transmit an ID for authenticating a user, image information, and the like together with the request for processing the captured image.

[0106] In step S 402 , the computer 304 receives the output image generated in the image estimation device 303 .

[0107] Next, the operation of the image estimation device 303 will be described.

[0108] First, in step S501, the image estimation device 303 receives a request for processing a captured image transmitted from the computer 304. The image estimation device 303 determines that processing for the captured image has been instructed, and executes the processing from step S502 onwards.

[0109] In step S502, the acquisition unit 303b acquires a captured image. In this embodiment, the captured image is transmitted from the computer 304. Note that image information may also be acquired together with the captured image and used in the steps described below. The acquisition unit 303b also acquires weights of a neural network (machine learning model) used to generate an upscaled image. Note that the weight information is transmitted from the learning device 301 and stored in the storage unit 303a. The numerical precision expressing the weights of the neural network is quantized to 16 bits.

[0110] In step S503, the determination unit 303d determines a normalization constant from the range information of the captured image.

[0111] The processes in steps S504 to S506 are the same as those in steps S205 to S207 in the first embodiment.

[0112] In step S507, the image estimation device 303 transmits the output image (high-resolution upscaled image) to the computer 304.

[0113] With the above configuration, according to this embodiment, it is possible to generate an upscaled image with high accuracy from a low-resolution image using a neural network.

[0114] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0115] According to each embodiment, it is possible to provide an image processing method, an image processing device, a program, and a storage medium that are capable of performing image processing with high accuracy on images of various shooting mode information and image formats using a machine learning model. The image processing device may be any device that has the image processing function of the present invention, and may be realized in the form of an imaging device or a personal computer.

[0116] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0117] The embodiments of the present invention include the following methods, configurations, and programs.

[0118] (Method 1) obtaining an input image and range information relating to the input image; selecting at least one machine learning model from a plurality of machine learning models based on the range information; and generating an estimated image by inputting the input image into the machine learning model selected in the machine learning model selection step.

[0119] (Method 2) obtaining an input image and range information relating to the input image; and generating an estimated image by inputting the input image and the range information into a machine learning model.

[0120] (Method 3) 3. The image processing method according to Method 1 or 2, comprising the step of generating the input image by normalizing a first image.

[0121] (Method 4) the first image is an image stored in a storage medium; 4. The image processing method according to Method 3, wherein the range information includes information about the image format in which the first image was stored on the storage medium.

[0122] (Method 5) the first image is an image acquired using an imaging device, The image processing method according to Method 3, wherein the range information includes information about the imaging mode of the imaging device when the first image was acquired.

[0123] (Method 6) The image processing method according to method 3, characterized in that in the step of generating the input image, a normalization constant is determined based on the range information, and the first image is normalized based on the normalization constant.

[0124] (Method 7) 7. The image processing method according to any one of Methods 1 to 6, comprising the step of generating an output image by denormalizing the estimated image.

[0125] (Method 8) the range information includes first range information and second range information; In the step of generating the input image, a normalization constant is determined based on the first range information, and the first image is normalized based on the normalization constant; An image processing method described in any one of methods 3 to 6, characterized in that in the step of selecting a machine learning model, the machine learning model is selected based on the second range information.

[0126] (Method 9) The image processing method according to method 8, wherein the first range information is image format information indicating that the image is HEIF.

[0127] (Method 10) The image processing method according to Method 8, wherein the first range information is shooting mode information indicating whether or not HDR shooting is performed.

[0128] (Method 11) 9. The image processing method according to method 8, wherein the second range information is information indicating a dynamic range.

[0129] (Method 12) 12. The image processing method according to method 11, wherein the greater the dynamic range, the greater the normalization constant.

[0130] (Method 13) An image processing method according to any one of methods 1 to 8, characterized in that the machine learning model is a first machine learning model when the first range information is image format information indicating JPEG, and is a second machine learning model when the first range information is image format information indicating HEIF.

[0131] (Method 14) the first machine learning model is a machine learning model trained using JPEG images as training data; The image processing method described in Method 13, wherein the second machine learning model is a machine learning model trained using HEIF images as training data.

[0132] (Method 15) 15. The image processing method according to any one of methods 1 to 14, wherein in the step of generating the estimated image, the machine learning model upscales the input image.

[0133] (Method 16) 15. The image processing method according to any one of methods 1 to 14, wherein in the step of generating the estimated image, the machine learning model reduces blur in the input image.

[0134] (Configuration 1) means for obtaining an input image and range information relating to the input image; means for selecting at least one machine learning model from a plurality of machine learning models based on the range information; and means for generating an estimated image by inputting the input image to the machine learning model selected in the step of selecting the machine learning model.

[0135] (Configuration 2) means for obtaining an input image and range information relating to the input image; and means for generating an estimated image by inputting the input image and the range information into a machine learning model.

[0136] (Configuration 3) An image processing system including the image processing device according to configuration 1 or 2 and a control device capable of communicating with the image processing device, the control device has a transmission means for transmitting a request to cause the image processing device to execute processing on the input image, The image processing system is characterized in that the image processing device has a receiving means for receiving the request, and executes processing on the input image in response to the request.

[0137] (Program 1) A program causing a computer to execute the image processing method according to any one of Methods 1 to 16.

[0138] (Method 17) obtaining a first training image and a first ground truth image; generating second training images by normalizing the first training images based on range information about the first training images; generating a second ground truth image by normalizing the first ground truth image based on range information about the first ground truth image; generating third training images based on the second training images using a machine learning model; A training method comprising a step of training the machine learning model based on the third training image, the second correct answer image, and an error.

[0139] (Program 2) A program that causes a computer to execute the training method according to Method 17. [Explanation of symbols]

[0140] 103b Image acquisition unit 103c Setting acquisition part 103d Model Selection Section 103e Decision section 103g Image processing unit

Claims

1. obtaining an input image and range information relating to the input image; selecting at least one machine learning model from a plurality of machine learning models based on the range information; and generating an estimated image by inputting the input image into the machine learning model selected in the machine learning model selection step.

2. obtaining an input image and range information relating to the input image; and generating an estimated image by inputting the input image and the range information into a machine learning model.

3. 3. The image processing method according to claim 1, further comprising the step of generating the input image by normalizing a first image.

4. the first image is an image stored in a storage medium; 4. The image processing method according to claim 3, wherein the range information includes information about the image format when the first image was stored on the storage medium.

5. the first image is an image acquired using an imaging device, 4. The image processing method according to claim 3, wherein the range information includes information about the imaging mode of the imaging device when the first image was acquired.

6. 4. The image processing method according to claim 3, wherein in the step of generating the input image, a normalization constant is determined based on the range information, and the first image is normalized based on the normalization constant.

7. The image processing method according to claim 3 , further comprising the step of generating an output image by denormalizing the estimated image.

8. the range information includes first range information and second range information; In the step of generating the input image, a normalization constant is determined based on the first range information, and the first image is normalized based on the normalization constant; 4. The image processing method according to claim 3, wherein in the step of selecting a machine learning model, the machine learning model is selected based on the second range information.

9. 9. The image processing method according to claim 8, wherein the first range information is information on an image format indicating HEIF.

10. 9. The image processing method according to claim 8, wherein the first range information is information on a shooting mode indicating whether or not HDR shooting is being performed.

11. 9. The image processing method according to claim 8, wherein the second range information is information indicating a dynamic range.

12. 12. The image processing method according to claim 11, wherein the normalization constant increases as the dynamic range increases.

13. The image processing method according to claim 1 or 2, characterized in that the machine learning model is a first machine learning model when the first range information is image format information indicating that the image format is JPEG, and is a second machine learning model when the first range information is image format information indicating that the image format is HEIF.

14. the first machine learning model is a machine learning model trained using JPEG images as training data; The image processing method according to claim 13 , wherein the second machine learning model is a machine learning model trained using HEIF images as training data.

15. The image processing method according to claim 1 or 2, wherein in the step of generating the estimated image, the machine learning model upscales the input image.

16. 3. The image processing method according to claim 1, wherein in the step of generating the estimated image, the machine learning model reduces blur in the input image.

17. means for obtaining an input image and range information relating to the input image; means for selecting at least one machine learning model from a plurality of machine learning models based on the range information; and means for generating an estimated image by inputting the input image to the machine learning model selected in the step of selecting the machine learning model.

18. means for obtaining an input image and range information relating to the input image; and means for generating an estimated image by inputting the input image and the range information into a machine learning model.

19. 19. An image processing system comprising the image processing device according to claim 17 or 18 and a control device capable of communicating with the image processing device, the control device has a transmission means for transmitting a request to cause the image processing device to execute processing on the input image, The image processing system is characterized in that the image processing device has a receiving means for receiving the request, and executes processing on the input image in response to the request.

20. 3. A program for causing a computer to execute the image processing method according to claim 1.

21. obtaining a first training image and a first ground truth image; generating second training images by normalizing the first training images based on range information about the first training images; generating a second ground truth image by normalizing the first ground truth image based on range information about the first ground truth image; generating third training images based on the second training images using a machine learning model; A training method comprising a step of training the machine learning model based on the third training image, the second correct image, and an error.

22. A program causing a computer to execute the training method according to claim 21.

Images