Image processing apparatus, control method, and storage medium

Abstract

To generate an image in which the effect of irreversible image processing is canceled.SOLUTION: A first acquisition unit configured to acquire a target image generated by applying the first image processing, a second acquisition unit configured to acquire a processing parameter related to the first image processing applied at the time of generation of the target image, and a control unit configured to, by inputting the target image and the processing parameter to the first trained model, A first inference unit configured to infer a first image in which an effect of first image processing is canceled from a target image, wherein the first image processing is irreversible image processing, and the first learned model is a machine learning model constructed by machine learning of training data in which a first learning image generated by applying the first image processing to an image signal obtained by imaging a predetermined scene and a processing parameter related to the first image processing are set as examples, and a second learning image generated without applying the first image processing to the image signal is set as a correct answer.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device, a control method, and a program, and more particularly to image processing for an image to which irreversible image processing has been applied. [Background technology]

[0002] When an image captured by an imaging device is recorded, various image processing processes such as gamma adjustment, white balance adjustment, noise reduction, sharpening, contrast adjustment, and saturation adjustment are applied (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-191768 Summary of the Invention [Problem to be solved by the invention]

[0004] However, some image processing, such as sharpness processing, is irreversible, making it difficult to restore a recorded image to a state before the image processing was applied.

[0005] The present invention has been made in consideration of the above-mentioned problems, and has as its object to provide an image processing device, a control method, and a program that generate an image in which the effects of irreversible image processing are negated. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the image processing device of the present invention is an image processing device that generates an image in which the effect of first image processing is negated using a first trained model, and includes a first acquisition means that acquires a target image generated by applying the first image processing, a second acquisition means that acquires processing parameters related to the first image processing applied when generating the target image, and a first inference means that infers a first image in which the effect of the first image processing is negated from the target image by inputting the target image and the processing parameters into the first trained model, wherein the first image processing is irreversible image processing, and the first trained model is a machine learning model constructed by machine learning of training data that uses as examples a first training image generated by applying the first image processing to an image signal capturing a specified scene and the processing parameters related to the first image processing, and a second training image generated without applying the first image processing to the image signal as a correct answer. [Effects of the Invention]

[0007] With this configuration, the present invention makes it possible to generate an image in which the effects of irreversible image processing are negated. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating a hardware configuration of an image capturing apparatus 100 according to an embodiment and a modification of the present invention. [Figure 2] FIG. 1 is a diagram for explaining machine learning related to construction of a first trained model according to an embodiment and a modification of the present invention. [Figure 3] 1 is a flowchart illustrating a cancellation process executed in the image capture device 100 according to an embodiment and a modification of the present invention; [Figure 4] 10 is a flowchart illustrating a super-resolution process executed in the image capturing device 100 according to the second embodiment of the present invention. [Figure 5] 10 is a flowchart illustrating a super-resolution process executed in the image pickup device 100 according to the fourth modification of the present invention. [Figure 6] FIG. 10 is a diagram for explaining machine learning related to the construction of a third trained model according to Variation 5 of the present invention. [Figure 7] FIG. 14 is a diagram illustrating additional learning of a third trained model according to a sixth modification of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] [Embodiment 1] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] In the embodiment described below, an example of applying the present invention to an image capture device, which is an example of an image processing device, has a function of canceling predetermined image processing on an image obtained by capturing using a machine learning model. However, the present invention can be applied to any device that can generate an image in which the effect of predetermined image processing is canceled using a machine learning model.

[0011] <Hardware configuration of imaging device> The hardware configuration of an imaging device 100 according to this embodiment will be illustrated below using the block diagram of Fig. 1. The imaging device 100 may include, for example, a device used for imaging, such as a digital camera or digital video camera, or an electronic device with an imaging function, such as a camera-equipped mobile phone or a camera-equipped computer.

[0012] The optical system 101 is an imaging optical system equipped with a group of lenses, a shutter, an aperture, etc. The group of lenses may include a correction lens that corrects camera shake and the like, a focus lens, etc. The optical system 101 forms an image of subject light on the imaging surface of the imaging element 102 based on a control signal received from a CPU 103 (described later). The imaging element 102 is an imaging sensor such as a CCD (charge-coupled device) image sensor or a CMOS (complementary metal-oxide semiconductor) image sensor. The imaging element 102 converts the optical image formed on the imaging surface by the optical system 101 into an image signal by photoelectric conversion.

[0013] The CPU 103 is a control device that controls the operation of each block of the imaging device 100. The CPU 103 reads out the operation program of each block stored in the secondary storage device 107, and controls the operation of each block by expanding and executing the program in the primary storage device 104. When controlling the operation of each block, the CPU 103 appropriately sends a control signal corresponding to the corresponding operation to each block.

[0014] The secondary storage device 107 is a non-volatile storage device such as an EEPROM (Electrically Erasable Programmable Read-Only Memory). The secondary storage device 107 stores various setting information required for the operation of each block, in addition to operation programs for each block of the imaging device 100 and firmware for the imaging device 100. The primary storage device 104 is a volatile storage device such as a RAM (Random Access Memory). The primary storage device 104 is used not only as a storage area for the operation programs of each block, but also as a work memory for storing temporary data output by the operation of each block. The information stored in the primary storage device 104 can be used by the image processing device 105 or recorded on a recording medium 106.

[0015] The recording medium 106 is a recording device, such as a semiconductor memory card, that is configured to be detachable from the imaging device 100. When an image is captured by the imaging device 100, the recording medium 106 is used to record image data that is stored in the primary storage device 104 by the image capture. The data recorded on the recording medium 106 can be used in an external device such as a PC (personal computer) by inserting the recording medium 106 into the external device.

[0016] The display unit 108 is a display device such as an LCD, etc. The display unit 108 is used to display a viewfinder image during shooting, a captured image, a GUI (Graphical User Interface) image for interactive operation, etc.

[0017] The operation unit 109 is a user interface such as a button, a lever, a touch panel, etc. When an operation input is made to various operation members, the operation unit 109 transmits a control signal corresponding to the operation input to the CPU 103. In addition, the operation unit 109 may also include an input device that uses voice, line of sight, etc.

[0018] The image processing device 105 is a device that performs various types of image processing on an image. In this embodiment, the image processing device 105 is configured to be able to apply multiple types of image processing to an image. The image processing to be performed by the image processing device 105 may be determined, for example, as a pattern for each shooting mode. In this embodiment, the image processing pattern to be applied by the image processing device 105 to the captured image is controlled based on information about the shooting mode set by the user. The image processing performed on the captured image includes so-called development processing as well as processing such as color tone adjustment. In some shooting modes provided in the imaging device 100 according to this embodiment, the image processing performed on the captured image includes irreversible image processing as the first image processing according to the present invention. In the following description, sharpness conversion processing that changes the sharpness of an image is exemplified as an example of irreversible image processing. That is, in some shooting modes, sharpness conversion processing is applied when converting the image signal (captured image) output by the image sensor 102 into image data to be recorded on the recording medium 106.

[0019] 1, the imaging device 100 is described as having the image processing device 105 as one piece of hardware, and the image processing device 105 executes image processing, but the implementation of the present invention is not limited to this. Part of the image processing may be realized by the CPU 103 executing a corresponding processing program.

[0020] <<Outline of the process to cancel the effect of sharpness conversion>> As described above, in the imaging device 100 of this embodiment, image data recorded on the recording medium 106 by shooting in some shooting modes is generated by applying sharpness conversion processing to image signals obtained by shooting. Sharpness conversion processing is irreversible image processing, and basically, even if a conversion processing with inverse characteristics is applied to image data after the sharpness conversion processing has been applied, it is not possible to restore the image data before or before the sharpness conversion processing has been applied. On the other hand, there may be a need to restore image data to the state before the irreversible image processing has been applied, for example, when the shooting mode has been set incorrectly or when the strength of the sharpness conversion processing needs to be changed. The image processing device 105 of this embodiment is configured to use a machine learning model to obtain an image similar to the image before or before the sharpness conversion processing has been applied from an image obtained after the sharpness conversion processing has been applied (hereinafter referred to as the target image).

[0021] <First step: Building a trained model> Below, we will explain machine learning for constructing a trained model (hereinafter referred to as the first trained model) that can infer an image in which the effect of sharpness conversion processing has been negated in the image processing device 105 of this embodiment.

[0022] When a target image is input, the image processing device 105 of this embodiment employs supervised learning to construct a first trained model in order to infer from the target image an image in which the effects of the sharpness conversion process have been negated. This learning can be performed using a technique such as Pix2Pix. Therefore, when constructing the first trained model, for example, a target image recorded by applying a sharpness conversion process to an image signal captured of the same scene and an image recorded without applying the sharpness conversion process (hereinafter referred to as an unprocessed image) are used as training data. That is, the training data includes the target image as the first training image according to the present invention as example data, and similarly includes the unprocessed image as the second training image as ground truth data. Furthermore, because the effect of the sharpness conversion process can vary depending on its strength (the degree to which the original image is sharpened), the training data also includes processing parameters related to the strength of the sharpness conversion process.

[0023] In this embodiment, in order to simplify the input of the machine learning model, the processing parameters are converted into a parameter map having the same number of dimensions as the target image and used for machine learning. More specifically, the processing parameters have the same pixel array as the target image and are configured as two-dimensional data in which the value of the processing parameter is stored in each pixel. In this way, the parameter map can also be treated as one of the channels of the target image input during machine learning, for example.

[0024] For example, when a target image 203 is generated through a sharpness conversion process using processing parameters 202, machine learning for constructing a first trained model is performed while providing training data to a machine learning model 205, as shown in FIG. 2. In the example shown in the figure, the machine learning model 205 is provided with a parameter map 204 generated based on the processing parameters 202 and the target image 203 as example data (hereinafter simply referred to as an example). For example, in an embodiment in which the machine learning model 205 is configured as a neural network having multiple layers of nodes, the machine learning model 205 repeatedly performs supervised learning using multiple sets of training data, thereby sequentially optimizing the connection weighting coefficients between the nodes. The machine learning model 205 outputs an inferred image 206 based on the connection weighting coefficients set at that stage for the input example data. Here, since the correct answer data (hereinafter simply referred to as the correct answer) is an unprocessed image 201 recorded without undergoing a sharpness conversion process, the machine learning model 205 adjusts the connection weighting coefficients so as to reduce the difference between the inferred image 206 and the unprocessed image 201.

[0025] By preparing multiple sets of training data and repeatedly learning in this way, the connection weighting coefficients of the machine learning model 205 are optimized, and a first trained model can be constructed.

[0026] Note that supervised learning may be performed on the sharpness conversion processing applied by the image processing device 105, for example, based on images obtained by capturing images (using sharpness conversion processing) performed before or after shipping of the imaging device 100. Alternatively, various data related to the first trained model may be supplied to the image processing device 105 from an external server (not shown) or the like by updating firmware of the imaging device 100. In this aspect, machine learning related to the construction of the first trained model may be performed not in the imaging device 100 but in another device.

[0027] Furthermore, in supervised learning, in order to learn the differences according to the strength of the sharpness conversion process, multiple types of target images generated by passing through sharpness conversion processes with multiple different processing parameters for one type of image signal capturing the same scene may be used as examples. In this way, it is possible to standardize the unprocessed image (an image of the scene recorded without passing through the sharpness conversion process) given as the correct answer, and therefore it is possible to efficiently perform multiple machine learning rounds (using multiple types of target images with different processing parameters).

[0028] <Use of the first trained model> By using the first trained model constructed by the above-described supervised learning, the image processing device 105 can obtain an image in a state in which the effect of the sharpness conversion processing has been negated from an image (target image) generated by applying the sharpness conversion processing. More specifically, the image processing device 105 acquires, for the target image, processing parameters used in the sharpness conversion processing when the target image was generated, and inputs a parameter map based on the processing parameters and the target image into the first trained model. With this input, the first trained model infers an image from the target image in which the effect of the sharpness conversion processing has been negated (hereinafter referred to as the first image), and the image processing device 105 can output the first image of the inference result as the image after the effect has been negated.

[0029] Counteraction The cancellation process for canceling the effect of the sharpness conversion process from an image, which is performed in the imaging device 100 of this embodiment, will be described in detail below with reference to the flowchart in Fig. 3. The process corresponding to this flowchart can be realized by the CPU 103 reading out a corresponding processing program stored in, for example, the secondary storage device 107, and then loading and executing the program in the primary storage device 104. This cancellation process will be described as being started, for example, when an operation input is made to specify the target image for which the effect is desired to be canceled.

[0030] In S301, the CPU 103 acquires a specified target image. The target image is acquired by, for example, reading the image data from the recording medium 106 and expanding it in the work memory of the primary storage device 104.

[0031] In S302, the CPU 103 acquires the processing parameters of the sharpness conversion process executed to generate the target image. The processing parameters can be acquired, for example, by referencing predetermined tag information from the header file of the image data of the target image.

[0032] In S303, under the control of the CPU 103, the image processing device 105 generates a parameter map based on the processing parameters acquired in S302.

[0033] In S304, under the control of the CPU 103, the image processing device 105 inputs the target image acquired in S301 and the parameter map generated in S303 into the first trained model. Then, the image processing device 105 outputs the first image inferred by the first trained model based on the input as an image in which the effect of the sharpness conversion process has been cancelled out from the target image, thereby completing the cancellation process.

[0034] As described above, the image processing device of this embodiment can generate an image in which the effects of irreversible image processing have been negated. Note that the negation process for negating the effects of irreversible image processing is not limited to a process that completely negates the effects of irreversible image processing, but may be a process that reduces the effects of irreversible image processing to a predetermined level or below. Similarly, an image in which the effects of irreversible image processing have been negated is not limited to an image in which the effects of irreversible image processing have been negated, but may be an image in which the effects of irreversible image processing have been reduced to a predetermined level or below, and may be an image that can be considered as if no irreversible image processing had been performed.

[0035] [Variation 1] In the above-described embodiment, in machine learning related to the construction of a first trained model, an unprocessed image recorded without applying a sharpness conversion process to an image signal captured of the same scene and a target image recorded after applying a sharpness conversion process are used as training data. However, the present invention is not limited to this example, and the target image used as training data may be an image generated by applying a sharpness conversion process to an unprocessed image.

[0036] [Variation 2] In the above-described embodiment, in machine learning for constructing a first trained model, an unprocessed image recorded without applying a sharpness conversion process to a captured image signal is used as the correct answer data. However, the present invention is not limited to this. The correct answer data does not necessarily have to be an image recorded without applying a sharpness conversion process. For example, in an embodiment in which the imaging device 100 is configured to apply a sharpness conversion process based on a reference value processing parameter to the image signal by default, image data recorded after undergoing a sharpness conversion process related to the reference value may be used as the correct answer data. In this case, the image data provided as example data may be image data recorded after undergoing a sharpness conversion process related to different processing parameters.

[0037] [Variation 3] In the above-described embodiment, in machine learning for constructing a first trained model, a target image recorded by applying a sharpness conversion process to a captured image signal is used as example data. However, the present invention is not limited to this. For example, the image used as example data may be an image obtained by extracting only the luminance component of an image signal to which a sharpness conversion process has been applied, which exhibits sharpness-related features.

[0038] In the above-described embodiment, a parameter map generated based on processing parameters related to a sharpness conversion process applied to a target image is used as example data so that machine learning can be performed in the form of a channel of the target image. However, the present invention is not limited to this example. For example, a first trained model can be constructed by machine learning in which processing parameters are provided as labels for the target image.

[0039] [Embodiment 2] In the above-described embodiment and modified examples, a first trained model is described that infers an image (first image) in which the effect of irreversible image processing has been canceled based on a target image to which the irreversible image processing has been applied. In this embodiment, an effective alternative embodiment using such a first trained model will be described.

[0040] <<Outline of further image processing for target image>> Some imaging devices 100 are equipped with a function for applying various image processing to image data recorded by capturing images. Since image processing functions are expected to be executed repeatedly and are preferably implemented with reduced power consumption, particularly in battery-powered devices, efforts have been made to implement them with reduced computational complexity. In one aspect, a trained model is introduced for some image processing functions, and by utilizing the corresponding trained model when using the function, an image in which the corresponding image processing has been applied to the input image is output by inference. In other words, by employing a trained model for some image processing functions that require high computational costs, a processed image can be generated without actually executing the image processing.

[0041] However, in order to build a trained model for such functions that can be used generally for images in various states, it is necessary to perform machine learning by providing various types of training data. On the other hand, in order to converge machine learning based on various types of training data, it may be necessary to perform training by providing a huge number of combinations of training data, for example, by using different processing parameters for each type. In other words, preparing various types of training data and performing machine learning to build a trained model for specific image processing requires high training costs. For this reason, in order to efficiently converge machine learning at an appropriate training cost, a method is adopted for building such a trained model by training training data limited to images with specific characteristics.

[0042] Therefore, a trained model constructed by such learning may not output a suitable inference result even if an image with different characteristics from the image used as training data is input. In other words, a trained model constructed by machine learning in which training data is limited to images with specific characteristics can reduce the learning cost for construction, but has the aspect of being less versatile in terms of input images that can produce suitable inference results.

[0043] In this embodiment, an aspect will be described in which a first trained model is used to obtain a suitable inference result in an imaging device 100 having an image processing function that uses inference by a trained model.

[0044] <Second trained model> Hereinafter, a trained model (hereinafter referred to as a second trained model) that infers a processed image for specific image processing that may involve an excessive amount of calculation and is used in the image processing device 105 of this embodiment will be described. In this embodiment, an example will be described in which the specific image processing is "super-resolution processing" that generates an image that reproduces detailed patterns while increasing the spatial resolution of an input image.

[0045] The second trained model for super-resolution processing is constructed by machine learning the characteristics of spatial resolution degradation that occurs during the imaging process, for example. In one aspect, the machine learning for constructing the second trained model is performed using training data in which a low-resolution image (low-resolution image) that is the source of the super-resolution processing is used as an example, and a high-resolution image (high-resolution image) output by applying the super-resolution processing to the low-resolution image is used as the correct answer. Alternatively, the machine learning for constructing the second trained model may be performed by generating low-resolution images by simulating degradation that occurs during the imaging process based on the high-resolution image, and using these as training data.

[0046] On the other hand, sharpness conversion processing has the property of increasing the sharpness of an image obtained by, for example, capturing, and acts in the direction of increasing the spatial resolution of the image. In other words, an image to which sharpness conversion processing has been applied may not be suitable for learning the characteristics of degradation of spatial resolution that occurs in the capturing process.

[0047] For this reason, the second trained model of this embodiment is constructed by machine learning using images excluding images to which sharpness conversion processing has been applied as training data. In other words, the low-resolution images given as examples when constructing the second trained model have different characteristics from images to which sharpness conversion processing has been applied. Therefore, the second trained model constructed by machine learning using such low-resolution images as training data may not output a suitable inference result even if an image generated by sharpness conversion processing is input.

[0048] Therefore, in the image processing device 105 of this embodiment, when an operation input is made to use the image processing function related to super-resolution processing for an image (target image) generated by sharpness conversion processing, the image processing device 105 first controls to input the target image to the first trained model. That is, the image processing device 105 causes the first trained model to infer a first image from the target image, with the effect of the sharpness conversion processing canceled out, and uses the first image as input to the second trained model. In this way, it is possible to obtain an image to which super-resolution processing has been applied for the target image while suppressing a decrease in the inference accuracy by the second trained model.

[0049] Note that the image obtained by inference of the second trained model using the first image as input (hereinafter referred to as the second image) does not have the effect of the sharpness conversion. Therefore, the image processing device 105 performs sharpness conversion processing on the second image to generate an image equivalent to the target image to which super-resolution processing has been applied (hereinafter referred to as the resultant image). That is, the image processing device 105 controls the reversible image processing applied to the target image to be reapplied to the second image. The sharpness conversion processing may be implemented, for example, by an image processing circuit, and the image processing device 105 can obtain the resultant image as an output by inputting the second image to the circuit. The resultant image obtained in this manner exhibits characteristics similar to those of an image to which both super-resolution processing and sharpness conversion processing have been applied, compared to an image to which the effect of the sharpness conversion processing has been negated from the target image (corresponding to an unapplied image).

[0050] Super-resolution processing Specific super-resolution processing executed in relation to the use of the image processing function in the imaging device 100 of this embodiment will be described below with reference to the flowchart in Fig. 4. The processing corresponding to this flowchart can be realized by the CPU 103 reading out a corresponding processing program stored in, for example, the secondary storage device 107, and deploying and executing the program in the primary storage device 104. This super-resolution processing will be described as being started when an operation input is made to use the image processing function related to super-resolution processing for a target image generated, for example, through sharpness conversion processing.

[0051] In S401, the image processing device 105 executes a cancellation process on the target image under the control of the CPU 103. Through the cancellation process, the image processing device 105 generates a first image from the target image in a state where the effect of the sharpness conversion process has been cancelled out.

[0052] In S402, under the control of the CPU 103, the image processing device 105 inputs the first image generated in S401 into a second trained model, thereby generating a second image as an inference result in which super-resolution processing has been applied to the first image.

[0053] In S403, the image processing device 105, under the control of the CPU 103, performs sharpness conversion processing on the second image generated in S402 based on the processing parameters acquired for the target image in the cancellation processing. Then, the image processing device 105 outputs the obtained resultant image, thereby completing this super-resolution processing.

[0054] By doing this, when using an image processing function that uses a second trained model constructed with reduced learning costs, it is possible to output a result image that has the desired effect while ensuring the inference accuracy of the second trained model.

[0055] [Variation 4] In the above-described second embodiment, a configuration was described in which the target image input to the second trained model associated with the image processing function is an image generated through a sharpness conversion process. However, the present invention is not limited to this configuration. As described above, to ensure the inference accuracy of the second trained model, it is sufficient to avoid inputting an image with characteristics different from those of the image used as training data in the machine learning for constructing the second trained model. In other words, if an image selected for use with the image processing function has characteristics similar to those of the training data associated with the second trained model, there is no need to input the image to the first trained model for inference. Furthermore, if such an image is input to the second trained model, there is no need to re-apply sharpness conversion processing to the inference result image. This modification describes a configuration in which, when using an image processing function that uses a second trained model, operation is selectively switched depending on whether the characteristics of the image to be processed are the same as those of the image used as training data for constructing the second trained model.

[0056] In the following description, it is assumed that a second image (output image) in a state where super-resolution processing has been applied to an image is inferred by a second trained model using an image processing function, as in embodiment 2. The second trained model is also assumed to be a machine learning model constructed by supervised learning using training data that includes, as at least examples, images generated without performing sharpness conversion processing.

[0057] Super-resolution processing Specific processing of super-resolution processing executed in relation to the use of the image processing function in the imaging device 100 of this modified example will be described below using the flowchart in Fig. 5. The processing corresponding to this flowchart can be realized by the CPU 103 reading out a corresponding processing program stored in, for example, the secondary storage device 107, and deploying and executing the program in the primary storage device 104. This super-resolution processing will be described as being started, for example, when an operation input is made to use the image processing function related to the super-resolution processing. Note that in the following description, steps that perform processing similar to that of the super-resolution processing of the second embodiment will be assigned the same reference numerals, and detailed description thereof will be omitted.

[0058] In S501, the CPU 103 receives an operation input related to the selection of an image (target image) to be processed in the image processing function. When the operation input is received, the CPU 103 shifts the processing to S502.

[0059] In S502, CPU 103 determines whether the selected target image is an image generated through a sharpness conversion process. That is, in this step, CPU 103 determines whether the target image is an image having characteristics different from those of an image used as training data in constructing a second trained model related to the image processing function. The determination in this step may be made, for example, by referencing generation information that describes information about the generation of the target image and is included in the header file of the target image. If CPU 103 determines that the target image is an image generated through a sharpness conversion process, it proceeds to S401. If CPU 103 determines that the target image is not an image generated through a sharpness conversion process, it proceeds to S503.

[0060] In S503, the image processing device 105, under the control of the CPU 103, inputs the target image to the second trained model, thereby generating a resultant image in which super-resolution processing has been applied to the target image as an inference result. The image processing device 105 then outputs the obtained resultant image, completing the super-resolution processing.

[0061] By doing this, the computational load of the image processing device 105 can be adaptively adjusted depending on whether the target image is an image in which the inference accuracy of the second trained model related to the image processing function is guaranteed.

[0062] [Variation 5] In the above-described second embodiment and fourth modification, when a first image inferred by a first trained model based on a target image is input to a second trained model, the sharpness conversion process is reapplied to the second image inferred by the second trained model. However, the present invention is not limited to this example, and a trained model for the sharpness conversion process (hereinafter referred to as a third trained model) may be used to apply the sharpness conversion process to the second image.

[0063] The sharpness conversion process performed during image capture is generally applied to the Bayer array image signal output from the image sensor 102, i.e., the image signal before development processing is performed. Therefore, the effect of the sharpness conversion process on the developed second image obtained as the inference result of the second trained model, as in Variation 4, may differ from the effect of the same process on the image signal before development. For this reason, in this Variation, when reapplying the sharpness conversion process, in order to bring the effect closer to that of the same process in the imaging process, the sharpness conversion process is not applied directly to the second image, but is instead realized by having the third trained model infer a resultant image based on the second image.

[0064] In one aspect, the construction of the third trained model can reuse the training data used in the construction of the first trained model. FIG. 6 outlines machine learning for the construction of the third trained model in this aspect. In the example shown in the figure, the unapplied image 201, the target image 203, and the parameter map 204, which were used as training data in the construction of the first trained model, are similarly provided as training data to the machine learning model 601, while machine learning for the construction of the second trained model is performed. More specifically, the unapplied image 201 and the parameter map 204 are provided as example problems to the machine learning model 601. The machine learning model 601 outputs an inferred image 602 for the input example problem by inference based on the connection weighting coefficients set at that stage. Furthermore, since the target image 203 is used as the correct answer, the machine learning model 601 adjusts the connection weighting coefficients so as to reduce the difference between the inferred image 602 and the target image 203. In other words, the machine learning involved in constructing the third trained model differs from the machine learning involved in constructing the first trained model in that the unapplied image 201 and the target image 203 are swapped in the example questions and correct answers given as training data.

[0065] By preparing multiple sets of training data and repeatedly learning in this way, the connection weighting coefficients of the machine learning model 601 are optimized, and a third trained model can be constructed.

[0066] Therefore, in S403 of the super-resolution processing, the image processing device 105 inputs the second image into the third trained model, thereby generating a resultant image in which the sharpness conversion processing has been applied to the second image as an inference result. By adopting the method of this modified example, the circuit scale and required calculation time related to the reapplication of the sharpness conversion processing may increase, but it is possible to obtain a resultant image that more accurately reproduces the effect of the sharpness conversion processing in the imaging process.

[0067] [Variation 6] In the above-described fifth variant, a second trained model is provided for an image processing function, such as super-resolution processing, and the sharpness conversion processing subsequently canceled using the first trained model is reapplied using the third trained model. However, the second trained model and the third trained model are essentially constructed through independent machine learning. Therefore, the characteristics of the image used as training data for the machine learning involved in constructing the third trained model may not be similar to the characteristics of the image output by the second trained model. In other words, depending on the combination of image processing related to the second trained model (e.g., super-resolution processing) and image processing related to the third trained model (e.g., sharpness conversion processing), the third trained model may not infer a suitable resulting image.

[0068] Therefore, for example, in an aspect where each trained model is constructed in advance and stored in a secondary storage device 107 or the like, additional learning may be performed to improve the inference accuracy of the third trained model. The additional learning may be performed by, for example, performing transfer learning on the third trained model constructed in advance.

[0069] Figure 7 illustrates an overview of additional learning related to the third trained model. In the example shown in the figure, a target image 701 selected as a processing target for the image processing function is input to a first trained model 703 together with a corresponding parameter map 702, and a first image 704 is output as an inference result. The first image 704 is also input to a second trained model 705 related to the image processing function, and the second trained model 705 outputs a second image 706 as an inference result. The second image 706 is input to a third trained model 707 together with the parameter map 702, and the third trained model 707 outputs a result image 708 as an inference result.

[0070] In the illustrated example, the additional learning unit 709 performs additional learning on the third trained model 707 based on the target image 701 used as input to the first trained model 703, its output first image 704, and the resultant image 708 output by the third trained model 707. During the additional learning process, the additional learning unit 709 sequentially constructs a new updated trained model 710 by updating the output layer of the third trained model 707. Then, for example, after a predetermined number of additional learnings have been performed, the additional learning unit 709 controls the operation of the image processing device 105 to use the constructed updated trained model 710 as the third trained model 707. Therefore, thereafter, the updated trained model 710 is used to reapply the image processing canceled by the first trained model 703, and the resultant image 711 generated by the updated trained model 710 with higher inference accuracy can be output.

[0071] [Variation 7] In the above-described second embodiment and modifications 4 to 6, when the effect of irreversible image processing is negated from a target image using a first trained model, the image processing is reapplied to a second image output from a second trained model. However, the present invention is not limited to this example. For example, if the effect of the irreversible image processing whose effect is negated by the first trained model is similar to the effect of image processing such as super-resolution processing realized by the second trained model, the reapplication of the irreversible image processing may be omitted. For example, super-resolution processing increases spatial resolution and sharpens details of a subject that were not represented in the input image, and therefore may have a similar effect to sharpness conversion processing. For this reason, the image processing device 105 may be configured to output the second image as a resultant image.

[0072] [Variation 8] In the above-described embodiment and modified examples, sharpness conversion processing has been described as an example of irreversible image processing that cancels the effect of the first trained model, but the implementation of the present invention is not limited to this. Such irreversible image processing can include, for example, color conversion, brightness conversion, noise removal, and optical characteristic conversion processing.

[0073] The color conversion process is a process of assigning a color to each pixel of the image signal by referring to a lookup table corresponding to a color conversion setting, for example. The brightness conversion process is a process of performing tone mapping based on settings such as contrast enhancement and brightness correction, for example. The noise reduction process and optical characteristic conversion process are processes that are adaptively performed based on settings of the image sensor 102, such as imaging sensitivity, and the lens characteristics of the optical system 101.

[0074] Therefore, the first trained model may be constructed to cancel out the effects of at least one of sharpness conversion, color conversion, brightness conversion, noise removal, and optical property conversion, or may be constructed for each of them and used adaptively. Furthermore, the combination of irreversible image processing whose effect is canceled by the first trained model and image processing realized by the second trained model may be other than the combination of sharpness conversion processing and super-resolution processing.

[0075] [Variation 9] In the above-described embodiment and modified examples, the present invention is embodied in an apparatus having an imaging function such as the imaging apparatus 100, but it goes without saying that the present invention is not limited to this.

[0076] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0077] [Summary of the embodiment and modifications] The disclosure of this specification includes the following image processing device, control method, and program. (Item 1) An image processing device that generates an image in which the effect of a first image processing is canceled out using a first trained model, a first acquisition means for acquiring a target image generated by applying the first image processing; a second acquisition means for acquiring processing parameters relating to the first image processing applied when generating the target image; a first inference means for inferring a first image from the target image by inputting the target image and the processing parameters into the first trained model, thereby canceling out the effect of the first image processing; and the first image processing is lossy image processing, The first trained model is a machine learning model constructed by machine learning of training data, in which a first training image generated by applying the first image processing to an image signal capturing a predetermined scene and the processing parameters related to the first image processing are used as examples, and a second training image generated without applying the first image processing to the image signal is used as a correct answer. 1. An image processing device comprising: (Item 2) 2. The image processing device according to item 1, wherein the first trained model is a machine learning model constructed by machine learning of training data in which a plurality of types of first training images generated by applying each of the first image processes of a plurality of types of processing parameters to one type of image signal are used as examples, and the second training image is used as a correct answer. (Item 3) 3. The image processing device according to item 1 or 2, further comprising a construction means for constructing the first trained model. (Item 4) The method further includes a second inference means for inferring an image obtained by applying second image processing to the image by inputting the image to a second trained model, the second trained model is a machine learning model constructed by machine learning of training data that uses as an example an image before the second image processing is applied, the image having characteristics different from those of the image to which the first image processing is applied, and sets as a correct answer an image after the second image processing is applied, The image processing device further includes a control means for causing the second inference means to perform inference based on the first image instead of the target image when causing the second inference means to perform inference based on the target image. 4. The image processing device according to any one of items 1 to 3, characterized in that: (Item 5) further comprising image processing means for outputting an image obtained by applying the first image processing to an input image; When the control means causes the second inference means to perform inference based on the first image instead of the target image, the control means controls the second image obtained by the inference to be input to the image processing means. 5. The image processing device according to item 4, (Item 6) The image processing means outputs an image inferred by inputting the image to a third trained model; The third trained model is a machine learning model constructed by machine learning of training data that uses an image before the application of the first image processing and the processing parameters related to the first image processing as examples, and an image after the application of the first image processing as a correct answer. 6. The image processing device according to item 5, (Item 7) 7. The image processing device according to item 6, further comprising a learning means for performing reinforcement learning of the third trained model based on the first image and the target image. (Item 8) 8. The image processing device according to any one of items 1 to 7, wherein the first image processing includes at least one of image sharpness conversion, color conversion, brightness conversion, noise removal, and optical characteristic conversion. (Item 9) An imaging means; a generation means for generating the target image by applying the first image processing to the captured image obtained by the imaging means; Further having 9. The image processing device according to any one of items 1 to 8, characterized in that: (Item 10) a first acquisition means for acquiring a target image; a second acquisition means for acquiring generation information relating to generation of the target image; a determination means for determining whether a first image processing has been performed to generate the target image based on the generation information; a generating means for generating an output image by applying a second image processing to the target image based on the target image; a control means for controlling the operation of the generating means; and When the first image processing is performed to generate the target image, the generation information includes processing parameters related to the first image processing, The generating means a first inference means for inferring an image in which the effect of the first image processing has been cancelled out from the input image by inputting an image generated by applying the first image processing and the processing parameters related to the first image processing into a first trained model; A second inference means for inferring an image obtained by applying second image processing to the input image by inputting the image to a second trained model; Including, The control means When the first image processing has not been performed to generate the target image, the second inference means generates the output image by inferring based on the target image; When the first image processing is performed to generate the target image, a first image is generated by causing the first inference means to perform inference based on the target image, and the output image is generated by causing the second inference means to perform inference based on the first image; the first trained model is a machine learning model constructed by machine learning of training data that uses a first training image generated by applying the first image processing to an image signal obtained by capturing a predetermined scene and the processing parameters related to the first image processing as examples, and a second training image generated without applying the first image processing to the image signal as a correct answer, The second trained model is a machine learning model constructed by machine learning of training data in which an example is an image before the second image processing is applied, the image having characteristics different from those of an image to which the first image processing is applied, and an image after the second image processing is applied is used as a correct answer. 1. An image processing device comprising: (Item 11) the generating means further includes image processing means for outputting an image obtained by applying the first image processing to the input image, When the first image processing is performed to generate the target image, the control means causes the second inference means to perform inference based on the first image to generate a second image, and causes the second image to be input to the image processing means to generate the output image. 11. The image processing device according to item 10. (Item 12) A control method for an image processing device that generates an image in which the effect of a first image processing is canceled out using a first trained model, a first acquisition step of acquiring a target image generated by applying the first image processing; a second acquisition step of acquiring processing parameters related to the first image processing applied when generating the target image; a first inference step of inferring a first image from the target image by inputting the target image and the processing parameters into the first trained model, thereby canceling out the effect of the first image processing; and the first image processing is lossy image processing, The first trained model is a machine learning model constructed by machine learning of training data, in which a first training image generated by applying the first image processing to an image signal capturing a predetermined scene and the processing parameters related to the first image processing are used as examples, and a second training image generated without applying the first image processing to the image signal is used as a correct answer. A control method comprising: (Item 13) 12. A program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 11.

[0078] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0079] 100: imaging device, 103: CPU, 104: primary storage device, 105: image processing device, 107: secondary storage device

Claims

1. An image processing device that generates an image in which the effect of a first image processing is canceled out using a first trained model, a first acquisition means for acquiring a target image generated by applying the first image processing; a second acquisition means for acquiring processing parameters related to the first image processing applied when generating the target image; a first inference means for inferring a first image from the target image by inputting the target image and the processing parameters into the first trained model, the first image being obtained by canceling out the effect of the first image processing; and the first image processing is lossy image processing, The first trained model is a machine learning model constructed by machine learning of training data, in which a first training image generated by applying the first image processing to an image signal obtained by capturing a predetermined scene and the processing parameters related to the first image processing are used as examples, and a second training image generated without applying the first image processing to the image signal is used as a correct answer.

1. An image processing device comprising:

2. The image processing device according to claim 1, characterized in that the first trained model is a machine learning model constructed by machine learning of training data in which a plurality of types of first training images generated by applying each of the first image processes of a plurality of types of processing parameters to one type of image signal are used as examples, and the second training image is used as a correct answer.

3. The image processing device according to claim 1 , further comprising a construction means for constructing the first trained model.

4. The method further includes a second inference means for inferring an image obtained by applying a second image processing to the image by inputting the image to a second trained model, the second trained model is a machine learning model constructed by machine learning of training data that uses as an example an image before the second image processing is applied, the image having characteristics different from those of the image to which the first image processing is applied, and sets as a correct answer an image after the second image processing is applied, The image processing device further includes a control means for causing the second inference means to perform inference based on the first image instead of the target image when causing the second inference means to perform inference based on the target image.

2. The image processing device according to claim 1, wherein:

5. further comprising image processing means for outputting an image obtained by applying the first image processing to an input image; When the control means causes the second inference means to perform inference based on the first image instead of the target image, the control means controls the second image obtained by the inference to be input to the image processing means.

5. The image processing device according to claim 4.

6. The image processing means outputs an image inferred by inputting the image to a third trained model; The third trained model is a machine learning model constructed by machine learning of training data, with an image before the application of the first image processing and the processing parameters related to the first image processing as examples, and an image after the application of the first image processing as a correct answer.

6. The image processing device according to claim 5,

7. The image processing device according to claim 6 , further comprising a learning means for performing additional learning of the third trained model based on the first image and the target image.

8. 2. The image processing device according to claim 1, wherein the first image processing includes at least one of image sharpness conversion, color conversion, brightness conversion, noise removal, and optical characteristic conversion.

9. An imaging means; a generation means for generating the target image by applying the first image processing to the captured image obtained by the imaging means; Further having 2. The image processing device according to claim 1, wherein:

10. a first acquisition means for acquiring a target image; a second acquisition means for acquiring generation information relating to generation of the target image; a determination means for determining whether a first image processing has been performed to generate the target image based on the generation information; a generating means for generating an output image by applying a second image processing to the target image based on the target image; a control means for controlling the operation of the generating means; and When the first image processing is performed to generate the target image, the generation information includes processing parameters related to the first image processing, The generating means a first inference means for inferring an image in which the effect of the first image processing has been cancelled out from the input image by inputting an image generated by applying the first image processing and the processing parameters related to the first image processing into a first trained model; A second inference means for inferring an image obtained by applying second image processing to the input image by inputting the image to a second trained model; Including, The control means When the first image processing has not been performed to generate the target image, the second inference means generates the output image by inferring based on the target image; When the first image processing is performed to generate the target image, the first inference means is caused to perform inference based on the target image to generate a first image, and the second inference means is caused to perform inference based on the first image to generate the output image, the first trained model is a machine learning model constructed by machine learning of training data that uses a first training image generated by applying the first image processing to an image signal obtained by capturing a predetermined scene and the processing parameters related to the first image processing as examples, and a second training image generated without applying the first image processing to the image signal as a correct answer, The second trained model is a machine learning model constructed by machine learning of training data, in which an example is an image before the second image processing is applied, the image having characteristics different from those of an image to which the first image processing is applied, and an image after the second image processing is applied is used as a correct answer.

1. An image processing device comprising:

11. the generating means further includes image processing means for outputting an image obtained by applying the first image processing to the input image, When the first image processing is performed to generate the target image, the control means causes the second inference means to perform inference based on the first image to generate a second image, and causes the second image to be input to the image processing means to generate the output image.

11. The image processing device according to claim 10.

12. A control method for an image processing device that generates an image in which the effect of a first image processing is canceled out using a first trained model, a first acquisition step of acquiring a target image generated by applying the first image processing; a second acquisition step of acquiring processing parameters related to the first image processing applied when generating the target image; a first inference step of inferring a first image from the target image by inputting the target image and the processing parameters into the first trained model, thereby canceling out the effect of the first image processing; and the first image processing is lossy image processing, The first trained model is a machine learning model constructed by machine learning of training data, in which a first training image generated by applying the first image processing to an image signal obtained by capturing a predetermined scene and the processing parameters related to the first image processing are used as examples, and a second training image generated without applying the first image processing to the image signal is used as a correct answer. A control method comprising:

13. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 11.

Images