Image processing device, control method, and program

Abstract

This image processing device uses a first trained model to generate an image in which the effect of first image processing is canceled, the image processing device comprising: a first acquisition means for acquiring a target image generated by applying the first image processing; a second acquisition means for acquiring a processing parameter related to the first image processing applied in the generation of the target image; and a first inference means for inputting the target image and the processing parameter to the first trained model to infer a first image corresponding to the target image in which the effect of the first image processing is canceled. The first image processing is irreversible image processing. The first trained model is a machine learning model constructed by machine learning that uses training data comprising an example and ground truth, wherein the example comprises a first training image generated by applying the first image processing to an image signal resulting from capturing a predetermined scene, and the processing parameter related to the first image processing; and the ground truth is a second training image generated without the application of the first image processing to the image signal.

Description

Image processing device, control method and program

[0001] The present disclosure relates to an image processing device, a control method, and a program, and more particularly to image processing for an image to which irreversible image processing has been applied.

[0002] When an image captured by an imaging device is recorded, various image processing processes such as gamma adjustment, white balance adjustment, noise reduction, sharpening, contrast adjustment, and saturation adjustment are applied (Patent Document 1).

[0003] JP 2019-191768 A

[0004] However, some image processing, such as sharpness processing, is irreversible, making it difficult to restore a recorded image to a state before the image processing was applied.

[0005] The present technology provides an image processing device, a control method, and a program for generating an image in which the effects of irreversible image processing are negated.

[0006] The image processing device of the present technology is an image processing device that generates an image in which the effect of first image processing has been negated using a first trained model, and includes: a first acquisition means that acquires a target image generated by applying the first image processing; a second acquisition means that acquires processing parameters related to the first image processing applied when generating the target image; and a first inference means that infers a first image in which the effect of the first image processing has been negated from the target image by inputting the target image and the processing parameters into the first trained model, wherein the first image processing is irreversible image processing; and the first trained model is a machine learning model constructed by machine learning of training data that uses as examples a first training image generated by applying the first image processing to an image signal capturing a predetermined scene and the processing parameters related to the first image processing, and a second training image generated without applying the first image processing to the image signal as a correct answer.

[0007] With this configuration, according to the present technology, it is possible to generate an image in which the effects of irreversible image processing are negated.

[0008] Other features and advantages of the present technology will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar components are designated by the same reference numerals.

[0009] The accompanying drawings are included in the specification and form a part thereof, show embodiments of the present technology, and are used to explain the principles of the present technology together with the description thereof.

[0023] A block diagram illustrating a hardware configuration of an imaging device 100 according to an embodiment and a modification of the present technology. A diagram for explaining machine learning related to the construction of a first trained model according to an embodiment and a modification of the present technology. A flowchart illustrating a cancellation process executed in the imaging device 100 according to an embodiment and a modification of the present technology. A flowchart illustrating a super-resolution process executed in the imaging device 100 according to an embodiment 2 of the present technology. A flowchart illustrating a super-resolution process executed in the imaging device 100 according to a modification 4 of the present technology. A diagram for explaining machine learning related to the construction of a third trained model according to a modification 5 of the present technology. A diagram for explaining additional learning of a third trained model according to a modification 6 of the present technology.

[0010] [Embodiment 1] Hereinafter, an embodiment will be described in detail with reference to the accompanying drawings. Note that the following embodiment does not limit the scope of the claims. Although multiple features are described in the embodiment, not all of these multiple features are required, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] In the embodiment described below, an example in which the present technology is applied to an imaging device, as an example of an image processing device, that has a function of canceling predetermined image processing on an image obtained by imaging using a machine learning model. However, the present technology can be applied to any device that can generate an image in which the effect of predetermined image processing is canceled using a machine learning model.

[0012] <Hardware Configuration of Imaging Device> The hardware configuration of an imaging device 100 according to this embodiment will be exemplified below using the block diagram of Fig. 1. The imaging device 100 may include, for example, a device provided for imaging purposes, such as a digital camera or a digital video camera, or an electronic device with an imaging function, such as a camera-equipped mobile phone or a camera-equipped computer.

[0013] The optical system 101 is an imaging optical system including a group of lenses, a shutter, an aperture, etc. The group of lenses may include a correction lens that corrects camera shake and the like, a focus lens, etc. The optical system 101 forms an image of subject light on the imaging surface of the imaging element 102 based on a control signal received from a CPU 103 (described later). The imaging element 102 is an imaging sensor such as a CCD (charge-coupled device) image sensor or a CMOS (complementary metal-oxide semiconductor) image sensor. The imaging element 102 converts the optical image formed on the imaging surface by the optical system 101 into an image signal by photoelectric conversion.

[0014] The CPU 103 is a control device that controls the operation of each block of the imaging device 100. The CPU 103 reads out the operation program of each block stored in the secondary storage device 107, and controls the operation of each block by expanding and executing the program in the primary storage device 104. When controlling the operation of each block, the CPU 103 appropriately sends a control signal corresponding to the corresponding operation to each block.

[0015] The secondary storage device 107 is a non-volatile storage device such as an EEPROM (Electrically Erasable Programmable Read-Only Memory). The secondary storage device 107 stores various setting information necessary for the operation of each block, as well as operation programs for each block of the imaging device 100 and firmware for the imaging device 100. The primary storage device 104 is a volatile storage device such as a RAM (Random Access Memory). The primary storage device 104 is used not only as a storage area for the operation programs of each block, but also as a work memory for storing temporary data output by the operation of each block. The information stored in the primary storage device 104 can be used by the image processing device 105 or recorded on the recording medium 106.

[0016] The recording medium 106 is a recording device, such as a semiconductor memory card, that is configured to be detachable from the imaging device 100. When an image is captured by the imaging device 100, the recording medium 106 is used to record image data that is stored in the primary storage device 104 by the image capture. The data recorded on the recording medium 106 can be used in an external device such as a PC (personal computer) by inserting the recording medium 106 into the external device.

[0017] The display unit 108 is a display device such as an LCD, and is used to display a viewfinder image during shooting, a captured image, a GUI (Graphical User Interface) image for interactive operation, and the like.

[0018] The operation unit 109 is a user interface such as a button, a lever, a touch panel, etc. When an operation input is made to various operation members, the operation unit 109 transmits a control signal corresponding to the operation input to the CPU 103. In addition, the operation unit 109 may also include an input device that uses voice, line of sight, etc.

[0019] The image processing device 105 is a device that performs various types of image processing on an image. In this embodiment, the image processing device 105 is configured to be able to apply multiple types of image processing to an image. The image processing to be performed by the image processing device 105 may be determined, for example, as a pattern for each shooting mode. In this embodiment, the image processing pattern to be applied by the image processing device 105 to the captured image is controlled based on information about the shooting mode set by the user. The image processing performed on the captured image includes so-called development processing as well as processing such as color tone adjustment. In some shooting modes provided in the imaging device 100 according to this embodiment, the image processing performed on the captured image includes irreversible image processing as the first image processing according to the present technology. In the following description, sharpness conversion processing that changes the sharpness of an image is exemplified as an example of irreversible image processing. That is, in some shooting modes, sharpness conversion processing is applied when converting the image signal (captured image) output by the image sensor 102 into image data to be recorded on the recording medium 106.

[0020] 1, the imaging device 100 is described as having the image processing device 105 as one piece of hardware, and the image processing device 105 executes image processing, but the implementation of the present technology is not limited to this. Part of the image processing may be realized by the CPU 103 executing a corresponding processing program.

[0021] <<Overview of the Process for Canceling the Effect of Sharpness Conversion>> As described above, in the imaging device 100 of this embodiment, image data recorded on the recording medium 106 by shooting in some shooting modes is generated by applying a sharpness conversion process to an image signal obtained by shooting. The sharpness conversion process is irreversible image processing, and basically, even if a conversion process with inverse characteristics is applied to image data after the sharpness conversion process has been applied, it is not possible to restore the image data before or before the sharpness conversion process has been applied. On the other hand, there may be a need to restore the image data to a state before the irreversible image processing has been applied, for example, when the shooting mode has been set incorrectly or when the strength of the sharpness conversion process needs to be changed. The image processing device 105 of this embodiment is configured to use a machine learning model to obtain an image similar to the image before or before the sharpness conversion process has been applied from an image obtained by applying the sharpness conversion process (hereinafter referred to as a target image).

[0022] <Construction of First Trained Model> Hereinafter, machine learning for constructing a trained model (hereinafter referred to as a first trained model) capable of inferring an image in which the effect of the sharpness conversion process has been negated in the image processing device 105 of this embodiment will be described.

[0023] When a target image is input, the image processing device 105 of this embodiment employs supervised learning to construct a first trained model in order to infer from the target image an image in which the effects of the sharpness conversion process have been negated. This learning can be performed using a technique such as Pix2Pix. Therefore, when constructing the first trained model, for example, a target image recorded by applying a sharpness conversion process to an image signal captured of the same scene and an image recorded without applying the sharpness conversion process (hereinafter referred to as an unapplied image) are used as training data. That is, the training data includes the target image serving as the first training image according to the present technology as example data, and similarly includes an unapplied image serving as the second training image as ground truth data. Furthermore, because the effect of the sharpness conversion process can vary depending on its strength (the degree to which the original image is sharpened), the training data also includes a processing parameter related to the strength of the sharpness conversion process.

[0024] In this embodiment, in order to simplify the input of the machine learning model, the processing parameters are converted into a parameter map having the same number of dimensions as the target image and used for machine learning. More specifically, the processing parameters have the same pixel array as the target image and are configured as two-dimensional data in which the value of the processing parameter is stored in each pixel. In this way, the parameter map can also be treated as one of the channels of the target image input during machine learning, for example.

[0025] For example, when a target image 203 is generated through a sharpness conversion process using processing parameters 202, machine learning for constructing a first trained model is performed while providing training data to a machine learning model 205, as shown in FIG. 2 . In the illustrated example, the machine learning model 205 is provided with a parameter map 204 generated based on the processing parameters 202 and the target image 203 as example data (hereinafter simply referred to as an example). For example, in an embodiment in which the machine learning model 205 is configured as a neural network having multiple layers of nodes, the machine learning model 205 repeatedly performs supervised learning using multiple sets of training data, thereby sequentially optimizing the connection weighting coefficients between the nodes. The machine learning model 205 outputs an inferred image 206 based on the connection weighting coefficients set at that stage for the input example data. Here, since the correct answer data (hereinafter simply referred to as the correct answer) is an unprocessed image 201 recorded without undergoing sharpness conversion processing, the machine learning model 205 adjusts the connection weighting coefficients to reduce the difference between the inferred image 206 and the unprocessed image 201.

[0026] By preparing multiple sets of training data and repeatedly learning in this way, the connection weighting coefficients of the machine learning model 205 are optimized, and a first trained model can be constructed.

[0027] Note that supervised learning may be performed on the sharpness conversion process applied by the image processing device 105, for example, based on images obtained by capturing images (using sharpness conversion process) performed before or after shipping of the imaging device 100. Alternatively, various data related to the first trained model may be supplied to the image processing device 105 from an external server (not shown) or the like by updating firmware of the imaging device 100. In this aspect, machine learning related to the construction of the first trained model may be performed not in the imaging device 100 but in another device.

[0028] Furthermore, in supervised learning, in order to learn the differences according to the strength of the sharpness conversion process, multiple types of target images generated by subjecting one type of image signal capturing the same scene to sharpness conversion processes with multiple different processing parameters may be used as examples. In this way, it is possible to standardize the unprocessed image (an image of the scene recorded without subjecting it to sharpness conversion processing) given as the correct answer, thereby enabling multiple machine learning rounds (using multiple types of target images with different processing parameters) to be performed efficiently.

[0029] <Use of First Trained Model> By using the first trained model constructed by the above-described supervised learning, the image processing device 105 can obtain an image in a state in which the effect of the sharpness conversion process has been negated from an image (target image) generated by applying the sharpness conversion process. More specifically, the image processing device 105 acquires, for the target image, processing parameters used in the sharpness conversion process when the target image was generated, and inputs a parameter map based on the processing parameters and the target image into the first trained model. With this input, the first trained model infers an image from the target image in which the effect of the sharpness conversion process has been negated (hereinafter referred to as the first image), and the image processing device 105 can output the first image of the inference result as an image after the effect has been negated.

[0030] <<Cancellation Process>> The cancellation process for canceling the effect of the sharpness conversion process from an image, which is performed in the imaging device 100 of this embodiment, will be described in detail below with reference to the flowchart in Fig. 3. The process corresponding to this flowchart can be realized by the CPU 103 reading out a corresponding processing program stored in, for example, the secondary storage device 107, and deploying and executing the program in the primary storage device 104. This cancellation process will be described as being started, for example, when an operation input is made to specify a target image for which it is desired to cancel the effect.

[0031] In step S301, the CPU 103 acquires a specified target image. The target image is acquired by, for example, reading the image data from the recording medium 106 and expanding it in the work memory of the primary storage device 104.

[0032] In S302, the CPU 103 acquires the processing parameters of the sharpness conversion process executed to generate the target image. The processing parameters can be acquired, for example, by referencing predetermined tag information from the header file of the image data of the target image.

[0033] In S303, under the control of the CPU 103, the image processing apparatus 105 generates a parameter map based on the processing parameters acquired in S302.

[0034] In S304, under the control of the CPU 103, the image processing device 105 inputs the target image acquired in S301 and the parameter map generated in S303 into the first trained model. Then, the image processing device 105 outputs the first image inferred by the first trained model based on the input as an image in which the effect of the sharpness conversion process has been cancelled out from the target image, thereby completing the cancellation process.

[0035] As described above, the image processing device of this embodiment can generate an image in which the effects of irreversible image processing have been negated. Note that the negation process for negating the effects of irreversible image processing is not limited to a process that completely negates the effects of irreversible image processing, but may be a process that reduces the effects of irreversible image processing to a predetermined level or below. Similarly, an image in which the effects of irreversible image processing have been negated is not limited to an image in which the effects of irreversible image processing have been negated, but may be an image in which the effects of irreversible image processing have been reduced to a predetermined level or below, and may be an image that can be considered as if no irreversible image processing had been performed.

[0036] [Variation 1] In the above-described embodiment, in machine learning related to the construction of a first trained model, an aspect has been described in which an unapplied image recorded without applying a sharpness conversion process to an image signal obtained by capturing an identical scene and a target image recorded after applying a sharpness conversion process are used as training data. However, the implementation of the present technology is not limited to this, and the target image used as training data may be an image generated by applying a sharpness conversion process to an unapplied image.

[0037] [Variation 2] In the above-described embodiment, in machine learning for constructing a first trained model, an unprocessed image recorded without applying a sharpness conversion process to a captured image signal is used as the correct answer data. However, the implementation of the present technology is not limited to this. The image used as the correct answer data does not necessarily have to be an image recorded without applying a sharpness conversion process. For example, in an embodiment in which the imaging device 100 is configured to apply a sharpness conversion process based on a reference value processing parameter by default to the image signal, image data recorded after undergoing a sharpness conversion process related to the reference value may be used as the correct answer data. In this case, the image data provided as example data may be image data recorded after undergoing a sharpness conversion process related to different processing parameters.

[0038] [Modification 3] In the above-described embodiment, in machine learning related to the construction of the first trained model, a target image recorded by applying a sharpness conversion process to a captured image signal is used as example data. However, the implementation of the present technology is not limited to this. For example, the image used as example data may be an image obtained by extracting only the luminance component of an image signal to which a sharpness conversion process has been applied, in which sharpness-related features are apparent.

[0039] In the above-described embodiment, a parameter map generated based on processing parameters related to a sharpness conversion process applied to a target image is used as example data so that machine learning can be performed in the form of a channel of the target image. However, the implementation of the present technology is not limited to this. For example, the first trained model can be constructed by machine learning in which processing parameters are provided as labels of the target image.

[0040] [Embodiment 2] In the above-described embodiment and modified examples, a first trained model is described that infers an image (first image) in which the effect of irreversible image processing has been canceled based on a target image to which the irreversible image processing has been applied. In this embodiment, an effective alternative embodiment using such a first trained model will be described.

[0041] <<Overview of Further Image Processing on Target Image>> Some imaging devices 100 are equipped with a function for applying various image processing to image data recorded by capturing an image. Since image processing provided as a single function is expected to be executed repeatedly and is preferably implemented with reduced power consumption, particularly in battery-powered devices, efforts have been made to implement the processing with reduced computational complexity. In one aspect, a trained model is introduced for some image processing, and by utilizing the corresponding trained model when using the function, an image in which the corresponding image processing has been applied to the input image is output by inference. In other words, by employing a trained model for some image processing that requires high computational costs, a processed image can be generated without actually executing the image processing.

[0042] However, in order to build a trained model for such functions that can be used generally for images in various states, it is necessary to perform machine learning by providing various types of training data. On the other hand, in order to converge machine learning based on various types of training data, it may be necessary to perform training by providing a huge number of combinations of training data, for example, by using different processing parameters for each type. In other words, preparing various types of training data and performing machine learning to build a trained model for specific image processing requires high training costs. For this reason, in order to efficiently converge machine learning at an appropriate training cost, a method is adopted for building such a trained model by training training data limited to images with specific characteristics.

[0043] Therefore, a trained model constructed by such learning may not output a suitable inference result even if an image with different characteristics from the image used as training data is input. In other words, a trained model constructed by machine learning in which training data is limited to images with specific characteristics can reduce the learning cost for construction, but has the aspect of being less versatile in terms of input images that can produce suitable inference results.

[0044] In this embodiment, an embodiment will be described in which a first trained model is used to obtain suitable inference results in an imaging device 100 having an image processing function that uses inference based on a trained model.

[0045] <Second Trained Model> Hereinafter, a trained model (hereinafter referred to as a second trained model) that infers an image after specific image processing that may involve an excessive amount of calculation and is used in the image processing device 105 of this embodiment will be described. In this embodiment, an example will be described in which the specific image processing is "super-resolution processing" that generates an image that reproduces detailed patterns while increasing the spatial resolution of an input image.

[0046] The second trained model for super-resolution processing is constructed, for example, by machine learning the characteristics of spatial resolution degradation that occurs during the imaging process. In one aspect, the machine learning for constructing the second trained model is performed using training data in which a low-resolution image (low-resolution image) that is the source of the super-resolution processing is used as an example, and a high-resolution image (high-resolution image) output by applying the super-resolution processing to the low-resolution image is used as the correct answer. Alternatively, the machine learning for constructing the second trained model may be performed by generating low-resolution images by simulating degradation that occurs during the imaging process based on the high-resolution image, and using these as training data.

[0047] On the other hand, sharpness conversion processing has the property of increasing the sharpness of an image obtained by, for example, capturing, and acts in the direction of increasing the spatial resolution of the image. In other words, an image to which sharpness conversion processing has been applied may not be suitable for learning the characteristics of degradation of spatial resolution that occurs in the capturing process.

[0048] For this reason, the second trained model of this embodiment is constructed by machine learning using images excluding images to which sharpness conversion processing has been applied as training data. In other words, the low-resolution images given as examples when constructing the second trained model have different characteristics from images to which sharpness conversion processing has been applied. Therefore, the second trained model constructed by machine learning using such low-resolution images as training data may not output a suitable inference result even if an image generated by sharpness conversion processing is input.

[0049] Therefore, in the image processing device 105 of this embodiment, when an operation input related to the use of the image processing function related to super-resolution processing is performed on an image (target image) generated by sharpness conversion processing, the image processing device 105 first controls the target image to be input to the first trained model. That is, the image processing device 105 causes the first trained model to infer a first image from the target image, with the effect of the sharpness conversion processing canceled out, and uses the first image as input to the second trained model. In this way, it is possible to obtain an image to which super-resolution processing has been applied to the target image while suppressing a decrease in the inference accuracy of the second trained model.

[0050] Note that the image obtained by inference of the second trained model using the first image as input (hereinafter referred to as the second image) does not have the effect of the sharpness conversion. Therefore, the image processing device 105 performs a sharpness conversion process on the second image to generate an image equivalent to the target image to which super-resolution processing has been applied (hereinafter referred to as the resultant image). That is, the image processing device 105 controls the reversible image processing applied to the target image to be reapplied to the second image. The sharpness conversion process may be implemented, for example, by an image processing circuit, and the image processing device 105 can obtain the resultant image as an output by inputting the second image to the circuit. The resultant image obtained in this manner exhibits characteristics similar to those of an image to which both super-resolution processing and sharpness conversion processing have been applied, compared to an image from which the effect of the sharpness conversion process has been negated (corresponding to an unapplied image) from the target image.

[0051] <<Super-resolution Processing>> Specific super-resolution processing executed in relation to the use of the image processing function in the imaging device 100 of this embodiment will be described below using the flowchart in Fig. 4. The processing corresponding to this flowchart can be realized by the CPU 103 reading out a corresponding processing program stored in, for example, the secondary storage device 107, and deploying and executing the program in the primary storage device 104. This super-resolution processing will be described as being started when an operation input is made to use the image processing function related to super-resolution processing for a target image generated, for example, through sharpness conversion processing.

[0052] In S401, the image processing device 105 executes a cancellation process on the target image under the control of the CPU 103. Through the cancellation process, the image processing device 105 generates a first image from the target image in a state where the effect of the sharpness conversion process has been cancelled out.

[0053] In S402, under the control of the CPU 103, the image processing device 105 inputs the first image generated in S401 into a second trained model, thereby generating a second image in which super-resolution processing has been applied to the first image as an inference result.

[0054] In S403, the image processing device 105, under the control of the CPU 103, performs sharpness conversion processing on the second image generated in S402 based on the processing parameters related to the target image acquired in the cancellation processing. The image processing device 105 then outputs the obtained resultant image, completing the super-resolution processing.

[0055] By doing this, when using an image processing function that uses a second trained model constructed with reduced learning costs, it is possible to output a result image that has the desired effect while ensuring the inference accuracy of the second trained model.

[0056] [Variation 4] In the above-described second embodiment, a configuration was described in which the target image input to the second trained model associated with the image processing function was an image generated through a sharpness conversion process. However, implementation of the present technology is not limited to this. As described above, to ensure the inference accuracy of the second trained model, it is sufficient to avoid inputting an image with characteristics different from those of the image used as training data in the machine learning for constructing the second trained model. In other words, if an image selected for use with the image processing function has characteristics similar to those of the training data associated with the second trained model, there is no need to input the image to the first trained model for inference. Furthermore, if such an image is input to the second trained model, there is no need to re-apply sharpness conversion processing to the image resulting from the inference. This variation describes a configuration in which, when using an image processing function that uses a second trained model, operation is selectively switched depending on whether the characteristics of the image to be processed are the same as those of the image used as training data for constructing the second trained model.

[0057] In the following description, it is assumed that a second image (output image) in a state where super-resolution processing has been applied to an image is inferred by a second trained model using an image processing function, as in embodiment 2. The second trained model is also assumed to be a machine learning model constructed by supervised learning using training data that includes, as at least example problems, images generated without performing sharpness conversion processing.

[0058] <<Super-resolution Processing>> Specific processing of the super-resolution processing executed in relation to the use of the image processing function in the imaging device 100 of this modified example will be described below using the flowchart in Fig. 5. The processing corresponding to this flowchart can be realized by the CPU 103 reading out a corresponding processing program stored in, for example, the secondary storage device 107, and deploying and executing the program in the primary storage device 104. This super-resolution processing will be described as being started, for example, when an operation input is made to use the image processing function related to the super-resolution processing. Note that in the following description, steps that perform processing similar to the super-resolution processing of the second embodiment will be assigned the same reference numbers, and detailed description thereof will be omitted.

[0059] In S501, the CPU 103 receives an operation input related to the selection of an image (target image) to be processed in the image processing function. When the operation input is received, the CPU 103 advances the processing to S502.

[0060] In S502, the CPU 103 determines whether the selected target image is an image generated through a sharpness conversion process. That is, in this step, the CPU 103 determines whether the target image is an image having characteristics different from those of an image used as training data in constructing a second trained model related to the image processing function. The determination in this step may be made, for example, by referencing generation information that describes information about the generation of the target image and is included in the header file of the target image. If the CPU 103 determines that the target image is an image generated through a sharpness conversion process, the CPU 103 proceeds to S401. If the CPU 103 determines that the target image is not an image generated through a sharpness conversion process, the CPU 103 proceeds to S503.

[0061] In S503, under the control of the CPU 103, the image processing device 105 inputs the target image into the second trained model, thereby generating a resultant image in a state in which super-resolution processing has been applied to the target image as an inference result. The image processing device 105 then outputs the obtained resultant image, completing the super-resolution processing.

[0062] By doing this, the computational load of the image processing device 105 can be adaptively adjusted depending on whether the target image is an image in which the inference accuracy of the second trained model related to the image processing function is guaranteed.

[0063] [Modification 5] In the above-described second embodiment and modification 4, when a first image inferred by a first trained model based on a target image is input to a second trained model, the sharpness conversion process is re-applied to the second image inferred by the second trained model. However, the implementation of the present technology is not limited to this, and a trained model for the sharpness conversion process (hereinafter referred to as a third trained model) may be used to apply the sharpness conversion process to the second image.

[0064] The sharpness conversion process performed during image capture is generally applied to the Bayer array image signal output from the image sensor 102, i.e., the image signal before development processing is performed. Therefore, the effect of the sharpness conversion process on the developed second image obtained as the inference result of the second trained model, as in Variation 4, may differ from the effect of the same process on the image signal before development. For this reason, in this Variation, when reapplying the sharpness conversion process, in order to bring the effect closer to that of the same process in the imaging process, the sharpness conversion process is not applied directly to the second image, but is instead realized by having the third trained model infer a resultant image based on the second image.

[0065] In one aspect, the construction of the third trained model can reuse the training data used in the construction of the first trained model. FIG. 6 shows an overview of machine learning for the construction of the third trained model in this aspect. In the example shown in the figure, the unapplied image 201, the target image 203, and the parameter map 204, which were used as training data in the construction of the first trained model, are similarly provided as training data to the machine learning model 601, while machine learning for the construction of the second trained model is performed. More specifically, the unapplied image 201 and the parameter map 204 are provided as example problems to the machine learning model 601. The machine learning model 601 outputs an inferred image 602 for the input example problem by inference based on the connection weighting coefficients set at that stage. Furthermore, since the target image 203 is used as the correct answer, the machine learning model 601 adjusts the connection weighting coefficients so as to reduce the difference between the inferred image 602 and the target image 203. In other words, the machine learning involved in constructing the third trained model differs from the machine learning involved in constructing the first trained model in that the unapplied image 201 and the target image 203 are swapped in the example questions and correct answers given as training data.

[0066] By preparing multiple sets of training data and repeatedly learning in this way, the connection weighting coefficients of the machine learning model 601 are optimized, and a third trained model can be constructed.

[0067] Therefore, in S403 of the super-resolution processing, the image processing device 105 inputs the second image to the third trained model, and can generate a resultant image in which the sharpness conversion processing has been applied to the second image as an inference result. By adopting the method of this modified example, the circuit scale and required calculation time related to the reapplication of the sharpness conversion processing may increase, but it is possible to obtain a resultant image that more accurately reproduces the effect of the sharpness conversion processing in the imaging process.

[0068] [Variation 6] In the above-described Variation 5, a second trained model is provided for an image processing function, such as super-resolution processing, and the sharpness conversion processing subsequently canceled using the first trained model is reapplied using a third trained model. However, the second trained model and the third trained model are essentially constructed through independent machine learning. Therefore, the characteristics of the image used as training data for the machine learning involved in constructing the third trained model may not be similar to the characteristics of the image output by the second trained model. In other words, depending on the combination of image processing related to the second trained model (e.g., super-resolution processing) and image processing related to the third trained model (e.g., sharpness conversion processing), the third trained model may not infer a suitable resultant image.

[0069] Therefore, for example, in an aspect in which each trained model is constructed in advance and stored in the secondary storage device 107 or the like, additional learning may be performed to improve the inference accuracy of the third trained model. The additional learning may be performed by, for example, performing transfer learning on the third trained model constructed in advance.

[0070] 7 illustrates an overview of additional learning related to the third trained model. In the example shown in the figure, a target image 701 selected as a processing target for the image processing function is input to a first trained model 703 together with a corresponding parameter map 702, and a first image 704 is output as an inference result. The first image 704 is also input to a second trained model 705 related to the image processing function, and the second trained model 705 outputs a second image 706 as an inference result. The second image 706 is input to a third trained model 707 together with the parameter map 702, and the third trained model 707 outputs a result image 708 as an inference result.

[0071] In the illustrated example, the additional learning unit 709 performs additional learning on the third trained model 707 based on the target image 701 used as input to the first trained model 703, its output first image 704, and the resultant image 708 output by the third trained model 707. During the additional learning process, the additional learning unit 709 sequentially constructs a new updated trained model 710 by updating the output layer of the third trained model 707. Then, for example, after a predetermined number of additional learnings have been performed, the additional learning unit 709 controls the operation of the image processing device 105 to use the constructed updated trained model 710 as the third trained model 707. Therefore, thereafter, the updated trained model 710 is used to reapply the image processing canceled by the first trained model 703, and the resultant image 711 generated by the updated trained model 710 with higher inference accuracy can be output.

[0072] [Variation 7] In the above-described second embodiment and variations 4 to 6, when the effect of irreversible image processing is negated from the target image using the first trained model, the image processing is reapplied to the second image output from the second trained model. However, the implementation of the present technology is not limited to this. For example, if the effect of the irreversible image processing whose effect is negated by the first trained model is similar to the effect of image processing such as super-resolution processing realized by the second trained model, the reapplication of the irreversible image processing may be omitted. For example, super-resolution processing has the effect of increasing spatial resolution and sharpening details of a subject that were not represented in the input image, and therefore may have a similar effect to sharpness conversion processing. For this reason, the image processing device 105 may be configured to output the second image as a resultant image.

[0073] [Modification 8] In the above-described embodiment and modification, sharpness conversion processing has been described as an example of irreversible image processing that cancels the effect of the first trained model, but implementation of the present technology is not limited to this. Such irreversible image processing may include, for example, color conversion, brightness conversion, noise removal, and optical characteristic conversion processing.

[0074] The color conversion process is a process of assigning a color to each pixel of the image signal by referring to a lookup table corresponding to a color conversion setting, for example. The brightness conversion process is a process of performing tone mapping based on settings such as contrast enhancement and brightness correction, for example. The noise reduction process and optical characteristic conversion process are processes that are adaptively performed based on settings of the image sensor 102, such as imaging sensitivity, and the lens characteristics of the optical system 101.

[0075] Therefore, the first trained model may be constructed to cancel out the effects of at least one of sharpness conversion, color conversion, brightness conversion, noise removal, and optical property conversion, or may be constructed for each of them and used adaptively. Furthermore, the combination of the irreversible image processing that cancels out the effect of the first trained model and the image processing realized by the second trained model may be other than the combination of sharpness conversion processing and super-resolution processing.

[0076] [Variation 9] In the above-described embodiment and variations, the present technology is implemented in an apparatus having an imaging function such as the imaging device 100, but it goes without saying that the implementation of the present technology is not limited to this.

[0077] The present technology can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present technology can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

[0078] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0079] This application claims priority based on Japanese Patent Application No. 2024-130007, filed August 6, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. An image processing device that uses a first trained model to generate an image in which the effect of first image processing has been negated, comprising: a first acquisition means that acquires a target image generated by applying the first image processing; a second acquisition means that acquires processing parameters related to the first image processing that was applied when generating the target image; and a first inference means that infers a first image in which the effect of the first image processing has been negated from the target image by inputting the target image and the processing parameters into the first trained model, wherein the first image processing is irreversible image processing, and the first trained model is a machine learning model constructed by machine learning of training data that uses, as examples, a first training image generated by applying the first image processing to an image signal that captures a predetermined scene and the processing parameters related to the first image processing, and a second training image generated without applying the first image processing to the image signal as a correct answer.

2. The image processing device described in claim 1, characterized in that the first trained model is a machine learning model constructed by machine learning of training data in which multiple types of first training images generated by applying each of the first image processing of multiple types of processing parameters to one type of image signal are used as examples, and the second training image is used as a correct answer.

3. The image processing device according to claim 1, further comprising a construction means for constructing the first trained model.

4. The image processing device described in claim 1 further comprises a second inference means for inferring an image after a second image processing has been applied to the image by inputting the image into a second trained model, wherein the second trained model is a machine learning model constructed by machine learning of training data that uses as an example an image before the second image processing has been applied, the image having characteristics different from those of the image to which the first image processing has been applied, and the image after the second image processing has been applied as a correct answer, and wherein the image processing device further comprises a control means for causing the second inference means to perform inference based on the first image instead of the target image when causing the second inference means to perform inference based on the target image.

5. An image processing device as described in claim 4, further comprising an image processing means for outputting an image in a state where the first image processing has been applied to an input image, and wherein the control means controls the second inference means to input a second image obtained by the inference to the image processing means when the second inference means is made to perform inference based on the first image instead of the target image.

6. The image processing device described in claim 5, characterized in that the image processing means outputs an inferred image by inputting the image into a third trained model, and the third trained model is a machine learning model constructed by machine learning of training data, using an image before the application of the first image processing and the processing parameters related to the first image processing as example problems, and an image after the application of the first image processing as a correct answer.

7. The image processing device according to claim 6, further comprising a learning means for additionally learning the third trained model based on the first image and the target image.

8. The image processing device according to claim 1, wherein the first image processing includes at least one of image sharpness conversion, color conversion, brightness conversion, noise removal, and optical characteristic conversion.

9. An image processing device according to claim 1, further comprising: an imaging means; and a generating means for generating the target image by applying the first image processing to an image obtained by the imaging means.

10. A system comprising: a first acquisition means for acquiring a target image; a second acquisition means for acquiring generation information related to the generation of the target image; a determination means for determining, based on the generation information, whether a first image processing was performed in generating the target image; a generation means for generating, based on the target image, an output image by applying a second image processing to the target image; and a control means for controlling the operation of the generation means, wherein the generation information includes processing parameters related to the first image processing if the first image processing was performed in generating the target image; the generation means comprising: a first inference means for inferring, from the input image, an image generated by applying the first image processing and the processing parameters related to the first image processing by inputting the image generated by applying the first image processing and the processing parameters related to the first image processing into a first trained model; and a second inference means for inferring an image in a state where the second image processing has been applied to the input image by inputting the image into a second trained model; and wherein the control means generates the output image by having the second inference means make an inference based on the target image if the first image processing was not performed in generating the target image, An image processing apparatus characterized in that, when the first image processing is performed to generate the target image, a first image is generated by having the first inference means make inference based on the target image, and the output image is generated by having the second inference means make inference based on the first image, wherein the first trained model is a machine learning model constructed by machine learning of teacher data that takes as examples a first training image generated by applying the first image processing to an image signal capturing a predetermined scene and the processing parameters related to the first image processing, and a second training image generated without applying the first image processing to the image signal as a correct answer, and the second trained model is a machine learning model constructed by machine learning of teacher data that takes as examples an image before the second image processing is applied, an image having characteristics different from the image to which the first image processing has been applied, and an image after the second image processing has been applied as a correct answer.

11. The image processing device described in claim 10, characterized in that the generation means further includes image processing means for outputting an image in a state where the first image processing has been applied to an input image, and the control means, when the first image processing has been performed to generate the target image, generates a second image by having the second inference means perform inference based on the first image, and generates the output image by inputting the second image to the image processing means.

12. A control method for an image processing device that generates an image in which the effect of first image processing is negated using a first trained model, comprising: a first acquisition step of acquiring a target image generated by applying the first image processing; a second acquisition step of acquiring processing parameters related to the first image processing applied when generating the target image; and a first inference step of inferring a first image in which the effect of the first image processing is negated from the target image by inputting the target image and the processing parameters into the first trained model, wherein the first image processing is irreversible image processing, and the first trained model is a machine learning model constructed by machine learning of training data that uses, as examples, a first training image generated by applying the first image processing to an image signal capturing a predetermined scene and the processing parameters related to the first image processing, and a second training image generated without applying the first image processing to the image signal as a correct answer.

13. A program for causing a computer to function as each means of the image processing device according to any one of claims 1 to 11.

Images