Image processing device, image processing method, and program

Abstract

The present disclosure relates to an image processing device, an image processing method, and a program that enable, at low cost and in large quantity, generation of appropriate learning datasets required for image conversion. A reverse converter invISP applies parameters, which have been applied to a prescribed function in a forward converter ISP, to an inverse function of the prescribed function, and converts a large number of RGB images of normal image quality into pseudo RAW images through calculation processing that adds a prescribed disturbance. The forward converter ISP converts the converted pseudo RAW images into RGB images corresponding to a user's preferences or purposes. This makes it possible to generate a large quantity of learning datasets at low cost from a large number of RGB images of normal image quality, the learning datasets pairing pseudo RAW images and RGB images that correspond to the user's preferences or purposes. The present invention can be applied to a system that learns ISPs.

Description

Image processing device, image processing method, and program

[0001] The present disclosure relates to an image processing device, an image processing method, and a program, and in particular to an image processing device, an image processing method, and a program that enable the generation of appropriate training datasets required for image conversion at low cost and in large quantities.

[0002] In ISPs (Image Signal Processors), style transfer devices, image generation devices, etc., the format of an original image is converted into an image format intended by the user. In recent years, a machine learning technique has been proposed for this image conversion, which uses a training dataset consisting of pairs of original images and corresponding correct images for the conversion.

[0003] In the case of an ISP, the original image is a RAW image, which is the format of an image captured by an image sensor, and the correct image is an RGB image in a color space format such as sRGB (standard RGB).

[0004] However, in the above-mentioned image conversion techniques, the RGB image that is the desired correct image differs depending on the user's preference or the intended use, and is not one and the same.

[0005] For this reason, it is conceivable to customize the above-mentioned image conversion device by machine learning using correct images according to user preferences and applications.

[0006] With an ISP, customization can be achieved by adjusting parameters related to image conversion, but there are many parameters, and complete adjustments are complicated and time-consuming.

[0007] It is also possible to customize the system by generating a training dataset consisting of a pair of a correct image (RGB image) and a corresponding image (RAW image) that matches the user's preferences and purpose, and then training the system.

[0008] However, to improve the accuracy of customization, it is necessary to generate a large amount of training datasets tailored to the user's preferences and purposes, which is time-consuming and costly.

[0009] Therefore, a technology has been proposed in which a forward transformer that converts a RAW image into a correct image (RGB image) and an inverse transformer that converts the correct image (RGB image) into a corresponding RAW image are trained as a pair, and a training dataset is generated using the trained forward transformer and inverse transformer (see Non-Patent Document 1).

[0010] Furthermore, in order to train a forward transformer that converts a RAW image into a ground truth image (RGB image), a technology has been proposed in which a training dataset is generated by converting one ground truth image (RGB image) into multiple RAW images while changing parameters using an inverse transformer (see Non-Patent Document 2).

[0011] https: / / openaccess.thecvf.com / content_CVPR_2020 / papers / Zamir_CycleISP_Real_Image_Restoration_via_Improved_Data_Synthesis_CVPR_2020_paper.pdfhttps: / / arxiv.org / abs / 1811.11127

[0012] In ISP and image conversion, images that are noisy or have insufficient illumination are converted into clean images, so original images with various problems are mapped into one clean, correct image.

[0013] Therefore, for example, the dataset required for training an ISP is a ground truth image (RGB image) and multiple RAW images, but the technology in Non-Patent Document 1 is based on the premise that RAW images and ground truth images (RGB images) are generated in a one-to-one correspondence.

[0014] For this reason, the technology of Non-Patent Document 1 can only create an inappropriate training dataset consisting of a pair of a correct image (RGB image) and a biased RAW image, and therefore, sufficient training cannot be achieved.

[0015] Furthermore, the technology in Non-Patent Document 2 can generate a training dataset by generating multiple RAW images for one ground truth image (RGB image) while changing parameters, thereby enabling diverse learning.

[0016] However, in the technology of Non-Patent Document 2, the inverse transformer is not paired with the forward transformer, and the exact inverse transformer is unknown. Therefore, many parameters can only be changed randomly without any restrictions. As a result, RAW images may be generated with parameters that are never set in reality, and an appropriate training dataset may not necessarily be generated.

[0017] The present disclosure has been made in light of such circumstances, and in particular, makes it possible to generate appropriate learning datasets required for image conversion at low cost and in large quantities.

[0018] An image processing device and a program according to one aspect of the present disclosure include a forward transformer that converts a first image into a second image having predetermined characteristics by an arithmetic process in which parameters set based on a first image are applied to a predetermined function, and an inverse transformer that converts the second image into the first image by an arithmetic process in which the parameters are applied to an inverse function of the predetermined function, wherein the inverse transformer converts the second image, which does not have the predetermined characteristics, into a pseudo-first image by an arithmetic process in which the parameters are applied to the inverse function of the predetermined function and the predetermined disturbance is added, and the forward transformer converts the pseudo-first image into a second image having the predetermined characteristics.

[0019] An information processing method according to one aspect of the present disclosure includes a forward transformation process that converts a first image into a second image having predetermined characteristics by applying parameters set based on a first image to a predetermined function, and an inverse transformation process that converts the second image into the first image by applying the parameters to an inverse function of the predetermined function, wherein the inverse transformation process converts the second image, which does not have the predetermined characteristics, into a pseudo-first image by applying the parameters to the inverse function of the predetermined function and adding the predetermined disturbance, and the forward transformation process converts the pseudo-first image into a second image having the predetermined characteristics.

[0020] In one aspect of the present disclosure, a calculation process applies parameters set based on a first image to a predetermined function, thereby converting the first image into a second image having predetermined characteristics; a calculation process applies the parameters to an inverse function of the predetermined function, thereby converting the second image into the first image; a calculation process applies the parameters to the inverse function of the predetermined function and adds the predetermined disturbance, thereby converting the second image not having the predetermined characteristics into a pseudo first image, and converting the pseudo first image into a second image having the predetermined characteristics.

[0021] 16 is a diagram explaining the need to increase the volume of a training dataset. FIG. 17 is a diagram explaining the relationship between RGB images and RAW images that constitute a training dataset. FIG. 18 is a diagram explaining an overview of the present disclosure. FIG. 19 is a diagram explaining an example configuration of an image processing system according to a preferred embodiment of the present disclosure. FIG. 20 is a diagram explaining an example configuration of an ISP learning server. FIG. 21 is a diagram explaining an example configuration of an ISP. FIG. 22 is a diagram explaining an example configuration of an invISP. FIG. 23 is a diagram explaining an ISP learning process. FIG. 24 is a diagram explaining the first process in FIG. 25. FIG. 26 is a diagram explaining the second process in FIG. 26. FIG. 27 is a diagram explaining the third and fourth processes in FIG. 27. FIG. 28 is a diagram explaining an example configuration of a noise addition unit. FIG. 29 is a diagram explaining an example configuration of an ISP that realizes forward transformation processing on a pixel-by-pixel basis. FIG. 29 is a diagram explaining an application example of the image processing system of the present disclosure. FIG. 21 is a diagram explaining processing by the ISP learning server of FIG. 16 to generate a corresponding RAW image from an RGB image that is in accordance with a new user's preferences and purpose, thereby generating a training dataset. FIG. 29 is a diagram explaining an example configuration of a general-purpose computer.

[0022] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0023] Hereinafter, embodiments of the present technology will be described in the following order.

[0024] 1. Overview of the present disclosure 2. Preferred embodiments 3. Application examples 4. Examples of implementation by software

[0025] <<1. Overview of the Present Disclosure>> The present disclosure is directed to enabling the generation of a large amount of appropriate training datasets required for image conversion achieved by training, particularly at low cost. Therefore, an overview of the present disclosure will first be described.

[0026] Imaging devices such as cameras and smartphones equipped with cameras are equipped with an ISP (Image Signal Processor) that converts RAW images, which are the actual image data captured by an image sensor, into RGB images expressed in a color space format such as sRGB (standard RGB).

[0027] The ISP consists of multiple types of converters, and each of the multiple converters converts the image based on the parameters set for it, so that the ISP as a whole converts the RAW image into an RGB image with a specific style or taste that is specified by the multiple parameters.

[0028] Although it is possible to set these multiple parameters individually by hand, the number of parameters is so diverse that setting them individually is not a realistic task except for specialized experts.For this reason, in recent years, a method has begun to be adopted in which these multiple parameters are set by learning using a training dataset consisting of pairs of RGB images that serve as the correct images and their corresponding RAW images.

[0029] However, there is no correct answer for the style or taste, which is a predetermined tendency specified by multiple parameters, and it varies depending on the user's preferences and use. For this reason, the multiple parameters need to be individually set according to the user's preferences and use.

[0030] Therefore, it is conceivable to set multiple parameters by learning using a learning data set according to the user's preferences and purpose.

[0031] To set multiple parameters for the ISP through learning, it is necessary to prepare a large amount of training datasets consisting of RGB images, which are correct images according to the user's preferences and purposes, and corresponding RAW images.

[0032] However, generating a training dataset consisting of RGB images as correct answers and their corresponding RAW images is a very costly task, so there is a limit to the amount that can be prepared. In particular, even if a certain number of RGB images as correct answers can be collected, generating the corresponding RAW images is not easy.

[0033] Therefore, as shown in Figure 1, it is conceivable to identify trends that correspond to user preferences and uses based on a small amount of training dataset Dt that can be prepared, and then use a large amount of existing RGB images Dg to generate a training dataset Db consisting of pairs of RAW images and RGB images that have trends that correspond to the identified user preferences and uses.

[0034] The training dataset Db generated from a large number of existing RGB images is also referred to as an augmented training dataset Db, as it is generated to increase the volume of the training dataset.

[0035] Here, the RAW images constituting the bulky training dataset Db are generated based on a large number of existing RGB images Dg. However, the RGB images constituting the bulky training dataset Db are RGB images obtained by improving the image quality of the RAW images constituting the bulky training dataset Db so as to correspond to the trends identified from the small number of training datasets Dt. Therefore, the RGB images constituting the bulky training dataset Db are improved in image quality to have a tendency to correspond to user preferences and uses more than the large number of existing RGB images Dg.

[0036] As a method for generating a bulky training dataset Db, a technology has been proposed (Non-Patent Document 1) that generates a training dataset by training a pair of a forward transformer that converts a RAW image into a correct image (RGB image) and an inverse transformer that converts the correct image (RGB image) into the corresponding RAW image.

[0037] However, the technology of Non-Patent Document 1 is based on the premise that a RAW image and a correct image (RGB image) are generated in a one-to-one correspondence, and that only a training dataset consisting of a pair of one correct image (RGB image) and one RAW image is created.

[0038] 2, a training dataset is essentially one in which multiple RAW images PR1 to PR4 are set for one RGB image Ps that is the ground truth image, and each pair constitutes an individual training dataset. Therefore, multiple RAW images PR should be generated for one RGB image Ps that is the ground truth image, and multiple training datasets should be generated from the multiple pairs.

[0039] However, the technology of Non-Patent Document 1 is based on the premise that a RAW image and a correct image (RGB image) are generated in a one-to-one correspondence. Therefore, as shown in FIG. 2, only a training data set is generated in which, for example, only the RAW image PR4 is paired with a single correct image, RGB image Ps.

[0040] For this reason, the technology of Non-Patent Document 1 can only create a training dataset consisting of a pair of a target image (RGB image) and a biased RAW image, and it is not possible to generate many training datasets, which may result in insufficient learning in terms of both accuracy and quantity.

[0041] In addition, a technology has been proposed in which multiple inverse transformations are formed using multiple transformation units that make up an ISP, and parameters in the multiple inverse transformation units are changed to generate multiple RAW data for a target image (RGB image), thereby generating multiple training datasets (see Non-Patent Document 2).

[0042] That is, the ISP has a configuration in which a plurality of converters are connected in multiple stages, as shown in the upper part of FIG. 3, and converts the input image, RAW image PR11, sequentially from the previous stage and finally converts it into an RGB image Ps.

[0043] 3 is composed of a first converter 1-1, a first converter 1-2, ..., an n-th converter 1-n. The first converter 1-1, the first converter 1-2, ..., the n-th converter 1-n sequentially perform conversion processing by calculating conversion functions based on predetermined parameters Pm1, Pm2, ..., Pmn, respectively, to convert the RAW image PR11 into an RGB image, which is the correct image.

[0044] On the other hand, the inverse transform unit invISP1' is composed of an n-th inverse transformer 1'-n, an (n-1)-th inverse transformer 1'-(n-1), ..., and a first inverse transformer 1'-1, as shown in the lower part of FIG.

[0045] The nth inverse converter 1'-n, the (n-1)th inverse converter 1'-(n-1), ... the first inverse converter 1'-1 are respectively composed of inverse functions of the conversion functions in the nth converter 1-n, the (n-1)th converter 1-(n-1), ... the first converter 1-1, and sequentially convert the RGB image in the reverse direction based on the parameters Pmn', Pm(n-1)', ... Pm1' supplied to each converter, to convert it into a RAW image.

[0046] At this time, invISP1' converts the RGB image, which is the ground truth image, into multiple RAW images while varying the parameters Pmn', Pm(n-1)', ..., Pm1' in various ways, thereby generating multiple RAW images for one RGB image and generating multiple training datasets. As a result, various variations of RAW images are generated based on one RGB image, which is the ground truth image, making it possible to generate more training datasets.

[0047] However, in the technology of Non-Patent Document 2, the parameters Pmn', Pm(n-1)', ..., Pm1' are changed completely randomly without any restrictions to generate multiple RAW images. As a result, the parameters Pmn', Pm(n-1)', ..., Pm1' may be set to values ​​or combinations that cannot actually occur, which may result in the generation of inappropriate RAW images.

[0048] Therefore, in the present disclosure, although the parameters Pmn', Pm(n-1)', ..., Pm1' set in each of the nth inverse transformer 1'-n and the (n-1)th inverse transformer 1'-(n-1) ..., first inverse transformer 1'-1 of invISP1' are changed, the changes are limited to the set of parameters Pm1, Pm2, ..., Pmn that are actually set in the ISP 1 or to the vicinity thereof. As a result, the parameters Pmn', Pm(n-1)', ..., Pm1' are changed among realistic values ​​and combinations, making it possible to generate a large number of appropriate RAW images while suppressing the generation of inappropriate RAW images that occur when values ​​or combinations that cannot actually occur are set.

[0049] In addition, by improving the image quality of the large number of appropriate RAW images generated in this way using an ISP trained with a small amount of training data set according to the user's preferences and purpose, a large number of RGB images that are appropriate correct images according to the user's preferences and purpose are generated.

[0050] As a result, by pairing a large number of generated appropriate RAW images with a large number of appropriate RGB images, it is possible to generate a large number of appropriate training datasets at low cost.

[0051] <<2. Preferred Embodiment>> Next, a configuration example of a preferred embodiment of an image processing system according to the present disclosure will be described with reference to FIG.

[0052] The image processing system 11 in FIG. 4 comprises a user PC 31 , an ISP learning server 32 , and an image storage server 33 .

[0053] The user PC (Personal Computer) 31 is a personal computer used by a user such as an image processing expert who mainly creates an RGB image that will be a desired ground truth image (GT (Ground Truth) image) for a RAW image.

[0054] The work of adding an RGB image that will be the desired correct answer image (GT image) (according to the user's preference (tendency) or the user's desired use) to a RAW image generated by the user PC 31 and generating a training dataset consisting of the generated RAW image and RGB image requires time and cost. For this reason, it is assumed that the training dataset generated by the user PC 31, consisting of a RAW image and an RGB image that will be the desired correct answer image (GT image), will be small in size.

[0055] Note that the "small amount" referred to here is not sufficient compared to the amount of images necessary and sufficient for training the ISP (Image Signal Processor), but it does not refer to a specific amount of images, but rather indicates an amount that allows the tendency of RGB images that will become correct images according to the user's preferences and purpose to be grasped to some extent as a result of training.

[0056] The user PC 31 supplies the ISP learning server 32 with a learning dataset consisting of a pair of a RAW image and an RGB image that is the correct image, which is generated in response to user input. In response to this, the user PC 31 acquires the learned ISP learned by the ISP learning server 32, stores it as ISP 31a, and uses it to convert images according to the user's preferences and purposes.

[0057] Based on a training dataset consisting of pairs of RAW images and RGB images serving as correct images supplied from the user PC 31, the ISP learning server 32 generates a training dataset consisting of pairs of RAW images and RGB images required for ISP training, using a large number of RGB images of normal image quality stored in the image storage server 33. Furthermore, the ISP learning server 32 trains the ISP using both the training dataset supplied from the user PC 31 and the generated training dataset. The ISP learning server 32 then provides the trained ISP to the user PC 31.

[0058] The image storage server 33 is a server that stores RGB images of normal image quality, and supplies the RGB images of normal image quality to the ISP learning server 32 when the ISP learning server 32 generates a learning data set for learning the ISP.

[0059] <Configuration Example of ISP Learning Server> Next, a configuration example of the ISP learning server 32 will be described with reference to FIG.

[0060] The ISP learning server 32 is composed of a control unit 101, an input unit 102, an output unit 103, a memory unit 104, a communication unit 105, a drive 106, and a removable storage medium 107, which are interconnected via a bus 108 and can send and receive data and programs.

[0061] The control unit 101 is composed of a processor and a memory, and controls the overall operation of the ISP learning server 32. The control unit 101 also includes a learning unit 131, a forward transformation processing unit 132, an inverse transformation processing unit 133, and a parameter set acquisition unit 134.

[0062] The learning unit 131 trains the ISP 151 based on a training data set consisting of pairs of RGB images and corresponding RAW images. The training data set used by the learning unit 131 is a combination of the training data set generated by the forward transformation processing unit 132 and the inverse transformation processing unit 133 and the training data set supplied by the user PC 31.

[0063] The forward conversion processing unit 132 uses the ISP 151 stored in the storage unit 104 to execute forward conversion processing for converting a RAW image into an RGB image with high image quality in accordance with the user's preferences and purpose. An example of the configuration of the ISP 151 will be described in detail later with reference to FIG. 6.

[0064] The inverse conversion processing unit 133 executes inverse conversion processing for converting an RGB image into a RAW image, using the invISP 152 stored in the storage unit 104. Note that an example configuration of the invISP 152 will be described in detail later with reference to FIG. 7 .

[0065] The parameter set acquisition unit 134 acquires the parameter sets set in the ISP 151 when the forward transformation processing unit 132 executes the forward transformation process, and stores the parameter sets in the parameter set group 153 .

[0066] In addition, when the reverse conversion processing unit 133 executes the reverse conversion processing, the parameter set acquisition unit 134 acquires the parameter set stored in the parameter set group 153 and supplies it to the invISP 152 to execute the reverse conversion processing.

[0067] The parameter set set in the ISP 151 and the parameter set supplied to the invISP 152 will be explained together with the explanation of the configurations of the ISP 151 and the invISP 152.

[0068] The input unit 102 is made up of input devices such as a keyboard, a mouse, and a touch panel for inputting various types of information, and supplies the control unit 101 with various signals corresponding to the input information.

[0069] The output unit 103 is controlled by the control unit 101 and includes a display unit and an audio output unit. The display unit displays data input by the input unit 102 and the results of various processes executed by the control unit 101.

[0070] The audio output unit is made up of an audio output device such as a speaker, and outputs various types of voice, music, sound effects, and the like as audio.

[0071] The storage unit 104 is composed of a hard disk drive (HDD), a solid state drive (SSD), or a semiconductor memory, and is controlled by the control unit 101 to write or read various data and programs. The storage unit 104 also stores an ISP 151, an invISP 152, and a parameter set group 153, which are written or read as needed under the control of the control unit 101.

[0072] The communication unit 105 is controlled by the control unit 101, and realizes wired or wireless communication represented by LAN (Local Area Network) or Bluetooth (registered trademark), and transmits and receives various data and programs to and from other information processing devices, such as the user PC 31 and the image storage server 33, via the network as necessary.

[0073] The drive 106 reads and writes data from and to a removable storage medium 107 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0074] <Configuration Example of ISP> Next, functions realized by the ISP 151 will be described with reference to the functional block diagram of FIG.

[0075] As described above, the ISP 151 is controlled by the forward conversion processing unit 132 to convert the input RAW image into a high-quality RGB image based on the learning dataset and in accordance with the preferences and intended use of a specific user, and then output the converted image.

[0076] More specifically, the ISP 151 is made up of a color correction unit 171 , a white balance adjustment unit 172 , a gain adjustment unit 173 , a noise removal unit 174 , a tone conversion unit 175 , a reduction unit 176 , a feature extraction unit 177 , and a parameter determination unit 178 .

[0077] The reduction unit 176 reduces the size of the RAW image PR, which is the input image, and outputs it to the feature extraction unit 177 .

[0078] The feature extraction unit 177 extracts feature amounts relating to a predetermined image from the reduced-size RAW image and supplies them to the parameter determination unit 178 .

[0079] The parameter determination unit 178 determines and supplies parameters Pc, Pv, Pg, Pn, and Pt required for the color correction unit 171, the white balance adjustment unit 172, the gain adjustment unit 173, the noise removal unit 174, and the gradation conversion unit 175, respectively, based on the feature amounts supplied from the feature extraction unit 177. At this time, the parameters Pc, Pv, Pg, Pn, and Pt supplied by the parameter determination unit 178 are acquired as a parameter set by the parameter set acquisition unit 134 and stored in the storage unit 104 as a parameter set group 153. Note that, although an example in which one parameter is set for each of the color correction unit 171 to the gradation conversion unit 175 has been described, the number of parameters that is set is not limited to one, and multiple parameters may be set.

[0080] The color correction unit 171 performs color correction on the input image, the RAW image PR, by executing a predetermined function calculation process related to color correction based on the color correction parameters Pc supplied from the parameter determination unit 178, and supplies the result to the subsequent white balance adjustment unit 172.

[0081] The white balance adjustment unit 172 adjusts the white balance by performing a predetermined function calculation process related to the white balance of the color-corrected RAW image PR based on the white balance adjustment parameter Pw supplied from the parameter determination unit 178, and supplies the result to the downstream gain adjustment unit 173.

[0082] The gain adjustment unit 173 adjusts the gain intensity by performing a predetermined function calculation process on the gain intensity of the white balance adjusted RAW image PR based on the gain intensity Pg, which is a parameter supplied from the parameter determination unit 178, and supplies the adjusted gain intensity to the downstream noise removal unit 174.

[0083] The noise removal unit 174 performs noise removal by performing a predetermined function calculation process related to noise removal of the gain-adjusted RAW image PR based on the denoising intensity Pn, which is a parameter supplied from the parameter determination unit 178, and supplies the result to the subsequent gradation conversion unit 175.

[0084] The gradation conversion unit 175 performs gradation conversion by performing a predetermined function calculation process related to gradation conversion of the noise-removed RAW image PR based on the gradation conversion parameters Pt supplied from the parameter determination unit 178, and outputs the result as an RGB image Ps.

[0085] It is assumed that, in the initial stage, the ISP 151 has been trained so that the parameter determination unit 178 determines various parameters based on a learning dataset supplied from the user PC 31 so as to convert the input image, which is a RAW image, into an RGB image that suits the user's preferences and purpose.

[0086] Furthermore, the ISP 151 is ultimately trained using a larger training dataset generated from RGB images of normal image quality.

[0087] Furthermore, the color correction unit 171, the white balance adjustment unit 172, the gain adjustment unit 173, the noise reduction unit 174, and the gradation conversion unit 175 are examples of components constituting the ISP 151, and other processing may be added, such as a sharpening unit. Furthermore, not all of the color correction unit 171, the white balance adjustment unit 172, the gain adjustment unit 173, the noise reduction unit 174, and the gradation conversion unit 175 are essential, and only some of them may be provided. Furthermore, as shown in FIG. 6 , the processing order of the color correction unit 171, the white balance adjustment unit 172, the gain adjustment unit 173, the noise reduction unit 174, and the gradation conversion unit 175 may be reversed.

[0088] <Configuration Example of invISP> Next, functions realized by the invISP 152 will be described with reference to the functional block diagram of FIG.

[0089] As described above, the invISP 152 is controlled by the inverse conversion processing unit 133, and performs inverse conversion processing, which is processing in the reverse direction of the forward conversion processing performed by the ISP 151, to convert the RGB image into a RAW image and output it.

[0090] More specifically, the invISP 152 is made up of a tone inverse conversion section 175', a noise addition section 174', a gain inverse adjustment section 173', a white balance inverse adjustment section 172', and a color inverse correction section 171'.

[0091] Here, the parameter set acquisition unit 134 reads out one of the parameter sets from the parameter set group 153, and supplies the read parameters Pt, Pn, Pg, Pv, and Pc to the tone inverse conversion unit 175', noise addition unit 174', gain inverse adjustment unit 173', white balance inverse adjustment unit 172', and color inverse correction unit 171', respectively.

[0092] The gradation inverse conversion unit 175′ executes processing in the opposite direction corresponding to that of the gradation conversion unit 175, and performs functional calculation processing using an inverse function of a predetermined function in the gradation conversion unit 175 related to the gradation conversion of the RGB image Pr based on the gradation conversion parameters Pt supplied from the parameter set acquisition unit 134, thereby performing gradation inverse conversion on the RGB image Ps and supplying it to the subsequent noise addition unit 174′.

[0093] The noise addition unit 174′ executes processing in the opposite direction to that of the noise removal unit 174, and adds noise to the tone-inversely adjusted RGB image Ps by performing a function calculation process using an inverse function of a predetermined function in the noise removal unit 174 based on the denoising intensity Pn, which is a parameter supplied from the parameter set acquisition unit 134, and supplies the result to the downstream gain inverse adjustment unit 173′.

[0094] The gain inverse adjustment unit 173′ executes processing in the opposite direction to that of the gain adjustment unit 173, and performs functional calculation processing using an inverse function of a predetermined function in the gain adjustment unit 173 related to the gain intensity of the noise-added RGB image Ps, based on the gain intensity Pg, which is a parameter supplied from the parameter set acquisition unit 134, to inversely adjust the gain and supply the result to the subsequent white balance inverse adjustment unit 172′.

[0095] The white balance inverse adjustment unit 172′ executes processing in the opposite direction to that of the white balance adjustment unit 172, and performs a function calculation process using an inverse function of a predetermined function in the white balance adjustment unit 172 related to the white balance of the RGB image Ps whose gain has been inversely adjusted, based on the white balance adjustment parameters Pw supplied from the parameter set acquisition unit 134, thereby performing inverse adjustment and supplying the result to a color inverse correction unit 171′ at the subsequent stage.

[0096] The color inverse correction unit 171′ executes processing in the opposite direction to that of the color correction unit 171, and performs calculation processing using an inverse function of a predetermined function in the color correction unit 171 related to color correction of the gain-inversely adjusted RGB image Ps based on the color correction parameters Pc supplied from the parameter set acquisition unit 134, thereby performing inverse correction and outputting the result as a RAW image.

[0097] It should be noted that the invISP 152 needs to have a configuration corresponding to the configuration of the ISP 151. Furthermore, even if the parameter set acquisition unit 134, for example, randomly selects any of the parameter sets registered in the parameter set group 153 and supplies the selected parameter sets to the tone inverse conversion unit 175′, the noise addition unit 174′, the gain inverse adjustment unit 173′, the white balance inverse adjustment unit 172′, and the color inverse correction unit 171′, these parameter sets are actually generated when the ISP 151 is trained, and therefore the RGB image can be converted into an appropriate RAW image.

[0098] Furthermore, by performing reverse conversion on the same RGB image to a RAW image using various parameter sets registered in the parameter set group 153, it becomes possible to generate RAW images with various variations from a single RGB image.

[0099] Furthermore, although the above has described an example in which one of the parameter sets 153 is selected, it is sufficient that the parameter set acquisition unit 134 supplies parameter sets that can actually occur to the tone inverse conversion unit 175', the noise addition unit 174', the gain inverse adjustment unit 173', the white balance inverse adjustment unit 172', and the color inverse correction unit 171'. Therefore, for example, at least one of the values ​​of any of the parameter sets in the parameter set group 153 may be changed to, for example, another value close to the value of the parameter set, thereby providing further variation. More specifically, the parameter set acquisition unit 134 may add a disturbance by changing the value of at least one of the parameter sets so as to increase or decrease it by about several percent relative to the value of the parameter set, thereby providing a parameter set.

[0100] Furthermore, the gradation inverse conversion unit 175′, noise addition unit 174′, gain inverse adjustment unit 173′, white balance inverse adjustment unit 172′, and color inverse correction unit 171′ are examples of components that make up the invISP 152, similar to the ISP 151, and other processing may be added thereto, for example, an inverse sharpening unit may be added.

[0101] Furthermore, not all of the gradation inverse conversion unit 175', noise addition unit 174', gain inverse adjustment unit 173', white balance inverse adjustment unit 172', and color inverse correction unit 171' are essential, and only some of them may be provided as long as they correspond to the ISP 151. Furthermore, the processing order of the gradation inverse conversion unit 175', noise addition unit 174', gain inverse adjustment unit 173', white balance inverse adjustment unit 172', and color inverse correction unit 171' does not necessarily have to be the order shown in Fig. 7, and as long as they correspond to the ISP 151, the processing order may be reversed.

[0102] <Processing Procedure for Generating a Large Training Data Set from a Small Training Data Set> Next, with reference to FIG. 8, a processing procedure for generating a large training data set from a small training data set will be described.

[0103] In step St1, the learning unit 131 trains the ISP 151 based on a training data set consisting of a pair of a DB (DataBase) 191 of small amounts of RAW images (small RAW DB) that corresponds to the user's preferences and purpose and a DB 192 of RGB images (small RGB DB), which are supplied from the user PC 31. Note that the "loss function" shown in Fig. 8 expresses that the ISP 151 is trained so as to minimize the loss function.

[0104] At this time, in the ISP 151, the parameter determination unit 178 acquires a parameter set set based on features extracted from the input image, which is a RAW image, and the parameter set acquisition unit 134 stores the parameter set in the memory unit 104 as a parameter set group 153.

[0105] 9, the details of step St1 will be described using, as an example, the white balance adjustment unit 172, the gain adjustment unit 173, and the noise removal unit 174 among the units from the color correction unit 171 to the gradation conversion unit 175 of the ISP 151. However, the same processing is also performed for the color correction unit 171 and the gradation conversion unit 175, and the description will be omitted.

[0106] That is, based on the features supplied from the feature extraction unit 177, the parameter determination unit 178 sets the white balance adjustment parameters Pw1 and Pw2 for the white balance adjustment unit 172 to 0.2 and 0.3, sets the gain intensity Pg for the gain adjustment unit 173 to 3.0, and sets the noise intensity Pn for the noise removal unit 174 to 2.1.

[0107] In this case, the parameter set acquisition unit 134 acquires parameters Pw1, Pw2, Pg, and Pn, and registers the parameter set P1={0.2, 0.3, 0.0, 2.1} in the parameter set group 153. Similarly, parameter sets P2, P3, ... corresponding to the feature amounts of the other input images are registered in the parameter set group 153.

[0108] 9 shows an example in which, in addition to the parameter set P1, parameter sets P2={0.2, 0.2, 7.0, 2.5} and P3={0.2, 0.2, 3.2, 2.4} are registered in the parameter set group 153.

[0109] Now, let us return to the description of FIG.

[0110] In step St2, the inverse conversion processing unit 133 reads out a DB of RGB images of normal image quality (Large RGB DB) 193 from the image storage server 33, and controls the invISP 152 to perform inverse conversion processing to generate a DB of RAW images (Large pseudo RAW DB) 194.

[0111] At this time, in the invISP 152, the parameter set acquisition unit 134 randomly selects at least one of the parameter sets registered in the parameter set group 153, thereby adding disturbance to the inverse transformation process.

[0112] 10, the details of step St2 will be described using, as an example, the noise addition unit 174', the gain inverse adjustment unit 173', and the white balance inverse adjustment unit 172' among the gradation inverse conversion unit 175' to the color inverse correction unit 171' of the invISP 152. However, the same processing is also performed for the gradation inverse conversion unit 175' and the color inverse correction unit 171', and only the description will be omitted.

[0113] That is, the parameter set acquisition unit 134 reads out a registered parameter set from the parameter set group 153, supplies it to a noise addition unit 174', a gain inverse adjustment unit 173', and a white balance inverse adjustment unit 172', and causes them to execute inverse conversion processing.

[0114] For example, in FIG. 10, parameter set P1 is read out, a noise intensity Pn of 2.1 is supplied to a noise addition unit 174′, a gain intensity Pg of 3.0 is supplied to a gain inverse adjustment unit 173′, and white balance adjustment parameters Pw1 and Pw2 of 0.2 and 0.3 are supplied to a white balance inverse adjustment unit 172′, and the noise addition unit 174′, gain inverse adjustment unit 173′, and white balance inverse adjustment unit 172′ each perform their corresponding processing, thereby performing an inverse conversion process on the RGB image and generating a RAW image.

[0115] Alternatively, as shown in FIG. 11, multiple parameter sets may be supplied to the same RGB image, and inverse transformation processing may be performed based on each parameter set to generate different RAW images.

[0116] That is, the top row of Figure 11 shows an example in which, for a single input image, RGB image PsX, in process St1, a parameter set P1 that was set when generating RGB image PP1 is supplied to invISP152, thereby generating a RAW image PRX1 by inverse conversion processing.

[0117] Also, the lower part of Figure 11 shows an example in which a parameter set P2 set when generating an RGB image PP2 in process St1 is supplied to invISP152 for an RGB image PsX, thereby generating a RAW image PRX2 by inverse conversion processing.

[0118] That is, as described above, the same RGB image PsX of normal image quality is subjected to inverse transformation processing based on different parameter sets to generate pseudo RAW images. In this way, a large number of common RGB images of normal image quality are subjected to inverse transformation processing using various parameters to generate a wide variety of RAW images.

[0119] The parameter set employed at this time is composed of values ​​actually generated when the ISP 151 that executes forward image processing is trained in the training process using the training data set of process St1.

[0120] For this reason, the parameter sets used are limited to those that actually exist or are close to them, excluding parameter sets that would not exist in reality, such as those that are set completely randomly without any restrictions. Furthermore, by using various parameter sets for a single RGB image, the inverse transformation process is performed with some degree of restricted disturbance added to the parameter set, making it possible to generate a large number of RAW images that could actually exist from a single RGB image.

[0121] Returning to the description of FIG.

[0122] In process St3, the forward transformation processing unit 132 uses the ISP 151 that has completed learning in process St1 to convert the RAW image DB (Large pseudo RAW DB) 194 generated in the second process St2 into an RGB image DB (Large RGB GT DB) 195 consisting of correct images (GT images) with image quality according to the user's preferences and purpose.

[0123] Furthermore, the forward transformation processing unit 132 may update a plurality of RGB images generated from the same RAW image in the RGB image DB (Large RGB GT DB) 195, which is made up of correct images (GT images) of image quality according to the user's preferences and purpose and generated by the ISP 151, to one that is deemed to have the highest image quality using an IQA (Image Quality Assessment) method or the like.

[0124] Furthermore, in process St4, the learning unit 131 further strengthens and trains the ISP 151 based on a pair of learning datasets: a DB of RAW images (large pseudo RAW DB) 194 added to a DB of RAW images (small RAW DB) 191 of a small number of images according to the user's preferences and use supplied from the user PC 31; and a DB of RGB images (large RGB GT DB) 195 made up of correct images (GT images) of image quality according to the user's preferences and use added to a DB of RGB images (small RGB DB) 192.

[0125] Here, in process St3, as shown in FIG. 12 , in a DB of RGB images (Large RGB GT DB) 195 made up of correct images (GT images) of image quality according to user preferences and use generated by the ISP 151, when a RAW image generated by reverse conversion processing of original image A using each of parameter sets P1 to P3 is converted into RGB images A′, A″, A′″ by forward conversion processing by the ISP 151, the original RGB image A and the RGB images A′, A″, A′″ are updated to the one that is deemed to have the highest image quality using an IQA (Image Quality Assessment) method or the like.

[0126] As a specific example, consider the case where, as shown in FIG. 13, RAW images PR11 and PR12 are generated by inverse transformation of the original RGB image Ps10 using predetermined parameter sets P11 and P12 in process St12.

[0127] In this case, processing St13 performs forward conversion processing on the RAW images PR11 and PR12, thereby generating RGB images Ps11 and Ps12 consisting of correct images (GT images) with image quality that suits the user's preferences and purpose.

[0128] At this time, the forward transformation processing unit 132 selects the image with the highest image quality from the original RGB image Ps10 and the generated RGB images Ps11 and Ps12 using an IQA (Image Quality Assessment) method or the like.

[0129] That is, it is assumed that the original RGB image Ps10 and the generated RGB images Ps11 and Ps12 have both been converted into images according to the user's preferences and applications. Here, by training the ISP 151 using a training data set that pairs the highest quality RGB image with a RAW image, it is possible to train the forward conversion process implemented by the ISP 151 to achieve the highest image quality.

[0130] Therefore, for example, if the RGB image with the highest image quality between the original RGB image Ps10 and the generated RGB images Ps11 and Ps12 is RGB image Ps12, then RGB image Ps12 is set as the correct image (GT image), and RGB images Ps10 and Ps11 are updated to RGB image Ps12.

[0131] Then, in process St4, the RAW image PR11 and the RGB image Ps12 are paired, and the RAW image PR12 and the RGB image Ps12 are paired, and each is used as a learning data set, and the ISP 151 is trained.

[0132] For information on learning the IQA (Image Quality Assessment) method, see RankIQA: Learning From Rankings for No-Reference Image Quality Assessment (thecvf.com). Using this method or similar makes it possible to create an IQA that can determine that an image selected by a user as a correct image is of high image quality without labeling. Also, the ISP 151 may be configured to perform learning using the IQA method each time a learning dataset is generated and learning is performed.

[0133] Furthermore, when multiple RGB images are generated by forward transformation using ISP 151 from a RAW image generated by inversely transforming the original image with multiple parameter sets, an example has been described in which the IQA method is used to select the image with the highest image quality from among these multiple RGB images, but methods other than IQA may also be used; for example, the image with the highest image quality may be selected by using a generation AI and a predetermined index.

[0134] Furthermore, the above process makes it possible to generate a large amount of appropriate training datasets at low cost (easily) based on the relatively large number of RGB images of normal image quality that do not have paired RAW images.

[0135] <Regarding the Noise Addition Unit> In the above, an example has been described in which a noise addition unit 174′ is provided as a corresponding component in the invISP 151′ that performs inverse transform processing, in contrast to the noise removal unit 174 provided in the ISP 151 that performs forward transform processing.

[0136] Originally, the configuration of the noise removal unit 174 should be a noise inverse removal unit, but since it is sometimes difficult to obtain an accurate noise inverse removal unit, a noise addition unit (Inv (Denoiser)) 174' with almost the same function is provided as an alternative.

[0137] 14, the noise addition unit 174′ converts the input image, RAW image PR31, into an RGB image Ps31 by superimposing a noise image PN on it. Note that the type of noise image PN to be superimposed on the RAW image PR31 can be selected using the methods described in Rawgment: Noise-Accounted RAW Augmentation Enables and Recognition in a Wide Variety of Environments (thecvf.com).

[0138] <Configuration example of an ISP that performs forward transform processing on a pixel-by-pixel basis> In the above, a so-called dynamic ISP has been described as an example of the format of the ISP 151, in which parameters are determined on an image-by-image basis in accordance with the feature amounts of the RAW image to dynamically change the content of the forward transform processing.

[0139] However, a pixel-by-pixel Dynamic ISP format may be used in which parameters are determined for each pixel constituting a RAW image, and forward transformation processing is dynamically performed for each pixel.

[0140] FIG. 15 shows an example of the configuration of an ISP 151' in the form of a dynamic ISP that realizes forward conversion processing in units of pixels.

[0141] The ISP 151 ′ in the Dynamic ISP format, which realizes forward transformation processing in pixel units, includes a transformation processing unit 181 , a reduction unit 182 , a feature extraction unit 183 , and a parameter determination unit 184 .

[0142] The reduction unit 182 has the same function as the reduction unit 176 (FIG. 6), and reduces the RAW image that is the input image, and outputs the reduced image to the feature extraction unit 183 .

[0143] The feature extraction unit 183 extracts feature amounts in pixel units from the RAW image supplied from the reduction unit 182 and outputs the extracted feature amounts to the parameter determination unit 184 .

[0144] The parameter determination unit 184 sets parameters according to the feature amount in pixel units, and supplies, for example, a filter based on the set parameters to the conversion processing unit 181 .

[0145] The conversion processing unit 181 performs filtering on a pixel-by-pixel basis using a filter according to parameters supplied from the parameter determination unit 184, thereby achieving forward conversion processing and converting the RAW image into an RGB image and outputting it.

[0146] FIG. 15 shows an example of a 3×3 filter, with the values ​​being 1.5, 1.6, and 1.4 from the upper left to the right, 0.7, 0.9, and 1.8 from the middle left to the right, and 0.8, 1.1, and 1.8 from the lower left to the right.

[0147] In the ISP 151 ′ of FIG. 15 as well, the parameters set on a pixel-by-pixel basis in the parameter determination unit 184 are acquired by the parameter set acquisition unit 134 and registered as a parameter set group 153 .

[0148] In this case, the invISP 152 also applies parameters set in pixel units in the forward transform process, and implements the inverse transform process by filtering.

[0149] The format of the ISP 151 may be a format other than Dynamic ISP, and for example, a model-based Image Adaptive Look Up Table format enhancement such as Image-Adaptive-3DLUT, which learns a 3D Look Up Table and converts an image, may be applied.

[0150] Furthermore, a model-based reversible ISP such as SRISP (Self-Supervised Reversed Image Signal Processing) or MBISPLD (Model-Based Image Signal Processors) may be applied as the format of the ISP 151.

[0151] Furthermore, while the forward transformation process has been described as being based on an ISP, it may also be applied to general image transformation processes based on a DNN (Deep Neural Network). Even in the case of a DNN, the forward transformation process and the inverse transformation process are generated separately, and by adding appropriate disturbances in the inverse transformation process, multiple RAW images are generated for each RGB image, thereby generating a training dataset.

[0152] In the above, an example has been described in which the process of converting a RAW image to an RGB image is considered to be a forward conversion process, and the process of converting an RGB image to a RAW image is considered to be a reverse conversion process. However, other conversions may also be applied, such as a task in which the process of converting an HSV (Hue, Saturation, Value) image to an RGB image is considered to be a forward conversion process, and the process of converting an RGB image to an HSV image is considered to be a reverse conversion process, or an enhancement or retouching task in which an RGB image is converted to an RGB image of higher image quality.

[0153] In either process, when generating multiple RAW images or HSV images from an RGB image using the inverse transformation process, an appropriate disturbance is applied based on the parameters generated in the forward transformation process, thereby generating a training dataset while suppressing conversion to RAW images or HSV images using parameters that do not actually exist.

[0154] <<3. Application Example>> In the above, an example has been described that assumes that the user of the user PC 31 is an expert in image processing and is able to prepare even a small amount of training data set consisting of pairs of RAW images and RGB images according to the user's preferences and purposes.

[0155] However, if the user of the user PC 31 is not an expert and has already received the ISP 31a but wants to change it to suit new preferences or applications, it is not possible to update the ISP 151 because it is not possible to prepare a learning dataset consisting of pairs of RAW images and RGB images that suit the user's preferences or applications.

[0156] Therefore, when the user is a general user who is not an expert and has already received ISP 31a but wants to update it to suit new preferences or uses, the user PC 31 prepares only new RGB images that suit the user's preferences and uses and supplies them to the ISP learning server 32, and the ISP learning server 32 generates corresponding RAW images from the RGB images that suit the user's preferences and uses, and trains the ISP 151 from the learning dataset of the generated RGB images and RAW images.

[0157] Figure 16 shows an example application of an image processing system in which the ISP learning server 32 acquires RGB images that correspond to the preferences and uses of a new user supplied from the user PC 31, generates corresponding RAW images from the acquired RGB images that correspond to the preferences and uses of the new user, and trains (updates) the ISP 151 using a learning dataset of the generated RGB images and RAW images.

[0158] The image processing system 11' in Fig. 16 is composed of a user PC 31, an ISP learning server 32', and an image storage server 33. In the image processing system 11' in Fig. 16, components corresponding to those in the image processing system 11 in Fig. 4 are given the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0159] The image processing system 11' in Figure 16 differs from the image processing system 11 in Figure 4 in that the user PC 31 supplies only RGB images that will be the correct images (GT images) according to the user's preferences and purpose to the ISP learning server 32, and an ISP learning server 32' is provided instead of the ISP learning server 32.

[0160] The ISP learning server 32' has the same basic functions as the ISP learning server 32, but in addition, it generates a corresponding RAW image from an RGB image that serves as a correct image (GT image) according to the user's preferences and purpose, which is supplied from the user PC 31, generates a learning dataset, and then learns (updates) the ISP 151.

[0161] <Process for generating a corresponding RAW image from an RGB image that will be a GT according to a new user's preferences and purpose, and generating a training dataset> First, in the process for generating training data, the ISP 151 and invISP 152 are used that have been trained in advance using another dataset. That is, before the user uses them, they are trained in advance using the ISP training server 32, or an ISP 151 and invISP 152 with general image quality are prepared using a pair dataset of an input RAW image and an output sRGB image of an existing camera ISP.

[0162] The functional configuration realized by the ISP learning server 32′ is the same as that of the ISP learning server 32. Here, with reference to Fig. 17, a process will be described in which a corresponding RAW image is generated from an RGB image that serves as a correct image (GT image) according to the preferences and purpose of a new user, and a learning dataset is generated.

[0163] That is, as shown in FIG. 17, when an RGB image Ps (new RGB image) 31 that meets the preferences and use of a new user is acquired from a user PC 31, in process St31, the inverse conversion processing unit 133 controls the invISP 152 to perform an inverse conversion process on the RGB image Ps31 that meets the preferences and use of the new user, thereby converting it into a RAW image (a RAW image that does not resemble the original) PR31.

[0164] Since the invISP 152 converts the RGB image that has been subjected to forward conversion processing by the previous ISP 151 into a RAW image, the RAW image PR31 into which the RGB image Ps31 according to the new user's preferences and use is converted is a RAW image that does not resemble the original RAW image corresponding to the RGB image according to the new user's preferences and use.

[0165] In processing St32, the forward conversion processing unit 132 controls the ISP 151 to perform forward conversion processing on the RAW image PR31, thereby converting it into an RGB image Ps32 (an RGB image close to the GT image of the current ISP).

[0166] That is, the ISP 151 performs forward conversion processing to convert the input RAW image into an RGB image that conforms to the previous user's preferences and purpose, and therefore the RGB image Ps32 is converted into an RGB image that is closer to the GT of the current ISP than the RGB image Ps31.

[0167] In processing St33, the inverse transformation processing unit 133 controls the invISP 152 to perform inverse transformation processing on the RGB image Ps32 to convert it into a RAW image (RAW DB similar to the GT image of the current ISP) PR32.

[0168] That is, since the RGB image Ps32 is an RGB image that is converted from the RGB image Ps31 into an RGB image that is closer to the GT of the current ISP, the RAW image PR32 that is the result of the reverse conversion process of the RGB image Ps32 is converted into a RAW image that is closer to the GT image of the current ISP 151 than the RAW image PR31.

[0169] In processing St34, the forward conversion processing unit 132 controls the ISP 151 to perform forward conversion processing on the RAW image PR32, thereby converting it into an RGB image (a GT image of approximately the current ISP) Ps33.

[0170] That is, since the RAW image PR32 has been converted into a RAW image closer to the GT image of the current ISP 151 than the RAW image PR31, the RGB image Ps33 obtained by forward conversion processing of the RAW image PR32 has been further converted into an RGB image closer to the GT image of the current ISP 151.

[0171] In processing St35, the inverse transformation processing unit 133 controls the invISP 152 to perform inverse transformation processing on the RGB image Ps33 to convert it into a RAW image (a RAW image corresponding to a GT image of the current ISP) PR33.

[0172] That is, since the RGB image Ps33 is converted into a GT image of the current ISP, the RAW image PR33 obtained by performing the reverse conversion process on the RGB image Ps33 is converted into a RAW image that is almost similar to the GT image of the current ISP 151.

[0173] In this way, by the processes from step St31 to step St35, the RGB image Ps31 according to the new user's preference and purpose is converted into an RGB image Ps33 consisting essentially of the current GT image of the ISP 151.

[0174] In other words, by repeating the reverse conversion process and forward conversion process by the current ISP 151 and the invISP 152, an RGB image Ps33 is generated that corresponds to the RGB image Ps31 and serves as the GT image of the current ISP 151. Furthermore, by converting the RGB image Ps33 by performing the reverse conversion process by the current invISP 152, an appropriate RAW image PR33 that corresponds to the RGB image Ps31 in accordance with the new user's preferences and purpose is generated.

[0175] Therefore, in process St36, the learning unit 131 learns the ISP 151 using a learning dataset consisting of a pair of an RGB image Ps31 that corresponds to the preferences and purpose of the new user and a corresponding appropriate RAW image PR33, thereby making it possible to update the ISP 151 to one that corresponds to the preferences and purpose of the new user.

[0176] 17 has been described as an example in which the ISP 151 is trained using a training data set in which a RAW image PR33 obtained after the inverse transformation process and the forward transformation process have been repeated three times through steps St31 to St35 is paired with a new RGB image Ps31 that meets the user's preferences and application. However, by further training the ISP 151 using a training data set in which a RAW image obtained after the inverse transformation process and the forward transformation process have been repeated is paired with a new RGB image Ps31 that meets the user's preferences and application, it is possible to further improve the image quality of the ISP 151.

[0177] <<4. Example of Execution by Software>> The above-described series of processes can be executed by hardware, but can also be executed by software. When the series of processes is executed by software, the program that constitutes the software is installed from a recording medium into a computer that is built into dedicated hardware, or into, for example, a general-purpose computer that can execute various functions by installing various programs.

[0178] 18 shows an example of the configuration of a general-purpose computer. This computer has a built-in CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

[0179] The input / output interface 1005 is connected to an input unit 1006 including input devices such as a keyboard and a mouse through which a user inputs operation commands, an output unit 1007 that outputs a processing operation screen and images of processing results to a display device, a storage unit 1008 including a hard disk drive or the like that stores programs and various data, and a communication unit 1009 including a LAN (Local Area Network) adapter or the like that executes communication processing via a network typified by the Internet. Also connected is a drive 1010 that reads and writes data from / to a removable storage medium 1011 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0180] The CPU 1001 executes various processes in accordance with a program stored in a ROM 1002 or a program read from a removable storage medium 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, installed in a storage unit 1008, and loaded from the storage unit 1008 into a RAM 1003. The RAM 1003 also stores data necessary for the CPU 1001 to execute various processes as appropriate.

[0181] In a computer configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0182] The program executed by the computer (CPU 1001) can be provided by being recorded on a removable storage medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0183] In a computer, a program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting a removable storage medium 1011 into the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be installed in advance in the ROM 1002 or the storage unit 1008.

[0184] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0185] 18. The CPU 1001 in FIG. 18 realizes the functions of the control unit 101 in FIG.

[0186] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device with multiple modules housed in a single housing, are both systems.

[0187] Furthermore, the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure.

[0188] For example, the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0189] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0190] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0191] The present disclosure may also be configured as follows: <1> An image processing device comprising: a forward transformer that converts a first image into a second image having predetermined characteristics by arithmetic processing in which parameters set based on the first image are applied to a predetermined function; and an inverse transformer that converts the second image into the first image by arithmetic processing in which the parameters are applied to an inverse function of the predetermined function, wherein the inverse transformer converts the second image not having the predetermined characteristics into a pseudo first image by arithmetic processing in which the parameters are applied to the inverse function of the predetermined function and the predetermined disturbance is added, and the forward transformer converts the pseudo first image into a second image having the predetermined characteristics. <2> The image processing device according to <1>, further including a parameter storage unit that acquires and stores a plurality of parameters set for each of the plurality of first images by the forward transformer, wherein the inverse transformer performs arithmetic processing with the predetermined disturbance added by applying the plurality of parameters to an inverse function of the predetermined function to convert a second image that does not have the predetermined characteristic into a plurality of pseudo first images, and the forward transformer converts the plurality of pseudo first images into a plurality of second images that have the predetermined characteristic. <3> The image processing device according to <2>, further including a learning unit that trains the forward transformer based on a training dataset that pairs the plurality of pseudo first images with second images that have the predetermined characteristic and are converted from the plurality of pseudo first images by the forward transformer. <4> The image processing device described in <3>, wherein the forward transformer is trained in advance based on a predetermined number of training datasets each consisting of a pair of a second image having the predetermined characteristic and a first image corresponding to the second image, and the training unit trains the forward transformer based on a training dataset that pairs the plurality of pseudo first images with a second image having the predetermined characteristic that is converted from the plurality of pseudo first images by the forward transformer, in addition to the predetermined number of training datasets.<5> The image processing device according to <3>, wherein the second image with the highest image quality is selected from a plurality of second images having the predetermined characteristics and which are transformation results of the plurality of pseudo first images, and the learning unit trains the forward transformer based on the training dataset that pairs the plurality of pseudo first images with the second image with the highest image quality from a plurality of second images having the predetermined characteristics. <6> The image processing device according to <5>, wherein the second image with the highest image quality is selected from a plurality of second images having the predetermined characteristics and which are transformation results of the plurality of pseudo first images using an IQA (Image Quality Assessment) method. <7> The image processing device according to <3>, wherein, after the inverse transformer has transformed a second image having characteristics different from the predetermined characteristics into a pseudo first image and the forward transformer has transformed the pseudo first image into a second image having the predetermined characteristics, the inverse transformer transforms the second image having characteristics different from the predetermined characteristics into the pseudo first image, and the learning unit trains the forward transformer based on a training dataset that pairs the second image having characteristics different from the predetermined characteristics with the pseudo first image transformed after the repeated processing. <8> The image processing device according to <1>, further including a parameter determination unit that determines the parameters on an image-by-image basis based on the first image. <9> The image processing device according to <1>, further including a parameter determination unit that determines the parameters on a pixel-by-pixel basis based on the first image.<10> The image processing device according to <2>, wherein the forward converter has a configuration in which a plurality of forward individual converters of different types are connected in series, and the plurality of forward individual converters forward convert the first image to the second image by an arithmetic process in which parameters set for each of the plurality of forward individual converters are applied to a function corresponding to the type based on the first image, and the backward converter has a configuration in which a plurality of backward individual converters corresponding to the plurality of forward individual converters are connected in series in an opposite direction to the forward converter, and the plurality of backward individual converters reversely convert the second image to the first image by an arithmetic process in which the parameters are applied to a function corresponding to the type and an inverse function. <11> The image processing device according to <10>, wherein the plurality of forward individual converters include a color correction unit, a white balance adjustment unit, a gain adjustment unit, a noise removal unit, and a gradation conversion unit, and the plurality of backward individual converters include a color reverse correction unit, a white balance inverse adjustment unit, a gain inverse adjustment unit, a noise addition unit, and a gradation inverse conversion unit. <12> The image processing device according to <1>, wherein the first image is a RAW image and the second image is an RGB image. <13> The image processing device according to <1>, wherein the first image is an HSV image and the second image is an RGB image. <14> The image processing device according to <1>, wherein the first image is an RGB image and the second image is an RGB image of higher image quality than the first image. <15> The image processing device according to <1>, wherein the predetermined characteristics are characteristics according to a user's preference and a characteristic according to the user's application.<16> An image processing method comprising: a forward transformation process that transforms a first image into a second image having predetermined characteristics by an arithmetic process in which parameters set based on a first image are applied to a predetermined function; and an inverse transformation process that transforms the second image into the first image by an arithmetic process in which the parameters are applied to an inverse function of the predetermined function, wherein the inverse transformation process transforms the second image not having the predetermined characteristics into a pseudo first image by an arithmetic process in which the parameters are applied to the inverse function of the predetermined function and the predetermined disturbance is added, and the forward transformation process transforms the pseudo first image into the second image having the predetermined characteristics. <17> A program that causes a computer to function as: a forward transformer that converts a first image into a second image having predetermined characteristics by an arithmetic process in which parameters set based on a first image are applied to a predetermined function; and an inverse transformer that converts the second image into the first image by an arithmetic process in which the parameters are applied to an inverse function of the predetermined function, wherein the inverse transformer converts the second image not having the predetermined characteristics into a pseudo first image by an arithmetic process in which the parameters are applied to the inverse function of the predetermined function and the predetermined disturbance is added; and the forward transformer converts the pseudo first image into the second image having the predetermined characteristics.

[0192] 11, 11' Image processing system, 31 User PC, 32, 32' ISP learning server, 33 Image storage server, 131 Learning unit, 132 Forward transformation processing unit, 133 Inverse transformation processing unit, 134 Parameter set acquisition unit, 151, 151' ISP, 152 invISP, 153 Parameter set group, 171 Color correction unit, 171' Color inverse correction unit, 172 White balance adjustment unit, 172' White balance inverse adjustment unit, 173 Gain adjustment unit, 173' Gain inverse adjustment unit, 174 Noise removal unit, 174' Noise addition unit, 175 Gradation conversion unit, 175' Gradation inverse conversion unit, 176 Reduction unit, 177 Feature extraction unit, 178 Parameter determination unit, 181 Conversion processing unit, 182 Reduction unit, 183 feature extraction unit, 184 parameter determination unit

Claims

1. An image processing device comprising: a forward transformer that converts a first image into a second image having predetermined characteristics by applying parameters set based on the first image to a predetermined function; and an inverse transformer that converts the second image into the first image by applying the parameters to the inverse function of the predetermined function, wherein the inverse transformer converts the second image, which does not have the predetermined characteristics, into a pseudo-first image by applying the parameters to the inverse function of the predetermined function and adding the predetermined disturbance; and the forward transformer converts the pseudo-first image into a second image having the predetermined characteristics.

2. An image processing device as described in claim 1, further comprising a parameter storage unit that acquires and stores a plurality of parameters set for each of the plurality of first images by the forward transformer, wherein the inverse transformer performs a calculation process that adds the predetermined disturbance by applying the plurality of parameters to an inverse function of the predetermined function, thereby converting a second image that does not have the predetermined characteristics into a plurality of pseudo first images, and the forward transformer converts the plurality of pseudo first images into a plurality of second images that have the predetermined characteristics.

3. The image processing device according to claim 2, further comprising a training unit that trains the forward transformer based on a training dataset that pairs the plurality of pseudo first images with second images having the predetermined characteristics that are converted from the plurality of pseudo first images by the forward transformer.

4. The image processing device described in claim 3, wherein the forward transformer is trained in advance based on a predetermined number of training datasets consisting of pairs of second images having the predetermined characteristics and first images corresponding to the second images, and the training unit trains the forward transformer based on a training dataset that pairs the plurality of pseudo first images with second images having the predetermined characteristics converted from the plurality of pseudo first images by the forward transformer, in addition to the predetermined number of training datasets.

5. The image processing device described in claim 3, wherein the second image with the highest image quality is selected from a plurality of second images having the specified characteristics that are the transformation results of the plurality of pseudo first images, and the learning unit trains the forward transformer based on the learning dataset that pairs the plurality of pseudo first images with the second image with the highest image quality from a plurality of second images having the specified characteristics.

6. The image processing device according to claim 5, wherein the second image with the highest image quality is selected from the plurality of second images having the predetermined characteristics that are the conversion results of the plurality of pseudo first images using an IQA (Image Quality Assessment) method.

7. The image processing device of claim 3, wherein, after the inverse transformer has transformed a second image having characteristics different from the predetermined characteristics into a pseudo first image and the forward transformer has transformed the pseudo first image into a second image having the predetermined characteristics, the learning unit trains the forward transformer based on a training dataset that pairs the second image having characteristics different from the predetermined characteristics with the pseudo first image transformed after the process has been repeated.

8. The image processing device according to claim 1, further comprising a parameter determination unit that determines the parameters for each image based on the first image.

9. The image processing device according to claim 1, further comprising a parameter determination unit that determines the parameters on a pixel-by-pixel basis based on the first image.

10. The image processing device of claim 2, wherein the forward transformer is configured to have a plurality of forward individual transformers of different types connected in series, and the plurality of forward individual transformers convert the first image into the second image in a forward direction by an arithmetic process in which parameters set for each of the plurality of forward individual transformers are applied to a function corresponding to the type based on the first image; and the backward transformer is configured to have a plurality of backward individual transformers corresponding to the plurality of forward individual transformers connected in series in a reverse direction to the forward transformer, and the plurality of backward individual transformers convert the second image in a reverse direction to the first image by an arithmetic process in which the parameters are applied to a function corresponding to the type and an inverse function corresponding to the function.

11. The image processing device according to claim 10, wherein the plurality of forward individual converters include a color correction unit, a white balance adjustment unit, a gain adjustment unit, a noise removal unit, and a gradation conversion unit, and the plurality of reverse individual converters include a color inverse correction unit, a white balance inverse adjustment unit, a gain inverse adjustment unit, a noise addition unit, and a gradation inverse conversion unit.

12. The image processing device according to claim 1, wherein the first image is a RAW image and the second image is an RGB image.

13. The image processing device according to claim 1, wherein the first image is an HSV image and the second image is an RGB image.

14. The image processing device according to claim 1, wherein the first image is an RGB image, and the second image is an RGB image of higher image quality than the first image.

15. The image processing device according to claim 1, wherein the predetermined characteristics are characteristics according to the user's preferences and characteristics according to the user's intended use.

16. An image processing method comprising: a forward transformation process for transforming a first image into a second image having predetermined characteristics by applying parameters set based on a first image to a predetermined function; and an inverse transformation process for transforming the second image into the first image by applying the parameters to an inverse function of the predetermined function, wherein the inverse transformation process transforms the second image, which does not have the predetermined characteristics, into a pseudo-first image by applying the parameters to the inverse function of the predetermined function and adding the predetermined disturbance; and the forward transformation process transforms the pseudo-first image into a second image having the predetermined characteristics.

17. A program that causes a computer to function as a forward transformer that converts a first image into a second image having predetermined characteristics by applying parameters set based on the first image to a predetermined function, and an inverse transformer that converts the second image into the first image by applying the parameters to the inverse function of the predetermined function, wherein the inverse transformer converts the second image, which does not have the predetermined characteristics, into a pseudo-first image by applying the parameters to the inverse function of the predetermined function and adding the predetermined disturbance, and the forward transformer converts the pseudo-first image into a second image having the predetermined characteristics.

Images