Image processing device, control method, and program

Abstract

To achieve both size reduction and accuracy maintenance of conversion information used when generating images with different dynamic ranges that are attached to resized images.SOLUTION: An image processing device includes first generation means for generating first conversion information used when generating an image with a different dynamic range from a first image, image resizing processing means for performing resizing processing on the first image, and control means for controlling whether to perform resizing processing on the first conversion information attached to the first image when resizing processing is performed on the first image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to image resizing. [Background technology]

[0002] Conventionally, high dynamic range (HDR) images and standard dynamic range (SDR) images have been mutually converted using conversion information called a gain map so that the images fit the dynamic range that a display device can display (Patent Document 1). The gain map is generated from a RAW image (main image) and stored in an image file. Patent Document 2 describes a method for reducing the processing load when generating a gain map by reducing the RAW image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2018-530281 [Patent Document 2] Japanese Patent Application Laid-Open No. 2007-180851 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 2, by storing a gain map attached to the main image in an image file, an HDR image can be converted into an SDR image and displayed on a display device compatible with SDR.

[0005] However, Patent Document 2 does not mention how to handle a gain map when a main image with an attached gain map is resized (enlarged or reduced). If the main image is resized and the gain map is recorded without being resized, the size of the gain map may become unnecessarily large or the accuracy of conversion may decrease.

[0006] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to realize a technology that achieves both size reduction and accuracy maintenance of conversion information used when generating images with different dynamic ranges that are attached to resized images. [Means for solving the problem]

[0007] In order to solve the above problems and achieve the object, the image processing device of the present invention has a first generation means for generating first conversion information used when generating an image with a different dynamic range from a first image, an image resizing processing means for performing resizing processing on the first image, and a control means for controlling whether or not to perform resizing processing on the first conversion information attached to the first image when resizing processing is performed on the first image. [Effects of the Invention]

[0008] According to the present invention, it is possible to reduce the size of conversion information that is attached to a resized image and that is used when generating an image with a different dynamic range, while maintaining the accuracy of the information. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating a hardware configuration of an image capturing apparatus to which an image processing apparatus according to an embodiment of the present invention is applied. [Figure 2] FIG. 2 is a block diagram illustrating the functional configuration of an image processing unit of the present embodiment. [Figure 3] 5A to 5C are diagrams for explaining a method of generating a gain map according to the present embodiment. [Figure 4] 5A to 5C are diagrams illustrating an example of region integration processing of a gain map according to the present embodiment. [Figure 5] 5A and 5B are diagrams illustrating an example of region division processing of a gain map according to the present embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of the data structure of a main image to which a gain map according to the present embodiment is added. [Figure 7] 4 is a flowchart illustrating a control process during shooting according to the first embodiment. [Figure 8] 10 is a flowchart illustrating a control process during image reduction according to the second embodiment. [Figure 9] 11A to 11C are diagrams for explaining a gain map generation process during image enlargement according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] In this embodiment, an example is described in which the image processing device of the present invention is applied to an imaging device such as a digital camera, a gain map is created from an original RAW image captured by the digital camera, and the gain map is resized when the original RAW image is resized (enlarged or reduced).

[0012] The imaging device of this embodiment is not limited to a digital camera, but may also be a personal computer (notebook PC or tablet PC), a smartphone, a web camera such as a surveillance camera, a medical camera, or the like.

[0013] <Device configuration> First, the configuration and functions of an image capturing apparatus 100 according to this embodiment will be described with reference to FIGS.

[0014] Fig. 1 is a block diagram illustrating the hardware configuration of an image capture device 100 according to this embodiment. Fig. 2 is a block diagram illustrating the functional configuration of an image processing unit 104 of the image capture device 100 according to this embodiment.

[0015] The imaging device 100 includes an optical unit 101, an imaging unit 102, an A / D conversion unit 103, an image processing unit 104, a display unit 105, a storage unit 106, a recording medium 107, a system control unit 108, and an operation unit 109.

[0016] The optical unit 101 includes a group of lenses including a zoom lens and a focus lens, and a shutter with an aperture function. The optical unit 101 adjusts the magnification, focus position, and light amount of the subject image. The optical unit 101 adjusts the magnification, focus state, and light amount of the subject image that reaches the imaging unit 102, and forms an image on the imaging surface of the imaging unit 102.

[0017] The imaging unit 102 includes an image sensor configured with a CCD, CMOS, etc. that converts the optical image of the subject formed by the optical unit 101 into an electrical signal. The imaging unit 102 generates still image data and video data configured with analog signals.

[0018] The A / D conversion unit 103 converts the analog signal generated by the imaging unit 102 into a digital signal. The A / D conversion unit 103 generates still image data or video data composed of a digital signal from still image data or video data composed of an analog signal.

[0019] The image processing unit 104 performs various image processing on the image data output from the A / D conversion unit 103. The various image processing includes, for example, development processing such as pixel interpolation, gamma conversion processing for generating a gain map, and color matrix processing. The image processing unit 104 generates image files by compressing and encoding the processed still image data in a format such as JPEG, or by encoding the moving image data in a moving image compression format such as MP4, and records the generated image files on the recording medium 107. The image processing unit 104 also decodes still image files read from the recording medium 107, or decodes moving image files read from the recording medium 107. The image processing unit 104 can perform similar image processing not only on the image data output from the A / D conversion unit 103, but also on image data read from the recording medium 107. The detailed configuration and functions of the image processing unit 104 will be described later with reference to FIG. 2.

[0020] The display unit 105 displays image data (live view) captured by the imaging unit 102, image data read from the recording medium 107, and a GUI (Graphical User Interface) for interactive operation. The display unit 105 includes a display device such as a liquid crystal display or an organic EL display. The display unit 105 includes a display device whose representable brightness range (dynamic range) conforms to SDR (Standard Dynamic Range). The image processing unit 104 converts an HDR (High Dynamic Range) image, which is the main image read from the recording medium 107, into an SDR image by applying a gain map attached to the HDR image. The display unit 105 displays an SDR image converted from the HDR image by applying the gain map in the image processing unit 104. This reduces the cases where an HDR image is displayed on the display unit 105 that is not HDR-compatible, and prevents an image from being displayed with image quality that the user does not expect.

[0021] The storage unit 106 is a volatile memory that stores various information such as an image processing program and a gain map required for image processing by the image processing unit 104, and stores image data during image processing.

[0022] The recording medium 107 is a non-volatile memory that records image files with gain maps. The recording medium 107 is, for example, a memory card or a hard disk, and is built into the imaging device 100 or is detachable.

[0023] The system control unit 108 includes a processor (CPU) that performs arithmetic processing and control processing of the image capture device 100, a volatile memory (ROM) that stores programs executed by the processor, and a work memory (RAM) into which programs read from the nonvolatile memory and constants and variables for executing the programs are loaded. The system control unit 108 controls each component of the image capture device 100 by loading the programs stored in the ROM into the RAM and executing them.

[0024] The operation unit 109 is an operation member such as a switch, button, or touch panel that receives various operations from the user and notifies the system control unit 108. The operation unit 109 includes at least a still image capture button, a video capture button, a mode dial, and a power switch.

[0025] The still image capture button is an operation member for instructing the system control unit 108 to perform still image capture processing. The video capture button is an operation member for instructing the system control unit 108 to perform video capture processing.

[0026] The mode dial is an operating member for switching the operation mode of the imaging device 100. The mode dial can switch the operation mode of the imaging device 100 between a still image capture mode, a video capture mode, and a playback mode.

[0027] The power switch is an operating member for switching the power of the imaging device 100 on and off.

[0028] When the still image shooting button is pressed halfway, it sends an instruction for shooting preparation processing to the system control unit 108. In still image shooting mode, when the still image shooting button is pressed halfway, the system control unit 108 starts shooting preparation processing (AE processing and AF processing) for a still image. When the still image shooting button is pressed all the way, it sends an instruction for shooting processing to the system control unit 108. When the still image shooting button is pressed all the way, the system control unit 108 executes still image shooting processing to record image data captured by the imaging unit 102 on the recording medium 107.

[0029] In addition, in the video shooting mode, when the video shooting button is pressed for the first time, the system control unit 108 performs shooting preparation processing (AE processing and AF processing) on ​​the image data (frames) captured by the imaging unit 102, continues the video shooting processing to record a video for a predetermined period of time on the recording medium 107, and stops the video shooting processing when the video shooting button is pressed again.

[0030] 1, the optical unit 101 is configured as a part of the imaging device 100 that includes the imaging unit 102, but the configuration is not limited to this. For example, an interchangeable optical unit (interchangeable lens) may be configured to be detachable from the imaging device 100, like a single-lens reflex camera.

[0031] FIG. 2 is a block diagram illustrating the functional configuration of the image processing unit 104 of this embodiment.

[0032] The image (RAW image) input to the image processing unit 104 is data generated by converting the analog image signal generated by the imaging unit 102 into a digital image signal by the A / D conversion unit 103, and is a Bayer image consisting of three components: red (R), green (G), and blue (B).

[0033] The development processing unit 211 generates an image in a predetermined format such as YUV422 by performing various image processing (development processing) on ​​the RAW image input to the image processing unit 104. The development processing unit 211 includes a white balance processing unit 205, a color matrix processing unit 206, and a gamma processing unit 207.

[0034] The white balance processing unit 205 calculates a white balance gain based on the RAW image input to the image processing unit 104, and adjusts the white balance by applying the gain to the signal values ​​of each red (R), green (G), and blue (B) pixel.

[0035] The color matrix processing unit 206 performs color matrix processing to convert the color gamut of the image data output from the white balance processing unit 205. The color matrix processing can change conversion coefficients depending on the spectral characteristics of the image sensor of the imaging unit 102, the color gamut of the output image such as BT.601 or BT.2020, the target value of color reproduction, and the like.

[0036] The gamma processing unit 207 performs gamma processing on the image data output from the color matrix processing unit 206, converting signal values ​​according to gamma characteristics (Opto-Electronic Transfer Function: OETF) for generating image signals that match the gamma characteristics of the output destination. For example, to generate an SDR image, gamma processing based on the gamma of the sRGB standard is performed, and to generate an HDR image, gamma processing based on the gamma characteristics of the OETF specified in ITU-R BT.2100 is performed.

[0037] The image resizing unit 208 resizes (reduces or enlarges) a RAW image input to the image processing unit 104 or a YUV image output from the development processing unit 211. The image resizing unit 208 generates a resized image using a known method such as bilinear interpolation or bicubic interpolation. Note that the enlargement processing may be performed as super-resolution processing using machine learning such as deep learning.

[0038] The gain map generation unit 220 generates a gain map based on a RAW image input to the image processing unit 104. The gain map is conversion information used to correct the signal value of each pixel of the main image to generate an image with different brightness. The gain map has a data structure in which gain amounts for correcting the signal value of each pixel are arranged two-dimensionally corresponding to each pixel. In this embodiment, an example of a gain map for converting an HDR image into an SDR image will be described.

[0039] FIG. 3 is a diagram illustrating a method for generating a gain map for converting an HDR image into an SDR image.

[0040] 3(a) illustrates the relationship between the signal value and display luminance of an output image when an SDR image and an HDR image (output image) output from the imaging device 100 are displayed on an HDR-compatible display device. Dashed line 304 illustrates the relationship between the signal value and display luminance of an SDR image. Dashed line 305 illustrates the relationship between the signal value and display luminance of an HDR image. The display luminance is set to a characteristic such that when the signal value of the output image is small, the display luminance of the SDR image and the HDR image matches, and as the signal value increases, the display luminance becomes different. By using such a display luminance characteristic, even when HDR and SDR images are mixed, the images can be displayed without causing discomfort to the viewer.

[0041] To achieve the display luminance characteristics shown in FIG. 3( a), gamma characteristics based on the EOTF (Electro-Optical Transfer Function) of an HDR-compatible display device may be set in the gamma processing unit 207. FIG. 3( b) illustrates gamma characteristics set in the gamma processing unit 207 when generating an SDR image and an HDR image. Gamma characteristics 301 are gamma characteristics based on the sRGB standard. Gamma characteristics 302 are SDR gammas set in the gamma processing unit 207 when generating an SDR image in the image capture device 100. The SDR gamma 302 has higher contrast than the gamma characteristics 301 based on the sRGB standard, and the gradation is adjusted to improve the appearance of images captured by the image capture device 100. Gamma characteristics 303 are HDR gammas set in the gamma processing unit 207 when generating an HDR image in the image capture device 100.

[0042] Figure 3(c) shows an example of the ratio of the signal value and display brightness of the HDR image and SDR image in Figure 3(a). Figure 3(d) shows an example of the ratio of Figure 3(c) converted from the gamma characteristic in Figure 3(b) into the relationship between the signal value of the RAW image and the display brightness of the HDR image and SDR image.

[0043] The base gain map generation unit 200 can calculate the gain amount for converting an HDR image (main image) into an SDR image from the input RAW image by storing gain information for the signal values ​​of the RAW image shown in FIG. 3(d) in the storage unit 106. The gain amount generated for each pixel of the RAW image or for each set of pixels (e.g., 2×2 pixels) is called a base gain map. Note that the base gain map is not limited to information for converting an HDR image into an SDR image, but may also be information for converting an SDR image into an HDR image, or information for converting an HDR image into an HDR image having a different dynamic range.

[0044] The gain map resizing unit 201 performs resizing (reduction or enlargement) to change the resolution of the entire area of ​​the base gain map generated by the base gain map generating unit 200. The resizing method may be a known method such as bilinear interpolation or bicubic interpolation, or may be resizing based on machine learning such as deep learning.

[0045] The gain map region integrating unit 202 integrates maps of adjacent regions (gain map region integration processing) to reduce the data volume of the gain map generated by the base gain map generating unit 200 or the gain map resizing processing unit 201. FIG. 4 illustrates an example of region integration processing of a gain map performed by the gain map region integrating unit 202. FIG. 4(a) illustrates an example of a gain map generated by the base gain map generating unit 200 or the gain map resizing processing unit 201. For example, the data volume of the gain map is reduced by consolidating the gain map of FIG. 4(a) into data of 2×2 regions as shown in FIG. 4(b). Note that while the example of FIG. 4 illustrates a case where the gains of adjacent regions are the same, gains may be integrated when the gains within a region can be considered to be the same, such as when all regions surrounding a region of interest have the same gain. Furthermore, a resolution other than 1×1 or 2×2 may be used.

[0046] 5 shows an example of region division of a gain map by the gain map region division unit 203. The gain map region division unit 203 performs region division on a gain map having regions that have already been integrated by the gain map region integration unit 202 or the like, to partially or entirely increase the resolution of the gain map.

[0047] The gain map encoding unit 204 performs processing to reduce the data amount of the gain map output from the gain map region integrating unit 202. For example, the data amount of the gain map may be reduced by taking the logarithm of the gain map to reduce the bit depth of the gain map, or by performing lossless or lossy compression. Alternatively, the base gain map generated by the base gain map generating unit 200 may be input directly to the gain map encoding unit 204.

[0048] The image comparison unit 212 calculates the difference between the two images for each pixel or each region. Based on the difference information calculated by the image comparison unit 212, the gain map region integration unit 202 performs gain map region integration processing, or the gain map region division unit 203 performs gain map region division processing.

[0049] The file storage unit 210 generates a file that combines the main image generated by the development processing unit 211, sub-images (reduced images and enlarged images) with a different resolution from the main image generated by the image resizing processing unit 208, and gain maps of the main image and sub-images. The file format is specified by the CIPA DC-007 Multi-Picture Format (MPF), a standard of the Camera and Imaging Products Association (CIPA).

[0050] FIG. 6 illustrates an example of the data structure of a main image to which a gain map of this embodiment is attached.

[0051] FIG. 6(a) shows an example of the data structure of a RAW image file.

[0052] The RAW image file includes a main image RAW, a reduced RAW resulting from resizing the main image RAW by the image resizing processor 208, a monitor display JPEG used for display on a display device, and a thumbnail JPEG used for multi-display when multiple images are displayed simultaneously. Gain maps corresponding to each image (main image gain map, reduced RAW gain map, monitor display JPEG gain map, thumbnail gain map) are also recorded in the RAW image file. Images other than the main image, such as the reduced RAW, monitor display JPEG, and thumbnail JPEG, are called sub-images, and gain maps other than the main image gain map are called sub-image gain maps. The number of gain maps may be reduced by using both the main image gain map and the reduced RAW gain map, or by using both the reduced RAW gain map and the monitor display JPEG gain map. Furthermore, various combinations are possible, such as a configuration that does not have both a sub-image and a sub-image gain map, or a configuration that has a sub-image but does not have a sub-image gain map.

[0053] FIG. 6(b) shows an example of the data structure of a JPEG image file.

[0054] The JPEG image file has the same data structure as the RAW image file in FIG. 6(a), except that the main image RAW is replaced with a main image JPEG, and there is no reduced RAW or reduced RAW gain map.

[0055] The image processing control unit 209 controls each component of the image processing unit 104 in Figure 2 to generate and regenerate a main image, a sub-image, a gain map, etc. from the input RAW image, and generates an output image for the display device.

[0056] [Embodiment 1] Hereinafter, with reference to FIG. 7, a gain map generation process during shooting by the imaging device 100 of the first embodiment will be described.

[0057] 7 is a flowchart illustrating a control process during shooting in embodiment 1. The process in FIG. 7 is realized by the system control unit 108 executing a program stored in ROM and controlling the image processing unit 104.

[0058] 7 starts when the power of the imaging device 100 is turned on and the system control unit 108 receives a shooting instruction from a shutter switch included in the operation unit 109 (step S700). The system control unit 108 controls the optical unit 101, the imaging unit 102, and the A / D conversion unit 103 to generate a RAW image.

[0059] In step S701, the image processing control unit 209 controls the development processing unit 211 to generate a main image (HDR image) based on the RAW image input to the image processing unit 104.

[0060] In step S702, the image processing control unit 209 outputs the main image generated in step S701 to the image resizing unit 208. The image resizing unit 208 performs resizing processing on the main image to generate a sub-image. Note that a reduced RAW image obtained by resizing a RAW image by the image resizing unit 208 may be output to the development processing unit 211, and the development processing unit 211 may perform development processing on the reduced RAW image to generate a sub-image.

[0061] In step S703, the image processing control unit 209 controls the gain map generation unit 220 to generate a main image gain map from the main image generated in step S701. The main image gain map may have the same resolution as the main image, or may have a resolution reduced to, for example, ¼ that of the main image by the gain map resizing unit 201. In addition, the gain map region integrating unit 202 may perform gain map region integrating processing.

[0062] In step S704, the image processing control unit 209 controls the gain map resizing unit 201 and the gain map region integrating unit 202 to generate a sub-image gain map from the main image gain map generated in step S703. Note that a reduced RAW image obtained by resizing the RAW image by the image resizing unit 208 may be output to the gain map generation unit 220, and the gain map generation unit 220 may generate the sub-image gain map.

[0063] In step S705, the image processing control unit 209 outputs the images and gain maps generated in steps S701 to S704 to the file storage unit 210, which then generates a file that combines these images and gain maps. The system control unit 108 stores the file generated by the file storage unit 210 in the recording medium 107, and the process ends.

[0064] [Embodiment 2] Hereinafter, with reference to FIG. 8, a gain map generation process performed by the imaging device 100 of the second embodiment when reducing an image will be described.

[0065] In the second embodiment, it is assumed that the recording medium 107 stores an image file having the structure shown in FIG. 6(b).

[0066] FIG. 8 is a flowchart illustrating a control process during image reduction according to the second embodiment.

[0067] The process starts when the user selects an image to be reduced from the images stored in the recording medium 107 via the operation unit 109 (step S800).

[0068] When the process starts in step S800, the system control unit reads the main image file selected by the user from the recording medium 107 and stores it in the storage unit .

[0069] In step S801, the image processing control unit 209 controls the image resizing unit 208 to perform reduction processing on the main image stored in the storage unit 106 according to the reduction ratio designated by the user via the operation unit 109.

[0070] In step S802, the image processing control unit 209 determines whether or not to regenerate the main image gain map. If the image processing control unit 209 determines not to regenerate the main image gain map, it skips the main image gain map regeneration process in step S803 and proceeds to step S804.

[0071] In step S804, the image processing control unit 209 outputs to the file storage unit 210 the main image file stored in the memory unit 106 in step S800 and the reduced image generated in step S801, generates a file in which the main image of the main image file stored in the memory unit 106 in step S800 has been replaced with the reduced image generated in step S801, stores this file on the recording medium 107, and terminates the processing.

[0072] If the image processing control unit 209 determines in step S802 that the main image gain map is to be regenerated, the process proceeds to step S803.

[0073] Here, the determination process of step S802 will be described. For example, if the resizing ratio specified by the user satisfies a predetermined condition, that is, if the reduction ratio is close to 1, that is, if the difference in pixel count between the main image and the resized image is small, the image processing control unit 209 determines that it is not necessary to regenerate the gain map of the main image. Also, if the reduction ratio specified by the user is close to the pixel count ratio between the main image and the sub-image, and the gain map of the main image can be substituted for the gain map of the sub-image, the image processing control unit 209 determines that it is not necessary to regenerate the gain map of the main image. Conversely, for example, if a sufficiently large gain map is available for the main image (e.g., the pixel count ratio is 1:1) or if a gain map without region integration has been saved, there is a high possibility that the data size can be significantly reduced by regenerating the gain map, and so the image processing control unit 209 determines that it is necessary to regenerate the gain map.

[0074] In step S803, the image processing control unit 209 regenerates the main image gain map. The regeneration of the main image gain map is achieved by a combination of one or more of the following data compression processes: resolution change processing by the gain map resizing unit 201, region division processing by the gain map region dividing unit 203, region integrating processing by the gain map region integrating unit 202, and bit depth change processing by the gain map encoding unit 204. If the regeneration of the gain map is completed in step S803, the image processing control unit 209 proceeds to step S804, outputs to the file storage unit 210 the main image file stored in the memory unit 106 in step S800, the reduced image generated in step S801, and the main image gain map regenerated in step S803, replaces the main image in the main image file stored in the memory unit 106 in step S800 with the reduced image generated in step S801, generates a file in which the main image gain map has been replaced with the main image gain map regenerated in step S803, stores this on the recording medium 107, and terminates the processing.

[0075] In the second embodiment, an image file having a configuration as shown in FIG. 6(b) has been described, but similar processing is also possible for an image file that does not have a sub-image or a sub-image gain map.

[0076] According to the second embodiment, when the main image is resized, the gain map attached to the main image file is also resized, thereby making it possible to reduce the size of the gain map while maintaining its accuracy.

[0077] [Embodiment 3] Hereinafter, with reference to FIG. 9, a gain map generation process performed by the imaging device 100 of the third embodiment when enlarging an image will be described.

[0078] The image enlargement process of the third embodiment is the same as the reduction process of the second embodiment shown in FIG. 8, where reduction is replaced by enlargement.

[0079] FIG. 9 is a diagram illustrating the gain map generation process during image enlargement according to the third embodiment.

[0080] The image processing control unit 209 performs known super-resolution processing on the main image using the image resizing unit 208, as shown in FIG. 9(a). In this case, the gain map corresponding to the main image includes both non-integrated and integrated regions, as shown in FIG. 9(b). Since it is considered acceptable to perform super-resolution processing only on the non-integrated regions of the gain map, the super-resolution processing is performed partially to save the time required for the super-resolution processing. In a similar manner, the super-resolution processing may be performed only on the gain map of the region where the resolution has been significantly improved by the super-resolution processing. Whether or not the resolution has been significantly improved can be determined by the image comparison unit 212 by determining the difference between the main image that has been subjected to super-resolution processing and an image enlarged using known interpolation technology.

[0081] According to embodiment 3, when a gain map attached to a main image file is also resized in response to resizing of the main image, the time required to resize the gain map can be saved by partially resizing the gain map.

[0082] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0083] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0084] The disclosure of this specification includes the following image processing device, control method, and program. [Configuration 1] a first generation means for generating first conversion information used when generating an image having a different dynamic range from the first image; an image resizing means for executing a resizing process on the first image; and a control means for controlling whether or not to perform resizing processing on the first conversion information attached to the first image when resizing processing is performed on the first image. [Configuration 2] 2. The image processing device according to configuration 1, wherein the control means controls the image processing device to perform resizing processing of the first conversion information when the resizing ratio of the first image does not satisfy a predetermined condition. [Configuration 3] a file storage means for generating a first file that stores the first image and the first conversion information generated from the first image; The image processing device according to configuration 2, characterized in that when resizing processing of the first conversion information is performed, the file storage means replaces the first image in the first file with a resized second image and generates a file in which the first conversion information is replaced with the resized second conversion information. [Configuration 4] the resizing process includes a reduction process, 4. The image processing device according to configuration 2 or 3, wherein the predetermined condition is that the first conversion information can be substituted with the resized second conversion information when the difference in resolution between the first image and the resized second image is smaller than a predetermined value. [Configuration 5] 2. The image processing device according to configuration 1, wherein the control means controls so as not to perform resizing processing of the first conversion information when the resizing ratio of the first image satisfies a predetermined condition. [Configuration 6] a file storage means for generating a first file that stores the first image and the first conversion information generated from the first image; The image processing device according to configuration 3, characterized in that, when the resizing process of the first conversion information is not performed, the file storage means generates a file in which the first image in the first file is replaced with a resized second image. [Configuration 7] the resizing process includes a reduction process, 7. The image processing device according to configuration 5 or 6, wherein the predetermined condition is that the first conversion information can be substituted with the resized second conversion information when the difference in resolution between the first image and the resized image is smaller than a predetermined value. [Configuration 8] the first conversion information forms a gain map in which a gain amount of a signal value for each pixel is determined; The gain map resizing process is a resolution change process for changing the resolution of the entire region of the gain map; a region integration process for integrating some regions of the gain map; The image processing device according to any one of the configurations 1 to 7, further comprising one or more processes among area division processes for dividing a part of the gain map. [Configuration 9] 9. The image processing device according to configuration 8, wherein the control means controls which of the resolution changing process, the region integrating process, and the region dividing process to perform depending on the resizing ratio of the first image. [Configuration 10] the image resizing processing means generates one or more second images by resizing the first image; 10. The image processing device according to any one of configurations 1 to 9, characterized in that, when the resizing process is performed on the first image, the control means performs the resizing process on the first conversion information attached to the first image, or the control means performs the resizing process on the second conversion information attached to the second image. [Configuration 11] 11. The image processing device according to configuration 10, wherein the first generating means generates the second conversion information attached to the second image from the first conversion information. [Configuration 12] the resizing of the first image includes enlargement; 7. The image processing device according to any one of configurations 1 to 6, wherein the control means executes enlargement processing partially on the first conversion information. [Configuration 13] 13. The image processing device according to claim 12, wherein the enlargement process includes super-resolution processing using machine learning. [Configuration 14] 14. The image processing device according to any one of configurations 1 to 13, wherein the first conversion information is any one of information for converting an image of a first dynamic range into an image of a second dynamic range, information for converting an image of the second dynamic range into an image of the first dynamic range, and information for converting an image of the first dynamic range into an image of the first dynamic range having a different dynamic range. [Configuration 15] An imaging means; 15. The image processing device according to any one of configurations 1 to 14, further comprising: second generating means for generating the first image from a RAW image generated by the imaging means. [Configuration 16] A control method for an image processing device, comprising: generating conversion information used when generating images with different dynamic ranges from the first image; performing a resizing process on the first image; a step of controlling whether or not to perform resizing processing on the conversion information attached to the first image when resizing processing is performed on the first image. [Configuration 17] A program for causing a computer to function as the image processing device according to any one of configurations 1 to 15. [Explanation of symbols]

[0085] 100... imaging device, 104... image processing unit, 108... system control unit, 200... base gain map generation unit, 201... gain map resizing processing unit, 208... image resizing processing unit, 209... image processing control unit, 220... gain map generation unit

Claims

1. a first generating means for generating first conversion information used when generating an image having a different dynamic range from the first image; an image resizing means for executing a resizing process on the first image; and a control means for controlling, when a resizing process is performed on the first image, whether or not a resizing process is to be performed on the first conversion information attached to the first image.

2. 2. The image processing apparatus according to claim 1, wherein said control means controls to perform resizing processing of said first conversion information when the resizing ratio of said first image does not satisfy a predetermined condition.

3. a file storage means for generating a first file that stores the first image and the first conversion information generated from the first image; 3. The image processing device according to claim 2, wherein, when resizing processing of the first conversion information is performed, the file storage means replaces the first image in the first file with a resized second image and generates a file in which the first conversion information is replaced with the resized second conversion information.

4. the resizing process includes a reduction process, 3. The image processing device according to claim 2, wherein the predetermined condition is that when the difference in resolution between the first image and the resized second image is smaller than a predetermined value, the first conversion information can be substituted with the resized second conversion information.

5. 2. The image processing apparatus according to claim 1, wherein said control means controls so as not to perform resizing processing of said first conversion information when a resizing ratio of said first image satisfies a predetermined condition.

6. a file storage means for generating a first file that stores the first image and the first conversion information generated from the first image; 4. The image processing device according to claim 3, wherein, when the resizing process of the first conversion information is not performed, the file storage means generates a file in which the first image in the first file is replaced with a resized second image.

7. the resizing process includes a reduction process, 6. The image processing device according to claim 5, wherein the predetermined condition is that the first conversion information can be substituted with the resized second conversion information when the difference in resolution between the first image and the resized image is smaller than a predetermined value.

8. the first conversion information forms a gain map in which a gain amount of a signal value for each pixel is determined; The gain map resizing process is a resolution change process for changing the resolution of the entire region of the gain map; a region integration process for integrating some regions of the gain map; 2. The image processing apparatus according to claim 1, further comprising one or more processes among area division processes for dividing a part of the gain map.

9. 9. The image processing device according to claim 8, wherein the control means controls which of the resolution change processing, the area integration processing, and the area division processing to perform depending on a resize ratio of the first image.

10. the image resizing processing means generates one or more second images by resizing the first image; 2. The image processing device according to claim 1, wherein, when the resizing process is performed on the first image, the control means performs the resizing process on the first conversion information attached to the first image, or the control means performs the resizing process on the second conversion information attached to the second image.

11. 11. The image processing apparatus according to claim 10, wherein the first generating means generates the second conversion information attached to the second image from the first conversion information.

12. the resizing of the first image includes enlarging the first image; 2. The image processing apparatus according to claim 1, wherein the control means executes enlargement processing partially on the first conversion information.

13. The image processing device according to claim 12 , wherein the enlargement processing includes super-resolution processing using machine learning.

14. 2. The image processing device according to claim 1, wherein the first conversion information is either information for converting an image of a first dynamic range into an image of a second dynamic range, information for converting an image of the second dynamic range into an image of the first dynamic range, or information for converting an image of the first dynamic range into an image of a first dynamic range having a different dynamic range.

15. An imaging means; 2. The image processing apparatus according to claim 1, further comprising: second generating means for generating the first image from the RAW image generated by the imaging means.

16. A control method for an image processing device, comprising: generating conversion information used when generating images with different dynamic ranges from the first image; performing a resizing process on the first image; a step of controlling whether or not to perform resizing processing on the conversion information attached to the first image when resizing processing is performed on the first image.

17. A program for causing a computer to function as the image processing device according to any one of claims 1 to 15.

Images