Image processing device, image processing method, and program

Abstract

This technology aims to suppress the unnecessarily large size of conversion information used when generating images in different formats. [Solution] The image processing device includes an acquisition means for acquiring a RAW image, an image processing means for generating a first image from the RAW image, and a generation means for generating conversion information used when generating a second image with a different format from the first image from the first image, wherein the generation means generates conversion information for a format according to the characteristics of the second image.

Description

Technical Field

[0001] The present invention relates to a technique for generating conversion information used when generating images of different formats.

Background Art

[0002] Conventionally, in order to conform to the dynamic range that a display device can display, images in HDR (High Dynamic Range) format and SDR (Standard Dynamic Range) format are mutually converted using conversion information called a gain map (Non-Patent Document 1).

[0003] Also, in Patent Document 1, it is described that an image with a wide dynamic range and a wide color gamut is generated with a low processing load based on base image data and difference data used in processing for expanding the dynamic range and color gamut of the image data.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

[0006] When generating an HDR image by applying a gain map to an SDR image, the hue and saturation of the HDR image may differ significantly depending on the tonal characteristics of the SDR image before conversion. If you want to adjust not only the dynamic range of the SDR image but also color information such as hue and saturation, you need to convert not only the luminance component but also the color component, thus requiring a multi-channel gain map. However, if you have a multi-channel gain map even when the color information characteristics of the SDR image and the HDR image are similar and no adjustment of color information is necessary, the data size of the gain map becomes unnecessarily large. Patent Document 1 does not consider cases where the color information of the SDR image and the HDR image does not need to be adjusted, so the data size of the gain map may become unnecessarily large.

[0007] This invention has been made in view of the above problems, and its purpose is to realize a technology that suppresses the unnecessary increase in the size of conversion information used when generating images of different formats. [Means for solving the problem]

[0008] To solve the above problems and achieve the objective, the image processing apparatus of the present invention comprises: acquisition means for acquiring a RAW image; image processing means for generating a first image from the RAW image; and generation means for generating conversion information used when generating a second image with a different format from the first image from the first image, wherein the generation means generates conversion information for a format according to the characteristics of the second image. [Effects of the Invention]

[0009] According to the present invention, it is possible to suppress the unnecessarily large size of the conversion information used when generating images of different formats. [Brief explanation of the drawing]

[0010] [Figure 1] A block diagram illustrating the configuration of the apparatus in this embodiment. [Figure 2] A diagram illustrating the relationship between the type of subject and the gamma characteristics in this embodiment. [Figure 3] A flowchart illustrating the gain map generation process of Embodiment 1. [Figure 4] A diagram illustrating the gain map generation process of Embodiment 1. [Figure 5] This diagram illustrates how to reduce the size of the multi-channel gain map in Embodiment 1. [Figure 6] A flowchart illustrating the gain map generation process of Embodiment 2. [Figure 7] A flowchart illustrating the scene recognition process of Embodiment 2. [Modes for carrying out the invention]

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention to the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] In this embodiment, an example of applying the image processing apparatus of the present invention to an imaging device such as a digital camera is described, but it is not limited to this and can be broadly applied to electronic devices having an imaging function. Such electronic devices include computer equipment (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, dashcams, medical devices, and the like.

[0013] <Device configuration> Next, with reference to Figure 1, the configuration and function of the imaging device 100 of this embodiment will be described.

[0014] Figure 1 is a block diagram showing the configuration of the imaging device 100 according to this embodiment.

[0015] The imaging device 100 includes an optical system 101, an imaging unit 102, a control unit 103, a primary storage device 104, a secondary storage device 105, a photometering sensor 106, an image processing unit 107, a recording medium 108, a communication unit 109, a display unit 110, and an operation unit 111.

[0016] Each functional block of the imaging device 100 can be implemented by software or a combination of software and hardware, excluding parts that can only be realized by hardware. For example, the functional block may be realized by dedicated hardware such as an ASIC. Also, the functional block may be realized by a processor such as a CPU executing a program stored in a memory. Note that a plurality of functional blocks may be realized by a common configuration (for example, one ASIC). Also, the hardware that realizes some functions of a certain functional block may be included in the hardware that realizes other functional blocks.

[0017] In this embodiment, based on the RAW image obtained by the imaging process of the imaging unit 102, the image processing unit 107 generates an image in SDR (Standard Dynamic Range) format (SDR image), an image in HDR (High Dynamic Range) format (HDR image), and a gain map. The gain map is conversion information that can generate images (SDR images or HDR images) with different dynamic ranges by being applied to the source image (HDR image or SDR image). The format of the gain map includes a multi-channel or color channel (3-channel) format and a monochannel (1-channel) format.

[0018] During display, an adaptive HDR expression can be realized by applying a weight from 0 to 1 to the gain map. For example, an image data file with a gain map applied to an SDR image as the source image is generated. During display, an HDR image for display can be generated by applying an appropriate weight according to the capabilities of the display device and the gain map to the SDR image. When the display device does not support HDR, the SDR image is displayed by ignoring the gain map (setting the weight to 0).

[0019] In this embodiment, PQ (Perceptual Quantization) and HLG (Hybrid Log Gamma) are assumed as signal characteristics representing the relationship between signal level and display brightness in HDR images. PQ is defined by the EOTF (Electro-Optical Conversion Function) in SMPTE ST 2084, and HLG is defined by the OETF (Opto-Electro-Telescopic Conversion Function) in ARIB STD-B67.

[0020] Furthermore, an SDR image is defined as an image with a narrower output range (dynamic range) than an HDR image. In the following explanation, the gamma of HDR images will be based on the OETF characteristics (PQ Inverse EOTF) according to SMPTE ST 2084, and the color gamut will conform to the ITU-R BT. 2020 standard. In addition, the gamma of SDR images will be sRGB gamma, and the color gamut will be sRGB.

[0021] The optical system 101 includes optical components such as lenses, shutters, and diaphragms, and a mechanism (such as a motor) for driving these optical components. The optical system 101 may be integrated with the imaging device 100, or it may have the form of an interchangeable lens. The optical system 101 forms an optical image of the subject on the imaging surface of the imaging unit 102.

[0022] The imaging unit 102 includes an image sensor, for example, a CCD or CMOS having a color filter with a primary color Bayer array. The imaging unit 102 has a pixel array in which multiple pixels are arranged in two dimensions, and peripheral circuits for reading signals from each pixel. Each pixel accumulates charge according to the amount of incident light by photoelectric conversion. By reading signals with a voltage corresponding to the amount of charge accumulated during the exposure period from each pixel, a group of pixel signals (analog image signals) consisting of electrical signals representing the optical image formed on the imaging surface is obtained.

[0023] The analog image signal is converted to a digital image signal (image data) via A / D conversion and then stored in the primary storage device 104. The A / D conversion may be performed by the imaging unit 102, or by another component outside the imaging unit 102 (for example, an A / D converter not shown). Image data consisting of pixel data having one color component, R, G, or B, which is stored in the primary storage device 104 at this stage, is called RAW image data.

[0024] The control unit 103 includes an arithmetic processing processor such as a CPU or MPU that executes control processing for the imaging device 100. By loading a program stored in the secondary storage device 105 into the primary storage device 104 and executing it, the control unit 103 controls the operation of each part of the imaging device 100 and realizes various functions of the imaging device 100. At least some of the functions described as being realized by the control unit 103 executing a program may be executed by hardware such as an ASIC or FPGA configured to perform those functions.

[0025] The control unit 103 determines the imaging conditions for the imaging unit 102 (shutter speed or exposure time, aperture value, and shooting sensitivity).

[0026] The primary storage device 104 is a volatile memory such as RAM. The primary storage device 104 temporarily stores various data, including the program executed by the control unit 103, the data necessary for program execution, data before processing by the image processing unit 107, data during processing, data after processing, display image data, and recording image data.

[0027] The secondary storage device 105 is a non-volatile memory such as flash ROM. The secondary storage device 105 stores programs executed by the control unit 103 to control the imaging device 100, various settings, information about the imaging device 100, GUI (Graphical User Interface) data, and so on.

[0028] The optical image formed by the optical system 101 is projected onto the photometering sensor 106. The photometering sensor 106 measures the brightness information of the optical image for each of the 96 photometric areas (for example, 12 horizontally and 8 vertically). The photometering sensor 106 outputs the photometric results to the control unit 103. The number, position, and size of the photometric areas change depending on the settings of the imaging device 100.

[0029] The image processing unit 107 performs various image processing operations on image data read from the imaging unit 102 and recording medium 108 and stored in the primary storage device 104, acquiring and / or generating signals, image data, and information according to the application. The image processing unit 107 may be a dedicated hardware circuit, such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the image processing unit 107 may be configured such that a processor, such as a DSP (Digital Signal Processor) or GPU (Graphics Processing Unit), executes software to realize a specific function. The image processing unit 107 outputs the acquired or generated signals, information, and data to the control unit 103 or stores them in the primary storage device 104, according to the application.

[0030] The image processing performed by the image processing unit 107 includes preprocessing, color interpolation, correction, detection, data processing, evaluation value calculation, and special effects processing.

[0031] Preprocessing includes signal amplification, reference level adjustment, and defective pixel correction.

[0032] Color interpolation is performed when a color filter is provided in the imaging unit 102, and it is a process that interpolates the values ​​of color components that are not included in the individual pixel data that make up the image data. Color interpolation is also called demosaicing.

[0033] The correction process includes white balance adjustment, gradation correction (gamma), correction of image degradation caused by optical aberrations in the optical system 101 (image recovery), correction of the effects of vignetting in the optical system 101, and color correction.

[0034] Detection processes include detecting feature regions (e.g., face regions and human body regions), detecting movement within feature regions, and recognizing people.

[0035] Data processing includes processes such as region extraction (trimming), merging, scaling, encoding and decoding, and header information generation (data file generation). The generation of display image data and recording image data, the generation of gain maps (described later), and conversion of color gamut and dynamic range are also included in data processing.

[0036] The evaluation value calculation process includes generating signals and evaluation values ​​used for autofocus detection (AF), and generating evaluation values ​​used for automatic exposure control (AE).

[0037] Special effects processing includes adding blur effects, changing color tones, and relighting.

[0038] The image processing described above is merely an example and does not limit the processing that the image processing unit 107 may perform.

[0039] The recording medium 108 is a storage device such as a memory card. The recording medium 108 stores recording image data and the like stored in the primary storage device 104. The recording medium 108 may be removable from the imaging device 100. The image data stored on the recording medium 108 can be read by an external device such as a personal computer. The imaging device 100 has a mechanism for attaching and detaching the recording medium 108, as well as functions for reading data from and writing data to the recording medium 108.

[0040] The communication unit 109 is an interface that enables communication between the imaging device 100 and an external device. The communication unit 109 conforms to one or more wired communication standards and / or wireless communication standards. The communication unit 109 has a transmitting and receiving circuit, connector or antenna, etc., according to the communication standard. Communication standards that the communication unit 109 conforms to include, but are not limited to, USB, wireless LAN, Bluetooth®, HDMI®, etc.

[0041] The display unit 110 is an output device such as a liquid crystal display or an organic EL display provided on the outer surface of the imaging device 100. The display unit 110 may be integrated with the imaging device 100 or it may be an external device connected to the imaging device 100. The imaging device 100 only needs to be able to connect to the display unit 110 and control the display of the display unit 110. The display unit 110 displays information from the imaging device 100, menu screens, GUIs, images captured by the imaging unit 102, images read from the recording medium 108, etc. The imaging device 100 can make the display unit 110 function as an electronic viewfinder (EVF) by continuously performing video recording and display on the display unit 110 in the shooting standby state. The operation of making the display unit 110 function as an EVF is called live view display, and the image used for live view display is called a live view image.

[0042] The operation unit 111 consists of various input devices such as switches, buttons, and dials that accept various operations from the user and output operation information to the control unit 103. The operation unit 111 includes, for example, a power button to turn the power on or off, a shooting button to start or stop taking still images or videos, a playback button to play back images, and a mode switching button to change the operating mode of the imaging device 100. The operation unit 111 also includes a dedicated connection button for starting communication with an external device such as a personal computer. The functions assigned to the same input device may be variable. The input device may also be a software button or key using a touch display.

[0043] In still image capture mode, the camera performs AF (autofocus) processing and AE (automatic exposure) processing based on the image signal generated by the imaging unit 102. The control unit 103 also performs a shooting process in which the image processing unit 107 processes the image signal generated by the imaging unit 102 and saves the resulting still image data to the recording medium 108.

[0044] In video recording mode, the control unit 103 performs AF (autofocus) processing and AE (automatic exposure) processing based on the image signal for each frame generated by the imaging unit 102. The control unit 103 also performs shooting processing to record video data, which has been processed by the image processing unit 107 on the image signal generated by the imaging unit 102, onto the recording medium. The control unit 103 starts the video data shooting process when the shooting button is pressed for the first time and continues the video data shooting process until the shooting button is pressed again. The control unit 103 stops the video data shooting process when the shooting button is pressed again and saves the video data for the time elapsed from the start to the stop of the shooting process to the recording medium 108.

[0045] Next, with reference to Figure 2, the characteristics of the SDR image and HDR image in this embodiment will be described.

[0046] Figure 2 illustrates the relationship between the type of subject and the gamma characteristics.

[0047] Because SDR images have a narrow expressible dynamic range 201, the actual input dynamic range 202 may be wider. Such input / output dynamic range characteristics are represented by gamma curves, but since a wide input dynamic range needs to be represented by a narrow output dynamic range, there are various methods for adjusting the dynamic range. For example, in the case of a portrait 204 with a main subject, by using a gamma curve that emphasizes the tonality 205 of the main subject, the impression of the image of the main subject can be brought closer to the actual impression. In that case, if the output dynamic range becomes insufficient in the high-brightness areas, the input dynamic range 202 can be adjusted to within the output dynamic range 201 by making the slope of the gamma curve gentler, as in gamma curve 206. Also, when it is important to emphasize the tonality of the entire subject, such as a landscape 207, it is often possible to bring the overall impression of the image closer to the actual impression by linearly scaling it to within the output dynamic range 201, as in gamma curve 208, or by performing local contrast processing.

[0048] On the other hand, HDR images have a wider dynamic range compared to SDR images, so portraits and landscapes can sometimes be represented with the same gamma curve characteristics.

[0049] As described above, the characteristics of SDR images differ depending on the scene, so it is desirable to use SDR images appropriately. However, depending on the characteristics, color information such as hue and saturation may change compared to HDR images.

[0050] <Gain map generation process of Embodiment 1> Next, with reference to Figure 3, the gain map generation of Embodiment 1 will be described.

[0051] Embodiment 1 describes a gain map generation process that reduces the data size depending on the mode.

[0052] Figure 3 is a flowchart illustrating the gain map generation process of Embodiment 1.

[0053] The process shown in Figure 3 is achieved by the control unit 103 loading a program stored in the secondary storage device 105 into the primary storage device 104 and executing it, thereby controlling each component of the imaging device 100. The process shown in Figure 6, which will be described later, is similar.

[0054] In step S301, the control unit 103 acquires the mode setting related to the tonal characteristics of the SDR image input by the user via the operation unit 111. The mode setting determines the conversion characteristics during image processing, such as the gamma curves 206 and 208 in Figure 2. For example, the user may choose whether to express the tonal characteristics of the HDR image within the dynamic range of the SDR image by generating an SDR image from a RAW image or by converting from an HDR image to generate an SDR image, or they may select a shooting scene such as portrait 204 or landscape 207, or select information indicating the width of the dynamic range as a percentage. Furthermore, a mechanism may be implemented to select and link a function that changes the width of the dynamic range, such as multi-shot.

[0055] In step S302, the control unit 103 controls the optical system 101 and the imaging unit 102 to capture a RAW image.

[0056] In step S303, the control unit 103 develops the RAW image acquired in step S302 into an HDR image by controlling the image processing unit 107.

[0057] In step S304, the control unit 103 switches the conversion characteristics during image processing based on the mode setting acquired in step S301. In this embodiment, the characteristics of the gamma curve 206 are described in relation to a mode in which the characteristics of the gamma curve 206 are generated from a RAW image as an SDR image, and a mode in which the characteristics of the gamma curve 208 are converted from an HDR image to generate an SDR image. If the control unit 103 generates an SDR image from a RAW image, it proceeds to step S305, and if it generates an SDR image by converting from an HDR image, it proceeds to step S307.

[0058] In step S305, the control unit 103 develops the RAW image acquired in step S302 into an SDR image by controlling the image processing unit 107.

[0059] In step S306, the control unit 103 controls the image processing unit 107 to generate a gain map with a greater number of channels (color format) than the number of channels generated in step S308.

[0060] In step S307, the control unit 103 controls the image processing unit 107 to convert the HDR image developed in step S303 into an SDR image.

[0061] In step S308, the control unit 103 controls the image processing unit 107 to generate a gain map with fewer channels (monochrome format) than the number of channels generated in step S306.

[0062] In step S309, the control unit 103 controls the image processing unit 107 to add the gain map generated in step S306 or step S308 to the SDR image generated in step S305 or step S307 and output it as a file.

[0063] By controlling it in this way, when displaying on an HDR-compatible display device, the gain map applied to the SDR image can be used to display an image similar to the HDR image generated in step S303. When displaying on a display device that is not HDR-compatible, the SDR image can be displayed as is without using the gain map.

[0064] Alternatively, a gain map may be applied to the HDR image in step S309. In this case, a gain map with the reciprocal of the gain amount of the generated gain map as its gain can be applied, or the same gain map applied to the SDR image can be applied to the HDR image, and the reciprocal can be multiplied when applying it. In this case, the gain map is not applied when displaying on an HDR-compatible display device, and the gain map is applied when displaying on an HDR-incompatible display device.

[0065] Next, referring to Figure 4, the processes in steps S305 to S308 in Figure 3 will be explained.

[0066] Figure 4 is an explanatory diagram of the gain map generation process in Embodiment 1.

[0067] Let 401 be the SDR image with the characteristics of gamma curve 206, 403 be the SDR image with the characteristics of gamma curve 208, 405 be the gain map corresponding to SDR image 401, 409 be the HDR image, 407 be the gain map corresponding to SDR image 403, and 411 be the HDR image.

[0068] First, let's explain the processes in steps S305 and S306 of Figure 3.

[0069] Since the SDR image 401 is generated based on characteristics such as the gamma curve 206 in Figure 2, the gradation of the high-luminance areas that can be represented in region 410 of the HDR image 409 may not be represented as in region 402. Therefore, if the slope of the high-luminance area of ​​the gamma curve of the SDR image 401 is below a predetermined threshold, in order to generate the HDR image 409 by applying the gain map 405 to the SDR image 401, the gain map 405 needs to have a multi-channel gain map that absorbs the difference in image quality, as in region 406.

[0070] To absorb differences in dynamic range, such as differences in brightness gradation, it is sufficient to apply the same gain map to each of the RGB components in the RGB color space, or to the Y component in the YUV color space; therefore, a single-channel gain map is sufficient. However, to absorb differences in hue and saturation, it is necessary to apply a gain map to each of the RGB components in the RGB color space, or to the UV component in the YUV color space, or to calculate the hue and saturation from the aforementioned color spaces and then apply a gain map to those values.

[0071] For example, if you only want to correct the saturation component as a color gradation, you can apply the same gain map to the UV component of the YUV color space or apply a gain map to the calculated saturation, so you will need at least two channels of gain maps for luminance and saturation. On the other hand, if you want to correct not only saturation but also the hue component as a color gradation, you will need to apply different gain maps to each of the RGB components of the RGB color space, or to the UV component of the YUV color space, or to the calculated hue and saturation, so you will need at least three channels of gain maps.

[0072] Next, we will explain the processes in steps S306 and S307 of Figure 3.

[0073] The SDR image 403 is generated by performing a transformation on the HDR image 411 that compresses the dynamic range so that it has characteristics similar to the gamma curve 208 in Figure 2. In this case, processing such as local contrast adjustment may be included.

[0074] Region 412 of the HDR image 411 is represented by compressing the dynamic range, similar to region 404 of the SDR image 403, but the color information of the HDR image 411 can be preserved. In cases where it is not necessary to absorb the difference in color information between the SDR image 403 and the HDR image 411, only the difference in brightness gradation mentioned above needs to be absorbed, and therefore a single-channel gain map is sufficient.

[0075] By changing the number of channels used when generating the gain map according to the characteristics of the SDR image, the data size of the gain map can be reduced.

[0076] Furthermore, in step S307, if there are significant changes in color information, such as when compressing the dynamic range and exceeding the color gamut, it is possible to have a multi-channel gain map.

[0077] Furthermore, if gain maps are used for multiple channels, the map size of the gain maps may be reduced to suppress the data size of the gain maps. By making the map size of the multi-channel gain maps smaller than or equal to that of the small-channel gain maps, the difference in data size between the two can be reduced.

[0078] Now, referring to Figure 5, we will explain the case where the gain map size of the multi-channel gain map is less than or equal to the map size of the gain map of the small-channel gain map.

[0079] Figure 5 is an explanatory diagram illustrating the case where the size of the multi-channel gain map in Embodiment 1 is reduced.

[0080] In Figure 5, region 501 is a contiguous pixel region of an HDR image, and pixels 502, 503, 504 and pixels 505, 506, 507 contained within pixel region 501 are assumed to have different color information, such as red and blue.

[0081] Next, let 508 be a contiguous pixel region in the SDR image that corresponds to a contiguous pixel region 501 in the HDR image. That is, pixel 509 in pixel region 508 corresponds to pixel 502, pixel 510 to pixel 503, pixel 511 to pixel 504, pixel 512 to pixel 505, pixel 513 to pixel 506, and pixel 514 to pixel 507.

[0082] Note that the color information of corresponding pixels 502-507 in the HDR image and pixels 509-514 in the SDR image will be different.

[0083] The gain map 515 is a gain map that converts from the pixel area 508 of an SDR image to the pixel area 501 of an HDR image. In order to reduce the data size, the map size is set to half the size of the image. In this embodiment, the map size is described as half for ease of understanding, but it is not limited to this, and any scale may be used.

[0084] Pixels 509 and 510 of the SDR image have similar color information characteristics, and pixels 502 and 503 of the target HDR image after applying the gain map also have similar color information characteristics, so the gain map can be combined as shown in region 516.

[0085] Similarly, pixels 513 and 514 of the SDR image and pixels 506 and 507 of the HDR image can also have their gain maps combined into a region like region 518.

[0086] However, as mentioned above, pixels 504 and 505 of the HDR image have different color information, and each also has different color information from pixels 511 and 512 of the SDR image. Therefore, if the gain map is grouped together as shown in region 517, there is a possibility of misconversion.

[0087] Therefore, in this embodiment, gain maps for regions with different color information are provided with higher resolution. For example, a sub-gain map 519 is generated separately from gain map 515, and regions 520 and 521 are given different gain amounts. Alternatively, one gain map may be provided for each region, such as gain map 522, where the number of pixels targeted differs. In the example in Figure 5, regions 516 and 523 target the same two pixels and have the same gain amount, and the same is true for regions 518 and 526. Also, regions 520 and 524 target the same one pixel and have the same gain amount, and the same is true for regions 521 and 525.

[0088] The color ratio is used to determine whether or not to have a higher resolution gain map for regions with different color information. When creating a gain map at 1:1 scale and then reducing it to half size, the color ratio of the regions being combined by the reduction is referenced. For example, if a gain map is in the RGB color space, if the RGB ratio of the gain amounts of each region being combined by the reduction is above a predetermined threshold, then it should be maintained at a higher resolution. In addition, if a gain map is in the YUV color space, the UV ratio may be used as a criterion. Furthermore, instead of using the gain map, the difference in color information of adjacent pixels in an SDR or HDR image may be used as a criterion.

[0089] Furthermore, when reducing the size to 1 / 3, instead of 3 regions, you can combine only 2 regions and divide them into 1 region and 2 regions to have a higher resolution gain map, or if they have the same gain as adjacent surrounding regions, you can combine them with those regions.

[0090] In this embodiment, we have described the system assuming that it has gain maps in the RGB color space and the YUV color space, but this is not limited to these, and it is also possible to have gain maps in other color spaces such as the L*a*b* color space and the ICtCp color space.

[0091] [Embodiment 2] Next, Embodiment 2 will be described with reference to Figure 6.

[0092] Embodiment 2 describes a gain map generation process that reduces data size according to scene recognition.

[0093] The apparatus configuration of Embodiment 2 is the same as that of Figure 1 of Embodiment 1.

[0094] Figure 6 is a flowchart illustrating the gain map generation process of Embodiment 2.

[0095] Steps S601 and S602 are the same as the processes in steps S302 and S303 in Figure 3.

[0096] In step S603, the control unit 103 performs scene recognition regarding the tonal characteristics of the SDR image based on the RAW image acquired in step S601, the HDR image developed in step S602, and photometric information acquired from the photometric sensor 106. Based on the scene recognition result, the control unit 103 switches the conversion characteristics during image processing. In this embodiment, the recognized scenes include scenes that prioritize the tonal characteristics of the main subject, such as a person, and scenes that prioritize the tonal characteristics of the entire subject, such as a landscape. Note that the information such as the RAW image used in scene recognition does not have to be information acquired at the same time as the actual SDR image generation; information acquired in advance may be used. In the case of still images, information such as live view may be used.

[0097] Now, referring to Figure 7, the scene recognition process in step S603 of Figure 6 will be explained.

[0098] Figure 7 is a flowchart illustrating the scene recognition process in step S603 of Figure 6.

[0099] In step S701, the control unit 103 controls the image processing unit 107 to perform subject recognition based on the RAW image acquired in step S601, the HDR image developed in step S602, and the photometric information acquired from the photometric sensor 106. In subject recognition, machine learning such as deep learning is used to detect the main subject, such as a person. If the main subject is detected, the control unit 103 proceeds to step S702; if the main subject is not detected, the process proceeds to step S704.

[0100] In step S702, the control unit 103 determines whether the proportion of the main subject detected in step S701 within the field of view is greater than or equal to a predetermined threshold. If a more important area is identifiable within the main subject, such as a person, the determination may be limited to the area such as the face.

[0101] If the control unit 103 determines in step S702 that the size of the main subject is greater than or equal to a predetermined threshold, it proceeds to step S703 and determines that it is a scene in which an SDR image prioritizing the gradation of the main subject is to be generated, and terminates the process.

[0102] If the control unit 103 determines in step S702 that the size of the main subject is less than a predetermined threshold, or if it determines in step S701 that there is no main subject, it proceeds to step S704.

[0103] In step S704, the control unit 103 calculates the dynamic range based on the RAW image acquired in step S601 and the HDR image developed in step S602 in Figure 6, and determines whether the calculated dynamic range is greater than or equal to a predetermined threshold.

[0104] If the control unit 103 determines in step S704 that the dynamic range is above a predetermined threshold, it proceeds to step S705.

[0105] In step S705, the control unit 103 determines that the scene is one in which a wide dynamic range SDR image is generated, prioritizing the overall tonality of the subject, such as a landscape.

[0106] If the control unit 103 determines in step S704 that the dynamic range is less than a predetermined threshold, it proceeds to step S703.

[0107] In step S703, the control unit 103 determines that there is no problem in compressing the dynamic range while prioritizing the tonal quality of the entire subject, and that it is a scene in which an SDR image with a narrow dynamic range that prioritizes the tonal quality of the main subject is to be generated.

[0108] In this embodiment, an example of switching the conversion characteristics during image processing based on two types of scenes was described. However, it is also possible to switch between three types of scenes, separating scenes that prioritize the tonal range of the main subject from scenes that generate SDR images with a narrow dynamic range. By doing so, it becomes possible to dynamically switch the SDR image generated according to the scene.

[0109] Returning to the explanation of Figure 6, in step S604, the control unit 103 switches the process by which the image processing unit 107 generates an SDR image based on the scene determined in step S603. In Embodiment 2, similar to Embodiment 1, the process will be described in correspondence with the mode in which the characteristics of the gamma curve 206 are generated as an SDR image from the RAW image, and the mode in which the characteristics of the gamma curve 208 are converted from the HDR image to generate an SDR image. Note that if the scene is one in which the gradation of the main subject is important, the process proceeds to step S605, which generates an SDR image from the RAW image, and if the scene is one in which the gradation of the entire subject is important, the process proceeds to step S607, which generates an SDR image by converting from the HDR image.

[0110] Steps S605 to S609 are the same as the processes in steps S305 to S309 in Figure 3.

[0111] In this embodiment, a person was used as the primary subject, but the system is not limited to this. Animals, vehicles, or other objects may be recognized as the primary subject, or the system may recognize scenes such as group photos or sports events and switch processing accordingly.

[0112] Furthermore, although this embodiment describes an example of switching the conversion characteristics during image processing based on the scene recognition result, control may also be performed based on the shooting mode of the imaging device 100. For example, control is possible based on the setting of the multi-shot mode or a mode that prioritizes the representation of high-luminance gradations.

[0113] [Other embodiments] The present invention may be applied to a system consisting of multiple devices (such as a host computer, interface devices, imaging devices, and web applications), or to a device consisting of a single device.

[0114] This embodiment can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiment to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0115] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention.

[0116] The disclosures herein include the following image processing apparatus, image processing methods, and programs. [Item 1] A means of acquiring RAW images, Image processing means for generating a first image from the aforementioned RAW image, The system includes a generation means for generating conversion information used when generating a second image with a different format from the first image from the first image, The image processing apparatus is characterized in that the generation means generates format conversion information according to the characteristics of the second image. [Item 2] The image processing apparatus according to item 1, characterized by having a determination means for determining the characteristics of the second image. [Item 3] The image processing apparatus according to item 2, characterized in that the determination means determines a mode relating to the gradation characteristics of the second image. [Item 4] The image processing apparatus according to item 3, characterized by having a selection means for selecting the aforementioned mode. [Item 5] The image processing apparatus according to item 3 or 4, characterized in that the generation means generates conversion information for the first format when the mode is a mode for generating the second image from the RAW image, and generates conversion information for the second format when the mode is a mode for generating the second image from the first image. [Item 6] The image processing apparatus according to item 2, characterized in that the determination means performs scene recognition relating to the grayscale characteristics of the second image. [Item 7] The image processing apparatus according to item 6, characterized in that the generation means generates conversion information for a first format if the scene is a first scene, and generates conversion information for a second format if the scene is a second scene. [Item 8] The aforementioned scene recognition is a process of recognizing a subject from the RAW image, The first scene described above is a mode that prioritizes the tonal range of a given subject, The image processing apparatus according to item 7, characterized in that the first scene is a mode that emphasizes the tonal gradation of the entire subject. [Item 9] The image processing apparatus according to item 8, characterized in that the determination means determines that it is the first scene if the size of the predetermined subject is greater than or equal to a threshold, and determines that it is the second scene if the size of the predetermined subject is less than a threshold. [Item 10] The image processing apparatus according to item 9, characterized in that the determination means determines the dynamic range based on the RAW image and the first image if the size of the predetermined subject is less than a threshold, determines it to be the first scene if the dynamic range is greater than or equal to a threshold, and determines it to be the second scene if the dynamic range is less than a threshold. [Item 11] An image processing apparatus according to any one of items 1 to 10, characterized by having a storage means for adding and storing conversion information generated by the generation means to the first image or the second image. [Item 12] The image processing apparatus according to any one of items 1 to 11, characterized in that the second image characteristic is the gradation characteristic of the gamma curve. [Item 13] The image processing apparatus according to item 12, characterized in that the gradation characteristics of the gamma curve are the gradation characteristics of the high-luminance area. [Item 14] The format of the aforementioned conversion information includes the number of channels corresponding to the color components, The image processing apparatus according to item 13, characterized in that it includes multiple channels when the slope of the high-luminance portion of the gamma curve is below a threshold. [Item 15] The image processing apparatus according to item 14, characterized in that the plurality of channels include at least two channels when generating conversion information for correcting the saturation component. [Item 16] The image processing apparatus according to item 14, characterized in that the plurality of channels include at least three channels when generating conversion information for correcting the hue component. [Item 17] The image processing apparatus according to item 5, characterized in that the size of the conversion information for the first format is less than or equal to the size of the conversion information for the second format. [Item 18] The image processing apparatus according to item 17, characterized in that, when the size of the conversion information for the first format is less than or equal to the size of the conversion information for the second format, the region in which the color ratio of the combined region exceeds a threshold is given a higher resolution. [Item 19] The format of the aforementioned conversion information includes the number of channels corresponding to the color components, The image processing apparatus according to item 17, wherein the number of channels for the conversion information of the first format is 3 channels, and the number of channels for the second format is 1 channel. [Item 20] The image processing apparatus according to item 7, characterized in that the size of the conversion information for the first format is less than or equal to the size of the conversion information for the second format. [Item 21] The image processing apparatus according to item 20, characterized in that, when the size of the conversion information for the first format is less than or equal to the size of the conversion information for the second format, the region in which the color ratio of the combined region exceeds a threshold is given a higher resolution. [Item 22] The format of the aforementioned conversion information includes the number of channels corresponding to the color components, The image processing apparatus according to item 20, wherein the number of channels for the conversion information of the first format is 3 channels, and the number of channels for the second format is 1 channel. [Item 23] The image processing apparatus according to any one of items 1 to 22, characterized in that the image format is HDR (High Dynamic Range) or SDR (Standard Dynamic Range). [Item 24] The image processing apparatus according to any one of items 1 to 23, characterized in that the conversion information is a gain map for mutually converting between an HDR (High Dynamic Range) format image and an SDR (Standard Dynamic Range) format image. [Item 25] An image processing method performed by an image processing device, Steps to acquire a RAW image, The steps include generating a first image from the aforementioned RAW image, The process includes the step of generating conversion information used when generating a second image with a different format from the first image from the first image, The image processing method is characterized in that, in the step of generating the conversion information, it generates conversion information in a format corresponding to the characteristics of the second image. [Item 26] A program for causing a computer to function as one of the means of an image processing apparatus as described in any of items 1 through 24. [Explanation of Symbols]

[0117] 100...Imaging device, 101...Optical system, 102...Imaging unit, 103...Control unit, 104...Primary storage device, 105...Secondary storage device, 106...Photometric sensor, 107...Image processing unit, 108...Recording medium, 109...Communication unit, 110...Display unit, 111...Operation unit

Claims

1. A means for acquiring RAW images, Image processing means for generating a first image from the RAW image, The system includes a generation means for generating conversion information used when generating a second image with a different format from the first image from the first image, The image processing apparatus is characterized in that the generation means generates format conversion information according to the characteristics of the second image.

2. The image processing apparatus according to claim 1, further comprising determination means for determining the characteristics of the second image.

3. The image processing apparatus according to claim 2, characterized in that the determination means determines a mode relating to the grayscale characteristics of the second image.

4. The image processing apparatus according to claim 3, characterized in that it has a selection means for selecting the aforementioned mode.

5. The image processing apparatus according to claim 3, characterized in that the generation means generates conversion information for the first format when the mode is a mode for generating the second image from the RAW image, and generates conversion information for the second format when the mode is a mode for generating the second image from the first image.

6. The image processing apparatus according to claim 2, characterized in that the determination means performs scene recognition relating to the grayscale characteristics of the second image.

7. The image processing apparatus according to claim 6, characterized in that the generation means generates conversion information for a first format if the scene is a first scene, and generates conversion information for a second format if the scene is a second scene.

8. The aforementioned scene recognition is a process of recognizing a subject from the RAW image, The first scene described above is a mode that prioritizes the tonal range of a given subject, The image processing apparatus according to claim 7, characterized in that the first scene is a mode that emphasizes the gradation of the entire subject.

9. The image processing apparatus according to claim 8, characterized in that the determination means determines that the scene is the first scene if the size of the predetermined subject is greater than or equal to a threshold, and determines that the scene is the second scene if the size of the predetermined subject is less than a threshold.

10. The image processing apparatus according to claim 9, wherein the determination means determines the dynamic range based on the RAW image and the first image if the size of the predetermined subject is less than a threshold, determines it to be the first scene if the dynamic range is greater than or equal to a threshold, and determines it to be the second scene if the dynamic range is less than a threshold.

11. The image processing apparatus according to claim 1, further comprising a storage means for adding and storing conversion information generated by the generation means to the first image or the second image.

12. The image processing apparatus according to claim 1, characterized in that the second image characteristic is the gradation characteristic of the gamma curve.

13. The image processing apparatus according to claim 12, characterized in that the gradation characteristics of the gamma curve are the gradation characteristics of the high-luminance area.

14. The format of the aforementioned conversion information includes the number of channels corresponding to the color components, The image processing apparatus according to claim 13, characterized in that it includes a plurality of channels when the slope of the high-luminance portion of the gamma curve is below a threshold.

15. The image processing apparatus according to claim 14, characterized in that the plurality of channels include at least two channels when generating conversion information for correcting the saturation component.

16. The image processing apparatus according to claim 14, characterized in that the plurality of channels include at least three channels when generating conversion information for correcting the hue component.

17. The image processing apparatus according to claim 5, characterized in that the size of the conversion information for the first format is less than or equal to the size of the conversion information for the second format.

18. The image processing apparatus according to claim 17, characterized in that, when the size of the conversion information for the first format is less than or equal to the size of the conversion information for the second format, the region in which the color ratio of the combined region exceeds a threshold is given a higher resolution.

19. The format of the aforementioned conversion information includes the number of channels corresponding to the color components, The image processing apparatus according to claim 17, wherein the number of channels for the conversion information of the first format is 3 channels, and the number of channels for the second format is 1 channel.

20. The image processing apparatus according to claim 7, characterized in that the size of the conversion information for the first format is less than or equal to the size of the conversion information for the second format.

21. The image processing apparatus according to claim 20, characterized in that, when the size of the conversion information for the first format is less than or equal to the size of the conversion information for the second format, the region in which the color ratio of the combined region exceeds a threshold is given a higher resolution.

22. The format of the aforementioned conversion information includes the number of channels corresponding to the color components, The image processing apparatus according to claim 20, wherein the number of channels for the conversion information of the first format is 3 channels, and the number of channels for the second format is 1 channel.

23. The image processing apparatus according to claim 1, characterized in that the image format is HDR (High Dynamic Range) or SDR (Standard Dynamic Range).

24. The image processing apparatus according to claim 1, characterized in that the conversion information is a gain map for mutually converting between an image in HDR (High Dynamic Range) format and an image in SDR (Standard Dynamic Range) format.

25. An image processing method performed by an image processing device, Steps to acquire a RAW image, The steps include generating a first image from the RAW image, The process includes the step of generating conversion information used when generating a second image with a different format from the first image from the first image, The image processing method is characterized in that, in the step of generating the conversion information, it generates conversion information in a format corresponding to the characteristics of the second image.

26. A program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 24.

Images