Image processing apparatus, image processing method, program and storage medium

Abstract

To provide a technique which allows a user to easily (intuitively) change a display range to a desired luminance range and which can properly display an image in the display range after change without changing the display of the image more than necessary.SOLUTION: An image processing apparatus comprises: input means which receives a prescribed user operation on an image displayed on a display part; change means which performs control so as to increase a display range of the display part when the input means receives the prescribed user operation; processing means which performs gradation change for improving the gradation of a high luminance part for the image on the basis of the display range increased by the change means; and control means which performs control so as to make the increase of the display range by the prescribed user operation effective when the image satisfies a prescribed condition, and so as not to make the increase of the display range by the prescribed user operation effective when the image does not satisfy the prescribed condition.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, a program, and a storage medium, and more particularly to a technique for changing the dynamic range (luminance range) of a display. [Background technology]

[0002] In recent years, HDR (High Dynamic Range) has become increasingly popular as an input device for smartphones and digital cameras. There are currently commercially available models capable of recording images with a wide dynamic range (brightness range), such as HDR images. HDR images include images with gradation values (signal values) that correspond to absolute luminance. Gradation values that correspond to absolute luminance are, for example, based on the ITU-R (Radiocommunication Sector of ITU) Gradation values conforming to the EOTF (Electro-Optical Transfer Function) specified in BT.2100 and SMPTE (Society of Motion Picture and Television Engineers) These are tone values that conform to the EOTF specified in ST.2084. These EOTFs are also called PQ (Perceptual Quantization) curves. EOTFs such as PQ curves correspond to functions that convert tone values (luminance tone values; pixel values) into luminance.

[0003] In addition, HDR displays that can display HDR images with a wide dynamic range (display range) are becoming more common, and smartphones equipped with HDR displays are also being commercialized. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-50840 Summary of the Invention [Problem to be solved by the invention]

[0005] However, smartphones and other devices are often used with their display brightness (the brightness of the display) set low to reduce power consumption and extend operating time. Setting the display brightness low lowers the upper limit of the display brightness, and therefore the maximum brightness of the display.

[0006] For this reason, taking the example of displaying an HDR image recorded according to the above-described PQ curve on an HDR display, a situation may arise in which the upper limit of luminance of the HDR display is lower than the luminance of the HDR image itself (the luminance corresponding to the pixel values of the HDR image in the PQ curve). In such a situation, areas of the HDR image where the luminance of the HDR image itself is higher than the upper limit of luminance of the HDR display (for example, highlight areas) may be displayed with blown-out highlights. While it is conceivable that a user could increase the upper limit of luminance of the HDR display to a desired luminance to suppress blown-out highlights (the user could change (expand) the display range to a desired luminance range), this is difficult, and the display range may end up being expanded more than necessary.

[0007] Patent Document 1 discloses a technique for converting (compressing) the dynamic range of an HDR image so that the image can be displayed on a display device with the display brightness set low. Specifically, the technique disclosed in Patent Document 1 compresses the entire dynamic range from dark to bright areas. This technique can suppress overexposure, but the display of dark areas changes. Specifically, the gradation in dark areas (gradation resolution; brightness gradation; brightness gradation resolution) decreases, and an overall dark image is displayed.

[0008] Therefore, the present invention aims to provide a technology that allows a user to easily (intuitively) change the display range to a desired brightness range, and that allows an image to be displayed favorably in the changed display range without changing the image display more than necessary. [Means for solving the problem]

[0009] The image processing device of the present invention is characterized by having an input means for accepting a specified user operation on an image displayed on a display unit, a modification means for controlling the display unit to increase the display range when the specified user operation is accepted by the input means, a processing means for performing gradation conversion on the image to improve the gradation of high-brightness areas based on the display range increased by the modification means, and a control means for enabling the increase in the display range due to the specified user operation when the image satisfies specified conditions, and for controlling the increase in the display range due to the specified user operation not to be enabled when the image does not satisfy the specified conditions. [Effects of the Invention]

[0010] According to the present invention, a user can easily (intuitively) change the display range to a desired brightness range. Furthermore, the image can be displayed appropriately in the changed display range without changing the image display more than necessary. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram illustrating an example of the configuration of an image processing device according to a first embodiment. [Figure 2] 2 is a block diagram showing an example of the configuration of a tone conversion processing unit according to the first embodiment. FIG. [Figure 3] 4 is a flowchart illustrating an example of display control according to the first embodiment. [Figure 4] 10 is a flowchart illustrating an example of an image display process according to the first embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of an EOTF and an OETF according to the first embodiment. [Figure 6] FIG. 4 is a diagram illustrating an example of a luminance conversion characteristic according to the first embodiment. [Figure 7] FIG. 2 is a diagram showing a display example according to the first embodiment. [Figure 8] FIG. 10 is a block diagram showing an example of the configuration of a tone conversion processing unit according to a second embodiment. [Figure 9] 10 is a flowchart illustrating an example of display control according to the second embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a blending ratio used in HDR blending according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Example 1 A first embodiment of the present invention will be described below. An image processing device according to the first embodiment performs gradation conversion of an image and displays the gradation-converted image on a display unit. When a user performs a range designation operation to designate a dynamic range (brightness range), the image processing device displays an item for the user operation together with the gradation-converted image. Then, in response to the range designation operation, the image processing device changes the display range setting, which is the dynamic range of display on the display unit, based on the designated range, which is the dynamic range designated by the user operation. Then, the image processing device changes the gradation conversion characteristics in accordance with the changed display range.

[0013] FIG. 1 is a block diagram showing an example of the configuration of an image processing device 100 according to a first embodiment. The image processing device 100 includes a control unit 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a UI (User Interface) unit 104, an image processing unit 105, and a display unit 106. They are connected to each other by a switch 107.

[0014] The control unit 101 controls the operation of each unit of the image processing device 100. For example, the control unit 101 is a CPU (Central Processing Unit), and controls the operation of each unit of the image processing device 100 by reading a program from a ROM 102, expanding it into a RAM 103, and executing it.

[0015] The ROM 102 is a non-volatile memory that can electrically erase and record various data, and stores programs executed by the control unit 101, parameters required for the operation of each unit of the image processing device 100, images (image data) to be processed, etc. The images to be processed may be acquired from an external device.

[0016] The RAM 103 is a volatile memory that can electrically erase and record various data. The control unit 101 loads programs stored in the ROM 102 into the RAM 103, and temporarily records parameters, images, and the like stored in the ROM 102 into the RAM 103. The control unit 101 also temporarily records parameters, images, and the like generated by each unit into the RAM 103.

[0017] The UI unit 104 accepts user operations on the image processing device 100. For example, the UI unit 104 is a pointing device such as a touch panel or a mouse, or a keyboard. Note that the UI unit 104 (operation device) may be an external device or a part of it, and the image processing device 100 may have an interface for connecting to the external device so as to acquire information corresponding to the user operations.

[0018] The image processing unit 105 performs various image processing such as white balance adjustment, color interpolation, gamma processing, etc. on the image (image to be processed) stored in the RAM 103. The image processing unit 105 has a gradation conversion processing unit 200 that performs gradation conversion of the image.

[0019] The display unit 106 displays an image (image after gradation conversion) stored in the RAM 103. The display unit 106 also displays items (UI) for user operation. For example, the display unit 106 is a display device such as a liquid crystal panel or an organic EL (Electro Luminescence) panel. The display unit 106 may be an external device such as a smartphone or a television device, or a part of the external device, and the image processing device 100 may have an interface for connecting to the external device so as to control the display unit 106.

[0020] 2 is a block diagram showing an example configuration of the gradation conversion processing unit 200. The gradation conversion processing unit 200 has a conversion characteristic determination unit 201 and a gradation conversion unit 202. The conversion characteristic determination unit 201 determines conversion characteristics for gradation conversion. The gradation conversion unit 202 performs gradation conversion of the image using the conversion characteristics determined by the conversion characteristic determination unit 201.

[0021] Fig. 3 is a flowchart showing an example of display control performed by the image processing device 100. For example, the display control in Fig. 3 is started when the control unit 101 reads a program from the ROM 102, expands it in the RAM 103, and executes it in response to startup of the image processing device 100.

[0022] In S301, the control unit 101 checks the setting of the display range, which is the dynamic range of the display on the display unit 106. In the first embodiment, the minimum luminance of the display range is fixed at 0 nit, and only the maximum luminance of the display range can be changed. Therefore, the setting of the display range can also be said to be the setting of the maximum luminance of the display range. For example, if the maximum luminance of the display range is set to 500 nit, the display range of 0 to 500 nit is set, and the display range If the maximum brightness of the image is set to 800 nit, the display range is set to 0 to 800 nit. Note that the maximum brightness of the display range may be fixed and only the minimum brightness of the display range may be changeable, or both the minimum and maximum brightness of the display range may be changeable.

[0023] In S302, the control unit 101 controls the display unit 106 to display the image to be processed according to the current display range (the display range confirmed in S301) in response to a user operation using the UI unit 104. The user operation is, for example, a user operation for selecting and displaying the image to be processed. The image to be processed is not particularly limited, but is assumed here to be a captured HDR image (captured HDR image). An HDR (High Dynamic Range) image is an image with a larger number of bits and a wider dynamic range than an SDR (Standard Dynamic Range) image, such as an image conforming to ITU-R BT.709.

[0024] A specific example of the process of S302 will be described with reference to the flowchart of Fig. 4. Fig. 4 is a flowchart showing an example of the process of S302 and S307 (image display process) described later.

[0025] In S401, the conversion characteristic determination unit 201 determines conversion characteristics for tone conversion of the captured HDR image according to the display range (maximum brightness of the display range) confirmed in S301. While the captured HDR image is not particularly limited, it is assumed here that the image conforms to a PQ (Perceptual Quantization) curve. Examples of PQ curves include the EOTF (Electro-Optical Transfer Function) defined in ITU-R (Radiocommunication Sector of ITU) BT.2100 and the EOTF defined in SMPTE (Society of Motion Picture and Television Engineers) ST.2084.

[0026] FIG. 5A shows an example of a PQ curve (EOTF). The EOTF corresponds to a function that converts gradation values (luminance gradation values; pixel values) to luminance. Specifically, the EOTF in FIG. 5A is expressed by the following equation (1). p_in is the input value of the EOTF, and is a value obtained by normalizing gradation values (such as R, G, and B values) to 0.0 to 1.0. p_in=1.0 corresponds to the upper limit of the gradation value (the upper limit according to the number of bits), and p_in=0.0 corresponds to the lower limit of the gradation value. For example, if the number of bits of the gradation value is 10 bits, the upper limit of the gradation value is 1023 and the lower limit of the gradation value is 0. p_out is the output value of the EOTF, and is a value obtained by normalizing gradation values (such as R, G, and B values) proportional to luminance to 0.0 to 1.0. For example, p_out=0.0 corresponds to 0 nit, and p_out=1.0 corresponds to 10,000 nit. max[x, y] is a function that outputs the larger value of x and y. In the first embodiment, the display unit 106 displays an image at a brightness according to the output value p_out.

number

[0027] Figure 5(B) shows an example of an OETF (Opto-Electronic Transfer Function) with characteristics opposite to those of the OETF in Figure 5(A). The OETF corresponds to a function that converts luminance into a gradation value. Specifically, the OETF in Figure 5(B) is expressed by the following equation 2. q_in is the input value of the OETF, and is a gradation value obtained by normalizing the gradation value (such as R value, G value, or B value) proportional to luminance to a range of 0.0 to 1.0. For example, q_in = 0. 0 corresponds to 0 nit, and q_in=1.0 corresponds to 10000 nit. q_out is the output value of the OETF, and is a value obtained by normalizing the gradation value (R value, G value, B value, etc.) to 0.0 to 1.0. q_out=1.0 corresponds to the upper limit of the gradation value (upper limit according to the number of bits), and q_out=0.0 corresponds to the lower limit of the gradation value. For example, if the number of bits for the gradation value is 10 bits, the upper limit of the gradation value is 1023 and the lower limit of the gradation value is 0.

number

[0028] As described above, in Example 1, the captured HDR image is an image that conforms to the EOTF (PQ curve) of Fig. 5(A). In other words, the captured HDR image is an image captured with the OETF of Fig. 5(B). The gradation value of the captured HDR image corresponds to the input value p_in of the EOTF and also corresponds to the output value q_out of the OETF.

[0029] The characteristic 600 (luminance conversion characteristic) in FIG. 6A corresponds to the characteristic obtained by multiplying the EOTF in FIG. 5A by the OETF in FIG. 5B. The horizontal axis in FIG. 6A represents the luminance of the image before tone conversion (the input value q_in of the OETF), and the vertical axis in FIG. 6A represents the luminance of the image after tone conversion (the output value p_out of the EOTF). Here, it is assumed that the maximum luminance of the display range is luminance Y1. In this case, the conversion characteristic determination unit 201 determines the characteristic 601 according to the luminance Y1 and corrects the EOTF in FIG. 5A so as to realize the characteristic 601. The characteristic 601 or the corrected EOTF is the conversion characteristic determined in S401. As shown in FIG. 6A, in the luminance range (luminance range equal to or less than luminance X1) corresponding to the display range (luminance range equal to or less than luminance Y1) in the characteristic 600, the characteristic 601 substantially matches (is identical to) the characteristic 600. For this reason, luminances below luminance X1 are converted to approximately the same luminance (same) in both characteristics 600 and 601 by gradation conversion, and are displayed according to the PQ curve. This enables display that matches the human visual characteristics. However, with characteristics 601, luminances higher than luminance X1 are converted (clipped) to luminance Y1 by gradation conversion, resulting in overexposed images.

[0030] In S402, the gradation conversion unit 202 performs gradation conversion on the captured HDR image in accordance with the conversion characteristics determined in S401. Specifically, the gradation conversion unit 202 performs gradation conversion on the captured HDR image in accordance with the corrected EOTF. As a result, an HDR image for display is generated as a gradation-converted image.

[0031] In S403, the control unit 101 outputs the HDR image for display generated in S402 to the display unit 106. Fig. 7(A) shows a display example of the HDR image for display generated in S402. As described above, in the characteristic 601 of Fig. 6(A), luminance higher than luminance X1 is converted to luminance Y1 by gradation conversion and is displayed as blown-out highlights. For this reason, although the sun and clouds exist in the captured HDR image (image before gradation conversion), the sun and clouds are blown-out highlights in the display example of Fig. 7(A).

[0032] Returning to the explanation of FIG. 3, in S303, the control unit 101 determines whether or not a blown-out suppression start operation has been performed as a user operation using the UI unit 104 to start suppressing blown-out highlights in the HDR image for display displayed in S302. For example, when the sun and clouds are blown out as in the display example of FIG. 7(A) and the user wants to suppress the blown-out highlights so that the sun and clouds can be recognized as in the display example of FIG. 7(B), the user performs the blown-out suppression start operation. The blown-out suppression start operation is, for example, touching a button displayed on the display unit 106 or pressing a button (physical button) provided on the image processing device 100. As a more specific example, An item such as "Burnt Out Suppression" is displayed in a menu of adjustment items such as "Exposure," "Contrast," "Color," "White Balance (Color Temperature)," "Sharpness," etc. When this "Burnt Out Suppression" item is selected, the control unit 101 determines that an operation to start blown out highlights suppression has been performed and proceeds to step S304; otherwise (when an operation to start blown out highlights suppression has not been performed), the display control in FIG. 3 ends.

[0033] In S304, the control unit 101 controls the display unit 106 to display an item (UI) for a user operation (range specification operation) to specify a dynamic range together with the HDR image for display. Specifically, as shown in FIG. 7C , the control unit 101 controls the display unit 106 to display a slider 701. The user can adjust the dynamic range (specified range) specified by the user by moving the slider 701. Specifically, the user can adjust the maximum luminance of the specified range by moving the slider 701. Therefore, the range specification operation can also be considered a user operation for specifying the maximum luminance of the specified range. As will be described in detail later, adjusting the specified range adjusts the degree of blown-out highlights in the HDR image for display. Therefore, the slider 701 can also be considered an item for adjusting the degree of blown-out highlights. The operation of moving the slider 701 is similarly used to adjust the aforementioned items such as "exposure," "contrast," "color," "white balance (color temperature)," and "sharpness." In other words, the user can perform blown-out highlight suppression operations with the same feel as when adjusting other adjustment items.

[0034] Furthermore, this blown-out highlight suppression is particularly effective for images, such as HDR images, in which a relatively large amount of gradation remains in the high-brightness areas as image information. Therefore, at least before step S303, it may be determined whether the image to be displayed is an HDR image. If it is an HDR image, the blown-out highlight suppression process of this embodiment may be enabled, and if it is an SDR image, the blown-out highlight suppression process may be disabled. Disabling the blown-out highlight suppression process may be achieved, for example, by hiding or graying out the "Blown-out Highlight Suppression" option in the aforementioned adjustment item menu. Alternatively, the slider 701 may be made unmovable. Whether an image is an HDR image may be determined, for example, by referencing the image extension or a specific item in the metadata. Furthermore, it may also be determined whether the image bit depth is a specific value. Furthermore, it may also be determined whether the image conforms to the PQ curve based on the image metadata, etc.

[0035] Note that items such as the slider 701 may be generated by the control unit 101 or may be stored in advance in the ROM 102. The initial position of the slider 701 may be, or may not be, a position on a bar indicating an area in which the slider 701 can move based on the maximum luminance of the current display range (the maximum luminance confirmed in S301). For example, the initial position of the slider 701 may be a position on the bar corresponding to a luminance that is approximately the same as (the same as) the maximum luminance of the current display range. If the upper limit of the maximum luminance of the specified range or the display range is 1000 nit and the maximum luminance of the current display range is 500 nit, the initial position of the slider 701 may be the center of the bar. If the minimum luminance of the display range is changeable, the item for the range specification operation may include an item (such as a slider) for specifying the minimum luminance of the specified range. The item for the range specification operation is not limited to a slider and may have a form similar to that of an item for specifying the exposure during shooting.

[0036] In S305, the control unit 101 acquires information on the current designated range (the dynamic range designated by the user in S304). Specifically, the control unit 101 acquires information on the brightness corresponding to the current position of the slider 701 as information on the maximum brightness of the current designated range. For example, if the upper limit of the maximum brightness of the designated range or display range is 1000 nits and the slider 701 is located at the center of the bar, the maximum brightness of the current designated range (the brightness corresponding to the current position of the slider 701) is 500 nits.

[0037] In S306, the control unit 101 changes the setting of the display range from the setting confirmed in S301 based on the current designated range. Specifically, the control unit 101 changes the setting of the maximum luminance of the display range from the setting confirmed in S301 based on the maximum luminance of the current designated range. The display range after the change does not have to be substantially identical to the designated range, but in the first embodiment, the control unit 101 changes the setting of the display range so that the display range becomes substantially identical (the same) as the designated range. Specifically, the control unit 101 changes the setting of the maximum luminance of the display range so that the maximum luminance of the display range becomes substantially identical (the same) as the maximum luminance of the designated range. For example, if the maximum luminance of the designated range is 500 nit, the maximum luminance of the display range is changed to 500 nit.

[0038] In S307, the control unit 101 performs control to update the display of the HDR image for display according to the display range changed in S306. In the first embodiment, the display of the HDR image for display is updated by updating the HDR image for display. Then, the control unit 101 ends the display control in Fig. 3. Note that if the operation to start whiteout suppression is performed again, the processes of S304 to S307 will be performed again.

[0039] A specific example of the process of S307 will be described using the flowchart of Fig. 4. However, a description of the same process as the process of S302 will be omitted as appropriate.

[0040] In S401, the conversion characteristic determination unit 201 changes the conversion characteristics for the gradation conversion of the captured HDR image according to the display range (maximum luminance of the display range) after the change in S306. Here, assume that the maximum luminance of the display luminance is changed from luminance Y1 to luminance Y2 (>Y1). In this case, the conversion characteristic determination unit 201 determines the characteristics 602 in FIG. 6B according to the luminance Y2 and corrects the EOTF in FIG. 5A so that the characteristics 602 are realized. In the characteristics 602, since the maximum luminance of the display range is increased from luminance Y1 to luminance Y2, the luminance range in which gradation can be expressed is expanded from the luminance range equal to or less than luminance X1 to the luminance range equal to or less than luminance X2 (>X1). Furthermore, in the portion of the changed display range that overlaps with the display range before the change (the luminance range equal to or less than luminance X1), the characteristics 602 approximately coincide with the characteristics 601. Therefore, even when the characteristics 601 are changed to the characteristics 602, the display of the HDR image for display can be maintained for the luminance range equal to or less than luminance X1.

[0041] In the characteristics 601 and 602, the luminance of the HDR image for display (luminance of the image after gradation conversion; output value p_out of the OETF) is approximately proportional to the luminance of the captured HDR image (luminance of the image before gradation conversion; input value q_in of the OETF) in the display range, but is not limited to this. In the display range, the luminance of the HDR image for display may change nonlinearly with changes in the luminance of the captured HDR image.

[0042] In S402, the gradation conversion unit 202 generates (updates) an HDR image for display by performing gradation conversion on the captured HDR image in accordance with the conversion characteristics (changed conversion characteristics) determined in S401.

[0043] In S403, the control unit 101 outputs the display HDR image generated in S402 to the display unit 106. This updates the display of the display HDR image. FIG. 7B shows a display example of the display HDR image generated in S402 (updated display HDR image). As described above, in the characteristic 602 of FIG. 6B, the luminance range in which gradation can be expressed is expanded from the luminance range equal to or less than luminance X1 to the luminance range equal to or less than luminance X2. Therefore, in the display example of FIG. 7B, blown-out highlights are suppressed, and the sun and clouds that are blown-out in the display example of FIG. 7A can be recognized.

[0044] As described above, according to the first embodiment, when a range designation operation is performed, an item for the range designation operation is displayed together with the image after gradation conversion. Then, in response to the range designation operation, the display range setting is changed based on the designated range, and the display of the image after gradation conversion is updated according to the changed display range. This allows the user to check the image after gradation conversion while performing the range designation operation, and easily (intuitively) change the display range to a desired brightness range. Furthermore, according to the first embodiment, the conversion characteristics of the gradation conversion are changed so that the display of the image after gradation conversion is maintained for a portion of the changed display range that overlaps with the display range before the change. This allows the image to be displayed favorably in the changed display range without changing the display of the image more than necessary.

[0045] Note that the control unit 101 may perform control so as to issue a predetermined notification (warning) when the specified range includes a luminance outside the maximum settable display range, for example, when the maximum luminance of the specified range is higher than the upper limit of the maximum luminance of the display range. The notification is performed, for example, by displaying an item, outputting a sound, or illuminating a lamp (such as a light-emitting diode). The display unit, speaker, lamp, etc. for notification may or may not be part of the image processing device 100.

[0046] The image processing device 100 may include a detection unit (light sensor) that detects ambient light relative to the display unit 106. The control unit 101 may then perform control to limit the luminance range that can be specified as the designated range based on the detection result of the ambient light by the detection unit. For example, the control unit 101 may limit the maximum luminance of the luminance range that can be specified as the designated range to a lower luminance as the ambient light becomes darker. This prevents the HDR image for display from appearing too dazzling due to the influence of ambient light. The control unit 101 may change the display range settings based on the detection result of the designated range and the ambient light, without limiting the luminance range that can be specified as the designated range. The detection unit may be an external device or a part of the external device, and the image processing device 100 may have an interface for connecting to the external device so as to acquire the detection result of ambient light.

[0047] When the image to be processed (image before tone conversion) is switched to another image, the control unit 101 may return the display range setting to the setting before the change (the setting confirmed in S301). If the maximum brightness of the display range is always increased based on the range specification operation, power consumption can be reduced by returning the display range setting to the setting before the change.

[0048] The control unit 101 may perform control so that the changed display range is recorded in the storage unit in association with the image before the tone conversion. The storage unit may be the ROM 102, or may not be a ROM. In this way, when an image whose display range has been changed in the past is displayed again on the image processing device 100 or another image processing device, the previous display can be reproduced based on the changed display range. The storage unit may be built into the image processing device 100, or may be a storage device that is detachable from the image processing device 100.

[0049] <Example 2> A second embodiment of the present invention will be described below. The image processing device according to the second embodiment performs the same processing as that described in the first embodiment. Furthermore, the image processing device according to the second embodiment further performs a process of determining an image range, which is the dynamic range of the image itself, based on the specified range, and a process of generating an image having the determined image range. Note that, below, differences from the first embodiment (such as configuration and processing) will be described in detail, and a description of similarities with the first embodiment will be omitted as appropriate.

[0050] The image processing device according to the second embodiment has the same configuration as the image processing device 100 (FIG. 1) according to the first embodiment. However, the image processing device according to the second embodiment has the same configuration as the image processing device 100 (FIG. 1) according to the first embodiment. 00, the gradation conversion processing unit 800 has a gradation conversion processing unit 800 shown in FIG. 8. FIG. 8 is a block diagram showing an example configuration of the gradation conversion processing unit 800. Like the gradation conversion processing unit 200, the gradation conversion processing unit 800 has a conversion characteristic determination unit 201 and a gradation conversion unit 202. The gradation conversion processing unit 800 further has an HDR synthesis unit 801. The HDR synthesis unit 801 synthesizes multiple images with different exposures to generate a single synthesized image. The dynamic range of the synthesized image can be freely changed by changing the multiple images to be synthesized.

[0051] Fig. 9 is a flowchart showing an example of display control performed by the image processing device according to the second embodiment. For example, the display control in Fig. 9 is started when the control unit 101 reads a program from the ROM 102, expands it in the RAM 103, and executes it in response to the start of the image processing device according to the second embodiment. In the display control in Fig. 9, the processes of S301 to S307 are the same as those in the first embodiment (Fig. 3).

[0052] In S901, the control unit 101 determines an image range, which is the dynamic range of the image itself, based on the current designated range. Specifically, the control unit 101 determines the maximum luminance of the image range based on the maximum luminance of the current designated range. The image range does not have to be approximately the same as the designated range, but in the second embodiment, the control unit 101 determines an image range that is approximately the same as (the same as) the designated range. Specifically, the control unit 101 determines a luminance that is approximately the same as (the same as) the maximum luminance of the designated range as the maximum luminance of the image range. For example, if the maximum luminance of the designated range is 4000 nit, 4000 nit is determined as the maximum luminance of the image range.

[0053] In S902, the HDR synthesis unit 801 synthesizes a plurality of images with different exposures to generate a synthesized image having the image range determined in S901 as an image before gradation conversion. At this time, the HDR synthesis unit 801 determines the plurality of images (exposure, number, etc.) to be used for synthesis based on the image range. In the second embodiment, in S307 (S402), the gradation conversion unit 202 performs gradation conversion on the synthesized image generated in S902 to generate an HDR image for display.

[0054] A specific example of the processing of S902 will be described below when the maximum brightness of the image range is 4000 nit, that is, the image range is a brightness range of 0 to 4000 nit. Here, it is assumed that the dynamic range of the properly exposed image is a brightness range of 0 to 1000 nit, and that a composite image having a dynamic range of 0 to 2000 nit can be generated by combining the properly exposed image with an image that is one step underexposed. It is also assumed that a composite image having a dynamic range of 0 to 4000 nit can be generated by combining the properly exposed image with an image that is two steps underexposed. The properly exposed image is an image captured at the proper exposure, the one-step underexposed image is an image captured at an exposure that is one step lower than the proper exposure, and the two-step underexposed image is an image captured at an exposure that is two steps lower than the proper exposure.

[0055] As described above, a composite image having a dynamic range of 0 to 4000 nit can be generated by combining a properly exposed image with a two-step underexposed image. However, it is not necessarily the case that a composite image in which gradation is expressed throughout the entire dynamic range of 0 to 4000 nit can be generated. For this reason, the HDR combining unit 801 combines the properly exposed image, the one-step underexposed image, and the two-step underexposed image so as to generate a composite image in which gradation is expressed throughout the entire dynamic range of 0 to 4000 nit (without gradation jumps). Note that, as long as a composite image in which gradation is expressed throughout the entire dynamic range of 0 to 4000 nit can be generated, the HDR combining unit 801 may combine the properly exposed image and the two-step underexposed image without using the one-step underexposed image. The number of images used for combining is not particularly limited, and may be more than three.

[0056] An example of the synthesis method (HDR synthesis; weighted synthesis) by the HDR synthesis unit 801 is shown in FIG. 10 shows an example of the composition ratios (weights) used when combining a properly exposed image, an underexposed image by one step, and an underexposed image by two steps. A composition ratio 1001 is the composition ratio for the properly exposed image, a composition ratio 1002 is the composition ratio for the underexposed image by one step, and a composition ratio 1003 is the composition ratio for the underexposed image by two steps.

[0057] As shown in FIG. 10 , the HDR synthesis unit 801 increases the luminance of the one-step underexposure image and the two-step underexposure image so that the luminance of the one-step underexposure image and the two-step underexposure image matches the luminance of the proper exposure image. Specifically, the HDR synthesis unit 801 increases the luminance of the one-step underexposure image so that the luminance of the one-step underexposure image approximately matches (matches) the luminance of the proper exposure image for portions where gradation can be expressed in both the one-step underexposure image and the proper exposure image. Similarly, the HDR synthesis unit 801 increases the luminance of the two-step underexposure image so that the luminance of the two-step underexposure image approximately matches (matches) the luminance of the proper exposure image for portions where gradation can be expressed in both the two-step underexposure image and the proper exposure image. In FIG. 10 , the luminance of the one-step underexposure image is increased by two times, and the luminance of the two-step underexposure image is increased by four times.

[0058] The HDR synthesis unit 801 then synthesizes the properly exposed image, the one-step underexposed image, and the two-step underexposed image according to synthesis ratios 1001 to 1003. The synthesis ratio 1001 of the properly exposed image is 100% up to brightness X1, and decreases linearly from 100% to 0% as brightness increases from X1 to X2. The synthesis ratio 1002 of the one-step underexposed image increases from 0% to 100% in proportion to the increase in brightness from X1 to X2, is 100% from X2 to X3, and decreases linearly from 100% to 0% as brightness increases from X3 to X4. The synthesis ratio 1003 of the two-step underexposed image increases from 0% to 100% in proportion to the increase in brightness from X3 to X4, and is 100% from X4 onwards. Note that the synthesis ratios 1001 to 1003 are not limited to those shown in FIG. 10 . The synthesis ratios 1001 to 1003 may be determined in any way as long as the sum of the synthesis ratios 1001 to 1003 is always 100%.

[0059] As described above, according to the second embodiment, an image range is determined based on the designated range, and multiple images with different exposures are combined to generate an image having the image range as an image before tone conversion. This makes it possible to display an image while further suppressing image quality degradation such as blown-out highlights.

[0060] While the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various modifications within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0061] The various controls described above as being performed by the control unit 101 may be performed by a single piece of hardware, or the entire device may be controlled by multiple pieces of hardware (e.g., multiple processors or circuits) sharing the processing.

[0062] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0063] 100: Image processing device 101: Control unit 104: UI unit 105: Image processing unit 200: Gradation conversion processing unit 201: Conversion characteristic determination unit 202: Gradation conversion unit 800: Gradation conversion processing section

Claims

1. an input means for receiving a predetermined user operation on an image displayed on the display unit; a change means for controlling the display unit to increase a display range when the predetermined user operation is accepted by the input means; a processing means for performing gradation conversion for improving the gradation of a high-brightness portion of the image based on the display range increased by the changing means; a control means for validating an increase in the display range due to the specified user operation when the image satisfies a predetermined condition, and for not validating an increase in the display range due to the specified user operation when the image does not satisfy the predetermined condition; 1. An image processing device comprising:

2. The increase in the display range includes an increase in the maximum display luminance of the display unit.

2. The image processing device according to claim 1, wherein:

3. The predetermined condition is a condition related to the dynamic range of the image.

3. The image processing device according to claim 1, wherein the image processing device is a computer.

4. The predetermined condition is a condition regarding the number of bits of the image.

3. The image processing device according to claim 1, wherein the image processing device is a computer.

5. The image that satisfies the predetermined condition is an HDR (High Dynamic Range) image recorded in a predetermined format.

5. The image processing device according to claim 1, wherein the image processing device is a computer.

6. The image that does not satisfy the predetermined conditions is an SDR (Standard Dynamic Range) image.

6. The image processing device according to claim 1, wherein the image processing device is a computer.

7. The processing means changes the conversion characteristics of the tone conversion so that the display luminance of the image after the tone conversion is approximately proportional to the display luminance of the image before the tone conversion in the display range increased by the changing means.

7. The image processing device according to claim 1, wherein the image processing device is a computer.

8. The control means further controls the display range increased by the change means to be recorded in a storage unit in association with the image before the gradation conversion.

8. The image processing device according to claim 1, wherein the image processing device is a computer.

9. further comprising an acquisition unit for acquiring a detection result of ambient light for the display unit; The control means further controls to limit the display range increased by the change means based on the detection result of the ambient light.

9. The image processing device according to claim 1, wherein the image processing device is a computer.

10. The control means limits the maximum display brightness of the display range increased by the change means to a lower display brightness as the ambient light becomes darker.

10. The image processing device according to claim 9,

11. The control means further issues a predetermined notification when the display range increased by the change means includes a display luminance outside the maximum settable display range. Control it so that 11. The image processing device according to claim 1,

12. The predetermined user operation is an operation of a slider displayed on the display unit.

12. The image processing device according to claim 1, wherein the image processing device is a computer.

13. the input means further accepts another user operation, different from the predetermined user operation, for adjusting at least one of exposure, contrast, and color with respect to the image displayed on the display unit; The control means validates the adjustment by the other user operation even if the image does not satisfy a predetermined condition.

13. The image processing device according to claim 1, wherein the image processing device is a computer.

14. an input step of accepting a predetermined user operation on the image displayed on the display unit; a change step of controlling the display unit to increase a display range when the predetermined user operation is accepted in the input step; a processing step of performing gradation conversion on the image based on the display range increased in the changing step to improve the gradation of a high-brightness portion; a control step of validating an increase in the display range due to the specified user operation when the image satisfies a predetermined condition, and not validating an increase in the display range due to the specified user operation when the image does not satisfy the predetermined condition; An image processing method comprising:

15. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 13.

16. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 13.

Images