Image processing apparatus, image processing method, program, and storage medium

Abstract

To provide a technique which allows a user to easily (intuitively) change a display range to a desired luminance range and which can properly display an image in the display range after change without changing the display of the image more than necessary.SOLUTION: An image processing apparatus comprises: input means which receives a prescribed user operation on an image displayed on a display part; change means which performs control so as to increase a display range of the display part when the input means receives the prescribed user operation; processing means which performs gradation change for improving the gradation of a high luminance part for the image on the basis of the display range increased by the change means; and control means which performs control so as to make the increase of the display range by the prescribed user operation effective when the image satisfies a prescribed condition, and so as not to make the increase of the display range by the prescribed user operation effective when the image does not satisfy the prescribed condition.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, a program, and a storage medium, and particularly to a technique for changing the dynamic range (luminance range) of display.

Background Art

[0002] In recent years, as input devices such as smartphones and digital cameras, models capable of recording images having a wide dynamic range (luminance range), such as HDR (High Dynamic Range) images, have been commercialized. HDR images include images having gradation values (signal values) corresponding to absolute luminance. The gradation values corresponding to absolute luminance are, for example, those conforming to the EOTF (Electro-Optical Transfer Function) defined in ITU-R (Radiocommunication Sector of ITU) BT.2100, or those conforming to the EOTF defined in SMPTE (Society of Motion Picture and Television Engineers) ST.2084. These EOTFs are called PQ (Perceptual Quantization) curves and the like. The EOTF such as the PQ curve corresponds to a function for converting gradation values (luminance gradation values; pixel values) into luminance.

[0003] In addition, HDR displays capable of displaying HDR images in a wide dynamic range (display range) are becoming popular, and smartphones equipped with HDR displays have also been commercialized.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, smartphones and the like are often used with the brightness of the display (the brightness of the display on the display) set low in order to reduce power consumption and extend the operating time. When the brightness of the display is set low, the upper limit of the brightness of the display becomes low, and the maximum brightness of the display becomes low.

[0006] The image processing apparatus of the present invention includes input means for receiving a predetermined user operation on an image displayed on a display unit, and change means for controlling to increase the display range of the display unit when the predetermined user operation is received by the input means. Determination means for determining a dynamic range based on a display range increased by the change means, and generation means for synthesizing a plurality of images to generate a synthesized image having the dynamic range determined by the determination means. Based on the display range increased by the change means, the Synthesis processing means for performing tone conversion for improving the tone of the high-brightness part of the image, and when the image satisfies a predetermined condition, the increase in the display range by the predetermined user operation is made effective, and the Displayed control means for controlling so that the increase in the display range by the predetermined user operation is not made effective when the image does not satisfy a predetermined condition. Displayed It is characterized by having.

[0007] Patent Document 1 discloses a technique for converting (compressing) the dynamic range of an HDR image so that the HDR image can be displayed on a display device with a low display brightness. Specifically, in the technique disclosed in Patent Document 1, the dynamic range from the dark part to the bright part is compressed as a whole. With this technique, blooming can be suppressed, but the display of the dark part changes will. Specifically, the tonality (tone resolution; luminance tonality; luminance tone resolution) of the dark part deteriorates, or an overall dark image is displayed.

[0008] Therefore, an object of the present invention is to provide a technique in which a user can easily (intuitively) change the display range to a desired brightness range, and an image can be suitably displayed within the changed display range without changing the display of the image more than necessary.

Means for Solving the Problem

[0009] The image processing apparatus of the present invention includes an input means for receiving a predetermined user operation on an image displayed on a display unit, a change means for controlling to increase the display range of the display unit when the predetermined user operation is received by the input means, a processing means for performing tone conversion for improving the tone of a high-luminance portion of the image based on the display range increased by the change means, and a control means for enabling an increase in the display range by the predetermined user operation when the image satisfies a predetermined condition, and controlling not to enable an increase in the display range by the predetermined user operation when the image does not satisfy the predetermined condition.

Effect of the Invention

[0010] According to the present invention, a user can easily (intuitively) change the display range to a desired luminance range. Further, the image can be suitably displayed in the changed display range without changing the display of the image more than necessary.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Mode for Carrying Out the Invention

[0012] <Example 1> Hereinafter, Example 1 of the present invention will be described. The image processing apparatus according to Example 1 performs tone conversion of an image and displays the tone-converted image on a display unit. When a user operation (range designation operation) for designating a dynamic range (luminance range) is performed, the image processing apparatus displays an item for the user operation together with the tone-converted image. Then, the image processing apparatus changes the setting of the display range and the conversion characteristics of the tone conversion in response to the range designation operation being performed. Specifically, the image processing apparatus changes the setting of the display range, which is the dynamic range of the display on the display unit, based on the designated range, which is the dynamic range designated by the user operation. Then, the image processing apparatus changes the conversion characteristics of the tone conversion according to the changed display range.

[0013] FIG. 1 is a block diagram showing a configuration example of an image processing apparatus 100 according to Example 1. The image processing apparatus 100 includes a control unit 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a UI (User Interface) unit 104, an image processing unit 105, and a display unit 106. These are connected to each other by a bus 107. They are connected to each other by a bus 107.

[0014] The control unit 101 controls the operations of each part of the image processing apparatus 100. For example, the control unit 101 is a CPU (Central Processing Unit), reads a program from the ROM 102, expands it in the RAM 103, and executes it to control the operations of each part of the image processing apparatus 100.

[0015] The ROM 102 is a non-volatile memory that can electrically erase and record various data, and stores programs executed by the control unit 101, parameters necessary for the operations of each part of the image processing apparatus 100, images (image data) to be processed, etc. Note that the image to be processed may be acquired from an external device.

[0016] The RAM 103 is a volatile memory that can electrically erase and record various data. By the control unit 101, the programs stored in the ROM 102 are expanded to the RAM 103, or the parameters, images, etc. stored in the ROM 102 are temporarily recorded in the RAM 103. By the control unit 101, the parameters, images, etc. generated in each part are also temporarily recorded in the RAM 103.

[0017] The UI unit 104 receives user operations on the image processing apparatus 100. For example, the UI unit 104 is a pointing device such as a touch panel or a mouse, or a keyboard. Note that the UI unit 104 (operation device) may be an external device or a part thereof, and the image processing apparatus 100 may have an interface for connecting to the external device so as to be able to acquire information according to user operations.

[0018] The image processing unit 105 performs various image processes such as white balance adjustment, color interpolation, and gamma processing on the images (images to be processed) stored in the RAM 103. The image processing unit 105 has a tone conversion processing unit 200 that performs tone conversion of images.

[0019] The display unit 106 displays the images (images after tone conversion) stored in the RAM 103. Also, the display unit 106 displays items (UI) for user operations, etc. For example, the display unit 106 is a display device such as a liquid crystal panel or an organic EL (Electro Luminescence) panel. Note that the display unit 106 may be an external device or a part thereof such as a smartphone or a television device, and the image processing apparatus 100 may have an interface for connecting to the external device so as to be able to control the display unit 106.

[0020] Figure 2 is a block diagram showing a configuration example of the gradation conversion processing unit 200. The gradation conversion processing unit 200 includes a conversion characteristic determination unit 201 and a gradation conversion unit 202. The conversion characteristic determination unit 201 determines the conversion characteristics of the gradation conversion. The gradation conversion unit 202 performs gradation conversion of an image with the conversion characteristics determined by the conversion characteristic determination unit 201.

[0021] Figure 3 is a flowchart showing an example of display control performed by the image processing apparatus 100. For example, the display control in Figure 3 starts when the control unit 101 reads a program from the ROM 102, expands it in the RAM 103, and executes it in response to the startup of the image processing apparatus 100.

[0022] In S301, the control unit 101 checks the setting of the display range, which is the dynamic range of the display on the display unit 106. In the first embodiment, it is assumed that the minimum luminance of the display range is fixed at 0 nit and only the maximum luminance of the display range can be changed. Therefore, the setting of the display range can also be said to be the setting of the maximum luminance of the display range. For example, when the maximum luminance of the display range is set to 500 nit, a display range of 0 to 500 nit is set, and when the maximum luminance of the display range is set to 800 nit, a display range of 0 to 800 nit is set. Note that the maximum luminance of the display range may be fixed and only the minimum luminance of the display range may be changeable, or both the minimum luminance and the maximum luminance of the display range may be changeable. When the maximum luminance of the display range is set to 800 nit, a display range of 0 to 800 nit is set. Note that the maximum luminance of the display range may be fixed and only the minimum luminance of the display range may be changeable, or both the minimum luminance and the maximum luminance of the display range may be changeable.

[0023] In S302, the control unit 101 controls the display unit 106 to display the image to be processed according to the current display range (the display range confirmed in S301) in response to a user operation using the UI unit 104. The user operation is, for example, a user operation for selecting and displaying the image to be processed. The image to be processed is not particularly limited, but here it is assumed to be a captured HDR image (captured HDR image). An HDR (High Dynamic Range) image is an image with a larger number of bits and a wider dynamic range compared to an SDR (Standard Dynamic Range) image such as an image compliant with ITU-R BT.709.

[0024] A specific example of the process of S302 will be described using the flowchart of FIG. 4. FIG. 4 is a flowchart showing an example of the processes of S302 and S307 (image display process) described later.

[0025] In S401, the conversion characteristic determination unit 201 determines the conversion characteristics in the tone conversion of the captured HDR image according to the display range (the maximum luminance of the display range) confirmed in S301. The captured HDR image is not particularly limited, but here it is assumed to be an image compliant with the PQ (Perceptual Quantization) curve. The PQ curve is, for example, the EOTF (Electro-Optical Transfer Function) defined in ITU-R (Radiocommunication Sector of ITU) BT.2100 or the EOTF defined in SMPTE (Society of Motion Picture and Television Engineers) ST.2084.

[0026] FIG. 5(A) shows an example of a PQ curve (EOTF). The EOTF corresponds to a function that converts a gradation value (luminance gradation value; pixel value) into luminance. Specifically, the EOTF in FIG. 5(A) is represented by Equation 1 below. p_in is the input value of the EOTF and is a value obtained by normalizing a gradation value (R value, G value, B value, etc.) to 0.0 to 1.0. p_in = 1.0 corresponds to the upper limit of the gradation value (the upper limit according to the number of bits), and p_in = 0.0 corresponds to the lower limit of the gradation value. For example, when the number of bits of the gradation value is 10 bits, the upper limit of the gradation value is 1023 and the lower limit of the gradation value is 0. p_out is the output value of the EOTF and is a gradation value obtained by normalizing a gradation value (R value, G value, B value, etc.) proportional to luminance to 0.0 to 1.0. For example, p_out = 0.0 corresponds to 0 nit, and p_out = 1.0 corresponds to 10000 nit. max[x, y] is a function that outputs the larger value of x and y. In Example 1, it is assumed that the display unit 106 displays an image with luminance corresponding to the output value p_out.

Number

[0027] FIG. 5(B) shows an example of an OETF (Opto-Electronic Transfer Function) having characteristics that are exactly the opposite of those of the EOTF in FIG. 5(A). The OETF corresponds to a function that converts luminance into a gradation value. Specifically, the OETF in FIG. 5(B) is represented by Equation 2 below. q_in is the input value of the OETF and is a gradation value obtained by normalizing a gradation value (R value, G value, B value, etc.) proportional to luminance to 0.0 to 1.0. For example, q_in = 0. 0 corresponds to 0 nit, and q_in = 1.0 corresponds to 10000 nit. q_out is the output value of the OETF and is a value obtained by normalizing a gradation value (R value, G value, B value, etc.) to 0.0 to 1.0. q_out = 1.0 corresponds to the upper limit of the gradation value (the upper limit according to the number of bits), and q_out = 0.0 corresponds to the lower limit of the gradation value. For example, when the number of bits of the gradation value is 10 bits, the upper limit of the gradation value is 1023 and the lower limit of the gradation value is 0.

Number

[0028] As described above, in the first embodiment, it is assumed that the captured HDR image is an image compliant with the EOTF (PQ curve) in FIG. 5(A). In other words, the captured HDR image is an image captured with the OETF in FIG. 5(B). The gradation value of the captured HDR image corresponds to the input value p_in of the EOTF and also corresponds to the output value q_out of the OEFT.

[0029] Characteristic 600 (luminance conversion characteristic) in FIG. 6(A) corresponds to the characteristic obtained by multiplying the EOTF in FIG. 5(A) and the OETF in FIG. 5(B). The horizontal axis in FIG. 6(A) indicates the luminance (input value q_in of the OETF) of the image before gradation conversion, and the vertical axis in FIG. 6(A) indicates the luminance (output value p_out of the EOTF) of the image after gradation conversion. Here, it is assumed that the maximum luminance of the display range is luminance Y1. In this case, the conversion characteristic determination unit 201 determines characteristic 601 according to luminance Y1, and corrects the EOTF in FIG. 5(A) so that characteristic 601 is realized. Characteristic 601 or the corrected EOTF is the conversion characteristic determined in S401. As shown in FIG. 6(A), in the luminance range (luminance range below luminance X1) corresponding to the display range (luminance range below luminance Y1) in characteristic 600, characteristic 601 substantially coincides (coincides) with characteristic 600. Therefore, by gradation conversion, the luminance below luminance X1 is converted to substantially the same (identical) luminance by both characteristic 600 and characteristic 601 and is displayed according to the PQ curve. This enables display that matches the visual characteristics of humans. However, in characteristic 601, the luminance higher than luminance X1 is converted (clipped) to luminance Y1 by gradation conversion and is displayed with blooming.

[0030] In S402, the gradation conversion unit 202 performs gradation conversion of the captured HDR image according to the conversion characteristic determined in S401. Specifically, the gradation conversion unit 202 performs gradation conversion of the captured HDR image according to the corrected EOTF. Thereby, a display HDR image is generated as the image after gradation conversion.

[0031] In S403, the control unit 101 outputs the display HDR image generated in S402 to the display unit 106. FIG. 7(A) shows an example of displaying the display HDR image generated in S402. As described above, in the characteristic 601 of FIG. 6(A), luminances higher than the luminance X1 are converted to the luminance Y1 by tone conversion and are displayed with blooming. Therefore, although the sun and clouds exist in the captured HDR image (the image before tone conversion), in the display example of FIG. 7(A), the sun and clouds are blooming.

[0032] Return to the description of FIG. 3. In S303, the control unit 101 determines whether a blooming suppression start operation for starting the suppression of blooming in the display HDR image displayed in S302 has been performed as a user operation using the UI unit 104. For example, when the user wants to suppress blooming and recognize the sun and clouds as shown in the display example of FIG. 7(B) although the sun and clouds are blooming as shown in the display example of FIG. 7(A), the user performs a blooming suppression start operation. The blooming suppression start operation is, for example, touching a button displayed on the display unit 106 or pressing a button (physical button) provided in the image processing apparatus 100. As a more specific example, for example items such as "blooming suppression" are displayed in the menu of adjustment items such as "exposure", "contrast", "color", "white balance (color temperature)", and "sharpness". When this "blooming suppression" item is selected, the control unit 101 determines that the blooming suppression start operation has been performed and proceeds to S304, and if not (if the blooming suppression start operation has not been performed), the display control in FIG. 3 is terminated.

[0033] In S304, the control unit 101 controls the display unit 106 to display an item (UI) for a user operation (range specification operation) that specifies the dynamic range, together with the display HDR image. Specifically, as shown in FIG. 7(C), the control unit 101 controls to display the slider 701. The user can adjust the dynamic range (specified range) specified by the user by moving the slider 701. Specifically, the user can adjust the maximum luminance of the specified range by moving the slider 701. Therefore, the range specification operation can also be said to be a user operation that specifies the maximum luminance of the specified range. Although details will be described later, when the specified range is adjusted, the degree of blooming in the display HDR image is adjusted. Therefore, the slider 701 can also be said to be an item for adjusting the degree of blooming. The operation of moving such a slider 701 is similarly used for adjusting items such as the aforementioned "exposure", "contrast", "color", "white balance (color temperature)", and "sharpness". That is, the user can perform a blooming suppression operation with the same operation feeling as other adjustment items.

[0034] In addition, this blooming suppression is particularly effective for images in which the gradation of the high-luminance portion remains relatively large as image information, such as HDR images. Therefore, it is possible to determine whether the image to be displayed is an HDR image at least before S303, enable the blooming suppression process of this embodiment when it is an HDR image, and disable the blooming suppression process when it is an SDR image. The process of disabling the blooming suppression process may be, for example, to make "blooming suppression" non-displayed or grayed out in the menu of the aforementioned adjustment items. Or the slider 701 may be made immovable. Whether it is an HDR image may be determined, for example, by referring to the image extension or a predetermined item in the metadata. Also, it may be determined by whether the number of bits of the image is a predetermined number of bits. Also, it may be determined whether it conforms to the PQ curve from the metadata of the image or the like.

[0035] Note that items such as the slider 701 may be generated by the control unit 101 or may be stored in advance in the ROM 102. The initial position of the slider 701 may or may not be a position based on the maximum luminance of the current display range (the maximum luminance confirmed in S301) on the bar indicating the movable range of the slider 701. For example, the initial position of the slider 701 may be a position corresponding to a luminance substantially the same as (identical to) the maximum luminance of the current display range on the bar. When the upper limit of the maximum luminance of the specified range or the display range is 1000 nit and the maximum luminance of the current display range is 500 nit, the initial position of the slider 701 may be the center of the bar. When the minimum luminance of the display range is changeable, the item for the range specifying operation may include an item (such as a slider) for specifying the minimum luminance of the specified range. The item for the range specifying operation is not limited to a slider and may have the same form as the item for specifying the exposure at the time of shooting.

[0036] In S305, the control unit 101 acquires information on the current specified range (the dynamic range specified by the user in S304). Specifically, the control unit 101 acquires information on the luminance corresponding to the current position of the slider 701 as information on the maximum luminance of the current specified range. For example, when the upper limit of the maximum luminance of the specified range or the display range is 1000 nit and the slider 701 is located at the center of the bar, the maximum luminance of the current specified range (the luminance corresponding to the current position of the slider 701) is 500 nit.

[0037] In S306, the control unit 101 changes the setting of the display range from the setting confirmed in S301 based on the current specified range. Specifically, the control unit 101 changes the setting of the maximum luminance of the display range from the setting confirmed in S301 based on the maximum luminance of the current specified range. The display range after the change does not have to be substantially the same as the specified range, but in the first embodiment, the control unit 101 changes the setting of the display range so that the display range is substantially the same (identical) as the specified range. Specifically, the control unit 101 changes the setting of the maximum luminance of the display range so that the maximum luminance of the display range is substantially the same (identical) as the maximum luminance of the specified range. For example, when the maximum luminance of the specified range is 500 nit, the maximum luminance of the display range is changed to 500 nit.

[0038] In S307, the control unit 101 controls to update the display of the display HDR image according to the display range after the change in S306. In the first embodiment, by updating the display HDR image, the display of the display HDR image is updated. Then, the control unit 101 ends the display control in FIG. 3. When the white-out suppression start operation is performed again, the processes of S304 to S307 are performed again.

[0039] A specific example of the process of S307 will be described with reference to the flowchart of FIG. 4. However, the description of the process similar to the process of S302 will be omitted as appropriate.

[0040] In S401, the conversion characteristic determination unit 201 changes the conversion characteristics in the tone conversion of the captured HDR image according to the changed display range (the maximum luminance of the display range) in S306. Here, assume that the maximum luminance of the display luminance is changed from luminance Y1 to luminance Y2 (>Y1). In this case, the conversion characteristic determination unit 201 determines the characteristic 602 in FIG. 6(B) according to luminance Y2, and corrects the EOTF in FIG. 5(A) so that the characteristic 602 is realized. In the characteristic 602, as the maximum luminance of the display range is increased from luminance Y1 to luminance Y2, the luminance range capable of expressing tones is expanded from the luminance range of luminance X1 or less to the luminance range of luminance X2 (>X1) or less. Further, in the portion of the changed display range that overlaps with the previous display range (the luminance range of luminance X1 or less), the characteristic 602 substantially coincides (coincides) with the characteristic 601. Therefore, even if the change from the characteristic 601 to the characteristic 602 is made, the display of the display HDR image can be maintained for the luminance range of luminance X1 or less.

[0041] In the characteristics 601 and 602, the luminance of the display HDR image (the luminance of the image after tone conversion; the output value p_out of the EOTF) is substantially proportional (proportional) to the luminance of the captured HDR image in the display range (the luminance of the image before tone conversion; the input value q_in of the OETF), but it is not limited to this. In the display range, the luminance of the display HDR image may change non-linearly with respect to the change in the luminance of the captured HDR image.

[0042] In S402, the tone conversion unit 202 generates (updates) the display HDR image by performing tone conversion of the captured HDR image according to the conversion characteristics (changed conversion characteristics) determined in S401.

[0043] In S403, the control unit 101 outputs the display HDR image generated in S402 to the display unit 106. As a result, the display of the display HDR image is updated. FIG. 7(B) shows a display example of the display HDR image (the updated display HDR image) generated in S402. As described above, in characteristic 602 of FIG. 6(B), the luminance range capable of expressing gradation is expanded from the luminance range of luminance X1 or less to the luminance range of luminance X2 or less. For this reason, in the display example of FIG. 7(B), white bleeding is suppressed, and the sun and clouds that are white bleeding in the display example of FIG. 7(A) can be recognized.

[0044] As described above, according to the first embodiment, when the range designation operation is performed, an item for the range designation operation is displayed together with the image after gradation conversion. Then, in response to the range designation operation being performed, the setting of the display range is changed based on the designated range, and the display of the image after gradation conversion is updated according to the changed display range. Thereby, while performing the range designation operation, the user can check the image after gradation conversion and easily (intuitively) change the display range to a desired luminance range. Further, according to the first embodiment, the conversion characteristics of gradation conversion are changed so as to maintain the display of the image after gradation conversion for the portion of the changed display range that overlaps with the display range before the change. Thereby, the image can be suitably displayed in the changed display range without changing the display of the image more than necessary.

[0045] Note that when the designated range includes a luminance outside the maximum display range in which the designated range can be set, for example, when the maximum luminance of the designated range is higher than the upper limit of the maximum luminance of the display range, the control unit 101 may control to perform a predetermined notification (warning). The notification is performed, for example, by display of an item, output of sound, emission of a lamp (such as a light emitting diode). The display unit, speaker, lamp, etc. for notification may be a part of the image processing apparatus 100, or may not be.

[0046] The image processing apparatus 100 may have a detection unit (optical sensor) that detects ambient light with respect to the display unit 106. Then, the control unit 101 may control to limit the luminance range that can be specified as the specified range based on the detection result of the ambient light by the detection unit. For example, the control unit 101 may limit the maximum luminance of the luminance range that can be specified as the specified range to a lower luminance as the ambient light is darker. Thereby, it is possible to suppress the display HDR image from looking extremely dazzling due to the influence of the ambient light. The control unit 101 may change the setting of the display range based on the specified range and the detection result of the ambient light without limiting the luminance range that can be specified as the specified range. The detection unit may be an external device or a part thereof, and the image processing apparatus 100 may have an interface for connecting to the external device so as to be able to acquire the detection result of the ambient light.

[0047] When the image to be processed (image before tone conversion) is switched to another image, the control unit 101 may return the setting of the display range to the setting before the change (the setting confirmed in S301). If the maximum luminance of the display range is always increased based on the range specification operation, the power consumption can be reduced by returning the setting of the display range to the setting before the change.

[0048] The control unit 101 may control to record the changed display range in the storage unit in association with the image before tone conversion. The storage unit may be the ROM 102 or may not be. By doing so, when an image whose display range has been changed in the past is displayed again on the image processing apparatus 100 or another image processing apparatus, the past display can be reproduced based on the changed display range. The storage unit may be built in the image processing apparatus 100 or may be a storage device detachable from the image processing apparatus 100.

[0049] <Example 2> Hereinafter, Example 2 of the present invention will be described. The image processing apparatus according to Example 2 performs the same processing as that described in Example 1. Further, the image processing apparatus according to Example 2 further performs a process of determining an image range, which is the dynamic range of the image itself, based on a specified range, and a process of generating an image having the determined image range. In the following, differences (configurations, processes, etc.) from Example 1 will be described in detail, and descriptions of the same points as in Example 1 will be omitted as appropriate.

[0050] The image processing apparatus according to Example 2 has the same configuration as the image processing apparatus 100 (FIG. 1) according to Example 1. However, the image processing apparatus according to Example 2 has a tone conversion processing unit 800 shown in FIG. 8 instead of the tone conversion processing unit 2 00. FIG. 8 is a block diagram showing a configuration example of the tone conversion processing unit 800. Similar to the tone conversion processing unit 200, the tone conversion processing unit 800 has a conversion characteristic determination unit 201 and a tone conversion unit 202. Further, the tone conversion processing unit 800 has an HDR synthesis unit 801. The HDR synthesis unit 801 synthesizes a plurality of images with different exposures to generate a single synthesized image. The dynamic range of the synthesized image can be freely changed by changing the plurality of images to be synthesized.

[0051] FIG. 9 is a flowchart showing an example of display control performed by the image processing apparatus according to Example 2. For example, the display control in FIG. 9 is started when the control unit 101 reads a program from the ROM 102, expands it in the RAM 103, and executes it in response to the activation of the image processing apparatus according to Example 2. In the display control in FIG. 9, the processes of S301 to S307 are the same as those in Example 1 (FIG. 3).

[0052] In S901, the control unit 101 determines an image range, which is the dynamic range of the image itself, based on the current specified range. Specifically, the control unit 101 determines the maximum luminance of the image range based on the maximum luminance of the current specified range. The image range may not be substantially the same as the specified range, but in the second embodiment, the control unit 101 determines an image range that is substantially the same (identical) as the specified range. Specifically, the control unit 101 determines a luminance that is substantially the same (identical) as the maximum luminance of the specified range as the maximum luminance of the image range. For example, when the maximum luminance of the specified range is 4000 nit, 4000 nit is determined as the maximum luminance of the image range.

[0053] In S902, the HDR composition unit 801 generates a composite image having the image range determined in S901 as an image before tone conversion by composing a plurality of images with different exposures. At this time, the HDR composition unit 801 determines a plurality of images (such as exposure and number of images) used for composition based on the image range. In the second embodiment, in S307 (S402), the tone conversion unit 202 generates a display HDR image by performing tone conversion of the composite image generated in S902.

[0054] A specific example of the process of S902 will be described when the maximum luminance of the image range is 4000 nit, that is, when the image range is a luminance range of 0 to 4000 nit. Here, it is assumed that the dynamic range of the properly exposed image is a luminance range of 0 to 1000 nit, and a composite image having a dynamic range of 0 to 2000 nit can be generated by compositing one stage of underexposed image with the properly exposed image. And it is assumed that a composite image having a dynamic range of 0 to 4000 nit can be generated by compositing two stages of underexposed images with the properly exposed image. The properly exposed image is an image taken at proper exposure, the one-stage underexposed image is an image taken at an exposure one stage lower than the proper exposure, and the two-stage underexposed image is an image taken at an exposure two stages lower than the proper exposure.

[0055] As described above, by synthesizing the two-stage low-exposure image with the proper exposure image, a composite image having a dynamic range of 0 to 4000 nits can be generated. However, a composite image in which gradations are expressed over the entire dynamic range of 0 to 4000 nits is not always generated. For this reason, the HDR synthesis unit 801 synthesizes the proper exposure image, the one-stage low-exposure image, and the two-stage low-exposure image so that a composite image in which gradations are expressed over the entire dynamic range of 0 to 4000 nits (no gradation jump occurs) is generated. Note that if a composite image in which gradations are expressed over the entire dynamic range of 0 to 4000 nits is generated, the HDR synthesis unit 801 may synthesize the proper exposure image and the two-stage low-exposure image without using the one-stage low-exposure image. The number of images used for synthesis is not particularly limited, and may be more than three.

[0056] A method example of synthesis (HDR synthesis; weighted synthesis) by the HDR synthesis unit 801 will be described with reference to FIG. 1 This will be described using 0. FIG. 10 shows an example of the synthesis ratio (weight) used when synthesizing the proper exposure image, the one-stage low-exposure image, and the two-stage low-exposure image. The synthesis ratio 1001 is the synthesis ratio of the proper exposure image, the synthesis ratio 1002 is the synthesis ratio of the one-stage low-exposure image, and the synthesis ratio 1003 is the synthesis ratio of the two-stage low-exposure image.

[0057] As shown in FIG. 10, the HDR synthesis unit 801 increases the brightness of the one-stage low-exposure image and the two-stage low-exposure image so that the brightness of the one-stage low-exposure image and the two-stage low-exposure image matches the brightness of the proper exposure image. Specifically, the HDR synthesis unit 801 increases the brightness of the one-stage low-exposure image so that the brightness of the one-stage low-exposure image substantially matches (matches) the brightness of the proper exposure image for portions where gradations can be expressed in both the one-stage low-exposure image and the proper exposure image. Similarly, the HDR synthesis unit 801 increases the brightness of the two-stage low-exposure image so that the brightness of the two-stage low-exposure image substantially matches (matches) the brightness of the proper exposure image for portions where gradations can be expressed in both the two-stage low-exposure image and the proper exposure image. In FIG. 10, the brightness of the one-stage low-exposure image is increased by a factor of 2, and the brightness of the two-stage low-exposure image is increased by a factor of 4.

[0058] Then, the HDR synthesis unit 801 synthesizes the proper exposure image, the one-step underexposed image, and the two-step underexposed image according to the synthesis ratios 1001 to 1003. The synthesis ratio 1001 of the proper exposure image is 100% up to the luminance X1, and linearly decreases from 100% to 0% with respect to the luminance increase from the luminance X1 to the luminance X2. The synthesis ratio 1002 of the one-step underexposed image increases from 0% to 100% in proportion to the luminance increase from the luminance X1 to the luminance X2, is 100% from the luminance X2 to the luminance X3, and linearly decreases from 100% to 0% with respect to the luminance increase from the luminance X3 to the luminance X4. The synthesis ratio 1003 of the two-step underexposed image increases from 0% to 100% in proportion to the luminance increase from the luminance X3 to the luminance X4, and is 100% from the luminance X4. Note that the synthesis ratios 1001 to 1003 are not limited to those shown in FIG. 10. The synthesis ratios 1001 to 1003 may be determined in any manner as long as the sum of the synthesis ratios 1001 to 1003 is always 100%.

[0059] As described above, according to the second embodiment, the image range is determined based on the specified range, and by synthesizing a plurality of images with different exposures, an image having the image range is generated as the image before tone conversion. Thereby, it is possible to display an image while further suppressing image quality degradation such as whiteout.

[0060] As described above, the present invention has been described in detail based on its preferred embodiments. However, the present invention is not limited to these specific embodiments, and various forms within the scope not departing from the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely shows one embodiment of the present invention, and it is also possible to appropriately combine the embodiments.

[0061] Note that each of the above-described various controls described as being performed by the control unit 101 may be performed by one piece of hardware, or the entire device may be controlled by a plurality of pieces of hardware (for example, a plurality of processors or circuits) sharing the processing.

[0062] (Other Embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. It can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

Explanation of Signs

[0063] 100: Image processing apparatus 101: Control unit 104: UI unit 105: Image processing unit 200: Tone conversion processing unit 201: Conversion characteristic determination unit 202: Tone conversion unit 800: Tone conversion processing unit

Claims

1. Input means for receiving a predetermined user operation on an image displayed on a display unit; When the predetermined user operation is received by the input means, change means for controlling to increase a display range of the display unit; Determination means for determining a dynamic range based on the display range increased by the change means; Generation means for synthesizing a plurality of images to generate a composite image having the dynamic range determined by the determination means; Processing means for performing gradation conversion for improving gradation of a high-luminance portion on the composite image based on the display range increased by the change means; Control means for enabling an increase in the display range by the predetermined user operation when the displayed image satisfies a predetermined condition, and disabling an increase in the display range by the predetermined user operation when the displayed image does not satisfy the predetermined condition An image processing apparatus, characterized by comprising:

2. The increase in the display range includes an increase in a maximum display luminance of the display unit The image processing apparatus according to claim 1, characterized in that:

3. The predetermined condition is a condition related to a dynamic range of the displayed image The image processing apparatus according to claim 1 or 2, characterized in that:

4. The predetermined condition is a condition related to a bit number of the displayed image The image processing apparatus according to claim 1 or 2, characterized in that:

5. An image satisfying the predetermined condition is an HDR (High Dynamic Range) image recorded in a predetermined manner The image processing apparatus according to any one of claims 1 to 4, characterized in that:

6. Images that do not meet the predetermined conditions are SDR (Standard Dynamic Range) images The image processing apparatus according to any one of claims 1 to 5, characterized in that

7. The processing means changes the conversion characteristics of the tone conversion so that the display luminance of the composite image after the tone conversion is substantially proportional to the display luminance of the composite image before the tone conversion in the display range increased by the changing means The image processing apparatus according to any one of claims 1 to 6, characterized in that

8. The control means further controls to record in the storage unit the display range increased by the changing means in association with the composite image before the tone conversion The image processing apparatus according to any one of claims 1 to 7, characterized in that

9. The apparatus further comprises acquisition means for acquiring a detection result of ambient light with respect to the display unit, The control means further controls to limit the display range increased by the changing means based on the detection result of the ambient light The image processing apparatus according to any one of claims 1 to 8, characterized in that

10. The control means limits the maximum display luminance of the display range increased by the changing means to a lower display luminance as the ambient light is darker The image processing apparatus according to claim 9, characterized in that

11. The control means further controls to perform a predetermined notification when the display luminance included in the display range increased by the changing means is outside the maximum display range that can be set The image processing apparatus according to any one of claims 1 to 10, characterized in that

12. The predetermined user operation is an operation of a slider displayed on the display unit The image processing apparatus according to any one of claims 1 to 11, characterized in that

13. The input means further accepts, with respect to the image displayed on the display unit, another user operation for adjusting at least one of exposure, contrast, and color, which is different from the predetermined user operation, The control means enables the adjustment by the other user operation even when the displayed image does not satisfy a predetermined condition. The image processing apparatus according to any one of claims 1 to 12, characterized in that.

14. An input step of accepting a predetermined user operation with respect to an image displayed on a display unit; A change step of controlling to increase the display range of the display unit when the predetermined user operation is accepted in the input step; A determination step of determining a dynamic range based on the increased display range in the change step; A generation step of generating a composite image having the dynamic range determined in the determination step by synthesizing a plurality of images; A processing step of performing tone conversion for improving the tone of the high-luminance portion with respect to the composite image based on the increased display range in the change step; A control step of enabling the increase in the display range by the predetermined user operation when the displayed image satisfies a predetermined condition, and controlling not to enable the increase in the display range by the predetermined user operation when the displayed image does not satisfy a predetermined condition. An image processing method, characterized by comprising:

15. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 13.

16. A computer-readable storage medium storing a program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 13.

Images