IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND IMAGING APPARATUS

Abstract

To provide an image processing apparatus that can appropriately deal with brightness data according to an encoding method applied to image data.SOLUTION: An image processing apparatus applies, to image data, processing using a brightness value of the image data and a value based on information on the maximum brightness value of an output dynamic range of the image data. When the image data is encoded in a first numerical range and first information is encoded in a second numerical range different from the first numerical range, the image processing apparatus converts the image data, the first information, or a first value into a value when the image data and the first value are encoded in the same numerical range, before applying the processing to the image data.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and an imaging device. [Background technology]

[0002] The luminance corresponding to the image data value may differ depending on the gradation characteristics and encoding method applied to the image data. The gradation characteristics are the correspondence between the input luminance range and the output luminance range, such as the so-called gamma curve or electro-optical transfer function (EOTF). The encoding method is a method of allocating the output luminance range to a specific numerical range. For example, when expressing the output luminance range as a 10-bit numerical value, a method of expressing it in the range of 0 to 1023 (full range) and a method of expressing it in the range of 64 to 940 (video range or limited range) are known. A range conversion technique for converting between each range is also known. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-182179 A Summary of the Invention [Problem to be solved by the invention]

[0004] Therefore, for example, when applying processing according to luminance to input image data, appropriate processing results may not be obtained unless the gradation characteristics and encoding method applied to the image data, especially the encoding method, are taken into consideration. For example, in the highlight warning display processing disclosed in Patent Document 1, a warning may be displayed in an incorrect range.

[0005] In one aspect, the present invention provides an image processing device and an image processing method capable of appropriately handling luminance data in accordance with an encoding method applied to image data. [Means for solving the problem]

[0006] The above-mentioned object can be achieved by an image processing device comprising: an acquisition means for acquiring image data and first information relating to the maximum luminance value of the output dynamic range of the image data; and a processing means for applying to the image data a process using a first value based on the luminance value of the image data and the first information, wherein when the image data is encoded in a first numerical range and the first information is encoded in a second numerical range different from the first numerical range, the processing means converts the image data, the first information, or the first value so that the image data and the first value become values ​​that would be obtained when encoded in the same numerical range, and then applies the process to the image data. Effect of the Invention

[0007] According to the present invention, it is possible to provide an image processing device and an image processing method capable of appropriately handling luminance data in accordance with the encoding method applied to image data. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of a basic functional configuration of an image capture apparatus as an image processing apparatus according to an embodiment of [Diagram 2] FIG. 13 is a diagram showing an example of a gradation characteristic applied to image data. [Diagram 3] 1 is a flowchart illustrating an example of a threshold calculation process according to a first embodiment. [Figure 4] 11 is a flowchart of another example of the threshold calculation process according to the first embodiment. [Diagram 5] 11 is a flowchart illustrating yet another example of the threshold calculation process according to the first embodiment. [Figure 6] 11 is a flowchart illustrating yet another example of the threshold calculation process according to the first embodiment. [Figure 7] FIG. 11 is a diagram for explaining a highlight warning display in the first embodiment; [Figure 8]13 is a flowchart illustrating an example of a histogram display process according to the second embodiment. [Figure 9] 13 is a flowchart of another example of the histogram display process in the second embodiment. [Figure 10] 13 is a flowchart of yet another example of the histogram display process in the second embodiment. [Figure 11] FIG. 13 is a diagram for explaining a histogram display in the second embodiment; [Figure 12] FIG. 13 is a diagram showing an example in which the second embodiment is applied to a waveform monitor display. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0010] In the following embodiment, the present invention will be described with respect to a case where the present invention is implemented by a digital camera (DSLR). However, the imaging function is not essential to the present invention, and the present invention can be implemented by any electronic device capable of handling image data. Such electronic devices include, but are not limited to, computer devices (personal computers, tablet computers, media players, PDAs, etc.), smartphones, game consoles, robots, drones, and drive recorders.

[0011] ●(First embodiment) FIG. 1 is a block diagram showing an example of a basic functional configuration of a digital camera 100 (hereinafter, referred to as camera 100) as an example of an image processing device according to an embodiment of the present invention. The optical system 101 is an imaging optical system that includes a lens group, a shutter, an aperture, and the like, and forms an optical image of a subject on an imaging surface of an image sensor 102. The lens group includes a fixed lens and a movable lens, and the movable lens includes a lens for image stabilization, a focus lens, a variable magnification lens, and the like. The aperture may also have a function of a mechanical shutter. Operations of the movable lens, the aperture, and the shutter are controlled by a CPU 103, which is a main control unit of the camera 100. The optical system 101 may or may not be replaceable.

[0012] The imaging element 102 is, for example, a CMOS image sensor, and has a plurality of pixels, each having a photoelectric conversion region, arranged two-dimensionally. The imaging element 102 also has a color filter having a specific color pattern, and each pixel is provided with a filter of one color according to the color pattern. Although the present invention does not depend on the color pattern of the color filter, it is assumed here that a primary color Bayer array color filter is provided. Therefore, each pixel is provided with a color filter of any one of R (red), G (green), and B (blue). The imaging element 102 photoelectrically converts the optical image at each pixel, and converts it into an analog image signal indicating luminance information for each pixel.

[0013] An analog image signal generated by the imaging element 102 is converted into a digital image signal by an A / D converter (not shown). The A / D converter may be included in the imaging element 102, or the CPU 103 may perform the A / D conversion. The pixel signals constituting the digital image signal output by the A / D converter are RAW data having only the luminance components of the color of the color filter provided in the pixel that generated the signal. The CPU 103 stores the RAW data in the primary storage device 104. The shooting sensitivity (hereinafter referred to as ISO sensitivity) of the imaging element 102 is set by the CPU 103.

[0014] CPU 103 transfers a program stored in secondary storage device 107 to primary storage device 104 and executes it to control each unit of camera 100 and realize various functions of camera 100. Note that in the following description, at least a part of the functions realized by CPU 103 executing a program may be realized by dedicated hardware such as an ASIC.

[0015] The primary storage device 104 is a volatile storage device such as a RAM, etc. The primary storage device 104 is used by the CPU 103 to execute programs, and is also used as a buffer memory for image data, a work area for image processing, a video memory for display, etc.

[0016] The secondary storage device 107 is a rewritable non-volatile storage device such as an EEPROM, etc. The secondary storage device 107 stores programs (instructions) executable by the CPU 103, settings of the camera 100, GUI data, and the like.

[0017] The recording medium 106 is a rewritable non-volatile storage device such as a semiconductor memory card. The recording medium 106 may or may not be detachable from the camera 100. Data (still image data, video data, audio data, etc.) generated by the camera 100 can be recorded on the recording medium 106. In other words, the camera 100 has a read / write function for the recording medium 106 and an attachment / detachment mechanism (when the recording medium 106 is detachable). Note that the recording destination of the data generated by the camera 100 is not limited to the recording medium 106. The data may be transmitted to an external device through a communication interface possessed by the camera 100 and recorded on a recording device accessible by the external device.

[0018] The display unit 108 is, for example, a liquid crystal display. The CPU 103 functions as a display control device for the display unit 108. In a shooting standby state or while a moving image is being recorded, the captured moving image is displayed in real time on the display unit 108, and the display unit 108 functions as an electronic viewfinder. The display unit 108 also displays image data recorded on the recording medium 106 and GUI images such as a menu screen.

[0019] The operation unit 109 is a general term for a group of input devices that accept user operations. For example, buttons, levers, and touch panels are used for the operation unit 109. The operation unit 109 may have input devices that do not require physical operations, such as voice and line of sight. The input devices of the operation unit 109 are given names according to the assigned functions. Representative examples include a shutter button, a menu button, a directional key, a decision (set) button, and a mode switching dial. Different functions may be selectively assigned to one input device.

[0020] The image processing unit 105 applies predetermined image processing to image data (which may be RAW data or image data after development processing), generates image data in different formats, and acquires and / or generates various information. The image processing circuit 105 may be a dedicated hardware circuit such as an ASIC designed to realize a specific function, or may be configured such that a programmable processor such as a DSP executes software to realize a specific function.

[0021] The image processing applied by the image processing unit 105 can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation processing is performed when an image sensor is provided with a color filter, and is a process for interpolating values ​​of color components that are not included in the individual pixel data that constitutes the image data. Color interpolation processing is also called demosaic processing. The correction processing can include white balance adjustment, gradation correction, correction of image degradation caused by optical aberration of the imaging optical system 101 (image restoration), correction of the effects of peripheral light falloff of the imaging optical system 101, color correction, and the like. The detection process may include detection of feature regions (for example, face regions or human body regions) and their movements, person recognition processing, and the like. Data processing may include processes such as area extraction (trimming), synthesis, scaling, encoding and decoding, header information generation (data file generation), etc. Data processing also includes the generation of image data for display or image data for recording. The evaluation value calculation process can include processes such as generation of a signal and evaluation value used in automatic focus detection (AF) and generation of an evaluation value used in automatic exposure control (AE). The special effects processing may include adding a blur effect, changing color tones, relighting, and the like. It should be noted that these are merely examples of processes that the image processing unit 105 can apply, and do not limit the processes that the image processing unit 105 can apply.

[0022] In this embodiment, a highlight warning process will be described as an example of a process that is applied to image data (moving image data) according to brightness, but other processes can also be applied in a similar manner.

[0023] In this embodiment, the highlight warning process is applied using a parameter called maxDRL (maximum dynamic range level). Although details will be described later, maxDRL is the maximum luminance value [nits] of the output dynamic range or the gradation value (signal level) corresponding to the maximum luminance value. Note that 1 nit is 1 cd / m 2 is equal to.

[0024] Camera 100 is provided with a plurality of gradation characteristics 201, 202 with different output dynamic ranges as shown in Fig. 2. The gradation characteristics are also called gamma curves, electro-optical transfer functions (EOTFs), etc., and represent the relationship between input gradation values ​​(signal levels) and output luminance. Camera 100 stores a plurality of gradation characteristics associated with, for example, shooting modes in secondary storage device 107. For example, gradation characteristic 201 having a maximum luminance value 203 is associated with a first shooting mode, and gradation characteristic 202 having a maximum luminance value 204 is associated with a second shooting mode.

[0025] It should be noted that there are gradation characteristics in which the output luminance is treated as a relative value and in which it is treated as an absolute value. In gradation characteristics in which the output luminance is treated as a relative value, the maximum luminance value is the maximum luminance value of the display device. Therefore, the brightness of the image may differ depending on the display device. On the other hand, in gradation characteristics in which the output luminance is treated as an absolute value, the maximum luminance value does not depend on the maximum luminance value of the display device. Therefore, within the range of displayable luminance, the brightness of the image will be the same regardless of the display device.

[0026] For example, in the international standard (ITU-R BT.2100) for HDR video, HLG is defined as a gradation characteristic that treats output luminance as a relative value, and PQ is defined as a gradation characteristic that treats output luminance as an absolute value. In PQ, the maximum luminance is defined as 10000 [nits]. For example, if the gradation characteristics 201 and 202 shown in FIG. 2 are compliant with PQ, the luminance 205 corresponds to 10000 [nits]. On the other hand, the gradation characteristics 201 and 202 each show a different maximum luminance value (peak luminance). As described above, since the gradation characteristics 201 and 202 are associated with the shooting mode, each captured image has a different output dynamic range depending on the shooting mode. Therefore, as information indicating what output dynamic range the captured image was captured with, the maximum luminance value [nits] of the output dynamic range or the gradation value (signal level) corresponding to the maximum luminance value is recorded as metadata. In this embodiment, this is called maxDRL. In this embodiment, image data (moving image data) using gradation characteristics that treat output luminance as an absolute value is handled. However, similar processing can be applied to image data (moving image data) using gradation characteristics that treat output luminance as a relative value. In addition, it may be HDR video or SDR video.

[0027] If maxDRL is the gradation value corresponding to the maximum luminance value, the numerical range of the gradation value differs depending on the encoding method, so either maxDRL is prepared for each encoding method, or the identification information of the encoding method is associated with maxDRL. Here, the encoding method is assumed to be either full range or video range (limited range). The numerical range of image data also differs depending on whether it is encoded in the full range or in the video range.

[0028] In this embodiment, it is assumed that the camera 100 can acquire the maxDRL of the image data (moving image data) to be handled. For example, when handling recorded moving image data, the camera 100 can acquire the maxDRL recorded in association with the moving image data. For moving image data captured by the camera 100, the maxDRL stored in the secondary storage device 107 in advance according to the shooting mode can be used.

[0029] The highlight warning process in the embodiment will be described below. The highlight warning process is a process for presenting a whiteout region of an image. For example, the process may be a process for superimposing a zebra pattern or an image of a specific color on the whiteout region, but the whiteout region may be presented by other methods. The highlight warning process is executed, for example, when the user sets the highlight warning to be valid. The image to which the highlight warning process is applied may be a still image or a video, but it is generally applied to a video, particularly a video used for live view display.

[0030] A whiteout area in an image can be detected as an area where the brightness is equal to or greater than a threshold value. In this embodiment, for an image to which a gradation characteristic that treats the output brightness as an absolute value is applied, a threshold value according to the encoding method is used, thereby enabling appropriate detection of whiteout areas regardless of the encoding method.

[0031] (Example 1) 3 is a flowchart related to the calculation process of the highlight warning threshold when maxDRL is encoded in the full range (encoded in the second numerical range). Here, it is assumed that the highlight warning process is executed when the video data recorded on the recording medium 106 is played back.

[0032] In S301, the CPU 103 acquires the maxDRL for the video data to be played back. Here, it is assumed that the maxDRL is acquired from the metadata of the video data file recorded on the recording medium 106.

[0033] In S302, the CPU 103 calculates a highlight warning threshold value using maxDRL. For example, when performing a highlight warning for an area having a brightness of 90% or more of maxDRL, the CPU 103 can calculate the threshold value by multiplying maxDRL by a coefficient of 0.9. Note that the coefficient may be changed according to user settings. The threshold value may also be calculated by other methods, such as finding the difference between maxDRL and an arbitrary constant.

[0034] In S303, the CPU 103 determines whether the video data to be played back is encoded in a video range (encoded in a first numerical range). The CPU 103 can obtain the encoding method of the video data from, for example, metadata of the video data file.

[0035] If it is determined that the video data is not encoded in the video range (encoded in the full range), the CPU 103 does not change the threshold calculated in S302 and ends the threshold calculation process. This is because the encoding method of the video data and the encoding method of the maxDRL are the same.

[0036] On the other hand, if it is determined that the video data is encoded in the video range, CPU 103 executes S304. In S304, CPU 103 converts the threshold calculated in S302 into a threshold corresponding to the video range. Specifically, CPU 103 converts the threshold so that the area for which a highlight warning is issued is the same as when a threshold based on maxDRL encoded in the full range is applied to video data encoded in the full range.

[0037] The threshold value encoded in the video range is Th_v, and the threshold value encoded in the full range is Th_f. When the gradation value is a 10-bit value, the CPU 103 can convert the threshold value using the following formula 1. Th_v=Th_f×(940-64) / 1023+64 Formula 1 It should be noted that any method can be used to round off the decimal point. After converting the threshold value, the CPU 103 ends the threshold value calculation process.

[0038] The CPU 103 instructs the image processing unit 105 to enable the application of highlight warning processing together with the threshold value calculated as above. Record Display video data is generated from the video data read from the medium 106. A threshold value is then applied to the luminance value of each pixel data constituting one frame of the display video data to detect pixel data having a luminance value equal to or greater than the threshold. Processing is then applied so that pixels having a luminance value equal to or greater than the threshold are displayed with a highlight warning. The image processing unit 105 can execute the highlight warning display, for example, by changing the display color of pixels having a luminance value equal to or greater than the threshold to a specific color or by superimposing a predetermined pattern image or the like.

[0039] When maxDRL is encoded in the video range and the video data is encoded in the full range, CPU 103 converts the threshold value using the following formula 1' in S304. Th_f=(Th_v-64)×1023 / (940-64) Equation 1'

[0040] The effect of the threshold calculated in the process of Fig. 3 will be described with reference to Figs. 7(a) and (b). Fig. 7(a) is a frame image represented by video data encoded in full range, with region 701 having a gradation value of 769, region 702 having a gradation value of 766, and region 703 having a gradation value of 762. Fig. 7(b) shows the image in Fig. 7(a) after the encoding method is converted to video range. Regions 704 to 706 correspond to regions 701 to 703, with region 704 having a gradation value of 722, region 705 having a gradation value of 720, and region 706 having a gradation value of 717.

[0041] For example, let us say that maxDRL is encoded over the full range and has a value of 769. Let us also assume that the highlight warning threshold is 764, which is maxDRL minus 5. When this threshold is applied to the image in Figure 7(a), a highlight warning is displayed for areas 701 and 702, excluding area 703. When the same threshold is applied to the image in Figure 7(b), the entire area has a gradation value smaller than the threshold, so there is no area where a highlight warning is displayed.

[0042] When the highlight warning threshold is converted to a threshold for the video range using equation 1, the result is 718. When the converted threshold is applied to the image in FIG. 7(b), a highlight warning is displayed for areas 704 and 705, excluding area 706. In this way, when the video data and maxDRL are encoded in different numerical ranges, the video data or the threshold is converted so that the video data and the threshold based on maxDRL are encoded in the same numerical range. This makes it possible to display an appropriate highlight warning regardless of the encoded numerical range.

[0043] (Example 2) Fig. 4 is a flow chart showing the calculation process of the highlight warning threshold when maxDRL is encoded and recorded in the same range as the video data. Note that the same reference numerals as in Fig. 3 are used for steps similar to those in Fig. 3.

[0044] In S301, the CPU 103 acquires the maxDRL for the video data to be played back. Here, it is assumed that the maxDRL is acquired from the metadata of the video data file recorded on the recording medium 106.

[0045] In S401, the CPU 103 determines whether or not the video data to be played back is encoded in the video range. The CPU 103 can obtain the encoding method of the video data from, for example, metadata of the video data file.

[0046] It is assumed that the calculation of the threshold value in S302 assumes that maxDRL is encoded in the full range. Therefore, if CPU 103 determines in S401 that the video data is encoded in the video range, CPU 103 executes S402. If CPU 103 determines that the video data is not encoded in the video range (encoded in the full range), CPU 103 skips S402 and executes S302.

[0047] In S402, the CPU 103 converts the maxDRL acquired in S301 into a value corresponding to the full range. The maxDRL encoded in the video range is defined as maxDRL_v, and the maxDRL encoded in the full range is defined as maxDRL_f. When the gradation value is a 10-bit value, the CPU 103 can convert the maxDRL using the following formula 2. maxDRL_f=(maxDRL_v-64)×1023 / (940-64) Formula 2 It should be noted that any method can be used to round off the decimal point. After converting maxDRL, the CPU 103 executes step S302.

[0048] In S302, the CPU 103 calculates the highlight warning threshold value to be applied to full-range video data encoded in the manner described above. The processing from S303 onward is omitted since it is the same as that described in FIG.

[0049] In this way, the process of calculating the threshold value from maxDRL in camera 100 is based on a maxDRL encoded in the full range, and appropriate highlight warning display can be achieved even when the maxDRL of the video data is encoded in the same way as the video data.

[0050] A similar method can be used when the process of calculating the threshold value from maxDRL in the camera 100 is based on the maxDRL encoded in the video range. That is, when it is determined in S401 that the video data is encoded in the full range, the CPU 103 converts the maxDRL to a value corresponding to the video range using the following formula 2' in S402. maxDRL_v=maxDRL_f×(940-64) / 1023+64 Formula 2' Thereafter, if it is determined in S303 that the video data has been encoded in the full range, the CPU 103 converts the threshold calculated in S302 into a value corresponding to the full range using the above-mentioned formula 1' in S304.

[0051] (Example 3) Fig. 5 is a flow chart of a calculation process for a highlight warning threshold value that is determined according to an encoding method by referring to a previously prepared table. Here, as in Fig. 4, it is assumed that maxDRL is recorded by the same encoding method as the video data encoding method. Note that the same reference numerals as in Figs. 3 and 4 are used for steps similar to those described in Figs. 3 and 4.

[0052] In S301, the CPU 103 acquires the maxDRL for the video data to be played back. Here, it is assumed that the maxDRL is acquired from the metadata of the video data file recorded on the recording medium 106.

[0053] In S401, the CPU 103 determines whether or not the video data to be played back is encoded in the video range. The CPU 103 can obtain the encoding method of the video data from, for example, metadata of the video data file.

[0054] If it is determined that the moving image data is encoded in the video range, the CPU 103 executes S503. If it is determined that the moving image data is not encoded in the video range (if it is determined that the moving image data is encoded in the full range), the CPU 103 executes S502.

[0055] In S502, the CPU 103 acquires a highlight warning threshold value by referring to the full range table based on the maxDRL acquired in S301. In S503, the CPU 103 acquires a threshold value for a highlight warning by referring to the table for the video range based on the maxDRL acquired in S301.

[0056] Figures 7(c) and (d) show examples of highlight warning threshold tables. A table is prepared for each encoding method and is stored in, for example, the secondary storage device 107. Figure 7(c) is a full range table, and Figure 7(d) is a video range table. Each table associates multiple discrete values ​​of maxDRL, including a minimum and maximum value, with a highlight warning threshold. If maxDRL is encoded in the full range and has a value of 712, S502 obtains a highlight warning threshold of 707. Similarly, if maxDRL is encoded in the video range and has a value of 674, S503 obtains a highlight warning threshold of 669.

[0057] If the acquired maxDRL value does not exist in the table, CPU 103 acquires highlight warning threshold values ​​corresponding to two values ​​between which the acquired maxDRL value is sandwiched among the maxDRL values ​​registered in the table. Then, CPU 103 can obtain a highlight warning threshold value corresponding to the acquired maxDRL by interpolating the two highlight warning threshold values ​​according to the difference between each of the two values ​​and the acquired maxDRL value.

[0058] In this way, by preparing a table that associates maxDRL with the highlight warning threshold for each encoding method, conversion to match different encoding methods is not necessary, and it is possible to simplify the threshold calculation process and realize an appropriate highlight warning display according to the encoding method.

[0059] (Example 4) Fig. 6 is a flow chart of a process for calculating a highlight warning threshold value according to an encoding method directly from maxDRL. As in Fig. 4, it is assumed that maxDRL is recorded using the same encoding method as the video data encoding method. Note that the same reference numerals as in Fig. 3 are used for steps similar to those described in Fig. 3.

[0060] In S301, the CPU 103 acquires the maxDRL for the video data to be played back. Here, it is assumed that the maxDRL is acquired from the metadata of the video data file recorded on the recording medium 106.

[0061] In S601, the CPU 103 judges whether the range of maxDRL for highlight warning is equal to or less than a threshold value. If it is judged that the range of maxDRL for highlight warning is equal to or less than the threshold value, the CPU 103 executes S602, and if not, executes S603. This judgment may be made by judging whether the ratio of the range of maxDRL for highlight warning to the numerical range of the maxDRL encoding method is equal to or less than a threshold value.

[0062] In S602, the CPU 103 calculates the highlight warning threshold by applying a predetermined method to the maxDRL without considering the encoding method of the maxDRL. This is because when the range of the maxDRL for highlight warning is narrow (below the threshold), the difference in the encoding method has little effect on the magnitude of the threshold.

[0063] 7(a) and (b), it is assumed that the maxDRL value 769 encoded in the full range corresponds to the maxDRL value 722 encoded in the video range. It is also assumed that the maxDRL range for highlight warning is 5.

[0064] Here, consider a case where the method of calculating the highlight warning threshold from maxDRL is to subtract 5 from maxDRL. In this case, if maxDRL is encoded in the full range, the threshold calculated by subtracting 5 from maxDRL is 764. When the threshold 764 is converted to a threshold equivalent to the video range using Equation 1, it becomes 718. On the other hand, if maxDRL is encoded in the video range, the threshold calculated by subtracting 5 from maxDRL is 717, which is almost equal to the value obtained by conversion using Equation 1.

[0065] The same applies when multiplying a value by a coefficient instead of subtracting it. If the maxDRL range for highlight warning is 5, the coefficient to multiply is 0.993 (=764 / 769) if maxDRL is encoded in the full range. Multiplying the same coefficient by the maxDRL value of 722 encoded in the video range results in 717.3, which becomes 717 or 718 when rounded to 1, so it is also almost equal to the value 718 obtained by converting 764 using formula 1.

[0066] Therefore, the threshold value used in the judgment in S601 can be determined, for example experimentally, as the maximum maxDRL range in which the difference between the highlight warning threshold value obtained by conversion using Equation 1 and the highlight warning threshold value obtained by the method applied in S602 can be ignored.

[0067] On the other hand, in S603, the CPU 103 calculates the highlight warning threshold value using another method, for example, any of the methods described in Examples 1 to 3.

[0068] In addition to maxDRL, the dynamic range of an image can be grasped using the minimum output luminance value [nits] or the corresponding gradation value minDRL, which changes depending on shooting conditions such as shooting mode and ISO sensitivity. In this case, the above-mentioned method may be applied to the calculation of a threshold value for issuing a highlight warning for a predetermined range on the highlight side of the dynamic range.

[0069] The threshold value used in false color display, which displays an image in a color according to the exposure state, can also be calculated by the method described above. In this case, the threshold value can be calculated after applying a gradation characteristic opposite to the gradation characteristic applied to the image, and making the gradation value and luminance have a linear relationship.

[0070] As described above, according to the present embodiment, it is possible to provide an image processing device and an image processing method capable of appropriately handling luminance data in accordance with the encoding method applied to image data.

[0071] ●(Second embodiment) Next, a second embodiment of the present invention will be described. In this embodiment, the brightness histogram display of an image is appropriately performed according to the encoding method. This embodiment is also executed by the camera 100, as in the first embodiment. In this embodiment, it is also assumed that minDRL is recorded in addition to maxDRL as metadata of the image data file.

[0072] The luminance histogram of image data is generally displayed on the same screen as the image represented by the image data, but it can be displayed in any form. The luminance histogram display process can be applied to still image data or video data, and can be applied before or after recording. Here, the luminance histogram display process is applied to one frame of recorded still image data or video data. It is also assumed that maxDRL and minDRL are encoded in the full range.

[0073] (Example 1) An example of a histogram display process will be described with reference to the flowchart shown in FIG. In S801, the CPU 103 acquires the maxDRL (first information) and minDRL (second information) for the target image data. Here, it is assumed that the maxDRL and minDRL are acquired from the metadata of the image data file recorded on the recording medium 106.

[0074] In S802, the CPU 103 reads out the image data and stores it in the primary storage device 104. Then, the CPU 103 instructs the image processing unit 105 to analyze the luminance distribution of the image data. The image processing unit 105 classifies each pixel into a plurality of bins having a predetermined width based on the luminance value of each pixel data.

[0075] In S803, the CPU 103 determines whether the target image data is encoded in the video range. The CPU 103 can obtain the encoding method of the image data from, for example, metadata of the image data file.

[0076] Here, it is assumed that maxDRL and minDRL are encoded in the full range. Therefore, if CPU 103 determines in S803 that the image data is encoded in the video range, CPU 103 executes S804. If CPU 103 determines that the image data is not encoded in the video range (encoded in the full range), CPU 103 skips S804 and executes S805.

[0077] In S804, the CPU 103 converts the maxDRL and minDRL acquired in S801 into values ​​corresponding to the video range. When the maxDRL and minDRL are 10-bit values, the CPU 103 can execute the conversion using the above-mentioned formula 2'. Formula 2' is a conversion formula for maxDRL, but the same formula can also be used for conversion of minDRL.

[0078] In S805, the CPU 103 instructs the image processing unit 105 to generate histogram display image data together with maxDRL and minDRL. In response to the instruction, the image processing unit 105 generates display image data representing the histogram generated in S802. At this time, the image processing unit 105 generates display image data for a histogram excluding a range with a gradation value larger than maxDRL and a range with a gradation value smaller than minDRL. Note that the bin including minDRL and the bin including maxDRL may or may not be excluded. The range with a gradation value larger than maxDRL and the range with a gradation value smaller than minDRL are called mask regions. The mask region is a luminance region that cannot exist in image data, and can also be called an invalid region. On the other hand, the range from minDRL to maxDRL can be treated as a valid range of luminance. The display image data of the histogram generated by the image processing unit 105 is displayed on the display unit 108.

[0079] The display image data generation process in S805 will be described with reference to FIG. 11(a) is a luminance histogram of image data encoded in the full range. If minDRL is 0 and maxDRL is 769, display image data is generated in which a mask area 1101 that is not included in the range of gradation values ​​0 to 769 is shaded. This allows the user to easily understand the relationship between the upper limit of the number of bits of image data and the upper limit of luminance based on MaxDRL. Of course, display image data may be generated for a histogram in the range of gradation values ​​0 to 769 excluding the mask area 1101.

[0080] Fig. 11(b) shows the luminance histogram of Fig. 11(a) converted into the range of the video range. When minDRL 0 and maxDRL 769 are applied to the luminance histogram of Fig. 11(b), display image data of the histogram is generated in which the areas 1102 and 1103 that should be masked are not shaded. In this case, the user may mistakenly think that the areas 1102 and 1103 are part of the dynamic range.

[0081] Fig. 11(c) shows the state where maxDRL and minDRL converted to values ​​corresponding to the video range are applied to the luminance histogram of Fig. 11(b). After conversion, minDRL is 64 and maxDRL is 722, so areas 1102 and 1103 in Fig. 11(b) are included in mask areas 1104 and 1105. This makes it possible to display an appropriate histogram even when image data is encoded in the video range.

[0082] When performing histogram display on unrecorded image data, such as image data for live view display, maxDRL and minDRL according to the shooting conditions can be obtained from the secondary storage device 107. The other processing is the same as that described with reference to FIG.

[0083] Also, if maxDRL and minDRL are encoded in the video range and image data is encoded in the full range, maxDRL and minDRL can be converted to values ​​equivalent to the full range using Equation 2. Equation 2 is a conversion equation for maxDRL, but the same equation can also be used to convert minDRL.

[0084] (Example 2) Another example of the histogram display process will be described with reference to the flowchart shown in Fig. 9. In this example, the maxDRL and minDRL are also assumed to be encoded in the full range. The same reference numerals as in Fig. 8 are used to designate the same steps as those in Fig. 8.

[0085] In S901, the CPU 103 reads the target image data from the recording medium 106 and stores it in the primary storage device 104. The CPU 103 also determines whether the target image data is encoded in the video range. The CPU 103 can obtain the encoding method of the image data from, for example, metadata of the image data file.

[0086] This assumes that maxDRL and minDRL are encoded in the full range. 1 If the CPU 103 determines that the image data is encoded in the video range, it executes S902. If the CPU 103 determines that the image data is not encoded in the video range (encoded in the full range), it skips S902 and executes S801.

[0087] In S902, the CPU 103 instructs the image processing unit 105 to convert the luminance value of the image data to a value equivalent to the full range. The image processing unit 105 can obtain the luminance value (gradation value) after conversion by substituting the luminance value (gradation value) before conversion into maxDRL_v in the formula 2. Note that, in order to reduce the processing load, the conversion may be applied after reducing the resolution of the image data for display.

[0088] Thereafter, the processes in S801, S802, and S805 are the same as those described with reference to FIG. 8, and therefore repeated description will be omitted. In this example as well, a histogram similar to that shown in FIG. 11(a) can be displayed.

[0089] In addition, when maxDRL and minDRL are encoded in the video range, the histogram display as shown in Fig. 11(c) can be realized by converting the image data to values ​​equivalent to the video range when the image data is encoded in the full range. The image data can be converted using Equation 2'.

[0090] (Example 3) Another example of the histogram display process will be described with reference to the flowchart shown in Fig. 10. In this example, the maxDRL and minDRL are also assumed to be encoded in the full range. The same reference numerals as in Fig. 8 are used to designate the same steps as those in Fig. 8.

[0091] S801 to S803 are as described with reference to FIG.

[0092] Here, it is assumed that maxDRL and minDRL are encoded in the full range. Therefore, if it is determined in S803 that the image data is encoded in the video range, CPU 103 executes S1002. Also, if it is determined that the image data is not encoded in the video range (encoded in the full range), CPU 103 skips S1002 and executes S805.

[0093] In S1002, the CPU 103 instructs the image processing unit 105 to convert the histogram calculated in S802 into a histogram equivalent to the full range. This corresponds to converting the histogram shown in Fig. 11(b) into the histogram shown in Fig. 11(a) (the mask area is not considered here).

[0094] The image processing unit 105 may, for example, regenerate the histogram by converting the luminance value of each pixel to a value equivalent to the full range, or may convert the histogram generated in S802 by modifying it.

[0095] When the histogram conversion process in the image processing unit 105 is completed, the CPU 103 instructs the image processing unit 105 to generate image data for displaying a histogram together with the maxDRL and minDRL in S805.

[0096] If maxDRL and minDRL are encoded in the video range, the histogram can be converted to a histogram corresponding to the video range when the image data is encoded in the full range. This results in a histogram with shaded mask areas 1104 and 1105 as shown in FIG. 11(c).

[0097] Although the present embodiment has been described with a luminance histogram as an example, the same processing can be performed with an RGB histogram. Furthermore, the present embodiment is not limited to the display of a luminance histogram, and can be applied to other uses. For example, the mask area in the luminance waveform monitor display can be determined based on maxDRL and minDRL. In this case, maxDRL and minDRL can be converted to values ​​in which the maximum luminance in the dynamic range of the camera 100 (image sensor) is set to 100%. FIG. 12 shows an example of the display on the waveform display monitor when minDRL is 0% and maxDRL is 70%, and an area 1201 is the mask area.

[0098] This embodiment also makes it possible to provide an image processing device and an image processing method capable of appropriately handling luminance data in accordance with the encoding method applied to image data.

[0099] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0100] The disclosure of the present embodiment includes the following image processing device, imaging device, image processing method, and program. (Item 1) an acquisition means for acquiring image data and first information relating to a maximum luminance value of an output dynamic range of the image data; and a processing means for applying to the image data a process using a luminance value of the image data and a first value based on the first information, The image processing device is characterized in that, when the image data is encoded in a first numerical range and the first information is encoded in a second numerical range different from the first numerical range, the processing means converts the image data, the first information, or the first value so that the image data and the first value become values ​​that would be obtained if encoded in the same numerical range. (Item 2) 2. The image processing device according to item 1, wherein the processing means converts the first value so that the image data and the first value are values ​​when encoded into the same numerical range. (Item 3) When the process of generating the first value from the first information assumes that the first information is encoded in the first numerical range and the image data is encoded in the second numerical range, converting the first information into a value encoded in the first numerical range; generating the first value based on the transformed value; converting the generated first value into a value encoded in the second range of values; applying the processing to the image data; 2. The image processing device according to item 1, (Item 4) The image processing device according to item 1, characterized in that when the image data and the first information are encoded in the same numerical range, the processing means generates the first value from the first information by referring to a table prepared in advance for each of the first numerical range and the second numerical range based on the encoded numerical range. (Item 5) 2. The image processing device according to item 1, wherein the conversion is not performed when the ratio of the first value to the second numerical range is equal to or less than a threshold value. (Item 6) 6. The image processing device according to any one of items 1 to 5, wherein the first numerical range is a part of the second numerical range. (Item 7) 7. The image processing device according to any one of items 1 to 6, wherein the processing is a highlight warning processing that presents an area of ​​the image data having a luminance equal to or greater than the first value. (Item 8) the acquiring means further acquires second information relating to a minimum luminance value of an output dynamic range of the image data; the processing means applies to the image data a process using a luminance value of the image data, the first value, and a second value based on the second information; When the image data is encoded in a first numerical range and the first and second information are encoded in a second numerical range different from the first numerical range, the processing means converts the image data, the first and second information, or the first and second values ​​so that the image data and the first and second values ​​are values ​​that are obtained when encoded in the same numerical range, and then applies the processing to the image data. 8. The image processing device according to any one of items 1 to 7, (Item 9) 9. The image processing device according to item 8, wherein the processing means converts the first and second values ​​so that the image data and the first and second values ​​become values ​​when encoded into the same numerical range. (Item 10) 9. The image processing device according to item 8, wherein the processing means converts the value of the image data so that the image data and the first and second values ​​are encoded into the same numerical range. (Item 11) 11. The image processing device according to any one of items 8 to 10, wherein the process is a process that treats a luminance range from the second value to the first value as a valid range. (Item 12) 12. The image processing device according to any one of items 8 to 11, wherein the processing is a display process of a luminance histogram or a luminance waveform monitor. (Item 13) the process is a display process of a luminance histogram, 13. The image processing device according to any one of items 8 to 12, wherein the processing means displays a luminance histogram excluding the range equal to or less than the second value and the range equal to or more than the first value. (Item 14) the process is a display process of a luminance histogram, The processing means includes: converting a histogram generated based on the image data into a histogram when the image data is encoded into the second numerical range; applying a process using the first and second values ​​to the converted histogram and displaying the result; 13. The image processing device according to any one of items 8 to 12, (Item 15) the processing is a waveform monitor display processing, the processing means performs a waveform monitor display with a range excluding the range from the second value to the first value as a mask region. 13. The image processing device according to any one of items 8 to 12, (Item 16) An imaging element; 16. The image processing device according to any one of items 1 to 15, which uses image data obtained by using the imaging element; An imaging device comprising: (Item 17) An image processing method executed by an image processing device, comprising: Obtaining image data and first information relating to a maximum luminance value of an output dynamic range of the image data; applying to the image data a process using a luminance value of the image data and a first value based on the first information; Applying the treatment comprises: If the image data is encoded in a first range of values ​​and the first information is encoded in a second range of values ​​different from the first range of values, converting the image data, the first information, or the first value such that the image data and the first value have values ​​when encoded into the same numerical range; applying said processing to said image data after said converting. (Item 18) A program for causing a computer to function as each of the means possessed by the image processing device according to any one of items 1 to 15.

[0101] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]

[0102] 100... digital camera, 101... optical system, 102... imaging element, 103... CPU, 104... primary storage device, 105... image processing unit, 106... recording medium, 107... secondary storage device, 108... display unit, 109... operation unit

Claims

1. acquiring means for acquiring image data and first information relating to a maximum luminance value of an output dynamic range of the image data; a processing means for applying to the image data a process using a luminance value of the image data and a first value based on the first information, The image processing device is characterized in that, when the image data is encoded in a first numerical range and the first information is encoded in a second numerical range different from the first numerical range, the processing means converts the image data, the first information, or the first value so that the image data and the first value become values ​​that would be obtained if encoded in the same numerical range.

2. 2. The image processing apparatus according to claim 1, wherein said processing means converts said first value so that said image data and said first value have values ​​in the same numerical range when encoded.

3. When the process of generating the first value from the first information assumes that the first information is encoded in the first numerical range, and the image data and the first information are encoded in the second numerical range, converting the first information into a value encoded in the first numerical range; generating the first value based on the transformed value; converting the generated first value into a value encoded in the second range of values; applying the processing to the image data; 2. The image processing device according to claim 1,

4. The image processing device according to claim 1, characterized in that, when the image data and the first information are encoded in the same numerical range, the processing means generates the first value from the first information by referring to tables prepared in advance for each of the first numerical range and the second numerical range based on the encoded numerical range.

5. 2 . The image processing apparatus according to claim 1 , wherein the conversion is not performed when a ratio of the range equal to or greater than the first value to the second numerical range is equal to or less than a threshold value.

6. 2. The image processing apparatus according to claim 1, wherein the first range of values ​​is a part of the second range of values.

7. 2. The image processing apparatus according to claim 1, wherein the processing is a highlight warning processing for presenting an area of ​​the image data having a luminance equal to or greater than the first value.

8. the acquiring means further acquires second information relating to a minimum luminance value of an output dynamic range of the image data; the processing means applies to the image data a process using a luminance value of the image data, the first value, and a second value based on the second information; When the image data is encoded in a first numerical range and the first and second information are encoded in a second numerical range different from the first numerical range, the processing means converts the image data, the first and second information, or the first and second values ​​so that the image data and the first and second values ​​become values ​​when encoded in the same numerical range, and then applies the processing to the image data.

2. The image processing device according to claim 1,

9. 9. The image processing device according to claim 8, wherein the processing means converts the first and second values ​​so that the image data and the first and second values ​​have values ​​in the same numerical range when encoded.

10. 9. The image processing apparatus according to claim 8, wherein said processing means converts the value of said image data so that said image data and said first and second values ​​have values ​​when encoded in the same numerical range.

11. 9. The image processing apparatus according to claim 8, wherein the processing is a processing that treats a luminance range from the second value to the first value as a valid range.

12. 9. The image processing apparatus according to claim 8, wherein the processing is a display process of a luminance histogram or a luminance waveform monitor.

13. the process is a display process of a luminance histogram, 9. The image processing apparatus according to claim 8, wherein said processing means displays a luminance histogram excluding the range equal to or less than said second value and the range equal to or more than said first value.

14. the process is a display process of a luminance histogram, The processing means includes: converting a histogram generated based on the image data into a histogram when the image data is encoded into the second numerical range; applying a process using the first and second values ​​to the transformed histogram and displaying the result; 9. The image processing device according to claim 8,

15. the processing is a display processing of a waveform monitor, the processing means displays a waveform monitor with a range excluding the range from the second value to the first value as a mask region.

9. The image processing device according to claim 8,

16. An imaging element; The image processing device according to claim 1 , which uses image data obtained by the imaging element; An imaging device comprising:

17. An image processing method executed by an image processing device, comprising: obtaining image data and first information relating to a maximum luminance value of an output dynamic range of the image data; applying to the image data a process using a luminance value of the image data and a first value based on the first information; Applying the treatment comprises: When the image data is encoded in a first range of values ​​and the first information is encoded in a second range of values ​​different from the first range of values, converting the image data, the first information, or the first value such that the image data and the first value have values ​​when encoded into the same numerical range; applying said processing to said image data after said converting.

18. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 15.

Images