Image display device and program

Abstract

To display beautiful skin colors even for contents in which environments (conditions) during shooting or editing are unknown.SOLUTION: An image display device which performs image processing on input video data and outputs it to a display unit, comprises: a face detection unit that detects a face of a person from the video data; a histogram detection unit that performs histogram detection related to color information from the video data; a continuous scene determination unit that determines a continuous scene using the histogram detection result of the histogram detection unit; and a color correction unit that performs color correction on the video data. The color correction unit maintains the previous color correction when the continuous scene determination unit has determined the continuous scene even in a case where the face detection result of the face detection unit has changed from face detection to no-face detection.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] Embodiments of the present invention relate to an image display device and Program mushi and are related thereto.

Background Art

[0002] Conventionally, in imaging devices such as digital cameras, technologies have been disclosed in which a face is detected from a captured image of a person using AI (Artificial Intelligence) technology or the like, and the skin color portion of the detected face is color-corrected.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when attempting to apply the conventional technology to an image display device such as a television, the environment during shooting and editing of the input video is unknown, and the video characteristics also differ for each content or scene. Therefore, there is a problem that it is difficult to perform color correction suitable for the content or scene.

Means for Solving the Problems

[0005] The image display device according to the embodiment is an image display device that performs image processing on input video data and outputs it to a display unit, and includes a face detection unit that detects a face of a person from the video data, a histogram detection unit that detects a histogram related to color information from the video data, and a continuous scene determination unit that calculates a similarity between a current frame and a frame before the current frame using a histogram detection result in the histogram detection unit, and determines that the scene is continuous when the similarity is equal to or greater than a predetermined value. Using the face detection result in the face detection unit and the histogram detection result in the histogram detection unit, the continuous scene determination in the continuous scene determination unit and a color shift detection unit that detects a shift degree of a hue or saturation of skin color in the video data, and a color correction parameter generation unit that generates a parameter for color correction based on the shift degree of the hue or saturation of the skin color detected by the color shift detection unit, a color correction unit that performs color correction of the video data based on the parameters for color correction; and is provided with the color shift detection unit, even when the detection result in the face detection unit changes from having detected a face to not having detected a face, wherein when it is determined as a continuous scene by the continuous scene determination unit, the previous shift degree detection information is maintained and the color correction unit is based on the parameters for color correction generated based on the previously maintained degree of shift and color correction is continued.

Brief Description of Drawings

[0006]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

[0007] An image display device will be described in detail below with reference to the accompanying drawings. and progra mushi in detail.

[0008] FIG. 1 is a schematic diagram showing an example of an image display device 1 according to an embodiment.

[0009] The image display device 1 includes an image processing unit 10 and a display unit 12. The image display device 1 transfers the data obtained by performing image processing in the image processing unit 10 to the display unit 12 and performs display on the display unit 16.

[0010] Video data 40 is input to the image processing unit 10 from a tuner unit (not shown), an HDMI (High-Definition Multimedia Interface) (registered trademark) terminal, etc. The image processing unit 10 outputs the video data 42 obtained by performing image processing on the input video data 40 to the display unit 12.

[0011] The image processing unit 10 includes an image quality adjustment unit 20 and a skin image correction unit 22.

[0012] The image quality adjustment unit 20 has a function of adjusting gamma, color tone, brightness, etc. for the input video data 40. The image quality adjustment unit 20 outputs the video data 41 obtained by adjusting the input video data 40 to the skin image correction unit 22.

[0013] The skin image correction unit 22 performs real-time determination of the correction target scene and color correction on the video data 41 input from the image quality adjustment unit 20 in order to perform good video display even when the environment (conditions) at the time of shooting or editing is unknown.

[0014] Here, FIG. 2 is a functional block diagram showing the functional configuration of the skin image correction unit 22.

[0015] As shown in FIG. 2, the skin image correction unit 22 includes a face detection unit 23, a histogram detection unit 24, a color shift degree calculation unit 25 which is a color shift detection unit, a color correction parameter generation unit 26, a color correction unit 27, and a continuous scene determination unit 28.

[0016] Generally, the skin image correction unit 22 uses the face detection result in the face detection unit 23, the histogram detection result in the histogram detection unit 24, and the continuous scene determination in the continuous scene determination unit 28 to detect the shift degree of the hue or saturation of the skin color of the video data 41 (input video) input from the image quality adjustment unit 20 in the color shift degree calculation unit 25. The skin image correction unit 22 generates correction parameters corresponding to the detected shift degree of the hue or saturation by the color correction parameter generation unit 26 based on the shift degree of the hue or saturation detected in the color shift degree calculation unit 25, and performs color correction by the color correction unit 27.

[0017] Note that the face detection unit 23, the histogram detection unit 24, the color shift degree calculation unit 25, the color correction parameter generation unit 26, and the color correction unit 27 may be realized by hardware, software, or both hardware and software.

[0018] The display unit 12 includes a display driving unit 14 and a display unit 16. The display unit 16 is a display for displaying an image. The display unit 16 is, for example, an organic EL (Organic Light Emitting Diode) display, a liquid crystal display, or the like.

[0019] The display driving unit 14 converts the video data 42 input from the image processing unit 10 into a voltage and current for driving the display unit 16, and drives the display unit 16.

[0020] Next, each part constituting the skin image correction unit 22 will be described in detail.

[0021] The face detection unit 23 detects a person's face from the video data 41 input from the image quality adjustment unit 20. Here, FIG. 3 is a diagram showing an example of a face detection method in the face detection unit 23. As shown in FIG. 3, as a face detection method, the video data 41 is divided into a plurality of areas using a DNN (Deep Neural Network) pre-trained to be able to detect various faces, and the probability (0 to 1) that each area contains a person's face is obtained. According to such a method, the numerical value of the area containing a person's face is high, and the numerical value of the area not containing a person's face is low. Therefore, the face detection unit 23 detects an area with a value larger than a certain threshold as having a person's face, and an area with a value smaller than the threshold as not having a person's face. When the area with a person's face is equal to or larger than a certain area, the face detection unit 23 detects it as a person's face.

[0022] The histogram detection unit 24 performs a hue histogram detection on the video data 41 input from the image quality adjustment unit 20. Here, FIG. 4 is a diagram showing an example of a histogram detection method in the histogram detection unit 24. FIG. 4(a) shows the hue distribution based on CbCr in the YCbCr signal, and FIG. 4(b) shows the YCbCr color space. The histogram detection by the histogram detection unit 24 is, for example, to obtain the number of pixels distributed in a specific hue in the entire image related to the video data 41 or a partial area of the image related to the video data 41 including a person's face and create a histogram.

[0023] Here, FIG. 5 is a diagram showing an example of imposing restrictions on histogram detection. As shown in FIG. 5, in histogram detection, the histogram detection unit 24 may impose restrictions on the detection range of saturation in the three-dimensional YCbCr color space (FIG. 5(a)), or may impose restrictions on the detection range of luminance in the three-dimensional YCbCr color space (FIG. 5(b)). In this embodiment, restrictions are imposed on the ranges of saturation and luminance in the three-dimensional YCbCr color space, but the present invention is not limited to this. As long as the color (hue, saturation) can be corrected so as to look natural and beautiful, the present invention is not limited to this.

[0024] Even when face detection is not performed, the continuous scene determination unit 28 performs continuous scene determination so that correction can be continued when the video characteristics for each content and scene are the same. Specifically, the continuous scene determination unit 28 determines whether the scene is a continuous scene or a discontinuous scene based on the degree of similarity between the previous frame and the current frame based on the histogram detection of the hue by the histogram detection unit 24.

[0025] The purpose of continuous scene determination is to distinguish whether scenes with the same tendency, such as the color being reddish or bluish due to lighting during shooting or color grading, continue, and whether a sense of incongruity occurs when the correction is switched.

[0026] The continuous scene determination unit 28 determines a case where there is movement in the subject, such as when the orientation of the face changes, or a case where the camera slowly zooms in or out as a continuous scene. In addition, the continuous scene determination unit 28 also determines a case where the cut is changed (angle switching) as an object of a continuous scene.

[0027] In this embodiment, as shown in FIG. 6 for example, the continuous scene determination unit 28 uses a method of calculating the similarity between the previous frame and the current frame based on the number of pixels in the hue region of the CbCr color space to determine a continuous scene. For example, the continuous scene determination unit 28 calculates the similarity according to the cumulative difference in the frequency of all hues or the frequency difference of a specific hue between the previous frame and the current frame of the video data, which will be described in detail below.

[0028] FIG. 6 is a diagram showing an example of determining a continuous scene based on the number of pixels in the hue region of the CbCr color space. The continuous scene determination unit 28 divides the hue into 32 parts in the CbCr color space for the YCbCr input signal as shown in FIG. 6(a), sets the number of pixels of the input data corresponding to each region as H[n], and the number of pixels of the previous frame as Ho[n]. When the following formula (1) is satisfied, it is determined as a continuous scene; when the following formula (1) is not satisfied, it is determined as a discontinuous scene.

[0029]

Equation

[0030] In addition, the continuous scene determination unit 28 may determine that it is a continuous scene when the following formula (2) without squaring is satisfied, and determine that it is a discontinuous scene when the following formula (2) is not satisfied.

[0031]

Equation

[0032] In addition, the continuous scene determination unit 28 may determine that it is a continuous scene when the following formula (3) which is only a part of the hue is satisfied, and determine that it is a discontinuous scene when the following formula (3) is not satisfied.

[0033]

Equation

[0034] The skin color hue shift degree calculation unit 25 detects the skin color hue shift degree of the video data 41 input from the image quality adjustment unit 20 by using the face detection result in the face detection unit 23, the histogram detection result in the histogram detection unit 24, and the continuous scene determination in the continuous scene determination unit 28. For example, the skin color hue shift degree calculation unit 25 detects the degree to which the skin color hue of the video data 41 input from the image quality adjustment unit 20 is shifted to red or yellow.

[0035] For example, when the skin color hue has shifted to red, the skin color hue shift degree calculation unit 25 calculates a "shift amount from red" indicating the degree of shift to red. Also, when the skin color hue has shifted to yellow, the skin color hue shift degree calculation unit 25 calculates a "shift amount from yellow" indicating the degree of shift to yellow. On the other hand, when the skin color hue has not shifted to red and when the skin color hue has not shifted to yellow, the skin color hue shift degree calculation unit 25 sets the "shift amount from red" and the "shift amount from yellow" indicating the degree of shift to red to 0.

[0036] In addition, even when the detection result in the face detection unit 23 changes from "face detected" to "no face detected", if continuous scene determination has been performed, the skin color hue shift degree calculation unit 25 maintains the previous skin color hue shift degree detection information.

[0037] Here, FIG. 7 is a diagram showing an example of calculating the shift degree by the skin color hue shift degree calculation unit 25. The calculation of the degree to which the skin color hue is shifted to red is, for example, to obtain the central value C of the skin color hue from the data of the obtained hue histogram as shown in FIG. 7(a) by the following formula (4).

[0038]

Equation

[0039] As shown in FIG. 7(b), the skin color hue shift degree calculation unit 25 determines that the smaller the central value C of the skin color hue, the larger the shift amount, and obtains the shift degree from red.

[0040] Based on the "shift amount from red" or "shift amount from yellow" detected by the color shift degree calculation unit 25, the color correction parameter generation unit 26 generates parameters for color correction to be applied to the color correction unit 27.

[0041] For example, assume that the color correction parameter generation unit 26 has a standard color management table (CMS[m]) to be applied when there is no hue shift, a color management table (CMR[m]) that is optimal when the red shift degree (RH; 0 to 1) is 1, and a color management table (CMY[m]) that is optimal when the yellow shift degree (YH; 0 to 1) is 1.

[0042] Based on the red shift degree, the color correction parameter generation unit 26 sets the parameter for color correction to be finally applied to the color correction unit 27 as the color management table (CM[m]) to be finally applied to the color correction unit 27. Here, CM[m] consists of elements of hue H[h], saturation S[s], and lightness V[v]. That is, m = h × s × v.

[0043] The color correction parameter generation unit 26 uses the standard color management table (CMS[m]) to be adapted when there is no hue shift. Also, the color correction parameter generation unit 26 uses the color management table (CMR[m]) that is optimal when the red shift degree (RH, 0 to 1) is 1. Also, the color correction parameter generation unit 26 uses the color management table (CMY[m]) that is optimal when the yellow shift degree (YH, 0 to 1) is 1.

[0044] When both the red shift degree and the yellow shift degree in the detection result by the color shift degree calculation unit 25 are 0, CM[m] = CMS[m] and sets it as the parameter for color correction to be applied to the color correction unit 27.

[0045] Also, when the red shift degree in the detection result by the color shift degree calculation unit 25 is not 0, CM[m] = RH × CMR[m] + (1 - RH) × CMS[m] is used as the color correction parameter applied to the color correction unit 27.

[0046] Also, when the degree of yellow shift in the detection result by the color shift degree calculation unit 25 is not 0, the color correction parameter generation unit 26 CM[m] = YH × CMY[m] + (1 - YH) × CMS[m] is used as the color correction parameter applied to the color correction unit 27.

[0047] Here, when CMS[m] is a value with a little red added to the skin color area, CMR[m] is a value obtained by slightly subtracting (minus) the amount of red added, and CMY[m] is a value obtained by slightly adding (plus) the amount of red added. When the video data 41 (input video) input from the image quality adjustment unit 20 is shifted more towards red, excessive correction can be suppressed. Also, when the video data 41 (input video) input from the image quality adjustment unit 20 is shifted more towards yellow, insufficient correction can be compensated for.

[0048] The color correction unit 27 performs color correction on the video data 41 input from the image quality adjustment unit 20 based on the color management table (CM[m]) applied by the color correction parameter generation unit 26.

[0049] FIG. 8 is a diagram showing an example of color correction in the color correction unit 27. CM[m] consists of elements of hue H[h], saturation S[s], and lightness V[v]. Hue H[h], saturation S[s], and lightness V[v] represent the degrees of correction of hue, saturation, and luminance at the point where h, s, and v overlap (see FIG. 8(d)) in the three-dimensional YCbCr space.

[0050] Note that the color correction unit 27 performs interpolation for the values in the middle of the points using the correction values of adjacent points.

[0051] Next, an example of the flow of the skin image correction process executed by the skin image correction unit 22 will be described.

[0052] FIG. 9 is a flowchart showing an example of the flow of skin image correction processing executed by the skin image correction unit 22. As shown in FIG. 9, the face detection unit 23 divides the video data 41 input from the image quality adjustment unit 20 into a plurality of areas, and detects whether a human face is included in each area (step S1).

[0053] When the area with a human face detected by the face detection unit 23 is equal to or larger than a certain area (Yes in step S2), the face detection unit 23 proceeds to step S3 on the assumption that a human face has been detected in the video data 41. On the other hand, when the area with a human face detected by the face detection unit 23 is smaller than a certain area (No in step S2), the face detection unit 23 proceeds to step S5 on the assumption that no human face has been detected in the video data 41.

[0054] In step S3, the histogram detection unit 24 acquires the hue histogram from the video data 41 input from the image quality adjustment unit 20.

[0055] Subsequently, the color shift degree calculation unit 25 calculates the "shift amount from red", which indicates the degree to which the hue of the skin color of the video data 41 input from the image quality adjustment unit 20 is shifted from red, using the face detection result in the face detection unit 23 and the histogram detection result in the histogram detection unit 24 (step S4).

[0056] Next, the color correction parameter generation unit 26 generates parameters for color correction to be applied to the color correction unit 27 based on the "shift amount from red" detected by the color shift degree calculation unit 25 (step S8).

[0057] Finally, the color correction unit 27 performs color correction on the video data 41 input from the image quality adjustment unit 20 based on the color correction parameters applied by the color correction parameter generation unit 26 (step S9).

[0058] On the other hand, when the area with a human face is smaller than a certain area (No in step S2), the continuous scene determination unit 28 determines whether the scene is continuous based on the hue histogram detection by the histogram detection unit 24 (step S5).

[0059] When the continuous scene determination unit 28 determines that it is a continuous scene (Yes in step S5), it proceeds to step S6. On the other hand, when the continuous scene determination unit 28 determines that it is not a continuous scene (No in step S5), it proceeds to step S7.

[0060] That is, even when no face is detected (No in step S2), if a continuous scene is determined (Yes in step S5), the color shift degree calculation unit 25 maintains (holds) the calculated value of the previous frame (step S6) and proceeds to step S8.

[0061] On the other hand, when no face is detected (No in step S2) and no continuous scene is determined (No in step S5), since it is not a scene to be corrected, the "shift amount from red", which indicates the degree to which the hue of the skin color of the video data 41 input from the image quality adjustment unit 20 has shifted from red, is set to 0 and reset (step S7), and the process proceeds to step S8.

[0062] As described above, according to the image display device of the embodiment, by detecting face detection, the degree of hue shift, and continuous scene determination in real time and performing skin color correction according to the characteristics of the input video, it is possible to display a beautiful skin color even for content where the environment (conditions) during shooting, editing, etc. are unknown, and to perform good video display.

[0063] In particular, according to the image display device of the embodiment, continuous scene determination is performed based on the acquired information of the hue histogram. Even when the face detection changes from presence to absence, when it is determined to be a continuous scene, by continuing the color correction, appropriate color correction can be performed even in a scene where no face is detected, and optimal color display without a sense of incongruity can be performed in more scenes.

[0064] Note that in this embodiment, the color shift degree calculation unit 25 is configured to hold the detection result when the detection result at the face detection unit 23 changes from having detected a face to not having detected a face, but only when it has been determined as a continuous scene by the continuous scene determination unit 28. However, this is not the only case. That is, as long as the color correction unit 27 can continue color correction when the detection result at the face detection unit 23 changes from having detected a face to not having detected a face, but only when it has been determined as a continuous scene by the continuous scene determination unit 28, it is not necessary to switch without correction. Instead, the change in the correction amount may be fixed, made small, or changed gradually.

[0065] In general scene change detection and the like, it is often determined by the difference in pixel values between the previous and subsequent frames or the difference in applications. However, when there is movement in the subject such as when the orientation of the face changes, or when the camera slowly zooms in or out, it is often possible to determine that it is a continuous scene. However, it is difficult to determine that it is a continuous scene when the cut ratio changes. On the other hand, if the elements such as the lighting, subject, and color grading at the time of shooting are the same, the hue histogram can obtain similar characteristics. Therefore, according to the image display device of the first embodiment, not only when there is movement in the subject such as when the orientation of the face changes or when the camera slowly zooms in or out, but also when the cut ratio changes, it is possible to accurately determine that it is a continuous scene.

[0066] Note that the program (software) executed by the image display device 1 of this embodiment is provided by being pre-embedded in a ROM or the like.

[0067] The program (software) executed by the image display device 1 of this embodiment may be configured to be recorded and provided on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, DVD (Digital Versatile Disc) in an installable format or an executable format file.

[0068] Furthermore, the program (software) executed by the image display device 1 of the present embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the program (software) executed by the image display device 1 of the present embodiment may be configured to be provided or distributed via a network such as the Internet.

[0069] Note that the embodiments of the present invention have been described above, but the above embodiments are presented as examples and are not intended to limit the scope of the invention. This novel embodiment can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. The above embodiments are included in the scope and gist of the invention and are included in the invention described in the claims and the equivalent scope thereof.

Explanation of Reference Numerals

[0070] 1 Image display device 12 Display unit 23 Face detection unit 24 Histogram detection unit 25 Color shift detection unit 26 Color correction parameter generation unit 27 Color correction unit 28 Continuous scene determination unit

Claims

1. An image display device that performs image processing on input video data and outputs it to a display unit, comprising: a face detection unit that detects a human face from the video data; a histogram detection unit that detects a histogram related to color information from the video data; a continuous scene determination unit that calculates the similarity between the current frame and a frame before the current frame using the histogram detection result in the histogram detection unit, and determines that the scene is continuous when the similarity is equal to or greater than a predetermined value; a color shift detection unit that detects the degree of shift in the hue or saturation of the skin color of the video data using the face detection result in the face detection unit, the histogram detection result in the histogram detection unit, and the continuous scene determination in the continuous scene determination unit; a color correction parameter generation unit that generates parameters for color correction based on the degree of shift in the hue or saturation of the skin color detected by the color shift detection unit; a color correction unit that performs color correction on the video data based on the parameters for color correction; and even when the detection result in the face detection unit changes from having detected a face to not having detected a face, the color shift detection unit maintains the detection information on the previous degree of shift when it has been determined as a continuous scene in the continuous scene determination unit; the color correction unit continues color correction based on the parameters for color correction generated based on the maintained previous degree of shift. An image display device characterized by the above.

2. The continuous scene determination unit determines whether it is a continuous scene or a discontinuous scene based on the degree of similarity to the hue histogram of the previous frame of the video data. The image display device according to claim 1, characterized by the above.

3. The continuous scene determination unit calculates the similarity according to the cumulative difference in the frequency of all hues or the cumulative difference in the frequency of a specific hue between the previous frame and the current frame of the video data. The image display device according to claim 2, characterized by the above.

4. A computer that controls an image display device that performs image processing on input video data and outputs it to a display unit, comprising: a face detection unit that detects a human face from the video data; a histogram detection unit that detects a histogram related to color information from the video data; Using the histogram detection result in the histogram detection unit, calculate the similarity between the current frame and the frame before the current frame. When the similarity is equal to or greater than a predetermined value, a continuous scene determination unit that determines that the scenes are continuous; A color shift detection unit that detects the degree of shift in the hue or saturation of the skin color of the video data using the face detection result in the face detection unit, the histogram detection result in the histogram detection unit, and the continuous scene determination in the continuous scene determination unit; A color correction parameter generation unit that generates parameters for color correction based on the degree of shift in the hue or saturation of the skin color detected by the color shift detection unit; A color correction unit that performs color correction on the video data based on the parameters for color correction; Function as; Even when the detection result in the face detection unit changes from having detected a face to not having detected a face, the color shift detection unit maintains the detection information on the previous degree of shift if it has been determined as a continuous scene by the continuous scene determination unit; The color correction unit continues color correction based on the parameters for color correction generated based on the maintained previous degree of shift; A program characterized by the above.

Images