Image display device, non-volatile storage medium, and continuous scene determination method

Abstract

The invention provides an image display apparatus, a non-volatile storage medium, and a continuous scene determination method. Even if the content of an environment (condition) at the time of shooting or editing is not known, beautiful skin color is displayed. An image display device that performs image processing on input video data and outputs the processed video data to a display unit, the image display device being provided with: a face detection unit that detects a face on the basis of the video data; a histogram detection unit that performs histogram detection relating to color information on the basis of the video data; a continuous scene determination unit that determines a continuous scene using a histogram detection result by the histogram detection unit; and a color correction unit that performs color correction of the video data, the color correction unit continuing the color correction when the continuous scene determination unit determines that the scene is continuous even if the detection result by the face detection unit changes from the detection of a human face to the detection of no human face.

Description

Image display device, non-volatile storage medium, and continuous scene determination method

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Japanese patent application No. 2023-058826 filed with the Japan Patent Office on March 31, 2023, and entitled “Image display device, program, and continuous scene determination method,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] Embodiments of the present application relate to an image display device, a non-volatile storage medium, and a method for determining continuous scenes. Background Art

[0004] Conventionally, there has been disclosed a technology for detecting a face from an image of a person using AI (Artificial Intelligence) technology in an imaging device such as a digital camera, and performing color correction on the skin color of the detected face.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent No. 4983643.

[0008] Summary of the Invention

[0009] However, when attempting to apply existing technologies to image display devices such as televisions, there is a problem: since the environment when the input video is shot and edited is unknown, and the visual characteristics of each content or scene are also different, it is difficult to perform color correction appropriate for the content and scene.

[0010] An image display device in an embodiment is an image display device that performs image processing on input video data and outputs the image to a display unit, and comprises: a face detection unit that detects a face based on the video data; a histogram detection unit that performs histogram detection related to color information based on the video data; a continuous scene determination unit that uses the histogram detection result of the histogram detection unit to determine continuous scenes; and a color correction unit that performs color correction on the video data, wherein the color correction unit continues to perform color correction even if the detection result in the face detection unit changes from detecting a face to detecting no face, if the continuous scene determination unit determines that the scene is continuous. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG1 is a schematic diagram showing an example of an image display device according to an embodiment;

[0012] FIG2 is a functional block diagram showing the functional structure of a skin image correction unit;

[0013] FIG3 is a diagram showing an example of a face detection method in a face detection unit;

[0014] FIG4 is a diagram showing an example of a histogram detection method in a histogram detection unit;

[0015] FIG5 is a diagram showing an example of adding restrictions to histogram detection;

[0016] 6 is a diagram showing an example of determining continuous scenes based on the number of pixels in the chrominance area of ​​the CbCr space;

[0017] FIG7 is a diagram showing an example of calculation of a deviation degree;

[0018] FIG8 is a diagram showing an example of color correction in a color correction section;

[0019] FIG. 9 is a flowchart showing an example of the flow of the skin image correction process performed by the skin image correction unit.

[0020] Description of Reference Numerals

[0021] 1 image display device, 12 display unit, 23 face detection unit, 24 histogram detection unit, 25 color deviation calculation unit, 26 color correction parameter generation unit, 27 color correction unit, 28 continuous scene determination unit. DETAILED DESCRIPTION

[0022] Hereinafter, an image display device, a program, and a continuous scene determination method will be described in detail with reference to the accompanying drawings.

[0023] FIG. 1 is a schematic diagram showing an example of an image display device 1 according to an embodiment.

[0024] The image display device 1 includes an image processing unit 10 and a display unit 12 . The image display device 1 transfers data subjected to image processing in the image processing unit 10 to the display unit 12 , and displays the data on the display unit 16 .

[0025] The image processing unit 10 receives video data 40 from a tuner (not shown), an HDMI (High-Definition Multimedia Interface) terminal, etc. The image processing unit 10 outputs video data 42 , which is obtained by subjecting the input video data 40 to image processing, to the display unit 12 .

[0026] The image processing unit 10 includes an image quality adjustment unit 20 and a skin image correction unit 22 .

[0027] The image quality adjustment unit 20 has a function of adjusting gamma, color, brightness, etc. of the input video data 40 . The image quality adjustment unit 20 outputs the video data 41 obtained by adjusting the input video data 40 to the skin image correction unit 22 .

[0028] The skin image correction unit 22 determines the scene to be corrected and performs color correction on the video data 41 input from the image quality adjustment unit 20 in real time in order to display good images even when the environment (conditions) are unknown during shooting or editing.

[0029] Here, FIG. 2 is a functional block diagram showing the functional configuration of the skin image correction unit 22 .

[0030] 2 , the skin image correction unit 22 includes a face detection unit 23 , a histogram detection unit 24 , a color shift detection unit (ie, a color shift degree calculation unit 25 ), a color correction parameter generation unit 26 , a color correction unit 27 , and a continuous scene determination unit 28 .

[0031] Generally speaking, the skin image correction section 22 uses the face detection results of the face detection section 23, the histogram detection results of the histogram detection section 24, and the continuous scene determination of the continuous scene determination section 28 to cause the color deviation calculation section 25 to detect the degree of deviation in the hue or saturation of the skin color of the video data 41 (input video) input from the image quality adjustment section 20. Based on the degree of deviation in hue or saturation detected by the color deviation calculation section 25, the skin image correction section 22 generates a correction parameter corresponding to the degree of deviation in hue or saturation by the color correction parameter generation section 26, and the color correction section 27 performs color correction.

[0032] Furthermore, the face detection unit 23 , the histogram detection unit 24 , the color shift degree calculation unit 25 , the color correction parameter generation unit 26 , and the color correction unit 27 may be implemented by hardware, software, or both.

[0033] The display unit 12 includes a display driver 14 and a display unit 16. The display unit 16 is a display for displaying images. The display unit 16 is, for example, an OLED (Organic Light Emitting Diode) display, a liquid crystal display, or the like.

[0034] The display driving unit 14 converts the voltage and current for driving the display unit 16 based on the video data 42 input from the image processing unit 10 , thereby driving the display unit 16 .

[0035] Next, each unit constituting the skin image correction unit 22 will be described in detail.

[0036] The face detection unit 23 performs face detection based on the video data 41 input from the image quality adjustment unit 20. FIG3 is a diagram showing an example of a face detection method in the face detection unit 23. As shown in FIG3 , as a face detection method, there is a method that uses a DNN (Deep Neural Network) that has been pre-learned to detect various faces to divide the video data 41 into multiple regions and calculate the probability (0 to 1) that each region contains a face. According to this method, since the value of the region containing a face is high and the value of the region not containing a face is low, the face detection unit 23 detects the region with a value greater than a fixed threshold as a face and detects the region with a value less than the threshold as a non-face. When the region with a face is larger than a fixed area, the face detection unit 23 detects it as a face.

[0037] The histogram detection unit 24 performs chromaticity histogram detection based on the video data 41 input from the image quality adjustment unit 20. FIG. 4 is a diagram illustrating an example of a histogram detection method employed by the histogram detection unit 24. FIG. 4(a) illustrates a chromaticity distribution based on CbCr in a YCbCr signal, and FIG. 4(b) illustrates a YCbCr color space. The histogram detection performed by the histogram detection unit 24 calculates the number of pixels with a specific chromaticity distribution, for example, in the entire image associated with the video data 41 or in a portion of the image associated with the video data 41 including a face, and then generates a histogram.

[0038] FIG5 illustrates an example of adding restrictions to histogram detection. As shown in FIG5 , during histogram detection, the histogram detection unit 24 can add restrictions to the saturation detection range in the three-dimensional YCbCr color space ( FIG5(a) ), and can also add restrictions to the brightness detection range in the three-dimensional YCbCr color space ( FIG5(b) ). It should be noted that while restrictions are added to the saturation / brightness range in the three-dimensional YCbCr color space in this embodiment, this is not a limitation. Any correction can be made to the color (hue, saturation) to achieve a natural and aesthetically pleasing appearance.

[0039] The continuous scene determination unit 28 performs continuous scene determination in such a manner that, even when no human face is detected, correction can continue if the visual characteristics of each content and each scene are the same. Specifically, the continuous scene determination unit 28 determines whether a scene is continuous or discontinuous based on the degree of similarity between the previous frame and the current frame, based on the chromaticity histogram detection performed by the histogram detection unit 24.

[0040] The purpose of continuous scene detection is to identify whether the scene has the same tendency of reddish or bluish colors displayed by lighting, color grading, etc. during shooting, and whether switching corrections will cause a sense of disharmony.

[0041] The continuous scene determination unit 28 determines as a continuous scene a situation where the subject moves, such as when the face changes direction, or when the camera slowly moves in and out of the subject (zooming in). Furthermore, the continuous scene determination unit 28 determines as an object of a continuous scene a situation where the cut-in position changes (angle transition).

[0042] As shown in FIG6 , in this embodiment, the continuous scene determination unit 28 uses a method to determine continuous scenes by calculating the similarity between the previous frame and the current frame based on the number of pixels in the chrominance region of the CbCr space. For example, the continuous scene determination unit 28 calculates the similarity based on the difference in the degree of all chrominance between the previous frame and the current frame of the video data, or the accumulation of the difference in the degree of a specific chrominance, as described in detail below.

[0043] FIG6 illustrates an example of continuous scene determination based on the number of pixels in the chromaticity region of the CbCr space. The continuous scene determination unit 28 divides the chromaticity into 32 regions in the CbCr space for the YCbCr input signal, as shown in FIG6(a). The continuous scene determination unit 28 defines the continuous scene as a continuous scene if the following equation (1) is satisfied, and defines the discontinuous scene as a non-continuous scene if the following equation (1) is not satisfied.

[0044]

Formula 1

[0045] (Th is the judgment threshold)……(1)

[0046] Furthermore, the continuous scene determination unit 28 may be configured to determine that a scene is a continuous scene when the following non-square equation (2) is satisfied, and to determine that a scene is a discontinuous scene when the following equation (2) is not satisfied.

[0047]

Formula 2

[0048] Furthermore, the continuous scene determination unit 28 may be configured to determine that a scene is a continuous scene if the following equation (3) is satisfied, which may be a part of the chromaticity, and to determine that a scene is a discontinuous scene if the following equation (3) is not satisfied.

[0049]

Formula 3

[0050] (a, b are constants)……(3)

[0051] The color shift calculation unit 25 detects the degree of chromaticity shift of the skin color in the video data 41 input from the image quality adjustment unit 20, using the face detection result of the face detection unit 23, the histogram detection result of the histogram detection unit 24, and the continuous scene determination of the continuous scene determination unit 28. For example, the color shift calculation unit 25 detects the degree to which the chromaticity of the skin color in the video data 41 input from the image quality adjustment unit 20 shifts from red or yellow.

[0052] For example, if the chromaticity of the skin color shifts from red, the color shift degree calculation unit 25 calculates a "shift from red" indicating the degree of shift from red. Furthermore, if the chromaticity of the skin color shifts from yellow, the color shift degree calculation unit 25 calculates a "shift from yellow" indicating the degree of shift from yellow. On the other hand, if the chromaticity of the skin color does not shift from red or does not shift from yellow, the color shift degree calculation unit 25 sets the "shift from red" and "shift from yellow" values, indicating the degree of shift from red, to zero.

[0053] Even if the detection result of the face detection unit 23 changes from "face detected" to "no face detected", the color shift calculation unit 25 maintains the previous chromaticity shift detection information as long as it is determined to be a continuous scene.

[0054] FIG7 is a diagram showing an example of the degree of deviation calculation performed by the color deviation calculation unit 25. For example, when calculating the degree of deviation of the chromaticity of skin color from red, the center value C of the chromaticity of the skin color is calculated using the following equation (4) based on the data of the chromaticity histogram acquired as shown in FIG7(a).

[0055]

Formula 4

[0056] As shown in FIG. 7( b ), the color shift degree calculation unit 25 determines that the smaller the center value C of the skin color chromaticity, the larger the shift amount, and calculates the degree of shift from red.

[0057] The color correction parameter generating section 26 generates color correction parameters to be applied to the color correction section 27 based on the “amount of shift from red” or the “amount of shift from yellow” detected by the color shift degree calculating section 25 .

[0058] For example, the color correction parameter generation unit 26 is configured to have a standard color management table (CMS[m]) applicable when there is no chromaticity shift, a color management table (CMR[m]) that is most suitable when the red shift degree (RH; 0 to 1) is 1, and a color management table (CMY[m]) that is most suitable when the yellow shift degree (YH; 0 to 1) is 1.

[0059] Based on the red shift, the color correction parameter generation unit 26 generates color correction parameters that are ultimately applied to the color correction unit 27 as a color management table (CM[m]). Here, CM[m] is composed of the elements hue H[h], saturation S[s], and hue V[v]. That is, m = h × s × v.

[0060] The color correction parameter generation unit 26 sets the appropriate standard color management table (CMS[m]) when there is no chromaticity shift. Furthermore, the color correction parameter generation unit 26 sets the optimal color management table (CMR[m]) when the red shift degree (RH, 0-1) is 1. Furthermore, the color correction parameter generation unit 26 sets the optimal color management table (CMY[m]) when the yellow shift degree (YH, 0-1) is 1.

[0061] When the red shift degree and the yellow shift degree in the detection result detected by the color shift degree calculation unit 25 are both 0, the color correction parameter generation unit 26 sets CM[m]=CMS[m] and uses it as a parameter for color correction in the color correction unit 27 .

[0062] In addition, when the red shift degree in the detection result detected by the color shift degree calculation unit 25 is not zero, the color correction parameter generation unit 26 sets CM[m]=RH×CMR[m]+(1-RH)×CMS[m] and uses it as a parameter for color correction applied by the color correction unit 27.

[0063] In addition, when the yellow shift degree is not zero in the detection result detected by the color shift degree calculation unit 25, the color correction parameter generation unit 26 sets CM[m]=YH×CMY[m]+(1-YH)×CMS[m] and uses it as a parameter for color correction in the color correction unit 27.

[0064] Here, when CMS[m] is set to a value that slightly increases red in the skin tone area, CMR[m] is set to a value that slightly decreases (-) the amount of red added, and CMY[m] is set to a value that slightly increases (+) the amount of red added. This can suppress overcorrection when the video data 41 (input video) input from the image quality adjustment unit 20 deviates from red. Furthermore, when the video data 41 (input video) input from the image quality adjustment unit 20 deviates from yellow, insufficient correction can be compensated.

[0065] The color correction section 27 performs color correction on the video data 41 input from the image quality adjustment section 20 based on the color management table (CM[m]) applied by the color correction parameter generation section 26 .

[0066] FIG8 is a diagram illustrating an example of color correction performed by the color correction unit 27. CM[m] is composed of the elements hue H[h], saturation S[s], and hue V[v]. Hue H[h], saturation S[s], and hue V[v] represent the degree of correction of hue, saturation, and brightness at a point where h, s, and v overlap in the three-dimensional YCbCr space (see FIG8(d)).

[0067] Furthermore, the color correction unit 27 interpolates the intermediate value of each point using the correction values ​​of adjacent points.

[0068] Next, an example of the flow of the skin image correction process executed by the skin image correction unit 22 will be described.

[0069] Fig. 9 is a flowchart showing an example of the flow of skin image correction processing performed by the skin image correction unit 22. As shown in Fig. 9, the face detection unit 23 divides the video data 41 input from the image quality adjustment unit 20 into a plurality of regions and detects whether each region contains a face (step S1).

[0070] If the face area is larger than the fixed area (Yes in step S2), the face detection unit 23 is configured to detect a face from the video data 41 and the process proceeds to step S3. On the other hand, if the face area is smaller than the fixed area (No in step S2), the face detection unit 23 is configured not to detect a face from the video data 41 and the process proceeds to step S5.

[0071] In step S3 , the histogram detection unit 24 obtains a chromaticity histogram based on the video data 41 input from the image quality adjustment unit 20 .

[0072] Next, the color shift degree calculation unit 25 uses the face detection result in the face detection unit 23 and the histogram detection result in the histogram detection unit 24 to calculate the "offset from red", where the "offset from red" represents the degree to which the chromaticity of the skin color of the video data 41 input from the image quality adjustment unit 20 is shifted from red (step S4).

[0073] Next, the color correction parameter generating section 26 generates parameters for color correction to be applied to the color correction section 27 based on the “amount of shift from red” detected by the color shift degree calculating section 25 (step S8 ).

[0074] Finally, the color correction section 27 performs color correction on the video data 41 input from the image quality adjustment section 20 based on the color correction parameters applied by the color correction parameter generation section 26 (step S9 ).

[0075] On the other hand, when the area containing the face is smaller than the fixed area (No in step S2), the continuous scene determination unit 28 determines the continuous scenes based on the chromaticity histogram detected by the histogram detection unit 24 (step S5).

[0076] If the continuous scene determination unit 28 determines that the scenes are continuous (Yes in step S5 ), the process proceeds to step S6 . On the other hand, if the continuous scene determination unit 28 determines that the scenes are not continuous (No in step S5 ), the process proceeds to step S7 .

[0077] That is, even if no face is detected (No in step S2), the color shift degree calculation unit 25 holds the calculated value of the previous frame (step S6) as long as it determines that the scene is continuous (Yes in step S5), and proceeds to step S8.

[0078] On the other hand, when no face is detected (No in step S2), if the color shift calculation unit 25 does not determine that it is a continuous scene (No in step S5), the scene is not the scene to be corrected, so the "offset from red" indicating the degree to which the chromaticity of the skin color of the video data 41 input from the image quality adjustment unit 20 is shifted from red is reset to 0 (step S7) and the process proceeds to step S8.

[0079] In this way, according to the embodiment, the image display device detects face detection, chromaticity deviation and continuous scene judgment in real time, and corrects the skin color to match the characteristics of the input video. As a result, even if the content of the environment (conditions) during shooting or editing is not understood, beautiful skin color can be displayed, and good video display can be achieved.

[0080] In particular, the image display device according to the embodiment performs continuous scene determination based on the information obtained from the chromaticity histogram. Even when the scene changes from face detection to no face detection, when it is determined to be a continuous scene, by continuing to perform color correction, appropriate color correction can be performed in scenes where no face is detected, and the most suitable color display without any sense of incongruity can be performed in more scenes.

[0081] Furthermore, in this embodiment, even if the detection result of the face detection unit 23 changes from detecting a human face to detecting no face, and the continuous scene determination unit 28 determines that the scene is continuous, the color shift degree calculation unit 25 maintains the detection result. However, this is not limiting. Specifically, even if the detection result of the face detection unit 23 changes from detecting a human face to detecting no face, and the continuous scene determination unit 28 determines that the scene is continuous, the color correction unit 27 can continue to perform color correction without switching to no correction. The change in the correction amount can be set to a fixed, small, or slowly changing value, for example.

[0082] In conventional scene change detection, judgments are often made using the difference in pixel values ​​between previous and next frames, differences in applications, and the like. It is often possible to determine that a scene is continuous when the subject moves, such as when the direction of a person's face changes, or when the camera slowly moves closer to or away from the subject. However, it is difficult to determine that a scene is continuous when the shot changes. On the other hand, if the lighting, subject, color grading, and other elements are the same during shooting, the chromaticity histogram can obtain similar characteristics. Therefore, the image display device according to the first embodiment can accurately determine that a scene is continuous not only when the subject moves, such as when the direction of a person's face changes, or when the camera slowly moves closer to or away from the subject, but also when switching shots.

[0083] In addition, the program (software) executed in the image display device 1 of the present embodiment is provided after being installed in a ROM or the like in advance.

[0084] The program (software) executed in the image display device 1 of this embodiment can be configured to be provided as a file in an installable or executable form stored in a computer-readable storage medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disc).

[0085] Alternatively, the program (software) executed by the image display device 1 of this embodiment may be stored in a computer connected to a network such as the Internet and provided via network download. Alternatively, the program (software) executed by the image display device 1 of this embodiment may be provided or distributed via a network such as the Internet.

[0086] It should be noted that, although the embodiments of the present application have been described above, the above embodiments are provided as examples and do not limit the scope of the application. The new embodiments can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the scope of the invention. The above embodiments are included in the scope or spirit of the invention and are included in the scope equivalent to the invention described in the claims.

Claims

1. An image display device that performs image processing on input video data and outputs the image data to a display unit, in, The image display device comprises: A face detection unit, which detects faces based on the video data; a histogram detection unit, which performs histogram detection related to color information based on the video data; a continuous scene determination unit, which uses the histogram detection result in the histogram detection unit to determine the continuous scenes; as well as A color correction unit performs color correction on the video data. Even when the detection result of the face detection unit changes from detection of a human face to detection of no human face, the color correction unit continues to perform color correction when the continuous scene determination unit determines that the scenes are continuous.

2. The image display device according to claim 1, in, The image display device comprises: a color shift detection unit for detecting a degree of shift of chromaticity or saturation of skin color of the video data using a face detection result of the face detection unit, a histogram detection result of the histogram detection unit, and a continuous scene determination result of the continuous scene determination unit; as well as a color correction parameter generating unit for generating a color correction parameter based on the degree of deviation of the chroma or saturation of the skin color detected by the color deviation detecting unit; Even when the detection result of the face detection unit changes from detection of a human face to detection of no human face, the color shift detection unit continues to perform color correction when the continuous scene determination unit determines that the scenes are continuous.

3. The image display device according to claim 1, in, The continuous scene determination unit determines whether the video data is a continuous scene or a discontinuous scene based on the degree of similarity between the video data and the chromaticity histogram of the previous frame.

4. The image display device according to claim 3, in, The continuous scene determination unit calculates the similarity based on the accumulation of differences in the degree of full chromaticity or differences in the degree of specific chromaticity between a previous frame and a current frame of the video data.

5. A non-volatile storage medium storing a program, in, The program causes a computer that controls an image display device that performs image processing on input video data and outputs the image data to a display unit to function as the following means: A face detection unit, which detects faces based on the video data; a histogram detection unit, which performs histogram detection related to color information based on the video data; a continuous scene determination unit, which uses the histogram detection result in the histogram detection unit to determine the continuous scenes; as well as A color correction unit, which performs color correction on the video data; Even when the detection result of the face detection unit changes from detection of a human face to detection of no human face, the color correction unit continues to perform color correction when the continuous scene determination unit determines that the scenes are continuous.

6. A method for determining continuous scenes, which is a method for determining continuous scenes in an image display device that performs image processing on input video data and outputs the image data to a display unit, in, Whether it is a continuous scene or a discontinuous scene is determined based on the degree of similarity between the video data and the chromaticity histogram of the previous frame.

Images