Image enhancement method and device for head mounted display

Abstract

The present application relates to an image enhancement method and device for a head-mounted display. An image enhancement method is disclosed, comprising: setting a reference color card at one or more locations in or near an area of ​​interest in a scene, wherein the sum of the colors of the reference color card includes at least black, white and one color, and the total area of ​​the area with the reference color card is sufficient to establish a color histogram of the area of ​​interest; obtaining a reference image of the scene under a preset lighting condition to generate the reference image color histogram; obtaining an image of the scene under different lighting conditions; processing the scene image to generate the scene image color histogram; matching the color histogram of the scene image based on the color histogram of the reference image to obtain a color conversion function of a corrected scene image; and obtaining an enhanced scene image based on the color conversion function of the corrected scene image.

Description

Technical Field

[0001] The present disclosure generally relates to an image enhancement method for a head mounted display, and more particularly, to a color correction technique in user status monitoring in a head mounted display. Background Art

[0002] With the development of the times, Metaverse and Virtual Reality (VR) have become emerging trends in many fields. Metaverse is a software designed to be like a "virtual space", while Virtual Reality refers to the type of technology used to access such software.

[0003] In Metaverse, users can interact through virtual characters, and this interaction can be in various forms such as games, social activities, and business transactions. Metaverse has a wide range of applications, including games, education, medical care, entertainment, and other fields. Virtual reality VR technology uses devices such as head-mounted displays (HMDs) to put users in a virtual environment and experience an immersive experience.

[0004] Head-mounted displays are increasingly used in VR technology, and with them comes the need to monitor the user's status during use. For example, long-term use of head-mounted displays can cause visual fatigue, which can usually be relieved by reducing the light. At this time, it is necessary to understand the user's real-time status and adjust the light accordingly to reduce the user's visual fatigue. In addition, some users will have a strong sense of dizziness, fatigue, dizziness, nausea, etc. after experiencing stereoscopic VR content, which are symptoms of VR motion sickness. In this case, an alarm system can be set up to issue an alarm to remind the user when the user's status is abnormal. VR technology can also be used for education and skills training, but if students are not concentrating on learning, the effect of teaching and training will be greatly reduced. Therefore, a monitoring system can be set up to notify teachers when students are not paying attention to the teaching content. In addition, head-mounted displays are usually set with a transparent mode, in which others can see the user's face through the helmet to increase the efficiency of communication and communication. Most commonly, it is necessary to perform security identification through user monitoring, and if the user's identity is wrong, it needs to be disabled or an alarm is issued. This can protect the privacy and security of users.

[0005] User status monitoring through physiological sensing technology based on non-contact imaging has become one of the hot topics in current research. For example, infrared cameras and eye trackers can monitor stress levels, eye fatigue, facial expression capture, and user recognition. For example, when providing stimulation to the user, a data set can be collected to obtain the corresponding parameter values ​​in order to obtain pupil diameter, FQIs score, number of blinks, etc. to indicate the state of stress and eye fatigue of the person. In addition, RGB video can monitor heart rate, heart rate variability, blood pressure, respiratory rate, and blood oxygen saturation. For example, physiological signals can be extracted from facial images through technologies such as facial electromyography (EMG) and remote photoencephalography (rPPG). These physiological signals can be used to measure various physiological indicators such as body temperature, heart rate, heart rate variability, blood pressure, respiratory rate, and blood oxygen saturation.

[0006] However, facial imaging in HMDs faces many challenges. First, in HMDs, there is often insufficient lighting for facial imaging of the user due to the absence of ambient light. Second, there are additional infrared backlights or white backlights in the HMD. In addition, the operation of the display makes it an unstable light source itself, resulting in mixed lighting color and brightness conditions. To solve this problem, some technical measures need to be taken to reduce the impact of unstable mixed light sources on facial imaging, thereby improving imaging stability and eliminating system imaging errors.

[0007] Low light enhancement is one of the common image enhancement techniques in digital image processing. In low light environments, the brightness and contrast of the image will be affected, resulting in a decrease in image quality. To solve this problem, many low light enhancement algorithms have been developed to increase the brightness and contrast of the image to improve image quality. Histogram equalization is a common low light enhancement technique: this technique increases the contrast and brightness of the image by redistributing the grayscale of the image. It can be achieved by mapping the grayscale values ​​of the image to a wider range. ML-driven technology, such as Zero-DCE, is a technology based on a deep neural network low light enhancement algorithm that can adaptively adjust the contrast and brightness of the image. In addition, the color balance of the image can also be corrected by automatic white balance technology. For example, light source estimation technology. This technology estimates the color temperature of the light source by analyzing the color information in the image, and adjusts the color balance of the image based on the estimation result to make the color in the image more real and natural. In addition, there is also DNN technology based on machine learning. This technology uses a deep neural network to learn how to adjust image features under different white balance conditions, and adjusts the color balance of the image based on the learning results.

[0008] However, the limitation of the above technology is that it can only correct the color under a single light source. When there are multiple light sources in the HMD at the same time (and changing), the corrected image is uneven and unstable.

[0009] Chinese patent application CN 112348754 A discloses a method and device for enhancing low-illumination color images. First, multiple groups of low-illumination color images are continuously acquired in a fixed field of view. The image sequence is sequentially converted from the RGB color space to the YUV color space, and the obtained YUV color space image is subjected to frame averaging to obtain a frame average image; the Y channel component of the frame average image is used as a reference image, and a non-global objective quality evaluation method is used to sort the quality of the blind source separation and denoised image, and the best quality image is extracted; the best quality image is subjected to contrast-limited adaptive histogram equalization processing, the fused image is subjected to adaptive illumination equalization processing, and the illumination equalization processed image is recombined with the U and V channel components of the frame average image to obtain an enhanced image of the low-illumination color image.

[0010] Chinese patent CN107392879B discloses a low-light monitoring image enhancement method based on reference frames, which utilizes the multi-frame resources provided by the monitoring video, introduces background frame information as a reference, and introduces multi-input frame information for fusion. The illumination component of the image to be enhanced is estimated by further filtering the illumination component calculated by the low-quality and high-quality background images, eliminating the texture information in the illumination component obtained by the ratio operation. The estimation model is more in line with the actual situation. In particular, the high-light quality background image H is first filtered. b As a low light quality background image L b The reflection component of the background image is obtained, and then the ratio operation is converted into a subtraction operation in logarithmic form to calculate the illumination component of the background image, so that the illumination component estimation of the image to be enhanced is more accurate, including obtaining a reference frame and a low-illumination image to be enhanced from a surveillance video, and extracting neighboring frames of the low-illumination image; calculating the illumination components of the background image and the image to be enhanced, extracting the reflection components of the low-illumination image and multiple neighboring frames respectively, and then fusing them to obtain the enhanced result after denoising the low-illumination image to be enhanced.

[0011] The paper "Auto White-Balance Correction for Mixed-Illuminant Scenes" discloses a white balance method to eliminate chromatic aberration caused by scene illumination under mixed lighting conditions. Unlike the traditional camera AWB method that only performs color correction on a single illuminance estimated by the illuminance estimation method, the paper proposes to render the captured scene with a set of predefined white balance settings, retaining the generation of small images (384×384) with a set of predefined color temperatures (such as 2850K, 3800K) related to common lighting conditions (such as incandescent, fluorescent), and also applies a single WB correction to correct for a fixed color temperature (such as 5500K), and further predicts the mixing weights to produce the final AWB result. However, this method still has chromatic aberration under some extreme lighting conditions.

[0012] US Patent US 8059913 B2 discloses a method and apparatus for image correction. A cumulative density function (CDF) is created based on a brightness histogram of an input image; multiple correction parameters that will be applied to multiple different brightness adjustment functions are calculated based on the CDF of the input image, and the brightness of the input image is corrected using the correction parameters and the brightness adjustment function. However, in this patent, the brightness of the reference image is not considered, and the artifact problem of CDF matching is not solved (only the linear slope of the CDF is used for brightness correction), resulting in loss of image details in bright areas. Summary of the invention

[0013] In view of this, the present disclosure aims to perform low-light enhancement and color correction for mixed lighting color and brightness conditions in an HMD to improve the stability and accuracy of user facial imaging.

[0014] Embodiments of the present disclosure provide a method and apparatus for correcting imaging in low-light conditions based on a reference color chart to enhance the image.

[0015] According to a first aspect of the present disclosure, there is provided an image enhancement method, comprising: setting a reference color card at one or more locations in or near a region of interest in a scene, wherein the sum of the colors of the reference color card includes at least black, white and one color, and the total area of ​​the region having the reference color card is sufficient to establish a color histogram of the region of interest; acquiring a reference image of the scene under preset lighting conditions to generate the reference image color histogram; acquiring an image of the scene under different lighting conditions; processing the scene image to generate the scene image color histogram; matching the color histogram of the scene image based on the color histogram of the reference image to obtain a color conversion function of a corrected scene image; and obtaining an enhanced scene image based on the color conversion function of the corrected scene image.

[0016] In one embodiment, the method also includes converting the reference image and the scene image into a color space including a luminance component and a chrominance component; respectively calculating the luminance component histogram and the chrominance component histogram of the reference image and the scene image, matching the luminance component histogram of the scene image based on the luminance component histogram of the reference image to obtain a corrected luminance component histogram of the scene image; matching the chrominance component histogram of the scene image based on the chrominance component histogram of the reference image to obtain a corrected chrominance component histogram of the scene image; and combining the corrected luminance component histogram and the corrected chrominance component histogram of the scene image.

[0017] According to a second aspect of the present disclosure, an image enhancement device is provided, wherein a reference color card is arranged at one or more locations in or near a region of interest in a scene, wherein the sum of the colors of the reference color card includes at least black, white and one color, and the total area of ​​the region having the reference color card is sufficient to establish a color histogram of the region of interest, and the device comprises: a color histogram component for generating a reference image of the scene acquired under preset lighting conditions; a component for generating a color histogram of an image of the scene acquired under different lighting conditions; a component for matching the scene image color histogram based on the reference image color histogram to obtain a color conversion function of a corrected scene image; and a component for obtaining an enhanced scene image based on the color conversion function of the corrected scene image.

[0018] In one embodiment, the device also includes a component for converting the reference image and the scene image into a color space including a luminance component and a chrominance component; a component for respectively calculating the luminance component histogram and the chrominance component histogram of the reference image and the scene image; a component for matching the luminance component histogram of the scene image based on the luminance component histogram of the reference image to obtain a corrected luminance component histogram of the scene image; a component for matching the chrominance component histogram of the scene image based on the chrominance component histogram of the reference image to obtain a corrected chrominance component histogram of the scene image; and a component for combining the corrected luminance component histogram and the corrected chrominance component histogram of the scene image.

[0019] According to a third aspect of the present disclosure, a head-mounted display device is provided, comprising: an image acquisition unit for acquiring an image of an area of ​​interest; an image processing unit, comprising a memory storing computer instructions, and when the computer instructions are executed by one or more processors, the method described in the first aspect of the present disclosure is executed to output an enhanced scene image.

[0020] These and other aspects of the disclosure will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 A conventional user monitoring system 100 for use in an HMD is schematically shown; Figure 2 A schematic diagram 200 schematically shows a conventional histogram matching using a reference image; Figure 3A Schematically shows a reference color card 300 according to an embodiment of the present disclosure; Figure 3B Schematically shows other reference color cards according to embodiments of the present disclosure; Figure 4 A flowchart 400 of an embodiment of image enhancement based on a reference color card according to an embodiment of the present disclosure is schematically shown; Figure 5 Schematically shows a flowchart 500 of performing image enhancement in a color space including a luminance component and a chrominance component according to an embodiment of the present disclosure; Figure 6 Schematically shows a flowchart 600 of image enhancement based on YUV color space according to an embodiment of the present disclosure; Figure 7 Schematically shows the Figure 6 A graph of the conversion functions of the Y channel brightness component and the UV channel chrominance component of the embodiment; Figure 8 A schematic diagram schematically shows an improved embodiment according to an embodiment of the present disclosure; Fig. 9 The block diagram of the image enhancement device 900 according to the embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0022] Figure 1 A conventional user monitoring system 100 in an HMD is schematically shown.

[0023] In order to monitor the user's emotions, physiological characteristics, attention and other states, physiological sensing technology based on non-contact imaging is usually used to obtain the required multiple physiological parameters. For example, a non-contact RGB camera set in the HMD can be used to capture the user's head and facial images 101, especially the images of the user's forehead and eye area to monitor the user's emotions, physiological characteristics and attention. The camera is usually set at the front or top of the HMD, and common RGB camera types include monocular cameras, binocular cameras, etc. As mentioned above, there is no ambient light in the HMD, and there is often a problem of insufficient light for the user's facial imaging. Therefore, the system processor will perform low-light enhancement 102 on the acquired image to improve the brightness and contrast of the image, so as to obtain more accurate user physiological data. According to the needs of user monitoring, the corresponding data set can be obtained based on the low-light enhanced image, and the identified parameter value can be retrieved 103 based on this. Based on the identified parameter value, the required physiological data can be obtained and a suitable response can be made. For example, the user's mental stress index and eye fatigue can be judged by the pupil diameter and blink rate. When it is detected that the user's eye fatigue exceeds a certain threshold, the display backlight can be controlled to reduce the user's eye fatigue 106. When in education and supervision scenarios, it is necessary to ensure that the user remains focused. In order to monitor the user's concentration, for example, the user's facial expressions can be captured based on the acquired facial images, and the user's concentration can be judged by analyzing these expressions. When it is detected that the user's concentration is insufficient, the notification system can be used to let the teacher or supervisor know the user's degree and then take corresponding measures to improve the user's concentration 107. User life feature monitoring is a very important function. If the user has an abnormal health state when using the head-mounted display, such as suddenly fainting or losing consciousness, the alarm system is required to issue an alarm and notify relevant personnel in time 108. In addition, when abnormal health signals are detected, such as abnormal heartbeat, shortness of breath, etc., the alarm system will also issue a warning 109. User identification is a commonly used function. In order to ensure the safety of the head-mounted display, the user's identity can be identified through user facial imaging, and it is ensured that only authorized users can use the head-mounted display 110.

[0024] Those skilled in the art will understand that in order to achieve the various functions mentioned above, it is necessary to obtain stable and accurate facial images, which is critical for accurately extracting physiological signals, facial skin color, blood vessel color, etc. for obtaining user status information. If the facial image cannot accurately reflect the user status, the user status information extracted therefrom is unreliable. Therefore, when acquiring facial images, it is necessary to perform real-time correction and restoration of the facial images acquired in the unstable mixed light source environment in the HMD to obtain stable and accurate user physiological status information.

[0025] Figure 2 A schematic diagram 200 schematically shows a conventional histogram matching using a reference image.

[0026] Histogram matching is a commonly used processing technique for image enhancement. The goal of histogram matching is to match the histogram of the input image with the histogram of the reference image so as to convert it into an output image with an image pixel intensity distribution similar to that of the reference image, thereby achieving image enhancement and color correction. This technology is widely used in digital image processing, computer vision, and medical image processing. A histogram is a statistical chart that represents the distribution of image pixel intensities, where the horizontal axis represents the pixel intensity value and the vertical axis represents the frequency of occurrence of the intensity value in the image. The histogram can show the distribution of image data, such as the center position of the data, the degree of discreteness of the data, etc. However, in order to more directly reflect the distribution of image color or brightness, it is necessary to further analyze it through the cumulative distribution function. CDF can be regarded as a supplement to Histogram, which represents the proportion of pixels whose image pixel values ​​are less than or equal to a certain value. CDF can be used to describe the distribution of color or brightness values ​​in an image, so as to understand the characteristics of the image more intuitively and clearly. The horizontal axis of CDF represents the pixel value, and the vertical axis represents the cumulative proportion under the pixel value. For example, if the cumulative ratio under a certain pixel value is 0.5, it means that the pixel points under this pixel value account for 50% of the total number of pixels. Through CDF, you can more intuitively understand the distribution of brightness values ​​in the image. For example, if the CDF shows a trend close to a straight line, it means that the brightness values ​​in the image are evenly distributed; if the CDF shows a curve trend, it means that the brightness values ​​in the image are unevenly distributed.

[0027] Figure 2 The original image 201 in FIG. 2 shows an image of the forehead and eye area of ​​the user in the HMD obtained by an RGB camera under a relatively bright light source, and a pixel value color histogram and CDF of the red channel component, green channel component and blue channel component obtained by analyzing and processing the RGB image. The conversion functions 207 of the red, green and blue channel components are obtained by matching the pixel value color histogram or CDF of the red channel component, green channel component and blue channel component of the reference image 203, respectively, and thus the output image 205 can be obtained.

[0028] Figure 2The original image 202 in FIG. 2 shows an image of the same user's forehead and eye area in the HMD obtained by an RGB camera under a relatively low-light source, as well as a color histogram and CDF of the pixel values ​​of the red channel component, the green channel component, and the blue channel component obtained by analyzing and processing the RGB image. After matching with the color histogram or CDF of the pixel values ​​of the red channel component, the green channel component, and the blue channel component of the reference image 203, the conversion functions 208 of the red, green, and blue channel components are obtained respectively, and thus the output image 206 can be obtained. By comparing the output image 205 with the output image 206, it can be seen that there are obvious differences in the output images obtained after the histogram matching of the scene image 201 and the scene image 202 imaged under different light source conditions based on the reference image 203. In addition, by comparing the conversion functions 207 and 208, it can be seen that there are obvious differences in the conversion functions of the red channel component, the green channel component, and the blue channel component.

[0029] It can be seen from the above examples that when the reference image is used for histogram matching, the output image obtained is unstable, with uneven hue and brightness, and cannot restore accurate and true facial images, such as facial skin color, blood vessel color, etc. In view of these problems existing in matching using reference images, the present disclosure proposes to obtain a more stable and accurate facial image by setting a fixed reference color card in the scene.

[0030] The reference color card according to the embodiment of the present disclosure can be set at one or more places in the scene. The reference color card can include black, white and color blocks, wherein the color can include, for example, the three primary colors of RGB. The total area of ​​the reference color card needs to be sufficient to establish a color histogram of a specific scene. The three primary colors of RGB can be a single color or a mixed color of RGB. In another embodiment, the reference color card can also include all fully saturated colors and color restoration test charts in the HSL color space. In addition, it can also include key colors in the image to be corrected, such as skin colors of different shades, as well as leaves, blue sky, etc.

[0031] Figure 3AA reference color card 300 according to an embodiment of the present disclosure is schematically shown. In this embodiment, the reference color card 300 includes a fully saturated color (SHSL=1) 301 in the HSL color space and a color reproduction test chart, a black color block 302. Marks can also be set in the reference color card 300 to obtain the position of the reference color card. For example, binary square reference marks 303 located at the four corners of the reference color card 300. (For example, ArUco marks). Setting these marks can provide sufficient correspondence (four corners) to obtain the position of the reference color card. In addition, the internal binary encoding makes them particularly reliable and can be used for error monitoring and correction. The reference color card can be made of materials with good light absorption properties, such as black flocking fabric, anti-reflective paint matte black, etc., to provide a wider brightness range.

[0032] In the embodiment shown in image 310 , the reference color card 300 may be disposed in the middle of the user's forehead and eye area within the HMD.

[0033] Figure 3B Other embodiments 320, 330, 340, 350 of the reference color card according to the present disclosure are shown. In one embodiment, the reference color card 320 is provided with color blocks 325 of different colors at four locations around the region of interest 321, one of the color blocks 325 is black and white B / W, and the other three color blocks 325 are red, green and blue respectively. The reference color card 320 including the color blocks 325 provided at multiple locations around the region of interest can realize the correction based on different regions. The reference color card 330 is provided with continuous reference color cards 325 around the region of interest 321, including black B, white W and red, green and blue RGB. The reference color card 340 is provided with color blocks 325 including black B, white W and red, green and blue RGB above the region of interest 321, and a color block 326 as a reference color card such as a pumpkin pattern is provided below the region of interest 321. The reference color card 350 is provided with a reference color card 326 with a butterfly pattern above the region of interest 321. In this embodiment, the white background surrounding the butterfly pattern is also a color block that is part of the reference color card.

[0034] Figure 4 A flowchart 400 of an embodiment of image enhancement based on a reference color card according to an embodiment of the present disclosure is schematically shown. The image enhancement based on a reference color card according to an embodiment of the present disclosure can be used in color spaces such as RGB, HSL, HSV, HCL, etc. In this embodiment, in order to monitor the user status when the user experiences VR technology, the forehead and eyes of the user in the HMD are imaged in real time to obtain the user's physiological status information. The following are set in the HMD: Figure 3AThe reference color card 300 shown in . First, in step S410, it is necessary to perform image preprocessing on the scene of interest, such as the forehead and eye parts of the user in this embodiment, to obtain the complete image of the reference image and the color histogram and CDF of the reference color card area respectively. In one embodiment. The preprocessing includes obtaining a reference image under preset light conditions in step S411. For example, the reference image of the forehead and eye area of ​​the user can be obtained under the illumination conditions of white light or daylight. Then, in step S412, the complete image of the obtained reference image is processed to derive the color histogram of the complete image pixels, and optionally, in step S413, the cumulative distribution function CDF Twr is obtained based on the color histogram of the complete image pixels. At the same time, the reference color card area image of the reference image can be extracted in parallel or sequentially in step S414, and the reference color card area image can be processed in step S415 to derive the color histogram of the reference color card area image pixels, and optionally, in step S416, the cumulative distribution function CDF Tbr is derived based on the color histogram of the reference color card area image pixels. Finally, in step S416, the derived color histogram and cumulative distribution function CDF Twr and Tbr of the reference image and the reference color card area image are stored respectively.

[0035] After obtaining the color histogram and cumulative distribution function CDF Twr and Tbr of the complete image of the reference image and the reference color card area image through preprocessing, the user status can be monitored in real time. First, in step S401, when the user experiences VR, an image of the same scene as preprocessing 410 is obtained in real time. For example Figure 3AAs shown in , an image 310 of the forehead and eye area of ​​the user in the HMD is obtained. Those skilled in the art will appreciate that in this case, the display running in the HMD becomes an unstable mixed light source that illuminates the user's face. After obtaining a real-time image of the scene, low-light image pre-enhancement is optionally performed in step S402, for example, image enhancement is performed using common techniques such as histogram matching or white balance. Then, in step S403, the complete image of the scene image is processed to derive a color histogram of the complete image pixels, and optionally, in step S404, a cumulative distribution function CDF Tw is obtained based on the color histogram of the complete image pixels. At the same time, a reference color card area image in the scene image can be extracted in parallel or sequentially in step S405, and the reference color card area image can be processed in step S406 to generate a color histogram of the reference color card area image pixels, and optionally, in step S407, a cumulative distribution function CDF Tbr is generated based on the color histogram of the reference color card area image pixels. At this time, the complete image Tw of the scene image can be matched based on the complete image Twr of the reference image in step S404 to obtain the color conversion function Tw' of the corrected complete image, and the reference color card area image Tb of the scene image can be matched based on the reference color card area Tbr of the reference image in step S409 to obtain the color conversion function Tb' of the corrected reference color card area of ​​the scene image. Finally, in step S410, an enhanced scene image is obtained based on the corrected color conversion function Tw' of the complete image and the corrected color conversion function Tb' of the reference color card area. Based on the enhanced scene image, more stable and accurate user physiological state information can be obtained.

[0036] The inventor of the present application has found that when the output image obtained by histogram matching conversion based on the RGB color model often appears too dark or overexposed, so there will still be artifacts (noise) or some details of the image will be lost. In view of this, the inventor of the present application proposes to perform image enhancement based on the reference color card of the embodiment of the present disclosure by converting the RGB color image to a color space including brightness components and chromaticity components. Because the brightness and darkness of the user image obtained in the HMD are different across the entire image, adjusting the brightness separately can make the chromaticity correction more accurate.

[0037] Figure 5 The flowchart 500 of performing image enhancement in a color space including a luminance component and a chrominance component according to an embodiment of the present disclosure is schematically shown. Figure 4The reference image and the scene image in the embodiment are converted to the color space including the brightness component and the chromaticity component respectively. In one embodiment, the color space including the brightness component and the chromaticity component can be a YUV color space, and in other embodiments, the color space including the brightness component and the chromaticity component can be Y'UV, YPbPr, YDbDr, YCbCr, YIQ etc. In step S520, the brightness component histogram and the chromaticity component histogram of the reference image and the scene image are calculated respectively. In step S530, the brightness component histogram of the scene image is matched based on the brightness component histogram of the reference image to obtain the brightness component histogram of the scene image after correction. Then, the chromaticity component histogram of the scene image is matched based on the chromaticity component histogram of the reference image in step S540 to obtain the chromaticity component histogram of the scene image after correction. Finally, the brightness component histogram and the chromaticity component histogram of the scene image after correction are combined in step S550.

[0038] Figure 6 A flowchart 600 of image enhancement based on the YUV color space according to an embodiment of the present disclosure is schematically shown.

[0039] YUV is a color encoding method. YUV images are represented by the brightness Y component, and the U and V components represent the chrominance. The U component represents the difference between the blue component and the brightness, and the V component represents the difference between the red component and the brightness. The YUV color space can be converted to the RGB model and other color spaces.

[0040] like Figure 4 Similar to the embodiment shown in FIG. 1 , in this embodiment, in order to obtain the user's state information when the user experiences the VR technology, the following is set in the HMD: Figure 3AThe reference color card 300 shown in FIG. Similarly, it is necessary to pre-process the scene of interest, such as the image of the forehead and eye part of the user in this embodiment, 610 to obtain the brightness histogram and CDF Yr of the Y channel component of the reference color card area and the complete image of the reference image, and the chromaticity histogram and CDFUr and Vr of the UV channel component. The pre-processing 610 includes obtaining a reference image under preset light conditions through an RGB camera in step S611, for example, an RGB reference image of the forehead and eye area of ​​the user can be obtained under the illumination conditions of white light or daylight. Then, in step S612, the RGB image is converted to the YUV color space, and in step S613, the reference color card area image in the reference image is extracted. Then, the complete image and the reference color card area image of the reference image can be processed separately to derive the histogram and cumulative distribution function CDF: Ybr, Ubr, Vbr of the Y component, U component and V component of the reference color card area image pixels in the reference image, and the histogram and cumulative distribution function CDF: Ywr of the Y component of the complete image. It is worth noting that, in this embodiment, for the UV channel component, only the reference color card area image is subjected to histogram matching to generate the UV channel component conversion function, and the UV channel component of the complete image is not processed. For the Y channel component, histogram matching is performed on both the reference color card area image and the complete image to generate the Y channel component conversion function.

[0041] After obtaining the Y channel component Ybr of the reference color card area image of the reference image, the Ubr and Vbr of the UV channel component, and the Ywr of the Y channel component of the complete image through preprocessing, real-time monitoring of the user status can be started. First, in step S601, when the user experiences VR, a scene image of the same scene as preprocessing 610 under unstable mixed light source illumination is obtained in real time. For example Figure 3A As shown in , an image 310 of the forehead and eye area of ​​the user in the HMD is obtained. Then, optionally, the scene image is pre-enhanced for low-light images using common techniques such as histogram matching or white balance. In step S602, for example, the RGB scene image is converted to the YUV color space, and in step S603, the reference color card area image in the scene image is detected and extracted. Then, the complete image of the scene image and the reference color card area image can be processed separately to derive the histograms and cumulative distribution functions CDF: Yb, Ub, Vb of the Y component, U component and V component of the reference color card area image in the scene image, and the histogram and cumulative distribution function CDF: Yw of the Y component of the scene area.

[0042] The histogram matching of the UV channel is shown in the box 620. In step S621, the U channel component Ub of the scene image reference color card area YUV color space is matched based on the Ubr derived from the U channel component of the reference image reference color card area YUV color space to obtain the conversion function Tu of the U channel. In step S622, the V channel component Vb of the reference color card area YUV color space obtained from the scene image is matched based on the Vbr derived from the V channel component of the reference image reference color card area YUV color space to obtain the conversion function Tv of the V channel component.

[0043] The histogram matching of the Y channel is shown in box 630. In step S631, the Y channel component Yb of the scene image reference color card area YUV color space is matched based on the Ybr derived from the Y channel component of the reference image reference color card area YUV color space to generate a corrected reference color card area Y channel conversion function Tm. In step S632, the Y channel component Yw of the scene image complete image YUV color space is matched based on the Y channel component Ywr of the reference image complete image YUV color space to generate a corrected complete image Y channel conversion function Ti. In addition, for the Y channel, a predefined reference conversion function T0 is specified for the Y channel in step S633. Figure 7 In the embodiment shown, T0 is a straight line. Depending on the image effect to be achieved, in other embodiments, T0 may also be a curve. Next, in step S633, weights W0, Wi, and Wm are respectively configured for the Y channel basic conversion function T0, the corrected reference color card area Y channel conversion function Tm, and the corrected complete image Y channel conversion function Ti based on the environmental conditions, and the conversion function Tm with the configured weights is converted to the Y channel conversion function T0. ×Wm、Ti ×Wi、To ×W0 is combined into the corrected Y channel transfer function T y The sum of W0, Wi, and Wm is 1. Figure 7 In the embodiment shown, W0 is 0.1, Wi is 0.5, and Wm is 0.4. The weights W0, Wi, and Wm can be adjusted as needed. For example, when the ambient light is dark, Wi can be increased, and when the ambient light is bright, Wm can be increased. Then, in step S606, based on the corrected Y channel conversion function T y , the conversion function Tu of the U channel and the conversion function Tv of the V channel to obtain a YUV enhanced image, and in step S607, the YUV enhanced image is converted into an RGB enhanced image. Finally, in step S608, the RGB enhanced image is output for extracting user status information.

[0044] Figure 7 Schematically shows the Figure 6The left figure of the chart 710 shows the conversion function of the Y channel brightness component and the UV channel chrominance component of the embodiment. Figure 6 The conversion curves T0, Ti and Tm corresponding to the reference conversion function T0, the corrected complete image Y channel conversion function Ti and the corrected reference color card area Y channel conversion function Tm in the embodiment. In this embodiment, T0 is a straight line. Depending on the image effect to be achieved, in other embodiments, T0 can also be a curve. The right figure of chart 710 shows the conversion function Tm configured with weights. ×Wm、Ti ×Wi、To ×W0 is the corrected Y channel transfer function T y The corresponding conversion curve T y . Where Wo = 0.1; Wi = 0.5; Wm = 0.4.

[0045] Graph 720 shows Figure 6 In the embodiment, the conversion function Tu of the U channel in the UV channel chromaticity component and the conversion function Tv of the V channel are based on the corrected Y channel conversion curve T y , the conversion curve Tu of the U channel and the conversion curve Tv of the V channel can obtain the enhanced YUV image 730.

[0046] Due to the instability of the light source in the HMD, sometimes the lighting conditions of the entire image of the user's face are significantly different. For example, there is a significant difference in the lighting conditions on the left and right sides of the forehead, such as the left side is darker and the right side is brighter. In order to improve the image enhancement in this case, the inventor proposes that in the present disclosure, for example, Figure 4 and Figure 6 Based on the illustrated embodiment, a step of mixed luminous white balance is added.

[0047] Figure 8The schematic diagram of the improved embodiment according to the embodiment of the present disclosure is schematically shown. In this embodiment, before generating the histogram and the conversion function, different white balance settings are added to find a nearly accurate white balance setting. For example, different color temperatures can be set. In step S801, when the user experiences VR, an image of the same scene as the preprocessing S810 is acquired in real time. After obtaining the real-time image of the scene, low-light image pre-enhancement is optionally performed in step S802. Before the scene image is matched with the histogram based on the reference image, step S820, i.e., mixed illumination white balance, can be added. Step S820 may include step S803, performing white balance under different color temperature settings through machine learning, for example, 7500k, 5500k, and 2850k can be set respectively for white balance, and then generating a weight map in step S804 to indicate that the entire image is close to the accurate white balance setting. For example, the setting with the most uniform skin color on the forehead indicates the closest to the accurate white balance setting. Next, in step S805, a histogram and CDF T of the scene image are generated, T is matched based on the histogram and CDF Tr of the reference image during preprocessing to generate a conversion function T', and an enhanced scene image is outputted thereby.

[0048] Adding mixed lighting white balance can reduce the obvious differences in lighting conditions across the image, resulting in a more uniform and stable scene image.

[0049] Fig. 9 Schematically shows a block diagram of an image enhancement device 900 according to an embodiment of the present disclosure, and various methods for image enhancement described herein may be implemented in the device 900. Fig. 9 As shown, the image enhancement device 900 may include a color histogram component 911 for generating a reference image of a scene acquired under a preset illumination condition, a color histogram component 912 for generating a color histogram of an image of a scene acquired under different illumination conditions, a component 913 for matching a scene image color histogram based on the reference image color histogram to obtain a color conversion function of a corrected scene image, and a component 914 for obtaining an enhanced scene image based on the color conversion function of the corrected scene image. In addition, in some embodiments, the image enhancement device 900 may also optionally include Fig. 9The illustrated components include a component 903 for converting a reference image and a scene image into a color space including a luminance component and a chrominance component, a component 904 for respectively calculating a luminance component histogram and a chrominance component histogram of the reference image and the scene image, a component 905 for matching a luminance component histogram of the scene image based on the luminance component histogram of the reference image to obtain a corrected luminance component histogram of the scene image, a component 906 for matching a chrominance component histogram of the scene image based on the chrominance component histogram of the reference image to obtain a corrected chrominance component histogram of the scene image, and a component 907 for combining the corrected luminance component histogram and the corrected chrominance component histogram of the scene image.

[0050] Although specific functions are discussed above with reference to specific components, it should be noted that the functions of the various components discussed herein can be divided into at least two components, and / or at least some functions of at least two components can be combined into a single component. In addition, the specific component execution of an action discussed herein includes the specific component itself executing the action, or alternatively the specific component calls or otherwise accesses another component or module that performs the action (or performs the action together with the specific component in conjunction with the specific component). Therefore, the specific component that performs an action can include the specific component itself that performs the action and / or another component that the specific component that performs the action calls or otherwise accesses.

[0051] Above about Fig. 9 The various components described may be implemented in hardware or in hardware in combination with software and / or firmware. For example, these modules may be implemented as computer program code / instructions configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules may be implemented as hardware logic / circuits. The features of the techniques described herein are carrier-independent, meaning that these techniques may be implemented on a variety of computing platforms with a variety of processors.

[0052] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices, equipment, and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0053] In a first embodiment, an image enhancement method includes: setting a reference color card at one or more locations in or near a region of interest in a scene, wherein the sum of the colors of the reference color card includes at least black, white and one color, and the total area of ​​the region with the reference color card is sufficient to establish a color histogram of the region of interest; acquiring a reference image of the scene under preset lighting conditions to generate the reference image color histogram; acquiring an image of the scene under different lighting conditions; processing the scene image to generate the scene image color histogram; matching the color histogram of the scene image based on the color histogram of the reference image to obtain a color conversion function of a corrected scene image; and obtaining an enhanced scene image based on the color conversion function of the corrected scene image.

[0054] A second embodiment may include a method according to the first embodiment, further comprising generating the reference image cumulative distribution function CDF and the scene image CDF based on the reference image color histogram and the scene image color histogram, respectively, wherein matching the scene image color histogram based on the reference image color histogram includes matching the scene image CDF based on the reference image CDF.

[0055] A third embodiment may include the method according to the second embodiment, further comprising: identifying a reference color card area in the reference image, wherein the reference image color histogram and CDF include the color histogram and CDF of the complete image of the reference image, and the color histogram and CDF of the reference color card area in the reference image; and including identifying the reference color card area of ​​the scene image, wherein the color histogram and CDF of the scene image include the color histogram and CDF of the complete image of the scene image, and the color histogram and CDF of the reference color card area in the scene image.

[0056] A fourth embodiment may include the method according to the third embodiment, further comprising: matching the scene image complete image CDF based on the reference image complete image CDF to obtain a corrected complete image color conversion function; The CDF of the reference color card area image in the scene image is matched based on the CDF of the reference color card area image in the reference image to obtain a color conversion function of a corrected reference color card area; and an enhanced scene image is obtained based on the corrected color conversion function of the complete image of the scene image and the corrected color conversion function of the reference color card area of ​​the scene image.

[0057] A fifth embodiment may include a method according to the first embodiment, wherein the predetermined lighting condition includes white light or sunlight.

[0058] A sixth embodiment may include the method according to the fourth embodiment, wherein the scene image is acquired in real time under an unstable light source environment.

[0059] A seventh embodiment may include the method according to the first embodiment, further comprising performing color histogram equalization on the scene image to obtain a low-light enhanced scene image.

[0060] An eighth embodiment may include the method according to the seventh embodiment, wherein the reference color card includes RGB single colors or mixed colors of RGB.

[0061] A ninth embodiment may include the method according to the eighth embodiment, wherein the reference color chart includes all fully saturated colors in the HSL color space and a color reproduction test chart.

[0062] A tenth embodiment may include the method according to the first embodiment, wherein the region of interest includes a forehead and eye regions of a user wearing the head mounted display.

[0063] An eleventh embodiment may include the method according to the tenth embodiment, wherein the reference color chart further includes dark and light skin tones.

[0064] A twelfth embodiment may include the method according to the eleventh embodiment, wherein the reference color card further includes a plurality of reference marks to provide sufficient correspondence to obtain the position of the reference color card, and wherein the reference color card is made of a material with good light absorption performance to provide a larger brightness range.

[0065] The thirteenth embodiment may include the method according to any one of the first to twelfth embodiments, further comprising: converting the reference image and the scene image into a color space including a luminance component and a chrominance component; respectively calculating the luminance component histogram and the chrominance component histogram of the reference image and the scene image, matching the luminance component histogram of the scene image based on the luminance component histogram of the reference image to obtain a corrected luminance component histogram of the scene image; matching the chrominance component histogram of the scene image based on the chrominance component histogram of the reference image to obtain a corrected chrominance component histogram of the scene image; and combining the corrected luminance component histogram and the corrected chrominance component histogram of the scene image.

[0066] The fourteenth embodiment may include the method according to the thirteenth embodiment, further comprising generating the luminance component CDF and chrominance component CDF of the reference image and the luminance component CDF and chrominance component CDF of the scene image based on the luminance component histogram and chrominance component histogram of the reference image and the scene image, respectively.

[0067] A fifteenth embodiment may include the method according to the fourteenth embodiment, wherein the color space including the luma component and the chroma components includes a YUV color space, wherein the luma component includes a Y channel component and the chroma components include a UV channel component.

[0068] A sixteenth embodiment may include the method according to the fifteenth embodiment, further comprising: matching the scene image Y channel CDF based on the reference image Y channel CDF to obtain a corrected Y channel conversion function T of the scene image y ; Matching the scene image UV channel CDF based on the reference image UV channel CDF to obtain a corrected UV channel conversion function T of the scene image uv ; and combining the scene image with the corrected Y channel conversion function T y and the corrected UV channel transfer function T uv Get the corrected color conversion function for the YUV color space.

[0069] The seventeenth embodiment may include the method according to the sixteenth embodiment, further comprising: calculating the histogram and CDF of the Y channel of the complete image of the reference image, and the histogram and CDF of the Y channel of the reference color card area in the reference image; calculating the histogram and CDF of the Y channel of the complete image of the scene image, and the histogram and CDF of the Y channel of the reference color card area in the scene image. An eighteenth embodiment may include the method according to the seventeenth embodiment, wherein matching the CDF of the Y channel of the scene image based on the CDF of the Y channel of the reference image comprises: matching the CDF of the Y channel of the scene image complete image based on the CDF of the Y channel of the reference image complete image to generate a Y channel conversion function T of the corrected complete image. i ; Matching the CDF of the Y channel of the reference color card area in the scene image based on the CDF of the Y channel of the reference color card area in the reference image to generate a corrected Y channel conversion function T of the reference color card area m ; Combine the corrected complete image Y channel conversion function T i and the Y channel conversion function T of the calibrated reference color card area m To obtain the scene image Y channel conversion function T y .

[0070] A nineteenth embodiment may include the method according to the eighteenth embodiment, wherein the corrected Y channel conversion function T of the scene image is obtained. y The invention also includes: specifying a predefined Y channel reference conversion function T0; and combining the Y channel reference conversion function T0 with the corrected complete image Y channel conversion function T0. iand the Y channel conversion function T of the calibrated reference color card area m To obtain the Y channel conversion function T y .

[0071] A twentieth embodiment may include the method according to the nineteenth embodiment, further comprising: providing a Y channel reference transfer function T0 based on the environmental condition, a corrected complete image Y channel transfer function T i and the Y channel conversion function T of the calibrated reference color card area m Configure weights W0, W respectively i , W m , and based on the configured weights W0, W i , W m To combine the Y channel reference transfer function T0, the corrected complete image Y channel transfer function T i and the Y channel conversion function T of the calibrated reference color card area m To obtain the Y channel conversion function T y .

[0072] A twenty-first embodiment may include the method according to the seventeenth embodiment, wherein calculating the histogram and CDF of the UV channels of the reference image and the scene image includes: calculating the histogram and CDF of the UV channel of only the reference color card area in the scene image.

[0073] A twenty-second embodiment may include the method according to any one of the seventeenth to twenty-first embodiments, wherein the CDF of the UV channel of the scene image is matched based on the CDF of the UV channel of the reference image to obtain a corrected UV channel conversion function T of the scene image. uv The method comprises: matching the CDF of the UV channel of the reference color card area of ​​the scene image based on the CDF of the UV channel of the reference color card area of ​​the reference image to obtain a corrected UV channel color conversion function T of the scene image. uv .

[0074] A twenty-third embodiment may include the method according to the twenty-first embodiment, wherein the UV channel includes a U channel and a V channel, and a corrected UV channel conversion function T of the scene image is obtained. uv Including the conversion function T for obtaining U channel and V channel respectively u and T v .

[0075] In a twenty-fourth embodiment, an image enhancement device, wherein a reference color card is provided at one or more locations in or near a region of interest in a scene, wherein the sum of the colors of the reference color card includes at least black, white, and one color, and the total area of ​​the regions having the reference color card is sufficient to establish a color histogram of the region of interest, the device comprising: a color histogram component for generating a reference image of the scene acquired under preset lighting conditions; A component for generating a color histogram of an image of the scene acquired under different lighting conditions; a component for matching the scene image color histogram based on the reference image color histogram to obtain a color conversion function for a corrected scene image; and a component for obtaining an enhanced scene image based on the color conversion function of the corrected scene image.

[0076] The twenty-fifth embodiment may include an apparatus according to the twenty-fourth embodiment, further comprising a component for converting the reference image and the scene image into a color space including a luminance component and a chrominance component; a component for respectively calculating a luminance component histogram and a chrominance component histogram of the reference image and the scene image; a component for matching the luminance component histogram of the scene image based on the luminance component histogram of the reference image to obtain a corrected luminance component histogram of the scene image; a component for matching the chrominance component histogram of the scene image based on the chrominance component histogram of the reference image to obtain a corrected chrominance component histogram of the scene image; and a component for combining the corrected luminance component histogram and the corrected chrominance component histogram of the scene image.

[0077] A twenty-sixth embodiment may include an apparatus according to the twenty-fifth embodiment, wherein the color space including the luminance component and the chrominance component includes a YUV color space, wherein the luminance component includes a Y channel component and the chrominance component includes a UV channel component, and the apparatus further includes a component for respectively calculating a histogram and a CDF of the Y channel of the reference image and the scene image and a histogram and a CDF of the UV channel; and a component for matching the CDF of the Y channel of the scene image based on the CDF of the Y channel of the reference image to obtain a corrected Y channel conversion function T of the scene image. y A component for matching the CDF of the scene image UV channel based on the CDF of the reference image UV channel to obtain a corrected UV channel conversion function T of the scene image uv and a Y channel conversion function T for combining the scene image corrected y and the corrected UV channel transfer function T uv Components that get the corrected color conversion function for the YUV color space.

[0078] In a twenty-seventh embodiment, a head-mounted display device includes: an image acquisition unit for acquiring an image of a region of interest; an image processing unit, comprising a memory storing computer instructions, which, when executed by one or more processors, executes a method as in any one of the first to twenty-third embodiments to output an enhanced scene image.

[0079] . A twenty-eighth embodiment may include the device according to the twenty-seventh embodiment, further comprising a physiological parameter detection unit to identify physiological parameters of the user based on the enhanced scene image.

[0080] By studying the drawings, the disclosure and the appended claims, those skilled in the art will be able to understand and implement variations to the disclosed embodiments in practicing the claimed subject matter. In the claims, the words "A and / or B" mean A, B, or A and B, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality; the words "first", "second", "third", "fourth" are used only to distinguish elements or steps and do not indicate the order of the elements or steps. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. An image enhancement method, comprising: Setting a reference color card at one or more locations in or near the region of interest in the scene, wherein the sum of the colors of the reference color card includes at least black, white and one color, and the total area of ​​the reference color card area is sufficient to establish a color histogram of the region of interest; Acquiring a reference image of the scene under preset lighting conditions to generate a color histogram of the reference image; Acquire images of the scene under different lighting conditions; Processing the scene image to generate a color histogram of the scene image; matching a color histogram of the scene image based on the color histogram of the reference image to obtain a color conversion function of a corrected scene image; and An enhanced scene image is obtained based on the color conversion function of the corrected scene image.

2. The method according to claim 1 further comprises generating the reference image cumulative distribution function CDF and the scene image CDF based on the reference image color histogram and the scene image color histogram, respectively, wherein: Matching the scene image color histogram based on the reference image color histogram includes matching the scene image CDF based on the reference image CDF.

3. The method according to claim 2 further includes identifying a reference color card area in the reference image, wherein the reference image color histogram and CDF include a color histogram and CDF of the complete image of the reference image, and a color histogram and CDF of the reference color card area in the reference image; and includes identifying a reference color card area of ​​the scene image, wherein the color histogram and CDF of the scene image include a color histogram and CDF of the complete image of the scene image, and a color histogram and CDF of the reference color card area in the scene image.

4. The method according to claim 3, further comprising: Matching the scene image complete image CDF based on the reference image complete image CDF to obtain a corrected complete image color conversion function; Matching the CDF of the reference color card area image in the scene image based on the CDF of the reference color card area image in the reference image to obtain a color conversion function of a corrected reference color card area; and An enhanced scene image is obtained based on the corrected color conversion function of the complete image of the scene image and the corrected color conversion function of the reference color card area of ​​the scene image. The method according to claim 1 , wherein the preset lighting condition comprises white light or sunlight. The method according to claim 4 , wherein the scene image is acquired in real time under an unstable light source environment. 7 . The method according to claim 1 , further comprising performing color histogram equalization on the scene image to obtain a low-light enhanced scene image.

8. The method according to claim 7, wherein: The reference color card includes RGB single color or RGB mixed color.

9. The method according to claim 8, wherein: The reference color chart includes all fully saturated colors in the HSL color space and a color reproduction test chart.

10. The method according to claim 1, wherein: The region of interest includes a forehead and eye areas of a user wearing the head mounted display.

Images