Method for obtaining biometric information through camera and electronic device thereof

Abstract

A method of obtaining biometric information through a camera is provided. The method may include sensing ambient brightness information, adjusting light output through the display by controlling the display based on the sensed ambient brightness information, obtaining an input image including a part of a body of a user through the camera, and obtaining biometric information of the user based on the input image.

Description

Technical Field

[0001] The present disclosure relates to a method and an electronic device for controlling a display when obtaining biometric information by using a remote photoplethysmography (rPPG) device capable of obtaining biometric information through a camera. Background Art

[0002] Remote photoplethysmography (rPPG) refers to measuring blood flow velocity, which is a type of biometric information, using light. When using rPPG technology, biometric information such as heart rate or oxygen saturation can be measured or calculated at a body part where blood vessels pass by using light. When the heart pumps blood, the blood flow velocity of corresponding blood vessels distributed throughout the body synchronizes with the pulse and increases and decreases, and the change in blood flow velocity in the microvascular layer of a living tissue can be detected by using rPPG. For example, when using rPPG technology, the widening or narrowing of blood vessels (the widening and narrowing of blood flow velocity changing over time) can be measured by using characteristics such as light absorption rate and transmittance.

[0003] rPPG technology can be implemented relatively simply and has thus been adopted in electronic devices such as mobile phones or home appliances and is widely used to allow users to conveniently obtain information about their health. Various factors may affect the results of rPPG measurements using an electronic device. For example, the measurement results obtained by using rPPG technology may be affected by factors such as the skin color or color of a user, motion artifacts caused by the user's movement, ambient light, or ambient temperature. When affected by various factors, the accuracy or reliability of such measurement results may decrease.

[0004] Therefore, in order to improve the accuracy of these measurement results, a technology for obtaining biometric information by considering factors that may affect the measurement is needed according to the type of electronic device using rPPG technology. Summary of the Invention

[0005] Solution to the Problem

[0006] In an embodiment of the present disclosure, a method of obtaining biometric information through a camera may include: sensing ambient brightness information, adjusting light output through the display by controlling the display based on the sensed ambient brightness information, obtaining an input image including a part of a user's body through the camera, and obtaining the user's biometric information based on the input image.

[0007] In an embodiment of the present disclosure, an electronic device for obtaining biometric information through a camera may include a camera, a display, an illuminance sensor, a memory storing a program including at least one instruction, and at least one processor. The at least one processor may be configured to execute the at least one instruction stored in the memory to sense ambient brightness information through the illuminance sensor, adjust light output through the display by controlling the display based on the sensed ambient brightness information, obtain an input image including a part of a user's body through the camera, and obtain the user's biometric information based on the input image.

[0008] In an embodiment of the present disclosure, a method of controlling a display may include sensing ambient brightness information and adjusting light output through the display by controlling the display based on the sensed ambient brightness information.

[0009] In an embodiment of the present disclosure, an electronic device for adjusting output light according to ambient brightness information may include a display, an illuminance sensor, a memory storing a program including at least one instruction, and at least one processor. The at least one processor may be configured to execute the at least one instruction stored in the memory to sense ambient brightness information through the illuminance sensor and adjust light output through the display by controlling the display based on the sensed ambient brightness information.

[0010] In an embodiment of the present disclosure, a method of controlling a display may include identifying distance information between the display and an object and adjusting light output through the display by controlling the display based on the identified distance information.

[0011] In an embodiment of the present disclosure, an electronic device for adjusting output light according to distance information between a display and an object may include a camera, a display, a memory storing a program including at least one instruction, and at least one processor. The at least one processor may be configured to execute the at least one instruction stored in the memory to identify distance information between the display and the object and adjust light output through the display by controlling the display based on the identified distance information.

[0012] In an embodiment of the present disclosure, a computer-readable recording medium may have recorded thereon a program for implementing a method of obtaining biometric information through a camera, the method including sensing ambient brightness information, adjusting light output through the display by controlling the display based on the sensed ambient brightness information, obtaining an input image including a part of a user's body through the camera, and obtaining the user's biometric information based on the input image.

[0013] In an embodiment of the present disclosure, a computer-readable recording medium may store therein a program for performing at least one of the embodiments of the methods disclosed herein on a computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a diagram showing an electronic device for obtaining biometric information from an image captured by a camera according to an embodiment of the present disclosure.

[0015] Figure 2 is a diagram for describing an operation of obtaining biometric information from an image captured by a camera, which is performed by an electronic device according to an embodiment of the present disclosure.

[0016] Figure 3 is a block diagram showing a configuration of an electronic device for obtaining biometric information from an image captured by a camera according to an embodiment of the present disclosure.

[0017] Figure 4 is a diagram for describing an operation of controlling a display based on ambient light information and obtaining biometric information of a user, which is performed by an electronic device according to an embodiment of the present disclosure.

[0018] Figure 5 is a diagram for describing an operation of controlling a display to adjust the intensity of output light, which is performed by an electronic device according to an embodiment of the present disclosure.

[0019] Figure 6 is a diagram for describing an operation of controlling a display to adjust output light to include auxiliary light, which is performed by an electronic device according to an embodiment of the present disclosure.

[0020] Figure 7 is a diagram for describing an operation of dividing a display screen of an electronic device into a plurality of regions according to an embodiment of the present disclosure.

[0021] Figure 8 is a diagram for describing an operation of controlling a backlight according to a type of a display, which is performed by an electronic device according to an embodiment of the present disclosure.

[0022] Figure 9 is a flowchart of a method for obtaining biometric information from an image captured by a camera according to an embodiment of the present disclosure.

[0023] Figure 10 is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein. In addition, to clearly describe the present disclosure, parts unrelated to the description of the present disclosure are omitted, and like reference numerals are assigned to like elements throughout the specification.

[0025] Although the terms used herein to describe the embodiments of the present disclosure are selected from commonly used terms that are currently widely used in consideration of their functions in the present disclosure, these terms may vary according to the intention of those of ordinary skill in the art, precedent, or the emergence of new technologies. In addition, in specific cases, the terms are arbitrarily selected by the applicant of the present disclosure, and in such cases, the meanings of those terms will be described in detail in the corresponding embodiments. Therefore, the terms used herein are not merely designations of the terms, but the terms are defined based on the meanings of the terms and the content throughout the present disclosure.

[0026] Singular expressions can also include plural meanings as long as they do not conflict with the context. All terms used herein, including technical and scientific terms, can have the same meanings as those commonly understood by those skilled in the art related to this specification.

[0027] Throughout the present disclosure, when a component "includes" an element, it should be understood that the component may additionally include other elements without excluding other elements, as long as there is no specific contrary statement. In addition, as used herein, terms such as "...er", "...unit", "...module", etc. represent a unit that performs at least one function or operation, which can be implemented as hardware or software or a combination thereof.

[0028] Throughout the specification, when a part is referred to as "connected to" another part, it may be "directly connected to" the other part or "electrically connected to" the other part through an intermediate element. In addition, when an element is referred to as "including" a component, the element may additionally include other components without excluding other components, as long as there is no specific contrary statement.

[0029] As used herein, depending on the context, the phrase "configured to" may be used interchangeably with, for example, "suitable for", "capable of", "designed to", "adapted to", "manufactured to", or "able to". The phrase "configured to" may not merely mean "specially designed to" in a hardware manner. Instead, in some cases, the phrase "a system configured to..." may indicate a system "capable of..." in conjunction with another device or component. For example, "a processor configured (or set) to perform A, B, and C" may imply a dedicated processor for performing the corresponding operations (e.g., an embedded processor) or a general-purpose processor (e.g., a central processing unit (CPU) or an application processor) capable of performing the corresponding operations by executing one or more software programs stored in a memory.

[0030] Functions related to artificial intelligence according to the present disclosure are executed by a processor and a memory. The processor may include one or more processors. In this case, the one or more processors may be a general-purpose processor (such as a CPU, an application processor (AP), or a digital signal processor (DSP)), a dedicated graphics processor (such as a graphics processing unit (GPU) or a vision processing unit (VPU)), or a dedicated artificial intelligence processor (such as a neural processing unit (NPU)). The one or more processors perform control to process input data according to predefined operation rules or an artificial intelligence model stored in the memory. In the case where the one or more processors are dedicated artificial intelligence processors, the dedicated artificial intelligence processors may be designed with a hardware structure dedicated to processing a specific artificial intelligence model.

[0031] A predefined operation rule or an artificial intelligence model is generated via a training process. Here, generating via a training process may mean generating a predefined operation rule or an artificial intelligence model that is set to perform a desired characteristic (or purpose) by training a basic artificial intelligence model (or a deep learning model) using a learning algorithm that utilizes a large amount of training data. The training process may be executed by the device itself on which the artificial intelligence according to the present disclosure is performed, or by a separate server and / or system. Examples of the learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning, but are not limited thereto.

[0032] An artificial intelligence model (or deep learning model) may include multiple neural network layers. Each neural network layer has multiple weight values and performs neural network arithmetic operations via arithmetic operations between the arithmetic operation results of the previous layer and the multiple weight values. As a result of training the artificial intelligence model, the multiple weight values in each of the multiple neural network layers can be optimized. For example, the multiple weight values can be modified to reduce or minimize the loss or cost value obtained by the artificial intelligence model during the training process. An artificial neural network may include, for example, a deep neural network (DNN), and may include, for example, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, etc., but is not limited thereto.

[0033] In the present disclosure, "video" or "image sequence" may refer to a moving picture or a moving image. A video or an image sequence may include a series of still images in chronological order.

[0034] In the present disclosure, "video frame" or "image" may refer to a single still image output on a display. That is, in a video that creates a moving image by displaying consecutive scenes at short time intervals, a 'video frame' or an 'image' may refer to a single still image of each scene.

[0035] Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

[0036] Figure 1 FIG. is a diagram showing an electronic device 100 for obtaining biometric information from an image captured by a camera 110 according to an embodiment of the present disclosure.

[0037] In an embodiment of the present disclosure, the electronic device 100 is a device capable of performing remote photoplethysmography and includes a camera 110 and a display 120. The electronic device 100 may analyze biometric signals (or biometric information) of a user U, such as a heart rate or a respiration rate, by analyzing an image obtained through the camera 110. For example, the camera 110 may include a red-green-blue (RGB) camera.

[0038] To measure biometric information of the user U (e.g., heart rate, respiration rate, heart rate variability, or oxygen saturation), the electronic device 100 may use remote photoplethysmography (rPPG) technology. The electronic device 100 may calculate biometric information by measuring the degree of light absorption of hemoglobin in the blood vessels of the user U based on a video obtained through the camera 110.

[0039] The electronic device 100 may be, for example, an electronic device including a display screen 120 and a camera 110, such as a television (TV), a tablet personal computer (PC), or a mobile phone. In addition, the electronic device 100 of the present disclosure is not limited to the above examples and may include various types of devices, and various types of devices include a camera 110 configured to capture an image for obtaining a biometric signal and a display screen 120 to be controlled.

[0040] Figure 2 is a diagram for describing an operation of obtaining biometric information from an image captured by a camera performed by an electronic device according to an embodiment of the present disclosure.

[0041] Reference Figure 2 , in operation 210, the electronic device may obtain an image through the camera. The image may include an image corresponding to a part of a human body, such as a face. Hereinafter, an example in which the face of a user is used as an object captured by the camera of the electronic device will be described. In an embodiment, the object captured by the camera may include various body parts of the body for obtaining a biometric signal, such as a shoulder or an arm.

[0042] In operation 220, the electronic device may identify a region of interest (ROI) from the obtained image. The ROI may be, for example, a region including a face image corresponding to the face of the user. The ROI may have a form of a block, a window, a bounding box, or a segmentation mask.

[0043] In an embodiment, in an operation of obtaining an ROI from the obtained image, a deep learning model trained to receive an image as an input and output an image region including a face region as the ROI may be used.

[0044] In an embodiment, in an operation of obtaining an ROI from the obtained image, a frame scanning operation or an edge analysis method may be used. In the frame scanning operation, the frame may be divided into a plurality of blocks of a certain size, and then the blocks may be scanned from the upper left to the lower right of the image to search for a region including a target image (for example, a face image). Here, the size of the block may be preset. For example, by the processor, when it is determined that there are a plurality of blocks respectively including parts of the face due to the small block size, the scanning operation may be performed again after increasing the block size, and when it is determined that the pixel region of the face included in a specific block is relatively small due to the large block size, the scanning operation may be performed again after reducing the block size (within the corresponding block). For example, the size of the finally selected ROI may vary depending on the distance between the electronic device and the face of the user. For example, when the face of the user is close to the camera, the image portion corresponding to the face of the user may be large, and thus, the block (window) size of the ROI may be large. For example, when the face of the user is far from the camera, the block (window) size of the ROI may be set small.

[0045] In an embodiment, the position of the ROI detected in the moving image may fluctuate over time. When calculating the eigenvalue for biometric information measurement in subsequent operations, difficulties may arise when the ROI fluctuates, and thus, a stabilization operation may be performed on the detected ROI by using an ROI stabilization module. In an embodiment, the ROI stabilization operation may be omitted.

[0046] In an embodiment, the ROI may be determined as a partial region of the image portion corresponding to the user's face. Although rPPG signals can be detected from various parts of the user's body surface, the regions from which eigenvalues with high saliency can be obtained may be limited, depending on the biometric information to be measured. For example, when obtaining biometric information such as heart rate, setting the ROI to a region corresponding to the cheek or forehead in the user's face may enable biometric information to be obtained with higher accuracy. In this case, the ROI may be determined as the cheek region or the forehead region, rather than the entire region of the user's face.

[0047] In operation 230, the electronic device obtains the eigenvalues necessary for obtaining the biometric information within the determined ROI. The eigenvalue may be an eigenvalue for obtaining a biometric signal. For example, since hemoglobin in the blood absorbs green light well, the value or average value of the green channel in the RGB channel data may be used as an eigenvalue. For example, various information for calculating biometric information may be used as an eigenvalue, such as an eigenvalue effective for biometric information calculation obtained through RGB channel operations (such as CHROME), or an eigenvalue effective for biometric signal calculation obtained by using DNN.

[0048] In operation 240, the electronic device may perform a post-processing operation on the obtained eigenvalues. Since the eigenvalues extracted from an image taken from a distance contain noise caused by various external factors (e.g., reflected light or camera noise), a post-processing operation may be performed on the obtained eigenvalues to minimize the influence of the noise on biometric information calculation. The post-processing operation may include, for example, applying a band-pass filter to the eigenvalues to remove signal components outside the target biometric signal range.

[0049] For example, when the target biometric information is the heart rate, since a person's heart rate is between approximately 48 bpm and 240 bpm, the post - processing operation may include applying a band - pass filter to the obtained eigenvalue. In the band - pass filter, the minimum value of the passable frequency component is set to 48 bpm and the maximum value is set to 240 bpm. For example, when the target biometric information is the respiration rate, since a person's respiration rate is between approximately 5 bpm and 30 bpm, the post - processing operation may include applying a band - pass filter to the obtained eigenvalue. In the band - pass filter, the minimum value of the passable frequency component is set to 5 bpm and the maximum value is set to 30 bpm.

[0050] In an embodiment, the post - processing operation (240) for removing noise from the obtained eigenvalue may be omitted. Alternatively, a DNN may be applied to the post - processing operation (240) instead of the band - pass filter.

[0051] In an embodiment, the operations of determining the ROI (220), extracting the eigenvalue (230), and performing the post - processing operation (240) for removing noise from the eigenvalue from the image obtained through the camera may be performed by one or more DNNs. For example, the series of operations of determining the ROI (220), extracting the eigenvalue (230), and performing the post - processing operation (240) for removing noise from the eigenvalue from the image obtained through the camera may be performed by a single DNN, or the DNN may be applied only to the post - processing operation (240) for removing noise from the eigenvalue, only to the operation of extracting the eigenvalue from the ROI (230), or only to the operation of determining the ROI from the image (220).

[0052] In operation 250, the electronic device may obtain biometric information by performing an operation such as a fast Fourier transform (FFT) on the obtained eigenvalue or the post - processed eigenvalue. For example, the frequency of the eigenvalue may be analyzed by using the FFT operation, and the frequency with the strongest energy may be selected as the signal for representing the biometric information. Thereafter, the biometric information may be obtained based on the obtained FFT result value.

[0053] In Figure 2 The operations performed in each of the stages shown may be modularized and may also be implemented in the form of an application executed in the electronic device.

[0054] However, although post - processing operations (240) are used to minimize the noise impact on the obtained biometric information, depending on the environment in which the electronic device performs the operation of obtaining biometric information through the camera, the measured biometric information values may be unstable. For example, the measured biometric information values may be affected by factors such as the distance between the electronic device and the user, ambient light, or the screen output by the electronic device through the display, and may be different from the actual biometric information values.

[0055] In an embodiment, when the electronic device includes a display, the illuminance or color of the screen output through the display may affect the skin color of the user in the ROI to be differently identified. Therefore, the value of the obtained biometric information may be different from the actual biometric information value of the user.

[0056] For example, when the environment in which the electronic device is used includes sufficient ambient light (i.e., the electronic device is used in a sufficiently bright environment), and the screen (light) output by the electronic device through the display is relatively dark, the influence (such as the change in the skin color of the captured body part of the user caused by the light directly output by the electronic device) is small. However, when the environment in which the electronic device is used becomes darker, or the distance between the user and the display of the electronic device decreases, due to the light directly output by the electronic device, the skin color of the captured body part of the user may be identified differently from the actual color, and the biometric information obtained based on such a captured image has an error compared with the actual biometric information of the user.

[0057] In an embodiment of the present disclosure, when an electronic device including a display performs an operation of obtaining biometric information through a camera (photoplethysmography or rPPG measurement), by adjusting the light amount, color, or output position, etc. of the screen output through the display, the influence of the light output from the display on the measured biometric information value can be minimized, and the accuracy of biometric information measurement can be improved.

[0058] Figure 3 is a block diagram showing the configuration of an electronic device for obtaining biometric information from an image captured by a camera according to an embodiment of the present disclosure.

[0059] Refer to Figure 3 , the electronic device 100 may include a camera 110, a display 120, an illuminance sensor 130, a processor 140, and a memory 150. In addition, the electronic device 100 may be implemented by more components than those shown in Figure 3 shown.

[0060] The electronic device 100 may be a device for obtaining biometric information through the camera 110. For example, the electronic device 100 may perform the above - mentioned reference Figure 2The described method. The electronic device 700 can be configured as at least one of, for example, a smart phone, a tablet PC, a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop PC, a netbook computer, a TV, a household appliance, and other mobile or non-mobile computing devices. Additionally, the electronic device 100 is not limited to the above examples, and the electronic device 100 can include various types of devices, including a camera 110 capable of capturing a video for obtaining biometric information of a user and a display 120.

[0061] The camera 110 can include a digital shooting device. The camera 110 can be a device for shooting light of various spectra. For example, the camera 110 can be a device for shooting light of various wavelengths, not limited to visible light. In an embodiment, the camera 110 can obtain a video, an image, or an image sequence including a part of the user's body. The electronic device 100 can obtain the user's biometric information through a change in the skin color of a part of the user's body included in the input video obtained through the camera 110.

[0062] The display 120 can be a device for externally displaying and outputting visual images. For example, the display 120 can output a video or an image sequence externally. In an embodiment, the display 120 can include a panel. The display 120 can be configured as at least one of, for example, a liquid crystal display, a digital mirror device, a liquid crystal on silicon display device, a thin film transistor liquid crystal display, an organic light emitting diode (OLED), a micro LED, a flexible display, a three-dimensional (3D) display, and an electrophoretic display.

[0063] The illuminance sensor 130 can be a device for sensing environmental brightness information of the electronic device 100 or the user of the electronic device 100. The illuminance sensor 130 can include an element having a photoelectric effect, in which when light energy (light) is received, moving electrons are generated inside, changing the conductivity. The illuminance sensor 130 can determine brightness and darkness by using the conductivity that varies depending on the amount of ambient light, and can obtain information about the environmental brightness of the electronic device 100 or the user of the electronic device 100.

[0064] The memory 150 can store programs to be executed by the processor 140, which will be described below, to control the operation of the electronic device 100. The memory 150 can store a program including at least one instruction for controlling the operation of the electronic device 100. The memory 150 can store instructions and program codes readable by the processor 140. In an embodiment, the processor 140 can be implemented to execute the instructions or code segments of the programs stored in the memory 150. The memory 150 can store data input to or output from the electronic device 100.

[0065] The memory 150 may include at least one of, for example, a flash memory type storage medium, a hard disk type storage medium, a multimedia card micro storage medium, a card type memory (e.g., SD or XD memory), a random access memory (RAM), a static RAM (SRAM), a read only memory (ROM), an electrically erasable programmable ROM (EEPROM), a programmable ROM (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0066] The programs stored in the memory 150 may be classified into a plurality of modules according to their functions. For example, the programs stored in the memory 150 may include a display control module and a biometric signal measurement module. The programs stored in the memory 150 will be described in more detail below with reference to Figure 4 The programs stored in the memory 150 will be described in more detail below with reference to

[0067] The processor 140 may control the overall operation of the electronic device 100. The processor 140 may perform operations according to embodiments of the present disclosure. For example, the processor 140 may execute programs stored in the memory 150 to control the overall operations of the camera 110, the display 120, the illuminance sensor 130, the memory 150, etc.

[0068] The processor 140 may include hardware components configured to perform arithmetic operations, logical operations, input / output operations, and signal processing. For example, the processor 140 may include at least one of a CPU, a microprocessor, a graphics processing unit, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), and a field programmable gate array (FPGA), but is not limited thereto.

[0069] The processor 140 may execute at least one instruction stored in the memory 150 to control the illuminance sensor 130 to sense the ambient brightness information of the electronic device 100, and control the display 120 to adjust the output light based on the sensed ambient brightness information. For example, the processor 140 may execute at least one instruction stored in the memory 150 to adjust the brightness and color of the output image, or the waveform and frequency of the light that constitutes the output of the output image. The processor 140 may execute at least one instruction stored in the memory 150 to control the camera 110 to obtain an input such as an image including a part of the user's body, and obtain the user's biometric information based on the input image.

[0070] The operation of the processor 140 to control the camera 110 to obtain an input such as an image including a part of the user's body may correspond to the above Figure 2 operation 210. The operation of the processor 140 to obtain the user's biometric information based on the input image may correspond to the above Figure 2Operations 220 to 250. The operations of controlling the display 120 to adjust the brightness, color, etc. of the output image performed by the processor 140 will be described in more detail below with reference to Figures 4 to 9 More specifically, the operations of controlling the display 120 to adjust the brightness, color, etc. of the output image performed by the processor 140 will be described.

[0071] In an embodiment, the processor 140 may execute at least one instruction stored in the memory 150 to control the display 120 to adjust the intensity of the output light.

[0072] For example, when the sensed ambient brightness value is less than a preset threshold, the processor 140 may control the display 120 to reduce the intensity of the output light. That is, when the surrounding environment of the electronic device 100 is dark, the electronic device 100 may adjust the screen output through the display 120 to be dark in order to prevent biometric information from being distorted when the directly output image is reflected from a part of the user's body (such as the face).

[0073] For example, when the sensed ambient brightness value is less than a preset threshold, the processor 140 may control the display 120 to adjust the output light to include auxiliary light. That is, when the surrounding environment of the electronic device 100 is dark, the electronic device 100 may display an area on the screen of the display 120 that serves as illumination for illuminating a part of the user's body (such as the face) in order to accurately obtain the user's biometric information.

[0074] In an embodiment, the processor 140 may execute at least one instruction stored in the memory 150 to control the display 120 to adjust the waveform of the output light. For example, the processor 140 may control the display 120 to change the waveform of the output light from a pulse wave to a continuous wave.

[0075] In an embodiment, the processor 140 may execute at least one instruction stored in the memory 150 to identify distance information between the display 120 and a part of the user's body included in the input image, and control the display 120 to further adjust the output light based on the identified distance information. For example, the distance information between the display 120 and a part of the user's body may be obtained through a separate distance sensor included in the electronic device 100, or may be calculated by using the time-of-flight (ToF) method based on the time it takes for the light emitted from the display 120 to be reflected from a part of the user's body and then received by the camera 110.

[0076] For example, when the recognized distance value is less than a preset threshold, the processor 140 may control the display 120 to reduce the intensity of the output light. That is, when the screen of the electronic device 100 (display 120) and a part of the user's body (e.g., face) are close to each other, the electronic device 100 may adjust the screen output through the display 120 to be dark to prevent biometric information from being distorted when the directly output image is reflected from a part of the user's body.

[0077] For example, when the recognized distance value is less than a preset threshold, the processor 140 may control the display 120 to adjust the output light to include auxiliary light. That is, when the screen of the electronic device 100 (display 120) and a part of the user's body (e.g., face) are close to each other, the electronic device 100 may display on the screen of the display 120 an area that serves as illumination for illuminating a part of the user's body, so as to accurately obtain the user's biometric information.

[0078] In an embodiment, when the distance value between the display 120 and a part of the user's body included in the input image is less than a preset threshold, the processor 140 may provide, through the display 120, a guidance for causing the user to move to an appropriate position. For example, when it is determined that the user is too close to the display 120 such that the accuracy of the obtained biometric information may be reduced, the electronic device 100 may provide a guidance such as "Please move back", so that the user can move to an appropriate distance from the electronic device 100. The guidance may include, for example, visual guidance or auditory guidance. A guidance for causing the user to move may also be provided, for example, when the output light cannot be adjusted.

[0079] In an embodiment, the processor 140 may execute at least one instruction stored in the memory 150 to divide the screen of the display 120 into multiple regions, and adjust the light output through each of the multiple divided regions based on each region.

[0080] For example, the processor 140 may adjust the output image displayed on a part of the display 120 to be dark, while displaying bright auxiliary light on another part to serve as illumination for illuminating a part of the user's body.

[0081] In this way, according to an embodiment of the present disclosure, when an input such as an image including a part of the user's body is obtained through the camera 110, the brightness, color, etc. of the output image may be adjusted by controlling the display 120 to prevent the biometric signal measurement result from being distorted due to the light output from the electronic device 100. Additionally, by controlling the display 120 to adjust the characteristics (such as brightness or color) of the output image, the biometric signal may be well measured even in a dark environment.

[0082] Figure 4It is a diagram for describing an operation of controlling a display based on ambient light information and obtaining biometric information of a user, which is performed by an electronic device according to an embodiment of the present disclosure.

[0083] Reference Figure 4 , the electronic device may receive a biometric information measurement request 401 from the user or another electronic device. In an embodiment, the biometric information measurement request 401 may be received from the user or another electronic device through a biometric information measurement application or another application that requires biometric information.

[0084] The electronic device that has received the biometric information measurement request 401 may correspond to the above-mentioned Figure 3 electronic device 100. In response to the biometric information measurement request 401, the processor 140 of the electronic device may control the illuminance sensor 130 to sense the ambient light information. The illuminance sensor 130 may transmit the sensed ambient light information to the processor 140.

[0085] The processor 140 may execute at least one instruction stored in the memory to adjust the output light to be output through the display based on the ambient light information sensed by the illuminance sensor 130.

[0086] The programs stored in the memory may be classified into multiple modules according to their functions. Referring to Figure 4 , the programs stored in the memory may include a display control module 410 and a biometric signal measurement module 420.

[0087] The display control module 410 may be further divided into one or more sub-modules according to the type of the display or the type of output light control. For example, in the case where the display included in the electronic device is a self-luminous display such as an OLED display, the display control module 410 may include an output screen control module 411 for adjusting the intensity of the output screen displayed on the display screen. For example, in the case where the display included in the electronic device is a non-self-luminous display such as an LCD, in addition to the output screen control module 411, the display control module 410 may further include a backlight control module 412 for adjusting the intensity of the output screen displayed on the display screen.

[0088] The processor 140 may execute at least one instruction included in the display control module 410 to adjust the output light based on the ambient light information.

[0089] For example, the processor 140 may execute at least one instruction included in the output screen control module 411 of the display control module 410 to adjust the brightness (intensity), color, etc. of the output image. By using the output screen control module 411, when the output light is used as illumination for illuminating a part of the user's body, the processor 140 may adjust the intensity of the output light to decrease, or may adjust the intensity of the output light to increase. By using the output screen control module 411, the processor 140 may adjust the color of the output light to white light, or may adjust the color of the output light to remove components that significantly affect the rPPG signal measurement.

[0090] For example, the processor 140 may execute at least one instruction included in the backlight control module 412 of the display control module 410 to adjust the waveform, intensity, etc. of the output light. In a display including a backlight, the waveform and intensity of the output light are determined based on the waveform and intensity of the backlight. By adjusting the intensity of the backlight using the backlight control module 412, the processor 140 may adjust the intensity of the output light emitted based on the backlight. Additionally, when the backlight has a pulse wave waveform, the biometric information obtained by the camera may be distorted as the backlight is reflected on a part of the user's body. Therefore, the processor 140 may change the waveform of the backlight from a pulse wave to a continuous wave.

[0091] Thereafter, the processor 140 may execute at least one instruction stored in the biometric signal measurement module 420 to obtain an input image including a part of the user's body through the camera and obtain the user's biometric information based on the input image. In an embodiment, the operation of obtaining the user's biometric information based on the input image by the processor 140 through the biometric signal measurement module 420 may correspond to operations 210 to 250 shown in the above Figure 2 as shown in

[0092] Figure 5 is a diagram for describing an operation of controlling the display 120 by an electronic device according to an embodiment of the present disclosure to adjust the intensity of the output light.

[0093] In an embodiment, the electronic device may control the display 120 based on at least one of ambient brightness information or distance information between the display 120 and a part of the user's body included in the input image. The electronic device may control the display 120 to adjust the intensity of the output light.

[0094] Refer to Figure 5, when the sensed ambient brightness value is less than a preset threshold, or when the distance between the display 120 and a part of the user's body included in the input image is less than a preset threshold, the electronic device may control the display 120 to adjust the intensity of the output light to decrease. That is, when the surrounding environment of the electronic device is dark or the screen (display 120) of the electronic device and a part of the user's body (e.g., face) are close to each other, the electronic device may adjust the screen 501 output through the display 120 to be dark so as to prevent biometric information from being distorted when reflected from a part of the user's body (such as the face) in the directly output image.

[0095] In an embodiment, the screen 501 output through the display 120 may include an execution screen of an application that obtains the user's biometric information. In an embodiment, in addition to the execution screen of the application that obtains the user's biometric information, the screen 501 output through the display 120 may further include an execution screen of another application that can apply multitasking.

[0096] Reference Figure 5 , the adjustment of the brightness of the output light may be uniformly performed over the entire area of the display 120, but the present disclosure is not limited to this exemplary embodiment, and as will be described with reference to Figure 7 , the brightness of the output light may be individually adjusted for each of a plurality of divided areas of the screen of the display 120.

[0097] In an embodiment, when the characteristics (such as brightness or color) of the light output by the display 120 have been adjusted, the screen of the display 120 may further include an indicator indicating that the characteristics of the output light have been adjusted. The indicator may prevent the user from confusing the color or brightness of the output screen.

[0098] In an embodiment, when the measurement of the user's biometric information is completed, the characteristics of the output light may be restored to the state before adjustment.

[0099] Figure 6 is a diagram for describing an operation of controlling the display 120 by an electronic device according to an embodiment of the present disclosure to adjust the output light to include auxiliary light.

[0100] In an embodiment, the electronic device may control the display 120 based on at least one of ambient brightness information or distance information between the display 120 and a part of the user's body included in the input image. The electronic device may control the display 120 to adjust the characteristics of the output light, such as intensity or color.

[0101] Reference Figure 6, when the sensed ambient brightness value is less than a preset threshold, or when the distance between the display 120 and a part of the user's body included in the input image is less than a preset threshold, the electronic device may control the display 120 to adjust the output light to include auxiliary light. That is, when the surrounding environment of the electronic device is dark, or when the screen of the electronic device (display 120) and a part of the user's body (e.g., face) are close to each other, the electronic device 100 may use the display area 602 on the screen of the display 120 as illumination to illuminate a part of the user's body (such as the face) so as to accurately obtain the user's biometric information.

[0102] When the output light is used as illumination to illuminate a part of the user's body, the electronic device may adjust the intensity of the light output through the auxiliary light output area 602 to increase. In addition, the electronic device may adjust the color of the light output through the auxiliary light output area 602 to white or a specific color, or may adjust the color of the output light to remove components that significantly affect the rPPG signal measurement.

[0103] In an embodiment, the screen output through the display 120 may include an execution screen 601 of an application for obtaining the user's biometric information. The electronic device may also adjust the light output through the application execution screen 601. For example, when the sensed ambient brightness value is less than a preset threshold, or when the distance between the display 120 and a part of the user's body included in the input image is less than a preset threshold, the electronic device may adjust the intensity of the light output through the application execution screen 601 to decrease, or may adjust the color of the output light to remove components that significantly affect the rPPG signal measurement. That is, when the surrounding environment of the electronic device is dark or the screen of the electronic device (display 120) and a part of the user's body (e.g., face) are close to each other, the electronic device may adjust the execution screen 601 of the application to be dark so as to prevent the biometric information from being distorted when reflected from a part of the user's body (such as the face) in the directly output image.

[0104] In this case, the adjusted output screen output through the display 120 may include, for example, an area where the brightness of the application execution screen 601 is adjusted to be low, and an auxiliary light output area 602 where the brightness is adjusted to be high.

[0105] In an embodiment, in addition to the execution screen 601 of the application for obtaining the user's biometric information and the auxiliary light output area 602, the screen output through the display 120 may further include an execution screen of another application that can apply multitasking. As will be described below with reference to Figure 7 It can be described that the brightness of the light output through the display 120 of the screen can be individually adjusted for each of multiple divided areas of the screen of the display 120.

[0106] In an embodiment, when the characteristics of the light output by the display 120, such as brightness or color, have been adjusted, the screen of the display 120 may further include an indicator indicating that the characteristics of the output light have been adjusted. The indicator can prevent the user from confusing the color or brightness of the output screen and can be displayed on the execution screen 601 of the application or on the auxiliary light output area 602.

[0107] In an embodiment, when the measurement of the user's biometric information is completed, the characteristics of the output light may return to the state before adjustment.

[0108] Figure 7 is a diagram for describing an operation of dividing a display screen of an electronic device into multiple regions according to an embodiment of the present disclosure.

[0109] In an embodiment, an electronic device may divide a display screen into multiple regions 701, 702, and 703, and adjust the light output through each of the multiple regions 701, 702, and 703 based on each region.

[0110] Reference Figure 7 In (a) of, the display screen may include a first region 701a in the central part and a second region 702a surrounding the first region 701a. For example, an execution screen of a biometric information measurement application may be displayed on the first region 701a, and an execution screen of another application or an auxiliary light screen (illumination screen) may be displayed on the second region 702a. In the embodiment of (a), when an execution screen of another application is displayed on the second region 702a, the first region 701a displaying the execution screen of the biometric information measurement application may be implemented in a pop-up form.

[0111] Reference Figure 7 In (b) of, the display screen may include a first region 701b in the upper right part and a second region 702b in the part other than the first region 701b. For example, an execution screen of a biometric information measurement application may be displayed on the first region 701b, and an execution screen of another application or an auxiliary light screen (illumination screen) may be displayed on the second region 702b. In the embodiment of (b), when an execution screen of another application is displayed on the second region 702b, the user may not stop the currently executing task even during the measurement of biometric information. In this case, biometric information can be measured without disturbing the user's task even when biometric information measurement is not performed according to a request from the user.

[0112] Reference Figure 7In (c), the display screen may include a first area 701c on the left side and a second area 702c on the right side. For example, the execution screen of the biometric information measurement application may be displayed on the first area 701c, and the execution screen of another application or the auxiliary light screen (illumination screen) may be displayed on the second area 702c, or the execution screen of the biometric information measurement application may be displayed on the second area 702c, and the execution screen of another application or the auxiliary light screen (illumination screen) may be displayed on the first area 701c. In the embodiment of (c), the user can monitor the biometric information measurement without stopping the currently executing task. Such a template may be effective during multitasking or when the biometric information measurement is executed in response to a request from the user but the user expects not to stop the currently executing task.

[0113] Reference Figure 7 In (d), the display screen may include a first area 701d in the upper right part, a third area 703d in the lower left part, and a second area 702d in a part other than the first area 701d and the third area 703d. For example, the execution screen of the biometric information measurement application or the execution screen of another application may be displayed on the first area 701d or the third area 703d, respectively, and the auxiliary light screen (illumination screen) may be displayed on the second area 702d. The embodiment of (d) may be effective when multitasking is required in a dark environment because even during the biometric information measurement, the user can continue the currently executing task without interruption and the output of the auxiliary light is possible.

[0114] In addition, the electronic device may divide the display screen into four or more areas, and the size and arrangement of the areas are not limited to the above embodiments. The display control method according to an embodiment of the present disclosure may be applied to various types of display screen templates.

[0115] Figure 8 is a diagram for describing an operation of controlling a backlight according to a type of a display performed by an electronic device according to an embodiment of the present disclosure.

[0116] In an embodiment, when the display included in the electronic device is a non-self-emitting display (such as an LCD), the display uses a backlight to output a screen. At least one of a current control method of controlling the backlight by using a current or a pulse width modulation (PWM) method of controlling the backlight by using a voltage may be applied. Among them, the LCD backlight applying the PWM method may output light having a pulse waveform as shown in (a) of Figure 8 shown in (a) of.

[0117] Reference Figure 8In (a), the time interval between pulses of the LCD backlight is so short that it is imperceptible to humans, and the pulse frequency of the LCD backlight can be about 200 Hz or greater. In addition, the light output in a pulse waveform by the display based on the LCD backlight also has a pulse waveform. When the light with a pulse waveform is reflected from a part of the user's body, the input image obtained by photographing the body part may be distorted, and the biometric information obtained based on the distorted input image may have an error from the actual biometric information.

[0118] For example, when an electronic device including an LCD display measures a user's heart rate through a camera and the pulse frequency of the LCD backlight is 1200 Hz, the rPPG signal corresponding to the frequency obtained by dividing the pulse frequency by an integer such as 30 Hz, 48 Hz, 50 Hz, 60 Hz, 80 Hz, 100 Hz, and 120 Hz undergoes constructive interference, and thus, the time point of constructive interference rather than the time point when the user's heart actually beats is more likely to be selected as the data associated with the user's heart rate, and the heart rate information of the user obtained in this way is different from the actual heart rate of the user.

[0119] In an embodiment of the present disclosure, the electronic device may adjust at least one of the waveform or intensity of the backlight. When the display backlight has a waveform of a pulse wave as shown in Figure 8 (a) of, the electronic device may change the waveform of the backlight from a pulse wave to a continuous wave as shown in Figure 8 (b) or (c) of. When the waveform of the backlight is changed to a continuous wave, it is possible to prevent the biometric information obtained through the camera from being distorted as the backlight is reflected on a part of the user's body.

[0120] In an embodiment, when the sensed ambient brightness value is less than a preset threshold, or when the distance between the display operated by the backlight and a part of the user's body included in the input image is less than a preset threshold, the electronic device may change the waveform of the backlight to a continuous wave. That is, when the surrounding environment of the electronic device is dark, or when the screen (display) of the electronic device and a part of the user's body (e.g., face) are close to each other, the electronic device may change the waveform of the backlight of the display to a continuous wave to prevent interference between the backlight and the biometric signal of the user, so as to prevent the biometric information from being distorted when the direct output image is reflected from a part of the user's body (such as the face).

[0121] Figure 9 is a flowchart of a method for obtaining biometric information from an image captured by a camera according to an embodiment of the present disclosure.

[0122] In operation 910, the electronic device may sense ambient brightness information. In an embodiment, the electronic device may include an illuminance sensor and may sense the ambient brightness information of the electronic device through the illuminance sensor.

[0123] In operation 920, the electronic device may control the display based on the sensed ambient brightness information to adjust the light output through the display. In an embodiment, the electronic device may adjust the brightness and color of the output image, or the waveform and frequency of the light output that constitutes the output image.

[0124] In an embodiment, the electronic device may control the display to adjust the intensity of the output light. For example, when the sensed ambient brightness value is less than a preset threshold, the electronic device may control the display to reduce the intensity of the output light. That is, when the surrounding environment of the electronic device is dark, the electronic device may adjust the screen output through the display to be dark to prevent biometric information from being distorted when the directly output image is reflected from a part of the user's body (such as the face). For example, when the sensed ambient brightness value is less than a preset threshold, the electronic device may control the display to adjust the output light to include auxiliary light. That is, when the surrounding environment of the electronic device is dark, the electronic device may display an area on the screen of the display that serves as illumination for illuminating a part of the user's body (such as the face) to accurately obtain the user's biometric information.

[0125] In an embodiment, the electronic device may control the display to adjust the waveform of the output light. For example, the electronic device may control the display to change the waveform of the output light from a pulse wave to a continuous wave.

[0126] In an embodiment, the electronic device may identify distance information between the display and a part of the user's body included in the input image, and control the display based on the identified distance information to further adjust the output light. For example, the distance information between the display and a part of the user's body may be obtained through a separate distance sensor included in the electronic device, or may be calculated by using the ToF method based on the time it takes for light emitted from the display to be reflected from a part of the user's body and then received by the camera.

[0127] For example, when the recognized distance value is less than a preset threshold, the electronic device can control the display to reduce the intensity of the output light. That is, when the screen (display) of the electronic device and a part of the user's body (e.g., the face) are close to each other, the electronic device can adjust the screen output through the display to be dark to prevent biometric information from being distorted when the directly output image is reflected from a part of the user's body. For example, when the recognized distance value is less than a preset threshold, the electronic device can control the display to adjust the output light to include auxiliary light. That is, when the screen (display) of the electronic device and a part of the user's body (e.g., the face) are close to each other, the electronic device can display on the screen of the display an area that serves as illumination for illuminating a part of the user's body, so as to accurately obtain the user's biometric information.

[0128] In an embodiment, when the distance value between the display and a part of the user's body included in the input image is less than a preset threshold, the electronic device can provide guidance to the user through the display to cause the user to move to an appropriate position. For example, when it is determined that the user is too close to the display such that the accuracy of the obtained biometric information may be reduced, the electronic device can provide guidance such as "Please move backward" so that the user can move to an appropriate distance from the electronic device. The guidance can include, for example, visual guidance or auditory guidance. Guidance for causing the user to move can also be provided, for example, when the output light cannot be adjusted.

[0129] In an embodiment, the electronic device can divide the display screen into multiple regions and adjust the light output through each of the multiple regions based on each region. For example, the electronic device can adjust the output image displayed on a part of the display to be dark while displaying bright auxiliary light on another part to serve as illumination for illuminating a part of the user's body.

[0130] In operation 930, the electronic device can obtain an input image including a part of the user's body through the camera. The operation of obtaining an input image including a part of the user's body performed by the electronic device through the camera can correspond to operation 210 Figure 2 described above.

[0131] In operation 940, the electronic device can obtain the user's biometric information based on the input image. The operation of obtaining the user's biometric information based on the input image performed by the electronic device can correspond to operations 220 to 250 Figure 2 described above.

[0132] Thus, according to an embodiment of the present disclosure, when an input such as an image including a part of a user's body is obtained through a camera, the characteristics (such as brightness or color) of the output image can be adjusted by controlling the display to prevent the biometric signal measurement result from being distorted due to the light output from the electronic device. In addition, by controlling the display to adjust the brightness, color, etc. of the output image, biometric signals can be measured well even in a dark environment.

[0133] Figure 10 is a block diagram of an electronic device 100 according to an embodiment of the present disclosure.

[0134] Figure 10 The electronic device 100 may be the embodiment of the electronic device 100 referred to above with reference to Figure 3 and Figure 4 described. For example, Figure 10 the electronic device 100 may be an electronic device such as a smart TV or a mobile phone.

[0135] Referring to Figure 10 , the electronic device 100 may include a camera 110, a display 120, a processor 140, a memory 150, a communication unit 1010, a sensor unit 1020, an input / output unit 1030, an image processing unit 1040, an audio processing unit 1050, an audio output unit 1060, and a power supply unit 1070.

[0136] Figure 10 The camera 110 of Figure 3 may correspond to the camera 110 of Figure 10 , Figure 3 the display 120 of Figure 10 may correspond to the display 120 of Figure 3 or Figure 4 the processor 140 of Figure 10 and Figure 3 or Figure 4 the memory 150 of

[0137] According to an embodiment, the communication unit 1010 may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, a local area network (LAN) module, an Ethernet module, a wired communication module, etc. Here, each communication module may be implemented as at least one hardware chip.

[0138] The Wi-Fi module and the Bluetooth module perform communication by using Wi-Fi and Bluetooth solutions, respectively. When using the Wi-Fi module or the Bluetooth module, various connection information such as a service set identifier (SSID) or a session key may be transmitted and received first, and then various pieces of information may be transmitted and received after establishing a communication connection by using the connection information. The wireless communication module may include at least one communication chip configured to perform communication according to various wireless communication standards such as Zigbee, third generation (3G), 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), LTE-Advanced (LTE-A), fourth generation (4G), fifth generation (5G), and the like.

[0139] According to an embodiment, the communication unit 1010 may receive a user input from an external device.

[0140] The sensor unit 1020 may sense a user signal around the electronic device 100 and may include at least one of a microphone 1021 and an optical receiver 1022.

[0141] The microphone 1021 receives voice emitted by a user. The microphone 1021 may convert the received voice into an electrical signal and output the electrical signal to the processor 140. The microphone 1021 may perform various noise cancellation algorithms for removing noise that occurs when receiving an external audio signal.

[0142] The camera 110 may obtain an image frame such as a still image or a moving image. An image captured by the image sensor may be processed by the processor 140 or a separate image processing unit.

[0143] The image frame processed by the camera 110 may be stored in the memory 150 or transmitted to the outside through the communication unit 1010. Two or more cameras 110 may be provided according to the configuration of the electronic device 100.

[0144] The optical receiver 1022 receives an optical signal (including a control signal) from an external remote control device. The optical receiver 1022 may receive an optical signal corresponding to a user input (e.g., touch, push, touch gesture, voice, or motion) from a remote control device (not shown). A control signal may be extracted from the received optical signal under the control of the processor 140. For example, the optical receiver 1022 may receive a control signal corresponding to a channel up / down button for channel switching from the remote control device.

[0145] Figure 10The sensor unit 1020 is shown as including a microphone 1021 and an optical receiver 1022, but is not limited thereto, and may include at least one of a magnetic sensor, an acceleration sensor, a temperature / humidity sensor, an infrared sensor, a gyro sensor, a position sensor (e.g., Global Positioning System (GPS)), a barometric pressure sensor, a proximity sensor, a Red-Green-Blue (RGB) sensor, an illuminance sensor, a radar sensor, a lidar sensor, and a Wi-Fi signal receiving unit, but is not limited thereto. Those skilled in the art can intuitively infer the functions of the sensors from their names, and thus, detailed descriptions thereof will be omitted.

[0146] Figure 10 The sensor unit 1020 is shown as being provided in the electronic device 100 itself, but is not limited thereto, and may be provided in a control device that is positioned independently of the electronic device 100 and communicates with the electronic device 100, such as a remote controller.

[0147] In the case where the sensor unit 1020 is provided in the control device of the electronic device 100, the control device may digitize the information detected by the sensor unit 1020 and send the digitized information to the electronic device 100. The control device may communicate with the electronic device 100 by using short-range communication such as infrared, Wi-Fi, or Bluetooth.

[0148] Under the control of the processor 140, the input / output unit 1030 receives video (e.g., moving images), audio (e.g., voice or music), and additional information (e.g., Electronic Program Guide (EPG)) from the outside of the electronic device 100. The input / output unit 1030 may include any one of a High-Definition Multimedia Interface (HDMI) port, a Mobile High-Definition Link (MHL) port, a Universal Serial Bus (USB) port, a DisplayPort (DP), a Thunderbolt port, a Video Graphics Array (VGA) port, an RGB port, a D-Subminiature (D-SUB) port, a Digital Visual Interface (DVI) port, a component jack, and a PC port.

[0149] The image processing unit 1040 processes the image data received by the electronic device 100. The image processing unit 1040 may perform various image processing operations on the image data, such as decoding, scaling, noise filtering, frame rate conversion, or resolution conversion.

[0150] The display 120 converts an image signal, a data signal, an on-screen display (OSD) signal, a control signal, etc., that have been processed by the processor 140 to generate a driving signal. The display 120 may be implemented as a Plasma Display Panel (PDP), an LCD, an OLED, a flexible display, or a 3D display. In addition, the display 120 may be configured as a touch screen to be used as both an output device and an input device.

[0151] The display 120 can output each piece of content input through the communication unit or the input / output unit 1030, or output an image stored in the memory 150. Additionally, the display 120 can output information input by the user through the input / output unit 1030 on the screen.

[0152] The display 120 can include a display panel. The display panel can be an LCD panel or a panel including various light-emitting elements such as LEDs, OLEDs, or cold cathode fluorescent lamps (CCFLs). Additionally, the display panel can include not only flat display devices but also curved display devices, which are screens with curvature or flexible display devices capable of adjusting curvature. The display panel can be a 3D display or an electrophoretic display.

[0153] The output resolution of the display panel can include, for example, high definition (HD), full HD, ultra HD, or a resolution higher than ultra HD.

[0154] The audio processing unit 1050 processes audio data. The audio processing unit 1050 can perform various processing operations on the audio data, such as decoding, amplification, or noise filtering. Additionally, the audio processing unit 1050 can include multiple audio processing modules configured to process audio corresponding to multiple pieces of content.

[0155] The audio output unit 1060 outputs the audio included in the broadcast signal under the control of the processor 140. The audio output unit 1060 can output audio (e.g., voice or sound) input through the communication unit 1010 or the input / output unit 1030. Additionally, the audio output unit 1060 can output the audio stored in the memory 150 under the control of the processor 140. The audio output unit 1060 can include at least one of a speaker or a headphone output port.

[0156] The power supply unit 1070 supplies the power input from an external power source to the components inside the electronic device 100 under the control of the processor 140. Additionally, the power supply unit 1070 can supply the power output from one or more batteries in the electronic device 100 to the internal components under the control of the processor 140.

[0157] The memory 150 may store various data, programs, or applications for driving and controlling the electronic device 100 under the control of the processor 140. The memory 150 may include a broadcast reception module, a channel control module, a volume control module, a communication control module, a voice recognition module, a motion recognition module, an optical reception module, a display control module, an audio control module, an external input control module, a power control module, a power control module for an external device connected wirelessly (e.g., Bluetooth), a voice database (DB), or a motion DB, which are not shown. The modules and DBs of the memory 150 that are not shown may be implemented in the form of software to enable the electronic device 100 to perform functions of controlling broadcast reception, channel control, volume control, communication control, voice identification, motion identification, optical reception control, display control, audio control, external input control, power supply control, or power supply control for an external device connected wirelessly (e.g., Bluetooth). The processor 140 may execute each function by using the software stored in the memory 150.

[0158] Figure 10 The block diagram of the illustrated electronic device 100 is a block diagram of an embodiment. Each component illustrated in the block diagram may be integrated, added, or omitted according to the specifications of the actually implemented electronic device 100. That is, two or more components may be integrated into one component as needed, or one component may be divided into two or more components. In addition, the functions performed by each block are for describing the embodiment, and their detailed operations or devices do not limit the scope of the present disclosure.

[0159] Various embodiments may be implemented or supported by one or more computer programs, which may be generated from computer-readable program code and stored in a computer-readable medium. In the present disclosure, the terms "application" and "program" refer to one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data, which are suitable for implementation in computer-readable program code or a part thereof. The term "computer-readable program code" may include various types of computer code, including source code, object code, and executable code. The term "computer-readable medium" may include various types of media accessible by a computer, such as ROM, RAM, a hard disk drive (HDD), a compact disc (CD), a digital versatile disc (DVD), or various types of memories.

[0160] In addition, a machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" refers to a tangible device and may exclude wired, wireless, optical, or other communication links that transmit temporary electrical signals or other signals. Additionally, the term "non-transitory storage medium" does not distinguish between cases where data is stored semi-permanently in the storage medium and cases where data is stored temporarily. For example, a non-transitory storage medium may include a buffer in which data is temporarily stored. A computer-readable medium may be any available medium that can be accessed by a computer and may include volatile or non-volatile media as well as removable or non-removable media. A computer-readable medium includes a medium in which data can be permanently stored and a medium in which data can be stored and later rewritten, such as a rewritable optical disc or an erasable memory device.

[0161] Embodiments disclosed herein may be implemented by a software (S / W) program including instructions stored in a computer-readable storage medium. A computer is a device capable of invoking instructions stored in a storage medium and performing operations according to embodiments disclosed herein, and may include an electronic device according to embodiments disclosed herein.

[0162] According to an embodiment, a method according to various embodiments disclosed herein may be included in a computer program product and then provided. A computer program product may include an S / W program and a computer-readable recording medium storing the S / W program. For example, a computer program product may include a product in the form of an S / W program (e.g., a downloadable application) electronically distributed by a manufacturer of a device or an electronic market. For electronic distribution, at least a part of the S / W program may be stored in a storage medium or temporarily generated. In this case, the storage medium may be a storage medium of a server of the manufacturer or an electronic market, or a relay server that temporarily stores the S / W program.

[0163] A computer program product may include a storage medium of a server or a storage medium of a device in a system composed of a server and a device. Alternatively, when there is a third device (e.g., a smart phone) communicatively connected to the server or the device, the computer program product may include a storage medium of the third device. Alternatively, a computer program product may include the S / W program itself that is transmitted from the server to the device or the third device, or from the third device to the device.

[0164] In this case, any one of the server, the device, and the third device may execute the computer program product and perform the method according to embodiments disclosed herein. Alternatively, two or more of the server, the device, and the third device may execute the computer program product in a distributed manner and perform the method according to embodiments disclosed herein.

[0165] For example, a server (e.g., a cloud server or an artificial intelligence server) may execute a computer program product stored in the server and may control devices communicatively connected to the server to execute the methods according to the embodiments disclosed herein.

[0166] As another example, a third device may execute a computer program product to control devices communicatively connected to the third device to execute the methods according to the embodiments disclosed herein. When the third device executes the computer program product, the third device may download the computer program product from a server and execute the downloaded computer program product. Alternatively, the third device may execute a computer program product provided in a pre-loaded state and execute the methods according to the embodiments disclosed herein.

[0167] Embodiments of the present disclosure may provide an electronic device and a method for improving the accuracy of obtaining biometric information and stabilizing performance by controlling a display when obtaining biometric information through a camera in an rPPG device.

[0168] In an embodiment of the present disclosure, a method for obtaining biometric information through a camera may include: sensing environmental luminance information, adjusting light output through the display by controlling the display based on the sensed environmental luminance information, obtaining an input image including a part of a user's body through the camera, and obtaining the user's biometric information based on the input image.

[0169] Adjusting the output light by controlling the display may include: adjusting the intensity of the output light.

[0170] Adjusting the output light by controlling the display may include: when the sensed environmental luminance information is less than a preset threshold, adjusting the intensity of the output light to decrease by controlling the display.

[0171] Adjusting the output light by controlling the display may include: when the sensed environmental luminance information is less than a preset threshold, adjusting the output light to include auxiliary light by controlling the display.

[0172] Adjusting the output light by controlling the display may include: adjusting the waveform of the output light.

[0173] Adjusting the output light by controlling the display may include: changing the waveform of the output light from a pulse wave to a continuous wave.

[0174] The method may further include: identifying distance information between the display and the part of the user's body included in the input image; and adjusting the output light by controlling the display based on the identified distance information.

[0175] Adjusting the output light by controlling the display may include: based on the recognized distance information being less than a preset threshold, adjusting the intensity of the output light to decrease by controlling the display.

[0176] The method may further include: based on the recognized distance information being less than the preset threshold, providing, through the display, guidance to cause the user to move to an appropriate position.

[0177] Adjusting the output light by controlling the display may include: dividing the screen of the display into a plurality of regions; and adjusting the light output through each of the plurality of regions based on each region.

[0178] In an embodiment of the present disclosure, an electronic device for obtaining biometric information through a camera may include a camera, a display, an illuminance sensor, a memory storing a program including at least one instruction, and at least one processor. The at least one processor may be configured to execute the at least one instruction stored in the memory to sense ambient brightness information through the illuminance sensor, adjust the light output through the display by controlling the display based on the sensed ambient brightness information, obtain an input image including a part of the user's body through the camera, and obtain the user's biometric information based on the input image.

[0179] The at least one processor may further be configured to execute the at least one instruction stored in the memory to adjust the intensity of the output light by controlling the display.

[0180] The at least one processor may further be configured to execute the at least one instruction stored in the memory to adjust the intensity of the output light to decrease by controlling the display based on the sensed ambient brightness information being less than a preset threshold.

[0181] The at least one processor may further be configured to execute the at least one instruction stored in the memory to adjust the output light to include auxiliary light by controlling the display based on the sensed ambient brightness information being less than a preset threshold.

[0182] The at least one processor may further be configured to execute the at least one instruction stored in the memory to adjust the waveform of the output light by controlling the display.

[0183] The at least one processor may further be configured to execute the at least one instruction stored in the memory to change the waveform of the output light from a pulse wave to a continuous wave.

[0184] The at least one processor may also be configured to execute the at least one instruction stored in the memory to identify distance information between the display and the portion of the user's body included in the input image; and adjust the output light by controlling the display based on the identified distance information.

[0185] The at least one processor may also be configured to execute the at least one instruction stored in the memory to adjust the intensity of the output light to decrease by controlling the display based on the identified distance information being less than a preset threshold.

[0186] The at least one processor may also be configured to execute the at least one instruction stored in the memory to divide the screen of the display into a plurality of regions; and adjust the light output through each of the plurality of regions based on each region.

[0187] In an embodiment of the present disclosure, a method of controlling a display may include sensing ambient brightness information, and adjusting the light output through the display by controlling the display based on the sensed ambient brightness information.

[0188] In an embodiment of the present disclosure, an electronic device for adjusting output light according to ambient brightness information may include a display, an illuminance sensor, a memory storing a program including at least one instruction, and at least one processor. The at least one processor may be configured to execute the at least one instruction stored in the memory to sense ambient brightness information through the illuminance sensor, and adjust the light output through the display by controlling the display based on the sensed ambient brightness information.

[0189] In an embodiment of the present disclosure, a method of controlling a display may include identifying distance information between the display and an object, and adjusting the light output through the display by controlling the display based on the identified distance information.

[0190] In an embodiment of the present disclosure, an electronic device for adjusting output light according to distance information between a display and an object may include a camera, a display, a memory storing a program including at least one instruction, and at least one processor. The at least one processor may be configured to execute the at least one instruction stored in the memory to identify distance information between the display and the object, and adjust the light output through the display by controlling the display based on the identified distance information.

[0191] In an embodiment of the present disclosure, a computer-readable recording medium may have a program recorded thereon for implementing a method of obtaining biometric information through a camera. The method includes sensing environmental luminance information, adjusting light output through the display by controlling the display based on the sensed environmental luminance information, obtaining an input image including a part of a user's body through the camera, and obtaining the user's biometric information based on the input image.

[0192] In an embodiment of the present disclosure, a computer-readable recording medium may store therein a program for executing at least one of the embodiments of the methods disclosed herein on a computer.

[0193] Although the present disclosure has been specifically shown and described, those of ordinary skill in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the present disclosure. Therefore, it should be understood that the above embodiments are exemplary in all respects and do not limit the scope of the present disclosure. For example, each element described in a single type may be executed in a distributed manner, and elements described in a distributed manner may also be executed in an integrated form.

[0194] The scope of the present disclosure is not limited by the detailed description of the present disclosure, but is defined by the appended claims, and all modifications or alternatives derived from the scope and spirit of the claims and their equivalents fall within the scope of the present disclosure.

Claims

1. A method of operating an electronic device (100) including a camera (110) and a display (120), the method comprising: Sensing ambient brightness information; Adjusting light output through the display (120) by controlling the display (120) based on the sensed ambient brightness information; Obtaining an input image including a part of a user's body through the camera (110); and Obtaining biometric information of the user based on the input image.

2. The method according to claim 1, wherein Adjusting the output light by controlling the display (120) includes adjusting the intensity of the output light.

3. The method according to claim 2, wherein, Adjusting the output light by controlling the display (120) includes: when the sensed ambient brightness information is less than a preset threshold, adjusting the intensity of the output light to decrease by controlling the display (120).

4. The method according to any one of claims 2 and 3, wherein Adjusting the output light by controlling the display (120) includes: when the sensed ambient brightness information is less than a preset threshold, adjusting the output light to include auxiliary light by controlling the display (120).

5. The method according to any one of claims 1 to 4, wherein Adjusting the output light by controlling the display (120) includes adjusting the waveform of the output light.

6. The method according to claim 5, wherein Adjusting the output light by controlling the display (120) includes changing the waveform of the output light from a pulse wave to a continuous wave.

7. The method according to any one of claims 1 to 6, further comprising: Identifying distance information between the display (120) and the part of the user's body included in the input image; And Adjusting the output light by controlling the display (120) based on the identified distance information.

8. The method according to claim 7, wherein Adjusting the output light by controlling the display (120) includes: when the identified distance information is less than a preset threshold, adjusting the intensity of the output light to decrease by controlling the display (120).

9. The method according to any one of claims 7 and 8, further comprising: Based on the identified distance information being less than the preset threshold, providing guidance to cause the user to move to an appropriate position through the display (120).

10. The method according to any one of claims 1 to 9, wherein Adjusting the output light by controlling the display (120) includes: Dividing the screen of the display (120) into a plurality of regions; and Adjusting the light output through each of the plurality of regions based on each region.

11. An electronic device (100) for obtaining biometric information through a camera (110), the electronic device (100) comprising: The camera (110); The display (120); An illuminance sensor (130); A memory (150) storing a program including at least one instruction; And At least one processor (140), Among them, the at least one processor (140) is configured to execute the at least one instruction stored in the memory (150) to sense ambient brightness information through the illuminance sensor (130), adjust the light output through the display (120) by controlling the display (120) based on the sensed ambient brightness information, obtain an input image including a part of the user's body through the camera (110), and obtain the user's biometric information based on the input image.

12. The electronic device (100) according to claim 11, wherein, The at least one processor (140) is further configured to execute the at least one instruction stored in the memory (150) to adjust the intensity of the output light by controlling the display (120).

13. The electronic device (100) according to claim 12, wherein, The at least one processor (140) is further configured to execute the at least one instruction stored in the memory (150) to adjust the output light to include auxiliary light by controlling the display (120) based on the sensed ambient brightness information being less than a preset threshold.

14. The electronic device (100) according to any one of claims 11 to 13, wherein, The at least one processor (140) is further configured to execute the at least one instruction stored in the memory (150) to adjust the waveform of the output light by controlling the display (120).

15. The electronic device (100) according to any one of claims 11 to 14, wherein, The at least one processor (140) is further configured to execute the at least one instruction stored in the memory (150) to identify distance information between the display (120) and the part of the user's body included in the input image, and adjust the output light by controlling the display (120) based on the identified distance information.

Images