Display device and method for measuring biometric information therefor

Abstract

A display device is disclosed. A processor controls a display to display content according to a user input, acquires data on a use state on the basis of photographing data of a camera and a sensing value of at least one sensor, newly acquires photographing data of a user through the camera when a state suitable for measuring biometric information is identified on the basis of the data on the use state and an artificial intelligence model, and acquires biometric information on the basis of the acquired photographing data of the user.

Description

Display device and method for measuring biometric information of the device

[0001] The present disclosure relates to a display device and a method for measuring biometric information of the device.

[0002] As the functionality of electronic devices becomes more sophisticated, development of biometric measurement technologies using electronic devices is becoming more active. Recently, with the growing demand for remote medical care due to the risk of infection in physical hospitals, the development of non-invasive biometric measurement technologies that do not require physical contact is gaining momentum.

[0003] Accordingly, it has become possible to measure biometric information using the remote Photoplethysmography (rPPG) method without physical contact with electronic devices.

[0004] However, conventional biometric information measurement technology has limitations in that it can only measure biometric information when the user requests it, and periodic measurement is difficult, making long-term monitoring and health status analysis difficult.

[0005] A display device according to at least one embodiment of the present disclosure includes a display, a camera, at least one sensor, a memory storing an artificial intelligence model learned based on learning data related to content viewing characteristics, and a processor.

[0006] The processor controls the display to display content according to user input, obtains data on a usage status based on the captured data of the camera and the sensing value of the at least one sensor, and, based on the identified usage status and the artificial intelligence model, if a state suitable for measuring biometric information is identified, acquires new user shooting data through the camera, and acquires biometric information based on the acquired user shooting data.

[0007] A method for measuring biometric information of a display device according to at least one embodiment of the present disclosure includes the steps of displaying content according to a user input, obtaining data on a user's usage status based on a camera and at least one sensing value, identifying a state suitable for biometric information measurement based on an artificial intelligence model learned based on learning data related to content viewing characteristics and the data on the usage status, and newly acquiring user's photographing data through the camera and acquiring biometric information based on the acquired photographing data.

[0008] According to at least one embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions that, when executed by a processor of a display device, cause the display device to perform an operation, the operation includes: displaying content according to a user input; acquiring data on a user's usage status based on a camera and at least one sensing value; identifying a state suitable for biometric information measurement based on an artificial intelligence model learned based on learning data related to content viewing characteristics and the identified usage status; and newly acquiring user's photographing data through the camera and acquiring biometric information based on the acquired photographing data.

[0009] FIG. 1 is a perspective view schematically illustrating a display device according to at least one embodiment of the present disclosure.

[0010] FIG. 2 is a block diagram illustrating a configuration of a display device according to at least one embodiment of the present disclosure.

[0011] FIG. 3 is a block diagram showing an example of a detailed configuration of a display device according to at least one embodiment of the present disclosure.

[0012] FIG. 4 is a drawing for explaining a biometric information measurement process of a display device according to at least one embodiment of the present disclosure.

[0013] FIG. 5 is a drawing for explaining a process of measuring biometric information through a camera of a display device according to at least one embodiment of the present disclosure.

[0014] FIGS. 6 and 7 are drawings for explaining an rPPG method according to at least one embodiment of the present disclosure.

[0015] FIGS. 8 and 9 are drawings for explaining a biometric information providing process of a display device according to at least one embodiment of the present disclosure.

[0016] FIG. 10 is a diagram for explaining an artificial intelligence model learning process of a display device according to at least one embodiment of the present disclosure.

[0017] FIG. 11 is a flowchart of a method for measuring biometric information of a display device according to at least one embodiment of the present disclosure.

[0018] FIG. 12 is a diagram for explaining the overall flow of biometric information measurement of a display device according to at least one embodiment of the present disclosure.

[0019] FIG. 13 is a diagram illustrating a software structure for implementing embodiments of a display device according to at least one embodiment of the present disclosure.

[0020] The terms used in the various embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should be defined based on the meaning of the terms and the overall content of this disclosure, rather than simply their names.

[0021] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0022] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0023] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0024] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0025] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this disclosure, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0026] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor (not shown), excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0027] In this disclosure, the term user may refer to a person using an electronic device or a device used by the person.

[0028] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0029] FIG. 1 is a perspective view schematically illustrating a display device according to at least one embodiment of the present disclosure.

[0030] A display device (100) refers to an electronic device that directly includes a display or is connected to an external display (e.g., a monitor). Specifically, the display device (100) may be implemented as a variety of devices, such as a monitor, TV, laptop PC, PC, kiosk, mobile phone, tablet PC, refrigerator, or air conditioner. When implemented in a form connected to an external display, the display device (100) may also be referred to as an electronic device or a terminal device, but in the present disclosure, it is collectively referred to as the display device (100).

[0031] Referring to FIG. 1, the display device (100) can measure biometric information of a user (10). Specifically, the display device (100) can capture a photo of the user (10) using a camera (120). The display device (100) can measure the biometric information of the user based on the captured image captured by the camera (120).

[0032] Biometric information refers to various pieces of information that describe a person's biological characteristics. Specifically, biometric information can include heart rate, stress level, oxygen saturation, respiration rate, heart rate variability, and body temperature.

[0033] The display device (100) can analyze the captured image to detect changes in the skin color of the user (10) and identify biometric information based on the detection results. At this time, the method for identifying biometric information may be rPPG (remote Photoplethysmography). rPPG may be a method for estimating various biometric information, such as heart rate signals, by analyzing a captured image of a person's face or body. Specific details regarding the rPPG (remote Photoplethysmography) method will be described later.

[0034] Once biometric information is identified, the display device (100) can provide the biometric information to the user. This provision of biometric information can be accomplished in various ways. For example, if connected to the user's mobile phone or an external server, the display device (100) may transmit the identified biometric information to the mobile phone or external server.

[0035] Alternatively, in the case of a display device (100) directly equipped with a display as in FIG. 1, biometric information can be displayed on the display. FIG. 1 illustrates a state in which a UI (30) including biometric information of a user (10) is displayed at the bottom of the display. When a user selects an item on the UI, the display device (100) can display biometric information measured for the selected item. Accordingly, the user can easily check biometric information of various items.

[0036] When measuring biometric information based on a captured image as described above, or when there are changes in the shooting environment or the user's movements, there is a high possibility that biometric information measurement may be impossible or may be measured inaccurately.

[0037] For example, if the lighting of the environment in which the user (10) and the display device (100) are placed is too bright or too dark, it may be difficult to detect changes in the user's skin color within the captured image, resulting in inaccurate measurement results. Alternatively, the measurement results may also be inaccurate if the user speaks or moves.

[0038] A display device (100) according to at least one embodiment of the present disclosure can identify a user's usage status using a captured image of a camera and a sensing value of at least one sensor, and based on the usage status, can first determine whether the user is in a state suitable for biometric information measurement. The user's usage status may include the type of content the user is viewing, the user's viewing posture, whether or not the user speaks, whether or not the user moves, lighting conditions, viewing time, viewing time, etc. In other words, the usage status may include various states related to viewing content.

[0039] The display device (100) can also extract usage patterns by accumulating and managing usage states. A usage pattern refers to information organized by patterning various pieces of information related to the user's normal usage of the display device (100).

[0040] For example, if the display device (100) is implemented as a TV that displays content, usage patterns can be organized according to various criteria such as the type of content, the viewing time of the content, the user's personal characteristics, the user's age, and gender. Specifically, while watching or listening to music content during the day, the user may sing or dance along to the music while maintaining the usual lighting. On the other hand, while watching or listening to music content at night, the user may quietly appreciate it. While watching content such as dramas or movies, the user may dim the lighting regardless of the time and stare at the screen without saying much. While watching content such as news, culture, or education, the user may stare at the screen without saying much while maintaining the usual lighting.

[0041] The display device (100) can identify a state suitable for measuring biometric information when the user is looking straight at the camera without any particular movement or speech, and the ambient illumination or distance from the user is within an optimal range.

[0042] The display device (100) can determine changes in the user's movements, whether the user speaks, etc. based on the captured images while the user selects and watches specific content. In addition, the display device (100) can sense the brightness of the surrounding environment using the light sensor (131). Based on the sensing value of the light sensor (131), the display device (100) can identify changes in the light level while the user selects and watches specific content. In other words, the display device (100) can detect a pattern of turning off the lights when the user watches movie content.

[0043] The display device (100) can identify and store a user's usage pattern based on values ​​sensed by a camera and at least one sensor over a certain period of time.

[0044] The display device (100) determines whether the current user's state is suitable for measuring biometric information based on the identified usage state, and if so, controls the camera to capture a photo of the user and measure the user's biometric information based on the captured image.

[0045] As a result, the display device (100) can determine a timing suitable for measuring biometric information on its own without the user (10) directly running the display device (100) or inputting a user operation.

[0046] For example, even if the user (10) does not directly input a command to check his / her heart rate, the display device (100) can measure the user's (10) heart rate by itself and display it on the display.

[0047] Alternatively, even if the user (10) does not run an application that measures biometric information to check his or her stress index, the display device (100) can determine a timing suitable for measuring biometric information, measure the user's stress index, and display it on the display.

[0048] Accordingly, the user's condition can be measured during their daily life, allowing for the early detection of sudden health risks. Furthermore, biometric data can be effectively measured for individuals unfamiliar with using technology, such as the elderly or children.

[0049] In the above, the case of automatically measuring biometric information has been described, but it is not necessarily limited to this, and the display device (100) can measure biometric information of the user (10) even when a user operation is input.

[0050] In the above, an embodiment has been described in which the display device (100) identifies the user's usage status, determines whether the current user's status is suitable for measuring biometric information, and performs an action based on the determination result. However, an artificial intelligence model may also be used to determine whether the user's status is suitable for measuring biometric information.

[0051] For example, an artificial intelligence model trained based on learning data related to content viewing characteristics can be used. The display device (100) can determine whether the user is in a state suitable for measuring biometric information based on the user's usage status and the artificial intelligence model, and then perform an action based on the determination result.

[0052] Below, the specific details of how the display device (100) measures the user's (10) biometric information by using an artificial intelligence model to determine a timing suitable for measuring biometric information will be described later.

[0053] FIG. 2 is a block diagram illustrating a configuration of a display device according to at least one embodiment of the present disclosure.

[0054] According to FIG. 2, the display device (100) may include a display (110), a camera (120), at least one sensor (130), a memory (140), and a processor (150). However, the present invention is not limited thereto, and the display device (100) may be implemented in a form in which some components are excluded, or may be implemented in a form in which other components are further included.

[0055] The display (110) is a configuration for displaying various screens such as content, biometric information, and notification messages. The display (110) can be implemented as various types of displays such as an LCD (liquid crystal display), an OLED (organic light-emitting diode), an LCoS (Liquid Crystal on Silicon), a DLP (Digital Light Processing), a QD (quantum dot) display panel, a QLED (quantum dot light-emitting diodes), a μLED (Micro light-emitting diodes), a Mini LED, etc. Meanwhile, the display (110) can also be implemented as a touch screen combined with a touch sensor, a flexible display, a rollable display, a 3D display, a display in which a plurality of display modules are physically connected, etc.

[0056] The camera (120) is configured to capture a subject. The captured data captured by the camera (120) may include both moving images and still images. Hereinafter, the captured images are referred to as such.

[0057] The camera (120) may include a lens and an image sensor. The type of lens may include a general-purpose lens, a wide-angle lens, a zoom lens, etc., and may be determined according to the type, characteristics, and usage environment of the display device (100). The image sensor may include a complementary metal oxide semiconductor (CMOS) and a charge-coupled device (CCD).

[0058] The camera (120) in FIG. 1 may include at least one RGB camera. If the camera (120) is implemented as an RGB camera, the processor (150) may analyze the user's (10) biometric information using the rPPG method. Specifically, the processor (150) may extract R, G, and B pixel values ​​of an area corresponding to the user's skin within an image captured by the RGB camera, and may calculate a change in skin color based on a change in the extracted pixel values.

[0059] The camera (120) can operate in one of a plurality of operating states, such as a low-resolution state and a high-resolution state, under the control of the processor (150). Meanwhile, the low-resolution state can be expressed as a low-resolution mode, and the high-resolution state can be expressed as a high-resolution mode. However, the present disclosure describes the low-resolution state and the high-resolution state.

[0060] The camera (120) may be provided in multiple units, such as a low-resolution camera and a high-resolution camera. The low-resolution camera and the high-resolution camera can be selectively activated and perform shooting under the control of the processor (150). Here, activation includes being supplied with power and switched to a shooting state.

[0061] In Fig. 1, a case is illustrated where one camera (120) is placed to selectively support low-resolution mode and high-resolution mode.

[0062] In FIG. 2, the camera (120) is illustrated as being included in the display device (100), but it need not necessarily be a built-in camera, and an external camera may be connected to and used with the display device (100). The external camera may be connected and used through various input / output interfaces, such as a USB port or HDMI port, provided in the display device (100). Alternatively, if the display device (100) further includes a communication unit, the camera may receive captured images from an external electronic device with a built-in camera.

[0063] At least one sensor (130) is configured to sense the surrounding conditions of the display device (100) or the user's condition. The at least one sensor (130) may include a light sensor (131), a distance sensor (132), etc. Details thereof will be described in FIG. 3.

[0064] The memory (140) can store at least one command, data, program, etc. necessary for the operation of the display device (100). For example, the memory (140) can store an artificial intelligence model trained based on learning data related to content viewing characteristics. In addition, the memory (140) can store data regarding the illumination range of the environment in which the user (10) is located.

[0065] The memory (140) may be implemented in the form of memory embedded in the display device (100) or in the form of memory detachable from the display device (100) depending on the purpose of data storage. For example, data for driving the display device (100) may be stored in a memory embedded in the display device (100), and data for the expansion function of the display device (100) may be stored in a memory detachable from the display device (100).

[0066] In the case of memory embedded in the display device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD)).

[0067] The memory (140) may be implemented as a single memory that stores data generated from various operations according to the present disclosure, but is not limited thereto, and the memory (140) may be implemented to include multiple memories that each store different types of data or each store data generated at different stages.

[0068] The processor (150) is a component for controlling the operation of the display device (100). The processor (150) may be implemented as a digital signal processor (DSP) for processing digital signals, a microprocessor, but is not limited thereto, and may include one or more of a central processing unit (CPU), a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP), a communication processor (CP), an ARM processor, and an artificial intelligence (AI) processor, or may be defined by the relevant terminology. In addition, the processor (150) may be implemented as a system on chip (SoC) having a built-in processing algorithm, a large scale integration (LSI), or may be implemented in the form of a field programmable gate array (FPGA). The processor (150) may perform various functions by executing computer executable instructions stored in the memory (140).

[0069] The processor (150) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors (150) are implemented as multicore processors, each of the multiple cores included in the multicore processor may include internal processor memory such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to an embodiment of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to an embodiment of the present disclosure.

[0070] The processor (150) can control the display (110) to display content according to user input. The processor (150) can identify the usage status of the user (10) using the shooting data of the camera (120) and the sensing value of at least one sensor (130). The processor (150) can obtain data on the usage status of the user based on the shooting data of the camera (120) and the sensing value of at least one sensor (130). The processor (150) can identify whether the status is suitable for measuring the biometric information of the user (10) through an artificial intelligence model learned based on learning data related to the identified usage status and content viewing characteristics.

[0071] When the processor (150) is identified as being in a state suitable for measuring biometric information, it can acquire new user photographing data through the camera (120) and acquire biometric information based on the acquired user photographing data. The processor (150) can measure the user's (10) biometric information based on the acquired photographing data.

[0072] The processor (150) can measure biometric information when an event occurs. Here, the event includes an event in which a preset time period arrives, an event in which a certain amount of time passes after the display device (100) is turned on, an event in which a certain amount of time changes after a user commands content output or changes the type, source, or brightness of content being viewed, an event in which the brightness changes, an event in which viewing time, usage time zone, device usage time, content / app session usage maintenance time, etc. are changed, etc. Events may be expressed as occurrences, activities, etc., but are referred to as events in the present disclosure. In addition, content in the present disclosure may be of various types, such as terrestrial broadcast content, cable broadcast content, satellite content, a PC screen, a web page screen provided by a web server, a playback screen of a multimedia playback device, a game execution screen of a game player, etc. The provision source may also be implemented in various ways, such as a broadcasting station, a cable broadcasting station, a satellite antenna, a PC, a multimedia playback device, a game player, a set-top box, etc.

[0073] If the processor (150) determines that at least one of the above-described events has occurred, it can automatically measure the user's biometric information.

[0074] Specifically, the processor (150) can measure biometric information of the user (10) using the camera (120) and at least one sensor (130) when the user (10) is watching content through the display device (100). The processor (150) can use an artificial intelligence model to determine biometric information of the user (10).

[0075] The artificial intelligence model may be a pre-trained model based on learning data related to content viewing characteristics. Manufacturers of display devices (100) or other related companies can collect pre-captured data capturing the user's state while viewing various content, and label various items, such as the user's appearance, content type, viewing time, and lighting conditions, to secure a large-scale data set. Manufacturers of display devices (100) or other related companies can train the artificial intelligence model by inputting the acquired data set into the artificial intelligence model and feeding back the analysis results of the artificial intelligence model.

[0076] The processor (150) can determine whether the current state of the user (10) viewing the content is suitable for measuring biometric information based on the user's (10) usage status and the artificial intelligence model described above. A specific determination method is described in FIG. 4.

[0077] If the processor (150) determines that the state is suitable for measuring the user's (10) biometric information, it can drive the camera (120) in a low-resolution state and a high-resolution state to photograph the user (10).

[0078] Meanwhile, if the processor (150) determines that the state is suitable for measuring the user's (10) biometric information, it can acquire new user shooting data through the camera (120) and acquire biometric information based on the acquired shooting data. In addition, the processor (150) can also acquire biometric information of the user (10) by reusing the shooting data of the camera used to acquire data on the usage state described above. Specifically, the processor (150) can acquire data on the user's (10) usage state through the camera (120) and store the shooting data of the camera used at this time in the memory (140). Thereafter, if the processor (150) determines that the state is suitable for measuring biometric information, it can reuse the shooting data stored in the memory (140) to acquire biometric information of the user (10).

[0079] The processor (150) can determine the movement status of the user (10) based on the captured image captured in a low-resolution state. Specifically, the processor (150) can divide all pixels included in each of a plurality of consecutive image frames captured in a low-resolution state into a plurality of block units each consisting of n*m pixels. The processor (150) can detect a representative value representing the characteristics of the pixels in each block. The representative value may be, but is not limited to, the average pixel value of the pixels in each block, and may also be the maximum pixel value, the minimum pixel value, or the RMS (Root Means Square) value.

[0080] The processor (150) can detect the edge of an object included in a photographed image by connecting blocks that have representative values ​​of a similar range and are positioned in consecutive locations among a plurality of blocks. The processor (150) can identify the size of the object based on the number of blocks included in the edge. In addition, the processor (150) can identify the shape of the object based on the shape of the edge, and thereby identify the type of the object.

[0081] In the case of a user, the processor (150) can identify a state corresponding to the shape of an edge by using a database that has previously stored shapes according to various states such as a standing state, a lying state, and a sitting state. When a user is recognized, the processor (150) compares blocks within the edge corresponding to the user in a plurality of continuously captured image frames to check for changes in their number and position. If a difference is identified that is greater than a preset error range as a result of the check, the processor (150) can determine that the user has moved. If the calculation is performed based on an image captured in a low-resolution state, the calculation burden can be reduced because the number of pixels is not large.

[0082] The processor (150) can operate the camera (120) in a high-resolution state if no movement of the user (10) is detected for a preset period of time. The processor (150) measures the biometric information of the user (10) based on the captured image captured in the high-resolution state. A specific measurement method will be described in FIG. 5.

[0083] FIG. 3 is a block diagram showing an example of a detailed configuration of a display device according to at least one embodiment of the present disclosure.

[0084] According to FIG. 3, the display device (100) may further include a display (110), a camera (120), a sensor (130), a memory (140), a processor (150), as well as a communication interface (310), an operation interface (320), an input / output interface (330), a microphone (340), etc. In the configuration of FIG. 3, descriptions of the display (110), the camera (120), the memory (140), and the processor (150) that are the same as those described in FIG. 2 will be omitted for redundant description.

[0085] The communication interface (310) is a component configured to communicate with at least one external device. The communication interface (310) may include at least one wireless communication module, at least one wired communication module, and the like. Each communication module may be implemented in the form of at least one hardware chip. For example, the processor (150) may transmit measured biometric information, etc. to an external device via the communication interface (310).

[0086] As another example, if the processor (150) identifies the user as being in a dangerous state based on measured biometric information, it may transmit a danger alert signal to an external device via the communication interface (310). Specifically, the processor (150) may identify the user as being in a dangerous state if no change in the user's heart rate is detected for a certain period of time or if it is detected to be above or below the normal range. The processor (150) may transmit a danger alert signal to a server device or terminal device operated by a hospital, police station, emergency rescue center, fire station, etc., or to a terminal device of a registered guardian. For this purpose, the memory (140) may store a phone number, email address, messenger ID, etc. of a recipient who will receive the danger alert signal.

[0087] The operation interface (320) is a configuration for receiving user operations. The operation interface (320) may include various buttons, a touch screen, etc. provided on the main body of the display device (100). However, the operation interface (320) is not limited thereto, and may also be implemented by other electronic devices such as a remote control. The user (10) may select a desired menu through the operation interface (320). In addition, the user (10) may input user operations for setting a biometric information measurement cycle, etc., through the operation interface (320). In FIG. 3, the operation interface (320) and the display (110) are illustrated as separate configurations, but if the operation interface (320) is implemented as a touch screen, it may be formed integrally with the display (110).

[0088] The input / output interface (330) is a configuration for inputting and outputting various external signals. The input / output interface (330) can receive at least one of audio and image signals from various content sources (e.g., web servers, media players, user terminal devices, etc.). In addition, the input / output interface (330) can also transmit and receive data or control signals between various external devices (e.g., other display devices, remote controls, mobile phones, speakers, set-top boxes, TVs, lights, etc.). The input / output interface (330) can be implemented as at least one wired input / output interface among HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), USB C-type, DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (Dsubminiature), and DVI (Digital Visual Interface). As described above, when using an external camera rather than the camera itself, the external camera can be connected via the input / output interface (330).

[0089] Meanwhile, at least one sensor (130) includes a light sensor (131) and a distance sensor (132).

[0090] The illuminance sensor (131) is configured to sense the illuminance surrounding the display device (100). The illuminance sensor (131) can measure the intensity of light using the photoelectric effect. The photoelectric effect refers to a phenomenon in which electrons are generated by light energy and current flows when light above a certain frequency is input to a metal. The processor (150) can identify the illuminance surrounding the display device (100) based on the sensing value of the illuminance sensor (131).

[0091] The processor (150) can also utilize the illuminance sensed when viewing each content to identify the user's usage status. For example, the illuminance value measured when watching a drama or movie can be stored as data regarding the usage status.

[0092] Additionally, the processor (150) determines that the identified illuminance is suitable for biometric information measurement if it falls within the illuminance range stored in the memory (140). Details regarding this will be described in FIG. 4.

[0093] The distance sensor (132) is configured to sense the distance to an external object. The processor (150) can identify the distance between the display device (100) and the user (10) based on the sensing value of the distance sensor (132). The distance sensor (132) can include at least one of an ultrasonic sensor, an infrared sensor, a laser sensor, an optical distance sensor, a radar (RADAR) sensor, a lidar (LIDAR) sensor, a photodiode sensor, and a time of flight (TOF) sensor. The processor (150) can additionally check whether the user is in a state suitable for measuring biometric information based on the identified distance.

[0094] For example, if the user (10) is positioned too close to or too far from the display device (100), the processor (150) may determine that accurate biometric information measurement is difficult.

[0095] The processor (150) may suspend biometric information measurement if accurate biometric information measurement is difficult. In another embodiment, the processor (150) may display a notification or message on the display (110) prompting the user to adjust the distance.

[0096] The microphone (340) is configured to receive various audio signals. The microphone (340) may receive the user's voice or other sounds and provide them to the processor (150). When the user's voice is input through the microphone (340), the processor (150) determines that the user is speaking and identifies the user as being unsuitable for biometric information. On the other hand, when the user's voice is not input for a certain period of time or longer, the processor (150) determines that the user is not speaking and identifies the user as being unsuitable for biometric information.

[0097] As described above, the display device (100) may include various configurations. Accordingly, the display device (100) may perform various tasks together to measure the biometric information of the user (10).

[0098] FIG. 4 is a drawing for explaining a biometric information measurement process of a display device according to at least one embodiment of the present disclosure.

[0099] Referring to FIG. 4, the processor (150) executes the aforementioned artificial intelligence model and generates data on the usage status identified by the camera (120) and at least one sensor (130) (S410). The data on the usage status may include data on at least one of various items, such as the type of content being viewed by the user, the user's viewing posture, whether or not they speak, whether or not they move, lighting conditions, viewing time, and viewing time required.

[0100] The processor (150) performs inference using data on the usage status as input values ​​for an artificial intelligence model (S420). The processor (150) may input data on the user's viewing posture into the artificial intelligence model in the form of individually set identification values ​​(e.g., digital values ​​combining 0 and 1) for standing, sitting, and lying postures, or may input data into the artificial intelligence model in the form of a photographed image of the user.

[0101] Inference refers to the process of deriving answers by performing predictions, classifications, and inferences on new input data after an artificial intelligence model has been trained. Thereafter, the processor (150) can obtain the measurement suitability score output from the artificial intelligence model (S430).

[0102] A measurement suitability score refers to information that numerically represents the degree to which a user's (10) usage status is suitable for measurement. The measurement suitability score can be called various names such as a usage pattern score, a characteristic score, a feature score, a target score, etc., but in this disclosure, it is referred to as a measurement suitability score.

[0103] The measurement suitability score may be expressed in percentages, but is not necessarily limited to this. The measurement suitability score can be a predictive value for the reliability score of biometric information based on signal-noise ratio (SNR) information, which will be described later. Details on this will be provided later.

[0104] For example, if a user (10) is watching the news during the evening, the processor (150) performs inference using data such as the viewing genre, the app used, the viewing time, and the viewing time as input values ​​for the artificial intelligence model. As in the example described above, if the user normally watches the news during the evening, quietly under normal lighting conditions without any particular movement or speech, the artificial intelligence model may output a high measurement suitability score.

[0105] As described above, the artificial intelligence model is a pre-trained model based on learning data related to various content viewing characteristics, but the processor (150) may additionally train the artificial intelligence model using the user's usage pattern.

[0106] Specifically, the processor (150) performs inference using an artificial intelligence model with data on the usage status of the user (10) as input values, and stores the data on the usage status in the memory (140). The processor (150) can accumulate the usage status at regular intervals to analyze the usage pattern, and use the data on the usage pattern as learning data to train the artificial intelligence model.

[0107] The processor (150) can identify a state suitable for measuring biometric information if the acquired measurement suitability score is higher than a preset score and the identified illuminance based on the sensing value of the illuminance sensor (131) among at least one sensor is within the illuminance range stored in the memory (140).

[0108] If the acquired measurement suitability score is greater than or equal to a preset score (S440), the processor (150) can identify the illuminance based on the sensing value of the illuminance sensor (131) among at least one sensor (S450). Conversely, if the measurement suitability score does not exceed the preset score, the processor (150) may not perform biometric information measurement. In this case, the processor (150) can update the learning data by adding the cases where the preset score is not exceeded to the usage pattern data stored in the memory (140).

[0109] For example, if the preset score is 0.7, the processor (150) can drive the illuminance sensor (131) to obtain a sensing value when the calculated measurement suitability score is 0.7 or higher, and then identify the illuminance based on the sensing value.

[0110] If the identified illuminance falls within the illuminance range stored in the memory (140) (S460), the processor (150) can determine that the user (10) is in a state suitable for measuring biometric information (S470). Conversely, if the identified illuminance does not fall within the illuminance range, biometric information measurement is not performed.

[0111] For example, if the illuminance range is set to 300 lx to 500 lx and the illuminance measured using the illuminance sensor (131) is 400 lx, the processor (150) can determine that the condition is suitable for measuring biometric information. The manufacturer of the display device (100) or a related company can repeatedly perform a task of comparing the results of measuring biometric information using captured images taken under various illuminances, set an illuminance range that allows for accurate biometric information measurement, and then store the result in the memory (140). Information about the illuminance range may be updated from time to time or periodically.

[0112] Meanwhile, the processor (150) can divide the measured illuminance using the illuminance sensor (131) into preset illuminance ranges and calculate illuminance scores representing each illuminance range. Specifically, the processor (150) can calculate illuminance scores in the manner shown in the following table.

[0113] Illuminance (lx) Illuminance score 0~1000.5 100~2000.6 200~3000.7 300~4000.8 400~5000.8 500~6000.7

[0114] According to Table 1, the processor (150) can calculate the illuminance score as 0.6 when the measured illuminance is 100 lx to 200 lx, and as 0.8 when the measured illuminance is 300 lx to 400 lx. The processor (150) can update the illuminance score by adding a bonus point according to the reliability score to be calculated later. That is, even if the measured illuminance is included in the illuminance range set to be suitable for measurement, if the reliability score to be calculated later is calculated low, the illuminance range can be adjusted and the illuminance score can be updated. Conversely, if the reliability score is calculated high, a bonus point can be added to the illuminance score. Table 2 shows an example of the result of updating the illuminance score based on the reliability score.

[0115] Illuminance (lx) Illuminance score 0~1000.5 100~2000.6 200~3000.7 300~4000.85 400~5000.8 500~6000.7

[0116] The processor (150) may assign a bonus point to a specific illuminance range if the confidence score is greater than or equal to a specific value. For example, if the confidence score is greater than or equal to 0.9 in the 300-400 lx range, the processor (150) assigns a bonus point of 0.05 to update the illuminance score to 0.85. On the other hand, if the confidence score is less than or equal to 0.7, the bonus point is assigned as -0.05 to update the illuminance score to 0.75. Table 2 shows a case where the confidence score is 0.9. As described above, the processor (150) may convert the illuminance range into an illuminance score format and use it by updating it according to the confidence score. If the updated illuminance score is greater than or equal to a specific value for measurement, the processor (150) may recognize that the illuminance environment is suitable for measuring biometric information. In this way, by using the updated illuminance score, the performance of predicting the measurement time point can be improved.

[0117] Below, the process of measuring biometric information in a processor (150) when it is determined that the state is suitable for measuring biometric information will be described.

[0118] FIG. 5 is a drawing for explaining a process of measuring biometric information through a camera of a display device according to at least one embodiment of the present disclosure.

[0119] Referring to FIG. 5, if the processor (150) is identified as being in a state suitable for measuring the biometric information of the user (10), the processor can detect the movement of the user (10) based on the captured image acquired through the camera (120) in a low-resolution state (S510).

[0120] A low-resolution state is a state in which a photographed image is created with a relatively low resolution by reducing the number of pixels that detect light in the image sensor within the camera (120).

[0121] By taking pictures in low resolution, the user's movements can be identified without the user's appearance being clearly expressed, thus protecting the user's privacy.

[0122] The processor (150) waits without performing measurements when the user (10) moves a lot.

[0123] On the other hand, if the user's movement is not detected for a preset period of time (S520), the processor (150) can acquire the user's (10) biometric information based on the captured image acquired through the camera (120) in a high-resolution state.

[0124] The processor (150) can drive the camera (120) in a high-resolution state to capture a user (10) (S530). The high-resolution state refers to a state in which the number of pixels detecting light in the image sensor is increased compared to the low-resolution state, thereby enabling a clear capture of the user's face or body parts.

[0125] The above describes a case where the operating state of one camera is selectively driven in a low-resolution state and a high-resolution state. However, if both a low-resolution camera and a high-resolution camera are provided, each camera can be used sequentially.

[0126] The processor (150) can identify at least one facial region or other body part from a photographed image acquired in a high-resolution state. The processor (150) can measure biometric information of the user (10) from the identified facial region using the rPPG (remote Photoplethysmography) method (S540) (S550).

[0127] FIGS. 6 and 7 are drawings for explaining an rPPG method according to at least one embodiment of the present disclosure.

[0128] Referring to FIG. 6, the processor (150) can acquire a photographed image captured in a high-resolution state (S610). The processor (150) can identify a face area within a plurality of consecutive image frames among the photographed images (S620).

[0129] Specifically, the processor (150) divides all pixels included in each of a plurality of consecutive image frames captured in a high-resolution state into a plurality of block units each consisting of n*m pixels, and detects a representative value for each block. The processor (150) can detect an edge by connecting a plurality of blocks that are positioned consecutively among the blocks in one image frame and have representative values ​​in a similar range. Examples of representative values ​​and an edge detection method have been specifically described in the above-described section, so a redundant description will be omitted.

[0130] If the camera (120) is installed at the center of the display device and the user is viewing the display device (100), the captured image may include the front of the user's face. Accordingly, the user's face may have a circular or vertically elliptical shape. If the connection shape of the blocks corresponding to the edge is circular or vertically elliptical, the processor (150) may determine that the blocks within the edge correspond to the user's face. The processor (150) may extract blocks corresponding to the user's eyes, nose, and mouth from among the blocks corresponding to the user's face to identify the characteristics of the user's entire face. The processor (150) may detect the R, G, and B pixel values ​​of each pixel within the block corresponding to the part of the user's face where the heart rate change is likely to appear (S630). The processor (150) may extract a pulse signal according to a change state of at least one of the R, G, and B pixel values ​​detected using the rPPG method (S640).

[0131] The rPPG method captures images of a person's face at a certain distance from the camera, extracting subtle movements from the captured image to measure biometric information. This noninvasive method remotely measures heart rate and blood flow.

[0132] Specifically, by measuring changes in the R, G, and B pixel values ​​of pixels constituting the same facial part in multiple image frames captured continuously, a signal in the form of a pulse signal can be extracted.

[0133] The processor (150) can obtain R, G, and B pixel values ​​of each pixel within a face area identified within a plurality of consecutive image frames among the captured images acquired in a high-resolution state, obtain a pulse signal according to a change state of at least one of the R, G, and B pixel values ​​acquired using the rPPG method, and obtain the user's biometric information based on changes in the size of peaks of the acquired pulse signal.

[0134] The processor (150) can detect peaks of a pulse signal and measure biometric information based on the magnitude of the detected peaks and the peak occurrence period, etc. (S650). For example, the color change period of some parts of the user's face may change or the color value may change depending on changes in heart rate. In the case of body temperature, the R value may be measured stronger as the body temperature rises. In the case of paleness due to poor blood circulation, the magnitude of all R, G, and B values ​​may be measured larger. The processor (150) estimates biometric information based on changes in the R, G, and B pixel values ​​(S660).

[0135] Figure 7 is a diagram illustrating a process for measuring heart rate variability (HRV), one of the user's biometric information, using the rPPG method. Prior to measuring biometric information using the rPPG method, the processor (150) can capture an image of the user's face area using a high-resolution camera (710).

[0136] The processor (150) can identify landmarks or representative values ​​of a facial region in units of pixels or pixel blocks for each of a plurality of consecutive image frames within a captured image (720). The processor (150) selects a region of interest (ROI) based on the identified landmarks, and can remove eye and mouth areas using the landmarks described above. The processor (150) can convert each frame into HSV (Hue, Saturation, Value) color space to extract a skin color area and remove hair and beard areas (740). Here, HSV refers to a color space that expresses hue, saturation, and value in contrast to the RGB color space.

[0137] The processor (150) can extract a pulse signal from color changes in a skin color region (750). Before applying the rPPG method, the processor (150) can calculate the spatial average of pixel values ​​in the skin color region and decompose them into RGB components. The processor (150) can extract a pulse signal using the rPPG method using the decomposed RGB components.

[0138] The processor (150) can convert a pulse signal into a frequency domain. The processor (150) can detect a peak corresponding to a frequency component of a heartbeat based on the pulse signal converted into the frequency domain (760). The processor (150) can track the detected peak to calculate a heart rate and produce a heart rate variability measurement value (770).

[0139] The processor (150) can measure the biometric information of the user (10) using the above-described rPPG method.

[0140] Meanwhile, the processor (150) can obtain signal-noise ratio (SNR) information to measure the reliability of biometric information measured using the rPPG method. The signal-noise ratio (SNR) is a numerical value representing the signal-to-noise ratio, expressed in decibels. The higher the SNR value, the higher the reliability of the measurement results.

[0141] The processor (150) can update the measurement suitability score according to the reliability score for the measured biometric information obtained based on the SNR information calculated from the measured biometric information.

[0142] The processor (150) can determine the quality and reliability of the measurement result based on the SNR information of the biometric information measured using the rPPG method and quantify the result to calculate a reliability score.

[0143] The processor (150) can distinguish between a signal (i.e., noise) having an intensity below a threshold value and a signal having an intensity above a threshold value within the rPPG signal, and then calculate the ratio and convert it into a reliability score.

[0144] There are various methods for calculating reliability scores using SNR information. In this disclosure, the final score can be calculated by linearly mapping values ​​between 0 and 1. However, the method for calculating reliability using SNR information is not limited to this method and can be calculated in various ways.

[0145] The processor (150) can update the measurement suitability score described above based on the calculated reliability score. Details regarding this are described in FIG. 10.

[0146] FIGS. 8 and 9 are drawings for explaining a biometric information providing process of a display device according to at least one embodiment of the present disclosure.

[0147] Referring to FIG. 8, the processor (150) automatically determines a timing suitable for measuring biometric information, measures the user's biometric information, and stores it in the memory (140). The processor (150) may analyze the biometric information stored in the memory (140) by day, month, and year, and provide the user with a graph for each biometric information. When the processor (150) measures the user's (10) biometric information, if the biometric information is measured worse than previously measured biometric information or a preset normal range, the processor (150) may display an abnormal signal detection message on the display (110) (810).

[0148] For example, when the display device (100) measures the heart rate among the biometric information of the user (10), and a heart rate value different from the resting heart rate data of the user (10) is measured, the display device (100) may provide a message such as detection of an abnormal heart rate (HR) signal on the display (110) or inform the user (10) of the abnormal signal detection result through a speaker or the like.

[0149] Fig. 9 illustrates a case where a display device (100) that measures biometric information notifies another external electronic device of the measurement results. According to Fig. 9, if the biometric information characteristics have deteriorated compared to before, or if the biometric information differs by a certain amount or more from the preset normal range, the display device (100) can provide a result of detecting an abnormal signal of the user's (10) biometric information to a mobile device (900) connected to the display device (100) (910, 920).

[0150] For example, if the user (10) falls asleep while watching content or uses a mobile device while watching content, the display device (100) can provide the result of detecting an abnormal signal in the user's (10) biometric information through a mobile device (900) linked to the display device (100) via a notification or message.

[0151] As another example, a user (100) can receive information about his / her biometric information measured through a display device (100) using a mobile device (900).

[0152] FIG. 9 illustrates a case where a message (920) is transmitted to a mobile device (900) owned by the user (10) of the display device (100). However, as described above, the measurement results or such notification messages may also be transmitted to other external terminal devices or server devices. Accordingly, when the user is identified as having lost consciousness and collapsed or as having a health problem, rescue can be performed immediately or a guardian can be notified.

[0153] FIG. 10 is a diagram for explaining an artificial intelligence model learning process of a display device according to at least one embodiment of the present disclosure.

[0154] The processor (150) can update learning data using data on the usage status identified through the camera (120) and at least one sensor (130). The processor (150) can further train an artificial intelligence model using the updated learning data. The processor (150) can update the measurement suitability score based on the reliability score.

[0155] Referring to Fig. 10, updated learning data (1020) is shown by adding new data to existing learning data (1010). Specifically, this shows a case where usage status data is included indicating that a user ran a DDD app and watched children's content (Kids) provided by the app for 3 hours, and that the illuminance value at the time of viewing was 420. As described above, the learning data may include data on multiple content viewing characteristics that combine at least one characteristic among the type of content, the content provision source, the viewing time, the viewing time, and the illuminance level, and multiple measurement suitability scores set for each of the multiple viewing characteristics. However, the present invention is not limited thereto, and may additionally include data on viewing characteristics.

[0156] For example, if biometric data is measured while a user (10) is watching a Kids genre using a display device (100), usage pattern data such as the type of content, viewing time, viewing time, illuminance, and measurement suitability score can be added to the learning data of the artificial intelligence model (1020).

[0157] On the other hand, if biometric data is measured while the user is watching news content (News) already included in the existing learning data, the existing learning data can be updated and the measurement suitability score can be updated according to the newly calculated reliability score (1020).

[0158] In this way, the display device (100) can newly measure the biometric information of the user (10), and in this process, add / update learning data by adding usage pattern data and reliability scores according to new usage status characteristics to the artificial intelligence model, and add / update other measurement suitability scores to the reliability scores.

[0159] FIG. 11 is a flowchart of a method for measuring biometric information of a display device according to at least one embodiment of the present disclosure.

[0160] Referring to FIG. 11, a display device displays content according to a user input (S1110). The display device acquires data on a user's usage status based on a camera and at least one sensing value (S1120). The display device identifies a state suitable for biometric information measurement based on an artificial intelligence model learned based on learning data related to content viewing characteristics and an identified usage status (S1130). The display device acquires new user shooting data through a camera and acquires biometric information based on the acquired shooting data (S1140).

[0161] Since the specific method for determining whether a state is suitable for measuring biometric information and measuring biometric information has been specifically described in the various embodiments described above, a duplicate description will be omitted.

[0162] FIG. 12 is a diagram for explaining the overall flow of biometric information measurement of a display device according to at least one embodiment of the present disclosure.

[0163] Referring to FIG. 12, when an event for measuring biometric information occurs (S1210), the display device executes an artificial intelligence model to obtain a measurement suitability score (S1220). If the measurement suitability score is greater than or equal to a threshold score, the display device determines whether the illumination is suitable (S1230). If the illumination is suitable, the display device detects movement using a low-resolution camera (S1240). If movement is not detected, the display device measures biometric information using the rPPG method using a high-resolution camera and calculates a reliability score (S1250).

[0164] Thereafter, the display device detects a usage pattern based on data about the user's usage status (S1260), updates learning data based on the usage pattern (S1270), and updates the illuminance range based on the illuminance score (S1280).

[0165] The operations and methods described in the various flowcharts above can be performed by a display device having the configuration shown in FIGS. 2 and 3, but are not necessarily limited thereto, and can also be performed by electronic devices having various configurations.

[0166] Meanwhile, the operations and methods according to the various embodiments described above can be performed according to the execution of an artificial intelligence model and other software modules.

[0167] FIG. 13 is a diagram illustrating a software structure for implementing embodiments of a display device according to at least one embodiment of the present disclosure.

[0168] According to FIG. 13, a learning data module (S1310), an artificial intelligence model module (S1320), a measurement condition detection module (S1330), an illuminance sensor preprocessing module (S1340), a low-resolution motion level preprocessing module (S1350), an rPPG method module (S1360), a biometric information storage database module (S1370), a measurement quality analysis module (S1380), and a viewing pattern detection module (S1390) may be stored in the memory of the display device (100). However, the present invention is not limited thereto and may further include other modules. The processor (150) may execute each software module to perform operations according to the various embodiments described above.

[0169] The learning data module (S1310) and the artificial intelligence model module (S1320) use an artificial intelligence model based on the user's learning data to calculate a measurement suitability score. As described in detail in the above section, a redundant description will be omitted.

[0170] The measurement condition detection module (S1330) is a module for detecting changes in viewing content, usage time, input source, app change, OTT content switching, and illuminance value output from the display device (100). The illuminance sensor preprocessing module (S1340) is a module for determining an illuminance range suitable for measuring biometric information using a illuminance sensor. The low-resolution movement level preprocessing module (S1350) is a module for detecting whether a user is moving using a low-resolution camera.

[0171] The rPPG method module (S1360) is a module for measuring bio-signals using the rPPG method from a user's high-resolution captured image. The bio-information storage database module (S1370) is a module for measuring bio-information based on bio-signals and storing it in a database. The measurement quality analysis module (S1380) is a module for calculating a reliability score of bio-signals using the SNR method. The viewing pattern detection module (S1390) is a module for detecting a viewing pattern based on data about the user's usage status. The processor can execute these modules in parallel and sequentially to perform the above-described process.

[0172] Meanwhile, although the above-described various embodiments have described cases in which biometric information is measured in a display device that directly includes a camera or is connected to an external camera or an external device including a camera, according to another embodiment, biometric information measurement may be performed by a server device connected to the display device. In this case, the display device may transmit the sensing results sensed by the camera and sensor to the server device, or transmit data on the usage status identified based on the sensing results to the server device. When the data is received, the server device may measure the user's status information using an artificial intelligence model and then transmit the measured data to the display device or other user terminal device. Since specific usage status identification methods, status information measurement methods, etc. have been described in the above-described various embodiments, redundant descriptions thereof will be omitted.

[0173] The various embodiments described above may be implemented as a single embodiment, or at least one of the embodiments may be combined in whole or in part and implemented together in one device.

[0174] According to the various embodiments described above, more accurate and efficient biometric information measurement is possible by automatically determining a timing suitable for biometric information measurement.

[0175] Meanwhile, the various embodiments described above may be applied to a product as an embodiment alone, but at least some of the contents may be implemented in combination with other embodiments of the present disclosure.

[0176] The various embodiments described above can be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call instructions stored from the storage medium and operate according to the called instructions, and may include an electronic device (e.g., a display device (100)) according to the disclosed embodiments. When an instruction is executed by a processor, the processor can perform a function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter. The machine-readable storage medium can be provided in the form of a non-transitory computer-readable storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0177] Additionally, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product.

[0178] Specifically, a non-transitory readable storage medium or a computer program product storing computer instructions that cause the computer to perform operations including a step of displaying content according to a user input, a step of identifying a user's usage status based on a camera and at least one sensing value, a step of identifying a state suitable for biometric information measurement based on an artificial intelligence model learned based on learning data related to content viewing characteristics and the identified usage status, and a step of newly acquiring user's photographing data through the camera and acquiring biometric information based on the acquired photographing data may be provided.

[0179] The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0180] In addition, computer instructions or programs for performing the biometric information measurement method of the display device according to the various embodiments described above may be stored in a non-transitory computer-readable medium. The computer instructions stored in such a non-transitory computer-readable medium, when executed by a processor of a specific device, cause the specific device to perform processing operations in the device according to the various embodiments described above. A non-transitory computer-readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of the non-transitory computer-readable medium may include a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, etc.

[0181] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In the display device, display; camera; At least one sensor; A memory storing an artificial intelligence model learned based on learning data related to content viewing characteristics; and a processor; including; The above processor, Control the display to display content based on user input; Obtaining data on the usage status based on the shooting data of the above camera and the sensing value of at least one sensor, Based on the data on the above usage status and the artificial intelligence model, if it is identified as a state suitable for measuring biometric information, A display device that acquires new user shooting data through the above camera and acquires biometric information based on the acquired user shooting data.

2. In paragraph 1, The above memory stores data for a preset illuminance range, The above processor, With the above artificial intelligence model running, data on the identified usage status is used as input values ​​for the artificial intelligence model to obtain a measurement suitability score output from the artificial intelligence model, The obtained above measurement suitability score is greater than or equal to the preset score, A display device that identifies the illuminance as being suitable for measurement of the biometric information when the illuminance identified based on the sensing value of the illuminance sensor among the at least one sensor is included within the illuminance range stored in the memory.

3. In paragraph 2, The above processor, If identified as being suitable for the above biometric measurement, Detecting the user's movement based on the captured image acquired through the camera in a low-resolution state, If the user's movement is not detected for a preset period of time, A display device that acquires biometric information of the user based on a photographed image acquired through the camera in a high-resolution state.

4. In paragraph 3, The above processor, A display device that identifies at least one facial region in a photographed image acquired in the above high-resolution state, and measures biometric information of the user from the identified facial region using the rPPG (remote Photoplethysmography) method.

5. In paragraph 4, The above processor, Obtaining the R, G, and B pixel values ​​of each pixel within the identified face area from among the consecutive image frames of the captured images obtained in the above high-resolution state, Using the above rPPG (remote Photoplethysmography) method, a pulse signal is obtained according to a change state of at least one of the acquired R, G, and B pixel values, A display device that acquires the user's biometric information based on changes in the size of peaks of the acquired pulse signal.

6. In paragraph 3, The above processor, A display device that updates the learning data using the identified usage status and further trains the artificial intelligence model using the updated learning data.

7. In paragraph 6, The above learning data is, Includes data on multiple content viewing characteristics that combine at least one of the following characteristics: type of content, content provider source, viewing time, viewing time, and light level; and multiple measurement suitability scores set for each of the multiple viewing characteristics. The above processor, A display device that updates the measurement suitability score according to a reliability score for the measured biometric information obtained based on SNR (signal-noise ratio) information calculated from the measured biometric information.

8. In a method for measuring biometric information of a display device, A step for displaying content based on user input; A step of acquiring data on a user's usage status based on a camera and at least one sensing value; A step of identifying a state suitable for biometric information measurement based on an artificial intelligence model learned based on learning data related to content viewing characteristics and data on the usage status; and A method for measuring biometric information, comprising: a step of newly acquiring a user's photographing data through the camera and acquiring biometric information based on the acquired photographing data.

9. In paragraph 8, A step of using data on the identified usage status as an input value of the artificial intelligence model while executing the artificial intelligence model to obtain a measurement suitability score output from the artificial intelligence model; and A method for measuring biometric information, comprising: a step of identifying a state suitable for measuring biometric information when the acquired measurement suitability score is equal to or greater than a preset score and the illuminance identified based on a sensing value of a light sensor among at least one sensor is within the illuminance range stored in the memory; 10. In paragraph 9, When identified as being suitable for measurement of the above biometric information, A step of detecting the movement of the user based on the captured image acquired through the camera in a low-resolution state; and If the user's movement is not detected for a preset period of time, A method for measuring biometric information, comprising: a step of obtaining biometric information of the user based on a photographed image acquired through the camera in a high-resolution state.

11. In paragraph 10, A method for measuring biometric information, comprising: a step of identifying at least one facial region in a photographed image acquired in the high-resolution state, and measuring biometric information of the user from the identified facial region using a remote photoplethysmography (rPPG) method.

12. In paragraph 11, A step of acquiring R, G, and B pixel values ​​of each pixel within the identified face area within a plurality of consecutive image frames among the captured images acquired in the above high-resolution state; A step of obtaining a pulse signal according to a change state of at least one of the acquired R, G, and B pixel values ​​using the above rPPG (remote Photoplethysmography) method; and A method for measuring bio-information, comprising: a step of obtaining bio-information of the user based on changes in the magnitude of peaks of the acquired pulse signal.

13. In paragraph 10, A method for measuring biometric information, comprising: a step of updating the learning data using data on the identified usage status, and further training the artificial intelligence model using the updated learning data.

14. In paragraph 13, The above learning data is, Includes data on multiple content viewing characteristics that combine at least one of the following characteristics: type of content, content provider source, viewing time, viewing time, and light level; and multiple measurement suitability scores set for each of the multiple viewing characteristics. A method for measuring biometric information, comprising: a step of updating the measurement suitability score according to a reliability score for the measured biometric information obtained based on SNR (signal-noise ratio) information calculated from the measured biometric information.

15. A non-transitory computer-readable storage medium storing computer instructions that, when executed by a processor of a display device, cause the display device to perform an operation, wherein the operation is: A step for displaying content based on user input; A step of acquiring data on a user's usage status based on a camera and at least one sensing value; A step of identifying a state suitable for biometric information measurement based on an artificial intelligence model learned based on learning data related to content viewing characteristics and data on the usage status; and A non-transitory computer-readable storage medium, comprising: a step of newly acquiring a user's photographing data through the camera, and acquiring biometric information based on the acquired photographing data.

Images