Display device recognizing user gestures through sensor, and control method thereof

Abstract

A display device is provided. The display device includes a display, a sensor and one or more processors configured to control the display to display an indicator, recognize a user's body in an image received through the sensor in real time, identify a first area of the body as a reference point, and identify a second area of the body as an auxiliary point, identify a change in the coordinates of the reference point and a change in the coordinates of the auxiliary point based on a plurality of image frames included in the image, and move the indicator based on the change in the coordinates of the reference point and the change in the coordinates of the auxiliary point.

Description

[Technical Field]

[0001] The present disclosure relates to a display device that recognizes a user gesture via a sensor and a controlling method thereof, and more particularly, to a display device that recognizes a user gesture and moves and displays an indicator and a controlling method thereof.[Background Art]

[0002] Recently, with technological advances in the field of display devices, various types of display devices have been developed and popularized.

[0003] Various types of user interfaces for controlling display devices have also been developed and popularized.

[0004] In particular, research on the so-called Natural User Interface (NUI) that detects natural human body movements and controls a display device using them rather than controlling the display device using peripheral devices such as a keyboard, a mouse, or a remote controller has been actively conducted.

[0005] When controlling a device using body movements, the challenge is to detect the body movements with high accuracy so that fine control can be achieved while ignoring unnecessary body movements to match the user's intent.

[0006] When controlling a display device by detecting the user's body movements and moving a cursor, etc. according to the detected body movements, there is a growing need for a device controlling method that accurately reflects even the slightest movements of the cursor.[Disclosure][Technical Solution]

[0007] A display device according to an embodiment includes a display, a sensor one or more processors configured to control the display to display an indicator, recognize a user's body in an image received through the sensor in real time, identify a first area of the body as a reference point, and identify a second area of the body as an auxiliary point, identify a change in the coordinates of the reference point and a change in the coordinates of the auxiliary point based on a plurality of image frames included in the image, and move the indicator based on the change in the coordinates of the reference point and the change in the coordinates of the auxiliary point, and the change in the coordinates of the auxiliary point corresponds to the relative coordinate change of the auxiliary point with reference to the reference point.

[0008] A controlling method of a display device according to an embodiment includes displaying an indicator, recognizing a user's body in an image received in real time, identifying a first area of the body as a reference point, and identifying a second area of the body as an auxiliary point, identifying a change in the coordinates of the reference point and a change in the coordinates of the auxiliary point based on a plurality of image frames included in the image, and moving the indicator based on the change in the coordinates of the reference point and the change in the coordinates of the auxiliary point, and the change in the coordinates of the auxiliary point corresponds to the relative coordinate change of the auxiliary point with reference to the reference point.

[0009] In a computer-readable recording medium including a program that executes a controlling method of a display device according to an embodiment, the controlling method of the display device includes displaying an indicator, recognizing a user's body in an image received in real time, identifying a first area of the body as a reference point, and identifying a second area of the body as an auxiliary point, identifying a change in the coordinates of the reference point and a change in the coordinates of the auxiliary point based on a plurality of image frames included in the image, and moving the indicator based on the change in the coordinates of the reference point and the change in the coordinates of the auxiliary point, and the change in the coordinates of the auxiliary point corresponds to the relative coordinate change of the auxiliary point with reference to the reference point.[Description of Drawings]

[0010] FIG. 1 is a view provided to explain a display device schematically according to an embodiment; FIG. 2 is a block diagram provided to explain configuration of a display device according to an embodiment; FIG. 3 is a view provided to explain a movement of an indicator in response to a movement of a first area according to an embodiment; FIG. 4 is a view provided to explain a movement of an indicator in response to a movement of a first area according to an embodiment; FIG. 5 is a view provided to explain a movement of an indicator in response to a movement of a second area according to an embodiment; FIG. 6 is a view provided to explain a movement of an indicator in response to a user position according to an embodiment; FIG. 7 is a view provided to explain a movement of an indicator in response to a movement of a second area according to an embodiment; FIG. 8 is a view provided to explain a selection command according to an embodiment; and FIG. 9 is a flowchart provided to explain a controlling method of a display device according to an embodiment. [Detailed Description of Embodiments]

[0011] Terms used in this specification will be described briefly and then, the present disclosure will be described in detail.

[0012] General terms that are currently widely used are selected as the terms used in the embodiments of the disclosure in consideration of their functions in the disclosure, but may be changed based on the intention of those skilled in the art or a judicial precedent, the emergence of a new technique, or the like. In addition, in a specific case, terms arbitrarily chosen by an applicant may exist, in which case, the meanings of such terms will be described in detail in the corresponding descriptions of the disclosure. Therefore, the terms used in the embodiments of the disclosure need to be defined on the basis of the meanings of the terms and the overall contents throughout the disclosure rather than simple names of the terms.

[0013] The embodiments of the present disclosure may be modified in various ways, and may have various embodiments, so specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, it is to be understood that the disclosure is not limited to specific exemplary embodiments, but include all modifications, equivalents, and / or alternatives according to exemplary embodiments of the disclosure. In describing the disclosure, when it is decided that a detailed description for the known functions or configurations related to the disclosure may unnecessarily obscure the gist of the disclosure, the detailed description therefor will be omitted.

[0014] Terms such as "first", "second", etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only to distinguish one component from another.

[0015] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, operations, components, parts, or a combination thereof described in the specification, but are not intended to exclude in advance the possibility of the presence or addition of one or more of other features, numbers, steps, operations, components, parts, or a combination thereof.

[0016] In exemplary embodiments, a "module" or a "unit" may perform at least one function or operation, and be implemented as hardware or software or be implemented as a combination of hardware and software. In addition, a plurality of "modules" or a plurality of "units" may be integrated into at least one module and be implemented as at least one processor (not shown) except for a 'module' or a 'unit' that needs to be implemented as specific hardware.

[0017] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those with ordinary skill in the art to which the present disclosure belongs can easily practice the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present disclosure in the drawings, parts that are not related to the description are omitted, and similar parts are given similar drawing reference numerals throughout the specification.

[0018] FIG. 1 is a view provided to explain a display device schematically according to an embodiment.

[0019] A display device 100 according to an embodiment may display various types of content.

[0020] According to an embodiment, the display device 100 may be any device having a display function, such as, but not limited to video wall, large format display (LFD), digital signage, digital information display (DID), projector display, electric whiteboard display, etc.

[0021] Further, the display device 100 may be may be placed in a wall-mounted form or may be placed in a stand form.

[0022] Meanwhile, the display device 100 may also be implemented as a touch screen coupled with a touch sensor, a flexible display, a rollable display, a three-dimensional (3D) display, a display where a plurality of display modules are physically connected, and the like.

[0023] The display device 100 according to an embodiment may display an indicator 10 through a display 110.

[0024] Further, the display device 100 may recognize a user gesture, and may move the indicator 10 in response to the recognized user gesture.

[0025] Here, the indicator 10 may be implemented as a pointer, a cursor (e.g., a character cursor, a mouse cursor, etc.), a highlighter that focuses (or emphasizes) one of the characters on the screen, an item, an object, etc.

[0026] Here, the user gesture may include a movement (e.g., shift) of a specific area (e.g., wrist, palm, etc.) of the user's body. However, the present disclosure is not limited thereto, and the user gesture may include tap, double tap, swipe, flick, pinch in / out, rotate, etc. of a specific area.

[0027] For example, a tap gesture may include a short and light tap with one finger, and a double tap gesture might include two quick, consecutive taps.

[0028] A swipe gesture may include a gesture that moves a certain distance while holding a tap, such as moving a certain distance to the left or right.

[0029] A flick gesture may include a gesture of quickly scrolling horizontally or vertically while holding a tap.

[0030] A pinch in / out gesture is a gesture for zooming in / out, and when the distance between two fingers increases, it may be a pinch out gesture for zooming in, and when the distance between two fingers decreases, it may be a pinch in gesture for zooming out.

[0031] A rotate gesture includes a gesture that changes from vertical to horizontal or horizontal to vertical, and may be a gesture that rotates a screen (or an object within the screen).

[0032] Referring to FIG. 1, the display device 100 may recognize a user's body in an image received in real time via the sensor 120, and may identify a movement of a specific area of the user's body. Subsequently, the display device 100 may move the indicator 10 based on the movement of the specific area.

[0033] Meanwhile, the display device 100 according to an embodiment may identify a first area and a second area on the user's body, and may move the indicator 10 based on the movement of the first area and the movement of the second area.

[0034] The display device 100 according to an embodiment moves the indicator 10 based on the movement of each of a plurality of areas (e.g., the first area and the second area), so that the indicator 10 can be moved more closely to the user's intent, and can be moved more precisely.

[0035] FIG. 2 is a block diagram provided to explain configuration of a display device according to an embodiment.

[0036] Referring to FIG. 2, the display device 100 according to an embodiment includes a display 110, a sensor 120, and one or more processors 130.

[0037] The display 110 according to an embodiment may be implemented as a display in various forms, such as a liquid crystal display (LCD), organic light-emitting diode (OLED), liquid crystal on silicon (LCoS), digital light processing (DLP), quantum dot (QD) display panel, quantum dot light-emitting diodes (QLEDs) micro light-emitting diodes (µLEDs), Mini LEDs, and the like.

[0038] However, the present disclosure is not limited thereto, and the display device 100 may be implemented as a projector device (or projector display) that projects an image, and the display 110 may be implemented as a projection that projects an image.

[0039] The sensor 120 according to an embodiment may detect a user located in the vicinity (e.g., at least one of the front, rear, or side) of the display device 100.

[0040] For example, the sensor 120 may be implemented as a camera, and the camera may capture a still image at a specific point in time, or may capture still images continuously.

[0041] The display device 100 according to an embodiment may identify a user located in the vicinity of, or adjacent to, the display device 100 based on the image received from the sensor 120.

[0042] Referring to FIG. 1, the display device 100 may recognize a user (i.e., a user's body) in an image received in real time via the sensor 120. Here, the sensor 120 photographs the front of the display device 100, and thus, the display device 100 may recognize a user located in front of the display device 100.

[0043] According to an embodiment, the sensor 120 is implemented as a camera, and here, the camera may obtain a color image, a depth image, etc. However, this is only an example, and the sensor 120 may be implemented as a time-of-flight (ToF) sensor, a lidar sensor, a radar sensor, an ultrasonic sensor, an infrared sensor, or the like. For example, the infrared sensor may recognize a user by detecting infrared light information.

[0044] The one or more processors 130 according to an embodiment are electrically connected to memory (not shown), and control the overall operations of the display device 100.

[0045] According to an embodiment, the processor 130 may be implemented as a digital signal processor (DSP) for processing digital signals, a microprocessor, or a time controller (TCON). However, the processor 130 is not limited thereto, and the processor 130 may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), an ARM processor, and an Artificial Intelligence (AI) processor, or may be defined as the corresponding term.

[0046] Further, the processor 130 may be implemented as a system on chip (SoC) with embedded processing algorithms, a large scale integration (LSI), or in the form of a field programmable gate array (FPGA). The processor 130 may perform various functions by executing computer executable instructions stored in the memory.

[0047] The processor 130 according to an embodiment may control the display 110 to display the indicator 10.

[0048] Subsequently, the processor 130 may recognize the user's body in the image received in real time via the sensor 120. For example, the processor 130 may recognize a body part of the user, such as face, hand, arm, leg, or the like, in the image received in real time via the sensor 120, and may track the movement of the recognized body part.

[0049] The processor 130 according to an embodiment may identify a first area of the body as a reference point and a second area as an auxiliary point. Here, specific descriptions of the reference point and the auxiliary point will be provided later with reference to FIG. 3.

[0050] The processor 130 may identify a change in the coordinates of the reference point and a change in the coordinates of the auxiliary point based on a plurality of image frames included in the image.

[0051] Subsequently, the processor 130 may move the indicator 10 based on the change in the coordinates of the reference point and the change in the coordinates of the auxiliary point.

[0052] FIG. 3 is a view provided to explain a movement of an indicator in response to a movement of a first area according to an embodiment.

[0053] The processor 130 according to an embodiment may identify a first area of the user's body as a reference point 1 in the image received from the sensor 120. Here, the first area may include a preset area of the body (e.g., wrist, palm, etc.).

[0054] Further, the processor 130 may identify a second area of the user's body as an auxiliary point 2 in the image received from the sensor 120. Here, the second area may include a preset area of the body (e.g., third finger, fourth finger, etc.).

[0055] However, the present disclosure is not limited thereto, and each of the first area and the second area may be preset as one of various body parts of the user.

[0056] For example, the processor 130 may identify the face of the body as the first area and the palm of the hand as the second area.

[0057] For example, the processor 130 may identify the pupil as the first area and the third finger as the second area.

[0058] Hereinafter, for convenience of explanation, it is assumed that the processor 130 identifies the wrist of the body as the first area and the fourth finger as the second area.

[0059] The processor 130 according to an embodiment may identify the first area as the reference point 1, and may move the indicator 10 to correspond to a change in the coordinates of the reference point 1.

[0060] Here, a change in the coordinates of the reference point 1 may refer to a vector value including a change in size and a change in direction. For example, a change in the coordinates of the reference point 1 may include a change in the coordinate values with reference to the x-axis and the y-axis. The present disclosure is not limited thereto, and a change in the coordinates of the reference point 1 may include a change in the coordinate values based on the x-axis, the y-axis, and the z-axis.

[0061] Referring to FIG. 3, the processor 130 may identify coordinate information (or location information) of the reference point 1 corresponding to the first area based on a frame corresponding to time t-1 among a plurality of image frames included in the image.

[0062] Subsequently, the processor 130 may identify coordinate information (or location information) of the reference point 1 corresponding to the first area based on a frame corresponding to time t among the plurality of image frames included in the image.

[0063] As shown in FIG. 3, the processor 130 may identify coordinate change A of the reference point 1 based on the coordinate information of the reference point 1 at time t-1 and the coordinate information of the reference point 1 at time t.

[0064] Subsequently, the processor 130 may move the indicator 10 by A' according to the coordinate change A of the reference point 1.

[0065] Meanwhile, in addition to the reference point 1, the processor 130 according to an embodiment may move the indicator 10 by additionally using the auxiliary point 2.

[0066] For example, the processor 130 may identify coordinate information (or location information) of the auxiliary point 2 corresponding to the second area based on a frame corresponding to time t-1 among the plurality of image frames included in the image. Subsequently, the processor 130 may identify relative coordinate information of the auxiliary point 2 with reference to the reference point 1 at time t-1. For example, the processor 130 may identify the relative coordinate information of the auxiliary point 2 with reference to the reference point 1 at time t-1 as (0, 0).

[0067] Subsequently, the processor 130 may identify coordinate information (or location information) of the auxiliary point 2 corresponding to the second area based on a frame corresponding to time t among the plurality of image frames included in the image. Subsequently, the processor 130 may identify coordinate information of the auxiliary point 2 with reference to the reference point 1 at time t as (0, 0).

[0068] For example, the processor 130 may identify relative coordinate information of the auxiliary point 2 with reference to the reference point 1 based on Equation 1 below. SubPositionDiff t = Distance BasePosition t ? SubPosition t

[0069] Here, BasePosition(t) may be the coordinate information of the reference point 1 at time t, SubPosition(t) may be the coordinate information of the auxiliary point 2 at time t, and SubPositionDiff(t) may be the relative coordinate information of the auxiliary point 2 with reference to the reference point 1 at time t.

[0070] Subsequently, the processor 130 may identify coordinate change B of the auxiliary point 2 based on the relative coordinate information of the auxiliary point 2 with reference to the reference point 1 at time t-1 and the relative coordinate information of the auxiliary point 2 with reference to the reference point 1 at time t, as shown in FIG. 3. Here, the coordinate change B of the auxiliary point 2 may correspond to the relative coordinate change of the auxiliary point 2 with reference to the reference point 1. For example, the processor 130 may identify the coordinate change B of the auxiliary point 2 by calculating SubPositionDiff(t) - SubPositionDiff(t-1).

[0071] Here, the coordinate change B of the auxiliary point 2 may refer to a vector value including a change in size and a change in direction. For example, a change in the coordinates of the auxiliary point 2 may include a change in the coordinate values with reference to the x-axis, the y-axis, and the reference point 1. The present disclosure is not limited thereto, and a change in the coordinates of the auxiliary point 2 may include a change in the coordinate values with reference to the x-axis, the y-axis, the z-axis, and the reference point 1 (0, 0, 0).

[0072] Subsequently, the processor 130 may move the indicator 10 based on the coordinate change A of the reference point 1 and the coordinate change B of the auxiliary point 2.

[0073] For example, the processor 130 may move the indicator 10 based on Equation 2 below. PointerPosition t = BasePosition t + SubPositionDiff t

[0074] Here, PointerPosition(t) may refer to the coordinate information of the indicator 10 at time t.

[0075] Thus, the processor 130 may move the indicator 10 by calculating PointerPosition(t-1) - PointerPosition(t).

[0076] Meanwhile, the processor 130 according to an embodiment may obtain the coordinate information of the indicator 10 by applying different weights to the coordinate change A of the reference point 1 and the coordinate change B of the auxiliary point 2.

[0077] In one example, the processor 130 may apply a first weight W1 to the coordinate change A of the reference point 1 and a second weight W2 to the coordinate change B of the auxiliary point 2. Subsequently, the processor 130 may obtain coordinate information of the indicator 10 by summing the coordinate change A where the first weight W1 is applied and the coordinate change B where the second weight W2 is applied, and may move the indicator 10 based on the coordinate information of the indicator 10.

[0078] For example, the processor 130 may move the indicator 10 based on Equation 3 below. PointerPosition t = BasePosition t * W 1 + SubPositionDiff t * W 2

[0079] According to an embodiment, the first weight W1 may be a value relatively greater than the second weight W2.

[0080] For example, it may be assumed that each of the first weight W1 and the second weight W2 has a value from 0 to 1, and the first weight W1 is 1 and the second weight W2 is 0.1.

[0081] In this case, the processor 130 may move the indicator 10 based on the coordinate information calculated by reflecting the coordinate change A of the reference point 1 as it is, and reflecting the coordinate change B of the auxiliary point 2 after reducing it by 1 / 10.

[0082] Meanwhile, the coordinate change B of the auxiliary point 2 is reflected after being reduced according to the second weight W2, so that the indicator 10 can be moved small (or slightly) according to the movement of the second area (e.g., the fourth finger) (or the auxiliary point 2).

[0083] However, this is only an example, and it may be assumed that the first weight W1 has a value of 0 to 10 and the second weight W2 has a value of 0 to 1, and that the first weight W1 is 5 and the second weight W2 is 0.1.

[0084] In this case, the processor 130 may move the indicator 10 based on the coordinate information calculated by reflecting the coordinate change A of the reference point 1 by magnifying it by 5 times and reflecting the coordinate change B of the auxiliary point 2 by reducing it by 1 / 10.

[0085] Meanwhile, the coordinate change A of the reference point is magnified and reflected according to the first weight W1, so that the indicator 10 can be moved large according to the movement of the first area (e.g., the wrist) (or, the reference point 1), and the coordinate change B of the auxiliary point 2 is reduced and reflected according to the second weight W2, so that the indicator 10 can be moved small (or slightly) according to the movement of the second area (e.g., the fourth finger) (or the auxiliary point 2).

[0086] A specific description of the coordinate change B of the auxiliary point 2 will be described later with reference to FIG. 5.

[0087] The processor 130 according to an embodiment applies different weights (e.g., the first weight W1 and the second weight W2) to each of the coordinate change A of the reference point and the coordinate change B of the auxiliary point 2, so that one of them is reflected as it is (or reflected after being magnified) and the other one is reflected after being reduced, thereby having the advantage of being able to move the indicator 10 more in line with the user's intent.

[0088] Here, the first weight W1 and the second weight W2 may be fixed values, or they may be changeable values.

[0089] FIG. 4 is a view provided to explain a movement of an indicator in response to a movement of a first area according to an embodiment.

[0090] When the coordinate change A of the reference point 1 is less than a threshold value for a preset period of time, the processor 130 according to an embodiment may adjust the first weight W1 downward.

[0091] Referring to FIG. 4, the processor 130 may identify the coordinate change A of the reference point 1 based on the coordinate information of the reference point 1 at time t-1 and the coordinate information of the reference point 1 at time t.

[0092] Subsequently, when the coordinate change A of the reference point 1 is less than a threshold value, the processor 130 may determine that the user does not intend to move the indicator 10 using the first area, and may adjust the first weight W1 downward (e.g., from 1 to 0.5).

[0093] For example, when the coordinate change A of the reference point 1 is less than a threshold value (e.g., 5 cm) for a preset period of time (e.g., from time t-5 to time t), the processor 130 may determine that the coordinate change A of the reference point 1 is caused by an unintended tremor of the first area, an unintended movement of the first area, or the like, and may reduce and reflect (or not reflect) the coordinate change A of the reference point 1 to move the indicator 10 more in line with the user's intent.

[0094] When the coordinate change B between the auxiliary points 2 is equal to or greater than a threshold value for a preset period of time, the processor 130 according to an embodiment may adjust the second weight W2 upward (e.g., from 0.1 to 0.5).

[0095] The processor 130 according to an embodiment may identify the coordinate change B of the auxiliary point 2 based on the relative coordinate information of the auxiliary point 2 with reference to the reference point 1 at time t-1 and the relative coordinate information of the auxiliary point 2 with reference to the reference point 1 at time t.

[0096] Subsequently, when the coordinate change B of the auxiliary point 2 is equal to or greater than a threshold value, the processor 130 may determine that the user's intent is to move the indicator 10 using the second area, and may adjust the second weight W2 upward.

[0097] For example, when the coordinate change of the auxiliary point 2 is equal to or greater than a threshold value (e.g., 5 cm) for a preset period of time (e.g., from time t-5 to time t), the processor 130 may determine that the second area has moved according to the user's primary intent and that the coordinate change of the auxiliary point 2 has occurred in response to the movement of the second area.

[0098] Accordingly, the processor 130 may magnify (or reflect as it is) the change in the coordinates of the auxiliary point 2 to move the indicator 10 more in line with the user's intent.

[0099] The specific numbers, units, etc. in the above examples are only examples provided for convenience of explanation, and the present disclosure is not limited thereto. For example, the preset period of time may mean from time t-10 to time t, and the threshold value may mean 10 cm.

[0100] FIG. 5 is a view provided to explain a movement of an indicator in response to a movement of a second area according to an embodiment.

[0101] Referring to FIG. 5, the processor 130 may identify the coordinate change B of the auxiliary point 2 based on the relative coordinate information of the auxiliary point 2 with reference to the reference point 1 at time t-1 and the relative coordinate information of the auxiliary point 2 with reference to the reference point 1 at time t.

[0102] Here, the coordinate change B of the auxiliary point 2 may correspond to the relative coordinate change of the auxiliary point 2 with reference to the reference point 1.

[0103] In one example, the processor 130 may obtain relative coordinate information of the auxiliary point 2 with reference to the reference point 1 based on coordinate information of the reference point 1 and coordinate information of the auxiliary point 2 at time t-1.

[0104] Subsequently, the processor 130 may obtain relative coordinate information of the auxiliary point 2 with reference to the reference point 1 at time t based on the coordinate information of the reference point 1 and the coordinate information of the auxiliary point 2. The processor 130 according to an embodiment may identify the coordinate change B of the auxiliary point 2 based on the relative coordinate information of the auxiliary point 2 at time t-1 and the relative coordinate information of the auxiliary point 2 at time t.

[0105] For example, when the coordinate information of the auxiliary point 2 has moved to the upper right with reference to the reference point 1 between time t-1 and time t, the processor 130 may move the indicator 10 to the upper right based on the coordinate change B of the auxiliary point 2, as shown in FIG. 5.

[0106] Meanwhile, the processor 130 may apply the second weight W2 to the coordinate change B of the auxiliary point 2 to move the indicator 10 to the upper right.

[0107] For example, when the coordinate information of the auxiliary point 2 has moved to the bottom with reference to the reference point 1 between time t-1 and time t, the processor 130 may apply the second weight W2 to the coordinate change B of the auxiliary point 2 to move the indicator 10 to the bottom.

[0108] Meanwhile, since the processor 130 applies the first weight W1 to the coordinate change A of the reference point 1 and the second weight W2 to the coordinate change B of the auxiliary point 2, the processor 130 may move the indicator 10 finely (or precisely) according to the movement of the auxiliary point 2.

[0109] For example, since the second weight W2 is a value relatively smaller than the first weight W1, the movement of the auxiliary point 2 (i.e., the coordinate change B of the auxiliary point 2) is reduced and reflected in the movement of the indicator 10, and the processor 130 may move the indicator 10 more precisely when the user moves the auxiliary point 2 than when the user moves the reference point 1.

[0110] FIG. 6 is a view provided to explain a movement of an indicator in response to a user position according to an embodiment.

[0111] The display device 100 according to an embodiment may further include memory (not shown).

[0112] According to an embodiment, the memory may store data required for various embodiments of the present disclosure. The memory may be implemented in the form of memory embedded in the display device 100, or may be implemented in the form of memory removably attached to the display device 100, depending on the purpose of storing data.

[0113] For example, in the case of data for driving the display device 100, the data may be stored in the memory embedded in the display device 100, and in the case of data for the expansion function of the display device 100, the data may be stored in the memory detachable from the display device 100. Meanwhile, The memory embedded in the display device 100 may be implemented as at least one of a volatile memory (e.g. a dynamic RAM (DRAM), a static RAM (SRAM), or a synchronous dynamic RAM (SDRAM)), or a non-volatile memory (e.g., a one-time programmable ROM (OTPROM), a programmable ROM (PROM), an erasable and programmable ROM (EPROM), an electrically erasable and programmable ROM (EEPROM), a mask ROM, a flash ROM, a flash memory (e.g. a NAND flash or a NOR flash), a hard drive, or a solid state drive (SSD)). The memory detachable from the display device 100 may be implemented in the form of a memory card (e.g., a compact flash (CF), a secure digital (SD), a micro secure digital (Micro-SD), a mini secure digital (Mini-SD), an extreme digital (xD), or a multimedia card (MMC)), an external memory connectable to a USB port (e.g., a USB memory), or the like.

[0114] In one example, the memory may store a computer program including at least one instruction or a set of instructions for controlling the display device 100.

[0115] In particular, the memory may store weight information for each position of the user.

[0116] In one example, the memory may store weight information corresponding to each of a plurality of positions, and the plurality of positions may include, for example, i) a lying position, ii) a sitting position, iii) a standing position, and the like. However, the plurality of positions may also include, but are not limited to, iv) a side lying position (recumbent position), and the like.

[0117] In one example, the processor 130 may identify a position corresponding to the user's body among the plurality of positions.

[0118] Subsequently, the processor 130 may identify weight information corresponding to the identified position.

[0119] Then, based on the identified weight information, the processor 130 may apply the first weight to the coordinate change A of the reference point 1 and the second weight to the distance change B between the auxiliary points 2.

[0120] Referring to FIG. 6, when the user's position is identified as a first position (e.g., sitting position) based on the image, the processor 130 may identify weight information corresponding to the first position.

[0121] In one example, the first weight included in the weight information corresponding to the first position may be a value relatively greater than the second weight.

[0122] For example, since it is easy for the user to move both the first area and the second area in the sitting position, the processor 130 may apply a relatively high weight (first weight W1) to the coordinate change A of the reference point 1 corresponding to the movement of the first area and a relatively low weight (second weight W2) to the coordinate change B of the auxiliary point 2 corresponding to the movement of the second area.

[0123] Meanwhile, when the user's position changes from the first position to the second position (e.g., lying position) between time t-1 and time t, the processor 130 may identify weight information corresponding to the second position.

[0124] Subsequently, the processor 130 may apply the first weight to the coordinate change A of the reference point 1 and the second weight to the coordinate change B of the auxiliary point 2 based on the weight information corresponding to the second position.

[0125] In one example, the first weight included in the weight information corresponding to the second position may be a value relatively smaller than the second weight. The embodiment thereof will be described with reference to FIG. 7.

[0126] FIG. 7 is a view provided to explain a movement of an indicator in response to a movement of a second area according to an embodiment.

[0127] For example, since it is not easy for the user to move the first area (e.g., the wrist), but it is easy for the user to move the second area (e.g., the fourth finger) in the lying position, the processor 130 may apply a relatively low weight (first weight W1) to the coordinate change A of the reference point 1 corresponding to the movement of the first area and a relatively high weight (second weight W2) to the coordinate change B of the auxiliary point 2 corresponding to the movement of the second area.

[0128] Referring to FIGS. 5 and 7, even though the coordinate change B of the auxiliary point 2 is the same, in the position assumed in FIG. 5 (e.g., standing position, sitting position, etc.), a relatively low weight is applied to the coordinate change B of the auxiliary point 2 to move the indicator 10 slightly, and in the posture assumed in FIG. 7 (e.g., lying position), a relatively high weight is applied to the coordinate change B of the auxiliary point 2 to move the indicator 10 significantly.

[0129] However, this is only an example, and the opposite of the above-described example is also possible.

[0130] For example, in the lying position, the processor 130 may apply a relatively high weight (first weight W1) to the coordinate change A of the reference point 1 corresponding to the movement in the first area, and a relatively low weight (second weight W2) to the coordinate change B of the auxiliary point 2 corresponding to the movement in the second area.

[0131] Meanwhile, in the various examples described above, time points t-1, t, and t+1 refer to one frame interval (e.g., one frame before time point t is t-1 and one frame after time point t is t+1), but are of course not limited thereto. For example, it is also possible that a preset time interval (for example, 0.5 seconds, 1 second, etc.) before time point t means t-1, and that a preset time interval after time point t means t+1.

[0132] FIG. 8 is a view provided to explain a selection command according to an embodiment.

[0133] The processor 130 according to an embodiment may identify a third area in the image.

[0134] Subsequently, when a preset action is detected in the third area, the processor 130 may obtain a selection command for the indicator 10. The processor 130 may perform an operation corresponding to the selection command (e.g., the operation of selecting a character, an item, an object, etc. corresponding to the current location of the indicator 10, a click operation).

[0135] Here, the processor 130 may identify as the third area an area in which each of the coordinate change A of the reference point 1 and the coordinate change B of the auxiliary point 2 is less than a threshold value while the preset action is being performed on the user's body.

[0136] For example, as the user performs a preset action in the third area of the user's body, causing an unintended tremor, an unintended movement, etc. in the first and second areas, a problem that the indicator 10 moves differently from the user's intent occurs.

[0137] Accordingly, the processor 130 may identify as the third area an area in which no unintended movement or the like occurs in the first and second areas while performing a preset action among various body parts of the user, that is, an area in which each of the coordinate change A of the reference point 1 and the coordinate change B of the auxiliary point 2 is less than a threshold value.

[0138] For example, referring to FIG. 8, the preset action is a joint action of two fingers, and the processor 130 may identify the first finger and the second finger as the third area.

[0139] As shown in FIG. 8, no unintended movement of the user may occur in the first area (e.g., the wrist) and the second area (e.g., the fourth finger) while the joint action of the first finger and the second finger is being performed.

[0140] As described above, the processor 130 may identify an area in which no unintended movement or the like occurs in the first and second areas while a preset action is being performed on the user's body as the third area, or may identify a preset area within the body (e.g., the first finger and the second finger) as the third area.

[0141] In the examples described above, it is assumed that the joint action of two fingers the preset action, but this is an example and the present disclosure is not limited thereto.

[0142] For example, the preset action may be a double tap action of any of the user's fingers, in which case the processor 130 may identify any of the first finger, the second finger in which no unintended movement or the like occurs in the first and second areas while a preset action is being performed on the user's body as the third area.

[0143] FIG. 9 is a flowchart provided to explain a controlling method of a display device according to an embodiment.

[0144] A method of controlling a display device, according to one embodiment of the present disclosure, first displays an indicator (S910).

[0145] Next, the user's body is recognized from the image received in real time (S920).

[0146] Next, the first area of the body is identified as the reference point, and the second area is identified as the auxiliary point (S930).

[0147] Subsequently, based on a plurality of image frames included in the image, the coordinate change of the reference point and the coordinate change of the auxiliary point are identified (S940).

[0148] The indicator is then moved based on the change in the coordinates of the reference point and the change in the coordinates of the auxiliary point (S950).

[0149] Here, the change in the coordinates of the auxiliary point corresponds to the change in the relative coordinates of the auxiliary point with reference to the reference point.

[0150] Wherein, the step S950 of moving the indicator includes applying a first weight to a coordinate change of the reference point, applying a second weight to a coordinate change of the auxiliary point, and moving the indicator by summing the coordinate change with the first weight and the coordinate change with the second weight, wherein the first weight may be a value that is relatively greater than the second weight.

[0151] The controlling method according to one embodiment of the present disclosure may further comprise the step of adjusting the first weight downwardly if the change in the coordinates of the reference point is below a threshold value for a period of time.

[0152] The controlling method according to an embodiment may further include the step of adjusting the second weight upward if the coordinate change of the auxiliary point is above a threshold value for a period of time.

[0153] The step S950 of moving the indicator according to an embodiment may include: identifying a position corresponding to the user's body; identifying weight information corresponding to the identified position; applying a first weight to a coordinate change of the reference point based on the identified weight information; applying a second weight to the coordinate change of the auxiliary point; and moving the indicator by summing the first weighted coordinate change and the second weighted coordinate change.

[0154] The step S950 of moving the indicator according to an embodiment, wherein, if the user's position is identified as a first position according to the image, applying a first weight included in the first weight information corresponding to the first position to the coordinate change of the reference point, applying a second weight included in the first weight information to the change in the coordinates of the auxiliary point, and, if the user's position changes from the first position to the second position according to the image, applying a first weight included in the second weight information corresponding to the second position to the change in the coordinates of the reference point, and applying a second weight included in the second weight information to the change in the coordinates of the auxiliary point.

[0155] The controlling method according to an embodiment may further include the steps of identifying a third area in the image, obtaining a selection command for the indicator when a preset behavior is detected in the third area, and performing an action corresponding to the selection command.

[0156] Here, the step of identifying the third area in the image may include identifying the area in which the coordinate change of the reference point and the coordinate change of the auxiliary point are each below a threshold value while the prescribed action is performed as the third area, or identifying the prescribed area within the body as the third area.

[0157] According to an embodiment, the first area may be a wrist of the user, and the second area may be any of the fingers of the user.

[0158] It should be appreciated, however, that various embodiments of the present disclosure can be applied to any type of electronic device having a display, not just the display device.

[0159] Meanwhile, the above-described various embodiments may be implemented in a recording medium that can be read by a computer or a similar device using software, hardware, or a combination thereof. In some cases, embodiments described herein may be implemented by a processor itself. According to software implementation, embodiments such as procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described in this disclosure.

[0160] Meanwhile, computer instructions for performing processing operations of the electronic device 100 according to the above-described various embodiments may be stored in a non -transitory computer-readable medium. When being executed by a processor of a specific device, the computer instructions stored in such a non-transitory computer-readable medium allows the specific device to perform processing operations of the electronic device 100 according to the above-described various embodiments.

[0161] The non-transitory computer-readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as registers, caches, memory, etc. Specific examples of the non-transitory computer-readable medium may include CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, etc.

[0162] Hereinabove, although preferred embodiments of the present disclosure have been shown and described above, the disclosure is not limited to the specific embodiments described above, and various modifications may be made by one of ordinary skill in the art without departing from the spirit of the disclosure as claimed in the claims, and such modifications are not to be understood in isolation from the technical ideas or prospect of the disclosure.

Claims

1. A display device comprising: a display; a sensor; and one or more processors configured to: control the display to display an indicator; recognize a user's body in an image received through the sensor in real time; identify a first area of the body as a reference point, and identify a second area of the body as an auxiliary point; identify a change in the coordinates of the reference point and a change in the coordinates of the auxiliary point based on a plurality of image frames included in the image; and move the indicator based on the change in the coordinates of the reference point and the change in the coordinates of the auxiliary point, wherein the change in the coordinates of the auxiliary point corresponds to a relative change in the coordinates of the auxiliary point with reference to the reference point.

2. The display device as claimed in claim 1, wherein the one or more processors are configured to: apply a first weight to the change in the coordinates of the reference point, and apply a second weight to the change in the coordinates of the auxiliary point; and move the indicator by summing the change in coordinates where the first weight is applied and the change in coordinates where the second weight is applied; and wherein the first weight is relatively greater than the second weight.

3. The display device as claimed in claim 2, wherein the one or more processors are configured to: based on the change in the coordinates of the reference point being less than a threshold value for a preset period of time, adjust the first weight downward.

4. The display device as claimed in claim 2, wherein the one or more processors are configured to: based on the change in the coordinates of the auxiliary point being equal to or greater than a threshold value for a preset period of time, adjust the second weight upward.

5. The display device as claimed in claim 1, further comprising: memory storing weight information for each user position, wherein the one or more processors are configured to: identify a position corresponding to the user's body; identify weight information corresponding to the identified position; apply a first weight to the change in the coordinates of the reference point and apply a second weight to the change in the coordinates of the auxiliary point based on the identified weight information; and move the indicator by summing the change in coordinates where the first weight is applied and the change in coordinates where the second weight is applied.

6. The display device as claimed in claim 5, wherein the one or more processors are configured to: based on the user position being identified as a first position according to the image, apply a first weight included in first weight information corresponding to the first position to the change in the coordinates of the reference point and apply a second weight included in the first weight information to the change in the coordinates of the auxiliary point; and based on the user position being changed from the first position to a second position according to the image, apply a first weight included in second weight information corresponding to the second position to the change in the coordinates of the reference point, and apply a second weight included in the second weight information to the change in the coordinates of the auxiliary point.

7. The display device as claimed in claim 1, wherein the one or more processors are configured to: identify a third area in the image; based on a preset operation being detected in the third area, obtain a selection command for the indicator; and perform an operation corresponding to the selection command.

8. The display device as claimed in claim 7, wherein the one or more processors are configured to: identify an area where each of the change in the coordinates of the reference point and the change in the coordinates of the auxiliary point is less than a threshold value while the preset operation is performed as the third area, or identify a preset area within the body as the third area.

9. The display device as claimed in claim 1, wherein the first area is the user's wrist; and wherein the second area is one of the user's fingers.

10. A controlling method of a display device, comprising; displaying an indicator; recognizing a user's body in an image received in real time; identifying a first area of the body as a reference point, and identifying a second area of the body as an auxiliary point; identifying a change in the coordinates of the reference point and a change in the coordinates of the auxiliary point based on a plurality of image frames included in the image; and moving the indicator based on the change in the coordinates of the reference point and the change in the coordinates of the auxiliary point, wherein the change in the coordinates of the auxiliary point corresponds to a relative change in the coordinates of the auxiliary point with reference to the reference point.

11. The method as claimed in claim 10, wherein the moving the indicator comprises: . applying a first weight to the change in the coordinates of the reference point, and applying a second weight to the change in the coordinates of the auxiliary point; and moving the indicator by summing the change in coordinates where the first weight is applied and the change in coordinates where the second weight is applied; and wherein the first weight is relatively greater than the second weight.

12. The method as claimed in claim 11, further comprising: based on the change in the coordinates of the reference point being less than a threshold value for a preset period of time, adjusting the first weight downward.

13. The method as claimed in claim 11, further comprising: based on the change in the coordinates of the auxiliary point being equal to or greater than a threshold value for a preset period of time, adjusting the second weight upward.

14. The method as claimed in claim 10, wherein the moving the indicator comprises: identifying a position corresponding to the user's body; identifying weight information corresponding to the identified position; applying a first weight to the change in the coordinates of the reference point and applying a second weight to the change in the coordinates of the auxiliary point based on the identified weight information; and moving the indicator by summing the change in coordinates where the first weight is applied and the change in coordinates where the second weight is applied.

15. The method as claimed in claim 14, wherein the moving the indicator comprises: based on the user position being identified as a first position according to the image, applying a first weight included in first weight information corresponding to the first position to the change in the coordinates of the reference point and applying a second weight included in the first weight information to the change in the coordinates of the auxiliary point; and based on the user position being changed from the first position to a second position according to the image, applying a first weight included in second weight information corresponding to the second position to the change in the coordinates of the reference point, and applying a second weight included in the second weight information to the change in the coordinates of the auxiliary point.

Images