Display device and control method therefor

Abstract

A display device is disclosed. One or more processors execute one or more instructions to: display a motion guide including a preset pose through a display; identify the head position of a user on the basis of an image acquired through at least one camera; identify a first distance from a display device to the head position; identify a preset point of the spine on the basis of a preset second distance downward from the head position; identify a first position of both hands of the user, corresponding to a preset pose, on the basis of distance information from at least one from among a plurality of motion controllers gripped by both hands through a communication interface; and display, through the display, a representation of both hands on the basis of the first position of both hands.

Description

Display device and control method thereof

[0001] The present disclosure relates to a display device and a control method thereof, and more particularly, to a display device for displaying a 3D stereoscopic image and a control method thereof.

[0002] Advances in electronic technology have led to the development and proliferation of various types of electronic devices. In particular, display devices, used in a variety of settings, including homes, offices, and public spaces, have been continuously evolving in recent years.

[0003] Stereoscopy refers to three-dimensional technology. Recently, commercialized 3D displays primarily utilize binocular parallax. Binocular parallax offers the advantage of creating a three-dimensional effect on a single screen, such as a TV or theater screen. Methods utilizing binocular parallax can be categorized into stereoscopic (using glasses or other auxiliary devices) and autostereocopic (glassless) methods.

[0004] Recently, commercialization of glasses-free light field displays and glasses-free 3D displays utilizing eye-tracking is being continuously researched.

[0005] A display device according to one or more embodiments of the present disclosure includes a display, a communication interface, at least one camera, a memory storing one or more commands, and one or more processors.

[0006] According to one or more embodiments, the one or more processors, by executing the one or more commands, display a motion guide including a preset pose through the display, identify a head position of the user based on an image acquired through the at least one camera, identify a first distance from the display device to the head position, identify a preset point of the spine based on a second preset distance downward from the head position, identify a first position of the two hands corresponding to the preset pose based on distance information from at least one of a plurality of motion controllers gripped by the two hands of the user through the communication interface, and display indications of the two hands through the display based on the first position of the two hands.

[0007] According to one or more embodiments, the one or more processors, by executing the one or more instructions, when the motion information of the two hands is received from the plurality of motion controllers, identify the second positions of the two hands based on the motion information, and display the marks of the two hands through the display based on the second positions of the two hands.

[0008] According to one or more embodiments, the motion information of the two hands from the plurality of motion controllers includes a movement direction and a movement distance of the two hands of the user, and the one or more processors identify a second position of the two hands based on the movement direction and the movement distance of the two hands by executing the one or more commands.

[0009] According to one or more embodiments, the one or more processors, by executing the one or more instructions, identify a first virtual space based on the head position, the first positions of the two hands, and the positions of the display device relative to the display device, identify a first coordinate corresponding to the head position and a second coordinate corresponding to the first positions of the two hands using the position of the display device as a reference point within the identified first virtual space, and display the motion information by mapping the head and the two hands to a content space displayed on the display based on the identified first coordinate and the second coordinate.

[0010] According to one or more embodiments, the content space is a second virtual space corresponding to a VR (Virtual Reality) content image, and the one or more processors, by executing the one or more commands, when a third coordinate corresponding to the second position of the two hands in the first virtual space is identified, identify a fourth coordinate in the second virtual space based on the third coordinate, and map and display the display of the two hands in the content space displayed on the display based on the identified fourth coordinate.

[0011] According to one or more embodiments, the one or more processors, by executing the one or more instructions, identify the positions of the user's two eyes based on a first captured image acquired through a depth camera, or identify the positions of the user's two eyes based on a second captured image and a third captured image acquired through a stereo camera, and identify the head position based on a center position of the two eyes.

[0012] According to one or more embodiments, the preset pose is a pose in which the user faces the front of the display while the hands are positioned on a horizontal line including the preset point of the spine, and a straight-line distance between the preset point and the user's head is perpendicular to the straight-line distance between the preset point and the display.

[0013] According to one or more embodiments, the display is a Light Field Display (LFD) display that provides 3D images.

[0014] According to one or more embodiments, the one or more processors adjust the size of the two-hand display by adjusting the preset second distance by executing the one or more commands.

[0015] According to one or more embodiments, the one or more processors adjust the size of the two-hand display based on the type of VR content by executing the one or more instructions.

[0016] According to one or more embodiments, a method for controlling a display device includes the steps of displaying a motion guide including a preset pose, identifying a head position of a user based on a captured image of the user, identifying a first distance from the display device to the head position, identifying a preset point of a spine based on a second preset distance downward from the head position, identifying a first position of both hands corresponding to the preset pose based on distance information from at least one of a plurality of motion controllers gripped by both hands of the user, and displaying indications of both hands based on the first positions of the both hands.

[0017] A non-transitory computer-readable storage medium storing computer instructions that, when executed by a processor of a display device, cause the display device to perform an operation, the operation includes: displaying a motion guide including a preset pose; identifying a head position of a user based on a captured image of the user; identifying a first distance from the display device to the head position; identifying a preset point of the spine based on a second preset distance downward from the head position; identifying a first position of the two hands corresponding to the preset pose based on distance information from at least one of a plurality of motion controllers gripped by the two hands of the user; and displaying indications of the two hands based on the first positions of the two hands.

[0018] The above-described contents, aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0019] FIG. 1 is a drawing for explaining the operation of a display device according to one or more embodiments.

[0020] FIG. 2 is a block diagram illustrating a configuration of a display device according to one or more embodiments.

[0021] FIG. 3 is a drawing for explaining a motion guide providing process of a display device according to one or more embodiments.

[0022] FIG. 4 is a diagram for explaining a head position and a first distance identification process of a display device according to one or more embodiments.

[0023] FIG. 5 is a diagram illustrating a process for identifying a preset point of a display device according to one or more embodiments.

[0024] FIG. 6 is a diagram for explaining a process for identifying the positions of two hands of a display device according to one or more embodiments.

[0025] FIG. 7 is a drawing for explaining an absolute coordinate generation process of a display device according to one or more embodiments.

[0026] FIG. 8 is a diagram illustrating a process for identifying the positions of both hands when motion information is received according to one or more embodiments.

[0027] FIG. 9 is a diagram illustrating a process for displaying a changed hand position based on motion information according to one or more embodiments.

[0028] FIG. 10 is a diagram illustrating a process for adjusting the size of a user's two hands according to one or more embodiments.

[0029] FIG. 11 is a drawing for explaining a method of controlling a display device according to one or more embodiments.

[0030] The embodiments described in this disclosure and the configurations illustrated in the drawings are merely examples of embodiments, and various modifications may be made without departing from the scope of this disclosure.

[0031] The terms used in the various embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should be defined based on the meaning of the terms and the overall content of this disclosure, rather than simply their names.

[0032] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0033] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0034] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0035] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0036] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this disclosure, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0037] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor (not shown), excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0038] In this disclosure, the term user may refer to a person using an electronic device or a device used by the person.

[0039] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0040] FIG. 1 is a drawing for explaining the operation of a display device according to one or more embodiments.

[0041] The display device (100) may be implemented as various types of display devices such as a TV, monitor, kiosk, tablet PC, electronic picture frame, mobile phone, LFD (large format display), Digital Signage (digital signage), DID (Digital Information Display), video wall, projector display, etc. However, in some cases, it may be implemented as an image processing device (e.g., set-top box, one connected box) that is connected to the display device and provides images.

[0042] The display device (100) can display a 3D stereoscopic image for a user (10) experiencing a glasses-free VR (Virtual Reality). The display device (100) can capture a user (10) through a camera (130) and display a VR content image (20) corresponding to the captured image. The VR content image (20) displayed by the display device (100) can include a content image for the user to experience and a GUI corresponding to the user's motion. For example, the GUI corresponding to the user's motion can include images of both hands (or one hand) (30-1, 30-2). The display device (100) can communicate with a plurality of motion controllers (200) gripped by both hands of the user to obtain motion information corresponding to both hands of the user, and display images (30-1, 30-2) of the user's both hands based on the obtained motion information.

[0043] Glasses-free VR refers to technology that allows users to experience virtual reality (VR) through displayed content without wearing glasses. For example, while existing VR devices used a head-mounted display (HMD), which attached the VR device to the user's head, glasses-free VR refers to technology that allows users to experience virtual reality (VR) through displayed content without wearing glasses or a device on their head.

[0044] The motion controller (200) is an electronic device for detecting the user's movements and may be implemented in a form that can be worn on a part of the user's body. For example, the motion controller (200) may be implemented in a form that can be gripped by the user's hand. However, the present invention is not limited thereto, and the motion controller (200) may also be implemented in a form that can be worn on the user's wrist or at least some of the fingers.

[0045] For example, the motion controller (200) can detect the user's movements in real time through various sensors such as an acceleration sensor, a gyro sensor, a magnetic sensor, and an optical sensor, and generate motion information. For example, the motion information may include at least one of movement distance information, movement direction information, and movement speed information. The motion information is not limited thereto, and may be referred to in various ways such as movement information, action information, and behavior information, but in the present disclosure, it is collectively referred to as motion information.

[0046] For example, the motion controller (200) can identify coordinate values ​​in a virtual space according to the user's motion. For example, the motion controller (200) can identify (x, y, z) values ​​in a virtual space through sensor values ​​of an acceleration sensor and a gyro sensor based on an arbitrary reference coordinate point. The motion controller (200) can identify coordinate values ​​in a virtual space through data measured by a plurality of sensors based on axes preset to the sensors. Accordingly, the motion controller (200) can detect user movement in a virtual space using a specific coordinate value as a reference coordinate. The motion controller (200) is not limited thereto and may be referred to in various ways, such as a motion detector, a motion interface, a motion tracker, a motion reader, etc., but will be collectively referred to as a motion controller in the present disclosure.

[0047] The VR content image (20) displayed by the display device (100) may be a 3D stereoscopic image. The display device (100) may be an LFD (Light Field Display) display device for displaying a 3D image. An LFD display device refers to a display device that generates a stereoscopic 3D image according to a user's line of sight, position, and depth. Unlike a 2D display that displays the same image regardless of the angle from which the user views the device, the LFD display device may be a 3D display that displays different images depending on the direction from which the user views the device. The LFD display device may display a 3D stereoscopic image by generating a light field expressed as a vector of light in space by a flat display and an optical element.

[0048] A light field is a concept of a light field that expresses a five-dimensional vector function that includes the direction and intensity of light propagation at all points in three-dimensional space. A light field can be implemented in various ways, such as using a diffraction grating, a method of controlling the direction and intensity of light through multiple pixels, and a method of using a pinhole or microlens array. For example, a method of using a microlens array can be a method of implementing a light field by having a series of display pixels assigned to a microlens so that the light from each pixel propagates in a specific direction only through the lens. Therefore, an LFD display device can display a three-dimensional 3D image to the user by implementing a light field.

[0049] According to one embodiment, the display device (100) can communicate with a plurality of motion controllers (200) gripped on the hands of the user (10). The display device (100) can display images of the user's two hands (30-1, 30-2) corresponding to the user's movements along with a VR content video (20) based on information received from the motion controllers (200).

[0050] Referring to FIG. 1, the display device (100) can display a 3D stereoscopic image including a VR content image (20) and images of the user's two hands (30-1, 30-2). The display device (100) can identify the user's position in the three-dimensional space where the user is located based on a captured image acquired through at least one camera (130) located at the top. For example, the display device (100) can identify the position of the user's head and the positions of both hands in the three-dimensional space based on the captured image acquired through at least one camera (130). According to one embodiment, the user's head and both hands always form a triangle, and a spine may be present at the bottom thereof. Accordingly, the position and / or size of the user's head and both hands can be reproduced in the virtual space provided by the display device (100) using only relative coordinates. Accordingly, the following describes various embodiments in which the movement of the user's two hands can be used as input control in VR content provided by the display device (100) based on information received from the motion controller (200).

[0051] FIG. 2 is a block diagram illustrating a configuration of a display device according to one or more embodiments.

[0052] According to FIG. 2, the display device (100) includes a display (110), a communication interface (120), at least one camera (130), a memory (140), and one or more processors (150). However, the present invention is not limited thereto, and the display device (100) may be implemented in a form in which some components are excluded, or may be implemented in a form in which other components are further included.

[0053] The display (110) is a configuration for providing VR content and 3D stereoscopic images of the user's two hands to the user. The display (110) may be implemented as a display including a self-luminous element or a display including a non-luminous element and a backlight. In addition, the display (110) may be implemented as an LFD display according to the above-described content. For example, the display may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc. The display (110) may also include a driving circuit, a backlight unit, etc., which may be implemented in a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc.

[0054] As an example, one or more processors (150) may provide a 3D stereoscopic image of the images of the user's two hands corresponding to the positions of the motion controller (200) through the display (110).

[0055] For example, one or more processors (150) may provide a 3D stereoscopic image of the image of the user's two hands corresponding to the position of the motion controller (200) through the display (110) based on the user's changed motion information.

[0056] The communication interface (120) includes a circuit and can communicate with a motion controller (server or user terminal). For example, one or more processors (150) can receive various data or information from a motion controller (200) connected via the communication interface (120), and can also transmit various data or information to the motion controller (200).

[0057] The communication interface (120) may include at least one of a WiFi module, a Bluetooth module, a wireless communication module, an NFC module, and a UWB module (Ultra Wide Band). At this time, the wireless communication module may perform communication according to various communication standards such as IEEE, Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), 5G (5th Generation), etc.

[0058] For example, one or more processors (150) may receive motion information from a motion controller (200) via a communication interface (120).

[0059] At least one camera (130) may be a device for generating a 3D stereoscopic image or photographing a user. The at least one camera may be implemented as various types of cameras, such as a plenoptic camera, an array camera, a stereo camera, a depth camera, an AI camera, an infrared camera, a motion camera, etc. A plenoptic camera can use a special lens array to convert a single image into images from multiple viewpoints and generate a 3D image including light direction and depth information. An array camera includes multiple lenses, captures images from different viewpoints, and combines the captured multiple images to generate a 3D image including spatial information.

[0060] In one example, one or more cameras (130) may be positioned to capture the front of the display (110). For example, one or more cameras (130) may be positioned in the central area of ​​the top bezel of the display (110).

[0061] For example, one or more cameras (130) may be positioned in a direction and angle to capture the front of the display (110). Depending on the work, the cameras (130) may be positioned in a direction and angle to be recognized as facing the front of the display (110) when the user's gaze is directed forward in the captured image. A stereo camera captures two images through two lenses and can obtain three-dimensional distance information by utilizing the gap between the lenses. The stereo camera receives an image of an object in space that is a certain distance away from the two lenses, and can obtain distance information about an object in space based on the parallax, focal length (the distance between the image plane and the lens), and baseline (the gap between the lenses) of the images input from each lens. Specifically, the stereo camera can obtain parallax information by a process of aligning two object images acquired from the two lenses based on the coordinates of a specific location, and obtain distance information about an object in space based on a preset focal length and baseline.

[0062] Depth cameras can acquire depth data by projecting laser or infrared light onto external objects, receiving the returning light with a stereo camera, and measuring the distance to the external object in three dimensions. AI cameras can identify human faces and detect movement based on artificial intelligence models.

[0063] In one example, one or more processors (150) can identify the distance from the user's head position to the display device (100) through a stereo camera.

[0064] The memory (140) can store at least one command, data, program, etc. required for the operation of the display device (100). For example, the memory (140) can store information on a motion guide including a preset pose. For example, the memory (140) can store information for identifying a user's movement space, for example, a second distance which is the distance from the user's head position to a preset point on the spine.

[0065] The memory (140) may be implemented in the form of memory embedded in the display device (100) or in the form of memory detachable from the display device (100) depending on the purpose of data storage. For example, data for driving the display device (100) may be stored in a memory embedded in the display device (100), and data for the expansion function of the display device (100) may be stored in a memory detachable from the display device (100).

[0066] In the case of memory embedded in the display device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD)).

[0067] The memory (140) may be implemented as a single memory that stores data generated from various operations according to the present disclosure, but is not limited thereto, and the memory (140) may be implemented to include multiple memories that each store different types of data or each store data generated at different stages.

[0068] One or more processors (150) control the overall operation of the display device (100). Specifically, one or more processors (150) may be connected to each component of the display device (100) to control the overall operation of the display device (100). For example, one or more processors (150) may be electrically connected to the display (110) and the memory (140) to control the overall operation of the display device (100). One or more processors (150) may be configured as one or more processors.

[0069] One or more processors (150) can perform operations of the display device (100) according to various embodiments by executing one or more commands stored in the memory (140).

[0070] The one or more processors (150) may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors (150) may control one or any combination of other components of the refrigeration device, and may perform operations related to communication or data processing. The one or more processors (150) may execute one or more programs or instructions stored in a memory. For example, the one or more processors may perform a method according to one or more embodiments of the present disclosure by executing one or more instructions stored in a memory.

[0071] When a method according to one or more embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0072] One or more processors (150) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors (150) are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure.

[0073] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0074] In the embodiments of the present disclosure, a processor may mean a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. Hereinafter, for the convenience of description, one or more processors (150) will be referred to as a processor (150).

[0075] According to one embodiment, the processor (150) may display a motion guide including a preset pose through the display (110). The motion guide refers to a guide that guides a specific action or movement to calculate absolute coordinates based on coordinate values ​​of the head and both hands. For example, the motion guide may be a guide for calculating absolute coordinates so that the positions of the user's head and both hands are located at each corner of a preset triangle. For example, the motion guide may be a guide that includes a motion in which the user's gaze looks toward the front of the display device (100) and raises both hands. The motion guide is not limited thereto and may be referred to in various ways such as a guidance guide, a movement guide, a gesture guide, etc., but will be collectively referred to as a motion guide in the present disclosure. A specific description of the motion guide will be described later with reference to FIG. 3.

[0076] According to one embodiment, the processor (150) may identify the position of the user's head based on an image acquired through at least one camera (130). Here, the head position may include the position between the user's eyes. According to one example, the processor (150) may identify the user's head position based on eye tracking that detects the user's eye movements in real time through at least one camera (130). According to one example, the processor (150) may identify the user's face in the form of a closed curve including the user's eyes, nose, and mouth through an AI camera, and identify the head position based on the identified face.

[0077] According to one embodiment, the processor (150) may identify a preset point of the spine based on a second preset distance downward from the head position. The preset point of the spine may refer to a point where a spine line extending downward from the neck of a person (hereinafter, the first line) and a horizontal line including both shoulders of the user (hereinafter, the second line) meet. Meanwhile, both end points of the second line may be where the user's both hands are positioned. In this case, the first line and the second line may meet perpendicularly, and the intersection of the first line and the second line where they meet vertically may be preset points of the spine.

[0078] The second distance may refer to the distance from the user's head position to a preset point on the spine. The second distance may be a preset value stored in the memory (140). For example, the second distance may be 12 cm, the average length of a human neck, and the value of the second distance may be changed based on user input.

[0079] According to one embodiment, the processor (150) may identify a position (hereinafter, a first position) of the user's two hands corresponding to a preset pose based on distance information received from at least one of a plurality of motion controllers gripped by the user's two hands through the communication interface (120). Here, the distance information may include a third distance between the plurality of motion controllers (200). The preset pose may include a pose in which the user's two hands are positioned on a horizontal line including a preset point of the spine and look at the front of the display (110). According to one example, the processor (150) may identify the first position of the user's two hands based on a distance value corresponding to half of the third distance.

[0080] According to one embodiment, the processor (150) may display motion information including the user's two hands through the display (110) based on the first positions of the user's two hands.

[0081] FIG. 3 is a drawing for explaining a motion guide providing process of a display device according to one or more embodiments.

[0082] According to one embodiment, the display device (100) may provide a motion guide to the user to generate absolute coordinates including the positions of the user's head and both hands. The display device (100) may display a motion guide (310, 320) including a preset pose and / or preset text through the display (110). The display device (100) may provide the motion guide (310, 320) for initializing to generate absolute coordinates including the coordinates of the user's head and both hands. Here, initializing refers to an initial setting process for setting absolute coordinates that serve as reference points for identifying the position of the motion controller (200).

[0083] Referring to FIG. 3, the display device (100) can display a preset pose image (310) in the form of a two-dimensional closed curve and preset text (320) such as "Spread your arms and look straight ahead" through the display (110). However, the preset pose image does not necessarily have to be provided in the form of a closed curve, and can be provided in various forms that can guide the user's pose, such as a three-dimensional closed curve, a two-dimensional / three-dimensional open curve, a straight line, etc.

[0084] For example, the display device (100) can capture a user (11) holding a motion controller (200) in both hands while looking at the display screen using a camera (130). When a user-captured image is acquired, the display device (100) can overlap the user-captured image with a preset pose image (310) and display them on the display (110). The display device (100) can display preset text (320) at the bottom of the display (110). However, the user-captured image and the preset pose image (310) do not necessarily need to be displayed overlappingly, and may be displayed in different screen areas. For example, the user-captured image and the preset pose image (310) may be displayed in a vertical or horizontal line.

[0085] Meanwhile, as illustrated in FIG. 3, the preset pose is not limited thereto and may include a pose with both arms raised to shoulder height. The preset text is not limited thereto and may include text corresponding to the preset pose, such as "Please raise both hands to shoulder height," or "Please look at the screen and raise your hands." Furthermore, while the preset text is indicated at the bottom, it is not limited thereto and may be displayed at any location, such as the top, left, or right.

[0086] For example, the preset pose image (310) and text (320) do not necessarily need to be provided together, and only one of the preset pose image (310) and text (320) may be provided.

[0087] According to an example, when a user (12) is identified as having changed a pose corresponding to a preset pose (310) based on a captured image acquired through a camera (130), the display device (100) may display a notification or message notifying that the initial setting is complete through at least one of the display (110) or the speaker. For example, when the user (12) is identified as having changed the pose by raising both hands to shoulder height to correspond to the preset pose or adjusting the distance from the display device to correspond to the size of the preset pose, the display device may display a notification or message notifying that the initial setting is complete.

[0088] FIG. 4 is a diagram for explaining a head position and a first distance identification process of a display device according to one or more embodiments.

[0089] According to one embodiment, the display device (100) can identify the positions of both eyes of the user (10) based on a captured image of the user (hereinafter, a first captured image) obtained through a depth camera. The display device (100) can obtain the captured image of the user during an initialization process based on a motion guide.

[0090] For example, the display device (100) can identify the position of the user's eyes from a first captured image captured by a depth camera. The display device (100) can track eye movements in real time through the camera and identify eye features such as the center of the eye, the position of the pupil, and the movement of the eyelids. The display device (100) can identify the position of the user's eyes based on the eye movements and eye features.

[0091] According to one embodiment, the display device (100) can identify the positions of the user's two eyes based on two captured images (hereinafter, a second captured image and a third captured image) acquired through a stereo camera.

[0092] According to one embodiment, the display device (100) can identify the center position of both eyes as the head position based on the positions of the user's both eyes.

[0093] Referring to FIG. 4, the display device (100) can identify the positions (410) of the user's eyes based on captured images acquired through a depth camera or a stereo camera. The display device (100) can identify the identified positions (410) of the eyes as the user's head position (430).

[0094] According to one embodiment, the display device (100) can identify a distance to the user's head (hereinafter, a first distance) (420) based on a depth camera or a stereo camera. For example, the display device (100) can identify the distance to the head (420) by detecting light that is projected onto the user's head (430) and then returned through an IR (Infrared Radiation) signal of the depth camera. For example, the display device (100) can detect light that is reflected and returned after an IR signal projected from the depth camera reaches the user's head, and identify the first distance (420) based on the time from the time of projection to the time of return.

[0095] For example, the display device (100) can identify a first distance (420) based on a plurality of captured images (hereinafter, second captured images, third captured images) acquired through a stereo camera. Specifically, the stereo camera includes two lenses (a first lens, a second lens), and each lens can be positioned at a predetermined distance (baseline) or more apart. Each lens of the stereo camera can be positioned at a predetermined distance (focal length) or more apart from an image plane on which the second captured images and the third captured images captured by each lens are formed. At this time, the second captured image captured by the first lens and the third captured image captured by the second lens can be positioned on an image plane on the rear side of the lenses. Therefore, the display device (100) can identify the first distance (420) based on the distance, focal length, and baseline distance to the second captured images and the third captured images located on the image plane.

[0096] FIG. 5 is a diagram illustrating a process for identifying a preset point of a display device according to one or more embodiments.

[0097] According to one embodiment, the display device (100) can identify a preset point of the spine based on a second distance downward from the head position. For example, the display device (100) can identify a preset point of the spine based on a second distance set to 12 cm, which is the average value of a human neck length. Referring to FIG. 5, the display device (100) can identify a preset point (510) of the spine located 12 cm downward from the head position (410).

[0098] In one embodiment, the straight-line distance between the preset point of the spine and the user's head may be perpendicular to the straight-line distance between the preset point of the spine and the display device (100).

[0099] According to one embodiment, the display device (100) can identify a distance (hereinafter, A distance) (530) from a preset point (510) of the spine to the display device (100). The display device (100) can identify the A distance (530) based on a simple mathematical formula of the first distance (420) and the second distance (520).

[0100] Referring to FIG. 5, the display device (100) can identify the intersection point where the line corresponding to the second distance and the line corresponding to distance A meet as a preset point (510) of the spine. Meanwhile, the first line corresponding to the second distance and the second line corresponding to distance A may be lines that meet perpendicularly to each other.

[0101] FIG. 6 is a diagram for explaining a process for identifying the positions of two hands of a display device according to one or more embodiments.

[0102] According to one embodiment, the display device (100) can identify the positions of the user's two hands based on the positions of the motion controllers (200). The display device (100) can communicate with a plurality of motion controllers (200) gripped by the user's two hands through a communication interface (120). The display device (100) can receive distance information between motion controllers from at least one of the plurality of motion controllers (200) through the communication interface (120).

[0103] For example, multiple motion controllers (200) can identify the position and distance between the motion controllers through UWB (Ultra-wideband) technology. UWB technology refers to a communication technology that transmits data using a wide frequency bandwidth in a very short time. For example, a motion controller (200-1) located on the left side of the user can generate a UWB signal for a very short time and transmit it to a motion controller (200-2) located on the right side. At this time, multiple motion controllers (200-1, 200-2) can identify the arrival time of the UWB signal and calculate the distance between the motion controllers (200-1, 200-2) based on the UWB signal.

[0104] For example, multiple motion controllers (200) can identify the position and distance between the motion controllers through laser tracker technology using laser beams. For example, a motion controller (200-1) located on the left can irradiate a laser beam to a motion controller (200-2) located on the right, and identify the time it takes for the reflected laser beam to return through a tracker device. At this time, the motion controller (200-1) can calculate the distance between the motion controllers (200-1, 200-2) based on the return time.

[0105] Referring to FIG. 6, the display device (100) can receive third distance information (610) between the motion controllers (200-1, 200-2) from at least one of the plurality of motion controllers (200-1, 200-2) through the communication interface (120). Based on the received third distance information (610), the display device (100) can identify the distance from a preset point (510) of the spine to each motion controller (200-1, 200-2). For example, the display device (100) can identify the B distance (620) from the preset point (510) of the spine to the motion controller (200-1) located on the left.

[0106] According to one embodiment, the display device (100) can identify the distances of each of the plurality of motion controllers (200-1, 200-2) from the display device (100). For example, the display device (100) can identify the C distance (630) based on the A distance and the B distance (620) using a simple mathematical formula.

[0107] FIG. 7 is a drawing for explaining an absolute coordinate generation process of a display device according to one or more embodiments.

[0108] According to one embodiment, the display device (100) may identify a virtual space (hereinafter referred to as a first virtual space) based on the position of the user's (10) head, the first positions of the user's two hands, and the position of the display device relative to the display device (100). The first virtual space may be a space in which the user is located, including the positions of the user's head and two hands.

[0109] According to one embodiment, the display device (100) may identify respective coordinate values ​​corresponding to the positions of the head and both hands of the user (10) based on the position of the display device (100) within the identified first virtual space. According to one example, the display device (100) may identify a first coordinate corresponding to the position of the user's head. According to one example, the display device (100) may identify a second coordinate corresponding to the first positions of the user's both hands.

[0110] Referring to FIG. 7, the display device (100) can set reference point coordinates (0, 0, 0) using the display device as a reference point. The display device (100) can identify the head coordinates of the user (10), the preset point coordinates of the spine, and / or the coordinates of both hands using the display device as a reference point.

[0111] For example, the display device (100) can identify the first coordinate (x, y, z) (720) based on the first distance, the second distance, and the A distance. For example, the display device (100) can identify the preset point coordinates (m, n, l) (750) of the spine based on the A distance and the first coordinate (720). For example, the display device (100) can identify the second coordinates (a, b, c) (730) of the left hand and the second coordinates (d, e, f) (740) of the right hand based on the A distance, the B distance, and the C distance.

[0112] According to one embodiment, the display device (100) may display motion information by mapping the head of the user (10) and both hands of the user (10) to the content space displayed on the display (110) based on the identified first coordinates (720) and second coordinates (730, 740). According to one example, the display device (100) may display a 3D stereoscopic image at a location in the virtual space corresponding to the first coordinates (720) and / or the second coordinates (730, 740) by mapping the first coordinates (720) and second coordinates (730, 740) to coordinates of a virtual space (hereinafter referred to as a second virtual space) corresponding to a VR content image. The display device (100) may input the coordinates in the first virtual space to the coordinates in the second virtual space by comparing the coordinates in the first virtual space with the coordinates in the second virtual space based on the coordinates in the second virtual space.

[0113] According to one embodiment, the display device (100) can generate absolute coordinates in a triangular shape based on the first coordinates (720) and the second coordinates (730, 740). When the display device (100) receives motion information of the user (10) through the communication interface (120), the display device (100) can identify the changed position of the user's hands based on the absolute coordinates. Here, the absolute coordinates may be fixed coordinates based on the positions of the user's (10) head and both hands within the first virtual space.

[0114] FIG. 8 is a diagram illustrating a process for identifying the positions of both hands when motion information is received according to one or more embodiments.

[0115] According to one embodiment, when the display device (100) receives motion information of the user's two hands from a plurality of motion controllers (200), the display device can identify the positions of the two hands (hereinafter referred to as second positions) based on the received motion information of the two hands. The motion information of the user's two hands refers to movement information identified by moving the two hands gripping the plurality of motion controllers (200) up and down and / or left and right. According to one example, the motion information of the user's two hands may include the movement direction and movement distance of the user's (10) two hands.

[0116] According to one embodiment, the display device (100) can identify second positions of the user's two hands based on the movement direction and movement distance of the user's two hands received from a plurality of motion controllers (200) via the communication interface (120). According to one example, the display device (100) can identify coordinates of the two hands (hereinafter referred to as third coordinates) based on the second positions.

[0117] According to one embodiment, the display device (100) can display motion information including both hands of the user (10) through the display (110) based on the second positions of the identified both hands.

[0118] Referring to FIG. 8, the display device (100) can receive motion information of the user's two hands from a plurality of motion controllers (200-3, 200-4). The display device (100) can receive movement direction and movement distance of the user's two hands from a plurality of motion controllers (200-3, 200-4). For example, the display device (100) can receive distance information of movement p to the right from the motion controller (200-3) gripped by the user's left hand. For example, the display device (100) can receive distance information of movement q upward from the motion controller (200-4) gripped by the user's right hand.

[0119] The display device (100) can receive motion information including a movement distance and a movement direction from a plurality of motion controllers (200-3, 200-4), and identify second positions of both hands of the user based on the received information. The display device (100) can identify third coordinates (810, 820) based on the generated absolute coordinates of the identified second positions. For example, the display device (100) can identify coordinates (a', b', c') (810) of the left hand gripping the motion controller (200-3). For example, the display device (100) can identify coordinates (d', e', f') (820) of the right hand gripping the motion controller (200-4).

[0120] FIG. 9 is a diagram illustrating a process for displaying a changed hand position based on motion information according to one or more embodiments.

[0121] According to one embodiment, the display device (100) can display motion information including the changed positions of both hands based on motion information received from a plurality of motion controllers (200) through the display (110).

[0122] According to one embodiment, the display device (100) can identify coordinates (hereinafter, fourth coordinates) within the second virtual space based on the third coordinates.

[0123] According to one embodiment, the display device (100) may display motion information by mapping the coordinates of both hands of the user (10) to a second virtual space displayed on the display (110) based on the identified fourth coordinates. According to one example, the display device (100) may display a 3D stereoscopic image at a location corresponding to the third coordinates (810, 820) by mapping the third coordinates (810, 820) to coordinates on the second virtual space corresponding to the VR content image.

[0124] Referring to FIG. 9, the display device (100) can display a 3D stereoscopic image at a location (920) on the second virtual space (910) corresponding to the second coordinates (730, 740) by mapping the second coordinates (730, 740) of the first virtual space to coordinates on the second virtual space. The display device (100) can display a 3D stereoscopic image at a location (930) on the second virtual space (910) corresponding to the third coordinates (810, 820) by mapping the third coordinates (810, 820) to the fourth coordinates on the second virtual space (910) based on the received motion information.

[0125] For example, the display device (100) can map the third coordinate (810, 820) to the fourth coordinate on the second virtual space based on motion information received through the communication interface (120) that moves to the right with respect to the user in the case of the left hand and upward with respect to the user in the case of the right hand.

[0126] FIG. 10 is a diagram illustrating a process for adjusting the size of a user's two hands according to one or more embodiments.

[0127] According to one embodiment, the display device (100) may adjust the size of the user's (10) hands displayed through the display (110) by adjusting the preset second distance. For example, the display device (100) may increase the size of the user's (10) hands by a ratio in which the second distance increases based on the increased second distance.

[0128] According to one embodiment, the display device (100) may adjust the second distance and / or the size of the user's hands based on the VR content type. For example, the display device (100) may adjust the second distance and / or the size of the user's hands based on a VR content image including a character with large hands or torsos.

[0129] Referring to FIG. 10, the display device (100) can adjust the size of the user's (10) hands based on a second distance according to a user input, by an increase or decrease ratio of the second distance. For example, the display device (100) can display a 3D image (1010) of the user's two hands based on a preset second distance. When the display device (100) increases the preset second distance according to the user input, the display device (100) can display a 3D image (1020) of the user's two hands that is increased by an increase ratio of the second distance.

[0130] For example, when a VR content image is changed from a general character mode to a giant character or animal character mode, the display device (100) can adjust the second distance and the size of the user's two hands based on the changed character mode.

[0131] FIG. 11 is a drawing for explaining a method of controlling a display device according to one or more embodiments.

[0132] Referring to FIG. 11, in operation 1110, the display device (100) can display a motion guide including a preset pose.

[0133] In operation 1120, the display device (100) can identify the user's head position based on the user's captured image.

[0134] In operation 1130, the display device (100) can identify a first distance from the display device (100) to the head position.

[0135] In operation 1140, the display device (100) can identify a preset point of the spine based on a second preset distance downward from the head position.

[0136] In operation 1150, the display device (100) can identify a first position of the user's two hands corresponding to a preset pose based on distance information from at least one of a plurality of motion controllers gripped by the user's two hands.

[0137] In operation 1160, the display device (100) can display motion information including both hands of the user based on the first positions of the both hands of the user.

[0138] The method for identifying the user's head position, the preset point of the spine, and the first position of both hands has been specifically described in the various embodiments described above, so a redundant description is omitted.

[0139] The control method described in FIG. 11 can be performed by a display device (100) having the configuration of FIG. 2 described above, but is not necessarily limited thereto, and can also be performed by a display device having various configurations.

[0140] The various embodiments described above may be implemented as a single embodiment, or at least one embodiment may be combined with each other in whole or in part and implemented together in one device.

[0141] According to the various embodiments described above, absolute coordinates can be generated based on the user's head position and the positions of both hands, and motion information that changes in real time can be identified based on the absolute coordinates.

[0142] Meanwhile, the various embodiments described above may be applied to a product as an embodiment alone, but at least some of the contents may be implemented in combination with other embodiments of the present disclosure.

[0143] The various embodiments described above can be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call instructions stored in the storage medium and operate according to the called instructions, and may include an electronic device (e.g., a display device (100)) according to the disclosed embodiments. When an instruction is executed by a processor, the processor can perform a function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter. The machine-readable storage medium can be provided in the form of a non-transitory computer-readable storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0144] Additionally, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product.

[0145] Specifically, a non-transitory readable storage medium or a computer program product storing computer instructions for causing the computer to perform an operation including a step of displaying a motion guide including a preset pose, a step of identifying a head position of the user based on a captured image of the user, a step of identifying a first distance from a display device to the head position, a step of identifying a preset point of the spine based on a second preset distance downward from the head position, a step of identifying a first position of the user's two hands corresponding to the preset pose based on distance information from at least one of a plurality of motion controllers gripped by the user's two hands, and a step of displaying motion information including the user's two hands based on the first position of the user's two hands may be provided.

[0146] The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0147] In addition, computer instructions or programs for performing the control method of the display device according to the various embodiments described above may be stored in a non-transitory computer-readable medium. The computer instructions stored in such a non-transitory computer-readable medium, when executed by a processor of a specific device, cause the specific device to perform processing operations in the device according to the various embodiments described above. A non-transitory computer-readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of non-transitory computer-readable media may include a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, etc.

[0148] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In the display device, display; communication interface; At least one camera; Memory for storing instructions; and One or more processors; The above instructions, when executed by the one or more processors, cause the display device to: A motion guide including a preset pose is displayed through the display, Identifying the user's head position based on an image acquired through at least one camera; Identifying a first distance from the display device to the head position, Identifying a preset point of the spine based on a second preset distance downward from the head position, Identifying a first position of the two hands corresponding to the preset pose based on distance information from at least one of a plurality of motion controllers gripped by the two hands of the user through the communication interface, A display device that displays the marks of the two hands through the display based on the first position.

2. In paragraph 1, The above instructions, when executed by the one or more processors, cause the display device to: When the motion information of the two hands is received from the plurality of motion controllers, the second position of the two hands is identified based on the motion information, A display device that displays the marks of the two hands through the display based on the second positions of the two hands.

3. In paragraph 2, The above motion information is, Including the direction and distance of movement of the two hands, The above instructions, when executed by the one or more processors, cause the display device to: A display device that identifies a second position of the two hands based on the movement direction and movement distance of the two hands.

4. In paragraph 1, The above instructions, when executed by the one or more processors, cause the display device to: Identifying a first virtual space based on the head position, the first position of both hands, and the position of the display device based on the display device, Identifying a first coordinate corresponding to the head position and a second coordinate corresponding to the first position of the two hands, using the position of the display device as a reference point within the identified first virtual space, A display device that displays the motion information by mapping the head and the two hands to a content space displayed on the display based on the identified first coordinate and the second coordinate.

5. In paragraph 4, The above content space is, It is a second virtual space corresponding to VR (Virtual Reality) content video, The above instructions, when executed by the one or more processors, cause the display device to: When a third coordinate corresponding to the second position of the two hands is identified within the first virtual space, a fourth coordinate within the second virtual space is identified based on the third coordinate, A display device that displays the marks of the two hands by mapping them to the content space displayed on the display based on the identified fourth coordinates.

6. In paragraph 1, The above instructions, when executed by the one or more processors, cause the display device to: Identifying the positions of the user's two eyes based on the first captured image acquired through the depth camera, or identifying the positions of the two eyes based on the second captured image and the third captured image acquired through the stereo camera, A display device that identifies the head position based on the center positions of the two eyes.

7. In paragraph 1, The above preset poses are, A pose in which both hands are positioned on a horizontal line including the preset points of the spine and face the front of the display, The first straight-line distance between the above-determined point and the user's head is A display device, wherein the second straight-line distance between the above-described preset point and the display is perpendicular.

8. In paragraph 7, The above display is, A display device that is a Light Field Display (LFD) display that provides 3D images.

9. In paragraph 1, The above instructions, when executed by the one or more processors, cause the display device to: A display device that adjusts the size of the two-hand display by adjusting the second distance set above.

10. In paragraph 1, The above instructions, when executed by the one or more processors, cause the display device to: A display device that adjusts the size of the two hand displays based on the VR content type.

11. In a method for controlling a display device, A step of displaying a motion guide including a preset pose; A step of identifying the user's head position based on the user's captured image; A step of identifying a first distance from the display device to the head position; A step of identifying a preset point of the spine based on a preset second distance downward from the head position; A step of identifying a first position of the two hands corresponding to the preset pose based on distance information from at least one of a plurality of motion controllers gripped by the two hands of the user; and A control method, comprising: a step of displaying indications of the two hands based on the first positions of the two hands; 12. In paragraph 11, When the motion information of the two hands is received from the plurality of motion controllers, a step of identifying the second position of the two hands based on the motion information; and A control method, comprising: a step of displaying the marks of the two hands based on the second positions of the two hands; 13. In paragraph 12, The above motion information is, Including the direction and distance of movement of the above hands, The step of identifying the second position of the two hands is: A control method further comprising: a step of identifying a second position of the two hands based on the movement direction and movement distance of the two hands.

14. In paragraph 11, The step of displaying motion information including both hands of the user is as follows: A step of identifying a first virtual space based on the head position, the first position of the two hands, and the position of the display device with respect to the display device; A step of identifying a first coordinate corresponding to the head position and a second coordinate corresponding to the first position of the two hands, using the position of the display device as a reference point within the identified first virtual space; and A control method further comprising: a step of displaying the motion information by mapping the head and the two hands to the content space based on the identified first coordinate and the second coordinate.

15. A non-transitory computer-readable storage medium storing computer instructions that, when executed by a processor of a display device, cause the display device to perform an operation, wherein the operation is: A step of displaying a motion guide including a preset pose; A step of identifying the user's head position based on the user's captured image; A step of identifying a first distance from the display device to the head position; A step of identifying a preset point of the spine based on a preset second distance downward from the head position; A step of identifying a first position of the two hands corresponding to the preset pose based on distance information from at least one of a plurality of motion controllers gripped by the two hands of the user; and A non-transitory computer-readable storage medium, comprising: a step of displaying marks of the two hands based on the first positions of the two hands;

Images