Display device and control method thereof

Abstract

A display device is disclosed. The display device includes: a memory that stores information on a reference coordinate system defined in an initial image and information on a plurality of reference points; a display; and at least one processor that controls the display to display a 3D image corresponding to an identified user face direction, wherein the at least one processor may, upon obtaining a current image in which the face and shoulders of a user are identified, track the plurality of reference points in the current image on the basis of the information on the plurality of reference points, identify coordinates of the plurality of reference points in the current image with respect to the reference coordinate system on the basis of the information on the reference coordinate system, and identify a user face direction on the basis of the coordinates of the plurality of reference points identified in the current image.

Description

Display device and control method thereof

[0001] The present disclosure relates to a display device and a control method thereof, and more particularly, to a display device for displaying 3D images and a control method thereof.

[0002] As display devices evolve, the types of images they display have diversified. Display devices capable of displaying not only 2D images but also 3D images are being developed.

[0003] To display immersive 3D images, a display device may need to accurately identify the user's gaze direction. To accurately identify the user's gaze direction, the display device may utilize technology that tracks the position and orientation of the user's face based on captured images, rather than relying on physical sensors.

[0004] A display device according to one or more embodiments of the present disclosure includes a memory that stores information about a reference coordinate system defined in an initial image and information about a plurality of reference points, a display, and at least one processor that controls the display to display a 3D image corresponding to an identified direction of a user's face, wherein the at least one processor tracks the plurality of reference points within the current image based on the information about the plurality of reference points when a current image in which a face and shoulders of a user are identified is acquired, identifies coordinates of the plurality of reference points within the current image with respect to the reference coordinate system based on the information about the reference coordinate system, and identifies the direction of the user's face based on the coordinates of the plurality of reference points identified within the current image, wherein the plurality of reference points include a reference point corresponding to a position of the user's face and a reference point corresponding to a position of the user's shoulder, and the information about the plurality of reference points may include coordinates of the plurality of reference points in an image of a previous frame and pixel values ​​of pixels corresponding to coordinates of the plurality of reference points in an image of a previous frame.

[0005] The plurality of reference points may include a first reference point corresponding to a central point of the user's face, and the at least one processor may, when an initial image in which the user's face and shoulders are identified is acquired, identify a plurality of feature points within the initial image, identify the plurality of reference points within the initial image based on positions of the plurality of feature points, and identify the reference coordinate system having the first reference point within the initial image as an origin based on positional relationships of the plurality of reference points within the initial image, and store information about the reference coordinate system in the memory, and the plurality of feature points may include a plurality of points used to recognize the user's face and shoulders.

[0006] The at least one processor can identify a circle corresponding to the user's face based on coordinates of the plurality of reference points identified within the current image, and identify a position of the circle with respect to the reference coordinate system based on information about the reference coordinate system, thereby identifying the direction of the user's face.

[0007] The plurality of reference points include a first reference point, a second reference point, and a third reference point, and the first reference point is a point corresponding to a central point of the user's face, the second reference point is a point corresponding to a left shoulder of the user, and the third reference point is a point corresponding to a right shoulder of the user, and the at least one processor, when the current image is acquired, tracks the first reference point, the second reference point, and the third reference point within the current image based on information about the plurality of reference points, identifies coordinates of the second reference point and the third reference point with respect to the reference coordinate system based on information about the reference coordinate system, and identifies a direction of the upper body of the user, and identifies a circle corresponding to the user's face based on the direction of the upper body of the user and the coordinates of the first reference point with respect to the reference coordinate system, and identifies a position of the circle with respect to the reference coordinate system to identify the direction of the user's face.

[0008] The plurality of reference points further include a fourth reference point and a fifth reference point, the fourth reference point being a point corresponding to a left jaw point of the user, the fifth reference point being a point corresponding to a right jaw point of the user, and the at least one processor, when the current image is acquired, can track the first reference point, the second reference point, the third reference point, the fourth reference point, and the fifth reference point within the current image based on information about the plurality of reference points, and can identify a circle corresponding to the face of the user based on the coordinates of the first reference point, the fourth reference point, and the fifth reference point with respect to the upper body direction of the user and the reference coordinate system.

[0009] The at least one processor may delete information about the reference coordinate system and information about the plurality of reference points from the memory if the plurality of reference points are not tracked within the image acquired for a preset number of frames, identify an image in which the user's face and shoulders are identified among images acquired after the preset number of frames as an initial image, identify the plurality of reference points and the reference coordinate system within the initial image, and store information about the identified plurality of reference points and information about the reference coordinate system in the memory.

[0010] The at least one processor may delete information about the reference coordinate system and information about the plurality of reference points from the memory at every preset number of frame intervals, identify an image in which the user's face and shoulders are identified among images acquired after the preset number of frame intervals as an initial image, identify the plurality of reference points and the reference coordinate system within the initial image, and store information about the identified plurality of reference points and information about the reference coordinate system in the memory.

[0011] The at least one processor can identify movement of the plurality of reference points based on coordinates of the plurality of reference points in the image of the previous frame and pixel values ​​of pixels corresponding to the coordinates of the plurality of reference points in the image of the previous frame when the current image is acquired, and track the plurality of reference points within the current image based on the identified movement.

[0012] The memory stores a neural network model, and the neural network model is a neural network model trained to receive information about the reference coordinate system and coordinates of the plurality of reference points with respect to the reference coordinate system, identify a circle corresponding to the user face based on the coordinates of the plurality of reference points, and output an angle of the direction of the user face based on a position of the identified circle with respect to the reference coordinate system, and the at least one processor can identify the direction of the user face based on the angle of the direction of the user face output by inputting coordinates of the plurality of reference points and information about the reference coordinate system in the current image into the neural network model.

[0013] The display device further includes a camera, and when the current image is acquired through the camera, the plurality of reference points can be tracked within the current image based on information about the plurality of reference points.

[0014] A method for controlling a display device according to one or more embodiments of the present disclosure includes, when a current image in which a user's face and shoulders are identified is acquired, a step of tracking a plurality of reference points within the current image based on information about the plurality of reference points, a step of identifying coordinates of the plurality of reference points within the current image with respect to a reference coordinate system based on information about a reference coordinate system, a step of identifying a direction of the user's face based on coordinates of the plurality of reference points identified within the current image, and a step of displaying a 3D image corresponding to the identified direction of the user's face, wherein the plurality of reference points include a reference point corresponding to a position of the user's face and a reference point corresponding to a position of the user's shoulder, and information about the plurality of reference points may include coordinates of the plurality of reference points in an image of a previous frame and pixel values ​​of pixels corresponding to coordinates of the plurality of reference points in an image of a previous frame.

[0015] The plurality of reference points may include a first reference point corresponding to a central point of the user's face, and the control method may include, when an initial image in which the user's face and shoulders are identified is acquired, a step of identifying a plurality of feature points within the initial image, a step of identifying the plurality of reference points within the initial image based on positions of the plurality of feature points, and a step of identifying the reference coordinate system having the first reference point within the initial image as an origin based on a positional relationship of the plurality of reference points within the initial image, and storing information about the reference coordinate system, wherein the plurality of feature points may include a plurality of points used to recognize the user's face and shoulders.

[0016] The step of identifying the direction of the user's face may include the step of identifying a circle corresponding to the user's face based on coordinates of the plurality of reference points identified within the current image, and the step of identifying the direction of the user's face by identifying a position of the circle with respect to the reference coordinate system based on information about the reference coordinate system.

[0017] The plurality of reference points may include a first reference point, a second reference point, and a third reference point, the first reference point being a point corresponding to a central point of the user's face, the second reference point being a point corresponding to a left shoulder of the user, and the third reference point being a point corresponding to a right shoulder of the user, and the control method may include, when the current image is acquired, a step of tracking the first reference point, the second reference point, and the third reference point within the current image based on information about the plurality of reference points, a step of identifying coordinates of the second reference point and the third reference point with respect to the reference coordinate system based on information about the reference coordinate system to identify an upper body direction of the user, a step of identifying a circle corresponding to the user's face based on the upper body direction of the user and the coordinates of the first reference point with respect to the reference coordinate system, and a step of identifying a position of the circle with respect to the reference coordinate system to identify a face direction of the user.

[0018] The plurality of reference points further include a fourth reference point and a fifth reference point, the fourth reference point being a point corresponding to a left jaw point of the user, the fifth reference point being a point corresponding to a right jaw point of the user, and the control method may include, when the current image is acquired, a step of tracking the first reference point, the second reference point, the third reference point, the fourth reference point, and the fifth reference point within the current image based on information about the plurality of reference points, and a step of identifying a circle corresponding to the face of the user based on coordinates of the first reference point, the fourth reference point, and the fifth reference point with respect to the upper body direction of the user and the reference coordinate system.

[0019] The control method may include a step of deleting information about the reference coordinate system and information about the plurality of reference points if the plurality of reference points are not tracked within an image acquired for a preset number of frames, a step of identifying an image in which the face and shoulders of the user are identified among images acquired after the preset number of frames as an initial image, and a step of identifying the plurality of reference points and the reference coordinate system within the initial image and storing information about the identified plurality of reference points and information about the reference coordinate system.

[0020] The control method may include a step of deleting information about the reference coordinate system and information about the plurality of reference points at every preset number of frame intervals, a step of identifying an image in which the user's face and shoulders are identified among images acquired after the preset number of frame intervals as an initial image, and a step of identifying the plurality of reference points and the reference coordinate system within the initial image and storing information about the identified plurality of reference points and information about the reference coordinate system.

[0021] The step of tracking the plurality of reference points may include, when the current image is acquired, identifying movement of the plurality of reference points based on coordinates of the plurality of reference points in the image of the previous frame and pixel values ​​of pixels corresponding to the coordinates of the plurality of reference points in the image of the previous frame, and tracking the plurality of reference points within the current image based on the identified movement.

[0022] The control method includes a step of identifying the direction of the user's face based on an angle of the direction of the user's face output by inputting coordinates of the plurality of reference points and information about the reference coordinate system in the current image into a neural network model, and the neural network model may be a neural network model trained to receive information about the reference coordinate system and coordinates of the plurality of reference points with respect to the reference coordinate system, identify a circle corresponding to the user's face based on the coordinates of the plurality of reference points, and output an angle of the direction of the user's face based on a position of the identified circle with respect to the reference coordinate system.

[0023] A computer-readable recording medium including a program for executing a method for controlling a display device according to at least one embodiment of the present disclosure, wherein the method for controlling a display device includes, when an image in which a user's face is identified is acquired, a step of identifying a plurality of reference points within the image based on information about the plurality of reference points, a step of identifying coordinates of the identified plurality of reference points with respect to the reference coordinate system based on information about the reference coordinate system, and a step of identifying a direction of the user's face based on the coordinates of the plurality of reference points, wherein the plurality of reference points include a reference point corresponding to a position of the user's face and a reference point corresponding to a position of the user's shoulder, and the information about the plurality of reference points may include coordinates of the plurality of reference points in an image of a previous frame and pixel values ​​of pixels corresponding to coordinates of the plurality of reference points in an image of a previous frame.

[0024] FIG. 1 is a drawing for explaining the operation of a display device according to one or more embodiments of the present disclosure.

[0025] FIG. 2 is a block diagram illustrating a configuration of a display device according to one or more embodiments of the present disclosure.

[0026] FIG. 3 is a detailed block diagram illustrating a configuration of a display device according to one or more embodiments of the present disclosure.

[0027] FIG. 4 is a drawing for explaining a method for setting a reference coordinate system of a display device according to one or more embodiments of the present disclosure.

[0028] FIG. 5 is a drawing for explaining a method for identifying a user's face direction of a display device according to one or more embodiments of the present disclosure.

[0029] FIG. 6 is a drawing for explaining a method for identifying a user's face direction of a display device according to one or more embodiments of the present disclosure.

[0030] FIG. 7 is a diagram illustrating a method for identifying a face direction based on a neural network model according to one or more embodiments of the present disclosure.

[0031] FIG. 8 is a flowchart illustrating a method for displaying a 3D image on a display device according to one or more embodiments of the present disclosure.

[0032] FIG. 9 is a flowchart illustrating a method for storing information on a reference coordinate system of a display device according to one or more embodiments of the present disclosure.

[0033] The present embodiments may be modified and have various embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, but should be understood to encompass various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0034] In describing the present disclosure, if it is determined that a specific description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, a detailed description thereof will be omitted.

[0035] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concepts of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to further faithfully and completely convey the technical concepts of the present disclosure to those skilled in the art.

[0036] The terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the scope of the rights. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0037] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0038] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0039] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0040] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that said component may be directly coupled to said other component, or may be coupled via another component (e.g., a third component).

[0041] On the other hand, when it is said that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it can be understood that no other component (e.g., a third component) exists between said component and said other component.

[0042] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0043] Instead, in some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0044] In the embodiments, a 'module' or 'part' performs at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of 'modules' or 'parts' may be integrated into at least one module and implemented as at least one processor, except for a 'module' or 'part' that needs to be implemented as a specific hardware.

[0045] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0046] Hereinafter, with reference to the attached drawings, embodiments according to the present disclosure will be described in detail so that a person having ordinary knowledge in the technical field to which the present disclosure pertains can easily implement the present disclosure.

[0047] FIG. 1 is a drawing for explaining the operation of a display device according to one or more embodiments of the present disclosure.

[0048] According to FIG. 1, the display device (100) can display a 3D image by providing light to the user based on the position and direction of the identified user's face.

[0049] According to one embodiment, the display device (100) can provide a first light (201) to a first user (301) looking in a first direction at a first location, and can provide a second light (202) to a second user (302) looking in a second direction at a second location.

[0050] For example, let's assume that a display device (100) is displaying a car. A first user (301) facing a first direction from a first position receives a first light (201), and thus can view an image including only the right side and front of the car. On the other hand, a second user (302) facing a second direction from a second position receives a second light (202), and thus can view an image including only the left side and front of the car.

[0051] Here, the first position may correspond to a “position moved to the right from the front of the display device (100)”, and the first direction may correspond to a “direction rotated clockwise by an angle greater than 0 degrees and less than 45 degrees around the direction in which the user looks at the display device (100) as the front axis.”

[0052] Additionally, the second position may correspond to a “position moved to the left from the front of the display device (100)”, and the second direction may correspond to a “direction rotated counterclockwise by an angle greater than 0 degrees and less than 45 degrees around the direction in which the user looks at the display device (100) as the front axis.”

[0053] As illustrated in FIG. 1, the display device (100) can acquire an image of a user through a camera (140) and identify the position and direction of the user's face based on the acquired image. However, the position of the camera (140) illustrated in FIG. 1 is merely exemplary, and the camera (140) may be located on the side or bottom of the display device (100), or may be located at a position spaced apart from the display device (100). Furthermore, the display device (100) may be connected to an external camera device through an interface.

[0054] As illustrated in FIG. 1, the display device (100) may be implemented as a light field display device, but this is only one example, and the display device (100) according to the present disclosure may be implemented as various devices capable of displaying 3D images, such as a head mounted display device and a hologram projector.

[0055] For example, the display device (100) according to the present disclosure may be implemented as a head-up display apparatus. When the display device (100) is implemented as a head-up display apparatus, the display device (100) may provide different 3D images related to the driver's driving according to the position and direction of the driver's face. In addition, the display device (100) may also determine the position at which to display the 3D images related to the driver's driving based on the position and direction of the driver's face.

[0056] As another example, the display device (100) according to the present disclosure may be implemented as a hologram projector. When the display device (100) is implemented as a hologram projector, the display device (100) can project different 3D hologram images depending on the position and direction of the user's face. In addition, the display device (100) can determine a space in which to project the 3D hologram image based on the position and direction of the driver's face.

[0057] As described above, the display device (100) can provide different images depending on the position and direction of the user's face, thereby allowing the user to experience a three-dimensional effect of objects included in the image. In the description of FIGS. 2 and 3 described below, the configuration of the display device (100) will be described along with a specific operating method of the display device (100).

[0058] FIG. 2 is a block diagram illustrating a configuration of a display device according to one or more embodiments of the present disclosure.

[0059] According to FIG. 2, the display device (100) may include a display (110), a memory (120), and at least one processor (130).

[0060] The display (110) is configured to display a 3D image. The display (110) may include a backlight, a base panel, and an optical layer.

[0061] In one embodiment, the backlight may generate light and provide it to the user. In this case, the light generated by the backlight may be white light.

[0062] In one embodiment, the base panel may include a plurality of pixels. If the base panel is a liquid crystal display, the base panel may be a filter layer including a plurality of color filters. In one embodiment, each of the plurality of pixels may correspond to a color filter.

[0063] In one embodiment, light provided by a backlight can pass through a base panel and reach an optical layer. The optical layer can refract, reflect, or disperse the light passing through the base panel. Specifically, the light passing through the base panel can be directed in a specific direction by the optical layer. The optical layer can include a configuration such as a microlens array (MLA), a lenticular lens, or a diffraction grating array.

[0064] Although the above description describes the display (110) as including a backlight, this is merely an example and the display (110) may not include a backlight. If the display (110) does not include a backlight, the base panel may be implemented as a panel including a self-light-emitting element.

[0065] The memory (120) can store data necessary for implementing various embodiments of the display device (100) according to one or more embodiments of the present disclosure. The memory (120) can store information about a reference coordinate system defined in an initial image and information about a plurality of reference points.

[0066] The memory (120) may be implemented in the form of memory embedded in the display device (100) or may be implemented in the form of memory that can be attached or detached from the display device (100) depending on the purpose of data storage. For example, data for driving the display device (100) may be stored in a memory embedded in the display device (100), and data for the expansion function of the display device (100) may be stored in a memory that can be attached or detached from the display device (100).

[0067] Meanwhile, the memory embedded in the display device (100) may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD)).

[0068] In addition, in the case of a memory that can be attached or detached to the display device (100), it can be implemented in the form of a memory card (e.g., CF (compact flash), SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.), an external memory that can be connected to a USB port (e.g., USB memory), etc.

[0069] The memory (120) may include various instructions necessary for the operation of at least one processor (130). Here, the instructions may include instructions for performing tracking of a plurality of reference points, instructions for performing identification of coordinates of a plurality of reference points, instructions for performing identification of a user's face direction, instructions for identifying a plurality of feature points in an initial image, instructions for identifying a plurality of reference points based on the positions of a plurality of feature points, etc.

[0070] In the present disclosure, the term "initial image" may refer to an image used by the display device (100) to identify multiple reference points and reference coordinate systems from the user's face and shoulders. For example, the "initial image" may include an image in which the user's face and shoulders are displayed frontally. The term "initial image" does not necessarily refer only to an image in which the user's face and shoulders are first recognized. If the user's face and shoulders are clearly displayed and multiple reference points and reference coordinate systems are clearly identified, even an image acquired later in time by the display device (100) may be identified as an "initial image."

[0071] Here, the term "initial image" may be replaced by various expressions representing the same or similar concepts. For example, it may be replaced by various terms such as "reference image," "original image," "first image," and "base image." In this disclosure, the term "initial image" will be used interchangeably.

[0072] In the present disclosure, the term "multiple feature points" may refer to characteristic points used by the display device (100) to identify the user's face and shoulders, and may include multiple points corresponding to each of multiple components included in the user's face and shoulders. For example, the "multiple feature points" may include eyes, nose, mouth, ears, chin, facial borders, etc. included in the user's face, and may include corner points, borders, etc. of the user's shoulders.

[0073] Here, the term "multiple feature points" may be replaced by various expressions representing the same or similar concepts. For example, it may be replaced by various terms such as "multiple landmarks," "multiple feature elements," "multiple marks," and "multiple singular points." In this disclosure, the term "multiple feature points" will be used interchangeably.

[0074] In the present disclosure, "multiple reference points" may refer to points that the display device (100) tracks to identify the position and direction of the user's face. For example, "multiple reference points" may include a point corresponding to the center point of the user's face, a point corresponding to the user's left shoulder, a point corresponding to the user's right shoulder, a point corresponding to the user's left chin, a point corresponding to the user's right chin, and the like.

[0075] Here, the term "multiple reference points" may be replaced by various expressions representing the same or similar concepts. For example, it may be replaced by various terms such as "multiple markers," "multiple comparison points," "multiple standard points," and "multiple comparison reference points." In this disclosure, the term "multiple reference points" will be used interchangeably.

[0076] In the present disclosure, the term "reference coordinate system" may refer to a coordinate system used by the display device (100) to identify the six-axis coordinate values ​​of the user's face. Here, the six-axis coordinate values ​​may include three-axis coordinate values ​​(x, y, z) indicating a position and three-axis rotation values ​​(yaw, pitch, roll) indicating a direction. For example, the "reference coordinate system" may include a 3D coordinate system defined with one of the user's multiple reference points as the origin.

[0077] Here, the term "reference coordinate system" may be replaced by various expressions representing the same or similar concepts. For example, it may be replaced by various terms such as "reference system," "standard coordinate system," "world coordinate system," and "reference frame." In this disclosure, the term "reference coordinate system" will be used interchangeably.

[0078] At least one processor (130) controls the overall operation of the display device (100). Specifically, at least one processor (130) is connected to the display (110) and the memory (120), and can control the overall operation of the display device (100) by executing at least one instruction stored in the memory (120).

[0079] At least one processor (130) may be implemented as a digital signal processor (DSP) that processes digital signals, a microprocessor, but is not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), an ARM processor, and an artificial intelligence (AI) processor, or may be defined by the relevant terms. In addition, the processor (130) may be implemented as a system on chip (SoC) having a built-in processing algorithm, a large scale integration (LSI), or may be implemented in the form of a field programmable gate array (FPGA). At least one processor (130) may perform various functions by executing computer executable instructions stored in the memory (120).

[0080] When a method according to various embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. At least one processor (130) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). Hereinafter, for convenience of explanation, at least one processor (130) will be referred to as a processor (130).

[0081] When a current image in which the user's face and shoulders are identified is acquired, the processor (130) can track multiple reference points within the current image based on information about the multiple reference points.

[0082] "Information on a plurality of reference points" may refer to information used by the display device (100) to track a plurality of reference points within images continuously acquired. For example, "information on a plurality of reference points" may include "coordinates of the plurality of reference points in an image of a previous frame" and "pixel values ​​of pixels corresponding to the coordinates of the plurality of reference points in an image of a previous frame."

[0083] A specific method for the display device (100) to track multiple reference points will be described in detail in the description of FIGS. 4 to 6 below.

[0084] The display device (100) of the present disclosure can identify the position and direction of a user's face and display a 3D image corresponding to the identified position and direction of the user's face by including the configuration illustrated in FIG. 2. Specifically, the display device (100) tracks only a plurality of reference points based on a reference coordinate system identified in an initial image and estimates the direction of the user's face, thereby eliminating the need to identify a plurality of feature points (e.g., eyes, nose, mouth, etc.) for each continuously acquired image, thereby enabling the direction of the user's face to be identified more quickly and accurately than a conventional face direction estimation method.

[0085] However, the display device (100) of the present disclosure may further include various configurations in addition to the display (110) and memory (120) illustrated in FIG. 2. The description of FIG. 3 described below will describe in detail various configurations that the display device (100) may include.

[0086] FIG. 3 is a detailed block diagram illustrating a configuration of a display device according to one or more embodiments of the present disclosure.

[0087] According to FIG. 3, the display device (100) may further include a display (110), a memory (120), a processor (130), a camera (140), and an interface (150).

[0088] The display (110) and memory (120) that can be included in the display device (100) have been described in detail in the description of FIG. 2 described above, and in the description of FIG. 3, any description that overlaps with the description of FIG. 2 will be omitted.

[0089] The camera (140) is configured to capture the user's face and shoulders.

[0090] For example, the camera (140) may be implemented as a stereo camera capable of simultaneously acquiring two images through two photographing lenses spaced a certain distance apart. When the camera (140) is implemented as a stereo camera, the display device (100) may acquire depth information for an object commonly included in the two images based on a triangulation method for the two images.

[0091] As another example, the camera (140) may be implemented as an RGB-D camera. If the camera is implemented as an RGB-D camera, the display device (100) may also obtain depth information of an object included in the image based on an image obtained through the RGB-D camera.

[0092] The description of the above-described camera (140) is only one example, and the camera (140) may be implemented with various cameras capable of acquiring depth information of an object, such as two or more RGB cameras and / or RGB-D cameras.

[0093] The interface (150) may include a communication interface, a manipulation interface, an input / output interface, and the like. For example, the communication interface is a configuration for performing communication with at least one external device. The communication interface may include at least one wireless communication module, at least one wired communication module, and the like. Each communication module may be implemented in the form of at least one hardware chip. The wireless communication module may include at least one module among a Wi-Fi module, a Bluetooth module, an infrared communication module, or other communication modules. In addition, the communication interface may include at least one communication chip that performs communication according to various wireless communication standards such as Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), and the like. The wired communication module may include, for example, at least one among a LAN (Local Area Network) module, an Ethernet module, a paired cable, a coaxial cable, a fiber optic cable, or a UWB (Ultra Wide-Band) module. The communication interface is implemented in various forms like this, and by performing communication with an external device, it can receive images of the user's face and shoulders captured from the external device.

[0094] The operation interface is a configuration for receiving user operation input. The operation interface may include various buttons, a touch screen, etc. provided on the main body of the display device (100).

[0095] The input / output interface is a configuration for inputting and outputting various external signals. The input / output interface can be connected to various external memories or external sources (e.g., web servers, user terminal devices, etc.) and can input various data. The input / output interface can be implemented as at least one interface among HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), USB C-type, DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (Dsubminiature), and DVI (Digital Visual Interface). At least some of the input / output interfaces may be connected to communication interfaces. For example, the input / output interface can transmit information received from an external device to the communication interface or transmit information received through the communication interface to the external device.

[0096] The interface (150) can be connected to a camera or an external device having a built-in camera.

[0097] For example, when a camera is connected through a USB port among the interfaces (150), the display device (100) can obtain an image captured by the user through the camera.

[0098] Specifically, the camera can capture an image incident at the time of photographing the user using an image sensor and store the captured photographed image data. The display device (100) can obtain an image corresponding to the user through the captured photographed image data and identify and / or track a plurality of reference points based on the image corresponding to the user.

[0099] The above description describes a case where an external camera is connected through an interface (150), but it goes without saying that, depending on the embodiment, the display device (100) may have a built-in camera.

[0100] The configurations illustrated in FIGS. 2 and 3 are merely exemplary, and it is to be understood that new configurations may be added or some configurations may be omitted in addition to the configurations illustrated in FIGS. 2 and 3 when implementing the present disclosure.

[0101] In the description of FIGS. 4 to 6 described below, a specific method by which the display device (100) identifies the position and direction of the user's face will be described.

[0102] FIG. 4 is a drawing for explaining a method for setting a reference coordinate system of a display device according to one or more embodiments of the present disclosure.

[0103] According to FIG. 4, the display device (100) can identify a reference coordinate system (420) and a plurality of reference points from an initial image (410).

[0104] When an initial image (410) in which a user's face and shoulders are identified is acquired, the display device (100) can identify a plurality of feature points within the initial image (410), and the display device (100) can identify a plurality of reference points within the initial image (410) based on the positions of the plurality of feature points.

[0105] According to one embodiment, the plurality of feature points may include a plurality of points corresponding to the user's eyes, nose, lips, jawline, and shoulder line. In this case, when the initial image (410) is acquired, the display device (100) may detect the presence of the user's face and shoulders within the initial image (410), extract feature points corresponding to the positions of the eyes, nose, lips, and jawline included in the user's face, and extract feature points corresponding to the position of the shoulder line included in the user's shoulder.

[0106] Specifically, the display device (100) can identify a plurality of feature points corresponding to the positions of the user's face and shoulders in the image based on various methods such as CNN (Convolutional Neural Network), Haar Cascasde, and HOG (Histogram of Oriented Gradients).

[0107] According to one embodiment, the display device (100) may identify a plurality of feature points corresponding to the positions of the user's face and shoulders based on the Haar Cascade method. In this case, the display device (100) may utilize a pre-trained classifier to identify feature points such as the user's eyes, nose, and lips. The classifiers used in the Haar Cascade method may be classifiers trained to classify feature points based on differences in pixel values ​​of adjacent pixels.

[0108] When the display device (100) identifies a plurality of feature points corresponding to the positions of the user's face and shoulders from the initial image (410), a plurality of reference points can be identified within the initial image (410) based on the positions of the plurality of feature points.

[0109] According to one embodiment, the display device (100) can identify the shape of the user's face and shoulders based on the positions of a plurality of feature points, and can identify a point corresponding to the center of the user's face as a first reference point. The display device (100) can identify a point corresponding to the user's left shoulder as a second reference point, and a point corresponding to the user's right shoulder as a third reference point. The display device (100) can also identify a point corresponding to the user's left chin as a fourth reference point, and a point corresponding to the user's right chin as a fifth reference point.

[0110] According to the above description, the display device (100) identifies points corresponding to the center point of the user's face, the left and right jaw points, and the left and right shoulders as multiple reference points, but this is only one example, and any point included in the user's face may be identified as a reference point, and one of any points included in the user's face and shoulders may be identified as a reference point. In the present disclosure, for convenience of explanation, the first reference point is referred to as a point corresponding to the center point of the user's face, the second reference point is referred to as a point corresponding to the user's left shoulder, the third reference point is referred to as a point corresponding to the user's right shoulder, the fourth reference point is referred to as a point corresponding to the user's left jaw, and the fifth reference point is referred to as a point corresponding to the user's right jaw.

[0111] When a plurality of reference points corresponding to the positions of the user's face and shoulders are identified, the display device (100) can identify a reference coordinate system (420) with the first reference point in the initial image (410) as its origin based on the positional relationship of the plurality of reference points in the initial image (410). In addition, the display device (100) can store information about the reference coordinate system (420) in the memory (120).

[0112] According to one embodiment, the display device (100) may convert the coordinates of each pixel corresponding to the first reference point, the second reference point, and the third reference point included in the image into coordinate values ​​in a world coordinate system having an arbitrary point in space as an origin based on extrinsic parameters and depth information of the camera. After the display device (100) obtains the coordinate values ​​of the first reference point, the second reference point, and the third reference point in the world coordinate system, the display device (100) may create a new coordinate system using the coordinate values ​​of the first reference point, the second reference point, and the third reference point, and may define the new coordinate system as the reference coordinate system (420).

[0113] According to one embodiment, the display device (100) may define a plane corresponding to the shape of the user's face and set the normal vector of the plane corresponding to the shape of the user's face as a coordinate axis of the reference coordinate system (420). Specifically, the display device (100) may define a plane including a first vector connecting a first reference point and a second reference point and a second vector connecting the first reference point and a third reference point, and then generate a normal vector of the plane through an outer product of the first vector and the second vector. In this case, the display device (100) may set the normal vector generated through the outer product as a coordinate axis of the reference coordinate system (420).

[0114] According to another embodiment, the display device (100) may convert the coordinates of each pixel corresponding to the first reference point, the fourth reference point, and the fifth reference point included in the image into coordinate values ​​in the global coordinate system. After the display device (100) obtains the coordinate values ​​of the first reference point, the fourth reference point, and the fifth reference point in the global coordinate system, the display device (100) may create a new coordinate system using the coordinate values ​​of the first reference point, the fourth reference point, and the fifth reference point in the global coordinate system, and may define the new coordinate system as the reference coordinate system (420). In this case as well, the display device (100) may define a plane including a first vector connecting the first reference point and the fourth reference point and a second vector connecting the first reference point and the fifth reference point as a plane corresponding to the user's face, and then set the normal vector of the defined plane as a coordinate axis of the reference coordinate system (420).

[0115] In the above description, only an example in which the display device (100) defines a plane corresponding to the user's face using three reference points and creates a reference coordinate system (420) has been described. However, it is of course possible for the display device (100) to create a reference coordinate system (420) based on three or more reference points.

[0116] In addition, although the above description explains that the display device (100) can generate the reference coordinate system (420) based only on the position of the reference point, this is only one example, and it is of course possible for the display device (100) to set the reference coordinate system based on a plurality of feature points. For example, the display device (100) can define a plane corresponding to the user's face based on the positions of a plurality of feature points corresponding to the user's jawline, and then generate the reference coordinate system (420).

[0117] After the display device (100) identifies a plurality of reference points and a reference coordinate system (420) from the initial image (410), information about the plurality of reference points and information about the reference coordinate system can be stored in the memory (120).

[0118] Here, "information about the reference coordinate system" may mean information about the location of the origin of the reference coordinate system and the directions of the x-axis, y-axis, and z-axis. For example, "information about the reference coordinate system" may include location information of a pixel corresponding to the location of the origin of the reference coordinate system and direction information of a unit vector of each coordinate axis.

[0119] The display device (100) according to the present disclosure identifies the position and direction of the user's face in images that are continuously acquired based on the reference coordinate system (420) defined in the initial image (410) in the manner described above, and can identify the direction of the user's face more quickly and accurately than a conventional user's face direction estimation method.

[0120] In the description of FIG. 5 described below, a method for the display device (100) to track multiple reference points will be described.

[0121] FIG. 5 is a drawing for explaining a method for identifying a user's face direction of a display device according to one or more embodiments of the present disclosure.

[0122] According to FIG. 5, when a current image (520) in which a user's face and shoulders are identified is acquired, the display device (100) can track a plurality of reference points within the current image based on information about a plurality of reference points, and identify the direction of the user's face based on the coordinates of the plurality of tracked reference points.

[0123] According to one embodiment, the display device (100) may track a plurality of reference points using a method of tracking the movement of pixels based on optical flow, such as the Lucas-Kanande algorithm. In addition, the display device (100) may also track a plurality of reference points using a deep learning model, such as FlowNet or LightFlowNet, and the method of tracking a plurality of reference points is not limited to the above-described method.

[0124] Specifically, the display device (100) can set a plurality of windows including pixels corresponding to the coordinates of each of the plurality of reference points and surrounding pixels based on the coordinates of the plurality of reference points in the image (510) of the previous frame. The display device (100) can estimate the motion vector of the window within the current image (520) based on the pixel values ​​of the pixels corresponding to the coordinates of the plurality of reference points in the image (510) of the previous frame and the pixel values ​​of the surrounding pixels included in the window. At this time, the display device (100) can use the assumption that the movement of all pixels included in the window is constant to estimate the motion vector. The display device (100) can track the plurality of reference points within the current image based on the estimated motion vector of the window.

[0125] The display device (100) can identify coordinates of a plurality of reference points within the current image (520) with respect to the reference coordinate system (420) based on information about the reference coordinate system (420) defined in the initial image. According to one embodiment, the display device (100) can identify coordinates of a first reference point, a second reference point, a third reference point, a fourth reference point, and a fifth reference point in the reference coordinate system (420).

[0126] The display device (100) can identify a plane corresponding to the user's face based on the coordinates of the first reference point, the second reference point, the third reference point, the fourth reference point, and the fifth reference point in the reference coordinate system (420), and can create an object coordinate system (530) with the plane corresponding to the user's face as a coordinate plane and the first reference point in the current image as the origin based on the same method as the method in which the display device (100) defined the reference coordinate system (420) in the initial image.

[0127] The display device (100) can identify the angle of the direction of the user's face by comparing the coordinate axes of the reference coordinate system (420) with the coordinate axes of the object coordinate system (530). For example, the display device (100) can identify the angle of the direction of the user's face by identifying how much the x-axis of the object coordinate system (530) is rotated clockwise or counterclockwise compared to the x-axis of the reference coordinate system (420). Specifically, the display device (100) can calculate the angle by which the coordinate axes of the object coordinate system (530) are rotated compared to the coordinate axes of the reference coordinate system (420) as yaw, roll, and pitch values.

[0128] In the above description, only the method of identifying the direction and position of the user's face by generating an object coordinate system (530) based on the coordinates of a plurality of reference points and comparing the object coordinate system (530) with the reference coordinate system (420) has been described, but this is only one example, and the display device (100) can identify the direction of the user's face based on various methods, such as identifying a circle corresponding to the user's face based on the coordinates of a plurality of reference points and identifying the direction of the user's face based on the identified circle. In the description of FIG. 6 described below, a method of identifying the direction of the user's face by identifying a circle corresponding to the user's face based on the coordinates of a plurality of reference points will be described.

[0129] FIG. 6 is a drawing for explaining a method for identifying a user's face direction of a display device according to one or more embodiments of the present disclosure.

[0130] According to FIG. 6, the display device (100) can identify a circle (620, 640) corresponding to the user's face.

[0131] The display device (100) can identify a circle corresponding to the user's face based on the coordinates of a plurality of reference points identified within the current image, and can identify the direction of the user's face by identifying the position of the circle with respect to the reference coordinate system based on information about the reference coordinate system.

[0132] According to one embodiment, the display device (100) can track a first reference point, a second reference point, and a third reference point within a current image based on information about a plurality of reference points, and can identify the coordinates of the second reference point and the third reference point with respect to the reference coordinate system based on information about the reference coordinate system to identify the upward direction of the user.

[0133] Specifically, the display device (100) can identify the direction of the user's upper body based on the distance between the second reference point and the third reference point in the current image and the change in the coordinate values ​​of the second reference point and the third reference point. For example, the display device (100) can compare the distance between the second reference point and the third reference point in the current image with the distance between the second reference point and the third reference point in the initial image that captures the front view of the user, and can identify that the upper body has rotated a lot as the distance between the second reference point and the third reference point in the current image becomes shorter. The display device (100) can identify that the upper body is facing the front if the distance between the second reference point and the third reference point in the current image is the same as the distance between the second reference point and the third reference point in the initial image.

[0134] Additionally, the display device (100) can identify that the direction of the upper body is facing right when the position of the second reference point moves to the right, and can identify that the direction of the upper body is facing left when the position of the third reference point moves to the left.

[0135] As described above, the display device (100) can identify the direction of the user's upper body by comparing the coordinates of the second and third reference points identified in the initial image in which the user's frontal view is captured with the coordinates of the second and third reference points identified in the current image. The display device (100) can identify a circle corresponding to the user's face using the identified upper body direction of the user based on the method described above.

[0136] The display device (100) can identify a circle corresponding to the user's face based on the direction of the user's upper body and the coordinates of the first reference point with respect to the reference coordinate system.

[0137] According to one embodiment, the display device (100) may identify a circle having a diameter corresponding to the size of the user's face and an eccentricity corresponding to the shape of the user's face within the initial image, based on a plurality of feature points identified from the initial image, with a first reference point as the center of the circle. The display device (100) may identify a circle corresponding to the user's face included in each of the continuously acquired images by changing only the position of the circle within the continuously acquired images without changing the diameter and eccentricity of the circle identified from the initial image.

[0138] Specifically, the display device (100) can identify a first circle (620) corresponding to the user's face included in the first frame image (610). Since the user is looking straight ahead in the first frame image (610), the location of the first reference point is the center point of the image, and the direction of the user's upper body also faces straight ahead. Accordingly, the display device (100) can generate a first circle (610) having a normal vector in the front direction.

[0139] Additionally, the display device (100) can identify a second circle (640) corresponding to the user's face included in the second frame image (630). In the second frame image (630), the user is looking to the right, so the position of the first reference point has moved to the right, and the direction of the user's upper body is also facing to the right. Accordingly, the display device (100) can generate a second circle (640) facing to the right based on the direction of the user's upper body and the coordinates of the first reference point. The plane including the second circle (640) can have a normal vector facing to the right.

[0140] The display device (100) can estimate the angle of the user's upper body direction and identify that the angle of the user's face direction is within a preset angle range from the estimated angle of the user's upper body direction. Here, the preset angle range can be set to various angle ranges by the user or the manufacturer of the display device. For example, the preset angle range can be set to ±10 degrees.

[0141] Here, the first circle (620) and the second circle (640) are circles having the same diameter and eccentricity. That is, the display device (100) can only change the position of a circle having a fixed diameter and eccentricity based on the direction of the user's upper body and the coordinates of the first reference point.

[0142] According to another embodiment, the display device (100) can track a first reference point, a second reference point, a third reference point, a fourth reference point, and a fifth reference point based on information about a plurality of reference points, and can identify a circle corresponding to the user's face based on the coordinates of the first reference point, the fourth reference point, and the fifth reference point with respect to the direction of the user's upper body and the reference coordinate system. In this case, since there are a total of five reference points for identifying the circle corresponding to the user's face, the circle corresponding to the user's face can be identified more accurately than a method of identifying the circle corresponding to the user's face based on three reference points.

[0143] Specifically, the display device (100) can identify a circle with the first reference point as the center of the circle and the fourth and fifth reference points as points on the circumference within continuously acquired images as a circle corresponding to the user's face.

[0144] Although the above description only explains that a circle corresponding to a user's face can be identified, this is only one example, and the display device (100) can also identify various shapes such as a square, a pentagon, and a hexagon corresponding to the shape of the user's face.

[0145] Additionally, although the above description explains that three reference points and five reference points can be used to identify a circle corresponding to a user's face, it is of course possible for the display device (100) to more accurately identify a circle corresponding to a user's face by using a greater number of reference points.

[0146] When the display device (100) identifies a circle corresponding to the user's face by the above-described method, the display device (100) can identify the direction of the user's face by identifying the position of the circle based on information about the reference coordinate system.

[0147] According to one embodiment, the display device (100) can identify a normal vector of a circle corresponding to a user's face in a reference coordinate system, identify an angle of the normal vector with respect to each of the xy plane, yz plane, and zx plane of the reference coordinate system, and calculate yaw, pitch, and roll values, and identify the calculated yaw, pitch, and roll values ​​as an angle of the direction of the user's face with respect to the reference coordinate system.

[0148] According to one embodiment, the display device (100) may identify the direction of the user's face by comparing the positions of a circle corresponding to the user's frontal face identified in the initial image with a circle corresponding to the user's face identified in the current image. The display device (100) may identify how much the circle corresponding to the user's face identified in the current image is rotated compared to the circle corresponding to the user's frontal face, thereby calculating yaw, pitch, and roll values, and may identify the calculated yaw, pitch, and roll values ​​as angles of the direction of the user's face with respect to a reference coordinate system.

[0149] The display device (100) identifies a circle corresponding to the user's face by tracking only a plurality of reference points in continuously acquired images while fixing the reference coordinate system defined in the initial image and identifying only the coordinates of the plurality of reference points in the reference coordinate system, and can quickly and accurately identify the direction of the user's face based on the position of the identified circle. In addition, unlike a conventional method for identifying the direction of the user's face, the display device (100) identifies the direction of the user's upper body and identifies the direction of the user's face by considering the direction of the user's upper body, so that the direction of the user's face can be identified more accurately than a conventional method.

[0150] The display device (100) may also update information about a plurality of reference points identified in the initial image and information about the reference coordinate system.

[0151] According to one embodiment, the display device (100) may delete information about the reference coordinate system and information about the plurality of reference points from the memory (120) if a plurality of reference points are not tracked within an image acquired for a preset number of frames.

[0152] If the display device (100) does not continuously track a plurality of reference points within an image acquired for a preset number of frames, the display device (100) determines that the user's face direction cannot be tracked based on information about the reference coordinate system and information about the plurality of reference points stored in the memory (120), and performs an operation to update information about the reference coordinate system and information about the plurality of reference points.

[0153] According to one embodiment, the display device (100) may identify an image in which the user's face and shoulders are identified among images acquired after a preset number of frames as an initial image, identify a plurality of reference points and a reference coordinate system within the initial image, and store information about the identified plurality of reference points and information about the reference coordinate system in the memory (120).

[0154] Specifically, the display device (100) can identify an image in which the user's frontal face and frontal upper body are identified among images acquired after a preset number of frames as an initial image. The display device (100) can identify a plurality of feature points from the user's frontal face and upper body, and identify a plurality of reference points within the initial image based on the positions of the identified plurality of feature points. The method by which the display device (100) identifies a plurality of reference points within the initial image and identifies a reference coordinate system based on the identified plurality of reference points has been described in detail in the above description, and therefore a description thereof will be omitted.

[0155] Additionally, the preset number of frames may be set to any number of frames (e.g., 60 frames) by the manufacturer or user of the display device (100).

[0156] According to one embodiment, the display device (100) can delete information about the reference coordinate system and information about a plurality of reference points from the memory (120) at every preset number of frame intervals, and identify an image in which the user's face and shoulders are identified among images acquired after the preset number of frames as an initial image. Even when a plurality of reference points are tracked in images acquired continuously, the display device (100) can identify the direction of the user's face more accurately by updating information about the reference coordinate system at every preset number of frame intervals.

[0157] As described above, the display device (100) can quickly and accurately identify the direction of the user's face by identifying a plurality of characteristic points (e.g., eyes, nose, mouth, shoulder line, etc.) only within the initial image, and by tracking only a plurality of reference points without identifying a plurality of characteristic points within images acquired after the initial image. The display device (100) according to the present disclosure can also implement the method for identifying the direction of the user's face as described above through a learned neural network model. In the description of FIG. 7 described below, a method for identifying the direction of the user's face based on the learned neural network model will be described.

[0158] FIG. 7 is a diagram illustrating a method for identifying a face direction based on a neural network model according to one or more embodiments of the present disclosure.

[0159] According to FIG. 7, the display device (100) can input information (710) about a reference coordinate system and coordinates (720) of a plurality of reference points into a neural network model (730) to output an angle in the direction of the user's face.

[0160] The neural network model (730) may be a neural network model trained to receive information (710) about a reference coordinate system, coordinates (720) of a plurality of reference points with respect to the reference coordinate system, identify a circle corresponding to the user's face based on the coordinates of the plurality of reference points, and output an angle of the direction of the user's face based on the position of the identified circle with respect to the reference coordinate system.

[0161] According to one embodiment, the neural network model (730) may be trained using the coordinates of a plurality of reference points and a circle corresponding to the user's face generated based on the coordinates of the plurality of reference points as training data. Specifically, the coordinates of the plurality of reference points included in the training data may be input data, and the circle corresponding to the user's face generated based on the coordinates of the plurality of reference points included in the training data may be target data.

[0162] The loss function of the neural network model (730) may be implemented as a function that measures the difference between the "circle inferred from the input data by the neural network model (730)" and the "circle corresponding to the face of the user given as target data to the neural network model (730)". At this time, the neural network model (730) may be trained to minimize the loss function. Specifically, the neural network model (730) may be trained to include parameters that minimize the loss function through a method such as the gradient descent method.

[0163] According to one embodiment, the neural network model (730) may be trained to identify a position in a reference coordinate system of a circle estimated from the coordinates of a plurality of reference points, and output an angle (yaw, pitch, roll value) of a user's face direction based on the position of the identified circle.

[0164] The neural network model (730) described above may be a neural network model including various neural network structures such as a CNN (Convolutional Neural Network), a DNN (Deep Neural Network), an RNN (Recurrent Neural Network), an RBM (Restricted Boltzmann Machine), a DBN (Deep Belief Network), a BRDNN (Bidirectional Recurrent Deep Neural Network), an LSTM (Long Short Term Memory), or a deep Q-network.

[0165] The neural network model (730) as described above may be stored in the memory (120). In addition, the neural network model (730) may be stored in an external server device. In this case, the display device (100) may transmit information about a reference coordinate system and information about a plurality of reference points to the external server device, and the external server device may input the received information about the reference coordinate system and information about a plurality of reference points into the neural network model (730) and transmit the output angle of the user's face direction to the display device (100).

[0166] It goes without saying that the method for identifying the direction of a user's face described in FIGS. 4 to 7 described above can be applied to various electronic devices capable of obtaining images in which the user's face and shoulders are identified.

[0167] For example, the method for identifying the direction of a user's face according to the present disclosure may be used in an electronic device for identifying the driver's state in a vehicle. Specifically, when an electronic device for identifying the driver's state acquires images of a driver in a vehicle in which the driver's face is continuously facing downward for a preset number of frames, the electronic device may detect that the driver is drowsy and output a warning sound to the driver. As another example, the electronic device may output a warning sound to the driver to look forward if the driver's face is continuously not looking forward for a preset number of frames.

[0168] As another example, the method for identifying the direction of a user's face according to the present disclosure can be used in a human-robot interaction (HRI). Specifically, when a robot acquires images of a user by continuously capturing the user, the robot can identify the direction of the user's face and provide the user with information about the object the user is looking at or provide a user interface (UI) that allows the user to control the electronic device the user is looking at. For example, if the user is looking at an air conditioner, the robot can provide the user with a UI that allows the user to turn the air conditioner on / off or change the air conditioner's set temperature or operation mode. As another example, the robot can initiate a conversation with the user if the user's face is facing the robot.

[0169] As another example, the method for identifying a user's face orientation according to the present disclosure may be used in an electronic device that assesses a user's concentration. In this case, the electronic device can assess the user's concentration by identifying the user's face orientation when continuously capturing images of the user, recording and storing how long the user gazes at a display or book.

[0170] As another example, the method for identifying the direction of a user's face according to the present disclosure can be used in various fields, such as in an electronic device for identifying the direction of an athlete's head in a sports field.

[0171] FIG. 8 is a flowchart illustrating a method for displaying a 3D image on a display device according to one or more embodiments of the present disclosure.

[0172] According to FIG. 8, when a current image in which a user's face and shoulders are identified is acquired, the display device (100) can track a plurality of reference points within the current image based on information about the plurality of reference points (S810).

[0173] Here, the display device (100) can identify the movement of the plurality of reference points based on the coordinates of the plurality of reference points in the image of the previous frame and the pixel values ​​of the pixels corresponding to the coordinates of the plurality of reference points in the image of the previous frame, and track the plurality of reference points within the current image based on the identified movement.

[0174] According to one embodiment, the plurality of reference points may include a first reference point, a second reference point, a third reference point, a fourth reference point, and a fifth reference point. In this case, the first reference point may be a point corresponding to the center point of the user's face, the second reference point may be a point corresponding to the user's left shoulder, the third reference point may be a point corresponding to the user's right shoulder, the fourth reference point may be a point corresponding to the user's left jaw, and the fifth reference point may be a point corresponding to the user's right jaw.

[0175] Next, the display device (100) can identify the coordinates of a plurality of reference points within the current image with respect to the reference coordinate system based on information about the reference coordinate system (S820).

[0176] Next, the display device (100) can identify the direction of the user's face based on the coordinates of a plurality of reference points identified within the current image (S830).

[0177] According to one embodiment, the display device (100) can identify a circle corresponding to the user's face based on coordinates of a plurality of reference points identified within the current image, and can identify the direction of the user's face by identifying the position of the circle with respect to the reference coordinate system based on information about the reference coordinate system.

[0178] According to one embodiment, the display device (100) can identify the direction of the user's upper body by identifying the coordinates of the second reference point and the third reference point with respect to the reference coordinate system based on information about the reference coordinate system, identify a circle corresponding to the user's face based on the direction of the user's upper body and the coordinates of the first reference point with respect to the reference coordinate system, and then identify the direction of the user's face by identifying the position of the circle with respect to the reference coordinate system.

[0179] According to one embodiment, the display device (100) can identify a circle corresponding to the user's face based on the direction of the user's upper body and the coordinates of the first reference point, the fourth reference point, and the fifth reference point with respect to the reference coordinate system, and then identify the position of the circle with respect to the reference coordinate system to identify the direction of the user's face.

[0180] According to one embodiment, the display device (100) may identify the direction of the user's face based on the angle of the direction of the user's face output by inputting information about the coordinates of a plurality of reference points and the reference coordinate system within the current image into a neural network model.

[0181] Next, the display device (100) can display a 3D image corresponding to the identified user face direction (S840).

[0182] By the above-described method, the display device (100) can quickly and accurately identify the direction of the user's face and display a 3D image corresponding to the identified direction of the user's face, so that the user can be provided with an optimal 3D image.

[0183] FIG. 9 is a flowchart illustrating a method for storing information about a reference coordinate system of a display device according to one or more embodiments of the present disclosure.

[0184] According to FIG. 9, when an initial image in which a user's face and shoulders are identified is acquired, the display device (100) can identify a plurality of feature points within the initial image (S910).

[0185] Here, the multiple feature points may include a plurality of characteristic points used to recognize the user's face and shoulders. For example, the multiple feature points may include points corresponding to the eyes, nose, mouth, and jawline of the user's face, as well as points corresponding to the corners and edges of the user's shoulders.

[0186] Next, the display device (100) can identify a plurality of reference points within the initial image based on the positions of the plurality of feature points (S920).

[0187] Next, the display device (100) can identify a reference coordinate system with the first reference point in the initial image as the origin based on the positional relationship of a plurality of reference points in the initial image (S930).

[0188] Next, the display device (100) can store information about the reference coordinate system within the initial image (S940).

[0189] According to one embodiment, the display device (100) may delete information about the reference coordinate system and information about the plurality of reference points if a plurality of reference points are not tracked within an image acquired after a preset number of frames. In this case, the display device (100) may identify an image in which the user's face and shoulders are identified among images acquired after a preset number of frames as an initial image, and may identify a plurality of reference points and reference coordinate systems within the initial image and store information about the identified plurality of reference points and information about the reference coordinate system.

[0190] According to one embodiment, the display device (100) may delete information about the reference coordinate system and information about a plurality of reference points at every preset number of frame intervals. In this case, the display device (100) may identify an image in which the user's face and shoulders are identified among images acquired after the preset number of frame intervals as an initial image, and may identify a plurality of reference points and reference coordinate systems within the initial image and store information about the identified plurality of reference points and information about the reference coordinate system.

[0191] According to the above-described method, the display device (100) stores information about a reference coordinate system defined in an initial image, and identifies the direction of the user's face in images that are continuously acquired based on the information about the reference coordinate system, thereby identifying the direction of the user's face more quickly than in a conventional method.

[0192] The various methods described in FIGS. 8 and 9 can be performed by a display device having the configuration shown in FIGS. 2 and 3, but are not necessarily limited thereto, and can also be performed by electronic devices having various configurations.

[0193] Meanwhile, in FIGS. 8 and 9, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0194] Meanwhile, the methods according to at least some of the various embodiments of the present disclosure described above may be implemented in the form of an application that can be installed on an existing electronic device.

[0195] Additionally, the methods according to at least some of the various embodiments of the present disclosure described above can be implemented with only a software upgrade or a hardware upgrade for an existing electronic device.

[0196] Additionally, the methods according to at least some of the various embodiments of the present disclosure described above may also be performed through an embedded server provided in an electronic device, or an external server of at least one of the electronic devices.

[0197] Meanwhile, according to one embodiment of the present disclosure, the various embodiments described above may be implemented as software including commands stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include a display device (e.g., display device (A)) according to the disclosed embodiments, which is a device that can call commands stored in the storage medium and operate according to the called commands. When the commands are executed by the processor, the processor may directly or under the control of the processor use other components to perform a function corresponding to the commands. The commands may include code generated or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term 'non-transitory storage medium' means a tangible device that does not contain signals (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently in the storage medium and cases where it is stored temporarily. For example, a 'non-transitory storage medium' may include a buffer in which data is temporarily stored. According to one embodiment, the method according to the various embodiments disclosed in the present document may be provided as a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones).In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily created in a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0198] Various embodiments of the present disclosure may be implemented as software including commands stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call commands stored in the storage medium and operate according to the called commands, and may include a display device (e.g., display device (100)) according to the disclosed embodiments.

[0199] When the above-described instruction is executed by the processor, the processor may perform the function corresponding to the instruction directly or by utilizing other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter.

[0200] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In the display device, A memory that stores information about a reference coordinate system defined in the initial image and information about multiple reference points; display; and At least one processor controlling the display to display a 3D image corresponding to the identified user face direction; At least one processor of the above, When a current image in which the user's face and shoulders are identified is acquired, the plurality of reference points within the current image are tracked based on information about the plurality of reference points, Identifying coordinates of the plurality of reference points within the current image with respect to the reference coordinate system based on information about the reference coordinate system, Identifying the direction of the user's face based on the coordinates of the plurality of reference points identified within the current image, The above plurality of reference points include a reference point corresponding to the position of the user's face and a reference point corresponding to the position of the user's shoulder, A display device, wherein information about the plurality of reference points includes coordinates of the plurality of reference points in an image of a previous frame and pixel values ​​of pixels corresponding to coordinates of the plurality of reference points in an image of a previous frame.

2. In paragraph 1, The above plurality of reference points include a first reference point corresponding to the central point of the user's face, At least one processor of the above, When an initial image in which the user's face and shoulders are identified is obtained, multiple feature points are identified within the initial image, Identifying the plurality of reference points within the initial image based on the positions of the plurality of feature points, Based on the positional relationship of the plurality of reference points within the initial image, the reference coordinate system having the first reference point within the initial image as its origin is identified, and information about the reference coordinate system is stored in the memory. A display device, wherein the plurality of feature points include a plurality of points used to recognize the user's face and shoulders.

3. In paragraph 1, At least one processor of the above, Identifying a circle corresponding to the user's face based on the coordinates of the plurality of reference points identified within the current image; A display device that identifies the direction of the user's face by identifying the position of the circle with respect to the reference coordinate system based on information about the reference coordinate system.

4. In paragraph 1, The above multiple reference points include a first reference point, a second reference point, and a third reference point, The above first reference point is a point corresponding to the central point of the user's face, The above second reference point corresponds to the left shoulder of the user, The third reference point corresponds to the user's right shoulder, At least one processor of the above, When the current image is acquired, the first reference point, the second reference point, and the third reference point are tracked within the current image based on information about the plurality of reference points, Identify the direction of the user's upper body by identifying the coordinates of the second reference point and the third reference point for the reference coordinate system based on information about the reference coordinate system, Identifying a circle corresponding to the user's face based on the direction of the user's upper body and the coordinates of the first reference point with respect to the reference coordinate system; A display device that identifies the direction of the user's face by identifying the position of the circle with respect to the reference coordinate system.

5. In paragraph 4, The above multiple reference points further include a fourth reference point and a fifth reference point, The above fourth reference point corresponds to the left jaw point of the user, The above fifth reference point corresponds to the user's right jaw point, At least one processor of the above, When the current image is acquired, the first reference point, the second reference point, the third reference point, the fourth reference point and the fifth reference point are tracked within the current image based on information about the plurality of reference points, A display device that identifies a circle corresponding to the user's face based on the upper body direction of the user and the coordinates of the first reference point, the fourth reference point, and the fifth reference point with respect to the reference coordinate system.

6. In paragraph 1, At least one processor of the above, If the plurality of reference points are not tracked within the image acquired for a preset number of frames, information about the reference coordinate system and information about the plurality of reference points are deleted from the memory, Among the images acquired after the above-mentioned preset number of frames, an image in which the user's face and shoulders are identified is identified as the initial image, A display device that identifies the plurality of reference points and the reference coordinate system within the initial image, and stores information about the identified plurality of reference points and information about the reference coordinate system in the memory.

7. In paragraph 1, At least one processor of the above, At each preset number of frame intervals, information about the reference coordinate system and information about the plurality of reference points are deleted from the memory, Among the images acquired after the above-described number of frame intervals, an image in which the user's face and shoulders are identified is identified as the initial image, A display device that identifies the plurality of reference points and the reference coordinate system within the initial image, and stores information about the identified plurality of reference points and information about the reference coordinate system in the memory.

8. In paragraph 1, At least one processor of the above, When the current image is acquired, the movement of the plurality of reference points is identified based on the coordinates of the plurality of reference points in the image of the previous frame and the pixel values ​​of the pixels corresponding to the coordinates of the plurality of reference points in the image of the previous frame, A display device that tracks the plurality of reference points within the current image based on the identified movement.

9. In paragraph 1, The above memory stores the neural network model, The above neural network model is, Input information about the above reference coordinate system, the coordinates of the above multiple reference points for the above reference coordinate system, Identifying a circle corresponding to the user's face based on the coordinates of the plurality of reference points, A neural network model trained to output an angle of the user's face direction based on the position of the identified circle with respect to the reference coordinate system, At least one processor of the above, A display device that identifies the direction of the user's face based on the angle of the direction of the user's face, which is output by inputting coordinates of the plurality of reference points within the current image and information about the reference coordinate system into the neural network model.

10. In paragraph 1, Including more cameras, A display device that tracks the plurality of reference points within the current image based on information about the plurality of reference points when the current image is acquired through the camera.

11. In a method for controlling a display device, When a current image in which a user's face and shoulders are identified is acquired, a step of tracking a plurality of reference points within the current image based on information about the plurality of reference points; A step of identifying coordinates of said plurality of reference points within said current image with respect to said reference coordinate system based on information about said reference coordinate system; A step of identifying the direction of the user's face based on the coordinates of the plurality of reference points identified within the current image; and A step of displaying a 3D image corresponding to the identified user face direction; The above plurality of reference points include a reference point corresponding to the position of the user's face and a reference point corresponding to the position of the user's shoulder, A control method, wherein information about the plurality of reference points includes coordinates of the plurality of reference points in an image of a previous frame and pixel values ​​of pixels corresponding to coordinates of the plurality of reference points in an image of a previous frame.

12. In paragraph 11, The above plurality of reference points include a first reference point corresponding to the central point of the user's face, When an initial image in which the user's face and shoulders are identified is acquired, a step of identifying a plurality of feature points within the initial image; A step of identifying the plurality of reference points within the initial image based on the positions of the plurality of feature points; and A step of identifying the reference coordinate system having the first reference point in the initial image as the origin based on the positional relationship of the plurality of reference points in the initial image and storing information about the reference coordinate system; including; A control method, wherein the plurality of feature points include a plurality of points used to recognize the user's face and shoulders.

13. In paragraph 11, The step of identifying the user's face direction is as follows: A step of identifying a circle corresponding to the user's face based on coordinates of the plurality of reference points identified within the current image; and A control method comprising: a step of identifying the direction of the user's face by identifying the position of the circle with respect to the reference coordinate system based on information about the reference coordinate system; 14. In paragraph 11, The above multiple reference points include a first reference point, a second reference point, and a third reference point, The above first reference point is a point corresponding to the central point of the user's face, The above second reference point corresponds to the left shoulder of the user, The third reference point corresponds to the user's right shoulder, When the current image is acquired, a step of tracking the first reference point, the second reference point, and the third reference point within the current image based on information about the plurality of reference points; A step of identifying the direction of the user's upper body by identifying coordinates of the second reference point and the third reference point with respect to the reference coordinate system based on information about the reference coordinate system; A step of identifying a circle corresponding to the user's face based on the direction of the user's upper body and the coordinates of the first reference point with respect to the reference coordinate system; and A control method comprising: a step of identifying the direction of the user's face by identifying the position of the circle with respect to the reference coordinate system; 15. A non-transitory computer-readable recording medium storing computer instructions that, when executed by a processor of a display device, cause the display device to perform a control method, wherein the control method comprises: When an image from which a user's face is identified is acquired, a step of identifying a plurality of reference points within the image based on information about the plurality of reference points; A step of identifying coordinates of the identified plurality of reference points for the reference coordinate system based on information about the reference coordinate system; and A step of identifying the direction of the user's face based on the coordinates of the plurality of reference points; The above plurality of reference points include a reference point corresponding to the position of the user's face and a reference point corresponding to the position of the user's shoulder, A non-transitory computer-readable storage medium comprising information about the plurality of reference points, the coordinates of the plurality of reference points in the image of the previous frame and the pixel values ​​of pixels corresponding to the coordinates of the plurality of reference points in the image of the previous frame.

Images