Head-mounted display device for displaying interface, and operation method of head-mounted display device

Abstract

The present disclosure includes a head-mounted display and an operation method thereof, the head-mounted display comprising: a camera; a display; a memory for storing at least one instruction; and at least one processor. The at least one processor, by executing the at least one instruction, enables the head-mounted display device to: display a first interface through the display; acquire an image by photographing a user through the camera; detect a pose of the user on the basis of the acquired image; compare the detected pose with a plurality of poses which are preset so as to correspond to respective plurality of interfaces; and display a second interface in place of the first interface through the display on the basis that the detected pose corresponds to the second interface which is one of the plurality of interfaces.

Description

Head-mounted display device displaying an interface and a method of operating the head-mounted display device

[0001] The present disclosure relates to a head-mounted display device and a method of operating the head-mounted display device. Specifically, the present disclosure relates to a head-mounted display device that displays an interface and a method of operating the head-mounted display device.

[0002] As technology advances, head-mounted display devices are being developed that provide images to users through displays placed close to the user's eyes while worn on the user's head.

[0003] The display included in the head-mounted display device may include an optical see-through display for overlaying a virtual image on a physical environment space of the real world or a real world object, or a video see-through display for providing a user with an image captured by a camera or the like of a physical environment space of the real world.

[0004] Recently, technologies are being developed to provide user input to images provided through head-mounted display devices via interfaces.

[0005] One embodiment of the present disclosure provides a head-mounted display device. The head-mounted display device may include a camera. The head-mounted display device may include a display. The head-mounted display device may include a memory storing at least one instruction and at least one processor. By executing at least one instruction, the head-mounted display device may display a first interface through the display. By executing at least one instruction, the head-mounted display device may capture an image of a user through a camera. By executing at least one instruction, the head-mounted display device may detect a pose of the user based on the acquired image. By executing at least one instruction, the head-mounted display device may compare the detected pose with a plurality of poses preset to correspond to a plurality of interfaces, respectively. By having at least one processor execute at least one command, the head mounted display device can display a second interface instead of the first interface through the display, as the detected pose corresponds to the second interface, which is one of the plurality of interfaces.

[0006] One embodiment of the present disclosure provides a method for operating a head-mounted display device. The method for operating a head-mounted display device may include a step of capturing an image of a user by using a camera. The method for operating a head-mounted display device may include a step of detecting a pose of the user based on the acquired image. The method for operating a head-mounted display device may include a step of comparing the detected pose with a plurality of poses preset to correspond to a plurality of interfaces, respectively. The method for operating a head-mounted display device may include a step of displaying a second interface by replacing a first interface through a display when the detected pose corresponds to a second interface, which is one of the plurality of interfaces.

[0007] As one embodiment of the present disclosure, a computer-readable recording medium having recorded thereon a program for performing at least one method of the disclosed operating method on a computer may be provided.

[0008] The present disclosure can be readily understood by the following detailed description and its accompanying drawings, wherein reference numerals refer to structural elements.

[0009] FIG. 1 is a drawing for explaining a head mounted display device according to one embodiment of the present disclosure.

[0010] FIG. 2 is a drawing for explaining the configuration of a head mounted display device according to one embodiment of the present disclosure.

[0011] FIG. 3 is a block diagram illustrating the configuration of a head mounted display device according to one embodiment of the present disclosure.

[0012] FIG. 4 is a flowchart for explaining an operation method of a head mounted display device according to one embodiment of the present disclosure.

[0013] FIG. 5 is a flowchart illustrating an operation of displaying a first interface and a second interface while first content is displayed, according to one embodiment of the present disclosure.

[0014] FIG. 6 is a diagram for explaining an operation of displaying an interface corresponding to a detected user's pose according to one embodiment of the present disclosure.

[0015] FIG. 7 is a flowchart illustrating an operation of determining an interface position based on a spatial map generated through a camera and a detected pose according to one embodiment of the present disclosure.

[0016] FIG. 8 is a flowchart illustrating an operation of acquiring a user's viewpoint through a gaze tracking sensor and detecting a user's pose based on the acquired viewpoint and acquired image according to one embodiment of the present disclosure.

[0017] FIG. 9 is a flowchart illustrating an operation of determining a content location for displaying second content based on a user's viewpoint acquired through a gaze tracking sensor according to one embodiment of the present disclosure.

[0018] FIG. 10A is a diagram for explaining an operation of determining a content location for displaying second content based on a user's viewpoint acquired through a gaze tracking sensor according to one embodiment of the present disclosure.

[0019] FIG. 10b is a diagram for explaining an operation of determining a content location for displaying second content based on a user's viewpoint acquired through a gaze tracking sensor according to one embodiment of the present disclosure.

[0020] FIG. 11A is a diagram for explaining an operation of displaying first content and second content according to one embodiment of the present disclosure.

[0021] FIG. 11b is a diagram for explaining an operation of displaying first content and second content according to one embodiment of the present disclosure.

[0022] FIG. 11c is a diagram for explaining an operation of displaying first content and second content according to one embodiment of the present disclosure.

[0023] FIG. 12 is a diagram for explaining whether an operation of comparing a detected pose with a preset start gesture and comparing a detected pose with a plurality of preset poses is performed according to one embodiment of the present disclosure.

[0024] The terms used in this disclosure will be briefly explained, and one embodiment of the present disclosure will be specifically described.

[0025] The terms used in this disclosure are selected from widely used, current terms, taking into account the functions of one embodiment of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments of the disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of the disclosure.

[0026] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0027] Throughout this disclosure, when a part is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," and the like described herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0028] As used herein, the expression "configured to" can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something is "specifically designed to" in terms of hardware. Instead, in some contexts, the expression "a system configured to" can mean that the system is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured to perform A, B, and C" can mean a dedicated processor for performing the operations (e.g., an embedded processor), or a general-purpose processor (e.g., a CPU or an application processor) that can perform the operations by executing one or more software programs stored in memory.

[0029] Additionally, when a component is referred to as being "connected" or "connected" to another component in the present disclosure, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated.

[0030] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, one embodiment of the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted to clearly describe one embodiment of the present disclosure, and similar parts are designated with similar drawing reference numerals throughout the present disclosure.

[0031] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0032] FIG. 1 is a drawing for explaining a head mounted display device according to one embodiment of the present disclosure.

[0033] Referring to FIG. 1, an electronic device according to one embodiment of the present disclosure can provide content (300) to a user (200) wearing the electronic device. In one embodiment, the electronic device can be a head mounted display (HMD) device (100) as a device capable of providing content (300) to the user (200).

[0034] In one embodiment, FIG. 1 illustrates an electronic device having a shape including a support portion that is hung over the face of a user (200). However, the present disclosure is not limited thereto, and the electronic device may also include a support portion that is hung over the face and head of the user (200). Hereinafter, for convenience of explanation, the electronic device is described as a head-mounted display device (100).

[0035] In one embodiment, the head-mounted display device (100) may be an optical see-through device that provides a virtual image by overlaying it on a physical environment space (1000) of the real world or a real world object to the user (200). The head-mounted display device (100) may provide data provided from an external source or a virtual image generated by the head-mounted display device (100) to the user (200).

[0036] However, the present disclosure is not limited thereto. The head-mounted display device (100) may be a video see-through electronic device that captures an image of a physical environment space (1000) toward which the user's (200) gaze is directed using a camera, and then provides the acquired image to the user (200).

[0037] Hereinafter, for convenience of explanation, the head-mounted display device (100) of the present disclosure is described as an optical see-through device, but the present disclosure is not limited thereto.

[0038] In one embodiment, the head mounted display device (100) may include a camera (130). In one embodiment, the camera (130) may include a first camera (110) for obtaining an image of the front of the head mounted display device (100) and a second camera (120) for obtaining an image of the lower portion of the head mounted display device (100). In one embodiment, the image of the front may refer to an image of an environment located in front of the head mounted display device (100) among images of the real world, i.e., the user's surrounding environment. The image of the lower portion may refer to an image of an environment located below the head mounted display device (100) among images of the real world.

[0039] In one embodiment, the first camera (110) may be positioned at the upper portion of the head-mounted display device (100). The second camera (120) may be positioned at the lower portion of the head-mounted display device (100). However, the present disclosure is not limited thereto. The positions of the first camera (110) and the second camera (120) positioned on the head-mounted display device (100) are not limited.

[0040] In one embodiment, FIG. 1 illustrates a head-mounted display device (100) including two first cameras and two second cameras. However, the present disclosure is not limited thereto, and the number of first cameras (110) and second cameras (120) may be one or three or more, respectively.

[0041] However, the present disclosure is not limited thereto, and the head-mounted display device (100) may include a camera capable of capturing both a front image and a downward image of the head-mounted display device (1000). In this case, the number of cameras capable of capturing both a front image and a downward image may be one or more.

[0042] Hereinafter, for convenience of explanation, the camera (130) is described as including a first camera (110) arranged at the upper end of the head mounted display device (100) to capture an image in front of the head mounted display device (100) and a second camera (120) arranged at the lower end of the head mounted display device (100) to capture an image below the head mounted display device (100).

[0043] In one embodiment, the head mounted display device (100) can display content (300) through a display (140) to provide content (300) to a user (200).

[0044] In one embodiment, the head-mounted display device (100) can determine the location, size, arrangement, etc. of the content (300) displayed on the display (140) so that the content (300) appears to the user (200) as being located in a specific area within the real space (1000) or as being overlaid with a specific object included in the real space (1000). Information about the location, size, arrangement, etc. of the content (300) to be displayed on the display (140) can be referred to as “content location.” The head-mounted display device (100) can display the content (300) on the determined content location of the display (140) based on the viewpoint of the user (200) and provide the content (300) to the user (200). Hereinafter, the content location and the method for determining the content location will be described later with reference to FIGS. 9 to 10B.

[0045] In one embodiment, while content (300) is provided to a user (200) through a head-mounted display device (100), the user may provide user input to the head-mounted display device (100) through an interface provided from the head-mounted display device (100). The head-mounted display device (100) may obtain user input through the interface provided to the user (200).

[0046] The head-mounted display device (100) of the present disclosure can capture an image by photographing a user (200) through a camera (130).

[0047] In one embodiment, the head mounted display device (100) can capture an image by capturing a motion using the user's (200) hand through the camera (130). The head mounted display device (100) can capture an image by capturing a motion using the user's (200) hand through the second camera (120) without movement of the head of the user (200) wearing the head mounted display device (100). The image here may mean an image including information about the user, captured by capturing the user's body to detect the user's pose.

[0048] In one embodiment, even if the user (200) moves his / her hand while looking at the content (300) with or without moving his / her head while wearing the head-mounted display device (100), the head-mounted display device (100) can capture an image of the user's (200) hand through the second camera (120).

[0049] In one embodiment, the head-mounted display device (100) can detect a user's pose based on an acquired image. In one embodiment, the head-mounted display device (100) can detect a pose (400) of a user's hand based on an image including the user's hand (200). Hereinafter, an operation of detecting a user's pose based on an image acquired by the head-mounted display device (100) will be described later with reference to FIGS. 3 and 6.

[0050] In one embodiment, the head-mounted display device (100) may compare the detected user's pose (400) with a plurality of preset poses. The plurality of preset poses may include a pose of a hand typing on a keyboard, a pose of a hand using a mouse, a pose of a hand drawing on a drawing pad (e.g., an object for drawing, such as paper or a pad) using a pen, or a pose of a hand using a touch pad (e.g., a pad that controls user input using touch). However, the present disclosure is not limited thereto, and the plurality of preset poses may include a pose of a person performing a specific action.

[0051] In one embodiment, each of the plurality of poses may be preset to correspond to a plurality of interfaces. In one embodiment, the interfaces may include a keyboard, a mouse, a drawing pad, or a touchpad. The pose of a hand typing may be preset to correspond to a keyboard. The pose of a hand using a mouse may be preset to correspond to a mouse. The pose of a hand drawing with a pen may be preset to correspond to a drawing pad.

[0052] In one embodiment, the head mounted display device (100) may display an interface set to correspond to one of the detected poses (400) through the display (140) when the detected pose (400) corresponds to one of a plurality of preset poses. The head mounted display device (100) may provide the user (200) with an interface corresponding to the detected pose (400) through the display (140).

[0053] In one embodiment, the head mounted display device (100) may determine the position, size, arrangement, etc. of an interface displayed on the display (140) so that the interface appears to the user (200) to be located in a specific area within the real space (1000) or to be overlaid with a specific object included in the real space (1000). Information about the position, size, arrangement, etc. of the interface to be displayed on the display (140) may be referred to as an “interface position.” The head mounted display device (100) may determine an interface corresponding to a detected pose (400) and determine the interface position based on at least one of the shape, size, or type of the determined interface and a spatial map. The head mounted display device (100) may display the interface at the determined interface position on the display (140) and provide it to the user (200).

[0054] In one embodiment, the head-mounted display device (100) can capture a space (1000) including the head-mounted display device (100) through a camera (130) to obtain a spatial image. The spatial image here may refer to an image including spatial information obtained by capturing the space (1000) to create a spatial map. The head-mounted display device (100) can create a spatial map representing the space (1000) based on the acquired spatial image. Hereinafter, an operation of creating a spatial map based on the spatial image acquired by the head-mounted display device (100) will be described later with reference to FIGS. 3 and 7. In addition, an operation of determining an interface position will be described later with reference to FIGS. 3, 11A, 11B, and 11C.

[0055] In one embodiment, if the detected pose (400) corresponds to a pose of a typing hand among a plurality of preset poses, the head mounted display device (100) may display a keyboard (500) corresponding to the pose of the typing hand through the display (140). In one embodiment, the head mounted display device (100) may determine an interface position at which to display the keyboard (500) on the display (140) based on the spatial map and the pose of the typing hand. The interface position at which to display the keyboard (500) may be a position at which to display the keyboard (500) on the display (140) so that the user (200) feels that the keyboard (500) is located in an area where the user's (200's) fingers are located among the tables included in the real space (1000).

[0056] In one embodiment, a user (200) may be provided with content (300) and a keyboard (500) through a head-mounted display device (100). The user (200) may experience the content (300) and the keyboard (500) displayed on the display (140) as being located within a real space (1000). In one embodiment, FIG. 1 illustrates virtual content (301) and a virtual keyboard (501) that appear to be located within a real space (1000) or overlaid on a real object, as experienced by the user (200) through the head-mounted display device (100) and as seen from the user's (200) field of view.

[0057] In one embodiment, the content location may be a location on the display (140) determined so that the content (300) displayed on the display (140) appears to the user (200) as virtual content (301). The interface location may be a location on the display (140) determined so that the keyboard (500) displayed on the display (140) appears to the user (200) as a virtual keyboard (501).

[0058] A user (200) can provide user input to the head mounted display device (100) through a keyboard (500) displayed on the head mounted display device (100). The user (200) can provide user input to the head mounted display device (100) through a keyboard (500) displayed on the display (140) recognized as a virtual keyboard (501).

[0059] Through the present disclosure, the user (200) may not perform any actions for requesting a keyboard (500) through the head mounted display device (100) other than taking a typing hand pose (e.g., operating a physical button of the head mounted display device (100) to display an interface, performing an action for stopping the operation of the content (300) being executed and displaying the interface, or performing an action for displaying the interface while looking at a screen other than the content (300).

[0060] In one embodiment, the head mounted display device (100) can detect the pose of the user (200) even after displaying the keyboard (500) on the display (140) by detecting the pose of the user's (200) typing hand. In one embodiment, the head mounted display device (100) can repeat the operation of detecting the pose of the user (200) at preset intervals.

[0061] In one embodiment, the head mounted display device (100) may perform an operation of detecting a pose of a user (200) while content (300) is displayed through the display (140), comparing the detected pose with a plurality of preset poses, and displaying a corresponding interface through the display (140).

[0062] In one embodiment, the head-mounted display device (100) may compare an interface (e.g., a keyboard (500), hereinafter referred to as a 'first interface') that was being displayed through the display (140) in a previous operation with an interface (hereinafter referred to as a 'second interface') corresponding to an additionally detected pose.

[0063] In one embodiment, the head mounted display device (100) may display the second interface through the display (140) and provide it to the user when it is determined that the first interface and the second interface are different. In one embodiment, the head mounted display device (100) may display the first interface displayed through the display (140) and the corresponding interface may be a note pad (510). Since the keyboard (500) and the note pad (510) displayed through the display (140) are different, the head mounted display device (100) may display the keyboard (500) as a note pad (510) through the display (140).

[0064] In one embodiment, the head mounted display device (100) can determine an interface location at which to display a note pad (510) based on a pose of a hand drawing a picture using a spatial map and a pen. The head mounted display device (100) can display the note pad (510) at the determined interface location on the display (140). The interface location at which to display the note pad (510) may be a location at which to display the note pad (510) on the display (140) so that the user (200) feels as if the note pad (510) is located in an area where a pen tip is located among tables included in real space (1000).

[0065] In one embodiment, a user (200) may experience a note pad (510) displayed on a display (140) as being located within a real space (1000). In one embodiment, FIG. 1 illustrates a virtual note pad (511) that appears to be located within a real-world space (1000) of a note pad (510) displayed on a display (140) within the user's (200) field of view, or overlaid on a real-world object, as experienced by the user (200) through a head-mounted display device (100).

[0066] In one embodiment, the interface location at which the note pad (510) is to be displayed may be a location on the display (140) determined so that the note pad (510) displayed on the display (140) appears to the user (200) as a virtual note pad (511).

[0067] In one embodiment, the head-mounted display device (100) may maintain the first interface displayed through the display (140) when it is determined that the first interface and the second interface are the same. In one embodiment, when the detected pose (410) corresponds to a pose of a hand typing among a plurality of preset poses, the interface corresponding to the pose may be a keyboard (500). In this case, the head-mounted display device (100) may maintain the keyboard (500) displayed through the display (140) and provide it to the user (200).

[0068] Through the present disclosure, a user (200) can assume a pose related to a second interface in order to input user input using a different type of second interface while inputting user input using a first interface. The user (200) does not need to perform other actions to request the second interface (e.g., operating a physical button of a head-mounted display device (100) to display an interface, performing an action to stop the operation of a currently running content (300) and display the interface, or performing an action to display the interface while looking at a screen other than the content (300). This provides a continuous user experience to the user (200) and improves the user's (200) operational convenience. In addition, the user (200) can provide user input using a type of interface desired depending on the content of the content (300).

[0069] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the present disclosure.

[0070] FIG. 2 is a drawing for explaining the configuration of a head mounted display device according to one embodiment of the present disclosure.

[0071] Referring to FIGS. 1 and 2, the head-mounted display device (100) may include a camera (130), a display (140), an optical lens (150), an eye tracking sensor (160), a frame (171, 172), a nose bridge (173), a nose support (174), glasses arms (175, 176), a battery (177), a display engine (178, 179), a memory (180), and at least one processor (190). In one embodiment, only components for explaining the structure of the head-mounted display device (100) are disclosed in FIG. 2, and the components included in the head-mounted display device (100) are not limited as illustrated in FIG. 2.

[0072] In one embodiment, the display (140) and the optical lens (150) may be disposed in frames (171, 172), respectively. The frames (171, 172) may have a shape similar to a frame of a typical eyeglass structure. The frames (171, 172) may have a rim shape surrounding the display (140) and the optical lens (150).

[0073] In one embodiment, the display (140) may include a first display (141) corresponding to the user's left eye (201) and a second display (142) corresponding to the user's right eye (202). In one embodiment, the optical lens (150) may include a first optical lens (151) corresponding to the user's left eye (201) and a second optical lens (152) corresponding to the user's right eye (202). In one embodiment, the distance between the optical lens (150) and the user's eye may be closer than the distance between the display (140) and the user's eye.

[0074] In one embodiment, the first optical lens (151) may be positioned closer to the user's left eye (201) than the first display (141). The second optical lens (152) may be positioned closer to the user's right eye (202) than the second display (142).

[0075] In one embodiment, the gaze tracking sensor (160) may include a first gaze tracking sensor (161) corresponding to the user's left eye (201) and a second gaze tracking sensor (132) corresponding to the user's right eye (302).

[0076] In one embodiment, the frames (171, 172) may include a first frame (171) corresponding to the user's left eye (201) and a second frame (172) corresponding to the user's right eye (202). The first frame (171) may include a rim surrounding the first display (141) and the first optical lens (151). The second frame (172) may include a rim surrounding the second display (142) and the second optical lens (152).

[0077] In one embodiment, the camera (130) may include a left camera (131) corresponding to the first frame (171) and a right camera (132) corresponding to the second frame (142). In one embodiment, FIG. 2 illustrates one left camera (131) and one right camera (132), but the present disclosure is not limited thereto. Referring to FIGS. 1 and 2, the left camera (131) and the right camera (132) may each include a first camera (110) and a second camera (120).

[0078] In one embodiment, the left camera (131) can capture an image by photographing the surrounding environment of the first frame (171). The left camera (131) includes a first camera (110) and a second camera (120), and can capture an image in front of and below the first frame (171). The right camera (132) can capture an image by photographing the surrounding environment of the second frame (172). The right camera (132) includes a first camera (110) and a second camera (120), and can capture an image in front of and below the second frame (172).

[0079] In one embodiment, the camera (130) can capture images of the user's surroundings by facing in a direction opposite to the direction in which the display (140) faces the user. However, the present disclosure is not limited thereto, and images can also be captured by capturing the upper, side, or rear of the first frame (171) and the second frame (172) depending on the arrangement or performance of each of the left camera (131) and the right camera (132).

[0080] In one embodiment, a first display (141), a first optical lens (151), a first camera (131), and a first gaze tracking sensor (161) may be arranged in a first frame (171). In one embodiment, a second display (142), a second optical lens (152), a second camera (132), and a second gaze tracking sensor (162) may be arranged in a second frame (172).

[0081] However, the present disclosure is not limited thereto. The first display (141) and the second display (142) may be formed integrally and placed in the frame (171, 172). The first optical lens (151) and the second optical lens (152) may be formed integrally and placed in the frame (171, 172). In addition, the first display (141) and the first optical lens (151) may be formed integrally and placed in the frame (171, 172). The second display (142) and the second optical lens (152) may be formed integrally and placed in the frame (171, 172).

[0082] In one embodiment, the glasses temples (175, 176) may be connected to the frames (171, 172). The glasses temples (175, 176) are portions that hang over the user's ears and support the head-mounted display device (100) when the user (200) wears the head-mounted display device (100). In one embodiment, the glasses temples (175, 176) may include a first glasses temple (175) connected to a first frame (171) and a second glasses temple (176) connected to a second frame (172).

[0083] In one embodiment, the display engines (178, 179) may include a first display engine (178) that projects a virtual image onto a first display (141) and a second display engine (179) that projects a virtual image onto a second display (142). In one embodiment, the display engines (178, 179) may perform functions such as a projector. The display engines (178, 179) may further include illumination optics, a light path converter, an image panel, a beam splitter, and projection optics. In one embodiment, the first display engine (178) may be embedded in the first temple of the glasses (175). The second display engine (179) may be embedded in the second temple of the glasses (176). In one embodiment, the virtual image may include content (300).

[0084] In one embodiment, the battery (177), the display engine (178, 179), the memory (180), and at least one processor (190) included in the head-mounted display device (100) may be built into the temples (175, 176) of the glasses. Although FIG. 2 illustrates that the battery (177), the memory (180), and the at least one processor (190) are built into the first temple (175), this is an exemplary structure, and the present disclosure is not limited to the structure illustrated.

[0085] In one embodiment, the nose support (174) may be connected to the frame (171, 172). The nose support (174) is a portion that hangs over the user's nose and supports the head-mounted display device (100) when the user wears the head-mounted display device (100). In one embodiment, the nose support (174) may include a nose bridge and a glasses nose. In addition, the nose bridge and the glasses nose may be formed as an integral part, but are not limited thereto. In one embodiment of the present disclosure, the nose support (174) and the frame (171, 172) may be formed as an integral part.

[0086] In one embodiment, the nose bridge (173) is a support connecting the first frame (171) and the second frame (172). The nose bridge (173) may be connected to the nose support (174). However, if the first frame (171) and the second frame (172) are formed integrally, the head mounted display device (100) may not include the nose bridge (173).

[0087] In one embodiment, the gaze tracking sensor (160) can track the direction of the user's gaze by irradiating light to the eyes (201, 202) of a user wearing a head-mounted display device (100) and receiving the reflected light reflected from the user's eyes (201, 202). The configuration, operation, and function of the gaze tracking sensor (160) will be described later with reference to FIG. 3.

[0088] In one embodiment, a battery (177) may be electrically connected to a camera (130), an eye tracking sensor (160), a memory (180), and at least one processor (190). The battery (177) may supply driving power to the camera (130) and the eye tracking sensor (160).

[0089] In one embodiment, the battery (177) may include at least one battery module composed of a rechargeable secondary battery. The battery (177) may be composed of, but is not limited to, a lithium ion battery (Li-ion Battery), a lithium ion polymer battery (Li-Ion Polymer Battery; LIPB), a nickel cadmium battery (Ni-Cd Battery), or a nickel metal hydride battery (Ni-MH Battery), for example.

[0090] The specific configuration, operation and function of the memory (180) and at least one processor (190) will be described later in FIG. 3.

[0091] FIG. 3 is a block diagram illustrating the configuration of a head mounted display device according to one embodiment of the present disclosure.

[0092] Referring to FIGS. 1, 2, and 3, a head-mounted display device (100) may include a camera (130), a display (140), an optical lens (150), an eye tracking sensor (160), a memory (180), at least one processor (190), a communication interface (191), and an inertial measurement unit (IMU, 192). However, not all of the components illustrated in FIG. 3 are essential components. The head-mounted display device (100) may be implemented with more components than the components illustrated in FIG. 3, or may be implemented with fewer components. In one embodiment, FIG. 3 does not show the configuration of the frame (171, 172), nose bridge (173), nose support (174), temples (175, 176), battery (177), and display engine (178, 179) compared to FIG. 2.

[0093] In one embodiment, a camera (130), a display (140), an optical lens (150), a gaze tracking sensor (160), a memory (180), at least one processor (190), a communication interface (191), and an inertial measurement device (192) may be electrically connected to each other. Hereinafter, the same components as those described in FIGS. 1 and 2 are given the same reference numerals, and redundant descriptions are omitted.

[0094] In one embodiment, the camera (130) can capture images by photographing a physical environment space (1000) of the real world or a physical object of the real world. In one embodiment, the camera (130) can include an RGB camera that can capture images containing RGB information by photographing the real world.

[0095] In one embodiment, the camera (130) may include multiple RGB cameras. At least one processor (190) may acquire depth information using images containing RGB information captured by each of the multiple RGB cameras. Each of the multiple RGB cameras may capture images of the real world from different locations.

[0096] In one embodiment, at least one processor (190) may acquire depth information of the real world based on the positional relationship of each of the plurality of RGB cameras, the distance between a plurality of pixels that capture the same point included in the images acquired from each of the plurality of RGB cameras, and the size of each of the plurality of pixels. In one embodiment, the camera (130) may include a stereo camera including two RGB cameras.

[0097] Additionally, the camera (130) may include an RGB-Depth camera that captures images of the real world and includes RGB information and depth information.

[0098] In one embodiment, the display (140) may include an optical element formed of a transparent material. The display (140) may be formed of a transparent material such that a portion of the back surface is visible when the user (200) wears the head-mounted display device (100). In one embodiment, the display (140) may include a waveguide. The waveguide may be formed of a single-layer or multi-layer flat plate made of a transparent material through which light can be propagated while being reflected within. However, the present disclosure is not limited thereto, and the display (140) may include other types of displays capable of providing a virtual image (e.g., content (300)) to the user.

[0099] In one embodiment of the present disclosure, when the head-mounted display device (100) is a video see-through device, the display (140) may include any one of a liquid crystal display, an organic light emitting diode (OLED) display, and an inorganic light emitting diode (iLED) display.

[0100] In one embodiment, the display (140) faces the emission surface of the display engine (178, 179) and can receive light of a virtual image projected from the display engine (178, 179). The light of the virtual image projected onto the display (140) can propagate within the display (140) by the principle of total reflection. The display (140) can change the path of the light and ultimately provide the virtual image to the user.

[0101] In one embodiment, the optical lens (150) may include a material that allows light provided from the real world toward the head-mounted display device (100) to pass through it. In one embodiment, the optical lens (150) may include a material that allows light that constitutes an image provided through the display (140) to pass through it.

[0102] In one embodiment, the gaze tracking sensor (160) can obtain the gaze direction of the user (200). In one embodiment, the gaze tracking sensor (160) can track the gaze direction of the user by detecting an image of the user's pupil or pupils, or by detecting the direction or amount of reflected light such as near-infrared light reflected from the cornea. In one embodiment, the gaze tracking sensor (160) can include a first gaze tracking sensor (161) and a second gaze tracking sensor (162). The first gaze tracking sensor (161) can obtain the gaze direction of the user's left eye by tracking it. The second gaze tracking sensor (162) can obtain the gaze direction of the user's right eye by tracking it.

[0103] In one embodiment, the gaze tracking sensor (160) may include a light source that irradiates light toward the user's eyes (201, 202) and a sensor that receives reflected light reflected from the user's eyes (201, 202). In this case, the light irradiated toward the user's eyes (201, 202) and the reflected light reflected from the user's eyes may each be infrared rays (IR).

[0104] In one embodiment, the memory (180) may store instructions, data structures, and program codes that can be read by at least one processor (190). In one embodiment, there may be more than one memory (180). In the disclosed embodiments, operations performed by at least one processor (190) may be implemented by executing instructions or codes of a program stored in the memory (180).

[0105] In one embodiment, the memory (180) may include at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a mask ROM, a flash ROM, a hard disk drive (HDD), or a solid state drive (SSD).

[0106] In one embodiment, the memory (180) may store instructions or program codes for performing functions or operations of the head-mounted display device (100). The instructions, algorithms, data structures, program codes, and application programs stored in the memory (180) may be implemented in a programming or scripting language such as, for example, C, C++, Java, or an assembler.

[0107] In one embodiment, the memory (180) may store an image acquisition module (181), a pose detection module (182), a pose comparison module (183), an interface comparison module (184), a space map generation module (185), a position determination module (186), and an interface display module (187). However, not all of the modules illustrated in FIG. 3 are essential modules. The memory (180) may store more or fewer modules than the modules illustrated in FIG. 3. In addition, the memory (180) may store a plurality of preset poses corresponding to at least one interface.

[0108] In one embodiment, a 'module' included in the memory (180) may mean a unit that processes a function or operation performed by at least one processor (190). The 'module' included in the memory (180) may be implemented as software such as instructions, an algorithm, a data structure, or a program code.

[0109] In one embodiment, the image acquisition module (181) may be configured with commands or program codes related to an operation or function of acquiring an image of the real world through the camera (130). The image acquisition module (181) may include commands or program codes related to an operation or function of acquiring an image by photographing the front of the head mounted display device (100) through the first camera (110). The image acquisition module (181) may include commands or program codes related to an operation or function of acquiring an image by photographing the lower part of the head mounted display device (100) through the second camera (120).

[0110] In one embodiment, at least one processor (190) can acquire an image by photographing the real world through the camera (130) by executing commands or program codes of the image acquisition module (181). At least one processor (190) can acquire an image by photographing the user (200) through the camera (130) by executing commands or program codes of the image acquisition module (181). Specifically, at least one processor (190) can acquire an image including the user's hand by photographing the user's (200) movements through the second camera (120) by executing commands or program codes of the image acquisition module (181).

[0111] In one embodiment, the pose detection module (182) may be configured with instructions or program codes related to an operation or function for detecting a pose of a user based on an acquired image. In one embodiment, the pose detection module (182) may be configured with instructions or program codes related to an operation or function for detecting a pose of a user's hand based on the user's hand included in an acquired image.

[0112] In one embodiment, the pose detection module (182) may include commands or program codes related to an operation or function of detecting a feature point of a user (200) included in an acquired image and obtaining location information of the feature point. The pose detection module (182) may include commands or program codes related to an operation or function of detecting a pose of a person based on the detected feature point.

[0113] Here, a "feature point" may refer to a point within an image that is distinct from the surrounding background or easily identifiable. A feature point may include a joint (skeleton), a hand, a face, etc. A "pose" may refer to a shape comprised of at least one feature point. In one embodiment, a pose may refer to a shape corresponding to a plurality of feature points and a shape formed by connecting the multiple feature points with lines.

[0114] In one embodiment, the pose detection module (182) may include commands or program codes related to an operation or function of detecting a plurality of feature points and connecting the detected plurality of feature points to detect a pose of a person. In one embodiment, the pose detection module (182) may include a Pose Estimation model that estimates a pose of a person. In one embodiment, the pose detection module (182) may include models such as OpenPose, AlphaPose, and CPN (Cascaded Pyramid Network). The pose detection module (182) may also detect a pose of a person from an image using a deep learning-based model such as CNN (Convolutional Neural Network), Transformer, and Vision Transformer.

[0115] In one embodiment, at least one processor (190) can detect a feature point of a user (200) included in an acquired image by executing instructions or program code of a pose detection module (182). At least one processor (190) can detect a plurality of feature points from an acquired image by executing instructions or program code of a pose detection module (182), and detect a pose of the user (200) by connecting the detected plurality of feature points.

[0116] In one embodiment, the pose comparison module (183) may be configured with instructions or program codes relating to an operation or function of comparing a detected pose with a plurality of preset poses corresponding to at least one interface. The pose comparison module (183) may be configured with instructions or program codes relating to an operation or function of comparing a detected pose with a plurality of preset poses to determine whether the detected pose corresponds to any one of the plurality of preset poses.

[0117] In one embodiment, the "multiple preset poses" may refer to poses having a specific preset shape. In one embodiment, the plurality of preset poses may include a pose set in the shape of a hand typing on a keyboard. The plurality of preset poses may include a pose set in the shape of a hand using a mouse. The plurality of preset poses may include a pose set in the shape of a hand drawing with a pen. However, the present disclosure is not limited thereto, and the plurality of preset poses may include preset poses in various shapes.

[0118] In one embodiment, the 'multiple preset poses' may be set to correspond to each of the plurality of interfaces. In one embodiment, the plurality of preset poses may be poses set to a specific shape when using a specific interface. The plurality of preset poses may be poses having a specific interface and a shape of a hand when using the interface. In one embodiment, a pose set to a shape of a hand typing may be set to correspond to a keyboard. A pose set to a shape of a hand using a mouse may be set to correspond to a mouse. A pose set to a shape of a hand drawing with a pen may be set to correspond to a drawing pad, which is an object of drawing.

[0119] However, the present disclosure is not limited thereto, and a plurality of preset poses may be set to correspond to an interface that uses preset poses in various shapes.

[0120] In one embodiment, each of the plurality of preset poses may include a plurality of feature points. The plurality of feature points included in each of the plurality of preset poses may include position information corresponding to a specific shape. The pose comparison module (183) may be configured with commands or program codes related to an operation or function that compares the position information of the plurality of feature points included in each of the plurality of preset poses with the position information of the plurality of feature points included in the detected pose to determine whether the detected pose corresponds to any one of the plurality of preset poses.

[0121] In one embodiment, the pose comparison module (183) may be configured with commands or program codes relating to an operation or function that determines that the detected pose corresponds to one of the plurality of preset poses when the difference between the position information of the plurality of feature points included in the detected pose and the position information of the plurality of feature points included in one of the plurality of preset poses is within an error range. At this time, the speed of the pose comparison operation can be increased as the error range is set larger. The accuracy of the pose comparison operation can be increased as the error range is set smaller. In one embodiment, the error range can be set in advance in consideration of the speed and accuracy of the pose comparison operation.

[0122] In one embodiment, at least one processor (190) can compare a detected pose with a plurality of preset poses by executing instructions or program codes of the pose comparison module (183). At least one processor (190) can determine one pose corresponding to the detected pose among the plurality of preset poses by executing instructions or program codes of the pose comparison module (183).

[0123] In one embodiment, the interface comparison module (184) may be configured with commands or program codes related to an operation or function of comparing an interface (hereinafter, a second interface) corresponding to one of a plurality of preset poses determined to correspond to a detected pose with an interface (hereinafter, a first interface) displayed through a display (140).

[0124] In one embodiment, the interface comparison module (184) may be configured with instructions or program codes related to operations or functions that determine whether the first interface and the second interface are identical. In one embodiment, at least one processor (190) may compare the first interface and the second interface by executing instructions or program codes related to the operations or functions of the interface comparison module (184).

[0125] In one embodiment, at least one processor (190) can compare a first interface displayed through a display (140) with a second interface corresponding to any one of the poses determined to correspond to the detected pose by executing instructions or program codes of an interface comparison module (184), thereby determining whether the first interface and the second interface are identical.

[0126] In one embodiment, the image acquisition module (181) may be configured with commands or program codes related to an operation or function of acquiring a spatial image by photographing a space (1000) including a head-mounted display device (100) through a camera (130). The space (1000) including the head-mounted display device (100) may refer to the real world. In one embodiment, the spatial image may include an image acquired by photographing the front of the head-mounted display device (100) through a first camera (110). The spatial image may include an image acquired by photographing the lower part of the head-mounted display device (100) through a second camera (120).

[0127] In one embodiment, the head-mounted display device (100) may include a LiDAR (Light Detection and Ranging) sensor or a RADAR (Radio Detection and Ranging) sensor, etc. The image generation module (181) may be configured with commands or program codes related to an operation or function of obtaining a space image by photographing a space (1000) including the head-mounted display device (100) through the LiDAR sensor or the RADAR sensor.

[0128] In one embodiment, at least one processor (190) can acquire a spatial image through the camera (130) by executing commands or program codes of the image acquisition module (181). In one embodiment, at least one processor (190) can acquire a spatial image through a LiDAR sensor or a RADAR sensor by executing commands or program codes of the image acquisition module (181).

[0129] In one embodiment, the spatial map generation module (185) may be configured with instructions or program codes related to an operation or function of generating a spatial map representing a space (1000) including a head mounted display device (100) based on an acquired spatial image. In one embodiment, the 'spatial map' is a map including information about the space (1000) including the head mounted display device (100), and may be a map generated based on background, type of object, location of object, shape of object, depth information of background and object, etc. included in the spatial image.

[0130] In one embodiment, the spatial map generation module (185) may include a Simultaneous Localization and Mapping (SLAM) algorithm. The spatial map generation module (185) may be configured with instructions or program codes related to operations or functions for generating a spatial map by applying a spatial image to the SLAM algorithm. At least one processor (190) may execute the instructions or program codes of the spatial map generation module (185) to generate a spatial map by applying a spatial image to the SLAM algorithm.

[0131] However, the present disclosure is not limited thereto, and the space map generation module (185) may include other types of models or algorithms for generating a space map representing a space (1000) including a head mounted display device (100).

[0132] In one embodiment, the position determination module (186) may be configured with instructions or program codes relating to an operation or function for determining an interface position at which the interface is to be displayed based on the generated space map and the detected pose. At this time, the 'interface position' may refer to the position, size, arrangement, etc. of the interface at which the interface is to be displayed on the display (140) so that the user sees that the interface is located in a specific area within the real space (1000) or is overlaid with a specific object included in the real space (1000). At this time, the fact that the interface appears to be overlaid with a specific object included in the real space (1000) may mean that the user sees that the interface displayed on the display (140) is placed on top of a specific object included in the real object (1000).

[0133] In one embodiment, the position determination module (186) may be configured with instructions or program codes relating to an operation or function for determining an interface position based on at least one of the position of the object in the real world, the shape of the object, the type of the interface corresponding to the detected pose, the shape of the interface corresponding to the detected pose, or the position information of the detected pose in the real world, based on the generated space map and the detected pose.

[0134] In one embodiment, the position determination module (186) may be configured with instructions or program codes for an operation or function to determine the interface position based on a 3 Degree of Freedom (3 DoF) measurement including a 3-axis angular velocity value (roll, yaw, pitch) obtained by a spatial map, a detected pose, and an inertial measurement device (192), or a 6 Degree of Freedom (6 DoF) measurement including a 3-dimensional position coordinate value (x-axis, y-axis, and z-axis coordinate value) of the user (200) and a 3-axis angular velocity value (roll, yaw, pitch). In one embodiment, the position determination module (186) may determine the interface position so that the position of the interface is displayed in response to a change in the 3DoF measurement value or so that the position of the interface is displayed in response to a change in the 6DoF measurement value in response to a movement of the user (200) wearing the head mounted display device (100). Accordingly, the head-mounted display device (100) can provide an interface that supports 3DoF or an interface that supports 6DoF. 3DoF and 6DoF will be described later in the inertial measurement device (192).

[0135] In one embodiment, the position determination module (186) may be configured with instructions or program codes related to an operation or function for determining a content position at which the content is to be displayed based on the acquired user's viewpoint. At this time, the 'content position' may refer to the position, size, arrangement, etc. of the content (300) displayed on the display (140) so that the content appears to be located in a specific area within the real space (1000) or overlaid with a specific object included in the real space (1000) to the user (200). At this time, the fact that the content appears to be overlaid with a specific object included in the real space (1000) may mean that the content displayed on the display (140) appears to the user to be placed on top of a specific object included in the real object (1000) or positioned in front of a specific object.

[0136] In one embodiment, the interface display module (187) may be configured with commands or program codes related to an operation or function of displaying an interface through the display (140). In one embodiment, the interface display module (187) may be configured with commands or program codes related to an operation or function of displaying an interface at a determined interface location through the display (140).

[0137] In one embodiment, at least one processor (190) can display an interface through the display (140) by executing instructions or program code of the interface display module (187). At least one processor (190) can display an interface at a determined interface location through the display (140) by executing instructions or program code of the interface display module (187).

[0138] In one embodiment, the memory (180) may further store a content display module comprising instructions or program codes related to operations or functions for displaying content through the display (140). At least one processor (190) may display content through the display (140) by executing the instructions or program codes of the content display module.

[0139] In one embodiment, at least one processor (190) may be configured as at least one of a Central Processing Unit, a microprocessor, a Graphic Processing Unit, an Application Processor (AP), an Application Specific Integrated Circuits (ASICs), a Digital Signal Processor (DSPs), a Digital Signal Processing Device (DSPDs), a Programmable Logic Device (PLDs), a Field Programmable Gate Array (FPGAs), and a Neural Processing Unit or an artificial intelligence (AI) processor designed with a hardware structure specialized for learning and processing an artificial intelligence (AI) model, but is not limited thereto.

[0140] In one embodiment of the present disclosure, if one or more processors included in at least one processor (190) are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0141] In one embodiment of the present disclosure, at least one processor (190) may be configured as a circuit (Circuitry) such as a System on Chip (SoC) or an Integrated Circuit (IC).

[0142] In one embodiment of the present disclosure, at least one processor (190) can execute various types of modules stored in the memory (180). At least one processor (190) can execute at least one instruction constituting the various types of modules stored in the memory (180). By executing the program or at least one instruction stored in the memory (180), at least one processor (190) can process data according to predefined operation rules or artificial intelligence models.

[0143] In one embodiment of the present disclosure, at least one processor (190) can execute at least one module among an image acquisition module (181), a pose detection module (182), a pose comparison module (183), an interface comparison module (184), a space map generation module (185), a position determination module (186), or an interface display module (187) stored in a memory (180).

[0144] In one embodiment of the present disclosure, at least one processor (190) may include a plurality of processors. In one embodiment of the present disclosure, at least one module among an image acquisition module (181), a pose detection module (182), a pose comparison module (183), an interface comparison module (184), a space map generation module (185), a position determination module (186), or an interface display module (187) stored in a memory (180) may be executed by any one of the plurality of processors.

[0145] In one embodiment, at least one processor (190) may control the operation of a head mounted display device (100) displaying an interface by executing at least one instruction stored in a memory (180).

[0146] In one embodiment, the communication interface (191) can perform data communication with an external server under the control of at least one processor (190). In addition, the communication interface (191) can perform data communication not only with the external server but also with other peripheral electronic devices.

[0147] In one embodiment, the communication interface (191) may perform data communication with a server or other peripheral electronic devices using at least one of data communication methods including, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, zigbee, Wi-Fi Direct (WFD), infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Wireless Broadband Internet (Wibro), World Interoperability for Microwave Access (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliance (WiGig), and RF communication.

[0148] In one embodiment, at least one processor (190) may receive a spatial map of the real world from an external server or peripheral electronic devices via a communication interface (191).

[0149] In one embodiment, the inertial measurement unit (IMU) 192 may include at least one of an accelerometer, a gyroscope, or a magnetometer. The inertial measurement unit (IMU) (192) may be a sensor configured to measure at least one of a moving speed, a direction, an angle, or a gravitational acceleration of the head-mounted display device (100) through at least one of the accelerometer, the gyroscope, or the magnetometer.

[0150] In one embodiment, when a user (200) wearing a head mounted display device (100) moves, at least one processor (190) can measure at least one of a moving speed, a direction, an angle, or a gravitational acceleration of the head mounted display device (100) through an inertial measurement device (192). In one embodiment, the head mounted display device (100) can obtain a 3 DoF (3 Degree of Freedom) measurement value using the inertial measurement device (192). In addition, the head mounted display device (100) can also obtain a 6 DoF (6 Degree of Freedom) measurement value using the inertial measurement device (192).

[0151] FIG. 4 is a flowchart illustrating an operation method of a head-mounted display device according to one embodiment of the present disclosure. FIG. 5 is a flowchart illustrating an operation of displaying a first interface and a second interface while first content is displayed according to one embodiment of the present disclosure.

[0152] Referring to FIGS. 1, 3, and 4, in one embodiment, FIG. 4 illustrates a method of operating a head-mounted display device (100) while an interface is displayed through a display (140). In one embodiment, an interface (500) displayed through the display (140) is referred to as a 'first interface (500)', and an interface (510) displayed through the display (140) according to a detected pose to be described later is referred to as a 'second interface (510)'.

[0153] In one embodiment, the first interface (500) may be an interface that detects a user's motion from an image obtained by photographing a user (200) according to the present disclosure, compares the detected motion with a plurality of preset poses, and displays an interface corresponding to a corresponding pose through the display (140). However, the present disclosure is not limited thereto, and the first interface (500) may also be displayed through the display (140) by another method, such as a user input provided through a user interface or the like.

[0154] In one embodiment, a method of operating a head mounted display device (100) may include a step (S100) of capturing an image of a user by using a camera (130).

[0155] In one embodiment, in the step (S100) of capturing an image by photographing a user, at least one processor (190) may capture an image by photographing the user through a camera (130) by executing commands or program codes of an image acquisition module (181).

[0156] In one embodiment, a method of operating a head mounted display device (100) may include a step (S200) of detecting a pose of a user based on an acquired image.

[0157] In one embodiment, in the step of detecting a user's pose (S200), at least one processor (190) can detect the user's pose from the acquired image by executing commands or program codes of the pose detection module (182).

[0158] In one embodiment, in the step (S200) of detecting a pose of a user, at least one processor (190) may detect, estimate, or segment a user included in an acquired image by executing commands or program codes of a pose detection module (182). At least one processor (190) may detect a plurality of feature points of the detected, estimated, or segmented user by executing commands or program codes of the pose detection module (182).

[0159] In one embodiment, in the step (S200) of detecting a pose of a user, at least one processor (190) may detect a pose of the user based on a plurality of detected feature points by executing commands or program codes of a pose detection module (182). In one embodiment, when the acquired image includes a hand of the user, at least one processor (190) may detect a pose of the hand of the user based on a plurality of feature points detected from the hand of the user.

[0160] In one embodiment, the operating method of the head mounted display device (100) may include a step (S300) of comparing a detected pose (410) with a plurality of preset poses corresponding to at least one interface.

[0161] In one embodiment, in the step (S300) of comparing a plurality of preset poses with the detected pose (410), at least one processor (190) can compare the detected pose (410) with a plurality of preset poses that correspond to each of the plurality of interfaces by executing commands or program codes of the pose comparison module (183).

[0162] In one embodiment, the operating method of the head mounted display device (100) may include a step (S400) of determining whether a detected pose (410) corresponds to any one of a plurality of preset poses.

[0163] In one embodiment, in the step (S400) of determining whether the detected pose (410) corresponds to any one of a plurality of preset poses, at least one processor (190) may compare the detected pose (410) with a plurality of preset poses by executing commands or program codes of the pose comparison module (183), thereby determining whether a pose corresponding to the detected pose (410) exists among the plurality of poses.

[0164] Although FIG. 4 illustrates a step (S300) of comparing a detected pose (410) with a plurality of preset poses corresponding to at least one interface and a step (S400) of determining whether the detected pose (410) corresponds to any one of the plurality of preset poses, the present disclosure is not limited thereto. The operation of comparing a plurality of preset poses with a detected pose (410) and the operation of determining whether the detected pose (410) corresponds to any one of the plurality of preset poses may be performed in a single step.

[0165] In one embodiment, in step S400, when it is determined that the detected pose (410) corresponds to one of a plurality of preset poses, the operating method of the head mounted display device (100) may include a step (S500) of comparing a first interface (500) displayed on the display (140) with a second interface (510) corresponding to one of the poses.

[0166] In one embodiment, at step S500, at least one processor (190) can compare the first interface (500) and the second interface (510) by executing instructions or program codes of the interface comparison module (184).

[0167] In one embodiment, since the first interface (500) and the second interface (510) are determined to be different interfaces in step S500, the operating method of the head mounted display device (100) may include a step (S600) of displaying the second interface (510) through the display (140).

[0168] In one embodiment, at least one processor (190) may display the second interface (510) through the display (140) by executing commands or program codes of the interface display module (187) in the step (S600) of displaying the second interface (510) through the display (140).

[0169] In one embodiment, in the step (S600) of displaying the second interface (510) through the display (140), at least one processor (190) may replace the first interface (500) displayed through the display (140) with the second interface (510) and display it. In one embodiment, in the step (S600) of displaying the second interface (510) through the display (140), at least one processor (190) may execute commands or program codes of the interface display module (187) to replace the first interface (500) displayed on the display (140) with the second interface (510) corresponding to the detected pose (410) and display it, thereby providing it to the user (200).

[0170] In one embodiment, since the pose (410) detected in step S400 is different from a plurality of preset poses and it is determined that the detected pose (410) does not correspond to the plurality of poses, the operating method of the head mounted display device (100) may include a step (S700) of maintaining the first interface (500) displayed through the display (140).

[0171] In one embodiment, when it is determined that the pose (410) detected in step S400 is different from a plurality of preset poses, at least one processor (190) may execute commands or program codes of the interface display module (187) to maintain the first interface (500) displayed through the display (140) and provide it to the user (200).

[0172] In one embodiment, if the detected pose does not correspond to a plurality of preset poses, the head-mounted display device (100) may determine that the user's (200) action is not an action intended to change the interface displayed on the display (140). Accordingly, the head-mounted display device (100) may continuously maintain the first interface (500) displayed through the display (140). This may prevent the interface displayed through the display (140) from being unnecessarily changed due to the user's (200) action.

[0173] In one embodiment, when the first interface (500) and the second interface (510) are determined to be the same interface in step S500, at least one processor (190) can maintain the first interface (500) displayed through the display (140) and provide it to the user (200) by executing instructions or program codes of the interface display module (187).

[0174] In one embodiment, the head mounted display device (100) may determine that the user's (200) action is an action to use the first interface (500) if it is determined that the second interface corresponding to the detected pose is the same interface as the first interface (500) displayed through the display (140). Accordingly, the head mounted display device (100) may continuously maintain the first interface (500) displayed through the display (140). Through this, it is possible to prevent the interface displayed through the display (140) from being changed while the user (200) is using the first interface (500).

[0175] Referring to FIGS. 1, 3, 4, and 5, in one embodiment, a method of operating a head-mounted display device (100) may include a step (S50) of displaying content (300) on a display (140).

[0176] In one embodiment, in the step (S50) of displaying content (300) on the display (140), the displayed content (300) is referred to as 'first content'.

[0177] In one embodiment, the first content is content that is displayed through the display (140) and provided to the user (200), and may include an image or a video, and is not limited to any one of an online lecture, an Internet browser, a game, a chat screen, a map, a book, an advertisement, a movie, a video chat, etc.

[0178] In one embodiment, the operation of step (S100) of capturing an image by photographing a user through a camera (130) may be performed after the operation of step (S50) of displaying content (300) on a display (140). In one embodiment, the operating method described in FIG. 4 may be performed after the operation of step (S50) of displaying content (300) on a display (140). In one embodiment, the operation of displaying the first interface (500) and the second interface (510) through the display (140) may be performed while the first content (300) is displayed through the display (140).

[0179] In one embodiment, the user (200) may display the first interface (500) on the display (140) by only taking a specific pose (e.g., a pose of a hand using a keyboard, a pose of a hand using a mouse) while the first content (300) is displayed on the display (140), or may change the first interface (500) displayed on the display (140) to a second interface (510) different from the first interface (500). Through this, the user's (200) convenience in using the head mounted display (100) may be improved.

[0180] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the present disclosure.

[0181] FIG. 6 is a diagram for explaining an operation of displaying an interface corresponding to a detected user's pose according to one embodiment of the present disclosure.

[0182] Referring to FIGS. 1, 3, 4, and 6, in one embodiment, at least one processor (190) may capture an image of a user through a second camera (120) by executing commands or program codes of an image acquisition module (181). Although FIG. 6 illustrates capturing a user through the second camera (120), the present disclosure is not limited thereto. Depending on the location of the user's motion to be captured, the arrangement of the camera (130), and the performance of the camera (130), the user's motion may be captured through the first camera (110) or another camera.

[0183] In one embodiment, at least one processor (190) may detect a feature point (600, 610) of a user included in an acquired image by executing instructions or program codes of a pose detection module (182). In one embodiment, if the acquired image includes a user's hand, at least one processor (190) may detect a feature point (600, 610) of the user's hand.

[0184] In one embodiment, at least one processor (190) can detect a pose of a user (200) based on detected feature points (600, 610) by executing instructions or program code of a pose detection module (182).

[0185] In one embodiment, when the second camera (120) captures the motion of a user's hand typing on a keyboard, the detected feature point may be a first feature point (600). In one embodiment, at least one processor (190) may detect the user's pose based on the detected first feature point (600) by executing commands or program codes of the pose detection module (182). In one embodiment, the pose detected from the first feature point (600) may be referred to as a first pose (410).

[0186] In one embodiment, at least one processor (190) may compare the detected first pose (410) with a plurality of preset poses by executing instructions or program codes of the pose comparison module (183). At least one processor (190) may determine that the detected first pose (410) corresponds to any one of the plurality of preset poses, which is set as a pose of a hand typing using a keyboard.

[0187] In one embodiment, at least one processor (190) may execute instructions or program codes of a pose comparison module (183) to determine that the detected first pose (410) corresponds to a pose of a hand typing using a keyboard among a plurality of preset poses, and may determine an interface corresponding to the detected first pose (410) as a keyboard (500).

[0188] In one embodiment, at least one processor (190) can display a keyboard (500) through the display (140) by executing instructions or program code of the interface display module (187) when there is no interface being displayed through the display (140).

[0189] In one embodiment, at least one processor (190) can determine whether the interface displayed through the display (140) is the same as the keyboard (500) by executing instructions or program codes of the interface comparison module (184) when there is an interface displayed through the display (140).

[0190] In one embodiment, at least one processor (190) may change the interface displayed through the display (140) to the keyboard (500) if the interface displayed through the display (140) is different from the keyboard (500) and display the same. At least one processor (190) may maintain the keyboard (500) displayed through the display (140) if the interface displayed through the display (140) is the keyboard (500).

[0191] In one embodiment, when the second camera (120) captures the motion of the user's hand drawing a picture using a pen, the detected feature point may be a second feature point (610). In one embodiment, at least one processor (190) may detect the user's pose based on the detected second feature point (610) by executing commands or program codes of the pose detection module (182). In one embodiment, the pose detected from the second feature point (610) may be referred to as a second pose (420).

[0192] In one embodiment, FIG. 6 depicts a user holding a pen and drawing, but the present disclosure is not limited thereto. The user may also draw using a stick-shaped object, not just a pen. Furthermore, the user may draw using the pen without holding anything. In this case, an image of the pen may be displayed on the display (140) and provided to the user (200).

[0193] In one embodiment, at least one processor (190) may determine that the detected second pose (420) corresponds to one of a plurality of preset poses set as a pose of a hand drawing with a pen by executing instructions or program codes of the pose comparison module (183).

[0194] In one embodiment, at least one processor (190) may execute instructions or program codes of a pose comparison module (183) to determine that the detected second pose (420) corresponds to a pose of a hand drawing with a pen among a plurality of preset poses, and may determine an interface corresponding to the detected second pose (420) as a drawing pad (510).

[0195] In one embodiment, at least one processor (190) can determine whether the interface being displayed through the display (140) is identical to the drawing pad (510) by executing instructions or program codes of the interface comparison module (184) when there is an interface being displayed through the display (140).

[0196] In one embodiment, at least one processor (190) may change the keyboard (500) displayed through the display (140) to a drawing pad (510) when the interface displayed through the display (140) is a keyboard (500).

[0197] FIG. 7 is a flowchart illustrating an operation for determining an interface position based on a spatial map generated through a camera and a detected pose, according to one embodiment of the present disclosure. Hereinafter, steps identical to those described in FIG. 4 are assigned the same reference numerals, and redundant descriptions are omitted.

[0198] Referring to FIGS. 1, 3, 4, 6, and 7, in one embodiment, the method of operating the head-mounted display device (100) may include a step (S610) of capturing a space (1000) including the head-mounted display device (100) through a camera (130) to obtain a space image.

[0199] In one embodiment, in the step of acquiring a spatial image (S610), at least one processor (190) can acquire a spatial image by photographing a space (1000) including a head-mounted display device (100) by executing commands or program codes of an image acquisition module (181).

[0200] In one embodiment, the operation of the step (S610) of acquiring a spatial image may be executed after the first interface (500) displayed on the display (140) and the second interface (510) corresponding to a pose are determined to be different interfaces in the step (S500). However, the present disclosure is not limited thereto. The operation of the step (S610) of acquiring a spatial image may be executed together with the operation of the step (S100) of acquiring an image by photographing the user's motion, or may be executed before the operation of the step (S500) of comparing the first interface (500) and the second interface (510).

[0201] In one embodiment, the operating method of the head-mounted display device (100) may include a step (S620) of generating a spatial map representing a space (1000) based on an acquired spatial image. In one embodiment, in the step (S620) of generating a spatial map, at least one processor (190) may generate a spatial map representing a space (1000) based on the spatial image by executing instructions or program codes of a spatial map generation module (185).

[0202] In one embodiment, the operating method of the head mounted display device (100) may include a step (S630) of determining an interface position at which an interface is to be displayed based on the generated space map and the detected pose. In one embodiment, in the step (S630) of determining the interface position, an interface position at which a second interface (510) corresponding to the detected second pose (420) is to be displayed may be determined. However, the present disclosure is not limited thereto, and when the first interface (500) is also displayed corresponding to the detected pose of the user, the step (S630) of determining the interface position at which the interface is to be displayed may also determine an interface position at which the first interface (500) corresponding to the detected first pose (410) is to be displayed.

[0203] In one embodiment, in the step (S630) of determining an interface position at which to display an interface, at least one processor (190) may determine an interface position at which to display an interface by executing instructions or program codes of a position determination module (186). At least one processor (190) may determine at least one of an interface position at which to display a first interface (500) or an interface position at which to display a second interface (510) by executing instructions or program codes of a position determination module (186).

[0204] In one embodiment, the method of operating the head mounted display device (100) may include a step (S640) of displaying an interface at a determined interface location through the display (140).

[0205] In one embodiment, in the step (S640) of displaying the interface at the determined interface location, the second interface (510) may be displayed at the determined interface location so as to display the second interface (510) through the display (140). However, the present disclosure is not limited thereto, and in the step (S640) of displaying the interface at the determined interface location, the first interface (500) may be displayed at the determined interface location so as to display the first interface (500) through the display (140).

[0206] In one embodiment, in the step (S640) of displaying an interface at a determined interface location, at least one processor (190) may display a first interface (500) or a second interface (510) at the determined interface location by executing instructions or program codes of an interface display module (187).

[0207] FIG. 8 is a flowchart illustrating an operation of acquiring a user's viewpoint through a gaze tracking sensor and detecting the user's pose based on the acquired viewpoint and acquired image, according to one embodiment of the present disclosure. Hereinafter, steps identical to those described in FIG. 4 are assigned the same reference numerals, and any redundant descriptions are omitted.

[0208] Referring to FIGS. 1, 3, 4, and 8, in one embodiment, the operating method of the head-mounted display device (100) may include a step (S60) of acquiring the user's viewpoint based on the user's gaze direction tracked through the gaze tracking sensor (160).

[0209] In one embodiment, in the step (S60) of acquiring the user's viewpoint, based on the acquired direction of the user's gaze, the point where the gazes of the user's two eyes (201, 202) intersect each other can be calculated, and the corresponding point can be acquired as the viewpoint. In one embodiment, in the step (S60) of acquiring the user's viewpoint, based on the image acquired through the camera (130) and the acquired direction of the user's gaze, the user's viewpoint can be acquired based on the distance between the head-mounted display device (100) and the surrounding environment and the direction of the user's gaze.

[0210] In one embodiment, in the step (S60) of acquiring the user's viewpoint, at least one processor (190) may acquire the user's viewpoint based on the user's gaze direction by executing instructions or program code for acquiring the user's viewpoint.

[0211] In one embodiment, the method of operating the head-mounted display device (100) may include a step (210) of detecting a pose of the user based on the acquired viewpoint of the user and the acquired image. In one embodiment, in the step (S210) of detecting the pose of the user, the pose of the user may be detected based on the viewpoint of the user acquired in the step (S60) of acquiring the viewpoint of the user and the image acquired in the step (S100) of acquiring the image.

[0212] In one embodiment, at least one processor (190) can detect a pose of a user based on the acquired viewpoint of the user and the acquired image by executing instructions or program codes of the pose detection module (182). In one embodiment, at least one processor can determine that a motion of the user to be detected exists in an area where the acquired viewpoint of the user is located among the acquired images, and can detect the pose of the user among the acquired images by assigning a relatively high weight to the area.

[0213] In one embodiment, if a user is performing a keyboard-typing motion with his or her hand and the user's gaze is directed toward the hand performing the keyboard-typing motion, the acquired user's viewpoint may correspond to an area of ​​the acquired image where the hand is present, such that the hand performing the keyboard-typing motion is included. In this case, at least one processor (190) may detect the user's pose by focusing on the user's hand in the acquired image.

[0214] FIG. 9 is a flowchart illustrating an operation for determining a content location for displaying second content based on a user's viewpoint acquired through a gaze tracking sensor, according to one embodiment of the present disclosure. The same steps as those described in FIG. 8 are assigned the same reference numerals, and any redundant descriptions are omitted.

[0215] Referring to FIGS. 1, 3, and 9, in one embodiment, the operating method of the head-mounted display device (100) may include, after the step of acquiring a user's viewpoint (S60), a step of determining a content location at which content is to be displayed based on a user input acquired through an interface based on the acquired user's viewpoint (S800).

[0216] In one embodiment, the content displayed based on user input obtained through the interface is referred to as 'second content'.

[0217] In one embodiment, the second content (730, see FIG. 10a) may be content that displays user input entered by a user through an interface. The second content (730) may be content that displays user input entered by a user through at least one of the first interface (500) and the second interface (510). In one embodiment, the second content (730) may include, but is not limited to, a notepad, a notebook, or a mouse cursor.

[0218] In one embodiment, the second content (730) displayed through the display (140) may display user input entered by the user (200) through the interface. The user (200) may confirm the user input entered through the second content (730) displayed through the display (140), thereby improving user convenience.

[0219] In one embodiment, in the step (S800) of determining the content location at which the second content (730) is to be displayed based on the acquired user's viewpoint, the content location at which the second content (730) is to be displayed may be determined based on the acquired user's viewpoint. In one embodiment, in the step (S800) of determining the content location at which the second content (730) is to be displayed based on the acquired user's viewpoint, the content location at which the second content (730) is to be displayed may be determined based on the acquired user's viewpoint and the first content (300) displayed on the display (140).

[0220] In one embodiment, in the step (S800) of determining a content location to display second content (730) based on the acquired user's viewpoint, at least one processor (190) may execute instructions or program codes of the location determination module (186) to determine a content location to display second content (730) based on the acquired user's viewpoint.

[0221] Below, the operation of determining the content location at which the second content (730) is to be displayed will be described later with reference to FIGS. 10a and 10b.

[0222] In one embodiment, the operating method of the head-mounted display device (100) may include a step (S900) of displaying second content (730) at a content location determined through the display (140). In one embodiment, in the step (S900) of displaying second content (730) at the content location determined through the display (140), at least one processor (190) may display the second content (730) at the determined content location by executing instructions or program codes for displaying the second content (730) through the display (140).

[0223] In one embodiment, the operation of the step (S800) of determining a content location at which the second content (730) is to be displayed and the operation of the step (S900) of displaying the second content (730) at the content location through the display (140) may be performed together with the operation of displaying at least one of the first interface (500) or the second interface (510) through the display (140).

[0224] In one embodiment, at least one processor (190) can determine an interface location at which at least one of the first interface (500) or the second interface (510) is to be displayed, and determine a content location at which the second content (730) is to be displayed. The at least one processor (190) can display at least one of the first interface (500) or the second interface (510) at the determined interface location through the display (140), and display the second content (730) at the determined content location.

[0225] FIG. 10A is a diagram illustrating an operation for determining a content location for displaying second content based on a user's viewpoint acquired through an eye tracking sensor, according to one embodiment of the present disclosure. FIG. 10B is a diagram illustrating an operation for determining a content location for displaying second content based on a user's viewpoint acquired through an eye tracking sensor, according to one embodiment of the present disclosure.

[0226] Referring to FIGS. 1, 3 and 10A, in one embodiment, FIG. 10A illustrates first content (300) and second content (730) displayed through a display (140).

[0227] In one embodiment, FIG. 10A illustrates that the acquired user's viewpoint (700) overlaps with the first content (300). In one embodiment, the user (200) may look at the first content (300) and perform a specific action using a body structure such as a hand. In one embodiment, the user (200) may look at the first content (300) and assume a pose such as typing on a keyboard with his or her hand, moving a mouse, or drawing with a pen.

[0228] In one embodiment, at least one processor (190) can display an interface (e.g., a keyboard, mouse, or drawing pad) corresponding to a pose detected by a user's motion at a determined interface location through a display (140).

[0229] In one embodiment, at least one processor (190) may display on the display (140) second content (730) that represents user input provided by a user through an interface displayed on the display (140). At this time, at least one processor (190) may determine a content location at which to display the second content (730) so that the second content (730) is not displayed overlapping with the first content (300) by executing instructions or program codes of the location determination module (186).

[0230] In one embodiment, at least one processor (190) may calculate a first distance (710) from the acquired user's viewpoint (700) to an area adjacent to the first content (300) by executing instructions or program codes of the location determination module (186). In one embodiment, the first distance (710) may be the shortest distance from the acquired user's viewpoint (700) to an area that does not overlap with the first content (300).

[0231] In one embodiment, at least one processor (190) may calculate a value obtained by adding a first distance (710) calculated at the acquired user's point of view (700) to a preset second distance (720) by executing instructions or program codes of the position determination module (186). In this case, the preset second distance (720) may be a preset distance such that the first content (300) and the second content (730) are displayed spaced apart from each other on the display (140) so that the first content (300) and the second content (730) are distinguished from each other.

[0232] In one embodiment, at least one processor (190) may determine a content location to display second content (730) based on a calculated value by adding a first distance (710) calculated from the acquired user's viewpoint (700) and a preset second distance (720).

[0233] Through this, it is possible to confirm user input entered by the user (200) through the interface without interfering with the user experience of the user (200) viewing the first content (300) displayed through the display (140).

[0234] At this time, FIG. 10A illustrates that the second content (730) is displayed on the right side of the first content (300) on the display (140), and the first distance (710) is illustrated as the shortest distance in the right direction from the user's viewpoint (700) that was acquired to an area that does not overlap with the first content (300). However, the present disclosure is not limited thereto, and the second content (730) may be displayed in the left, upper, lower, or diagonal direction of the first content (300). At this time, the first distance (710) may be the shortest distance from the user's viewpoint (700) that was acquired to the direction in which the second direction (730) is displayed based on the first content (300).

[0235] In one embodiment, the direction in which the second content (730) is displayed relative to the first content (300) may be set based on at least one of the type, shape, and size of the first content (300), the type, shape, and size of the second content (730), and the shape or size of the display (140).

[0236] Referring to FIGS. 10A and 10B , in one embodiment, FIG. 10B illustrates that the acquired user's viewpoint (700) does not overlap with the first content (300). In one embodiment, the user (200) may look at an area other than the first content (300) and perform a specific action using a hand or the like.

[0237] In one embodiment, at least one processor (190) may determine a content location at which to display second content (730) based on the acquired user's viewpoint (700) by executing instructions or program codes of the location determination module (186). At this time, if it is determined that the second content (730) displayed based on the acquired user's viewpoint (700) overlaps with the first content (300) based on a preset second distance (720), the at least one processor (190) may determine a content location at which to display the second content (730) based on a value obtained by adding the second distance (720) to the acquired user's viewpoint (700).

[0238] However, the present disclosure is not limited thereto. In one embodiment, at least one processor (190) may determine a content location to display the second content (730) so that the second content (730) is displayed overlapping the first content (300) based on at least one of the type, shape, and size of the first content (300), the type, shape, and size of the second content (730), and the shape or size of the display (140) by executing instructions or program codes of the position determination module (186).

[0239] FIG. 11A is a diagram illustrating an operation of displaying first content and second content according to an embodiment of the present disclosure. FIG. 11B is a diagram illustrating an operation of displaying first content and second content according to an embodiment of the present disclosure. FIG. 11C is a diagram illustrating an operation of displaying first content and second content according to an embodiment of the present disclosure. Hereinafter, the same components as those described in FIG. 1 are assigned the same reference numerals, and redundant descriptions are omitted.

[0240] Referring to FIG. 1, FIG. 10A and FIG. 11A, in one embodiment, FIG. 11A illustrates a head-mounted display device (100) detecting a pose (400) of a user typing using a keyboard, and providing a keyboard (500) to the user (200) through a display (140) based on the detected pose (400).

[0241] In one embodiment, the head-mounted display device (100) may display, through the display (140), first content (300), a keyboard (500), and second content (730) indicating user input input by a user (200) using the keyboard (500). In one embodiment, even if the user (200) provides user input while looking at the first content (300) and not at the keyboard (500), the user (200) may confirm the user input input by the user through the second content (730) displayed adjacent to the first content (300).

[0242] In one embodiment, the user (200) may experience the first content (300), the second content (730), and the keyboard (500) displayed on the display (140) as being located within a real space (1000). In one embodiment, FIG. 11A illustrates the first virtual content (301), the second virtual content (731), and the virtual keyboard (501) displayed on the display (140) as being located within the real space (1000) or appearing to be overlaid on a real object, as experienced by the user (200) through the head-mounted display device (100) and in the field of view of the user (200).

[0243] In one embodiment, the head mounted display device (100) can detect the user's pose (400) through the second camera (120) and display a keyboard (500), which is an interface corresponding to the detected pose (400), through the display (140), even if the user (200) takes a pose of typing using a keyboard while looking at the first content (300). Referring to FIGS. 1, 10A, 11A, and 11B, in one embodiment, FIG. 11B illustrates that the head mounted display device (100) detects the user's pose (410) of drawing using a pen, and provides a drawing pad (510) to the user (200) through the display (140) based on the detected pose (410).

[0244] In one embodiment, the head-mounted display device (100) may display, through the display (140), third content (740) that represents user input input by the user (200) using the picture pad (510), in addition to the first content (300) and the picture pad (510). In one embodiment, even if the user (200) provides user input while looking at the first content (300) and not at the picture pad (510), the user (200) may confirm the user input input by the user through the third content (740) that is displayed adjacent to the first content (300).

[0245] In one embodiment, the head mounted display device (100) can detect the user's pose (410) through the second camera (120) and display a drawing pad (510), which is an interface corresponding to the detected pose (410), through the display (140), even if the user (200) takes a pose of drawing using a pen while looking at the first content (300).

[0246] In one embodiment, compared to FIG. 11A, if the head mounted display device (100) determines that the interface corresponding to the detected pose (410) is a drawing pad (510), the head mounted display device (100) may change the keyboard (500) displayed through the display (140) to the drawing pad (510) and display it. Through this, the user (200) may change his / her pose to correspond to any one of a plurality of preset poses while using the first content (300), thereby allowing the head mounted display device (100) to provide a desired interface.

[0247] In one embodiment, the user (200) may experience the first content (300), the third content (740), and the picture pad (510) displayed on the display (140) as being located within a real space (1000). In one embodiment, FIG. 11B illustrates the first virtual content (301), the third virtual content (741), and the virtual picture pad (511) as being located within a real space (1000) or appearing to be overlaid on a real object, as seen by the user (200) through the head-mounted display device (100) and in the field of view of the user (200).

[0248] Referring to FIGS. 1, 10A, 11A and 11C, in one embodiment, FIG. 11C illustrates a head-mounted display device (100) detecting a pose (420) of a user controlling a mouse cursor using a touch pad, and providing a touch pad (520) to a user (200) through a display (140) based on the detected pose (420).

[0249] In one embodiment, the head-mounted display device (100) may display, through the display (140), the first content (300), the touchpad (520), and, in addition, fourth content (750) indicating user input input by the user (200) using the touchpad (520). In one embodiment, even if the user (200) provides user input while looking at the first content (300) and not at the touchpad (520), the user (200) may confirm the user input input through the fourth content (750) displayed adjacent to the first content (300). In one embodiment, the fourth content (750) may be a mouse cursor controlled by the user input provided by the user (200) through the touchpad (520). In one embodiment, the fourth content (750) may be displayed overlapping the first content (300).

[0250] In one embodiment, the head-mounted display device (100) can detect the user's pose (420) through the second camera (120) and display the touch pad (520), which is an interface corresponding to the detected pose (420), through the display (140), even if the user (200) assumes a pose of looking at the first content (300) and controlling the mouse cursor using the touch pad.

[0251] In one embodiment, compared to FIG. 11A, if the head-mounted display device (100) determines that the interface corresponding to the detected pose (420) is the touch pad (520), the keyboard (500) displayed through the display (140) can be changed to the touch pad (520) and displayed. Through this, the user (200) can change his / her pose to correspond to any one of a plurality of preset poses while using the first content (300), thereby allowing the head-mounted display device (100) to provide a desired interface.

[0252] In one embodiment, the user (200) may experience the first content (300), the fourth content (750), and the touch pad (520) displayed on the display (140) as being located within a real space (1000). In one embodiment, FIG. 11C illustrates the first virtual content (301), the fourth virtual content (751), and the virtual picture pad (521) displayed on the display (140) as being located within a real space (1000) or appearing to be overlaid on a real object, as experienced by the user (200) through the head-mounted display device (100) in the user's (200) field of view.

[0253] FIG. 12 is a diagram illustrating whether an operation of comparing a detected pose with a preset starting pose and comparing the detected pose with a plurality of preset poses is performed according to one embodiment of the present disclosure. Hereinafter, steps identical to those described in FIG. 4 are assigned the same reference numerals, and any redundant descriptions are omitted.

[0254] Referring to FIGS. 1, 3, 4, and 12, in one embodiment, the method of operating the head-mounted display device (100) may include a step (S250) of comparing a detected pose (400) with a preset starting pose. In one embodiment, the operation of the step (S250) of comparing the detected pose (400) with the preset starting pose may be executed after the operation of the step (S200) of detecting a user's pose.

[0255] In one embodiment, at least one processor (190) can compare a detected pose (400) with a preset starting pose by executing instructions or program code of a pose comparison module (183).

[0256] In one embodiment, the 'start pose' may be a preset pose for comparing the detected pose (400) with a plurality of preset poses and determining whether to execute an operation of the head mounted display device (100) that displays an interface (500) corresponding to the detected pose (400) through the display (140).

[0257] In one embodiment, the "start pose" may be set to a pose unrelated to the user's pose for providing user input through the interface, rather than a pose the user takes. In one embodiment, the "start pose" may include a pose in which the user (200) holds a hand in a mouse-holding position and moves the index finger twice, a pose in which the user makes a specific shape with the fingers, or a pose in which the user touches the nose with the hand.

[0258] In one embodiment, in the step (S250) of comparing the detected pose (400) with a preset start pose, if it is determined that the detected pose (400) corresponds to the preset start pose, the operating method of the head mounted display device (100) can execute the operation of the step (S300) of comparing the detected pose (400) with at least one preset pose.

[0259] In one embodiment, in the step (S250) of comparing the detected pose (400) with the preset start pose, if it is determined that the detected pose (400) does not correspond to the preset start pose, the operating method of the head mounted display device (100) may terminate the operation.

[0260] Through this, when a user (200) wishes to receive an interface through a head-mounted display device (100), the user (200) can assume a pose corresponding to a start pose, and then, by assuming a pose corresponding to the interface desired to be received, the interface can be displayed through the display (140). Accordingly, the user (200) can be provided with an interface as intended, thereby improving the convenience of the user (200).

[0261] In addition, even if a user (200) unconsciously assumes a pose corresponding to a specific interface while using a head-mounted display device (100), if the user (200) has not assumed the starting pose, the interface is not displayed through the display (140), thereby preventing unintentional disruption of the user's (200) usage experience.

[0262] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the present disclosure.

[0263] To solve the above-described technical problem, in one embodiment, a head-mounted display device is provided. The head-mounted display device may include a camera. The head-mounted display device may include a display. The head-mounted display device may include a memory storing at least one instruction and at least one processor. When the at least one processor executes the at least one instruction, the head-mounted display device may display a first interface through the display. When the at least one processor executes the at least one instruction, the head-mounted display device may capture an image of a user through a camera. When the at least one processor executes the at least one instruction, the head-mounted display device may detect a pose of the user based on the acquired image. When the at least one processor executes the at least one instruction, the head-mounted display device may compare the detected pose with a plurality of poses preset to correspond to a plurality of interfaces, respectively. By having at least one processor execute at least one command, the head mounted display device can display a second interface instead of the first interface through the display, as the detected pose corresponds to the second interface, which is one of the plurality of interfaces.

[0264] In one embodiment, the head-mounted display device can compare the first interface and the second interface by having at least one processor execute at least one command. When the head-mounted display device determines that the first interface and the second interface are different interfaces by having at least one processor execute at least one command, the head-mounted display device can display the second interface instead of the first interface through the display.

[0265] In one embodiment, the head mounted display device can maintain the first interface displayed through the display when the detected pose and the preset plurality of poses are different, or when the first interface and the second interface are determined to be the same interface, by having at least one processor execute at least one command.

[0266] In one embodiment, the head-mounted display device can display first content through the display by having at least one processor execute at least one command. The head-mounted display device can display a first interface and a second interface through the display while the first content is displayed by having at least one processor execute at least one command.

[0267] In one embodiment, by at least one processor executing at least one command, the head-mounted display device can capture a space (1000) including the head-mounted display device through a camera to acquire a space image. By at least one processor executing at least one command, the head-mounted display device can generate a space map representing the space (1000) based on the acquired space image. By at least one processor executing at least one command, the head-mounted display device can determine an interface position at which a second interface is to be displayed based on the space map and the detected pose. By at least one processor executing at least one command, the head-mounted display device can display the second interface at the interface position through the display.

[0268] In one embodiment, the head-mounted display device may further include a gaze tracking sensor that tracks the direction of the user's gaze. By having at least one processor execute at least one command, the head-mounted display device can acquire the user's viewpoint based on the direction of the gaze.

[0269] In one embodiment, the head mounted display device can detect a pose of the user based on the acquired viewpoint of the user and the acquired image by having at least one processor execute at least one instruction.

[0270] In one embodiment, by having at least one processor execute at least one command, the head-mounted display device can determine a content location at which to display second content, wherein the second content is displayed based on a user input obtained through at least one of the first interface and the second interface, based on the acquired user's viewpoint. By having at least one processor execute at least one command, the head-mounted display device can display the second content at the content location through the display.

[0271] In one embodiment, the first interface may be an interface displayed on the display based on whether the detected pose corresponds to one of a plurality of preset poses set to correspond to the first interface. The pose set to correspond to the first interface and the pose set to correspond to the second interface may be different gestures.

[0272] In one embodiment, at least one processor executes at least one command, thereby allowing the head-mounted display device to compare the detected pose with a preset starting pose. At least one processor executes at least one command, thereby allowing the head-mounted display device to compare the detected pose with a plurality of preset poses, based on whether the detected pose corresponds to the preset starting pose.

[0273] In order to solve the above-described technical problem, one embodiment of the present disclosure provides a method for operating a head-mounted display device. The method for operating a head-mounted display device may include a step of capturing an image of a user by using a camera. The method for operating a head-mounted display device may include a step of detecting a pose of the user based on the acquired image. The method for operating a head-mounted display device may include a step of comparing the detected pose with a plurality of poses preset to correspond to each of a plurality of interfaces. The method for operating a head-mounted display device may include a step of displaying a second interface by replacing a first interface through a display when the detected pose corresponds to a second interface, which is one of the plurality of interfaces.

[0274] In one embodiment, a method of operating a head-mounted display device may include comparing a first interface with a second interface. If the first interface and the second interface are determined to be different in the comparing step, the second interface may be displayed on the display in place of the first interface.

[0275] In one embodiment, a method of operating a head-mounted display device may include, in a step of comparing a plurality of preset poses with a detected pose, maintaining a first interface displayed through a display when the plurality of preset poses and the detected pose are different, or in a step of comparing a first interface with a second interface, the first interface and the second interface are determined to be the same interface.

[0276] In one embodiment, a method of operating a head-mounted display device may include displaying first content via a display. The method of operating the head-mounted display device may display a first interface and a second interface via the display while the first content is displayed.

[0277] In one embodiment, a method of operating a head-mounted display device may include a step of capturing a space (1000) including the head-mounted display device through a camera to acquire a space image. The method of operating the head-mounted display device may include a step of generating a space map representing the space (1000) based on the acquired space image. The method of operating the head-mounted display device may include a step of determining an interface position at which a second interface is to be displayed based on the space map and a detected pose. The method of operating the head-mounted display device may display the second interface at the interface position in the step of displaying the second interface through the display.

[0278] In one embodiment, a method of operating a head-mounted display device may include acquiring a user's viewpoint based on the user's gaze direction tracked by an eye tracking sensor. In the step of detecting the user's pose, the user's pose may be detected based on the acquired user's viewpoint and the acquired image.

[0279] In one embodiment, a method of operating a head-mounted display device may include determining a content location at which to display second content, wherein the second content is displayed based on a user input obtained through at least one of a first interface and a second interface, based on an acquired user viewpoint. The method of operating a head-mounted display device may include displaying the second content at the content location through the display.

[0280] In one embodiment, a method of operating a head-mounted display device may include comparing a detected pose with a preset starting pose. The method of operating a head-mounted display device may compare the detected pose with a plurality of preset poses, based on whether the detected pose corresponds to the preset starting pose.

[0281] In order to solve the above-described technical problem, a computer-readable recording medium having recorded thereon a program for performing at least one method of the method of operating a head-mounted display device disclosed in one embodiment can be provided.

[0282] The program executed by the head-mounted display device described in the present disclosure may be implemented as hardware components, software components, and / or a combination of hardware components and software components. The program may be executed by any system capable of executing computer-readable instructions.

[0283] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to do a desired thing or may independently or collectively command a processing device to do a desired thing.

[0284] Software may be implemented as a computer program containing instructions stored on a computer-readable storage medium. Examples of computer-readable storage media include magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disks, hard disks, etc.) and optical readable media (e.g., CD-ROMs, DVDs (Digital Versatile Discs)). The computer-readable storage media may be distributed across network-connected computer systems, so that computer-readable code may be stored and executed in a distributed manner. The storage media may be readable by a computer, stored in a memory, and executed by a processor.

[0285] Computer-readable storage media may be provided in the form of non-transitory storage media. Here, the term "non-transitory storage media" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage media and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0286] Additionally, programs according to the embodiments disclosed herein may be provided as part of a computer program product. The computer program product may be traded as a commodity between sellers and buyers.

[0287] A computer program product may include a software program, or a computer-readable storage medium storing the software program. For example, the computer program product may include a product in the form of a software program (e.g., a downloadable application) distributed electronically by the manufacturer of the head-mounted display device (100) or through an electronic marketplace (e.g., the Samsung Galaxy Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily generated. In this case, the storage medium may be a storage medium of a server of the manufacturer of the head-mounted display device (100), a server of an electronic marketplace, or a relay server that temporarily stores the software program.

Claims

1. In a head mounted display device (100), Camera (130); display (140); A memory (180) storing at least one instruction; and comprising at least one processor (190), The head mounted display device (100) executes the at least one command by the at least one processor (190). Through the above display (140), the first interface (500) is displayed, Through the above camera (130), the user is photographed and an image is acquired, Detecting the user's pose based on the acquired image, Compare the detected pose with a plurality of poses preset to correspond to each of the plurality of interfaces, A head mounted display device (100) that displays the second interface (510) by replacing the first interface (500) through the display (140) as the detected pose corresponds to the second interface (510) which is one of the plurality of interfaces.

2. In paragraph 1, The head mounted display device (100) executes the at least one command by the at least one processor (190). Comparing the first interface (500) and the second interface (510), A head mounted display device (100) that displays the second interface (510) by replacing the first interface (500) through the display (140) as the first interface (500) and the second interface (510) are determined to be different interfaces.

3. In paragraph 2, The head mounted display device (100) executes the at least one command by the at least one processor (190). Since the above-detected pose and the above-detected plurality of poses are different, or the first interface (500) and the second interface (510) are determined to be the same interface, A head mounted display device (100) that maintains the first interface (500) displayed through the display (140).

4. In paragraph 1, The head mounted display device (100) executes the at least one command by the at least one processor (190). The first content is displayed through the above display (140), A head mounted display device (100) that displays the first interface (500) and the second interface (510) through the display (140) while the first content (300) is displayed.

5. In any one of clauses 1 to 4, The head mounted display device (100) executes the at least one command by the at least one processor (190). Through the above camera (130), the space including the head mounted display device (100) is photographed to obtain a space image, Generate a spatial map representing the space based on the acquired spatial image, Determine the interface position to display the second interface (510) based on the above space map and the detected pose, A head mounted display device (100) that displays the second interface (510) at the interface location through the display (140).

6. In any one of clauses 1 to 5, The above head mounted display device (100) is Further comprising a gaze tracking sensor for tracking the gaze direction of the user; The head mounted display device (100) executes the at least one command by the at least one processor (190). A head mounted display device (100) that acquires the user's viewpoint based on the above gaze direction.

7. In paragraph 6, The head mounted display device (100) executes the at least one command by the at least one processor (190). A head mounted display device (100) that detects the pose of the user based on the acquired user's viewpoint and the acquired image.

8. In either of paragraphs 6 or 7, The head mounted display device (100) executes the at least one command by the at least one processor (190). Determine the content location where the second content is to be displayed based on the user input acquired through at least one of the first interface (500) or the second interface (510) based on the acquired user's viewpoint, A head mounted display device (100) that displays the second content at the content location through the display (140).

9. In any one of clauses 1 to 8, The first interface (500) is an interface displayed on the display (140) as the detected pose corresponds to one of the preset plurality of poses set to correspond to the first interface (500). A head mounted display device (100) in which one pose set to correspond to the first interface (500) and one pose set to correspond to the second interface (510) are different poses.

10. In any one of clauses 1 to 9, The head mounted display device (100) executes the at least one command by the at least one processor (190). Compare the above detected pose with the preset starting pose, A head mounted display device (100) that compares the detected pose with a plurality of preset poses, as the detected pose corresponds to the preset start pose.

11. In the operating method of a head mounted display device, Step (S100) of capturing an image by photographing a user through a camera; Step (S200) of detecting the user's pose based on the acquired image; A step (S300) of comparing the detected pose with a plurality of poses preset to correspond to each of a plurality of interfaces; A method for operating a head mounted display device, comprising a step (S600) of displaying the second interface by replacing the first interface through the display, when the detected pose corresponds to a second interface, which is one of the plurality of interfaces.

12. In paragraph 11, The operating method of the head mounted display device includes a step (S500) of comparing the first interface and the second interface, In the step (S500) of comparing the first interface and the second interface, since the first interface and the second interface are determined to be different interfaces, in the step (S600) of displaying the second interface through the display A method of operating a head mounted display device that displays a second interface by replacing the first interface through the display.

13. In paragraph 12, The method of operation of the head mounted display device is as follows: In the step (S300) of comparing the detected pose with the plurality of preset poses, the plurality of preset poses and the detected pose are different, or In the step (S500) of comparing the first interface and the second interface, since the first interface and the second interface are determined to be the same interface, A method of operating a head mounted display device, comprising a step (S700) of maintaining the first interface displayed through the display.

14. In paragraph 11, The method of operation of the head mounted display device is as follows: Including a step (S50) of displaying first content through the above display, A method of operating a head mounted display device that displays the first interface and the second interface through the display while the first content is displayed.

15. A computer-readable recording medium having recorded thereon a program for performing the method of any one of claims 11 to 14 on a computer.

Images