Augmented reality device for displaying virtual keyboard and operation method thereof

Abstract

Provided are an augmented reality device and an operation method thereof, the augmented reality device adaptively determining a virtual keyboard and an area on which the virtual keyboard is to be overlaid on the basis of attribute information of a surrounding real space and profile information of the virtual keyboard. The augmented reality device may: scan a real space to detect at least one area including a plane in which no object has been detected; determine a type of virtual keyboard overlayable on the detected at least one area on the basis of at least one of the shape, size, and input language of the virtual keyboard; and perform rendering so that the virtual keyboard of the type is overlaid and displayed on the at least one area.

Description

Augmented reality device displaying a virtual keyboard and method for operating the same

[0001] Embodiments of the present disclosure relate to an augmented reality device that displays a virtual keyboard by overlaying it on a real space, and a method for operating the same. Specifically, embodiments of the present disclosure relate to an augmented reality device that detects an optimal area for displaying a virtual keyboard by overlaying it on a surrounding real space, and renders the virtual keyboard in the detected area, and a method for operating the same.

[0002] Augmented reality (AR) is a technology that overlays virtual objects onto the physical environment or objects of the real world, presenting them alongside one another. Its advantage lies in the fusion of virtual objects and information within the real world. AR devices (e.g., smart glasses) utilizing AR technology are being used in everyday life for various purposes, such as information retrieval, route guidance, and camera photography. In particular, smart glasses are often worn as fashion accessories and are primarily used for outdoor activities.

[0003] Unlike general PCs that use a physical keyboard or mobile devices that use a keyboard consisting of a graphical user interface (UI) displayed on a touchscreen, augmented reality devices can display a virtual keyboard overlaid on the surrounding real world due to the nature of the device, and provide input means through interactions such as the user's hand gestures touching the virtual keyboard. A virtual keyboard is a concept distinct from a physical keyboard, and refers to a virtual keyboard implemented through software.

[0004] When an augmented reality device displays a virtual keyboard in the air, the speed at which it receives key inputs, such as hand gestures, from the user is very slow, and when the virtual keyboard is displayed on a plane where the user's hand is located, a space larger than the size of the keyboard is required. Conventional augmented reality devices display a virtual keyboard in an arbitrary area regardless of the conditions of the user's surrounding environment, and there was a problem that the virtual keyboard was inconvenient to use if the area where the user's hand was located did not have sufficient empty space. For example, when a virtual keyboard is displayed by overlaying it on a plane such as a desk with many objects on it, the entire virtual keyboard may not be displayed completely or the size of the virtual keyboard may be reduced, resulting in low visibility of the virtual keyboard, which may reduce user convenience.

[0005] According to one embodiment of the present disclosure, a method performed by an augmented reality device is provided. The method may include: scanning a surrounding real world to detect at least one area including a plane where no object is detected; determining a type of virtual keyboard that can be overlaid on the detected at least one area based on at least one of a shape, a size, and an input language of the virtual keyboard; and performing rendering so that a virtual keyboard having the type is overlaid and displayed on the at least one area.

[0006] According to one embodiment of the present disclosure, an augmented reality device is provided. The augmented reality device comprises: at least one camera; a sensor including at least one of an infrared sensor, a depth camera, and a LiDAR sensor; a memory storing one or more instructions; and at least one processor for executing the one or more instructions, wherein when the one or more instructions are executed by the at least one processor, the augmented reality device is configured to scan a surrounding real world using at least one of the camera and the at least one sensor, thereby detecting at least one area including a plane where no object is detected, and, based on at least one of a shape, a size, and an input language of the virtual keyboard, determine a type of a virtual keyboard that can be overlaid on the at least one area, and perform rendering so that a virtual keyboard having the type is overlaid and displayed on the at least one area.

[0007] According to one embodiment of the present disclosure, a computer program product is provided comprising a non-transitory computer-readable recording medium. The non-transitory computer-readable recording medium is configured to, when executed by at least one processor of an augmented reality device, cause the augmented reality device to: scan a surrounding real world to detect at least one area including a plane where no object is detected, determine a type of virtual keyboard that can be overlaid on the detected at least one area based on at least one of a shape, a size, and an input language of the virtual keyboard, and perform rendering so that a virtual keyboard having the type is overlaid and displayed on the at least one area.

[0008] The present disclosure can be readily understood by the following detailed description and its accompanying drawings, wherein reference numerals refer to structural elements.

[0009] FIG. 1 is a conceptual diagram illustrating an operation of an augmented reality device of the present disclosure to display a virtual keyboard in real space.

[0010] FIG. 2 is a flowchart illustrating a method for an augmented reality device to display a virtual keyboard in a real space according to one embodiment of the present disclosure.

[0011] FIG. 3 is a block diagram illustrating components of an augmented reality device according to one embodiment of the present disclosure.

[0012] FIG. 4 is a diagram illustrating data input / output between a software module stored in a memory and a camera, a sensor, and a display unit of an augmented reality device according to one embodiment of the present disclosure.

[0013] FIG. 5 is a flowchart illustrating a method for an augmented reality device according to one embodiment of the present disclosure to scan the surrounding real world and detect at least one area.

[0014] FIG. 6 is a flowchart illustrating a method for determining a type of virtual keyboard that can be overlaid on at least one area by an augmented reality device according to one embodiment of the present disclosure, and determining a virtual keyboard of one of the determined types and an area on which the virtual keyboard is to be overlaid.

[0015] FIG. 7 is a flowchart illustrating a method for determining a type of virtual keyboard that can be overlaid on at least one area by an augmented reality device according to one embodiment of the present disclosure, and determining a virtual keyboard of one of the determined types and an area on which the virtual keyboard is to be overlaid.

[0016] FIG. 8A is a diagram illustrating an embodiment in which the augmented reality device of the present disclosure overlays a virtual keyboard in QWERTY format on an area.

[0017] FIG. 8b is a diagram illustrating an embodiment in which the augmented reality device of the present disclosure overlays and displays a virtual keyboard of the Cheonjiin input method on an area.

[0018] FIG. 8c is a diagram illustrating an embodiment in which the augmented reality device of the present disclosure overlays a virtual keyboard in the form of number keys on an area.

[0019] FIG. 8d is a diagram illustrating an embodiment in which the augmented reality device of the present disclosure overlays a virtual keyboard in a 12-key English keyboard input method on an area.

[0020] FIG. 9 is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure to overlay a detachable virtual keyboard on multiple areas.

[0021] FIG. 10 is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure to overlay a virtual keyboard on a part of a user's body part.

[0022] FIG. 11 is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure to display a virtual keyboard by tracking the movement of a user's body part.

[0023] FIG. 12 is a flowchart illustrating a method for an augmented reality device according to one embodiment of the present disclosure to change the color of a virtual keyboard based on color information of an area where the virtual keyboard is overlaid.

[0024] FIG. 13 is a flowchart illustrating a method for an augmented reality device to display a virtual keyboard on an area determined based on a user's hand gesture according to one embodiment of the present disclosure.

[0025] FIG. 14 is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure to display a virtual keyboard in an area determined based on a user's hand gesture.

[0026] FIG. 15 is a flowchart illustrating a method for an augmented reality device according to one embodiment of the present disclosure to determine a virtual keyboard and an area in which the virtual keyboard is to be displayed based on context.

[0027] FIG. 16 is a flowchart illustrating a method for an augmented reality device according to one embodiment of the present disclosure to determine a virtual keyboard and an area in which the virtual keyboard is to be displayed based on context.

[0028] The terms used in the non-limiting exemplary embodiments of the present disclosure have been selected from widely used and general terms as much as possible, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, the applicant may arbitrarily select terms, and in such cases, the meanings thereof will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as the names of the terms, but rather based on the meanings of the terms and the overall content of the present disclosure.

[0029] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0030] Throughout this disclosure, when a part is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," etc., used herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0031] The expression "configured to" as used herein can be used interchangeably with, for example, "suitable for", "having the capacity to", "designed to", "adapted to", "made to", or "capable of", depending on the context. The term "configured to" may not only mean "specifically designed to" in terms of hardware. Instead, in some contexts, the expression "a system configured to" may mean that the system is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured to perform A, B, and C" can mean a dedicated processor for performing the operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or an application processor) that can perform the operations by executing one or more software programs stored in memory.

[0032] Additionally, when a component is referred to as being "connected" or "connected" to another component in the present disclosure, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated.

[0033] In this disclosure, 'Augmented Reality' means displaying a virtual image together within a physical environment space of the real world or displaying a real object and a virtual image together.

[0034] In this disclosure, "real world" refers to the real world space viewed by a user through an augmented reality device. In one embodiment of the present disclosure, "real world" may refer to an indoor space. Real world objects may be placed within the real world.

[0035] In the present disclosure, an 'augmented reality device' is a device capable of expressing augmented reality, and may be, for example, not only augmented reality glasses in the shape of glasses worn by a user on the face, but also a head-mounted display apparatus (HMD) worn on the head, an augmented reality helmet, etc. However, the embodiments of the present disclosure are not limited thereto, and the augmented reality device may be implemented as various electronic devices such as a laptop computer, a desktop, an e-book reader, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a navigation system, an MP3 player, a camcorder, an IPTV (Internet Protocol Television), a DTV (Digital Television), a wearable device, etc.

[0036] In one embodiment of the present disclosure, the electronic device may be an augmented reality device. An 'augmented reality device' is a device capable of expressing augmented reality, and may be implemented as, for example, augmented reality glasses in the shape of glasses that a user wears on the face. However, the embodiments of the present disclosure are not limited thereto, and the augmented reality device may also be implemented as a head-mounted display apparatus (HMD) worn on the user's head, an augmented reality helmet, or the like.

[0037] In the present disclosure, a "virtual keyboard" is a concept distinct from a physical keyboard, and refers to a virtual keyboard implemented through software. The virtual keyboard may be a graphical user interface (UI) composed of pixels overlaid on a real space. In one embodiment of the present disclosure, an augmented reality device may render a virtual image constituting the virtual keyboard, generate light from the rendered virtual image, and project the light from the virtual image onto a waveguide through an optical engine, thereby overlaying the virtual keyboard on a surrounding real space. The optical engine may include, for example, an image panel, an illumination optical system, a projection optical system, and the like.

[0038] Below, non-limiting exemplary embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the embodiments of the present disclosure may be implemented in various different forms and are not limited to the embodiments described herein.

[0039] Hereinafter, non-limiting exemplary embodiments of the present disclosure will be described in detail with reference to the drawings.

[0040] FIG. 1 is a conceptual diagram illustrating an operation of an augmented reality device (100) of the present disclosure displaying a virtual keyboard in real space.

[0041] The augmented reality device (100) is a device capable of expressing augmented reality, and may be configured as, for example, augmented reality glasses in the shape of glasses that a user (1) wears on the face. In FIG. 1, the augmented reality device (100) is illustrated as augmented reality glasses, but is not limited thereto. For another example, the augmented reality device (100) may be implemented as a head-mounted display apparatus (HMD) or an augmented reality helmet that is worn on the head of the user (1).

[0042] Referring to FIG. 1, the augmented reality device (100) scans the surrounding environment and detects at least one area (e.g., a first area (P1), a second area (P2), a third area (P3), a fourth area (P4)) (operation A1).

[0043] The augmented reality device (100) determines the type of virtual keyboard that can be overlaid on at least one area (P1, P2, P3, P4) based on at least one of the shape, size, and input language of the virtual keyboard (e.g., the first keyboard (k1), the second keyboard (k2), the third keyboard (k3), and the fourth keyboard (k4)) (operation A2).

[0044] The augmented reality device (100) renders a virtual keyboard of a determined type (e.g., a first keyboard (k1), a second keyboard (k2), a third keyboard (k3), a fourth keyboard (k4)) and displays the virtual keyboard (e.g., a first keyboard (k1), a second keyboard (k2), a third keyboard (k3), a fourth keyboard (k4)) by overlaying it on at least one area (e.g., a first area (P1), a second area (P2), a third area (P3), a fourth area (P4)) (operation A3).

[0045] Hereinafter, the function and / or operation of the augmented reality device (100) to display a virtual keyboard in real space will be described in detail with reference to FIGS. 1 and 2 together.

[0046] FIG. 2 is a flowchart illustrating a method for displaying a virtual keyboard on a real space of an augmented reality device (100) according to one embodiment of the present disclosure.

[0047] In step S210, the augmented reality device (100) scans the surrounding real world to detect at least one area including a plane where no object is detected. The augmented reality device (100) includes a camera (110, see FIG. 3) and can obtain image data of the surrounding real world by taking pictures of the surrounding real world using the camera (110). In one embodiment of the present disclosure, the augmented reality device (100) includes an infrared sensor (122, see FIG. 3), a depth camera (124, see FIG. 3), and / or a LiDAR sensor (126, see FIG. 3), and can obtain three-dimensional data about the real world by scanning the surrounding environment using at least one of the infrared sensor (122), the depth camera (124), and the LiDAR sensor (126). '3D data' may be data that explicitly represents the 3D shape of the surrounding real space, such as a point cloud or mesh, or may include 3D data in an abstract form, such as a signed distance function.

[0048] The augmented reality device (100) can perform plane detection to detect at least one area including a plane where no object is detected from three-dimensional data of a real space. The augmented reality device (100) can recognize horizontal surfaces, such as walls, floors, or desk surfaces in an office, using a plane detection algorithm.

[0049] However, the embodiments of the present disclosure are not limited thereto, and an augmented reality device (100) according to one embodiment of the present disclosure may detect at least one area including a surface having a predetermined curvature from three-dimensional data of the surrounding environment. For example, the augmented reality device (100) may recognize a surface having a curvature similar to a cylinder.

[0050] In one embodiment of the present disclosure, the augmented reality device (100) can recognize an object placed on a flat or curved surface from image data acquired through a camera by performing vision recognition using an object recognition model composed of a learned artificial intelligence model. Here, the 'object' refers to a real world object placed in a real space, and may mean, for example, a desk, chair, personal computer (PC), tablet PC, keyboard, mouse, or bag in an office. The augmented reality device (100) can detect an area in which a real object is not detected among the detected flat or curved surfaces.

[0051] Referring to operation A1 of FIG. 1, the augmented reality device (100) can acquire three-dimensional data about an office by scanning a real space, and detect a first area (P1) to a fourth area (P4) where no object is detected from the acquired three-dimensional data. In the embodiment illustrated in FIG. 1, the first area (P1) and the second area (P2) are planes formed by walls of an office, the third area (P3) is an area on the surface of a desk where no object is placed, and the fourth area (P4) may be a surface of a part of a body part of a user (1). For example, the fourth area (P4) may be a surface having a predetermined curvature on a thigh among the body parts of the user (1). The shape, size, and number (e.g., '4') of the first area (P1) to the fourth area (P4) illustrated in FIG. 1 are examples for convenience of explanation, and the shape, size, and number of 'at least one area' of the embodiments of the present disclosure are not limited to those illustrated in FIG. 1.

[0052] The augmented reality device (100) can determine an area among the detected areas (P1, P2, P3, P4) on which a virtual keyboard cannot be overlaid. In one embodiment of the present disclosure, if the distance between the detected area and the user (1) exceeds a preset threshold, the augmented reality device (100) can determine that the area is an area on which a virtual keyboard cannot be overlaid. The 'overlaid-impossible area' may include, for example, an area outside the range of about 60 centimeters to 80 centimeters, which is the average human arm length. The augmented reality device (100) determines, for example, an area exceeding 80 centimeters as an area on which a virtual keyboard cannot be overlaid, and thus determines at least one area (for example, a first area (P1)) It can be excluded from the fourth area (P4)).

[0053] Referring back to FIG. 2, in step S220, the augmented reality device (100) determines the type of virtual keyboard that can be overlaid on at least one area based on at least one of the shape, size, and input language of the virtual keyboard. In one embodiment of the present disclosure, the augmented reality device (100) can obtain profile information of all types of virtual keyboards that can be provided. The 'profile information' can include information about at least one of the shape, size, and input language of the virtual keyboard. The profile information of the virtual keyboard can be pre-stored in the memory (140, see FIG. 3) of the augmented reality device (100). The augmented reality device (100) can load the profile information of the virtual keyboard from the memory (140) to obtain the profile information. However, the present disclosure is not limited thereto, and in one embodiment of the present disclosure, the profile information of the virtual keyboard is stored in a server or an external device, and the augmented reality device (100) can also obtain the profile information of the virtual keyboard from the server or the external device by performing data communication.

[0054] Among the profile information of the virtual keyboard, the 'virtual keyboard type' may include at least one of, for example, a full-sized type including all 106 keys, a split type that can be divided into multiple keyboard areas, a type including only number keys, and a 12-key English keyboard type provided by a mobile device such as a cell phone (12-key telephone keypad). The 'virtual keyboard size' may include information about the minimum and maximum displayable sizes in which the virtual keyboard is rendered. The 'virtual keyboard input language' may include Korean, English, Chinese, Japanese, numbers, or special characters.

[0055] The augmented reality device (100) can configure an area-virtual keyboard combination by matching at least one area detected in step S210 with all types of virtual keyboards that can be provided. Referring to operation A2 of FIG. 1, the augmented reality device (100) can configure a first combination (10) by including a first keyboard (k1) which is a QWERTY-type virtual keyboard, a second keyboard (k2) which is a virtual keyboard of the Cheonjiin input method, a third keyboard (k3) which is a virtual keyboard of the 12-key English keypad input method, and a fourth keyboard (k4) which is a virtual keyboard of the number input method as types of virtual keyboards that can be matched to the first area (P1). Similarly, the augmented reality device (100) can configure a first combination (10) by including a first keyboard (k1) which is a QWERTY-type virtual keyboard, a second keyboard (k2) which is a virtual keyboard of the Cheonjiin input method, a third keyboard (k3) which is a virtual keyboard of the 12-key English keypad input method, and a fourth keyboard (k4) which is a virtual keyboard of the number input method as types of virtual keyboards that can be matched to the first area (P1). 2) For the first keyboard (k1), the second keyboard (k2), the third keyboard (k3), and the fourth keyboard (k4) can be matched to form a second combination (20), for the third area (P3) the second keyboard (k2), the third keyboard (k3), and the fourth keyboard (k4) can be matched to form a third combination (30), and for the fourth area (P4) the third keyboard (k3) and the fourth keyboard (k4) can be matched to form a fourth combination (40).

[0056] In one embodiment of the present disclosure, the augmented reality device (100) may match a plurality of virtual keyboards that can be overlaid on at least one area.

[0057] In one embodiment of the present disclosure, the augmented reality device (100) may separate a virtual keyboard configured as a detachable keyboard and match it to multiple areas.

[0058] The augmented reality device (100) can perform an evaluation of a region-virtual keyboard combination based on attribute information of a region including the size and shape of at least one region and at least one of the shape, size, and input language of virtual keyboards. In one embodiment of the present disclosure, the augmented reality device (100) can calculate an evaluation score for a region-virtual keyboard combination by considering the distance between the at least one region and the user, together with profile information including the attribute information of the at least one region and at least one of the shape, size, and input language of virtual keyboards. Referring to the embodiment illustrated in FIG. 1, the augmented reality device (100) can calculate an evaluation score for the first combination (10) based on profile information regarding the size and shape of the first region (P1) and at least one of the shape, size, and input language of each of the first to fourth keyboards (k1 to k4). In the same manner as described above for the first combination (10), the augmented reality device (100) can calculate evaluation scores for the second combination (20), the third combination (30), and the fourth combination (40).

[0059] The augmented reality device (100) may determine the type of virtual keyboard that can be overlaid on at least one area based on the evaluation result of the area-virtual keyboard combination. In one embodiment of the present disclosure, the augmented reality device (100) may determine that a virtual keyboard can be overlaid on an area only for a combination in which the calculated evaluation score exceeds a preset reference score. Referring also to the embodiment illustrated in FIG. 1, for example, the augmented reality device (100) may determine, based on the evaluation score for the first combination (10), that the types of virtual keyboards that can be overlaid on the first area (P1) include all of the first keyboard (k1), the second keyboard (k2), the third keyboard (k3), and the fourth keyboard (k4). For example, the augmented reality device (100) may determine, based on the evaluation score for the third area (P3), that the types of virtual keyboards that can be overlaid on the third area (P3) include the second keyboard (k2), the third keyboard (k3), and the fourth keyboard (k4). For example, the augmented reality device (100) may determine, based on the evaluation score for the fourth area (P4), that the fourth area (P4) includes a third keyboard (k3) and a fourth keyboard (k4) as types of virtual keyboards that can be overlaid on the fourth area (P4). For example, if the evaluation score is lower than a reference score, the augmented reality device (100) may determine that there is no virtual keyboard that can be overlaid on the corresponding area.

[0060] Referring back to FIG. 2, in step S230, the augmented reality device (100) performs rendering to display a virtual keyboard of a determined type by overlaying it on at least one area. In one embodiment of the present disclosure, the augmented reality device (100) may perform rendering to enlarge or reduce the size of the virtual keyboard by scaling the virtual keyboard to suit the size and shape of at least one area. Referring to operation A3 of FIG. 1, the augmented reality device (100) may render a first keyboard (k1), which is a QWERTY type virtual keyboard, on a first area (P1) and a second area (P2), which are relatively large in size. The augmented reality device (100) may render a second keyboard (k2), which is a Cheonjiin type virtual keyboard, on a third area (P3), which is a relatively narrow area on a desk where no objects are placed. In one embodiment of the present disclosure, the augmented reality device (100) can render the second keyboard (k2) based on the size and shape of the third area (P3).

[0061] In one embodiment of the present disclosure, when it is determined to overlay a virtual keyboard on a fourth area (P4), which is a surface of a body part of a user (1) (e.g., a thigh), the augmented reality device (100) may perform rendering that warps the determined virtual keyboard (e.g., the fourth keyboard (k4), which is a numeric keyboard) based on the curvature of the surface of the body part.

[0062] In one embodiment of the present disclosure, the augmented reality device (100) determines a virtual keyboard and an area on which the virtual keyboard is to be overlaid from a combination of at least one area and a type of virtual keyboard, based on at least one of an input language, an input field, and usage history information. The augmented reality device (100) may determine to overlay a virtual keyboard included in the selected area-virtual keyboard combination on the selected area. Referring also to the embodiment illustrated in FIG. 1, the augmented reality device (100) may select a third combination (30) from among the area-virtual keyboard combinations (10, 20, 30, 40) based on at least one of an input language, an input field, and usage history information. The augmented reality device (100) may determine a second keyboard (k2) constituting the selected third combination (30) to be overlaid on the third area (P3) as a virtual keyboard to be displayed.

[0063] The augmented reality device (100) can display a virtual keyboard rendered on a determined area by overlaying it. In one embodiment of the present disclosure, when the augmented reality device (100) is implemented as augmented reality glasses in the shape of glasses worn on the face of a user (1), the augmented reality device (100) can include a display unit (150, see FIG. 3) configured with a lens optical system and including a waveguide and an optical engine. The augmented reality device (100) can display a virtual keyboard by overlaying it on an area by generating light of graphic objects composed of letters, numbers, special symbols, virtual images, or a combination thereof constituting the rendered virtual keyboard through the optical engine of the display unit (150) and projecting the light onto the waveguide.

[0064] The augmented reality device of the comparative embodiments displays a virtual keyboard in an arbitrary area regardless of the properties of the user's (1) surrounding environment. However, there was a problem in that the virtual keyboard was inconvenient to use when there was not enough free space in the area where the user's hand was positioned. For example, when the virtual keyboard was overlaid on a flat surface such as a desk with many objects placed on it, the entire virtual keyboard was not displayed completely or the size of the virtual keyboard was reduced, resulting in low visibility of the virtual keyboard. This may reduce the usability and user convenience of the virtual keyboard.

[0065] The embodiments of the present disclosure aim to provide an augmented reality device (100) and an operating method thereof that adaptively determine a virtual keyboard and an area on which the virtual keyboard is to be overlaid based on attribute information such as the size and shape of the real space around the user (1) and the size, shape, input language, etc. of the virtual keyboard.

[0066] The augmented reality device (100) according to the embodiment illustrated in FIGS. 1 and 2 adaptively determines an optimal area (in the embodiment illustrated in FIG. 1, the 'third area (P3)') and an optimal virtual keyboard (in the embodiment illustrated in FIG. 1, the 'second keyboard (k2)') based on at least one of attribute information of a real space around a user (1) and a shape, size, and input language of a virtual keyboard, thereby improving the visibility of the virtual keyboard and providing a technical effect of improving usability and operational convenience of the virtual keyboard.

[0067] FIG. 3 is a block diagram illustrating components of an augmented reality device (100) according to one embodiment of the present disclosure.

[0068] Referring to FIG. 3, the augmented reality device (100) may include a camera (110), a sensor (120), a processor (130), a memory (140), and a display unit (150). The camera (110), the sensor (120), the processor (130), the memory (140), and the display unit (150) may be electrically and / or physically connected to each other, respectively. FIG. 3 illustrates exemplary components for explaining the operation of the augmented reality device (100), and the components included in the augmented reality device (100) are not limited to the components illustrated in FIG. 3. In one embodiment of the present disclosure, the augmented reality device (100) may further include a communication interface for performing data communication with an external device or server. In one embodiment of the present disclosure, the augmented reality device (100) is implemented as a portable device, and in this case, the augmented reality device (100) may further include a battery that supplies driving power to a camera (110), a sensor (120), a processor (130), and a display unit (150).

[0069] The camera (110) is configured to capture images by photographing the real space around the user. The camera (110) may include a lens module, an image sensor, and an image processing module. The camera (110) may capture still images or video of an object by means of an image sensor (e.g., CMOS or CCD). The video may include a plurality of image frames continuously captured by photographing the object through the camera (110). The image processing module may encode still images composed of a single image frame captured by the image sensor or video data composed of a plurality of image frames and transmit the encoded data to the processor (130).

[0070] In one embodiment of the present disclosure, the camera (110) may be implemented in a small form factor so that it can be mounted on a portable augmented reality device (100), and may be implemented as a lightweight RGB camera that consumes low power.

[0071] The camera (110) may include one or more cameras. In one embodiment of the present disclosure, when implemented as Augmented Reality Glasses in the form of glasses worn on the user's face, the camera (110) may include two cameras, each positioned on the left eye lens and the right eye lens of the augmented reality device (100). In this case, the two cameras may be configured as stereo cameras.

[0072] However, the embodiments of the present disclosure are not limited thereto, and in one embodiment of the present disclosure, the augmented reality device (100) may include a plurality of cameras configured to capture an object in a real space located in front of the user, and a plurality of cameras configured to capture the user's hand with lenses facing downward. For example, the camera (110) may include two cameras for capturing the front and two cameras facing downward to capture the user's hand. In this case, the two cameras for capturing the front may be respectively positioned at the top of frames surrounding the left and right lenses of the augmented reality device (100), and the two cameras facing downward to capture the user's hand may be respectively positioned at the bottom of the frames of the left and right lenses.

[0073] The sensor (120) is configured to acquire three-dimensional data regarding the real space around the user. In one embodiment of the present disclosure, the sensor (120) may include at least one of an infrared sensor (122), a depth camera (124), and a lidar sensor (126).

[0074] The infrared sensor (122) is a sensor configured to emit infrared rays to an object in the real world and detect an infrared signal reflected from the object. The infrared sensor (122) can detect the intensity, emission angle, and emission position of the infrared signal. The infrared sensor (122) can provide information about the intensity, emission angle, and emission position of the infrared signal to the processor (130). The processor (130) can obtain a depth value for an object in real space based on the sensing information obtained from the infrared sensor (122) and obtain three-dimensional data such as a depth map of the real space.

[0075] A depth camera (124) is configured to obtain depth information about an object in real space. 'Depth information' means information about the distance from the depth camera (124) (e.g., depth sensor) to a specific object. In one embodiment of the present disclosure, the depth camera (124) may be configured as a stereo camera that includes a plurality of cameras and obtains depth information about an object based on disparity and the relative positional relationship between the cameras. However, the embodiments of the present disclosure are not limited thereto, and the depth camera (124) (e.g., depth sensor) may include a ToF (Time of Flight) sensor that uses a light source to irradiate a pattern light to an object and obtains depth information based on the time taken for the irradiated pattern light to be reflected from the object and detected again, i.e., the time of flight.

[0076] A LIDAR sensor (Light Detection And Ranging sensor) (126) is a sensor configured to detect at least one of distance, direction, speed, temperature, material distribution, and concentration characteristics by emitting a pulsed laser to an object and measuring the time and intensity it takes for the pulsed laser to reflect from the object and return. The processor (130) can obtain three-dimensional data, such as a depth map, of spatial structures, such as walls and objects in a real space, using the sensing information acquired through the LIDAR sensor (126).

[0077] The processor (130) can execute one or more instructions of a program stored in the memory (140). The processor (130) may be configured with hardware components that perform arithmetic, logic, and input / output operations and image processing. Although the processor (130) is illustrated as a single element in FIG. 3 , it is not limited thereto. In one embodiment of the present disclosure, the processor (130) may be configured with one or more elements. The processor (130) may be, for example, a general-purpose processor such as a Central Processing Unit (CPU), an Application Processor (AP), a Digital Signal Processor (DSP), a graphics-only processor such as a Graphics Processing Unit (GPU), a Vision Processing Unit (VPU), or an artificial intelligence-only processor such as a Neural Processing Unit (NPU). The processor (130) may be controlled to process input data according to predefined operating rules or artificial intelligence models. Alternatively, if the processor (130) is an artificial intelligence-only processor, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0078] The processor (130) may include various processing circuits and / or multiple processors. For example, the term "processor" as used in this disclosure, including the claims, may include various processing circuits, including at least one processor. One or more processors in at least one processor may be configured to perform various functions described in this disclosure individually and / or collectively in a distributed manner. As used herein, "processor," "at least one processor," and "one or more processors" may be configured to perform various functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, at least one processor may include a combination of processors that perform various functions of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0079] The memory (140) may be configured as at least one type of storage medium, for example, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), or an optical disk.

[0080] The memory (140) may store instructions related to functions and / or operations for determining an optimal area for overlaying a virtual keyboard among areas detected from the surrounding real space by the augmented reality device (100) and adaptively displaying the virtual keyboard in the determined area. In one embodiment of the present disclosure, the memory (140) may store at least one of instructions (e.g., program code including an application program), an algorithm, and a data structure that can be read by the processor (130). The instructions (e.g., program code including an application program), an algorithm, and a data structure stored in the memory (140) may be implemented in a programming or scripting language such as, for example, C, C++, Java, an assembler, etc.

[0081] The memory (140) may store instructions (e.g., program codes), algorithms, and data structures related to the area detection module (142), the virtual keyboard determination module (144), and the rendering module (146). A 'module' included in the memory (140) refers to a unit that processes a function or operation performed by the processor (130), and this may be implemented as software such as instructions (e.g., program codes), algorithms, or data structures. In one embodiment of the present disclosure, the memory (140) may also include a virtual keyboard data storage unit (148).

[0082] The processor (130) performs functions (e.g., realizing an area detection module (142), a virtual keyboard determination module (144), and / or a rendering module (146)) by executing instructions or program codes stored in the memory (140). Hereinafter, with reference to FIGS. 3 and 4 together, the functions and / or operations performed by the processor (130) by executing instructions (e.g., program codes) of each of the plurality of modules stored in the memory (140) and the data input / output between the plurality of modules and components (e.g., a camera (110), a sensor (120), and a display unit (150)) will be described in detail.

[0083] FIG. 4 is a diagram illustrating data input / output between a software module stored in a memory (140) and a camera (110), a sensor (120), and a display unit (150) of an augmented reality device (100) according to one embodiment of the present disclosure. According to one embodiment of the present disclosure, a processor (130, see FIG. 3) may execute commands (e.g., program codes) of an area detection module (142), a virtual keyboard determination module (144), and a rendering module (146) to perform related functions and / or operations.

[0084] Referring to FIGS. 3 and 4 together, the area detection module (142) may include (or be configured to include) instructions (e.g., program codes) for executing a function and / or operation of detecting at least one area on which a virtual keyboard can be overlaid from three-dimensional data of a real space acquired through at least one of a camera (110) and a sensor (120). The processor (130) may acquire a spatial image of a real space around the location of the augmented reality device (100) from the camera (110) and acquire sensing data about the real space from the sensor (120). The processor (130) may acquire three-dimensional data about the real space based on the spatial image and the sensing data. In one embodiment of the present disclosure, the 'three-dimensional data' may be data that explicitly represents a three-dimensional shape of the surrounding real space, such as a point cloud or a mesh, or may include three-dimensional data in an abstract form, such as a signed distance function. The processor (130) can detect at least one area where a virtual keyboard can be displayed from three-dimensional data about a real space by executing commands (e.g., program code) of the area detection module (142).

[0085] In one embodiment of the present disclosure, the processor (130) may use a plane detection algorithm to detect at least one area including a plane on which a virtual keyboard can be overlaid from three-dimensional data of a real space. The processor (130) may detect a horizontal plane and a vertical plane from the three-dimensional data of the real space, recognize a plane included in a wall and a floor in the real space from the detected horizontal and vertical planes, and detect an area on the recognized plane. However, the embodiments of the present disclosure are not limited thereto, and the processor (130) may also recognize a plane composed of a window or a door as well as a wall and a floor from the three-dimensional data. For example, the processor (130) may recognize a horizontal plane, such as a wall, a floor, or a desk surface in an office, from three-dimensional data of the surrounding environment.

[0086] However, the embodiments of the present disclosure are not limited thereto, and the processor (130) may also detect a surface having a predetermined curvature from three-dimensional data of the surrounding environment. In one embodiment of the present disclosure, the processor (130) may recognize a surface having a curvature similar to a cylinder. The processor (130) may recognize a surface having a curvature of a part of the user's body, such as a palm, the back of the hand, or the thigh, as at least one area on which to overlay a virtual keyboard.

[0087] In one embodiment of the present disclosure, if the distance between the detected area and the user exceeds a preset threshold, the processor (130) may determine that the area is an area where overlay of the virtual keyboard is not possible.

[0088] In one embodiment of the present disclosure, three-dimensional data of the surrounding environment of the augmented reality device (100) may be stored in advance. In this case, the processor (130) may not acquire three-dimensional data of the surrounding real space based on images or sensing data acquired through the camera (110) or sensor (120), but may load the pre-stored three-dimensional data from the memory (140) to acquire the data. An embodiment in which three-dimensional data regarding the surrounding real space of the augmented reality device (100) is pre-stored will be described in detail with reference to FIG. 5.

[0089] The region detection module (142) can provide region detection information regarding at least one detected region to the virtual keyboard determination module (144).

[0090] The virtual keyboard determination module (144) may include (or be configured to include) instructions (e.g., program code) for executing a function and / or operation of determining a virtual keyboard that can be overlaid on at least one area based on profile information of the virtual keyboard. In the present disclosure, the 'virtual keyboard profile information' may include information regarding at least one of the shape, size, and input language of the virtual keyboard. The shape of the virtual keyboard may include, for example, at least one of a full-sized shape including all 106 keys, a split shape that can be divided into multiple keyboard areas, a shape including only number keys, or a 12-key English keyboard shape provided by a mobile device such as a cell phone (12-key telephone keypad). The size of the virtual keyboard may include information regarding the minimum and maximum displayable sizes in which the virtual keyboard is rendered. The input language of the virtual keyboard may include Korean, English, Chinese, Japanese, numbers, or special characters.

[0091] In one embodiment of the present disclosure, profile information of a virtual keyboard is stored in a virtual keyboard data storage unit (148) in a memory (140), and the processor (130) can obtain (e.g., load) profile information of the virtual keyboard from the virtual keyboard data storage unit (148). However, the embodiments of the present disclosure are not limited thereto, and the augmented reality device (100) further includes a communication interface configured to perform data communication with an external device or a server, and the processor (130) can also receive profile information of the virtual keyboard from the server or the external device through the communication interface.

[0092] The processor (130) can determine the type of virtual keyboard that can be overlaid on at least one area detected from the surrounding real space based on the profile information of the virtual keyboard by executing the instructions (e.g., program code) of the virtual keyboard determination module (144). In the present disclosure, the 'type of virtual keyboard' can include, for example, a QWERTY keyboard, a Cheonjiin keyboard, a numeric keyboard, or a 12-key English keyboard.

[0093] In one embodiment of the present disclosure, the processor (130) can configure a region-virtual keyboard combination by matching at least one region with all types of virtual keyboards available. The processor (130) can match multiple types of virtual keyboards to a single region. A specific embodiment of how the processor (130) configures a region-virtual keyboard combination will be described in detail in operation B1 of FIG. 7.

[0094] However, the embodiments of the present disclosure are not limited thereto, and in one embodiment of the present disclosure, the processor (130) may separate a virtual keyboard that can be separated into multiple keyboards and match them to multiple areas. An embodiment in which the processor (130) matches a separate virtual keyboard to multiple areas will be described in detail in FIG. 9.

[0095] The processor (130) may perform an evaluation on a region-virtual keyboard combination based on attribute information of the region including the size and shape of at least one region and at least one of the shape, size, and input language of the virtual keyboards, and may calculate an evaluation score. For example, if the size of a first region among at least one region is smaller than the minimum displayable size of the first virtual keyboard, the processor (130) may provide a low score lower than a reference score to the first combination composed of the first region and the first virtual keyboard. In one embodiment of the present disclosure, the processor (130) may calculate an evaluation score on a region-virtual keyboard combination by considering the distance between the at least one region and the user together with attribute information of the at least one region and profile information including at least one of the shape, size, and input language of the virtual keyboards.

[0096] The processor (130) may determine the type of virtual keyboard that can be overlaid on at least one region based on the evaluation results of the region-virtual keyboard combination. In one embodiment of the present disclosure, the processor (130) may determine that a virtual keyboard can be overlaid on a region only for combinations in which the calculated evaluation score exceeds a preset reference score. The processor (130) may determine that a virtual keyboard cannot be overlaid on a region that constitutes the region-virtual keyboard combination for combinations in which the calculated evaluation score is equal to or lower than the reference score.

[0097] The processor (130) determines a virtual keyboard and an area on which the virtual keyboard is to be overlaid from a combination of at least one area and a type of virtual keyboard based on at least one of an input language, an input field, and usage history information. In one embodiment of the present disclosure, the processor (130) may select an optimal area-virtual keyboard combination based on at least one of an input language, an input field, and usage history information among the area-virtual keyboard combinations. The processor (130) may determine a virtual keyboard included in the selected area-virtual keyboard combination, and may determine to overlay the determined virtual keyboard on an area included in the selected area-virtual keyboard combination. A specific embodiment in which the processor (130) configures an area-virtual keyboard combination using at least one area and a type of virtual keyboard, and determines a virtual keyboard and an area on which the virtual keyboard is to be overlaid from the area-virtual keyboard combination will be described in detail with reference to FIGS. 6 and 7.

[0098] The virtual keyboard determination module (144) can provide information about the determined virtual keyboard and the determined area to the rendering module (146).

[0099] The rendering module (146) may include (or be configured to include) instructions (e.g., program codes) for executing virtual keyboard rendering to display the virtual keyboard on the determined area. The processor (130) may perform rendering to display the virtual keyboard by overlaying it on the determined area by executing the instructions (e.g., program codes) of the rendering module (146). The processor (130) may perform rendering to enlarge or reduce the size of the virtual keyboard by scaling the virtual keyboard to suit the size and shape of the determined area. In one embodiment of the present disclosure, the processor (130) may load and obtain rendering data including image data, text data, or API (Applicable Programming Interface) regarding the size, shape, and input language of the virtual keyboard from the virtual keyboard data storage (148), and render the virtual keyboard using the obtained rendering data.

[0100] In one embodiment of the present disclosure, if the determined area is a surface having a curvature, such as a portion of a user's body part, such as a thigh, the processor (130) may perform rendering by warping the determined virtual keyboard based on the curvature of the surface of the body part. A specific embodiment of the processor (130) warping and rendering the virtual keyboard on the body part will be described in detail in FIG. 10.

[0101] In one embodiment of the present disclosure, when a virtual keyboard is rendered and overlaid on a surface of a user's body part and the surface moves due to the user's movement, the processor (130) tracks the movement and rotation of the surface by photographing the body part through the camera (110), and renders the virtual keyboard based on the moved position and rotation value of the surface obtained as a result of the tracking. A specific embodiment of the processor (130) rendering the virtual keyboard when there is movement of the user's body will be described in detail with reference to FIG. 11.

[0102] In one embodiment of the present disclosure, the processor (130) acquires color information of a determined area and compares the acquired color information with the color of the virtual keyboard, thereby changing the color of all or part of the virtual keyboard. A specific embodiment in which the processor (130) changes the color of the virtual keyboard to contrast with the color of the area to improve the visibility of the virtual keyboard will be described in detail in FIG. 12.

[0103] In one embodiment of the present disclosure, the processor (130) may recognize a user's hand gesture from an image acquired via the camera (110), recognize an area pointed by the user based on the hand gesture, and render and display a virtual keyboard on the recognized area. A specific embodiment in which the processor (130) renders and displays a virtual keyboard on the recognized area based on the user's hand gesture will be described in detail with reference to FIGS. 13 and 14.

[0104] The virtual keyboard data storage unit (148) is a data storage space that stores data regarding a virtual keyboard that can be provided by the augmented reality device (100). In one embodiment of the present disclosure, the virtual keyboard data storage unit (148) may store profile information regarding at least one of the size, shape, and input language of the virtual keyboard. However, the present disclosure is not limited thereto, and the virtual keyboard data storage unit (148) may further include rendering data such as image data, text data, or an API (Applicable Programming Interface) for rendering the virtual keyboard.

[0105] The virtual keyboard data storage unit (148) may be composed of non-volatile memory. Non-volatile memory refers to a storage medium that stores and maintains information even when power is not supplied, and can use the stored information again when power is supplied. The non-volatile memory may include, for example, at least one of flash memory, a hard disk, an SSD (Solid State Drive), a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a ROM (Read Only Memory; ROM), a magnetic memory, a magnetic disk, and an optical disk.

[0106] In FIG. 3, the virtual keyboard data storage unit (148) is illustrated as a component included in the memory (140), but is not limited thereto. In one embodiment of the present disclosure, the virtual keyboard data storage unit (148) may be configured in the form of a database separate from the memory (140). For example, the virtual keyboard data storage unit (148) may be configured as a web storage or cloud server that is accessible via a network and performs a storage function. In this case, the augmented reality device (100) may receive data related to the virtual keyboard by communicating with the web storage or cloud server via a communication interface and performing data transmission and reception to access the virtual keyboard data storage unit (148).

[0107] The display unit (150) is configured to display a virtual keyboard by overlaying it on a determined area under the control of the processor (130). When the augmented reality device (100) is implemented as an augmented reality glass in the shape of glasses, the display unit (150) may be configured with a lens optical system and may include a waveguide and an optical engine. The optical engine may be configured with a projector that generates light of a virtual object composed of a virtual image and projects the light onto a waveguide. The optical engine may include, for example, an image panel, an illumination optical system, a projection optical system, etc. In one embodiment of the present disclosure, the optical engine may be disposed on the frame or temples of the augmented reality glasses. In one embodiment of the present disclosure, the optical engine can generate light of a graphic object rendered as a character, number, special symbol, virtual image, or a combination thereof constituting a virtual keyboard under control of a processor (130), and project the light onto a waveguide to overlay and display the virtual keyboard on an area.

[0108] However, the embodiments of the present disclosure are not limited thereto, and the display unit (150) may be configured as at least one of, for example, a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a 3D display, and an electrophoretic display.

[0109] FIG. 5 is a flowchart illustrating a method for an augmented reality device (100) according to one embodiment of the present disclosure to scan a surrounding real space and detect at least one area.

[0110] Steps S510 to S540 of FIG. 5 are steps that embody step S210 illustrated in FIG. 2. After an operation by either step S530 or S540 illustrated in FIG. 5 is performed, step S220 of FIG. 2 may be performed.

[0111] In step S510, the augmented reality device (100) determines whether three-dimensional data of a real space (real world) is previously stored. In one embodiment of the present disclosure, three-dimensional data regarding a real space including a surrounding real space, for example, a wall, a floor, a desk, etc. in an office, may be previously stored in a memory (140, see FIG. 3). In the present disclosure, 'three-dimensional data' may be data that explicitly expresses a three-dimensional shape of a surrounding real space, such as a point cloud or a mesh, or may include three-dimensional data in an abstract form, such as a signed distance function. The augmented reality device (100) may scan the memory (140) to determine whether three-dimensional data of a real space is stored in the memory (140).

[0112] In one embodiment of the present disclosure, the memory (140) of the augmented reality device (100) may store three-dimensional data regarding a portion of a real space. For example, three-dimensional data regarding half of an area of ​​desks in an office may be stored in the memory (140). If the entire three-dimensional data regarding the real space is not stored, the augmented reality device (100) may determine that the three-dimensional data is not stored in the memory (140).

[0113] In one embodiment of the present disclosure, an augmented reality device (100) includes a communication interface and can receive three-dimensional data about a real space from an external server or external device. In this case, the augmented reality device (100) can determine whether the received three-dimensional data includes three-dimensional data about a real space.

[0114] If three-dimensional data about a real space is not previously stored (step S520), the augmented reality device (100) scans the surrounding environment using at least one of an RGB camera, an infrared sensor, a depth camera, and a LiDAR sensor, thereby acquiring three-dimensional data about a real space. In one embodiment of the present disclosure, the augmented reality device (100) can acquire at least one image frame by photographing the surrounding real space using the RGB camera. In one embodiment of the present disclosure, the augmented reality device (100) can acquire sensing data about a real space using an infrared sensor, a depth camera, or a LiDAR sensor. Since the infrared sensor, the depth camera, and the LiDAR sensor are the same as the infrared sensor (122), the depth camera (124), and the LiDAR sensor (126) described in FIG. 3, overlapping descriptions may be omitted. The augmented reality device (100) can acquire three-dimensional data about a real space based on at least one image frame acquired through the RGB camera and sensing data acquired through the infrared sensor, the depth camera, or the LiDAR sensor.

[0115] In step S530, the augmented reality device (100) performs plane detection to detect at least one area on which a virtual keyboard can be overlaid from the three-dimensional data. The augmented reality device (100) can detect at least one area composed of a plane or a surface with a curvature where no object is detected from the three-dimensional data of the real space using a plane detection algorithm. In one embodiment of the present disclosure, the augmented reality device (100) can detect a horizontal plane and a vertical plane from the three-dimensional data of the real space, recognize a plane included in a wall and a floor in the real space from the detected horizontal plane and vertical plane, and detect an area on the recognized plane where no object is placed. However, the embodiments of the present disclosure are not limited thereto, and the augmented reality device (100) can also recognize a plane composed of a window or a door as well as a wall and a floor from the three-dimensional data. For example, the augmented reality device (100) can recognize a horizontal surface, such as a wall, floor, or desk surface in an office, from three-dimensional data of the surrounding environment. However, the present invention is not limited thereto, and the augmented reality device (100) can also detect a surface having a predetermined curvature from three-dimensional data of the surrounding environment. In one embodiment of the present disclosure, the augmented reality device (100) can recognize a surface having a curvature similar to a cylinder. The processor (130) can also recognize a surface having a curvature of a part of the user's body, such as a palm, the back of the hand, or the thigh.

[0116] The augmented reality device (100) can determine an area on which a virtual keyboard cannot be overlaid among the detected flat or curved surfaces. In one embodiment of the present disclosure, if the distance from the user among the detected flat or curved areas exceeds a preset threshold, the augmented reality device (100) can determine that the area is an area on which a virtual keyboard cannot be overlaid. The 'overlaid-unable area' may include, for example, an area outside the range of approximately 60 centimeters to 80 centimeters, which is the average human arm length.

[0117] If three-dimensional data regarding a real space is previously stored (step S540), the augmented reality device (100) loads the previously stored three-dimensional data and detects at least one area where a virtual keyboard can be overlaid. The augmented reality device (100) can scan the memory (140) and load the previously stored three-dimensional data regarding a real space from the memory (140). The augmented reality device (100) can detect at least one area where a virtual keyboard can be overlaid based on the loaded three-dimensional data.

[0118] However, the embodiments of the present disclosure are not limited thereto, and in one embodiment of the present disclosure, the augmented reality device (100) may receive three-dimensional data about a real space from an external server or an external device, and detect at least one area on which a virtual keyboard may be overlaid from the received three-dimensional data.

[0119] FIG. 6 is a flowchart illustrating a method for determining a type of virtual keyboard that can be overlaid on at least one area by an augmented reality device (100) according to one embodiment of the present disclosure, and determining a virtual keyboard of one of the determined types and an area on which the virtual keyboard is to be overlaid.

[0120] Steps S610 to S630 of FIG. 6 are steps that embody step S220 illustrated in FIG. 2. After the operation according to step S640 illustrated in FIG. 6 is performed, step S230 of FIG. 2 may be performed.

[0121] FIG. 7 is a flowchart illustrating a method for determining a type of virtual keyboard (e.g., a first keyboard (k1), a second keyboard (k2), a third keyboard (k3), a fourth keyboard (k4)) that can be overlaid on at least one area (e.g., a first area (P1), a second area (P2), a third area (P3), a fourth area (P4)) by an augmented reality device (100) according to one embodiment of the present disclosure, and determining a virtual keyboard of any one of the determined types and an area on which the virtual keyboard is to be overlaid. The shape, size, and number (e.g., '4') of at least one area (e.g., first area (P1), second area (P2), third area (P3), fourth area (P4)) illustrated in FIG. 7 and the type, shape, size, and number (e.g., '4') of virtual keyboards (k1, k2, k3, k4) are examples for convenience of explanation, and at least one area and virtual keyboard of embodiments of the present disclosure are not limited to those illustrated in FIG. 7.

[0122] Hereinafter, the functions and / or operations of the augmented reality device (100) will be described in detail with reference to FIGS. 6 and 7.

[0123] Referring to FIG. 6, in step S610, the augmented reality device (100) matches at least one area with all types of virtual keyboards that can be provided by the augmented reality device to configure an area-virtual keyboard combination. Referring to operation B1 of FIG. 7, the processor (130, refer to FIG. 3) of the augmented reality device (100) can generate a total of 16 area-virtual keyboard combinations (701 to 716) by combining the first area (P1) to the fourth area (P4) and the first keyboard (k1) to the fourth keyboard (k4). For example, the processor (130) can configure the first combination (701) by matching the first area (P1) with the first keyboard (k1), which is a QWERTY-type virtual keyboard. In the same manner, the processor (130) can configure a second combination (702) by matching the first area (P1) with the second keyboard (k2), which is a virtual keyboard of the Cheonjiin input method, a third combination (703) by matching the first area (P1) with the third keyboard (k3), which is a virtual keyboard of the 12-key English keypad input method, and a fourth combination (704) by matching the first area (P1) with the fourth keyboard (k4), which is a virtual keyboard of the number input method. Through the above-described method, the augmented reality device (100) can obtain combinations between areas and virtual keyboards including the first combination (701) to the sixteenth combination (716). The configuration and number (for example, 16) of combinations between areas and virtual keyboards illustrated in FIG. 7 are merely examples for the convenience of explanation, and the area-virtual keyboard combinations of the present disclosure are not limited as illustrated in FIG. 7.

[0124] Referring back to FIG. 6, in step S620, the augmented reality device (100) performs an evaluation on a region-virtual keyboard combination based on attribute information of at least one region and at least one of the shape, size, and input language of the virtual keyboard. In one embodiment of the present disclosure, the 'region attribute information' may include information on the size and shape of the region. The augmented reality device (100) may obtain profile information of all types of virtual keyboards that can be provided. In one embodiment of the present disclosure, the 'profile information' may include information on at least one of the shape, size, and input language of the virtual keyboard. Since the profile information of the virtual keyboard and the specific method by which the augmented reality device (100) obtains the profile information of the virtual keyboard are the same as those described in FIGS. 1 to 4, redundant descriptions will be omitted.

[0125] The processor (130) of the augmented reality device (100) may perform an evaluation of a region-virtual keyboard combination based on attribute information of the region, including the size and shape of at least one region, and profile information, including information about at least one of the shape, size, and input language of virtual keyboards. In one embodiment of the present disclosure, the augmented reality device (100) may calculate an evaluation score for the region-virtual keyboard combination by considering the distance between the at least one region and the user, together with the attribute information of the at least one region and the profile information of the virtual keyboards. Referring to the embodiment illustrated in FIG. 7, the augmented reality device (100) may perform an evaluation of the region-virtual keyboard combination based on attribute information of the region included in each of 16 region-virtual keyboard combinations and profile information of the virtual keyboard. For example, the processor (130) may perform an evaluation for each of the first combination (701) to the sixteenth combination (716) and calculate an evaluation score.

[0126] In step S630 of FIG. 6, the augmented reality device (100) determines the type of virtual keyboard that can be overlaid on at least one area based on the evaluation result regarding the area-virtual keyboard combination. In one embodiment of the present disclosure, the augmented reality device (100) may determine that a virtual keyboard can be overlaid on an area only for combinations in which the calculated evaluation score exceeds a preset reference score. Referring to operation B2 of FIG. 7, for example, the processor (130) of the augmented reality device (100) may determine that the first keyboard (k1), the second keyboard (k2), the third keyboard (k3), and the fourth keyboard (k4) are all overlaid on the first area (P1) based on the calculated evaluation scores for the first combination (701) to the fourth combination (704). Likewise, the processor (130) may determine that the first keyboard (k1), the second keyboard (k2), the third keyboard (k3), and the fourth keyboard (k4) are all overlayable in the second area (P2) based on the evaluation scores calculated for the fifth combination (705) to the eighth combination (708). The processor (130) may determine that the first keyboard (k1), which is a QWERTY-type virtual keyboard, is not overlayable in the third area (P3) constituting the ninth combination (709) having an evaluation score lower than a preset threshold among the evaluation scores calculated for the ninth combination (709) to the twelfth combination (712). The processor (130) may determine that the second keyboard (k2), the third keyboard (k3), and the fourth keyboard (k4) are all overlayable in the third area (P3). The processor (130) may determine that the virtual keyboard cannot be overlaid on the corresponding area for the 13th combination (713) and the 14th combination (714) that have an evaluation score lower than a preset threshold among the evaluation scores calculated for the 13th combination (713) to the 16th combination (716).For example, the processor (130) may determine that only the third keyboard (k3) and the fourth keyboard (k4) can be overlaid for the fourth area (P4).

[0127] Referring to step S640 of FIG. 6, the augmented reality device (100) may select an optimal area-virtual keyboard combination based on at least one of an input language, an input field, and usage history information among the area-virtual keyboard combinations. The 'input field' may include, for example, a field for inputting English or Korean characters, such as a login field or a text input field, a field for inputting numbers, such as a password input field, etc. The 'usage history information' may include, for example, history information about the frequency of use of a specific type of keyboard (e.g., QWERTY keyboard, Cheonjiin keyboard, etc.), the type and size of the keyboard most recently used by the user, etc.

[0128] The augmented reality device (100) may determine to overlay a virtual keyboard included in the combination of the selected area and the virtual keyboard on the selected area. Referring to operation B3 of FIG. 7, the processor (130) of the augmented reality device (100) may determine the tenth combination (710) as the optimal combination among the first combination (701) to the sixteenth combination (716) based on at least one of the input language, the input field, and the usage history information. For example, if the language that the user wants to input is Korean, the input field is a field for entering text, and the type of keyboard frequently used by the user is the Cheonjiin keyboard, the processor (130) may comprehensively consider the input language, the input field, and the usage history information to determine the second keyboard (k2) as the third area (P3) and the virtual keyboard to be overlaid on the third area (P3).

[0129] FIG. 8a is a drawing illustrating an embodiment in which the augmented reality device (100) of the present disclosure overlays and displays a first virtual keyboard (810) in the form of QWERTY on a first area (800a).

[0130] Referring to FIG. 8A, the processor (130, see FIG. 3) of the augmented reality device (100) may determine a first area (800a) as an area on which to overlay a virtual keyboard, and may determine a type of virtual keyboard that can be overlaid on the first area (800a) based on attribute information including the size and shape of the first area (800a) and profile information of the virtual keyboard. In the embodiment illustrated in FIG. 8A, the first area (800a) may be a surface of a desk on which real objects such as a doll, a water bottle, a can, and a clock are placed. The processor (130) may obtain attribute information regarding the size and shape of the first area (800a) on the desk from an image obtained through a camera (110, see FIG. 3) or sensing data obtained through a sensor (120). The processor (130) can determine a first virtual keyboard (810) in QWERTY format as a virtual keyboard to be overlaid on the first area (800a) based on the attribute information of the first area (800a) and profile information including at least one of the shape, size, and input language of the virtual keyboard in QWERTY format.

[0131] FIG. 8b is a drawing illustrating an embodiment in which the augmented reality device (100) of the present disclosure overlays and displays a second virtual keyboard (820) of the Cheonjiin input method on a second area (800b).

[0132] Referring to FIG. 8b, the processor (130, see FIG. 3) of the augmented reality device (100) may determine a second area (800b) as an area on which to overlay a virtual keyboard, and may determine the type of virtual keyboard that can be overlaid on the second area (800b) based on attribute information including the size and shape of the second area (800b) and profile information of the virtual keyboard. In the embodiment illustrated in FIG. 8b, the second area (800b) is a surface of a desk on which real objects such as a monitor, a water bottle, and a note are arranged, and the size of the second area (800b) may be smaller than the size of the first area (800a, see FIG. 8a). The processor (130) may obtain attribute information regarding the size and shape of the second area (800b) on the desk from an image obtained through a camera (110, see FIG. 3) or sensing data obtained through a sensor (120). The processor (130) can determine a second virtual keyboard (820) of the Cheonjiin input method as a virtual keyboard to be overlaid on the second area (800b) based on the attribute information of the second area (800b) and profile information including at least one of the shape, size, and input language of the virtual keyboard in QWERTY format.

[0133] FIG. 8c is a diagram illustrating an embodiment in which the augmented reality device (100) of the present disclosure overlays and displays a third virtual keyboard (830) in the form of number keys on a third area (800c). In the embodiment illustrated in FIG. 8c, the size and shape of the third area (800c) are substantially the same as the size and shape of the second area (800b) illustrated in FIG. 8b, and therefore, any redundant description regarding the third area (800c) may be omitted.

[0134] Referring to FIG. 8c, the processor (130, see FIG. 3) of the augmented reality device (100) determines a third area (800c) as an area to overlay a virtual keyboard, and determines a third keyboard (830), which is a virtual keyboard of a numeric key input method, as a virtual keyboard that can be overlaid on the third area (800c) based on attribute information including the size and shape of the third area (800c) and profile information of the virtual keyboard. The embodiment illustrated in FIG. 8c is the same as the embodiment illustrated in FIG. 8b except that the virtual keyboard to be overlaid on the third area (800c) is the third keyboard (830), and therefore, redundant descriptions may be omitted.

[0135] FIG. 8d is a diagram illustrating an embodiment in which the augmented reality device (100) of the present disclosure overlays and displays a fourth virtual keyboard (840) in a 12-key English keypad input method on a fourth area (800d). In the embodiment illustrated in FIG. 8d, the size and shape of the fourth area (800d) are substantially the same as the size and shape of the second area (800b) illustrated in FIG. 8b, and therefore, any redundant description regarding the fourth area (800d) may be omitted.

[0136] Referring to FIG. 8d, the processor (130, see FIG. 3) of the augmented reality device (100) determines a fourth area (800d) as an area on which to overlay a virtual keyboard, and determines a fourth keyboard (840), which is a virtual keyboard of a 12-key English keypad input method, as a virtual keyboard that can be overlaid on the fourth area (800d) based on attribute information including the size and shape of the fourth area (800d) and profile information of the virtual keyboard. The embodiment illustrated in FIG. 8d is the same as the embodiment illustrated in FIG. 8b except that the virtual keyboard to be overlaid on the fourth area (800d) is the fourth keyboard (840), and therefore, redundant descriptions may be omitted.

[0137] In one embodiment of the present disclosure, the processor (130, see FIG. 3) of the augmented reality device (100) may determine a virtual keyboard to be overlaid on an area based on at least one of an input language, an input field, and user usage history information among the types of virtual keyboards. For example, if the input language is Korean, the input field is a field for entering text, and history information indicating that the user frequently used the Cheonjiin keyboard is confirmed, the processor (130) may determine a second keyboard (820) of the Cheonjiin input method as illustrated in FIG. 8b as the virtual keyboard to be overlaid on the second area (800b). For example, if the input language is a number, and the input field is a number input field for entering an account number or amount in a calculator or banking application, the processor (130) may determine a third keyboard (830), which is a virtual keyboard for number input, as illustrated in FIG. 8c as the virtual keyboard to be overlaid on the third area (800c). For example, if the input language is English and numbers, and history information is confirmed that the user frequently used a 12-key input method keyboard, the processor (130) may determine the fourth keyboard (840), which is a virtual keyboard of the 12-key English keypad input method, as shown in FIG. 8d, as the virtual keyboard to be overlaid on the fourth area (800d).

[0138] FIG. 9 is a diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure to overlay and display a detachable keyboard (910) on a plurality of areas (e.g., a first area (900-1), a second area (900-2), and a third area (900-3)).

[0139] Referring to FIG. 9, the augmented reality device (100) can match a splittable keyboard (910) that can be split into multiple parts to be displayed overlaid on multiple areas (e.g., a first area (900-1), a second area (900-2), and a third area (900-3)). In one embodiment of the present disclosure, the splittable keyboard (910) can be split into multiple keyboards (e.g., a left keyboard (910-1) and a right keyboard (910-2)). For example, the splittable keyboard (910) is a virtual keyboard having an input method of a QWERTY keyboard, and can be split into two keyboards including a left keyboard (910-1) and a right keyboard (910-2). In the embodiment illustrated in FIG. 9, the processor (130, see FIG. 3) of the augmented reality device (100) can match the left keyboard (910-1) among a plurality of regions (e.g., the first region (900-1), the second region (900-2), and the third region (900-3)) to be overlaid on the first region (900-1), and can match the right keyboard (910-2) to be overlaid on the third region (900-3).

[0140] Although the split keyboard (910) illustrated in FIG. 9 is illustrated and described as being separated into two keyboards including a left keyboard (910-1) and a right keyboard (910-2), this is exemplary and the split keyboards of the embodiments of the present disclosure are not limited to as illustrated in FIG. 9.

[0141] As in the embodiment illustrated in FIG. 9, when many real objects (e.g., monitors, bags, stand lights, etc.) are arranged on a plane on a desk, and the size of the area is smaller than the size of a virtual keyboard of a QWERTY input method, the augmented reality device (100) can divide the split keyboard (910) into a plurality of keyboards (e.g., a left keyboard (910-1) and a right keyboard (910-2)) and overlay them on a plurality of areas (e.g., a first area (900-1), a second area (900-2), and a third area (900-3)). The augmented reality device (100) according to one embodiment of the present disclosure can improve the usability of the area by overlaying the split keyboard (910) even when the size of the overlayable area of ​​the virtual keyboard is small due to the arrangement of real objects on the plane.

[0142] FIG. 10 is a drawing illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure to overlay a virtual keyboard on a part of a user's body part (1000).

[0143] Referring to FIG. 10, the augmented reality device (100) detects at least one area among a portion of a user's body part (1000) (operation C1). In one embodiment of the present disclosure, the augmented reality device (100) may detect a flat or curved surface among the user's body parts (1000) from an image obtained by photographing the user's body through a camera (110, see FIG. 3) or sensing data obtained through a sensor (120, see FIG. 3). The processor (130, see FIG. 3) of the augmented reality device (100) may detect an area (1010) on which a virtual keyboard can be overlaid among the detected flat or curved surfaces. In one embodiment of the present disclosure, the 'area (1010) on which a virtual keyboard can be overlaid' may include an area on a flat surface or a curved surface having a curvature where no object is detected. The area (1010) on which the virtual keyboard can be overlaid may be one area or a plurality of areas.

[0144] In the embodiment illustrated in FIG. 10, the processor (130) may detect the thigh among the user's body parts as an area (1010) on which a virtual keyboard may be overlaid. For example, the area (1010) on which the virtual keyboard may be overlaid may be two separate areas, such as a portion of the user's left thigh and a portion of the user's right thigh. However, the embodiments of the present disclosure are not limited to the area illustrated in FIG. 10, and the processor (130) may detect other body parts of the user, such as the palm, the back of the hand, the wrist, the shoulder, etc., as an area (1010) on which a virtual keyboard may be overlaid.

[0145] The augmented reality device (100) determines the type of virtual keyboard that can be overlaid on at least one area (1010) (Operation C2). The processor (130) of the augmented reality device (100) can determine the type of virtual keyboard that can be overlaid on at least one area (1010) based on attribute information of the at least one area (1010) including the size and shape of the at least one area and profile information including information on at least one of the shape, size, and input language of virtual keyboards. In one embodiment of the present disclosure, the processor (130) can configure an area-virtual keyboard combination by matching all types of virtual keyboards that can be provided with the at least one area (1010), perform an evaluation on the area-virtual keyboard combination based on the attribute information of the area and the profile information of the virtual keyboard, and determine the type of virtual keyboard that can be overlaid on the at least one area (1010) based on the evaluation result. In the embodiment illustrated in FIG. 10, the processor (130) may determine a detachable virtual keyboard (1020) as a virtual keyboard that can be overlaid on the area (1010) on the user's left thigh and right thigh.

[0146] The augmented reality device (100) warps the determined virtual keyboard (1020) based on the curvature of the area (Operation C3). In one embodiment of the present disclosure, if the determined area is a surface having a curvature of a part of a user's body (in the embodiment illustrated in FIG. 10, a 'thigh'), the processor (130) of the augmented reality device (100) may perform rendering to warp the determined virtual keyboard (1020) based on the curvature of the surface of the body part. Here, 'warping' refers to an image processing technique that changes the positions of pixels constituting an image. Image warping is a type of geometric transformation and may change the positions of pixels of an original image. Image warping may be performed based on parameters including a transformation function for changing the positions of pixels.

[0147] In one embodiment of the present disclosure, the processor (130) may obtain depth value information of a body part, obtain a warping parameter based on the obtained depth value information, and warp the virtual keyboard (1020) based on the obtained warping parameter. However, the embodiments of the present disclosure are not limited thereto, and in one embodiment of the present disclosure, the processor (130) may warp the virtual keyboard (1020) based on a warping parameter value preset for each body part. Referring to the embodiment illustrated in FIG. 10, the augmented reality device (100) may obtain a warped virtual keyboard (1030) based on the curvature of an area (1010) on the thigh by performing warping.

[0148] In the case where a non-planar curved surface such as a thigh, palm, wrist, or back of the hand of a user is determined as the area (1010) on which a virtual keyboard (1020) is overlaid, and the virtual keyboard (1020) is overlaid in a planar form, the virtual keyboard (1020) and the area (1010) formed by the curved surface do not completely match, and there is a gap, which may cause inconvenience to the user when operating the virtual keyboard (1020), and the recognition rate may decrease when a key is input. The augmented reality device (100) according to the embodiment illustrated in FIG. 10 can provide a technical effect of improving the convenience of operation and the recognition rate of a key input by overlaying a warped virtual keyboard (1030) obtained by warping the virtual keyboard (1020) based on the curvature of an area (1010) (e.g., a partial area) of the user's body part (1000).

[0149] FIG. 11 is a drawing illustrating an operation in which an augmented reality device (100) according to one embodiment of the present disclosure tracks the movement of a user's body part (1100) and displays a virtual keyboard (1120).

[0150] Referring to FIG. 11, the augmented reality device (100) overlays a virtual keyboard on a part of a user's body part (1100) (operation D1). The processor (130, see FIG. 3) of the augmented reality device (100) can detect a flat or curved surface among the user's body parts (1100) from an image obtained by photographing the user's body through a camera (110, see FIG. 3) or sensing data obtained through a sensor (120, see FIG. 3), and can detect an area (1110) on which a virtual keyboard can be overlaid among the detected flat or curved surfaces. The processor (130) can determine a type of virtual keyboard that can be overlaid on the detected area (1110) and display the determined virtual keyboard (1120) by overlaying it on the detected area (1110). The specific method of the processor (130) detecting a certain area (1110) of a user's body part (1100) and determining the type of virtual keyboard (1120) that can be overlaid on the area (e.g., the area (1110) detected) is the same as that described in FIGS. 1 to 4 and FIG. 10, so redundant descriptions are omitted. In the embodiment illustrated in FIG. 11, the processor (130) can detect an area (1110) on the user's body part (1100) (e.g., the palm) and display a virtual keyboard (1120) of the Cheonjiin input method by overlaying it on the detected area (1110).

[0151] The augmented reality device (100) tracks the movement and rotation of the area (1110) due to the movement of the body part (1100) (operation D2). In one embodiment of the present disclosure, when the area (1110) of the user's body part (1100) is moved due to the user's movement while the virtual keyboard (1120) is overlaid on the area (1110), the processor (130) can capture the body part (1100) through the camera (110) to obtain multiple image frames, recognize the moved area (1110') from the obtained multiple image frames, and track the position and rotation of the moved area (1110'). As a result of the tracking, the processor (130) can obtain the position and rotation values ​​of the moved area (1110'). In the embodiment illustrated in FIG. 11, the processor (130) can obtain information on the distance (d) and rotation angle (θ) between the area (1110') moved as a result of tracking and the area (1110) before moving.

[0152] The augmented reality device (100) renders a virtual keyboard (1120) based on the position and rotation value of the area obtained as a result of tracking (Operation D3). In the embodiment illustrated in FIG. 11, the processor (130) of the augmented reality device (100) can render the virtual keyboard (1120) based on the distance (d) and the rotation angle (θ) between the moved area (1110') and the area before being moved. The processor (130) can display the rendered virtual keyboard (1120) by overlaying it on the moved area (1110').

[0153] In the augmented reality device (100) according to the embodiment illustrated in FIG. 11, when a virtual keyboard (1120) is displayed overlaid on a user's body part (1100) and a relative position movement or rotation value change occurs due to movement of the body part (1100) or movement of the head, the virtual keyboard (1120) is rendered and displayed on the moved area (1110'), thereby enabling the user to operate the virtual keyboard (1120) with continuity, thereby improving operational convenience.

[0154] FIG. 12 is a flowchart illustrating a method for changing the color of a virtual keyboard based on color information of an area where the virtual keyboard is overlaid by an augmented reality device (100) according to one embodiment of the present disclosure.

[0155] Steps S1210 to S1230 illustrated in FIG. 12 are steps that embody the operation of step S230 of FIG. 2. Step S1210 illustrated in FIG. 12 may be performed after the operation of step S220 of FIG. 2 is performed.

[0156] In step S1210, the augmented reality device (100) acquires color information of the determined area. In one embodiment of the present disclosure, the camera (110, see FIG. 3) of the augmented reality device (100) is controlled to capture an area and acquire an image, and the processor (130, see FIG. 3) can analyze the image acquired by the camera (110) to acquire color information of the area. For example, the processor (130) can acquire color information of the area from the image using a known image processing technique. However, the embodiments of the present disclosure are not limited thereto, and the processor (130) can also acquire color information of the area using an artificial intelligence model trained to output color information when an image is input.

[0157] In step S1220, the augmented reality device (100) compares the acquired color information with the determined color of the virtual keyboard. In one embodiment of the present disclosure, the processor (130) of the augmented reality device (100) can acquire color information of the virtual keyboard from profile information of the virtual keyboard. The processor (130) can compare the color information of the area acquired in step S1210 with the color of the virtual keyboard.

[0158] In step S1230, the augmented reality device (100) changes the color of all or part of the virtual keyboard based on the comparison result. For example, if the color of the area is black, the processor (130) of the augmented reality device (100) may change the color of the virtual keyboard to a color with higher visibility than black, such as white, yellow, or gray. In one embodiment of the present disclosure, the processor (130) may change the color of the virtual keyboard to a color that has a complementary color relationship with the color of the area. For example, if the color of the area is green, the processor (130) may change the color of the virtual keyboard to red, which is a complementary color to green. As another example, if the color of the area is yellow, the processor (130) may change the color of the virtual keyboard to purple, which is a complementary color to yellow.

[0159] The processor (130) may change the overall color of the virtual keyboard, but is not limited thereto. In one embodiment of the present disclosure, the processor (130) may change the color of a portion of the virtual keyboard or each of the character keys of the virtual keyboard.

[0160] The augmented reality device (100) according to the embodiment illustrated in FIG. 12 can provide a technical effect of improving the visibility of the virtual keyboard by obtaining color information of an area and changing the color of the virtual keyboard to a color that can be contrasted with the color of the area.

[0161] FIG. 13 is a flowchart illustrating a method for an augmented reality device (100) according to one embodiment of the present disclosure to display a virtual keyboard on an area determined based on a user's hand gesture.

[0162] FIG. 14 is a diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure to display a virtual keyboard (e.g., a first keyboard (1410)) on an area (e.g., a second area (P2)) determined based on a user's hand gesture.

[0163] Hereinafter, the functions and / or operations of the augmented reality device (100) will be described in detail with reference to FIGS. 13 and 14.

[0164] In step S1310, the augmented reality device (100) captures a user's hand through a camera (110, see FIG. 14) and recognizes a hand gesture for displaying a virtual keyboard. Referring also to the embodiment illustrated in FIG. 14, the augmented reality device (100) captures a user's hand through the camera (110) to obtain an image, and the processor (130, see FIG. 3) recognizes the user's hand from the obtained image and can recognize a hand gesture made by the recognized user's hand. Since the hand gesture recognition technology is a well-known technology in the art, a detailed description thereof may be omitted.

[0165] In step S1320, the augmented reality device (100) recognizes an area pointed by the user based on the recognized hand gesture. In one embodiment of the present disclosure, the augmented reality device (100) can recognize an area pointed by the user's hand from an image based on a recognition result of the hand gesture. Referring to the embodiment illustrated in FIG. 14, the first area (P1) to the third area (P3) are areas determined as areas on which a virtual keyboard can be overlaid, and the first area (P1) and the second area (P2) may be a wall surface, and the third area (P3) may be a flat surface on a desk. In operation E1 of FIG. 14, the processor (130) of the augmented reality device (100) can recognize a hand gesture of the user pointing to any one of the first area (P1), the second area (P2), and the third area (P3) included in the surrounding real space, and recognize the second area (P2) pointed by the user based on the recognition result of the hand gesture.

[0166] In step S1330, the augmented reality device (100) determines whether there is a virtual keyboard that can be overlaid on the recognized area. In one embodiment of the present disclosure, the processor (130) of the augmented reality device (100) can identify the type of virtual keyboard that can be overlaid on the area based on attribute information including the size and shape of the recognized area and profile information regarding at least one of the shape, size, and input language of the virtual keyboard. The processor (130) matches the recognized area with all types of virtual keyboards that can be provided by the augmented reality device (100) to configure an area-virtual keyboard combination, performs an evaluation on the area-virtual keyboard combination based on the attribute information of the recognized area and the profile information of the virtual keyboard, and determines whether there is a virtual keyboard that can be overlaid on the area based on the evaluation result. A specific method by which the processor (130) determines or identifies the type of virtual keyboard that can be overlaid on the area is the same as the method described in FIGS. 6 and 7, and therefore, redundant descriptions may be omitted. Referring to the embodiment illustrated in FIG. 14, the processor (130) can identify a first keyboard (1410) as an overlayable virtual keyboard, which is a virtual keyboard of the QWERTY input method, a second keyboard (1420) as a virtual keyboard of the Cheonjiin input method, a third keyboard (1430) as a virtual keyboard for number input, and a fourth keyboard (1440) as a 12-key English keypad.

[0167] If it is determined that there is a virtual keyboard that can be overlaid on the recognized area (step S1340), the augmented reality device (100) determines a virtual keyboard that can be overlaid on the recognized area among at least one area. In one embodiment of the present disclosure, the processor (130) of the augmented reality device (100) may determine any one of the types of virtual keyboards that can be overlaid on the area based on at least one of an input language, an input field, and usage history information. Referring also to the example of operation E2 of FIG. 14, if the input language is Korean or English, the input field is a field for inputting text, and the keyboard most recently used by the user is a virtual keyboard of the QWERTY input method, the processor (130) may determine a first keyboard (1410) that is a QWERTY keyboard as the virtual keyboard to be overlaid on the recognized area.

[0168] Steps S1350 to S1380 illustrated in FIG. 13 are steps that specify the operation of step S230 of FIG. 2. In step S1350, the augmented reality device (100) outputs a notification message (1450, see FIG. 14) asking whether to display the determined virtual keyboard. Referring also to operation E3 of FIG. 14, the augmented reality device (100) may output a notification message (1450) asking, "Do you want to display the Cheonjiin keyboard?"

[0169] In step S1360, the augmented reality device (100) determines whether to receive user input regarding consent.

[0170] However, embodiments of the present disclosure are not limited thereto. In one embodiment of the present disclosure, steps S1350, S1360, and operation E3 of FIG. 14 may not be performed and may be omitted.

[0171] When a user's consent input for a notification message is received (step S1370), the augmented reality device (100) renders and displays the determined virtual keyboard on the recognized area. Referring to operation E4 of FIG. 14, the processor (130) of the augmented reality device (100) may render the determined virtual keyboard, the first keyboard (1410), and control the display unit (150, see FIG. 3) to overlay the first keyboard (1410) on the second area (P2). In one embodiment of the present disclosure, the processor (130) may control the optical engine of the display unit (150) to generate light of a graphic object composed of letters, numbers, special symbols, virtual images, or a combination thereof that constitute the rendered first keyboard (1410), and project the light onto a waveguide, thereby overlaying and displaying the first keyboard (1410) on the second area (P2).

[0172] In step S1330, if it is determined that there is no virtual keyboard that can be overlaid on the recognized area (step S1380), the augmented reality device (100) renders and displays a virtual keyboard set as default. The virtual keyboard set as default may be preset by the user. However, the embodiments of the present disclosure are not limited thereto, and the virtual keyboard set as default may be preset when the augmented reality device (100) is shipped from the factory.

[0173] In step S1360, if the user's consent input for the notification message is not received (step S1380), the augmented reality device (100) renders and displays a virtual keyboard set as default.

[0174] In the embodiments of FIGS. 13 and 14, the augmented reality device (100) is illustrated and described as overlaying and displaying a virtual keyboard (for example, the 'first keyboard (1410)' in the embodiment illustrated in FIG. 14) on a recognized area (for example, the 'second area (P2)' in the embodiment illustrated in FIG. 14) based on a user's hand gesture input, but the embodiments of the present disclosure are not limited thereto. In one embodiment of the present disclosure, the augmented reality device (100) may include at least one gaze tracking sensor, detect a gaze direction of both eyes of the user through the gaze tracking sensor, determine an area where the user gazes based on a gaze point where the detected gaze directions of both eyes converge, and display a virtual keyboard overlayed on the determined area.

[0175] FIG. 15 is a flowchart illustrating a method for an augmented reality device (100) according to one embodiment of the present disclosure to determine a virtual keyboard and an area in which the virtual keyboard is to be displayed based on a context.

[0176] Steps S1510 and S1520 illustrated in FIG. 15 are steps performed between the operations of steps S220 and S230 of FIG. 2. Step S1510 illustrated in FIG. 15 may be performed after the operation of step S220 of FIG. 2 is performed. After step S1520 illustrated in FIG. 15 is performed, the operation of step S230 of FIG. 2 may be performed.

[0177] FIG. 16 is a flowchart illustrating a method by which an augmented reality device (100) according to one embodiment of the present disclosure determines a virtual keyboard (1620) and an area (1610) in which the virtual keyboard is to be displayed based on a context.

[0178] Hereinafter, the functions and / or operations of the augmented reality device (100) will be described in detail with reference to FIGS. 15 and 16.

[0179] In step S1510 of FIG. 15, the augmented reality device (100) recognizes the context based on at least one of the user's location, the characteristics of the space, and the usage history of the augmented reality device. The augmented reality device (100) can recognize the characteristics of the location and space from an image acquired by photographing the surrounding real space through a camera (110, see FIG. 16). In one embodiment of the present disclosure, the augmented reality device (100) includes a location sensor such as a GPS sensor, and can obtain information about the user's current location using the location sensor. Referring also to the embodiment illustrated in FIG. 16, the augmented reality device (100) obtains an image by photographing a front door (1600) through a camera (110), and the processor (130, see FIG. 3) recognizes the door (1600) and the door lock (1602) from the image, and can recognize the user's location and the characteristics of the space based on the recognized door (1600) and the door lock (1602). In one embodiment of the present disclosure, the processor (130) may acquire history information on the user's frequent use of the virtual keyboard (1620) for entering a password in a situation where a door (1600) and a door lock (1602) are recognized. In this case, the processor (130) may recognize the context for entering the password of the door lock (1602) based on the user's current location (e.g., in front of the door), characteristics of the space (e.g., a hallway in front of the door), and usage history information (e.g., password entry).

[0180] Referring back to FIG. 15, in step S1520, the augmented reality device (100) determines a virtual keyboard and an area on which the virtual keyboard is to be overlaid based on the input language, input field, usage history information, and context. Referring to the embodiment illustrated in FIG. 16, since the password of the door lock (1602) is composed of numbers, the input language and input field are numbers, and if history information that the user frequently used the numeric keypad in a context in which the door lock (1602) is recognized is confirmed, the processor (130) of the augmented reality device (100) may determine a virtual keyboard (1620) of a numeric input method as the virtual keyboard to be overlaid based on the context. For example, the processor (130) may display the determined virtual keyboard (1620) of the numeric input method to be overlaid on the door (1600). However, the embodiments of the present disclosure are not limited thereto, and the processor (130) may also display the virtual keyboard (1620) by overlaying it on a wall next to the door (1600).

[0181] According to one embodiment of the present disclosure, a method for displaying a virtual keyboard by an augmented reality device (100) is provided. In one embodiment of the present disclosure, the method may include a step (S210) of scanning a surrounding real world to detect at least one area including a plane where no object is detected. The method may include a step (S220) of determining a type of virtual keyboard that can be overlaid on at least one detected area based on at least one of a shape, a size, and an input language of the virtual keyboard. The method may include a step (S230) of performing rendering to overlay and display the determined type of virtual keyboard on the at least one area.

[0182] In one embodiment of the present disclosure, the step of detecting at least one area (S210) may include a step of obtaining three-dimensional data about a real space by scanning a surrounding environment using at least one of an RGB camera, an infrared sensor (122), a depth camera (124), and a LiDAR sensor (126). The step of detecting at least one area (S210) may include a step of performing plane detection to detect at least one area composed of a surface having a plane on which a virtual keyboard can be overlaid from the obtained three-dimensional data.

[0183] In one embodiment of the present disclosure, the step of detecting at least one area may include a step of detecting at least one area including a curved surface having curvature from the acquired three-dimensional data.

[0184] In one embodiment of the present disclosure, profile information of a virtual keyboard including at least one of the shape, size, and input language of the virtual keyboards may be stored in the memory (140) of the augmented reality device (100). The method may further include a step of loading the profile information of the virtual keyboard from the memory (140) to obtain the profile information of the virtual keyboards.

[0185] In one embodiment of the present disclosure, the step of determining the type of the virtual keyboard (S220) may include a step of configuring a region-virtual keyboard combination (S610) by matching at least one region with all types of virtual keyboards that can be provided by the augmented reality device. The step of determining the type of the virtual keyboard (S220) may include a step of evaluating the region-virtual keyboard combination based on attribute information of the region, including the size and shape of the at least one region, and at least one of the shape, size, and input language of the virtual keyboards. The step of determining the type of the virtual keyboard (S220) may include a step of determining the type of virtual keyboard that can be overlaid on at least one region based on the evaluation result of the region-virtual keyboard combination (S630).

[0186] In one embodiment of the present disclosure, the step (S610) of configuring the area-virtual keyboard combination may include a step of matching a plurality of virtual keyboards that can be overlaid on at least one area.

[0187] In one embodiment of the present disclosure, the virtual keyboard may include a detachable, split keyboard. The step (S610) of configuring the region-virtual keyboard combination may include a step of dividing the split keyboard into multiple virtual keyboards and matching them to multiple regions.

[0188] In one embodiment of the present disclosure, the method may further include a step (S640) of determining a virtual keyboard and an area on which the virtual keyboard is to be overlaid from a combination of at least one area and a type of overlayable virtual keyboard, based on at least one of an input language, an input field, and a user's usage history information.

[0189] In one embodiment of the present disclosure, the step of detecting at least one area (S210) may include a step of detecting a surface having a curvature of a part of the user's body. The step of performing the rendering (S230) may include a step of warping the determined virtual keyboard based on the curvature of the surface.

[0190] In one embodiment of the present disclosure, the method may further include a step of tracking the movement and rotation of the surface by photographing the body part via a camera (110) when the surface moves due to movement of the user's body part. The step of performing the rendering (S230) may include a step of rendering a virtual keyboard based on the moved position and rotation value of the surface obtained as a result of the tracking.

[0191] In one embodiment of the present disclosure, the step of performing the rendering (S230) may include a step of obtaining color information of a determined area (S1210) and a step of comparing the obtained color information with a color of a determined virtual keyboard (S1220). The step of performing the rendering (S230) may include a step of changing the color of all or part of the virtual keyboard based on the comparison result (S1230).

[0192] In one embodiment of the present disclosure, the method may further include a step (S1310) of recognizing a hand gesture of the user for displaying a virtual keyboard by photographing the user's hand through a camera (110), and a step (S1320) of recognizing an area pointed by the user based on the recognized hand gesture. The step (S220) of determining the type of the virtual keyboard may include a step (S1340) of determining a type of virtual keyboard that can be overlaid on the recognized area among at least one area. The step (S230) of performing the rendering may include a step (S1370) of rendering the determined virtual keyboard on the recognized area.

[0193] According to one embodiment of the present disclosure, an augmented reality device (100) displaying a virtual keyboard may be provided. The augmented reality device (100) according to one embodiment of the present disclosure may include at least one camera (110); a sensor (120) including at least one of an infrared sensor (122), a depth camera (124), and a LiDAR sensor (126); a memory (140) storing at least one instruction; and at least one processor (130) executing the at least one instruction. The at least one processor (130) may scan a surrounding real world using at least one of the camera (110) and the sensor (120), thereby detecting at least one area including a plane where no object is detected. The at least one processor (130) may determine a type of virtual keyboard that can be overlaid on the at least one detected area based on at least one of a shape, a size, and an input language of the virtual keyboard. The above at least one processor (130) can perform rendering to display a virtual keyboard of a determined type by overlaying it on at least one area.

[0194] In one embodiment of the present disclosure, the at least one processor (130) can acquire three-dimensional data about a real space by scanning the surrounding environment using at least one of the at least one camera (110), the infrared sensor (122), the depth camera (124), and the lidar sensor (126). The at least one processor (130) can perform plane detection to detect at least one area including a plane on which a virtual keyboard can be overlaid from the three-dimensional data.

[0195] In one embodiment of the present disclosure, the at least one processor (130) can detect at least one area including a curved surface having a curvature from the acquired three-dimensional data.

[0196] In one embodiment of the present disclosure, profile information of a virtual keyboard, including at least one of the shape, size, and input language of the virtual keyboards, may be stored in a memory (140). The at least one processor (130) may load the profile information of the virtual keyboard from the memory (140) to obtain the profile information of the virtual keyboards.

[0197] In one embodiment of the present disclosure, the at least one processor (130) can configure a region-virtual keyboard combination by matching at least one region with all types of virtual keyboards that can be provided by the augmented reality device. The at least one processor (130) can perform an evaluation on the region-virtual keyboard combination based on attribute information of the region, including the size and shape of the at least one region, and at least one of the shape, size, and input language of the virtual keyboards. The at least one processor (130) can determine a type of virtual keyboard that can be overlaid on the at least one region based on the evaluation result on the region-virtual keyboard combination.

[0198] In one embodiment of the present disclosure, the virtual keyboard may include a detachable, split keyboard. The at least one processor (130) may divide the split keyboard into multiple virtual keyboards and match them to multiple areas.

[0199] In one embodiment of the present disclosure, the at least one processor (130) can determine a virtual keyboard and an area on which the virtual keyboard is to be overlaid from a combination of at least one area and a type of overlayable virtual keyboard based on at least one of an input language, an input field, and a user's usage history information.

[0200] In one embodiment of the present disclosure, the at least one processor (130) can detect a surface having a curvature of a part of a user's body, and warp the determined virtual keyboard based on the curvature of the surface.

[0201] In one embodiment of the present disclosure, the at least one processor (130) can track the movement and rotation of the surface from an image obtained by photographing the body part via the camera (110) when the surface moves due to movement of the user's body part. The at least one processor (130) can render a virtual keyboard based on the moved position and rotation value of the surface obtained as a result of the tracking.

[0202] In one embodiment of the present disclosure, the at least one processor (130) may acquire color information of a determined area and compare the acquired color information with the color of a determined virtual keyboard. The at least one processor (130) may change the color of all or part of the virtual keyboard based on the comparison result.

[0203] In one embodiment of the present disclosure, the at least one processor (130) controls the camera (110) to capture a picture of the user's hand, thereby recognizing a hand gesture of the user for displaying a virtual keyboard, and recognizing an area pointed by the user based on the recognized hand gesture. The at least one processor (130) can determine a type of virtual keyboard that can be overlaid on the recognized area among at least one area. The at least one processor (130) can render the determined virtual keyboard on the recognized area.

[0204] Another aspect of the present disclosure provides a computer program product comprising a computer-readable storage medium. The storage medium may include instructions readable by an augmented reality device (100), such that the augmented reality device (100) performs the following operations: scanning a surrounding real world to detect at least one area including a plane where no object is detected; determining a type of virtual keyboard that can be overlaid on the detected at least one area based on at least one of a shape, a size, and an input language of the virtual keyboard; and performing rendering to overlay and display the determined type of virtual keyboard on the at least one area.

[0205] The program executed by the augmented reality device (100) described in the present disclosure may be implemented as hardware components, software components, and / or a combination of hardware components and software components. The program may be executed by any system capable of executing computer-readable instructions.

[0206] Software may include instructions (e.g., computer programs and / or code) that can configure a processing device or command the processing device, either independently or collectively, to do something as desired.

[0207] Software may be implemented as a computer program containing instructions stored on a computer-readable storage medium. Examples of computer-readable storage media include magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disks, hard disks, etc.) and optical readable media (e.g., CD-ROMs, DVDs (Digital Versatile Discs)). The computer-readable storage media may be distributed across network-connected computer systems, so that computer-readable code may be stored and executed in a distributed manner. The media may be readable by a computer, stored in a memory, and executed by a processor.

[0208] A computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium does not contain signals and is tangible, but does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0209] Additionally, programs according to the embodiments disclosed herein may be provided as part of a computer program product. The computer program product may be traded as a commodity between sellers and buyers.

[0210] A computer program product may include a software program, a computer-readable storage medium having the software program stored thereon. For example, the computer program product may be provided by a manufacturer of an augmented reality device (100) or an electronic market (e.g., Samsung Galaxy Store). TM ) may include a product in the form of a software program (e.g., a downloadable application) that is distributed electronically. For electronic distribution, at least a portion of the software program may be stored in a storage medium or temporarily created. In this case, the storage medium may be a storage medium of a manufacturer's server of the augmented reality device (100), a server of an electronic market, or a relay server that temporarily stores the software program.

[0211] The computer program product may include a storage medium of the server or the storage medium of the augmented reality device (100) in a system comprising an augmented reality device (100) and / or a server. Alternatively, if there is a third device (e.g., a wearable device) that is communicatively connected to the augmented reality device (100), the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself that is transmitted from the augmented reality device (100) to the third device, or from the third device to an electronic device.

[0212] In this case, either the augmented reality device (100) or the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, at least one of the augmented reality device (100) and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0213] For example, the augmented reality device (100) may execute a computer program product stored in the memory (140, see FIG. 3) to control another electronic device (e.g., a wearable device) that is in communication with the augmented reality device (100) to perform a method according to the disclosed embodiments.

[0214] As another example, a third device may execute a computer program product to control an electronic device in communication with the third device to perform a method according to the disclosed embodiment.

[0215] When a third device executes a computer program product, the third device may download the computer program product from the augmented reality device (100) and execute the downloaded computer program product. Alternatively, the third device may execute a computer program product provided in a pre-loaded state to perform the method according to the disclosed embodiments.

[0216] Although the non-limiting exemplary embodiments of the present disclosure have been particularly illustrated and described with reference to the drawings, it will be understood by those skilled in the art that various modifications and variations are possible without departing from the spirit and scope of the present disclosure. For example, suitable results may still be achieved if the described techniques are performed in a different order than described, and / or components such as the described computer system or modules are combined or combined in a different form than described, or are replaced or substituted by other components or equivalents.

Claims

1. A method performed by an augmented reality device, A step of scanning the surrounding real world to detect at least one area containing a plane where no object is detected; A step of determining a type of the virtual keyboard that can be overlaid on the at least one area based on at least one of the shape, size, and input language of the virtual keyboard; and A step of performing rendering so that the virtual keyboard having the above type is overlaid and displayed on at least one of the above areas; A method comprising:

2. In paragraph 1, The step of detecting at least one region comprises: A step of acquiring three-dimensional data about the surrounding real space by scanning the surrounding environment using at least one of an RGB camera, an infrared sensor, a depth camera, or a LiDAR sensor; and A step of performing plane detection to detect at least one area from the three-dimensional data, wherein the at least one area includes a surface having a plane on which the virtual keyboard can be overlaid; A method comprising:

3. In paragraph 1, A step of obtaining profile information of the virtual keyboards by loading profile information of the virtual keyboards from the memory; Including more, A method wherein the virtual keyboard profile information includes at least one of the shape, size, and input language of the virtual keyboards.

4. In paragraph 1, The step of determining the type of the above virtual keyboard is: A step of configuring an area-virtual keyboard combination by matching at least one area with all kinds of virtual keyboards that can be provided by the augmented reality device; A step of performing an evaluation on the region-virtual keyboard combination based on at least one of the attribute information of the region and at least one of the shape, size, and input language of the virtual keyboards, wherein the attribute information of the region includes the size and shape of the at least one region; and A step of determining a type of virtual keyboard that can be overlaid on at least one area based on an evaluation result regarding the area-virtual keyboard combination; A method comprising:

5. In paragraph 4, The steps to configure the above area-virtual keyboard combination are: A method comprising the step of matching a plurality of virtual keyboards that can be overlaid on each of the at least one area.

6. In paragraph 4, A step of determining an area on which the virtual keyboard is to be overlaid among the plurality of virtual keyboards and the at least one area, based on at least one of an input language, an input field, and a user's usage history information, from a combination of the at least one area and the type of the overlayable virtual keyboard; A method further comprising:

7. In paragraph 1, The step of detecting at least one area comprises the step of detecting a surface having a curvature of a part of the user's body, A method wherein the step of performing the rendering comprises the step of warping the virtual keyboard based on the curvature of the surface.

8. In augmented reality devices, At least one camera; A sensor comprising at least one of an infrared sensor, a depth camera, and a LiDAR sensor; A memory that stores one or more instructions; and At least one processor executing one or more of the above instructions; Including, When said one or more instructions are executed by said at least one processor, said augmented reality device: By scanning the surrounding real world using at least one of the above camera and the at least one sensor, at least one area including a plane where no object is detected is detected, Based on at least one of the shape, size, and input language of the virtual keyboard, determine the type of the virtual keyboard that can be overlaid on the at least one detected area, An augmented reality device configured to perform rendering such that the virtual keyboard having the type is overlaid and displayed on at least one area.

9. In paragraph 8, When said one or more instructions are executed by said at least one processor, said augmented reality device: By scanning the surrounding environment using at least one of the above at least one camera and at least one of the above at least one sensor, three-dimensional data about the surrounding real space is obtained, An augmented reality device that performs plane detection to detect at least one area from the three-dimensional data, wherein the at least one area includes a surface having a plane on which the virtual keyboard can be overlaid.

10. In paragraph 9, An augmented reality device, wherein at least one of the regions comprises a curved surface having a curvature.

11. In paragraph 8, The above one or more commands, when executed by the at least one processor, are configured to cause the augmented reality device to obtain profile information of the virtual keyboards by loading the profile information of the virtual keyboards from a memory, An augmented reality device, wherein the profile information includes at least one of the shape, size, or input language of the virtual keyboard.

12. In paragraph 8, When said one or more instructions are executed by said at least one processor, said augmented reality device: By matching at least one of the above areas with the types of virtual keyboards that can be provided by the augmented reality device, a region-virtual keyboard combination is formed, An evaluation is performed on the region-virtual keyboard combination based on at least one of the attribute information of the region and at least one of the shape, size, and input language of the virtual keyboards, wherein the attribute information of the region includes the size and shape of the at least one region, An augmented reality device that determines a type of virtual keyboard that can be overlaid on at least one area based on an evaluation result regarding the above area-virtual keyboard combination.

13. In paragraph 12, When said one or more instructions are executed by said at least one processor, said augmented reality device: An augmented reality device, wherein, based on at least one of an input language, an input field, and a user's usage history information, a combination of the at least one area and the type of the overlayable virtual keyboard is determined, among the plurality of virtual keyboards, a virtual keyboard, and among the at least one area, an area on which the virtual keyboard is to be overlaid.

14. In paragraph 8, When said one or more instructions are executed by said at least one processor, said augmented reality device: Detecting surfaces that have the curvature of a part of the user's body, An augmented reality device that warps the virtual keyboard based on the curvature of the surface.

15. A computer program product comprising a non-transitory computer-readable recording medium, wherein when the non-transitory computer-readable recording medium is executed by at least one processor of an augmented reality device, the augmented reality device: Scan the surrounding real world to detect at least one area containing a plane where no objects are detected, Based on at least one of the shape, size, and input language of the virtual keyboard, determine the type of the virtual keyboard that can be overlaid on the at least one area, Perform rendering so that the virtual keyboard having the above type is overlaid and displayed on at least one of the above areas A computer program product configured to do something.

Images