Method for generating movement path of virtual object and electronic device therefor

Abstract

An electronic device, in one embodiment of the present disclosure, may comprise: a camera; a display; a memory; and at least one processor electrically connected to the camera, the display, and the memory, wherein the memory may store instructions. The instructions may, when executed by the at least one processor, instruct the electronic device to: generate a virtual object on the basis of a user input; identify an action pattern of the generated virtual object; determine a movement path from a first point to a second point of the virtual object on the basis of the identified action pattern; and display the motion of the virtual object on the display, on the basis of the determined movement path.

Description

Method for generating a movement path of a virtual object and electronic device therefor

[0001] Embodiments disclosed in this document relate to a method for generating a movement path of a virtual object based on behavioral properties and an electronic device therefor.

[0002] Virtual reality (VR) refers to a specific virtual environment or situation created using computer technology, or the technology for it. Augmented reality (AR) refers to a technology that superimposes a three-dimensional virtual image on a real-world image or background to display it as a single image. Mixed reality (MR) is a technology that combines virtual reality (VR) and augmented reality (AR) to further enhance the user's interaction with the virtual world. For example, there are two types of mixed reality (MR): mixing virtual objects into the real world and mixing real objects into the virtual world. Extended reality (XR) is a general term for extended reality, including virtual reality (VR), augmented reality (MR), and mixed reality (MR).

[0003] The above information may be provided as background information to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art in connection with the present disclosure.

[0004] An electronic device according to one embodiment disclosed in the present document includes a camera, at least one display, a memory, and at least one processor electrically connected to the camera, the at least one display, and the memory, wherein the memory can store instructions. When the instructions are individually or in combination executed by the at least one processor, the electronic device can generate a virtual object based on a user input, identify an action pattern of the generated virtual object, determine a movement path of the virtual object from a first point to a second point based on the identified action pattern, and display a motion of the virtual object on the display based on the determined movement path.

[0005] A method using an electronic device according to an embodiment disclosed in this document may include an operation of generating a virtual object based on a user input of a user, an operation of identifying an action pattern of the generated virtual object, an operation of determining a movement path of the virtual object from a first point to a second point based on the identified action pattern, and an operation of displaying a motion of the virtual object based on the determined movement path.

[0006] In a computer-readable storage medium storing instructions according to one embodiment disclosed in the present document, the instructions, when executed by at least one processor, may cause the at least one processor to create a virtual object based on a user input of a user, identify an action pattern of the created virtual object, determine a movement path of the virtual object from a first point to a second point based on the identified action pattern, and display a motion of the virtual object based on the determined movement path.

[0007] FIG. 1 is a block diagram of a system for generating a movement path of a virtual object according to one embodiment of the present disclosure.

[0008] FIG. 2 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure.

[0009] FIG. 3 is a flowchart of a method for an electronic device to determine a movement path of a virtual object and provide motion according to one embodiment of the present disclosure.

[0010] FIG. 4 is a flowchart of a method for an electronic device to generate a 3D virtual object based on user input according to one embodiment of the present disclosure.

[0011] FIG. 5 is a flowchart of a method for an electronic device to assign properties to a behavior type of a virtual object according to one embodiment of the present disclosure.

[0012] FIG. 6 is a flowchart of a method for an electronic device to generate a movement path of a virtual object according to an embodiment of the present disclosure.

[0013] FIGS. 7A and 7B are exemplary diagrams of an embodiment of an electronic device obtaining movable map information according to one embodiment of the present disclosure.

[0014] FIG. 8 is an exemplary diagram of an embodiment in which an electronic device specifies an initial position of a virtual object according to one embodiment of the present disclosure.

[0015] FIGS. 9A and 9B are exemplary diagrams of an embodiment in which an electronic device generates a movement path of a virtual object according to one embodiment of the present disclosure.

[0016] FIG. 10 is a flowchart of a method for an electronic device to display motion of a virtual object according to one embodiment of the present disclosure.

[0017] FIG. 11 is an example diagram of an electronic device obtaining a notification from an external electronic device and providing it to a user, according to one embodiment of the present disclosure.

[0018] Figure 12 is a schematic diagram of an electronic device according to one embodiment.

[0019] FIG. 13 is a schematic diagram of a method for tracking and displaying gaze through a transparent member according to one embodiment.

[0020] FIG. 14 is a block diagram of an electronic device within a network environment according to various embodiments.

[0021] FIG. 15 illustrates an example of a block diagram of a wearable device according to one embodiment.

[0022] FIG. 16A illustrates an example of a perspective view of a wearable device according to one embodiment.

[0023] FIG. 16b illustrates an example of one or more hardware elements disposed within a wearable device, according to one embodiment.

[0024] FIGS. 17A and 17B illustrate an example of an appearance of a wearable device according to one embodiment.

[0025] In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0026] Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that various modifications, equivalents, and / or alternatives of the embodiments of the present invention are included.

[0027] FIG. 1 is a block diagram of a system for generating a movement path of a virtual object according to one embodiment of the present disclosure.

[0028] Referring to FIG. 1, in one embodiment of the present disclosure, a system for generating a movement path of a virtual object may include at least one of an electronic device (10), at least one external electronic device (e.g., a first external electronic device (20a) and / or a second external electronic device (20b)), a server (30), or a network (199). In one example, the electronic device (10) may communicate with the at least one external electronic device and / or the server (30) via the network (199). The electronic device (10) may also communicate with the at least one external electronic device via a direct connection (e.g., a wired connection and / or an end-to-end wireless connection).

[0029] For example, the network (199) may include, but is not limited to, at least one of a 3rd Generation Partnership Project (3GPP) network, a Long Term Evolution (LTE) network, a 5G network, a World Interoperability for Microwave Access (WIMAX) network, the Internet, a Local Area Network (LAN), a Wireless Local Area Network (Wireless LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), a Wi-Fi network, a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, or a Digital Multimedia Broadcasting (DMB) network.

[0030] A server (30) according to one embodiment of the present disclosure may include a server that provides content to allow a user to be immersed in a virtual environment including at least one of augmented reality, virtual reality, mixed reality, or extended reality. For example, the server (30) may include a server (an MEC server) and / or a cloud server for providing MEC (multi-access edge computing or mobile-edge computing) services.

[0031] An electronic device (10) according to one embodiment of the present disclosure may receive image information and / or notifications from at least one external electronic device (e.g., a first external electronic device (20a) and / or a second external electronic device (20b)). In one example, the electronic device (10) may receive image information stored in at least one external electronic device. For example, the electronic device (10) may receive image information regarding an image stored in an external electronic device associated with the electronic device (10) (e.g., a user device of a user of the electronic device (10) and / or a device associated with the same account as the electronic device (10). For example, the notification may include at least one of a phone notification, a message notification, or a text notification obtained through any application.

[0032] For example, at least one external electronic device (e.g., the first external electronic device (20a) and / or the second external electronic device (20b)) may include any electronic device such as a smart phone, a laptop, a desktop, a TV, a smart pad, a tablet PC, a wearable device, a connected car, and / or a portable terminal.

[0033] For example, the electronic device (10) may include at least one of a head mounted display (HMD), a virtual reality headset (VRH), augmented reality glasses (e.g., the electronic device (1201) of FIG. 12), or an augmented reality helmet that provides content for virtual reality, augmented reality, mixed reality, or extended reality. For example, the electronic device (10) may include any video see through (VST) device.

[0034] Hereinafter, a person skilled in the art will understand that the description related to the first external electronic device (20a) can be similarly applied to the second external electronic device (20b). The number of at least one external electronic device (e.g., the first external electronic device (20a) and / or the second external electronic device (20b)) illustrated in FIG. 1 is merely an example, and embodiments of the present disclosure are not limited thereto.

[0035] An electronic device (10) according to one embodiment of the present disclosure can provide augmented reality for a surrounding environmental space. In one example, the electronic device (10) can provide augmented reality within a field of view. For example, the electronic device (10) can provide augmented reality based on the current user's space corresponding to the point of view of the user wearing the electronic device (10).

[0036] An electronic device (10) according to one embodiment of the present disclosure can generate a virtual object. For example, the virtual object can include a 3D virtual object. In one example, the electronic device (10) can generate the virtual object based on a user input. For example, the user input can include an input to an interface on the electronic device (10), an input received through an external electronic device in communication with the electronic device (10), a voice input, and / or a touch input. For example, the virtual object can include any animal, person, and / or any virtual object having mobility. For example, the virtual object can correspond to a default image, an image specified by the user, or an image acquired from an external device.

[0037] An electronic device (10) according to one embodiment of the present disclosure can add (e.g., display) a generated virtual object to augmented reality. In one example, the electronic device (10) can create a virtual object within or outside the user's field of view. In the present disclosure, the user's field of view can be referred to as a predetermined area (e.g., field of view) set around the user's viewpoint or a preset display area (e.g., at least a portion of the display area of ​​the electronic device (10). For example, the electronic device (10) can display at least one image in an area beyond the user's field of view. For example, the field of view of the electronic device (10) can be the same as the user's field of view or wider than the user's visual angle. For example, the electronic device (10) can add a virtual object to augmented reality to provide the user with content such as the virtual object existing in the current location.

[0038] An electronic device (10) according to one embodiment of the present disclosure can set a movement path of a virtual object. In one example, the electronic device (10) can set a movement path in augmented reality and move the virtual object along the movement path. For example, the electronic device (10) can analyze the behavioral properties of the virtual object and set a movement path in a location corresponding to augmented reality. A method for setting a movement path of a virtual object will be described below with reference to FIG. 6 .

[0039] An electronic device (10) according to one embodiment of the present disclosure can identify and display the motion of a virtual object based on a movement path. In one example, the electronic device (10) can identify the motion of a virtual object corresponding to the movement path. For example, if the movement path is simply flat, the electronic device (10) can provide a walking motion and / or a running motion of the virtual object along the movement path. For example, if there is a table in the movement path, the electronic device (10) can provide a walking or running motion to the table, a jumping motion to step on the table and jump up, and a movement motion (e.g., a walking motion or a running motion) to move back to the destination.

[0040] An electronic device (10) according to one embodiment of the present disclosure may receive notification information from a first external electronic device (20a) and generate a person virtual object. In one example, the electronic device (10) may receive a notification from the first external electronic device (20a) and generate a person virtual object corresponding to the external user who sent the notification. The electronic device (10) may generate the person virtual object and provide the notification to the user. A method for the electronic device (10) to provide a notification received by an external electronic device to the user may be further described and referenced in FIG. 11.

[0041] FIG. 2 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure.

[0042] Referring to FIG. 2, the electronic device (10) may include at least one of at least one processor (e.g., processor (220)), memory (230), at least one display (e.g., display (260)), at least one camera (e.g., camera (280)), or communication circuit (290).

[0043] In one embodiment of the present disclosure, at least one processor (e.g., processor (220)) may be connected to memory (230), a display (260), at least one camera (e.g., camera (280)), and a communication circuit (290). For example, at least one processor may be electrically connected to components of the electronic device (10). For example, at least one processor may be connected to components of the electronic device (10) by wire or wirelessly. At least one processor may be composed of a single chip or multiple chips. For example, at least one processor may include at least one processing circuitry such as a central processing unit (CPU), an application processor (AP), a microprocessor unit (MPU), a communication processor (CP), a system on chip (SoC), or an integrated circuit (IC).

[0044] In one embodiment of the present disclosure, at least one processor may perform operations necessary for the operation of the electronic device (10). In one example, the operations of the electronic device (10) may be performed by at least one processor individually or collectively executing instructions. Some of the operations of the electronic device (10) may be performed by a first processor executing instructions, and at least some of the remaining operations may be performed by a processor different from the first processor executing instructions. At least one processor may control components of the electronic device (10). For example, the operations of the electronic device (10) described in the present disclosure may be referred to as being performed by at least one processor. For example, the operations of the electronic device (10) may be performed by at least one processor executing instructions stored in the memory (230). At least one processor may be referred to by processor (1420) of FIG. 14.

[0045] In one embodiment of the present disclosure, the memory (230) may include built-in memory or external memory. For example, the built-in memory may include at least one of a volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM), or synchronous DRAM (SDRAM)), a nonvolatile memory (e.g., programmable read-only memory (PROM), one time PROM (OTPROM), erasable PROM (EPROM), electrically erasable and PROM (EEPROM), mask ROM, flash ROM, flash memory, a hard drive, or a solid state drive (SSD). The external memory may include at least one of a flash drive (e.g., compact flash), secure digital (SD), micro-SD, mini-SD, extreme digital (xD), multi-media card (MMC), or a memory stick. The memory (230) may be referenced by the memory (1430) of FIG. 14.

[0046] In one embodiment of the present disclosure, the memory (230) may store instructions that can be executed by at least one processor (e.g., processor (271)). The memory (230) may store at least one data related to the operation of the electronic device (10) or a command related to the functional operation of components of the electronic device (10). For example, the memory (230) may store at least one application that is preloaded upon manufacturing the electronic device (10) or downloaded as a third party from an online market (e.g., app store). For example, the at least one application may include a voice recognition application that supports the operation of a voice recognition service.

[0047] In one embodiment of the present disclosure, the electronic device (10) may include a microphone (not shown). In one example, the electronic device (10) may obtain voice input from a user using the microphone. For example, the voice input may include information associated with creating a virtual object. For example, the voice input may include a voice command for the virtual object. The microphone (not shown) may be referenced by the input module (1450) of FIG. 14.

[0048] In one embodiment of the present disclosure, the electronic device (10) may include a speaker (not shown). In one example, the electronic device (10) may provide a notification to a user using the speaker. For example, the electronic device (10) may provide a notification received from an external electronic device to the user using the speaker. An embodiment related to providing a notification may be referred to by FIG. 11. For example, the electronic device (10) may provide a sound of a virtual object (e.g., a barking sound when the virtual object is a puppy) to the user using the speaker. The speaker (not shown) may be referred to by the audio output module (1455) of FIG. 14.

[0049] In one embodiment of the present disclosure, the display (260) may be configured to display an image in an XR environment. For example, the processor (220) may use the camera (280) to acquire a real-world image in front of the user (e.g., the direction in which the user's face is facing) and display content by overlaying it on the real-world image. In one example, the processor (220) may overlay content on a virtual space and display it. In one example, the display (260) may include multiple displays. For example, the display (260) may include a left-eye display and a right-eye display. For example, the display (260) may include an internal display and / or an external display. For example, the external display may include a display of the user's appearance as seen from the outside when worn by the user. The display (260) may include at least one of a see-through display, a flexible display, a rollable display, a foldable display, and / or a rigid display.

[0050] In one example, the display (260) may be configured to generate light for displaying a virtual image. For example, the display (260) may include an optical engine of a projector including an image panel, an illumination optical system, a projection optical system, etc. The display (260) may include a light source that outputs light, an image panel that forms a two-dimensional virtual image using the light output from the light source, and a projection optical system that projects the light of the virtual image formed on the image panel. The light source is an optical component that illuminates light and can generate light by controlling the color of RGB. The light source may be composed of, for example, a light emitting diode (LED). The image panel may be composed of a reflective image panel that modulates light illuminated by the light source into light containing a two-dimensional image and reflects it. The reflective image panel may be, for example, a DMD (Digital Micromirror Device) panel, an LCoS (Liquid Crystal on Silicon) panel, or any other known reflective image panel. The display (260) may be placed on the back of the electronic device (10). The display (260) may be referenced by the display module (1460) of FIG. 14.

[0051] In one embodiment of the present disclosure, at least one camera (e.g., camera 280) may capture images of the surroundings and / or the interior space of the electronic device (10). For example, the electronic device (10) may include at least one camera configured to acquire images of the surrounding space of the electronic device (10). In one example, the electronic device (10) may include at least one camera configured to acquire images of the front of the electronic device (10) (e.g., the direction in which the eyes of a wearer of the electronic device (10) are directed). The electronic device (10) may acquire parallax information using at least two or more cameras and generate a depth map of the surrounding space based on the parallax information. For example, the electronic device (10) may include at least one internal camera configured to acquire images of a direction toward the wearer of the electronic device (10) (e.g., the direction toward the interior space). Using the internal camera, the electronic device (10) may track the user's gaze. The at least one camera (e.g., camera 280) may include one or more lenses. For example, at least one camera may include at least one of a standard lens, a wide-angle lens, a telephoto lens, a fisheye lens, a micro lens, or a zoom lens based on a focal length.

[0052] The communication circuit (290) may include at least one circuit for processing a signal. For example, the communication circuit (290) may include at least one converter for frequency modulation of a signal, at least one filter for noise removal of a signal, at least one modem for modulating and / or demodulating a signal, at least one antenna, and / or at least one transceiver. The communication circuit (290) may be controlled based on a signal (e.g., a control signal) from the processor (220), for example. The communication circuit (290) may be referenced by the communication module (1490) of FIG. 14.

[0053] In one embodiment of the present disclosure, the electronic device (10) may provide an indoor surrounding space as an augmented reality when at least one instruction stored in the memory (230) is executed by at least one processor. In one example, the electronic device (10) may obtain image information about the user's surroundings using at least one camera (e.g., the camera (280)). The electronic device (10) may obtain map information about the indoor space around the user using an artificial intelligence model and / or big data learning information. For example, the electronic device (10) may recognize objects around the user and identify the types of the objects. The electronic device (10) may obtain map information about the indoor space by identifying the types of the objects. For example, the electronic device (10) may map the indoor space around the user using simultaneous localization and mapping (SLAM) technology. The electronic device (10) may identify objects around the user through indoor space mapping. The electronic device (10) can obtain map information including object characteristics of the space around the user and provide the user's current indoor space as augmented reality (AR).

[0054] In one embodiment of the present disclosure, the electronic device (10) can generate a virtual object using a generative artificial intelligence model when at least one instruction stored in the memory (230) is executed by at least one processor. In one example, the electronic device (10) may be equipped with on-device artificial intelligence technology. For example, the electronic device (10) can generate a virtual object using an artificial intelligence model based on on-device AI. The electronic device (10) can also generate a virtual object using an artificial intelligence model based on an external device.

[0055] In one example, the electronic device (10) may generate a virtual object corresponding to a user input in augmented reality (AR) and display it to the user. For example, the electronic device (10) may identify the external properties of the virtual object to be generated based on the user input. For example, the external properties may include any parameters that constitute the external appearance of the virtual object, such as color, size, type (e.g., breed in the case of a dog). The electronic device (10) may generate a virtual object corresponding to the user input from information about the external properties of the virtual object using a generative artificial intelligence model.

[0056] In one embodiment of the present disclosure, the electronic device (10) can identify an action pattern of a generated virtual object when at least one instruction stored in the memory (230) is executed by at least one processor. For example, the action pattern may include an action type and / or an action attribute that the virtual object can perform. For example, the action type may include at least one of running, walking, sitting, jumping, rolling, barking, sleeping, or lying down. For example, the action attribute may correspond to an attribute of the virtual object for each action type. For example, if the virtual object is a poodle, which is a medium-sized dog, the action attribute for running may be a speed of up to 15 km / h.

[0057] In one embodiment of the present disclosure, when at least one instruction stored in the memory (230) is executed by at least one processor, the electronic device (10) may determine a movement path of a virtual object based on a behavior pattern. In one example, the electronic device (10) may determine a movement path along which a virtual object moves in virtual reality. For example, the movement path may include at least one path extending from an arbitrary starting point to a point where a user is present. The electronic device (10) may determine the movement path based on a behavioral attribute of the virtual object. For example, the electronic device (10) may identify an object that may be included in the movement path of the virtual object in an indoor space provided in augmented reality based on the behavioral attribute of the virtual object. For example, if the jump attribute of the virtual object is that it can jump up to a maximum height of 50 cm, the electronic device (10) may exclude an object with a height exceeding 50 cm from the movement path. A specific method by which the electronic device (10) determines the movement path can be described later in FIG. 6.

[0058] In one embodiment of the present disclosure, the electronic device (10) may display the motion of a virtual object through the display (260) based on a determined movement path when at least one instruction stored in the memory (230) is executed by at least one processor. In one example, the electronic device (10) may display the motion of the virtual object moving along the movement path according to a frame rate. For example, if the movement path only moves on the same plane, the electronic device (10) may display a walking motion and / or a running motion of the virtual object. For example, if the movement path includes an object that the virtual object can pass through, the electronic device (10) may provide a motion of jumping and moving in front of the object.

[0059] FIG. 3 is a flowchart of a method for an electronic device to determine a movement path of a virtual object and provide motion according to one embodiment of the present disclosure.

[0060] The operations described below with reference to FIG. 3 may be referred to as the operations of the electronic device (10) of FIG. 2. The order of the operations described below with reference to FIG. 3 is merely an example, and embodiments of the present disclosure are not limited thereto. For example, at least some of the operations may be executed differently from the order of FIG. 3, or may be executed substantially simultaneously with other operations of FIG. 3. At least some of the operations described below with reference to FIG. 3 may be omitted.

[0061] Referring to FIGS. 1 and 3, in operation 310, an electronic device (10) according to an embodiment of the present disclosure may obtain information about a virtual object to be created from a user. In one example, the electronic device (10) may obtain information about the virtual object based on a user's touch input and / or voice input. For example, the electronic device (10) may obtain a voice input or touch input having a meaning such as "create a brown poodle puppy" as a prompt for a 3D virtual object creation model. For example, the electronic device (10) may obtain a user's touch input for a puppy photo stored in a gallery. For example, the electronic device (10) may obtain a selection input for any one of a virtual object library that is already stored.

[0062] In operation 320, the electronic device (10) may generate a virtual object. In one example, the electronic device (10) may generate the virtual object using a 3D generation artificial intelligence model. For example, the electronic device (10) may generate the virtual object by inputting information about the virtual object as a prompt.

[0063] The method of generating a virtual object by obtaining the user input described above in operations 310 and 320 can be further described in FIG. 4.

[0064] In operation 330, the electronic device (10) according to one embodiment of the present disclosure may assign an action attribute to the generated virtual object. In one example, the electronic device (10) may register and add an action type that the virtual object can perform to the information of the virtual object. For example, if the virtual object is a puppy, the action type may include any action that a puppy can perform, such as walking, running, jumping, rolling, lying down, sitting, barking, and / or sleeping. The electronic device (10) may assign an action attribute to each action type based on the external attributes of the virtual object. For example, if the generated virtual object is a poodle, which is a medium-sized dog, the electronic device (10) may assign an action attribute, such as a maximum jump height of 50 cm and a maximum running speed of 15 km / h, based on the external attributes (e.g., medium-sized dog, poodle). The method of assigning behavioral properties to a virtual object can be further described in FIG. 5.

[0065] In operation 340, an electronic device (10) according to an embodiment of the present disclosure may display objects in a space surrounding the user. In one example, the electronic device (10) may obtain indoor space map information surrounding the user using an artificial intelligence model and / or big data learning information. For example, the electronic device (10) may recognize objects surrounding the user and identify the types of the objects. The electronic device (10) may obtain map information of an indoor space by identifying the types of objects surrounding the user. For example, the electronic device (10) may map an indoor space surrounding the user using simultaneous localization and mapping (SLAM) technology. The electronic device (10) may obtain indoor space map information by processing images acquired using a camera (280) using an artificial intelligence model. The electronic device (10) may identify objects surrounding the user through indoor space mapping. For example, when a user is in a living room, the electronic device (10) can identify any object existing in the living room, such as a TV, a table, and / or a sofa. After identifying objects surrounding the indoor space where the user is, the electronic device (10) can provide the objects surrounding the indoor space as augmented reality (AR). For example, the electronic device (10) can display at least one object surrounding the user to the user. The electronic device (10) can store object characteristics of the identified object. For example, the object characteristics can include at least one of the height, area, curvature, or inclination of the object.

[0066] In operation 350, an electronic device (10) according to an embodiment of the present disclosure may determine a movement path of a virtual object. In one example, the electronic device (10) may obtain movable map information based on the surrounding objects displayed in operation 340. For example, the movable map may correspond to map information including only objects that may be included in the movement path of the virtual object. For example, the electronic device (10) may identify objects that may be included in the movement path of the virtual object by using object characteristics of the surrounding objects. For example, the electronic device (10) may obtain movable map information that includes objects that may be included in the movement path, excluding objects that may not be included in the movement path of the virtual object. The electronic device (10) may determine a movement path of the virtual object based on the movable map. An embodiment of determining a movement path of a virtual object may be further described later in FIG. 9A or 9B.

[0067] In operation 360, an electronic device (10) according to an embodiment of the present disclosure may generate a motion of a virtual object. In one example, the electronic device (10) may generate a motion of a virtual object corresponding to a determined movement path. For example, the electronic device (10) may generate the next motion of the virtual object along the movement path from the time the virtual object is generated. For example, the electronic device (10) may generate a movement motion (e.g., a walking motion, a running motion) of the virtual object along the movement path. For example, if the movement path includes a plane object with different heights, the electronic device (10) may generate a virtual object that takes a jumping motion in front of the object.

[0068] In operation 370, an electronic device (10) according to an embodiment of the present disclosure may display the motion of a generated virtual object. In one example, the electronic device (10) may display the motion of a virtual object corresponding to a movement path along the movement path and provide it to a user. For example, the electronic device (10) may add the movement motion of a virtual object to an augmented reality (AR) of a surrounding indoor space, thereby providing an image of the virtual object moving in AR. A user wearing the electronic device (10) may see the motion of a virtual object, such as a puppy, moving around his or her surroundings in AR.

[0069] FIG. 4 is a flowchart of a method for an electronic device to generate a 3D virtual object based on user input according to one embodiment of the present disclosure.

[0070] The operations described below with reference to FIG. 4 may be referred to as operations of the electronic device (10) of FIG. 2. The order of the operations described below with reference to FIG. 4 is merely an example, and embodiments of the present disclosure are not limited thereto. For example, at least some of the operations may be executed differently from the order of FIG. 4, or may be executed substantially simultaneously with other operations of FIG. 4. At least some of the operations described below with reference to FIG. 4 may be omitted.

[0071] Referring to FIG. 4, an electronic device (e.g., the electronic device (10) of FIG. 1) according to an embodiment of the present disclosure may obtain a user input. For example, the user input may include a voice input and / or a touch input from a user of the electronic device (10). In one example, the user input may include information associated with the creation of a virtual object. For example, the electronic device (10) may obtain a voice input and / or a touch input from a user that includes natural language information, such as “create a puppy.” In this case, the electronic device (10) may use the user input as a prompt input for a generative artificial intelligence model. In another example, the electronic device (10) may obtain a voice input and / or a touch input from a user that includes natural language information, such as “bring me the puppy I created before.” In this case, the electronic device (10) may retrieve a 3D virtual object corresponding to the user input from a 3D virtual object library stored in a memory (e.g., the memory (230) of FIG. 2).

[0072] In one example, the electronic device (10) can obtain a user input input from an external electronic device (e.g., the first external electronic device (20a) and the second external electronic device (20b) of FIG. 1). For example, if the external electronic device is a smartphone connected to the electronic device (10), the user can select a puppy image stored in the gallery. Alternatively, the user can select a puppy image stored in the gallery by performing a voice input and / or a touch input including natural language information such as “Generate a puppy image stored in the gallery.” The external electronic device can transmit information about the selected puppy image to the electronic device (10).

[0073] In operation 420, an electronic device (10) according to an embodiment of the present disclosure may identify external attribute information of a virtual object to be created. For example, if the virtual object is a puppy, the external attribute information may include external characteristics of any puppy, such as breed (e.g., poodle, husky, Pomeranian, or Shamoyed), size (e.g., large, medium, or small), and color.

[0074] In one example, the electronic device (10) can identify external attribute information of a virtual object based on a user input. For example, if the user input is for “create a puppy” as described in operation 410, the electronic device (10) can obtain specific external attribute information of a 3D virtual puppy object to be created using a generative artificial intelligence model. For another example, if the user input is for selecting an image from an external electronic device as described in operation 410, the electronic device (10) can analyze the image information using an artificial intelligence model to identify external attribute information of a virtual object to be created. For example, the electronic device (10) can identify external attribute information from image information of a puppy selected by the user input using an image analysis artificial intelligence model. For another example, if the user input is for “bring me the puppy I created previously” as described in operation 410, the electronic device (10) can identify external attribute information from information of a virtual object retrieved from a 3D virtual object library. For example, a 3D virtual object library may store information about the external properties of each virtual object.

[0075] In one example, the electronic device (10) may be equipped with on-device artificial intelligence technology. For example, the electronic device (10) may acquire external attribute information using an artificial intelligence model based on on-device AI. The electronic device (10) may also acquire external attribute information using an artificial intelligence model based on an external device.

[0076] In operation 430, an electronic device (10) according to an embodiment of the present disclosure may generate a 3D virtual object. In one example, the electronic device (10) may generate a virtual object using a 3D generative artificial intelligence model with external attribute information as an input prompt. For example, the electronic device (10) may generate a 3D virtual object corresponding to the external attribute using the external attribute information of the identified virtual object. For example, if the identified external attribute information includes information that the breed is a poodle, the size is a medium-sized dog, and the color is brown, the electronic device (10) may generate a brown poodle the size of a medium-sized dog using the 3D generative artificial intelligence model. The electronic device (10) may generate a virtual object outside the field of view of a user wearing the electronic device (10) (e.g., outside the boundary of the field of view). For example, the field of view may correspond to the field of view range of an augmented reality being viewed by a user wearing the electronic device (10). For example, the outside of the field of view may include the boundary of the field of view. When the electronic device (10) generates a virtual object outside the user's field of view and provides the movement motion of the virtual object according to the subsequent movement path, it can provide a scene in which the virtual object naturally enters the user's field of view. The electronic device (10) can obtain external attribute information using an artificial intelligence model based on on-device AI and / or an artificial intelligence model based on an external device.

[0077] FIG. 5 is a flowchart of a method for an electronic device to assign properties to a behavior type of a virtual object according to one embodiment of the present disclosure.

[0078] The operations described below with reference to FIG. 5 may be referred to as operations of the electronic device (10) of FIG. 2. The order of the operations described below with reference to FIG. 5 is merely an example, and embodiments of the present disclosure are not limited thereto. For example, at least some of the operations may be executed differently from the order of FIG. 5, or may be executed substantially simultaneously with other operations of FIG. 5. At least some of the operations described below with reference to FIG. 5 may be omitted.

[0079] Referring to FIGS. 4 and 5, in operation 510, an electronic device according to an embodiment of the present disclosure (e.g., the electronic device (10) of FIG. 1) may acquire a 3D virtual object and external attribute information of the virtual object. In one example, the electronic device (10) may identify the external attribute information of the virtual object based on a user input. In addition, the electronic device (10) may generate a 3D virtual object based on the user input. Operation 510 may be referenced by the contents described above with reference to FIG. 4.

[0080] In operation 520, the electronic device (10) may register a behavior type of the virtual object. For example, the behavior type may correspond to any motion that the virtual object can perform. In one example, the electronic device (10) may identify at least one behavior type of the generated virtual object using an artificial intelligence model. For example, if the virtual object is a puppy, the behavior type may include multiple motions that any puppy can perform, such as running, walking, sitting, jumping, rolling, barking, sleeping, and lying down. For example, if the virtual object is an RC car (radio control car), the behavior type may include any motion that an RC car can perform, such as driving and stopping. The electronic device (10) may register the behavior type by adding the behavior type to the object information of the virtual object. For example, the object information may include at least one of the external attribute information of the generated virtual object, the behavior type, or the behavior attribute information described below. The electronic device (10) can identify a behavior type by using an artificial intelligence model based on on-device AI and / or an artificial intelligence model based on an external device.

[0081] In operation 530, the electronic device (10) according to one embodiment of the present disclosure may assign a behavior attribute to each behavior type of a virtual object. In one example, the electronic device (10) may generate a behavior attribute of a virtual object for each behavior type using an artificial intelligence model. The behavior attribute may correspond to an attribute of the virtual object for each behavior type. For example, if the virtual object is a medium-sized poodle puppy, the behavior attribute for running may be a running speed of up to 15 km / h, and the behavior attribute for jumping may be a jumping height of up to 50 cm. For example, if the virtual object is a large-sized retriever puppy, the behavior attribute for running may be a running speed of up to 25 km / h, and the behavior attribute for jumping may be a jumping height of up to 100 cm. For example, if the virtual object is an RC car, the behavior attribute for driving may be a driving speed of up to 10 km / h. The electronic device (10) can add the attributes of the identified behavior types using an artificial intelligence model to object information by mapping them to each behavior type. For example, the electronic device (10) can add behavior attributes to object information and assign behavior attributes to a generated virtual object. The electronic device (10) can identify behavior attributes using an artificial intelligence model based on on-device AI and / or an artificial intelligence model based on an external device.

[0082] FIG. 6 is a flowchart of a method for an electronic device to generate a movement path of a virtual object according to an embodiment of the present disclosure.

[0083] The operations described below with reference to FIG. 6 may be referred to as operations of the electronic device (10) of FIG. 2. The order of the operations described below with reference to FIG. 6 is merely an example, and embodiments of the present disclosure are not limited thereto. For example, at least some of the operations may be executed differently from the order of FIG. 6, or may be executed substantially simultaneously with other operations of FIG. 6. At least some of the operations described below with reference to FIG. 6 may be omitted.

[0084] Referring to FIGS. 4 to 6, in operation 610, an electronic device (10) according to an embodiment of the present disclosure may obtain a surrounding indoor space map including height information of objects in an environment surrounding a user. In one example, when a user wears the electronic device (10) in an indoor space, the electronic device (10) may obtain image information of the user's surroundings using a camera (e.g., the camera (280) of FIG. 2). The electronic device (10) may perform SLAM (simultaneous localization and mapping) using an artificial intelligence model. For example, the electronic device (10) may perform SLAM to obtain spatial map information of the user's surroundings from image information of objects surrounding the user. The electronic device (10) may identify at least one object surrounding the user using the spatial map information. The spatial map information may include height information of the object. For example, the electronic device (10) may recognize a height plane of the object using the spatial map information.

[0085] In operation 620, an electronic device (10) according to an embodiment of the present disclosure may verify a height-related behavioral attribute of the generated virtual object. In one example, the electronic device (10) may obtain information about the height-related behavioral attribute based on the assigned behavioral attribute of the virtual object. For example, the information about the height-related behavioral attribute may include a jump attribute of the virtual object. For example, the jump attribute may correspond to an attribute about the maximum height to which the virtual object can jump.

[0086] In operation 630, the electronic device (10) according to one embodiment of the present disclosure may obtain a movable map of a virtual object. For example, the movable map may correspond to map information including objects that may be included in the movement path of the virtual object. In one example, the electronic device (10) may identify objects that may be included in the movement path of the virtual object based on height-related behavioral attributes of the virtual object and spatial map information. For example, the electronic device (10) may compare height information of objects included in the spatial map information with height-related behavioral attributes of the virtual object to identify objects that the virtual object may jump over. The electronic device (10) may identify at least one object that may be included in the movement path of the virtual object and obtain a movable map that includes only the at least one identified object. In addition, the electronic device (10) may obtain the movable map by excluding objects that may not be included in the movement path from the spatial map information obtained in operation 610.

[0087] In operation 640, an electronic device (10) according to an embodiment of the present disclosure may designate an initial position of a virtual object. In one example, the electronic device (10) may determine and designate a starting point of a movement path of the virtual object. For example, the electronic device (10) may designate the starting point of the virtual object as a boundary outside the user's field of view (e.g., an end point, boundary, or corner of the field of view). For example, the height of the virtual object's viewpoint may be the ground where height information corresponds to 0. An embodiment of a virtual object generated based on the initial position designation may be further described later with reference to FIG. 9.

[0088] In operation 650, an electronic device (10) according to an embodiment of the present disclosure may determine a movement path of a virtual object. In one example, if a user's voice command is received, a movement path may be generated for an action corresponding to the voice command. For example, the voice command may include any command that allows the user to interact with the virtual object. For example, the voice command may include commands that can affect the motion and / or movement path of the virtual object, such as "come here" or "sit." In one example, the electronic device (10) may generate a movement path of the virtual object based on the user's voice command and movable map information. For example, the electronic device (10) may determine a recommended action of the virtual object corresponding to the user's voice command using voice recognition and natural language processing technologies. The electronic device (10) may generate a movement path of the virtual object based on the recommended action determined using an artificial intelligence model and the movable map. For example, the movement path may correspond to a path along which the virtual object moves from an initial location designated in operation 640 to an arbitrary destination.

[0089] In one embodiment, the electronic device (10) can generate a movement path for an arbitrary motion of a virtual object in the absence of a user's voice command. In one example, the electronic device (10) can generate a movement path of a virtual object from a movement map and behavior patterns of the virtual object using an artificial intelligence model. For example, the behavior patterns may include the behavior types and behavioral attributes of the virtual object, as described above with reference to FIG. 2 . For example, the electronic device (10) can generate a movement path of a virtual object using height information of surrounding objects included in the movement map, and the behavior types and behavioral attributes of the virtual object. For example, the electronic device (10) can provide the user with an appearance of a virtual object performing realistic actions by including at least one object having a plane of different heights in the movement path. In the absence of a user's voice command, the electronic device (10) can generate an arbitrary movement path to provide motion of the virtual object in a manner similar to reality.

[0090] In one example, the electronic device (10) can generate a movement path using an artificial intelligence model based on on-device AI and / or an artificial intelligence model based on an external device.

[0091] FIGS. 7A and 7B are exemplary diagrams of an embodiment of an electronic device obtaining movable map information according to one embodiment of the present disclosure.

[0092] Referring to FIGS. 7A and 7B , an electronic device (e.g., the electronic device (10) of FIG. 1 ) according to an embodiment of the present disclosure may obtain movable map information of virtual objects (711, 721). In one example, the electronic device (10) may obtain indoor space map information around a user (719) to obtain movable map information of virtual objects (711, 721). The electronic device (10) may provide a space around the user (719) wearing the electronic device (10) as augmented reality (AR). The electronic device (10) may provide a surrounding space within a field of view (713) of the user (719) as AR. A method for the electronic device (10) to obtain movable map information may be referred to by operation 630 of FIG. 6 .

[0093] Referring to FIG. 7A, an electronic device (10) according to an embodiment of the present disclosure may identify an object that may be included in a movement path of a virtual object (711) based on a behavioral attribute of a puppy, which is a virtual object (711). In one example, the electronic device (10) may identify an object that the virtual object (711) may jump onto by using jump attribute information of the virtual object (711) and height information of the object. For example, the electronic device (10) may identify a desk (717) as an object that may be included in a movement path of the virtual object (711) because the height information of the desk (717) is lower than the height information based on the behavioral attribute of the virtual object (711). Conversely, the electronic device (10) may identify a television (715) as an obstacle that is not included in a movement path of the virtual object (711) because the height information of the television (715) is higher than the height information based on the behavioral attribute of the virtual object (711). The electronic device (10) can obtain a movable map by identifying only objects that can be included in the movement path.

[0094] Referring to FIG. 7B, an electronic device (10) according to an embodiment of the present disclosure may identify an object that may be included in a movement path of a virtual object (721), which is a radio control car (RC car), based on a behavioral attribute of the virtual object (721). In one example, the electronic device (10) may identify an object that the virtual object (721) may pass through by using height attribute information of the virtual object (721) and height information of the object. For example, since the RC car, which is a virtual object (721), does not have a height-related behavioral attribute, the electronic device (10) may include only objects whose height information is 0 in the movement path. For example, the electronic device (10) may identify a television (715) as an obstacle that is not included in the movement path of the virtual object (721). For example, the electronic device (10) may identify a desk (727) as an obstacle that is not included in the movement path of the virtual object (721). The electronic device (10) can obtain a movable map by identifying only objects that can be included in the movement path.

[0095] FIG. 8 is an exemplary diagram of an embodiment in which an electronic device specifies an initial position of a virtual object according to one embodiment of the present disclosure.

[0096] Referring to FIGS. 6 to 8, an electronic device (10) according to an embodiment of the present disclosure may provide an indoor space surrounding a user (719) as an augmented reality (AR) within a field of view range (713), as described above in FIG. 7. The electronic device (10) may designate an initial position of a virtual object (811), as described above in operation 640 of FIG. 6. In one example, the electronic device (10) may designate the initial position of the virtual object (811) to an end point of the user's field of view. In addition, the electronic device (10) may designate the initial position on the ground where the height information of the initial position of the virtual object (811) corresponds to 0. As illustrated in FIG. 8, the electronic device (10) may designate the initial position of the virtual object (811) so that it is in contact with a horizontal dotted line and a vertical dotted line. For example, the horizontal dotted line may be a virtual dotted line corresponding to the ground where the height information is 0 in the augmented reality (AR). For example, the vertical axis dotted line may be a virtual dotted line corresponding to the end boundary of the user's field of view range (713).

[0097] FIGS. 9A and 9B are exemplary diagrams of an embodiment in which an electronic device generates a movement path of a virtual object according to one embodiment of the present disclosure.

[0098] Referring to FIGS. 6 to 9A, an electronic device (10) according to an embodiment of the present disclosure can provide an indoor space surrounding a user (719) as an augmented reality (AR) within a field of view range (713), as described above in FIG. 7. The electronic device (10) can generate a movement path of a virtual object (811) based on a user's command, as described above in operation 650 of FIG. 6. In one example, the electronic device (10) can obtain a voice command (910) from the user (719). The electronic device (10) can generate a movement path of the virtual object (811) from the user's (719) voice command using an artificial intelligence model. For example, the electronic device (10) can obtain a voice command (910) such as “come here” from the user (719) and generate a first movement path (905). For example, the first movement path (905) may include a movement path moving toward the user (719) based on the location information of the user (719). The first movement path (905) may include a desk (717), which is an object identified from the movable map described above in FIG. 6.

[0099] Referring to FIG. 9b, an electronic device (10) according to an embodiment of the present disclosure may generate an arbitrary movement path of a virtual object (811) in the absence of a user command, as described above in operation 660 of FIG. 6 . In one example, the electronic device (10) may generate an arbitrary movement path from the behavioral pattern and movable map information of the virtual object (811) using an artificial intelligence model. For example, the electronic device (10) may generate a second movement path (915) based on the behavioral pattern of a puppy, which is the virtual object (811). For example, the second movement path (915) may include a desk (717), which is an object identified from the movable map described above in FIG. 6 .

[0100] FIG. 10 is a flowchart of a method for an electronic device to display motion of a virtual object according to one embodiment of the present disclosure.

[0101] Referring to FIG. 10, an electronic device (e.g., the electronic device (10) of FIG. 1) according to an embodiment of the present disclosure may obtain location information (1005) of a next movement path of a virtual object, an action type (1010) corresponding to the next movement path, and information (1015) of a current state of the virtual object. In one example, the electronic device (10) may generate a movement path of the virtual object as described above with reference to FIG. 6. The electronic device (10) may identify location information (1005) corresponding to the next movement path of the current virtual object based on the generated movement path. For example, the location information may include height information in augmented reality (AR). The electronic device (10) may identify a behavior type (1010) corresponding to the next movement path. For example, when the location information of the next movement path is greater than 0, the electronic device (10) may identify that the action type (1010) corresponding to the next movement path of the virtual object is a jumping action. The electronic device (10) can identify the current state (1015) of the virtual object. For example, the electronic device (10) can identify the walking or stationary state of the virtual object when the virtual object is currently walking or stationary.

[0102] In one example, the electronic device (10) may display a virtual object taking the next motion using a motion-generating artificial intelligence model based on the location information (1005) of the next movement path of the virtual object, the action type (1010) corresponding to the next movement path, and the information (1015) of the current state of the virtual object. For example, if the location information (1005) of the next movement path has height information greater than 0, the action type (1010) is jumping, and the current state (1015) of the virtual object is walking, the electronic device (10) may display a virtual object taking a jumping motion while stretching out its legs while walking. For example, the electronic device (10) may display the motion of the virtual object corresponding to the next movement path to provide the user with natural movements of the virtual object in augmented reality. The electronic device (10) may generate the motion of the virtual object using an artificial intelligence model based on on-device AI and / or an artificial intelligence model based on an external device.

[0103] FIG. 11 is an example diagram of an electronic device obtaining a notification from an external electronic device and providing it to a user, according to one embodiment of the present disclosure.

[0104] Referring to FIG. 11, an electronic device (10) according to an embodiment of the present disclosure may generate a person object (1111) that provides a notification to a user (1119). In one example, the electronic device (10) may obtain movable map information of a virtual object (1111). In one example, the electronic device (10) may obtain indoor space map information around the user (1119) to obtain movable map information of the virtual object (1111). The electronic device (10) may provide the space around the user (1119) wearing the electronic device (10) as augmented reality (AR). The electronic device (10) may provide the surrounding space within the user's field of view (1113) as AR to the user (1119). A method for the electronic device (10) to obtain movable map information may be referred to by operation 630 of FIG. 6.

[0105] In one example, the electronic device (10) may receive notification information from an external electronic device (e.g., the first external electronic device (20a) and the second external electronic device (20b) of FIG. 1). The electronic device (10) may create a person virtual object (1111) corresponding to the external user who sent the notification. For example, the person virtual object may correspond to an image of the external user stored in the external electronic device. The electronic device (10) may create the person virtual object (1111) and provide the notification to the user. For example, the notification may include at least one of a phone call notification, a message notification, or a text notification obtained through any application.

[0106] In one example, when an external electronic device receives a message from an external user saying “Let’s meet for lunch tomorrow,” the electronic device (10) may receive information about the notification from the external electronic device. The electronic device (10) may provide information (1110) indicating the notification received from the external electronic device to the user (1119) using a person virtual object (1111). For example, the electronic device (10) may generate a movement path of the person virtual object (1111) so that it approaches the side of the user (1119) and provides information (1110) indicating the notification. The electronic device (10) may display the motion of the person virtual object (1111) so that the person virtual object moves along the movement path and provides information (1110) indicating the notification to the user (1119).

[0107] Figure 12 is a schematic diagram of an electronic device according to one embodiment.

[0108] Referring to FIG. 12, in the example of FIG. 12, the electronic device (1201) may be referred to as a head mounted display (HMD) device, a wearable device, smart glasses, or eyewear. The form of the electronic device (1201) illustrated in FIG. 12 is exemplary, and the embodiments of the present document are not limited thereto. For example, the electronic device (1201) may be any electronic device configured to provide augmented reality (AR) or virtual reality (VR).

[0109] According to one embodiment, the electronic device (1201) may include at least some of the components of the electronic device (1401) of FIG. 14. For example, the electronic device (1201) may include at least one of a display (e.g., a display module (1460) of FIG. 14), a camera (e.g., a camera module (1480) of FIG. 14), at least one sensor (e.g., a sensor module (1476) of FIG. 14), a processor (e.g., a processor (1420) of FIG. 14), a battery (e.g., a battery (1489) of FIG. 14), a memory (e.g., 1430) of FIG. 14), or a communication circuit (e.g., a communication module (1490) of FIG. 14). At least some of the components of the electronic device (1201) may be located inside a housing of the electronic device (1201) or may be exposed to the outside of the housing.

[0110] The electronic device (1201) may include a display. For example, the electronic device (1201) may include a first display (1261-1) and / or a second display (1261-2). The first display (1261-1) and / or the second display (1261-2) may include at least one of a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon device (LCoS device), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED). For example, the display of the electronic device (1201) may include at least one light source for emitting light. When the first display (1261-1) and / or the second display (1261-2) includes one of a liquid crystal display, a digital mirror device, or a silicon liquid crystal display, the electronic device (1201) may include at least one light source that irradiates light to the screen output area (1260-1 and / or 1260-2) of the display. For another example, when the display of the electronic device (1201) can generate light on its own, the display may not include a separate light source other than the light source included in the display. When the first display (1261-1) and / or the second display (1261-2) includes at least one of an organic light emitting diode or a micro LED, the electronic device (1201) may provide images to the user even without including a separate light source. When the display is implemented with an organic light emitting diode or a micro LED, the weight of the electronic device (1201) may be reduced by omitting the separate light source.

[0111] According to one embodiment, the electronic device (1201) may include a first transparent member (1296-1) and / or a second transparent member (1296-2). For example, when a user wears the electronic device (1201), the user can see through the first transparent member (1296-1) and / or the second transparent member (1296-2). The first transparent member (1296-1) and / or the second transparent member (1296-2) may be formed of at least one of a glass plate, a plastic plate, or a polymer, and may be transparent or translucent. For example, when worn, the first transparent member (1296-1) may be positioned to face the user's right eye, and the second transparent member (1296-2) may be positioned to face the user's left eye.

[0112] In one embodiment, at least a portion of the first transparent member (1296-1) and / or the second transparent member (1296-2) may be an optical waveguide. For example, the optical waveguide may transmit an image generated by a display (e.g., the first display (1261-1) and / or the second display (1261-2)) to a user's eyes. The optical waveguide may be formed of glass, plastic, or a polymer. For example, the optical waveguide may include a nano-pattern formed inside or on one surface (e.g., a grating structure having a polygonal or curved shape). For example, light incident on one end of the optical waveguide may be propagated inside the optical waveguide by the nano-pattern and provided to the user's eyes. For example, an optical waveguide composed of a free-form prism may be configured to provide the incident light to the user via a reflective mirror.

[0113] According to one embodiment, the optical waveguide may include at least one diffractive element (e.g., a Diffractive Optical Element (DOE), a Holographic Optical Element (HOE)) or at least one reflective element (e.g., a reflective mirror). The optical waveguide may guide display light emitted from a light source to a user's eye by using the at least one diffractive element or reflective element included in the optical waveguide. For example, the diffractive element may include an input optical element (e.g., 1262-1 and / or 1262-2) and / or an output optical element (not shown). The first input optical element (1262-1) and / or the second input optical element (1262-2) may be referred to as an input grating area, and the output optical element (not shown) may be referred to as an output grating area. The input grating region can diffract or reflect light to transmit light output from a light source (e.g., Micro LED) to a transparent member (e.g., first transparent member (1296-1) and / or second transparent member (1296-2)) of the display unit. The output grating region can diffract or reflect light transmitted to a transparent member (e.g., first transparent member (1296-1) and / or second transparent member (1296-2)) of the optical waveguide toward the user's eye. For example, the reflective element can include a total reflection optical element or a total reflection waveguide for total internal reflection (TIR). Total reflection can be referred to as one way of guiding light, and can mean creating an angle of incidence such that light (e.g., an image) input through the input grating region is 100% reflected from one side (e.g., a specific side) of the optical waveguide, so that it is 100% transmitted to the output grating region. In one embodiment, the optical path of light emitted from the display can be guided into an optical waveguide by an input optical element.Light traveling within the optical waveguide can be directed toward the user's eye via the output optical element. The screen output area (1260-1 and / or 1260-2) can be determined based on the light emitted toward the eye.

[0114] Although FIG. 12 illustrates an electronic device (1201) providing an image to a user using an optical waveguide, the embodiments of the present disclosure are not limited thereto. In one embodiment, the display of the electronic device (1201) may be a transparent or translucent display. In this case, the display may be positioned in a location facing the user's eyes (e.g., the first screen output area (1260-1) and / or the second screen output area (1260-2)).

[0115] According to one embodiment, the electronic device (1201) may include at least one camera. For example, the electronic device (1201) may include a first camera (1280-1), a second camera (1280-2), and / or a third camera (1280-3). For example, the first camera (1280-1) and the second camera (1280-2) may be used for external image recognition. The first camera (1280-1) and the second camera (1280-2) may be configured to acquire an image corresponding to a direction corresponding to the user's gaze (e.g., +x direction). The electronic device (1201) may perform head tracking (e.g., 3-degrees of freedom (DoF) tracking), hand image detection, hand image tracking, and / or space recognition using the first camera (1280-1) and the second camera (1280-2). For example, the first camera (1280-1) and the second camera (1280-2) may be GS (global shutter) cameras having the same specifications and performance (e.g., angle of view, shutter speed, resolution, and / or color bit count, etc.). The electronic device (1201) may support SLAM (simultaneous localization and mapping) technology by performing spatial recognition (e.g., 6-degree-of-freedom spatial recognition) and / or acquiring depth information using stereo cameras positioned on the left and right. In addition, the electronic device (1201) may recognize the user's gestures using the stereo cameras positioned on the left and right. The electronic device (1201) may detect faster hand movements and fine movements by using the GS camera, which has relatively less distortion compared to the RS (rolling shutter) camera. For example, the third camera (1280-3) may be used for external image recognition. The third camera (1280-3) may be set to acquire an image corresponding to a direction corresponding to the user's gaze (e.g., +x direction).In one example, the third camera (1280-3) may be a camera having a relatively high resolution compared to the first camera (1280-1) and the second camera (1280-2). The third camera (1280-3) may be referred to as a high resolution (HR) camera or a photo video (PV) camera. The third camera (1280-3) may support functions for acquiring high-quality images, such as auto focus (AF) and / or optical image stabilization (OIS). The third camera (1280-3) may be a GS camera or an RS camera.

[0116] According to one embodiment, the electronic device (1201) may include at least one eye-tracking sensor. For example, the electronic device (1201) may include a first eye-tracking sensor (1276-1) and a second eye-tracking sensor (1276-2). The first eye-tracking sensor (1276-1) and the second eye-tracking sensor (1276-2) may be, for example, cameras configured to acquire images in a direction corresponding to the user's eyes. The first eye-tracking sensor (1276-1) and the second eye-tracking sensor (1276-2) may be configured to acquire images of the user's right eye and the user's left eye, respectively. The electronic device (1201) may be configured to detect the user's pupil using the first eye-tracking sensor (1276-1) and the second eye-tracking sensor (1276-2). The electronic device (1201) can acquire the user's gaze from the user's eye image and provide an image based on the acquired gaze. For example, the electronic device (1201) can display the image so that the image is positioned in the direction of the user's gaze. For example, the first gaze tracking sensor (1276-1) and the second gaze tracking sensor (1276-2) can be GS (global shutter) cameras having the same specifications and performance (e.g., angle of view, shutter speed, resolution, and / or color bit count, etc.).

[0117] According to one embodiment, the electronic device (1201) may include at least one illumination unit. The illumination unit may include, for example, at least one LED. In FIG. 12, the electronic device (1201) may include a first illumination unit (1281-1) and a second illumination unit (1281-2). The electronic device (1201) may provide auxiliary illumination for the first camera (1280-1), the second camera (1280-2), and / or the third camera (1280-3) using, for example, the first illumination unit (1281-1) and the second illumination unit (1281-2). In one example, the electronic device (1201) may provide illumination for acquiring a pupil image using the illumination unit (not shown). For example, the electronic device (1201) may provide illumination for the gaze tracking sensor using an infrared wavelength LED. In this case, the gaze tracking sensor may include an image sensor for acquiring an infrared wavelength image.

[0118] According to one embodiment, the electronic device (1201) may include at least one printed circuit board (PCB). For example, the electronic device (1201) may include a first PCB (1287-1) positioned in a first temple (1298-1) and a second PCB (1287-2) positioned in a second temple (1298-2). The first PCB (1287-1) and / or the second PCB (1287-2) may be electrically connected to other components of the electronic device (1201) via signal lines and / or a flexible PCB (FPCB). For example, a communication circuit, a memory, at least one sensor, and / or a processor may be disposed on the first PCB (1287-1) and / or the second PCB (1287-2). For example, each of the first PCB (1287-1) and the second PCB (1287-2) may be composed of multiple PCBs separated by an interposer.

[0119] According to one embodiment, the electronic device (1201) may include at least one battery. For example, the electronic device (1201) may include a first battery (1289-1) positioned at one end of a first temple (1298-1) and a second battery (1289-2) positioned at one end of a second temple (1298-2). The first battery (1289-1) and the second battery (1289-2) may be configured to supply power to components of the electronic device (1201).

[0120] According to one embodiment, the electronic device (1201) may include at least one speaker. For example, the electronic device (1201) may include a first speaker (1270-1) and a second speaker (1270-2). The electronic device (1201) may be configured to provide stereo sound using speakers positioned on the left and right sides.

[0121] According to one embodiment, the electronic device (1201) may include at least one microphone. For example, the electronic device (1201) may include a first microphone (1271-1), a second microphone (1271-2), and / or a third microphone (1271-3). The first microphone (1271-1) may be positioned on the right side of the frame (1297), the second microphone (1271-2) may be positioned on the left side of the frame (1297), and the third microphone (1271-3) may be positioned at a bridge of the frame (1297). In one example, the electronic device (1201) may perform beamforming using the first microphone (1271-1), the second microphone (1271-2), and / or the third microphone (1271-3).

[0122] According to one embodiment, the electronic device (1201) may include a first temple (1298-1), a second temple (1298-2), and a frame (1297). The first temple (1298-1), the second temple (1298-2), and the frame (1297) may be referred to as a housing. The first temple (1298-1) may be physically connected to the frame (1297) via a first hinge portion (1299-1) and may support the frame (1297) when worn. The second temple (1298-2) may be physically connected to the frame (1297) via a second hinge portion (1299-2) and may support the frame (1297) when worn.

[0123] The configuration of the electronic device (1201) described above is exemplary, and the embodiments of the present document are not limited thereto. For example, the electronic device (1201) may not include at least some of the components described with respect to FIG. 12 , or may further include other components in addition to the described components. For example, the electronic device (1201) may include at least one sensor (e.g., an acceleration sensor, a gyro sensor, and / or a touch sensor, etc.) and / or an antenna.

[0124] FIG. 13 is a schematic diagram of a method for tracking and displaying gaze through a transparent member according to one embodiment.

[0125] Referring to FIG. 13, a display (1361) (e.g., a first display (1261-1) or a second display (1261-2) of FIG. 12) can provide an image through a transparent member (1396) (e.g., a first transparent member (1296-1) or a second transparent member (1296-2) of FIG. 12). According to one embodiment, the display (1361) can input light corresponding to the image to an input optical member (1362) (e.g., a first input optical member (1262-1) or a second input optical member (1262-2) of FIG. 12) through a lens (1351). The input optical member (1362) can reflect or diffract the incident light and input it into an optical waveguide (1360). The output optical member (1364) can output light transmitted through the optical waveguide (1360) toward the user's eye (1399). In one example, the lens (1351) can be included in the display (1361). In one example, the position of the lens (1351) can be determined based on the distance between the transparent member (1396) and the user's eye (1399).

[0126] The gaze tracking sensor (1371) (e.g., the first gaze tracking sensor (1276-1) or the second gaze tracking sensor (1276-2) of FIG. 12) can acquire an image corresponding to at least a portion of the user's eye (1399). For example, light corresponding to the image of the user's eye (1399) can be reflected and / or diffracted through a first splitter (splitter) 1381 and input to an optical waveguide (1382). Light transmitted to a second splitter (1383) through the optical waveguide (1382) can be reflected and / or diffracted by the second splitter (1383) and output toward the gaze tracking sensor (1371).

[0127] FIG. 14 is a block diagram of an electronic device within a network environment according to various embodiments.

[0128] Referring to FIG. 14, in a network environment (1400), an electronic device (1401) may communicate with an electronic device (1402) via a first network (1498) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (1404) or a server (1408) via a second network (1499) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (1401) may communicate with the electronic device (1404) via the server (1408). According to one embodiment, the electronic device (1401) may include a processor (1420), a memory (1430), an input module (1450), an audio output module (1455), a display module (1460), an audio module (1470), a sensor module (1476), an interface (1477), a connection terminal (1478), a haptic module (1479), a camera module (1480), a power management module (1488), a battery (1489), a communication module (1490), a subscriber identification module (1496), or an antenna module (1497). In some embodiments, the electronic device (1401) may omit at least one of these components (e.g., the connection terminal (1478)), or may have one or more other components added. In some embodiments, some of these components (e.g., sensor module (1476), camera module (1480), or antenna module (1497)) may be integrated into a single component (e.g., display module (1460)).

[0129] The processor (1420) may, for example, execute software (e.g., a program (1440)) to control at least one other component (e.g., a hardware or software component) of the electronic device (1401) connected to the processor (1420) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (1420) may store commands or data received from other components (e.g., a sensor module (1476) or a communication module (1490)) in a volatile memory (1432), process the commands or data stored in the volatile memory (1432), and store result data in a non-volatile memory (1434). According to one embodiment, the processor (1420) may include a main processor (1421) (e.g., a central processing unit or an application processor) or an auxiliary processor (1423) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (1421). For example, when the electronic device (1401) includes the main processor (1421) and the auxiliary processor (1423), the auxiliary processor (1423) may be configured to use less power than the main processor (1421) or to be specialized for a given function. The auxiliary processor (1423) may be implemented separately from the main processor (1421) or as a part thereof.

[0130] The auxiliary processor (1423) may control at least a portion of functions or states associated with at least one component (e.g., the display module (1460), the sensor module (1476), or the communication module (1490)) of the electronic device (1401), for example, on behalf of the main processor (1421) while the main processor (1421) is in an inactive (e.g., sleep) state, or together with the main processor (1421) while the main processor (1421) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (1423) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (1480) or a communication module (1490)). In one embodiment, the auxiliary processor (1423) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (1401) where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (1408)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0131] The memory (1430) can store various data used by at least one component (e.g., the processor (1420) or the sensor module (1476)) of the electronic device (1401). The data can include, for example, software (e.g., the program (1440)) and input data or output data for commands related thereto. The memory (1430) can include volatile memory (1432) or non-volatile memory (1434).

[0132] The program (1440) may be stored as software in memory (1430) and may include, for example, an operating system (1442), middleware (1444), or an application (1446).

[0133] The input module (1450) can receive commands or data to be used in a component of the electronic device (1401) (e.g., a processor (1420)) from an external source (e.g., a user) of the electronic device (1401). The input module (1450) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0134] The audio output module (1455) can output audio signals to the outside of the electronic device (1401). The audio output module (1455) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0135] The display module (1460) can visually provide information to an external party (e.g., a user) of the electronic device (1401). The display module (1460) may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling the device. In one embodiment, the display module (1460) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0136] The audio module (1470) can convert sound into an electrical signal, or vice versa. According to one embodiment, the audio module (1470) can acquire sound through the input module (1450), output sound through the sound output module (1455), or an external electronic device (e.g., electronic device (1402)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (1401).

[0137] The sensor module (1476) can detect the operating status (e.g., power or temperature) of the electronic device (1401) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (1476) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0138] The interface (1477) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (1401) with an external electronic device (e.g., the electronic device (1402)). In one embodiment, the interface (1477) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0139] The connection terminal (1478) may include a connector through which the electronic device (1401) may be physically connected to an external electronic device (e.g., the electronic device (1402)). In one embodiment, the connection terminal (1478) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0140] The haptic module (1479) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (1479) may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0141] The camera module (1480) can capture still images and videos. In one embodiment, the camera module (1480) may include one or more lenses, image sensors, image signal processors, or flashes.

[0142] The power management module (1488) can manage the power supplied to the electronic device (1401). According to one embodiment, the power management module (1488) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0143] A battery (1489) may power at least one component of the electronic device (1401). In one embodiment, the battery (1489) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0144] The communication module (1490) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (1401) and an external electronic device (e.g., electronic device (1402), electronic device (1404), or server (1408)), and the performance of communication through the established communication channel. The communication module (1490) may operate independently from the processor (1420) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1490) may include a wireless communication module (1492) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (1494) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, a corresponding communication module can communicate with an external electronic device (1404) via a first network (1498) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (1499) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network or a wide area network)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1492) can verify or authenticate the electronic device (1401) within a communication network such as the first network (1498) or the second network (1499) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (1496).

[0145] The wireless communication module (1492) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency communications (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (1492) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (1492) may support various technologies for securing performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (1492) may support various requirements specified in the electronic device (1401), an external electronic device (e.g., the electronic device (1404)), or a network system (e.g., the second network (1499)). According to one embodiment, the wireless communication module (1492) may support a peak data rate (e.g., 20 Gbps or more) for eMBB implementation, a loss coverage (e.g., 164 dB or less) for mMTC implementation, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC implementation.

[0146] The antenna module (1497) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (1497) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (1497) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (1498) or the second network (1499), may be selected from the plurality of antennas by, for example, the communication module (1490). A signal or power may be transmitted or received between the communication module (1490) and the external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (1497).

[0147] According to various embodiments, the antenna module (1497) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0148] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0149] According to one embodiment, commands or data may be transmitted or received between the electronic device (1401) and an external electronic device (1404) via a server (1408) connected to a second network (1499). Each of the external electronic devices (1402 or 1404) may be the same or a different type of device as the electronic device (1401). According to one embodiment, all or part of the operations executed in the electronic device (1401) may be executed in one or more of the external electronic devices (1402, 1404, or 1408). For example, when the electronic device (1401) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (1401) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (1401). The electronic device (1401) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (1401) may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (1404) may include an Internet of Things (IoT) device. The server (1408) may be an intelligent server utilizing machine learning and / or a neural network.According to one embodiment, an external electronic device (1404) or server (1408) may be included within the second network (1499). The electronic device (1401) may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology and IoT-related technology.

[0150] FIG. 15 illustrates an example of a block diagram of a wearable device according to one embodiment.

[0151] Referring to FIG. 15, a wearable device (1503) according to one embodiment may include at least one of a processor (1510), a memory (1515), a display (1520), a camera (1525), a sensor (1530), or a communication circuit (1535). The processor (1510), the memory (1515), the display (1520), the camera (1525), the sensor (1530), and the communication circuit (1535) may be electrically and / or operably coupled with each other by an electronic component such as a communication bus (1502). The type and / or number of hardware components included in the wearable device (1503) are not limited to those illustrated in FIG. 15. For example, the wearable device (1503) may include only some of the hardware components illustrated in FIG. 15. The elements (e.g., layers and / or modules) within the memory described below may be logically separated. The elements within the memory (1515) may be included within a hardware component that is separate from the memory (1515). The operation performed by the processor (1510) using each element within the memory (1515) is one embodiment, and the processor (1510) may perform a different operation from the above operation through at least one of the elements within the memory (1515).

[0152] A processor (1510) of a wearable device (1503) according to one embodiment may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), and / or a central processing unit (CPU). The number of processors (1510) may be one or more. For example, the processor (1510) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core.

[0153] A memory (1515) of a wearable device (1503) according to one embodiment may include a hardware component for storing data and / or instructions input and / or output to the processor (1510). The memory (1515) may include, for example, a volatile memory such as a random-access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, and an embedded multi media card (eMMC).

[0154] In one embodiment, a display (1520) of a wearable device (1503) can output visualized information to a user of the wearable device (1503). For example, the display (1520) can be controlled by a processor (1510) including a circuit such as a graphic processing unit (GPU) to output visualized information to the user. The display (1520) can include a flat panel display (FPD) and / or electronic paper. The FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs).

[0155] In one embodiment, the camera (1525) of the wearable device (1503) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. The plurality of optical sensors included in the camera (1525) may be arranged in the form of a two-dimensional array. The camera (1525) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously to generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera (1525) may mean one (a) two-dimensional frame data acquired from the camera (1525). For example, video data captured using the camera (1525) may mean a sequence of a plurality of two-dimensional frame data acquired from the camera (1525) according to a frame rate. The camera (1525) may further include a flash light positioned toward the direction in which the camera (1525) receives light and outputs light toward the direction.

[0156] According to one embodiment, the wearable device (1503) may include a plurality of cameras, for example, cameras (1525), arranged in different directions. A first camera among the plurality of cameras may be referred to as a motion recognition camera (e.g., motion recognition cameras 1660-2 and 1660-3 of FIG. 16B ), and a second camera may be referred to as a gaze tracking camera (e.g., gaze tracking camera 1660-1 of FIG. 16B ). The wearable device (1503) may identify a position, shape, and / or gesture of a hand using an image acquired using the first camera. The wearable device (1503) may identify a direction of a gaze of a user wearing the wearable device (1503) using an image acquired using the second camera. For example, the direction in which the first camera faces may be opposite to the direction in which the second camera faces.

[0157] According to one embodiment, a sensor (1530) of a wearable device (1503) may generate electrical information that may be processed by a processor (1510) and / or a memory (1515) of the wearable device (1503) from non-electronic information related to the wearable device (1503). The information may be referred to as sensor data. The sensor (1530) may include a global positioning system (GPS) sensor, an image sensor, an ambient light sensor, and / or a time-of-flight (ToF) sensor for detecting a geographic location of the wearable device (1503), and an inertial measurement unit (IMU) for detecting a physical motion of the wearable device (1503).

[0158] In one embodiment, the communication circuit (1535) of the wearable device (1503) may include hardware components for supporting transmission and / or reception of electrical signals between the wearable device (1503) and an external electronic device. The communication circuit (1535) may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit (1535) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G new radio (NR), and / or 6G.

[0159] According to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by a processor (1510) of the wearable device (1503) may be stored in a memory (1515) of the wearable device (1503). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the wearable device (1503) and / or the processor (1510) may perform at least one of the operations of FIGS. 3 to 6 when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and / or an application is executed. Hereinafter, the fact that an application is installed in a wearable device (1503) may mean that one or more instructions provided in the form of an application are stored in a memory (1515), and that the one or more applications are stored in a format executable by the processor (1510) (e.g., a file having an extension specified by the operating system of the wearable device (1503)). As an example, the application may include a program and / or a library related to a service provided to a user.

[0160] Referring to FIG. 15, programs installed in the wearable device (1503) may be classified into any one of different layers, including an application layer (1540), a framework layer (1550), and / or a hardware abstraction layer (HAL) (1580), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware of the wearable device (1503) (e.g., a display (1520), a camera (1520), and / or a sensor (1530)) may be classified within the hardware abstraction layer (1580). The framework layer (1550) may be referred to as an XR framework layer in that it includes one or more programs for providing an XR (extended reality) service. For example, FIG. 15 illustrates layers within the memory (1515) as being divided, but the layers may be logically divided. However, the present invention is not limited thereto. Depending on the embodiment, the layers may be stored in a designated area within the memory (1515).

[0161] For example, within the framework layer (1550), programs designed to target at least one of the hardware abstraction layer (1580) and / or the application layer (1540) (e.g., a position tracker (1571), a space recognizer (1572), a gesture tracker (1573), and / or an eye tracker (1574), a face tracker (1575)) may be classified. Programs classified within the framework layer (1550) may provide an executable API (application programming interface) based on other programs.

[0162] For example, within the application layer (1540), programs designed to target users controlling wearable devices (1503) may be classified. Examples of programs classified within the application layer (1540) include, but are not limited to, an extended reality (XR) system user interface (UI) and / or an XR application (1542). For example, programs (e.g., software applications) classified within the application layer (1540) may call an application programming interface (API) to cause execution of functions supported by programs classified within the framework layer (1550).

[0163] For example, the wearable device (1503) may display one or more visual objects on the display (1520) for performing interaction with a user for using a virtual space based on the execution of the XR system UI (1541). A visual object may refer to an object that can be deployed on a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (1503) may provide a service for controlling functions available in a virtual space to the user based on the execution of the XR system UI (1541).

[0164] Referring to FIG. 15, a lightweight renderer (1543) and / or an XR plug-in (1544) are illustrated as being included within the XR system UI (1541), but are not limited thereto. For example, the XR system UI (1541) may cause execution of a function supported by the lightweight renderer (1543) and / or the XR plug-in (1544) included within the application layer (1540).

[0165] For example, the wearable device (1503) may acquire resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial modifications based on the execution of a lightweight renderer (1543). The lightweight renderer (1543) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline that allows partial modifications. The lightweight renderer (1543) may include a renderer that is built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (1503) may acquire resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (1544). The XR plugin (1544) can be referred to as an open XR native client from the perspective of defining (or configuring) the entire rendering pipeline.

[0166] For example, the wearable device (1503) may display a screen representing at least a portion of a virtual space on the display (1520) based on the execution of the XR application (1542). The XR plug-in (1541-1) included in the XR application (1542) may be referenced by the XR plug-in (1544) of the XR system UI (1541). Descriptions of the XR plug-in (1541-1) that overlap with the description of the XR plug-in (1544) may be omitted. The wearable device (1503) may cause the execution of the screen composition manager (1551) based on the execution of the XR application (1542).

[0167] According to one embodiment, the wearable device (101) may provide a virtual space service based on the execution of the screen composition manager (1551). For example, the screen composition manager (1551) may include a platform (e.g., an Android platform) for supporting the virtual space service. The wearable device (1503) may display the posture of a virtual object representing the user's posture rendered using data acquired through a sensor (1530) based on the execution of the screen composition manager (1551) on the display. The screen composition manager (1551) may be referred to as a composition presentation manager (CPM).

[0168] For example, the screen composition manager (1551) may include a runtime service (1552). As an example, the runtime service (1552) may be referred to as an OpenXR runtime module. The wearable device (1503) may be used to provide at least one of a pose prediction function, a frame timing function, and / or a spatial input function to a user through the wearable device (1503) based on the execution of the runtime service (1552). As an example, the wearable device (1503) may be used to perform rendering for a virtual space service to a user based on the execution of the runtime service (1552). For example, an application (e.g., unity or an OpenXR native application) may be implemented based on the execution of the runtime service (1552).

[0169] For example, the screen configuration manager (1551) may include a pass-through library (1553). Based on the execution of the pass-through library (1553), the wearable device (1503) may display a screen representing a virtual space on the display (1520), while simultaneously displaying another screen representing a real space acquired through the camera (1525) by overlaying at least a portion of the screen.

[0170] For example, the screen composition manager (1551) may include a renderer. The wearable device (101) may render a screen to be displayed on the display by compositing virtual layers (or virtual nodes) rendered based on sensor data (e.g., sensing data acquired through a camera (1525) or a sensor (1530)) and pass-through layers (or pass-through nodes) acquired through a pass-through library (1553) using the renderer through the screen composition manager (1551). The virtual layers may be referred to as virtual nodes and / or virtual surfaces. The wearable device (1503) may render each of the virtual layers or all of the virtual layers through the screen composition manager (1551).

[0171] For example, the screen configuration manager (1551) may include an input manager (1554). The wearable device (1503) may identify acquired data (e.g., sensor data) by executing one or more programs included in the recognition service layer (1570) based on the execution of the input manager (1554). The wearable device (1503) may initiate execution of at least one of the functions of the wearable device (1503) using the acquired data.

[0172] For example, the perception abstract layer (1560) can be used for data exchange between the screen composition manager (1551) and the perception service layer (1570). From the perspective of being used for data exchange between the screen composition manager (1551) and the perception service layer (1570), the perception abstract layer (1560) can be referred to as an interface. For example, the perception abstract layer (1560) can be referred to as OpenPX and / or PPAL (perception platform abstract layer). The perception abstract layer (1560) can be used for a perception client and a perception service.

[0173] According to one embodiment, the recognition service layer (1570) may include one or more programs for processing data acquired from a sensor (1530) (or a camera (1525)). The one or more programs may include at least one of a position tracker (1571), a space recognizer (1572), a gesture tracker (1573), an eye tracker (1574), and / or a face tracker (1575). The type and / or number of the one or more programs included in the recognition service layer (1570) are not limited to those illustrated in FIG. 15.

[0174] For example, the wearable device (1503) can identify the pose of the wearable device (1503) using the sensor (1530) based on the execution of the position tracker (1571). The wearable device (1503) can identify the 6 degrees of freedom pose (6 DOF pose) of the wearable device (1503) using data acquired using the camera (1525) and the IMU based on the execution of the position tracker (1571). The position tracker (1571) may be referred to as a head tracking (HeT) module.

[0175] For example, the wearable device (1503) may be used to construct a three-dimensional virtual space surrounding the wearable device (1503) (or a user of the wearable device (1503)) based on the execution of the space recognizer (1572). The wearable device (1503) may reconstruct the three-dimensional surroundings of the wearable device (1503) using data acquired using the camera (1525) based on the execution of the space recognizer (1572). The wearable device (1503) may identify at least one of a plane, a slope, or stairs based on the three-dimensionally reconstructed surroundings of the wearable device (1503) based on the execution of the space recognizer (1572). The space recognizer (1572) may be referred to as a scene understanding (SU) module.

[0176] For example, the wearable device (1503) may be used to identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (1503) based on the execution of the gesture tracker (1573). As an example, the wearable device (1503) may identify a pose and / or gesture of a hand of a user using data acquired from a sensor (1530) based on the execution of the gesture tracker (1573). As an example, the wearable device (1503) may identify a pose and / or gesture of a hand of a user based on data (or images) acquired using a camera (1525) based on the execution of the gesture tracker (1573). The gesture tracker (1573) may be referred to as a hand tracking (HaT) module and / or a gesture tracking module.

[0177] For example, the wearable device (1503) can identify (or track) eye movements of a user of the wearable device (1503) based on the execution of the gaze tracker (1574). As an example, the wearable device (1503) can identify eye movements of the user using data acquired from at least one sensor based on the execution of the gaze tracker (1574). As an example, the wearable device (1503) can identify eye movements of the user based on data acquired using a camera (1525) (e.g., the gaze tracking camera (1660-1) of FIGS. 16A and 16B) and / or an infrared light emitting diode (IR LED) based on the execution of the gaze tracker (1574). The gaze tracker (1574) may be referred to as an eye tracking (ET) module and / or a gaze tracking module.

[0178] For example, the recognition service layer (1570) of the wearable device (1503) may further include a face tracker (1575) for tracking the user's face. For example, the wearable device (1503) may identify (or track) the movement of the user's face and / or the user's expression based on the execution of the face tracker (1575). The wearable device (1503) may estimate the user's expression based on the movement of the user's face based on the execution of the face tracker (1575). As an example, the wearable device (1503) may identify the movement of the user's face and / or the user's expression based on data (e.g., an image) acquired using a camera based on the execution of the face tracker (1575).

[0179] FIG. 16A illustrates an example of a perspective view of a wearable device according to one embodiment. According to one embodiment, the wearable device (1503) may have a form of glasses that can be worn on a body part (e.g., head) of a user. The wearable device (1503) of FIGS. 16A and 16B may be an example of the electronic device (10) of FIG. 1. The wearable device (1503) may include a head-mounted display (HMD). For example, the housing of the wearable device (1503) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (1503) may include one or more straps capable of being twined around the user's head, and / or one or more temples attachable to the ears of the head.

[0180] Referring to FIG. 16A, according to one embodiment, a wearable device (1503) may include at least one display (1650) and a frame (1600) supporting at least one display (1650).

[0181] According to one embodiment, a wearable device (1503) can be worn on a part of a user's body. The wearable device (1503) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (1503). For example, the wearable device (1503) can display a virtual reality image provided from at least one optical device (1682, 1684) of FIG. 16B on at least one display (1650) in response to a user's designated gesture acquired through a motion recognition camera (or motion tracking camera) (1660-2, 1660-3) of FIG. 16B.

[0182] According to one embodiment, at least one display (1650) may provide visual information to a user. For example, at least one display (1650) may include a transparent or translucent lens. At least one display (1650) may include a first display (1650-1) and / or a second display (1650-2) spaced apart from the first display (1650-1). For example, the first display (1650-1) and the second display (1650-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0183] Referring to FIG. 16B, at least one display (1650) can provide a user with visual information transmitted from external light and other visual information distinct from the visual information through a lens included in the at least one display (1650). The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, the at least one display (1650) can include a first surface (1631) and a second surface (1632) opposite the first surface (1631). A display area can be formed on the second surface (1632) of the at least one display (1650). When a user wears the wearable device (1503), external light can be transmitted to the user by being incident on the first surface (1631) and transmitted through the second surface (1632). As another example, at least one display (1650) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (1682, 1684) on a real screen transmitted through external light, in a display area formed on the second surface (1632).

[0184] In one embodiment, at least one display (1650) may include at least one waveguide (1633, 1634) that diffracts light emitted from at least one optical device (1682, 1684) and transmits the diffracted light to a user. The at least one waveguide (1633, 1634) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (1633, 1634). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (1633, 1634) may be propagated to the other end of the at least one waveguide (1633, 1634) by the nano-pattern. At least one waveguide (1633, 1634) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)), or at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (1633, 1634) may be positioned within the wearable device (1503) to guide a screen displayed by at least one display (1650) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (1633, 1634).

[0185] The wearable device (1503) can analyze an object included in a real-world image collected through a camera (1660-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (1650). The virtual object can include at least one of text and an image regarding various information related to the object included in the real-world image. The wearable device (1503) can analyze the object based on a multi-camera, such as a stereo camera. For the object analysis, the wearable device (1503) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using the multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (1503) can view an image displayed on at least one display (1650).

[0186] According to one embodiment, the frame (1600) may be configured as a physical structure that allows the wearable device (1503) to be worn on the user's body. According to one embodiment, the frame (1600) may be configured so that, when the user wears the wearable device (1503), the first display (1650-1) and the second display (1650-2) can be positioned corresponding to the user's left and right eyes. The frame (1600) may support at least one display (1650). For example, the frame (1600) may support the first display (1650-1) and the second display (1650-2) to be positioned corresponding to the user's left and right eyes.

[0187] Referring to FIG. 16A, the frame (1600) may include an area (1620) that at least partially contacts a portion of the user's body when the user wears the wearable device (1503). For example, the area (1620) of the frame (1600) that contacts a portion of the user's body may include an area that contacts a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (1503) makes contact with. According to one embodiment, the frame (1600) may include a nose pad (1610) that contacts a portion of the user's body. When the wearable device (1503) is worn by the user, the nose pad (1610) may contact a portion of the user's nose. The frame (1600) may include a first temple (1604) and a second temple (1605) that contact another part of the user's body that is distinct from the part of the user's body.

[0188] For example, the frame (1600) may include a first rim (1601) that surrounds at least a portion of the first display (1650-1), a second rim (1602) that surrounds at least a portion of the second display (1650-2), a bridge (1603) that is disposed between the first rim (1601) and the second rim (1602), a first pad (1611) that is disposed along a portion of the edge of the first rim (1601) from one end of the bridge (1603), a second pad (1612) that is disposed along a portion of the edge of the second rim (1602) from the other end of the bridge (1603), a first temple (1604) that extends from the first rim (1601) and is fixed to a portion of the ear of the wearer, and a second temple (1605) that extends from the second rim (1602) and is fixed to a portion of the ear opposite the ear. There are. The first pad (1611) and the second pad (1612) can be in contact with a part of the user's nose, and the first temple (1604) and the second temple (1605) can be in contact with a part of the user's face and a part of the user's ear. The temples (1604, 1605) can be rotatably connected to the rim through the hinge units (1606, 1607) of FIG. 16B. The first temple (1604) can be rotatably connected to the first rim (1601) through the first hinge unit (1606) disposed between the first rim (1601) and the first temple (1604). The second temple (1605) may be rotatably connected to the second rim (1602) via a second hinge unit (1607) disposed between the second rim (1602) and the second temple (1605). In one embodiment, the wearable device (1503) may use a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (1600) to identify an external object (e.g., a user's fingertip) touching the frame (1600) and / or a gesture performed by the external object.

[0189] According to one embodiment, the wearable device (1503) may include hardwares that perform various functions (e.g., hardwares to be described later based on the block diagram of FIG. 15). For example, the hardwares may include a battery module (1670), an antenna module (1675), at least one optical device (1682, 1684), speakers (e.g., speakers 1655-1, 1655-2), a microphone (e.g., microphones 1665-1, 1665-2, 1665-3), a light-emitting module (not shown), and / or a printed circuit board (PCB) (1690) (e.g., a printed circuit board). The various hardwares may be arranged within the frame (1600).

[0190] According to one embodiment, microphones (e.g., microphones 1665-1, 1665-2, 1665-3) of a wearable device (1503) may be disposed on at least a portion of a frame (1600) to acquire sound signals. A first microphone (1665-1) disposed on a bridge (1603), a second microphone (1665-2) disposed on a second rim (1602), and a third microphone (1665-3) disposed on a first rim (1601) are illustrated in FIG. 16B , but the number and arrangement of the microphones (1665) are not limited to the embodiment of FIG. 16B . When the number of microphones (1665) included in the wearable device (1503) is two or more, the wearable device (1503) can identify the direction of a sound signal by using a plurality of microphones arranged on different parts of the frame (1600).

[0191] According to one embodiment, at least one optical device (1682, 1684) may project a virtual object onto at least one display (1650) to provide various image information to a user. For example, at least one optical device (1682, 1684) may be a projector. At least one optical device (1682, 1684) may be disposed adjacent to at least one display (1650) or may be included within at least one display (1650) as a part of at least one display (1650). According to one embodiment, the wearable device (1503) may include a first optical device (1682) corresponding to a first display (1650-1) and a second optical device (1684) corresponding to a second display (1650-2). For example, at least one optical device (1682, 1684) may include a first optical device (1682) positioned at an edge of a first display (1650-1) and a second optical device (1684) positioned at an edge of a second display (1650-2). The first optical device (1682) may transmit light to a first waveguide (1633) positioned on the first display (1650-1), and the second optical device (1684) may transmit light to a second waveguide (1634) positioned on the second display (1650-2).

[0192] In one embodiment, the camera (1660) may include a recording camera (1660-4), an eye tracking camera (ET CAM) (1660-1), and / or a motion recognition camera (1660-2, 1660-3). The recording camera (1660-4), the eye tracking camera (1660-1), and the motion recognition cameras (1660-2, 1660-3) may be positioned at different locations on the frame (1600) and may perform different functions. The eye tracking camera (1660-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (1503). For example, the wearable device (1503) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (1660-1). The wearable device (1503) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (1660-1). The wearable device (1503) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (1503) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (1660-1). The wearable device (1503) can render an image (or screen) displayed on at least one display (1650) based on the position of the user's eyes. For example, the visual quality of a first area related to the gaze within the image and the visual quality (e.g., resolution, brightness, saturation, grayscale, or PPI (pixels per inch)) of a second area distinguished from the first area may be different from each other.The wearable device (1503) can obtain an image (or screen) having a visual quality of a first area matching the user's gaze and a visual quality of a second area using foveated rendering. For example, if the wearable device (1503) supports an iris recognition function, user authentication can be performed based on iris information obtained using a gaze tracking camera (1660-1). Although an example in which the gaze tracking camera (1660-1) is positioned toward both eyes of the user is illustrated in FIG. 16B, the embodiment is not limited thereto, and the gaze tracking camera (1660-1) can be positioned solely toward the user's left eye or right eye.

[0193] In one embodiment, the capturing camera (1660-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (1660-4) can be used to obtain a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (1660-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (1650). The at least one display (1650) can display a single image in which information about an actual image or background including an image of the specific object obtained using the capturing camera (1660-4) and a virtual image provided through at least one optical device (1682, 1684) are superimposed. The wearable device (1503) can compensate for depth information (e.g., the distance between the wearable device (1503) and an external object acquired through a depth sensor) using an image acquired through the capture camera (1660-4). The wearable device (1503) can perform object recognition using an image acquired through the capture camera (1660-4). The wearable device (1503) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the capture camera (1660-4). The wearable device (1503) can perform a pass-through function to display an image acquired through the capture camera (1660-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (1650). The shooting camera (1660-4) may be referred to as a high resolution (HR) camera or a photo video (PV) camera.The camera (1660-4) may provide auto focus (AF) and optical image stabilization (OIS) functions. The camera (1660-4) may include a global shutter (GS) camera and / or a rolling shutter (RS) camera. In one embodiment, the camera (1660-4) may be positioned on a bridge (1603) positioned between the first rim (1601) and the second rim (1602).

[0194] The gaze tracking camera (1660-1) can implement more realistic augmented reality by tracking the gaze of a user wearing a wearable device (1503) and matching the user's gaze with visual information provided to at least one display (1650). For example, when the wearable device (1503) looks straight ahead, the wearable device (1503) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (1650). The gaze tracking camera (1660-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (1660-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (1660-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (1660-1) may be positioned within the first rim (1601) and / or the second rim (1602) to face the direction in which the user wearing the wearable device (1503) is positioned.

[0195] The gesture recognition cameras (1660-2, 1660-3) can recognize the movement of the user's entire body, such as the user's torso, hands, or face, or a part of the body, and thereby provide a specific event on a screen provided on at least one display (1650). The gesture recognition cameras (1660-2, 1660-3) can recognize the user's gesture (gesture recognition), obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (1650). The processor can identify the signal corresponding to the gesture, and perform a designated function based on the identification. The gesture recognition cameras (1660-2, 1660-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 DOF pose). The processor may perform gesture recognition and / or object tracking functions using the motion recognition cameras (1660-2, 1660-3). In one embodiment, the motion recognition cameras (1660-2, 1660-3) may be disposed on the first limb (1601) and / or the second limb (1602). The motion recognition cameras (1660-2, 1660-3) may include a global shutter (GS) camera (e.g., a global shutter (GS) camera) used for head tracking, hand tracking, and / or spatial recognition based on one of a three-degree-of-freedom pose or a six-degree-of-freedom pose. The GS camera may include two or more stereo cameras to track fine movements. As an example, the GS camera may be included in the gaze tracking camera (1660-1) for tracking the gaze of a user.

[0196] The camera (1660) included in the wearable device (1503) is not limited to the above-described gaze tracking camera (1660-1) and motion recognition cameras (1660-2, 1660-3). For example, the wearable device (1503) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (1503) can identify an external object based on a sensor for identifying the distance between the wearable device (1503) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (1660) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (1503) may include a camera (1660) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (1503).

[0197] Although not shown, in one embodiment, the wearable device (1503) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being captured using the camera (1660). The light source may include an infrared wavelength LED. The light source may be disposed on at least one of the frame (1600) and the hinge units (1606, 1607).

[0198] In one embodiment, the battery module (1670) may supply power to the electronic components of the wearable device (1503). In one embodiment, the battery module (1670) may be disposed within the first temple (1604) and / or the second temple (1605). For example, the battery module (1670) may be a plurality of battery modules (1670). The plurality of battery modules (1670) may be disposed within each of the first temple (1604) and the second temple (1605). In one embodiment, the battery module (1670) may be disposed at an end of the first temple (1604) and / or the second temple (1605).

[0199] The antenna module (1675) can transmit signals or power to the outside of the wearable device (1503), or receive signals or power from the outside. In one embodiment, the antenna module (1675) can be positioned within the first temple (1604) and / or the second temple (1605). For example, the antenna module (1675) can be positioned close to one surface of the first temple (1604) and / or the second temple (1605).

[0200] The speaker (1655) can output an audio signal to the outside of the wearable device (1503). The audio output module may be referred to as a speaker. In one embodiment, the speaker (1655) can be positioned within the first temple (1604) and / or the second temple (1605) so as to be positioned adjacent to the ear of a user wearing the wearable device (1503). For example, the speaker (1655) can include a second speaker (1655-2) positioned within the first temple (1604) and thus adjacent to the user's left ear, and a first speaker (1655-1) positioned within the second temple (1605) and thus adjacent to the user's right ear.

[0201] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state to visually provide information regarding a specific state of the wearable device (1503) to the user. For example, when the wearable device (1503) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (1601) and / or the second rim (1602).

[0202] Referring to FIG. 16B, according to one embodiment, a wearable device (1503) may include a printed circuit board (PCB) (1690). The PCB (1690) may be included in at least one of the first temple (1604) or the second temple (1605). The PCB (1690) may include an interposer positioned between at least two sub-PCBs. One or more hardwares included in the wearable device (1503) (e.g., hardwares illustrated by different blocks in FIG. 15) may be positioned on the PCB (1690). The wearable device (1503) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0203] According to one embodiment, a wearable device (1503) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (1503) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (1503). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (1503) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (1503) based on the IMU.

[0204] Figures 17a and 17b illustrate an example of an exterior appearance of a wearable device according to one embodiment. The wearable device (1503) of Figures 17a and 17b may be an example of the wearable device (1503) of Figure 1. An example of an exterior appearance of a first side (1710) of a housing of the wearable device (1503) according to one embodiment is illustrated in Figure 17a, and an example of an exterior appearance of a second side (1720) opposite to the first side (1710) may be illustrated in Figure 17b.

[0205] Referring to FIG. 17A, a first surface (1710) of a wearable device (1503) according to one embodiment may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (1503) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (1604) and / or the second temple (1605) of FIGS. 16A and 16B). A first display (1650-1) for outputting an image to a left eye among the user's two eyes, and a second display (1650-2) for outputting an image to a right eye among the two eyes, may be disposed on the first surface (1710). The wearable device (1503) may be formed on the first surface (1710) and may further include a rubber or silicone packing to prevent interference from light (e.g., ambient light) different from the light emitted from the first display (1650-1) and the second display (1650-2).

[0206] According to one embodiment, a wearable device (1503) may include cameras (1660-1) for photographing and / or tracking both eyes of a user adjacent to each of the first display (1650-1) and the second display (1650-2). The cameras (1660-1) may be referred to as the gaze tracking camera (1660-1) of FIG. 16B. According to one embodiment, a wearable device (1503) may include cameras (1660-5, 1660-6) for photographing and / or recognizing a face of a user. The cameras (1660-5, 1660-6) may be referred to as FT cameras. The wearable device (1503) can control an avatar representing the user in a virtual space based on the facial motion of the user identified using the cameras (1660-5, 1660-6). For example, the wearable device (1503) can change the texture and / or shape of a part of the avatar (e.g., a part of the avatar representing a human face) using information obtained by the cameras (1660-5, 1660-6) (e.g., an FT camera) and representing the facial expression of the user wearing the wearable device (1503).

[0207] Referring to FIG. 17b, a camera (e.g., cameras (1660-7, 1660-8, 1660-9, 1660-10, 1660-11, 1660-12)) and / or a sensor (e.g., a depth sensor (1730)) for obtaining information related to the external environment of the wearable device (1503) may be disposed on a second surface (1720) opposite to the first surface (1710) of FIG. 17a. For example, the cameras (1660-7, 1660-8, 1660-9, 1660-10) may be disposed on the second surface (1720) for recognizing external objects. Cameras (1660-7, 1660-8, 1660-9, 1660-10) may be referenced to the motion recognition cameras (1660-2, 1660-3) of FIG. 16b.

[0208] For example, using cameras (1660-11, 1660-12), the wearable device (1503) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (1660-11) can be placed on a second surface (1720) of the wearable device (1503) to obtain an image to be displayed through a second display (1650-2) corresponding to the right eye among the two eyes. The camera (1660-12) can be placed on a second surface (1720) of the wearable device (1503) to obtain an image to be displayed through a first display (1650-1) corresponding to the left eye among the two eyes. As an example, the wearable device (1501) can obtain a single screen using a plurality of images obtained through the cameras (1660-11, 1660-12). Cameras (1660-11, 1660-12) may be referenced to the shooting camera (1660-4) of FIG. 16b.

[0209] According to one embodiment, the wearable device (1503) may include a depth sensor (1730) disposed on the second face (1720) to identify a distance between the wearable device (1503) and an external object. Using the depth sensor (1730), the wearable device (1503) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (1503). Although not shown, a microphone may be disposed on the second face (1720) of the wearable device (1503) to obtain sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0210] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0211] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0212] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0213] Various embodiments of the present document may be implemented as software (e.g., a program (1440)) including one or more instructions stored in a storage medium (e.g., an internal memory (1436) or an external memory (1438)) readable by a machine (e.g., an electronic device (1401)). For example, a processor (e.g., a processor (1420)) of the machine (e.g., an electronic device (1401)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0214] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0215] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, camera; At least one display; memory; and comprising at least one processor electrically connected to the camera, at least one display, and the memory; The above memory stores instructions, When the above instructions are individually or collectively executed by the at least one processor, the electronic device: Create virtual objects based on user input from the user, Identify the action pattern of the generated virtual object, Based on the identified behavior pattern, the movement path of the virtual object from the first point to the second point is determined, and An electronic device that displays the motion of the virtual object on at least one display based on the determined movement path.

2. In claim 1, When the above instructions are individually or collectively executed by the at least one processor, the electronic device: Based on the above user input, identifying the external properties of the virtual object, and An electronic device that identifies the behavioral pattern based on the above external attributes.

3. In claim 2, When the above instructions are individually or collectively executed by the at least one processor, the electronic device: Obtaining voice input from the above user, An electronic device that identifies the behavior pattern corresponding to the voice input.

4. In claim 3, An electronic device, wherein the above behavior pattern includes at least one behavior type of the virtual object and a behavior attribute of each behavior type.

5. In claim 1, An electronic device wherein the first point corresponds to a boundary of the user's field of view.

6. In claim 1, When the above instructions are individually or collectively executed by the at least one processor, the electronic device: Using an artificial intelligence model to explore the user's surroundings, Based on the above surrounding environment, a map of the above surrounding environment is generated, An electronic device that determines the movement path based on the generated map and the behavior pattern.

7. In claim 6, When the above instructions are individually or collectively executed by the at least one processor, the electronic device: Identify the characteristics of multiple objects included in the above map, Based on the characteristics and behavior patterns of the above plurality of objects, at least one interactable object that can be included in the movement path of the virtual object is determined, An electronic device that determines a movement path including at least one interactable object.

8. In claim 7, When the above instructions are individually or collectively executed by the at least one processor, the electronic device: If the behavioral pattern of the above virtual object is the first behavioral pattern, the first interactable object among the plurality of objects is determined as the at least one interactable object, An electronic device that determines a second interactable object, different from the first interactable object among the plurality of objects, as the at least one interactable object if the behavioral pattern of the virtual object is the second behavioral pattern.

9. In claim 7, When the above instructions are individually or collectively executed by the at least one processor, the electronic device: Based on the characteristics and behavior patterns of the plurality of objects, at least one obstacle is identified among the plurality of objects, An electronic device that determines a travel path that detours at least one obstacle.

10. In claim 1, The electronic device further comprises a communication circuit; When the above instructions are individually or collectively executed by the at least one processor, the electronic device: Obtain notifications from external electronic devices, Create a person object associated with the above notification as the above virtual object, An electronic device, wherein the motion further comprises providing information of the notification.

11. In a method using an electronic device, An action that creates a virtual object based on user input; An action for identifying the action pattern of the generated virtual object; An action of determining a movement path of the virtual object from a first point to a second point based on the identified behavior pattern; and A method comprising: displaying motion of the virtual object based on the determined movement path.

12. In claim 11, An operation of identifying an external property of the virtual object based on the user input; and A method comprising: identifying a behavioral pattern based on said external attributes; 13. In claim 12, An operation of obtaining voice input from said user; and A method comprising: an operation for identifying the behavioral pattern corresponding to the voice input; 14. In claim 13, A method wherein the above behavior pattern includes at least one behavior type of the virtual object and a behavior attribute of each behavior type.

15. In claim 12, A method wherein the first point corresponds to a boundary of the user's field of view.

Images