Wearable device, method, and non-transitory computer-readable storage medium for generating global movement of avatar

Abstract

This wearable device may comprise: a memory for storing instructions; a communication circuit; a display assembly including at least one display; and at least one processor. The wearable device may be caused to: display an avatar, which includes a first part controllable by a first control device and a second part controllable by the first control device and a second control device, and an object related to the avatar; among first data on movement of the first control device and second data on movement of the second control device, receive the first data; generate third data on the first part of the avatar by using the first data; generate fourth data on the second part of the avatar by using the third data and information on the object; generate fifth data on global movement of the avatar by using the third data and the fourth data; and display the avatar through the display assembly by using the fifth data.

Description

Wearable device, method, and non-transitory computer-readable storage medium for generating global motion of an avatar

[0001] The present disclosure relates to a wearable device, a method, and a non-transitory computer-readable storage medium for generating global motion of an avatar.

[0002] A device utilizing VST (video see-through) can acquire an image of a user's frontal view through a camera mounted on the front of an HMD (head-mounted display) and display it on the display of the HMD. A device utilizing VST can be utilized as a tool for implementing virtual reality, augmented reality, and mixed reality. Within a virtual space, a user of the HMD can be represented by an avatar corresponding to the user. The user can control the movement of the avatar through input to the HMD. The avatar controlled by the user can interact with objects within the environment provided through the HMD. The user can control the movement of the avatar considering the environment.

[0003] The above information may be provided as background art to aid in understanding the present disclosure.

[0004] No claim or determination is made as to whether any of the above is applicable as prior art to the present disclosure.

[0005] A wearable device is described. The wearable device may include a memory storing instructions and including one or more storage media. The wearable device may include communication circuitry. The wearable device may include a display assembly including at least one display. The wearable device may include at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display, through the display assembly, an avatar including a first portion controllable by a first control device and a second portion controllable by a second control device paired with the first control device, and an object associated with the avatar. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to receive, through the communication circuit, first data of movement of the first control device transmitted from the first control device and second data of movement of the second control device transmitted from the second control device while displaying the avatar and the object. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate, based on the reception, third data about the first part of the avatar moving relative to the object using the first data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate, based on the reception, fourth data about the second part of the avatar moving relative to the object using the third data and information about the object.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate fifth data about global movement of the avatar using the third data and the fourth data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the avatar moving relative to the object through the display assembly using the fifth data.

[0006] A method is provided. The method can be executed in a wearable device having a communication circuit and a display assembly including at least one display. The method can include an operation of displaying, through the display assembly, an avatar and an object related to the avatar, the avatar including a first portion controllable by a first control device and a second portion controllable by a second control device paired with the first control device. The method can include an operation of receiving, through the communication circuit, first data from among first data regarding movement of the first control device transmitted from the first control device and second data regarding movement of the second control device transmitted from the second control device, while displaying the avatar and the object. The method can include an operation of generating, based on the reception, third data regarding the first portion of the avatar moving relative to the object using the first data. The method can include an operation of generating, using the third data and information regarding the object, fourth data regarding the second portion of the avatar moving relative to the object. The method may include an operation of generating fifth data regarding the global movement of the avatar using the third data and the fourth data. The method may include an operation of displaying the avatar moving with respect to the object through the display assembly using the fifth data.

[0007] A non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by a wearable device having a communication circuit and a display assembly including at least one display, cause the wearable device to display, through the display assembly, an avatar and an object associated with the avatar, the avatar including a first portion controllable by a first control device and a second portion controllable by a second control device paired with the first control device. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to receive, through the communication circuit, first data regarding movement of the first control device transmitted from the first control device and second data regarding movement of the second control device transmitted from the second control device while displaying the avatar and the object. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to generate third data about the first part of the avatar moving relative to the object using the first data based on the reception. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to generate fourth data about the second part of the avatar moving relative to the object using the third data and information about the object.The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to generate fifth data about global movement of the avatar using the third data and the fourth data. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display, through the display assembly, the avatar moving relative to the object using the fifth data.

[0008] A wearable device is described. The wearable device may include a memory storing instructions and including one or more storage media. The wearable device may include a camera assembly including a plurality of cameras. The wearable device may include communication circuitry. The wearable device may include a display assembly including at least one display. The wearable device may include at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display, through the display assembly, an avatar including a first part controllable by a first part of a user's body and a second part controllable by a second part of the user's body paired with the first part of the user's body, and an object associated with the avatar. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to acquire first data of movement of the first part of the body of the user and second data of movement of the second part of the body of the user using images acquired through the camera assembly while displaying the avatar and the object. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate, based on the acquisition, third data of the first part of the avatar moving relative to the object using the first data.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate fourth data about the second part of the avatar moving relative to the object using the third data and information about the object. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate fifth data about global movement of the avatar using the third data and the fourth data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display, through the display assembly, the avatar moving relative to the object using the fifth data.

[0009] A method is provided. The method can be executed in a wearable device having a camera assembly including a plurality of cameras, a communication circuit, and a display assembly including at least one display. The method can include an operation of displaying, through the display assembly, an avatar including a first part controllable by a first part of a user's body and a second part controllable by a second part of the user's body paired with the first part of the user's body, and an object associated with the avatar. The method can include an operation of acquiring, while displaying the avatar and the object, first data on movement of the first part of the user's body and second data on movement of the second part of the user's body using images acquired through the camera assembly, the first data. The method can include an operation of generating, based on the acquisition, third data on the first part of the avatar moving with respect to the object using the first data. The method may include an operation of generating fourth data about the second part of the avatar moving with respect to the object using the third data and information about the object. The method may include an operation of generating fifth data about the global movement of the avatar using the third data and the fourth data. The method may include an operation of displaying the avatar moving with respect to the object through the display assembly using the fifth data.

[0010] A non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by a wearable device having a camera assembly including a plurality of cameras, a communication circuit, and a display assembly including at least one display, cause the wearable device to display, through the display assembly, an avatar and an object associated with the avatar, the avatar including a first part controllable by a first part of a user's body and a second part controllable by a second part of the user's body paired with the first part of the user's body. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to acquire, using images acquired through the camera assembly, first data regarding movement of the first part of the user's body and second data regarding movement of the second part of the user's body, the first data. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to generate third data about the first part of the avatar moving relative to the object using the first data based on the acquisition. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to generate fourth data about the second part of the avatar moving relative to the object using the third data and information about the object.The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to generate fifth data about global movement of the avatar using the third data and the fourth data. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display, through the display assembly, the avatar moving relative to the object using the fifth data.

[0011] Figure 1 illustrates an example of an environment including a wearable device and a control device.

[0012] Figure 2 illustrates an example of controlling an avatar with one control device included in a pair of control devices.

[0013] Figure 3 is a simplified block diagram of an exemplary wearable device.

[0014] FIG. 4 is a flowchart illustrating an exemplary method for generating global movement of an avatar using data received from a first control device among a first control device and a second control device.

[0015] Figure 5 shows an example of a control device.

[0016] Figure 6 illustrates an example of an object related to an avatar.

[0017] Figure 7 shows an example of a model trained through machine learning.

[0018] Figure 8 illustrates an example of generating global motion of an avatar using inverse kinematics.

[0019] Figure 9 illustrates an example of generating global movement of an avatar using one control device.

[0020] Figure 10 is a flowchart illustrating an exemplary method for generating global motion of an avatar by further utilizing other information about the environment around the avatar.

[0021] Figure 11 shows an example of the environment around an avatar.

[0022] Figure 12 is a flowchart illustrating an exemplary method for identifying whether a mode for compensating avatar movement is activated.

[0023] FIG. 13 is a flowchart illustrating an exemplary method for generating global movement of an avatar using data received from a first control device and a second control device.

[0024] Figure 14 is a flowchart illustrating an exemplary method for generating global motion of an avatar using images acquired through a camera assembly.

[0025] FIG. 15 is a block diagram of an electronic device within a network environment according to various embodiments.

[0026] Figure 16a shows an example of a perspective view of a wearable device.

[0027] FIG. 16b illustrates an example of one or more hardware elements placed within a wearable device.

[0028] Figures 17a and 17b show an example of the appearance of a wearable device.

[0029] Figure 18 shows an example of a block diagram of a wearable device.

[0030] Figure 19 shows an example of a block diagram of an electronic device for displaying an image in virtual space.

[0031] Figure 1 illustrates an example of an environment including a wearable device and a control device.

[0032] Referring to FIG. 1, a wearable device (100) can be worn on the body of a user (e.g., the user (200) of FIG. 2). For example, the wearable device (100) can be worn on the head of the user. For example, the wearable device (100) can provide virtual reality (VR) to the user through a display assembly (e.g., the display assembly (308) of FIG. 3). For example, the wearable device (100) can provide the user with an image captured through a camera assembly of the wearable device (100) (e.g., the camera assembly (309) of FIG. 3). For example, the wearable device (100) can provide augmented reality (AR) or mixed reality (MR) to a user (e.g., a user (200) of FIG. 2) through a display assembly (e.g., a display assembly (308) of FIG. 3). The wearable device (100) can be connected to a first control device (110) and a second control device (120). For example, the wearable device (100) can be connected to the first control device (110) and the second control device (120) wirelessly. For example, the wearable device (100) can be connected to the first control device (110) and the second control device (120) through a Bluetooth communication technique. Although an operation of wirelessly connecting the wearable device (100) and the control device is described, this is merely exemplary. For example, the wearable The device (100) can be connected to the first control device (110) and the second control device (120) by wire.

[0033] The wearable device (100) may include a display assembly (e.g., display assembly (308) of FIG. 3). For example, the wearable device (100) may display an avatar (130) and an object (e.g., a violin) through the display assembly (e.g., display assembly (308) of FIG. 3). The display assembly (e.g., display assembly (308) of FIG. 3) is described and illustrated in more detail with reference to FIG. 3. A user of the wearable device (100) may control the movement of the avatar (130) through a first control device (110) and a second control device (120). For example, the avatar (130) may be controlled by the user of the wearable device (100). For example, the wearable device (100) can control the movement of the avatar (130) by manipulating the first control device (110) or the second control device (120) connected to the wearable device (100). For example, the first control device (110) can correspond to a first part of the avatar (130). For example, the second control device (120) can correspond to a second part of the avatar (130).

[0034] The wearable device (100) can display an avatar (130) and the object through a display assembly (e.g., the display assembly (308) of FIG. 3). For example, the avatar (130) can exist within an environment provided through the display assembly (e.g., the display assembly (308) of FIG. 3). For example, the object can exist within an environment provided through the display assembly (e.g., the display assembly (308) of FIG. 3). For example, the object can be a violin (210). For example, the avatar (130) can interact with the violin (210). As an example of the interaction, the avatar (130) can play the violin using a first part of the avatar (130) (e.g., the left arm) and a second part of the avatar (130) (e.g., the right arm). The wearable device (100) can display an avatar (130) playing the object violin (210) within the environment through a display assembly (e.g., the display assembly (308) of FIG. 3). The violin (210) can be described as a virtual object.

[0035] Figure 2 illustrates an example of controlling an avatar with one control device included in a pair of control devices.

[0036] Referring to FIG. 2, a user (200) can use a wearable device (100) and a first control device (110). For example, the wearable device (100) can display a violin through a display assembly (e.g., the display assembly (308) of FIG. 3). For example, the wearable device (100) can control an avatar (130) using data received from a control device. For example, the user (200) can control an avatar (130) displayed on the display assembly using a control device (e.g., the first control device (110)). The user (200) can grip the first control device (110) with his left hand. The user (200) can refrain from gripping the second control device (120) with his right hand. The first control device (110) can be connected to the wearable device (100). The second control device (120) may not be connected to the wearable device (100). The wearable device (100) can identify whether the first control device (110) and the second control device (120) are connected to the wearable device (100).

[0037] An avatar (130) corresponding to a user (200) can interact with an object related to the avatar (130). For example, the wearable device (100) can display an image of the avatar (130) playing a violin (210) through the display assembly. For example, the user (200) can assume a pose of playing the violin using the first control device (110). The wearable device (100) can obtain data on the movement of the wearable device (100) from at least one sensor (not shown). Based on the obtained data, the wearable device (100) can display the avatar (130) through the display assembly (e.g., the display assembly (308) of FIG. 3). The wearable device (100) can display the left hand of the avatar (130) corresponding to the first control device (110) through a display assembly (e.g., the display assembly (308) of FIG. 3) by obtaining data on the movement of the left hand of the avatar (130) from the first control device (110). Since the wearable device (100) does not receive data on the movement from the second control device (120), the wearable device (100) may not display the movement of the right hand of the avatar (130) corresponding to the second control device (120) through a display assembly (e.g., the display assembly (308) of FIG. 3). For example, the first control device (110) may correspond to a left hand joint of the avatar (130). For example, the second control device (120) may correspond to a right hand joint of the avatar. For example, a user (200) of a wearable device (100) may imagine playing a violin and strumming the violin with the right hand of an avatar (130). The user (200) may assume a pose of playing the violin. For example, the avatar (130) may hold a violin (210-1) with his left hand and a violin bow (210-2) with his right hand. The violin (210-1) may be referred to as a violin body.

[0038] The wearable device (100) can receive movement data from the first control device (110) among the first control device (110) and the second control device (120). The wearable device (100) can receive data on the left hand of the avatar (130). The wearable device (100) may not reflect the movement of the right hand of the user (200) to the avatar (130). The right arm of the avatar (130) can assume a basic pose. For example, the state (291) can be described as a state in which the left hand of the avatar (130) holds a violin (210-1) and the right hand of the avatar (130) assumes a basic pose while holding a violin bow (210-2). The right hand of the avatar (130) can be described as a second part of the avatar (130). For example, the basic pose can be the A-pose. Since the wearable device (100) receives data on the movement of the first control device (110) from the first control device (110) among the first control device (110) and the second control device (120), the movement of the user's (200) right hand may not be reflected in the right arm of the avatar (130). The avatar (130) may take the A pose with its right arm. The wearable device (100) may display an unnatural avatar (130). For example, in state (291), the pose of the avatar (130) may be unusual.

[0039] State (292) can be described as a state in which the wearable device (100) generates data for a second part of the avatar (130) (e.g., the right arm of the avatar (130)). The wearable device (100) can display an avatar (130) controlling a violin using a first part of the avatar (130) (e.g., the left arm of the avatar (130)) and a second part of the avatar (130). For example, it can be natural for the avatar (130) to interact with the violin using the first part of the avatar (130) and the second part of the avatar (130). For example, the wearable device (100) can generate data for a second part of the avatar (130) using data for the first part of the avatar (130) and information about the violin. For example, in state (292), the wearable device (100) may display an avatar (130) normally playing the violin through a display assembly (e.g., display assembly (308) of FIG. 3). For example, the wearable device (100) may include hardware components used to perform or execute the above operations. The hardware components are described and exemplified with reference to FIG. 3.

[0040] Figure 3 is a simplified block diagram of an exemplary wearable device.

[0041] Referring to FIG. 3, the wearable device (100) may include at least one processor (307), a communication circuit (305), and a memory (306).

[0042] At least one processor (307) may include a hardware component for processing data using instructions stored in the memory (306). The hardware component for processing data may include a central processing unit (CPU) (e.g., including processing circuitry). The hardware component for processing data may include a graphic processing unit (GPU) (e.g., including processing circuitry). The hardware component for processing data may include a display processing unit (DPU) (e.g., including processing circuitry). The hardware component for processing data may include a neural processing unit (NPU) (e.g., including processing circuitry).

[0043] At least one processor (307) may include one or more cores. For example, at least one processor (307) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core.

[0044] The memory (306) may include hardware components for storing data and / or instructions input to and / or output from at least one processor (307). The memory (306) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multimedia card (EMMC).

[0045] The communication circuit (305) may include hardware components for supporting transmission and / or reception of signals between the wearable device (100) and an external electronic device. The communication circuit (305) may include, for example, at least one of a modem, an antenna, and an optical / electronic (O / E) converter. The communication circuit (305) may support transmission and / or reception of signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), zigbee, long term evolution (LTE), and 5G new radio (NR).

[0046] The display assembly (308) can output visualized information. For example, the display assembly (308) can output visualized information to the user under the control of at least one processor (307). The display assembly (308) can include hardware components of the wearable device (100) used to display a screen. For example, the display assembly (308) can include light-emitting elements and circuits (e.g., transistors) that control the light-emitting elements to emit light. For example, each of the light-emitting elements can include an organic light emitting diode (OLED) or a micro LED. However, the present invention is not limited thereto. For example, the display assembly (308) can include a liquid crystal display (LCD).

[0047] As a non-limiting example, the display assembly (308) may include a first display positioned in front of the left eye of a user wearing the wearable device (100) and a second display positioned in front of the right eye of the user wearing the wearable device (100). For example, first content provided through a screen displayed through the first display may be (substantially) identical to second content provided through a screen displayed through the second display. While the first content and the second content are identical, the screen displayed through the second display may have a disparity with respect to the screen displayed through the first display. For example, the disparity may cause the display assembly (308) to present content (e.g., corresponding to the first content and the second content) in three dimensions.

[0048] The camera assembly (309) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. For example, the camera assembly (309) may be described as one or more image sensors. For example, the camera assembly (309) may be utilized to acquire images of the environment surrounding the wearable device (100). For example, at least a portion of the camera assembly (309) may have a field of view (FOV) corresponding to the FOV of a user's eye. For example, the FOV of a portion of the camera assembly (309) may be different from the FOV of another portion of the camera assembly (309). For example, the camera assembly (309) may be utilized to identify an input means positioned around the wearable device (100).

[0049] FIG. 4 is a flowchart illustrating an exemplary method for generating global movement of an avatar using data received from a first control device among a first control device and a second control device.

[0050] Referring to FIG. 4, the wearable device (100) can display an avatar through the display assembly (308). For example, in operation 410, at least one processor (307) can display an avatar (130) including a first part controllable by a first control device (110) and a second part controllable by a second control device (120) paired with the first control device (110), and an object related to the avatar through the display assembly (308). At least one processor (307) can display an environment surrounding the avatar (130) through the display assembly (308). An exemplary operation of at least one processor (307) displaying an avatar controllable by a control device through the display assembly (308) is described, but the embodiment is not limited thereto. At least one processor (307) can display an avatar and an object related to the avatar, through a display assembly (308), including a first part controllable by a first part of the user's body and a second part controllable by a second part of the user's body paired with the first part of the user's body.

[0051] In operation 420, at least one processor (307) may receive, through the communication circuit (305), first data among first data on movement of the first control device (110) transmitted from the first control device (110) and second data on movement of the second control device (120) transmitted from the second control device (120), while displaying the avatar and the object. At least one processor (307) may identify whether it is connected to the first control device (110) and the second control device (120). For example, the first data on movement of the first control device (110) received from the first control device (110) may be referred to as the first data. For example, the second data on movement of the second control device (120) received from the second control device (120) may be referred to as the second data.

[0052] At least one processor (307) may receive the second data from the second control device (120) through the communication circuit (305) based on the connection. For example, since the at least one processor (307) is not connected to the second control device (120), the at least one processor (307) may not receive the second data regarding the movement of the second control device (120) from the second control device (120) through the communication circuit (305). For example, the at least one processor (307) may be connected to the first control device (110) among the first control device (110) and the second control device (120). For example, the second control device (120) may not be connected to the wearable device (100). For example, the user (200) may only be able to use one hand. For example, a user (200) of a wearable device (100) may hold a first control device (110) with one hand and hold another object with the other hand. For example, a second control device (120) may not be held by the user (200). At least one processor (307) may disconnect or terminate the connection with the second control device (120) while displaying an avatar (130) through the display assembly (308). For example, the second control device (120) may disconnect from the wearable device (100) when it detects that there is no input for a preset period of time. For example, a sensor of the second control device (120) may detect the hand of the user (200). For example, the second control device (120) may disconnect from the wearable device (100) when there is no user input. For example, at least one processor (307) may disconnect or terminate the connection with the second control device (120) based on the battery of the second control device (120) being discharged.

[0053] The first control device (110) and the second control device (120) can be operated by the user's (200) hand. An exemplary operation in which the user (200) moves the first part of the avatar through the control devices is described, but the embodiment is not limited thereto. The wearable device (100) can control the first part of the avatar and the second part of the avatar by obtaining data on the movement of the first part of the user's (200) body and data on the movement of the second part using images obtained through the camera assembly (309) of the wearable device (100). The types of control devices are described and exemplified in more detail with reference to FIG. 5.

[0054] Figure 5 shows an example of a control device.

[0055] Referring to FIG. 5, the wearable device (100) can be connected to a first control device (110) and a second control device (120). The wearable device (100) can control an avatar (130) corresponding to a user (200) using the first control device (110) and the second control device (120). The first control device (110) can correspond to a first part of the avatar (130). The second control device (120) can correspond to a second part of the avatar (130). For example, the control device (540) can correspond to the first control device (110). For example, the control device (550) can correspond to the second control device (120). For example, the wearable device (100) can generate a movement of an arm of the avatar (130) using the movement of the control device (540). The movement of the control device (540) can be described as a movement of the position of the control device (540) and a rotation of the control device (540). The movement of a part of the avatar (130) can be described as a movement of the position of a part of the avatar (130) (or a movement of a representative position of a part of the avatar (130)) and a rotation of a part of the avatar (130) (or a change in the orientation of a part of the avatar (130)).

[0056] The first control device (110) and the second control device (120) can be attached to the body of the user (200). The first control device (110) can correspond to the control device (510) or the control device (520). The control device (510) and the control device (520) can be attached to the ankle of the user (200). The control device (e.g., 510, 520) attached to the ankle can generate movement of a part of the avatar corresponding to the control device. For example, the movement of the leg part of the avatar can be generated by the control device (e.g., 510, 520) attached to the ankle. For example, when the user (200) attaches a control device (540) capable of controlling a part of the hand of the avatar (130) to the ankle of the user (200), the user (200) can control a part of the hand of the avatar (130) using the ankle.

[0057] For example, the control device (530) may be attached to the waist of the user (200). The first control device (110) may correspond to the control device (530). The control device (530) attached to the waist may generate movement of a part of the avatar corresponding to the control device. For example, the movement of the waist part of the avatar may be generated by the control device (530) attached to the waist. For example, the wearable device (100) may control the avatar (130) using the control device (530), the control device (540), and the control device (550). For example, the wearable device (100) may generate data on the global movement of the avatar (130) using data on the movement of the control device (530), data on the movement of the control device (540), and data on the movement of the control device (550). For example, the wearable device (100) can generate data on the movement of the control device (550) using data on the movement of the control device (530) and data on the movement of the control device (540).

[0058] Referring again to FIG. 4 , at operation 430, at least one processor (307) may generate third data for a first portion of an avatar (130) that moves relative to an object associated with the avatar using first data regarding the movement of the first control device (110) transmitted from the first control device (110). For example, the third data may be generated for movement of the first portion of the avatar (130) corresponding to a change in the position of the first control device (110) and a change in the direction of the first control device (110). For example, the third data may indicate a change in the position of the first portion of the avatar (130) and a change in the direction of the first portion of the avatar (130). An exemplary operation of generating the third data for the first portion of the avatar (130) using data regarding the movement of the first control device (110) received from the first control device (110) is described, but the embodiment is not limited thereto. At least one processor (307) can generate third data for a first part of the avatar (130) by obtaining data on movement of a first part of the body of the user (200) using images acquired through the camera assembly (309).

[0059] In operation 440, at least one processor (307) may generate fourth data about a second part of the avatar (130) moving with respect to the object using the third data and information about an object related to the avatar (130). The at least one processor (307) may identify an object interacting with the avatar (130) or an environment surrounding the avatar. For example, the at least one processor (307) may obtain information about an object related to the avatar (130) at a preset time interval. The at least one processor (307) may obtain other information about the environment surrounding the avatar (130) at a preset time interval. For example, the at least one processor (307) may obtain the information and the other information from a software application (not shown) stored in the memory (306). For example, the at least one processor (307) may obtain the information and the other information from a server (not shown) where a software application is stored. At least one processor (307) can generate fourth data for a second part of the avatar (130) using third data for a first part of the avatar (130) and other information about the environment around the avatar (130). At least one processor (307) can generate the fourth data considering the type of the object. At least one processor (307) can generate the fourth data considering the relative distance between the avatar (130) and the object. At least one processor (307) can generate the fourth data considering the type of the object and the relative distance between the avatar (130) and the object. An exemplary operation of a user (200) moving a first part of the avatar through a control device is described, but the embodiment is not limited thereto.The wearable device (100) can control a first part of an avatar by obtaining data on the movement of a first part of the user's (200) body using images acquired through the camera assembly (309) of the wearable device (100). The object is described and illustrated in more detail with reference to FIG. 6.

[0060] Figure 6 illustrates an example of an object related to an avatar.

[0061] Referring to FIG. 6, the object may be an object that interacts with the avatar (130). For example, the object may be a violin (210). For example, the object may be a box (620). For example, the object may be a bow (630). A portion (e.g., a first portion) of the avatar (130) may correspond to the object. For example, a portion of the avatar (130) may correspond to a hand of the avatar. For example, the avatar (130) may hold the violin (210-1) with one hand and the violin bow (210-2) with the other hand. For example, the avatar (130) may hold the box (620) with both hands. For example, the avatar (130) may hold the bow (630-1) with one hand and pull the bow string (630-2) with the other hand.

[0062] Movement of a part of the avatar (130) may be determined according to the type of the object and / or the positional relationship between the avatar (130) and the object. For example, if the left hand of the avatar (130) is holding a violin (210-1), the right hand of the avatar (130) holding the violin bow (210-2) may be positioned in front of the face of the avatar (130). For example, if the avatar (130) is holding a box (620), the left and right hands of the avatar (130) may be positioned in front of the stomach of the avatar (130). For example, if the left hand of the avatar (130) is holding a bow (630-1), the right hand of the avatar (130) pulling the bow string (630-2) paired with the bow may be positioned next to the right cheek of the avatar (130).

[0063] At least one processor (307) may generate fourth data for a second part of the avatar (130) using third data for a first part of the avatar (130) and information about the object. As a non-limiting example, the at least one processor (307) may generate the fourth data using a model trained through machine learning (e.g., model (710) of FIG. 7). As a non-limiting example, the model may be included in the wearable device (100). As a non-limiting example, the model may also be included in another electronic device (not shown) that is distinct from the wearable device (100).

[0064] For example, at least one processor (307) may determine a model for generating fourth data from among models (e.g., model (710)) trained through machine learning using the third data and the information. For example, at least one processor (307) may determine a model for generating fourth data by considering the information. For example, at least one processor (307) may determine a model for generating fourth data by considering other information about the environment around the avatar (130). For example, at least one processor (307) may determine a model for generating fourth data by considering the information and the other information. The model is described and exemplified in more detail with reference to FIG. 7.

[0065] Figure 7 shows an example of a model trained through machine learning.

[0066] Referring to FIG. 7, at least one processor (307) can generate fourth data (750) by providing third data and information (730) about the object to the model (710). At least one processor (307) can generate fourth data by providing third data and other information (not shown) about the environment around the avatar (130) to the model (710). For example, the model (710) can receive third data about a first part of an avatar (e.g., avatar (130)) and information about an object related to the avatar. The model (710) can receive other information about the environment around the avatar (130). The model (710) can generate fourth data (750) about a second part of the avatar using the third data about the first part of the avatar and the information about the object related to the avatar. The model (710) can output the fourth data about the second part of the avatar. For example, the model (710) can be trained through machine learning. For example, the model (710) can be trained through deep learning. For example, the model (710) can be trained through a neural network. An external electronic device (not shown) can train the model (710). The external electronic device can be described as a server (not shown). The wearable device of the user (760) can be connected to the control devices of the user (760). The wearable device of the user (760) can receive data on the movement of the control devices from the control devices. The wearable device of the user (760) can control an avatar corresponding to the user (760) based on the data on the movement of the control devices of the user (760). For example, the number of control devices of the user (760) can be 2 or more.The wearable device of the user (770) can receive data on the movement of the control devices from the control devices. The wearable device of the user (770) can control an avatar corresponding to the user (770) based on the data on the movement of the control devices of the user (770). For example, the number of the control devices of the user (770) may be 2 or more. The wearable device of the user (780) can receive data on the movement of the control devices from the control devices. The wearable device of the user (780) can control an avatar corresponding to the user (780) based on the data on the movement of the control devices of the user (780). For example, the number of the control devices of the user (780) may be 2 or more.

[0067] The server can collect data on the movements of control devices (e.g., the first control device (110), the second control device (120)) from users (e.g., 760, 770, 780). For example, the server can receive data on the movements of the control devices of the users from wearable devices (e.g., the wearable device (100)) used by the users at preset intervals. For example, the server can train models using the received data at preset intervals. The server can store information on the movements of avatars (e.g., an avatar corresponding to the user (760)) of the users (760), (770), and (780). For example, the server can store data that affects the movements of a part of the avatars of the users (760), (770), and (780) together. For example, the data that affects the movements of a part of the avatars of the users (760), (770), and (780) may be data on objects that interact with the avatars of the users. For example, data affecting the movement of each of the avatars may be data about the environment surrounding each of the avatars. For example, the server may store information about objects associated with the avatars of each of the users (760), users (770), and users (780). For example, the server may store other information about the environment surrounding each of the avatars of each of the users (760), users (770), and users (780). Although the operation of the server storing data of the users (760), users (770), and users (780) is described, this is exemplary and the embodiments are not limited thereto.

[0068] The server can train multiple models (e.g., model (710)) using information about the movements of the avatars of each of the users (760), users (770), and users (780). The models can learn information about the movements of the avatars of each of the users (760), users (770), and users (780). The server can classify the collected data according to the type of the object. The server can classify the collected data according to the environment surrounding the avatar. The server can train multiple models based on the classified data. The server can train the multiple models by considering the type of object interacting with each avatar and the environment surrounding each avatar. The server can collect the distance between the center of the object interacting with each avatar and a part of the avatar. The environment surrounding each avatar can be background music. For example, the environment surrounding each avatar can be a background location. For example, the environment surrounding each avatar may not cause movement of each avatar. For example, the environment surrounding each of the avatars may not cause movements of users (e.g., 760, 770, 780) corresponding to each of the avatars. For example, each of the avatars may run, walk, perform bodyweight exercises, or speak. Each of the avatars may move according to the type of the object or the environment surrounding it. For example, the movement of a first part of the avatar (130) and the movement of a second part of the avatar (130) may be determined to play the violin (210). For example, the wearable device (100) may display the avatar (130) playing the violin (210) through the display assembly (308).For example, the model (710) can learn information about the distance, trajectory, and direction in which the right hand of the avatar (130) moves with respect to the left hand of the avatar (130) by identifying the violin (210). For example, the model (710) that has completed training can generate data about a second part of the avatar using data about a first part of the avatar (130) and information about the violin (210) associated with the avatar (130). For example, the data about the first part of the avatar (130) can be acquired using the first control device. For example, the data about the first part of the avatar (130) can be acquired using images acquired through the camera assembly (309).

[0069] At least one processor (307) may generate fourth data for a second part of the avatar (130) using a suitable model for a third data for a first part of the avatar (130) among the models and an object related to the avatar (130). For example, the suitable model may be a model corresponding to an object related to the avatar (130) among the models. For example, the suitable model may be a model corresponding to an environment surrounding the avatar (130) among the models. For example, the suitable model may be a model corresponding to an object related to the avatar (130) and an environment surrounding the avatar (130) among the models. For example, at least one processor (307) may determine a model (710) among the trained models to which the third data (720) and information (730) will be input in order to generate the fourth data for the second part of the avatar (130). At least one processor (307) can obtain information about the movement of the second control device (120) by inputting information about the movement of the first control device (110) and information about the object (730) to the model (710). For example, at least one processor (307) can obtain fourth data (750) about the second part of the avatar (130) by inputting third data (720) about the first part of the avatar (130) and information about the object (730) related to the avatar (130) to the model (710). For example, at least one processor (307) can obtain fourth data (750) about the second part of the avatar (130) by inputting third data (720) about the first part of the avatar (130) and other information about the environment around the avatar (130) to the model (710). For example, the model (710) can estimate or calculate fourth data (750) for a second part of the avatar (130) using the third data (720) and the above information (730).For example, the estimation of the fourth data (750) of the model (710) may be related to the third data (720) about the first part of the avatar (130) and information (730) about the object. For example, the estimation of the fourth data (750) of the model (710) may be related to the third data (720) about the first part of the avatar (130) and other information about the environment around the avatar (130).

[0070] At least one processor (307) can store the model (710) in the memory (306). The wearable device (100) can generate fourth data using the model (710) stored in the memory (306). For example, the at least one processor (307) can obtain fourth data (750) for the second part of the avatar (130) by inputting third data (720), information (730), and sixth data (740) into the model (710). An exemplary operation of the wearable device (100) generating the fourth data using the model (710) is described, but the embodiment is not limited thereto. A server (not shown) can store the model (710) in the server. The server can be referenced as an external electronic device. At least one processor (307) can transmit third data (720), information (730), and sixth data (740) to an external electronic device via a communication circuit (305). The external electronic device (not shown) can receive the third data (720), information (730), and sixth data (740) from the wearable device (100). The external electronic device can obtain fourth data (750) for a second part of the avatar (130) by inputting the third data (720), information (730), and sixth data (740) into the model (710). The external electronic device can transmit the fourth data (750) to the wearable device (100). At least one processor (307) can receive the fourth data (750) via the communication circuit (305).

[0071] At least one processor (307) may further include at least one sensor. For example, at least one processor (307) may obtain sixth data (740) about the movement of the wearable device (100) from the sensor. At least one processor (307) may further input information about the movement of the wearable device (100) into the model (710). For example, at least one processor (307) may obtain fourth data (750) by further inputting sixth data (740) about the movement of the wearable device (100) into the model (710). At least one processor (307) may further generate fourth data using the sixth data (740) in operation 440. For example, at least one processor (307) can generate fourth data (750) using third data (720), information about the object (730), and sixth data (740).

[0072] At least one processor (307) can generate fourth data (750) for a second part of the avatar by considering an object related to the avatar (130) and an environment around the avatar (130). At least one processor (307) can input third data and information about an object related to the avatar (130) into the model (710) to generate the fourth data (750). For example, at least one processor (307) can consider whether the object is a violin (210), a box (620), or a bow (630). For example, at least one processor (307) can determine the position and rotation of a part of the avatar (130) according to the type of the object and the relative position between the avatar (130) and the object. For example, at least one processor (307) can generate the fourth data (750) by calculating the distance between the center of the object interacting with the avatar (130) and a part of the avatar (130). For example, the environment surrounding the avatar (130) may be background music. For example, the environment surrounding the avatar (130) may be a background location. For example, the environment surrounding the avatar (130) may be an orchestra. For example, the environment surrounding the avatar (130) may be a playground. At least one processor (307) may generate fourth data (750) for a second part of the avatar (130) using information (730) about objects related to the avatar (130), other information about the environment surrounding the avatar (130), and third data (720) about a first part of the avatar (130).

[0073] Referring again to FIG. 4, at operation 450, at least one processor (307) may generate fifth data about global movement of the avatar (130) using the third data and the fourth data. The at least one processor (307) may further generate fifth data about global movement of the wearable device (100) using sixth data about movement of the wearable device (100). For example, the at least one processor (307) may generate fifth data about global movement of the avatar (130) using the third data, the fourth data, and the sixth data. The at least one processor (307) may generate global movement of the avatar (130) using movement of a portion of the avatar (130). The at least one processor (307) may generate the fifth data using inverse kinematics. Inverse kinematics is described and exemplified in more detail with reference to FIG. 8.

[0074] Figure 8 illustrates an example of generating global motion of an avatar using inverse kinematics.

[0075] Referring to FIG. 8, at least one processor (307) may obtain information about the global movement of the avatar (130) using inverse kinematics. For example, at least one processor (307) may obtain information about the joints of the avatar (130) from information about a portion of the avatar (130) using inverse kinematics. The joints (e.g., 810, 820, 821) of the avatar (130) may each correspond to at least one joint of the user (200). For example, joint (810) may correspond to the neck of the user (200). For example, joint (820) may correspond to the left shoulder of the user (200). For example, joint (821) may correspond to the left elbow of the user (200). For example, joint (822) may correspond to the left wrist of the user (200). For example, joint (830) may correspond to the right shoulder of the user (200). For example, joint (831) may correspond to the right elbow of the user (200). For example, joint (832) may correspond to the right wrist of the user (200). For example, joint (840) may correspond to the waist of the user (200). For example, joint (850) may correspond to the right hip joint of the user (200). For example, joint (851) may correspond to the right knee of the user (200). For example, joint (852) may correspond to the right ankle of the user (200). For example, joint (860) may correspond to the left hip joint of the user (200). For example, joint (861) may correspond to the left knee of the user (200). For example, joint (862) may correspond to the left ankle of the user (200). For example, at least one processor (307) may obtain information about joints (820) to (840) of the avatar (130) as information about the movement of some joints (e.g., 822, 832) of the avatar (130).The above information may include information about the position and rotation of the joint (822). At least one processor (307) may generate global movement of the avatar (130) using the acquired information about the joint (820) to the joint (840). At least one processor (307) may further use data about the movement of the wearable device (100) to generate global movement of the avatar (130). For example, the data about the movement of the wearable device (100) may include information about the position and rotation of the wearable device (100). For example, at least one processor (307) may generate data about the movement of each of the joints (850) to (862) using data about the movement of each of the joints (852) and (862). For example, at least one processor (307) may generate (or calculate) (or obtain) information about the movement of each of the parts of the avatar (130) by using information about each movement obtained from the first control device (110) and the second control device (120). At least one processor (307) may further use information about the movement of the wearable device (100) to calculate information about the movement of each of the parts of the avatar (130).

[0076] Although Fig. 8 illustrates an example in which an avatar (130) includes 14 joints, this is for convenience of explanation. For example, if the avatar (130) is in the form of an animal distinct from a human, the number of joints included in the avatar (130) may change. For example, if the standard for distinguishing joints is changed, the number of joints in the avatar (130) may change even if the avatar (130) is in the form of a human.

[0077] Referring back to FIG. 4, at operation 460, at least one processor (307) may use the fifth data to display an avatar (130) moving with respect to an object through the display assembly (308). For example, the fifth data may include information indicating a global movement of the avatar (130). For example, even when the wearable device (100) is connected to the first control device (110) among the first control device (110) and the second control device (120), at least one processor (307) may generate a global movement of the avatar (130) and display the avatar (130) through the display assembly (308). The global movement of the avatar (130) is described and exemplified in more detail with reference to FIG. 9.

[0078] Figure 9 illustrates an example of generating global movement of an avatar using one control device.

[0079] Referring to FIG. 9, a wearable device (100) can be connected to a first control device (110). A user (200) can wear the wearable device (100). The user (200) can use the first control device (110) with one hand. The wearable device (100) can display an avatar (130) through a display assembly (308). For example, the avatar (130) can interact with an object associated with the avatar (130). For example, the object associated with the avatar (130) can be a violin (210). For example, the avatar (130) can play the violin (210). For example, when a user (200) controls the first control device (110) among the first control device (110) and the second control device (120), at least one processor (307) can generate data on the movement of the second control device (120). For example, at least one processor (307) can obtain data on a part of an avatar (130) corresponding to the second control device (120) using the model (710). For example, at least one processor (307) can obtain data on the global movement of the avatar (130) using the data on the movement of the second control device (120). For example, the avatar (130) can play the violin (210) using the first part of the avatar (130) and the second part of the avatar (130).

[0080] State (900) can be described as a state in which an avatar (130) playing a violin (210) is displayed through a display assembly (308) using a first part (950) of the avatar (130) and a second part (960) of the avatar (130) while the user (200) controls the avatar (130) using a first control device (110). The first control device (110) can correspond to the first part (950) of the avatar (130). For example, the first part (950) can correspond to the left arm of the avatar (130). The second control device (120) may not correspond to the second part (960) of the avatar (130). For example, the second part (960) can correspond to the right arm of the avatar (130). The wearable device (100) can display an avatar (130) holding a violin (210-1) using a first portion (950) and a violin bow (210-2) using a second portion (960) through the display assembly (308). The wearable device (100) can generate data on the global movement of the avatar (130) using information about the first portion (950) and the second portion (960). An exemplary motion of the avatar (130) playing the violin (210) is described, but the embodiment is not limited thereto.

[0081] At least one processor (307) can generate natural movements of the avatar (130) by receiving first data among first data about movements of the first control device (110) and second data about movements of the second control device (120), and generating fourth data about a second part (960) of the avatar (130) and fifth data about global movements of the avatar (130). By generating the fifth data, the at least one processor (307) can cause an object that requires the first part of the avatar (130) and the second part of the avatar (130) to interact with the avatar (130). For example, the at least one processor (307) can cause the avatar (130) to perform more work by generating the fourth data.

[0082] At least one processor (307) may use objects associated with the avatar (130) and other information about the environment surrounding the avatar (130) to generate data for a second portion of the avatar (130). The generation of data for the second portion of the avatar (130) by the at least one processor (307) using other information about the environment surrounding the avatar (130) is described and exemplified in more detail with reference to FIGS. 10 and 11.

[0083] Figure 10 is a flowchart illustrating an exemplary method for generating global motion of an avatar by further utilizing other information about the environment around the avatar.

[0084] Referring to FIG. 10, in operation 1010, at least one processor (307) may display an avatar (130) and an object related to the avatar (130), including a first part controllable by a first control device (110) and a second part controllable by a second control device (120) paired with a second control device (120), through a display assembly (308). Operation 1010 may correspond to operation 410 of FIG. 4. An exemplary operation in which a user (200) moves a first part of the avatar using a control device is described, but the embodiment is not limited thereto. At least one processor (307) may display the avatar through the display assembly (308) by obtaining data on movement of a first part of the body of the user (200) using images obtained through a camera assembly (309) of the wearable device (100).

[0085] In operation 1020, at least one processor (307) may receive, through the communication circuit (305), first data among first data regarding movement of the first control device (110) transmitted from the first control device (110) and second data regarding movement of the second control device (120) transmitted from the second control device (120). Operation 1020 may correspond to operation 420 of FIG. 4.

[0086] In operation 1030, at least one processor (307) may generate third data for a first part of an avatar moving relative to an object using the first data. Operation 1030 may correspond to operation 430 of FIG. 4.

[0087] In operation 1040, at least one processor (307) may generate fourth data about a second part of the avatar moving with respect to the object using third data, information about the object, and other information about the environment around the avatar (130). The at least one processor (307) may display the environment around the avatar (130) through the display assembly (308). The at least one processor (307) may display the environment around the avatar (130) through the display assembly (308) using a software application stored in the wearable device (100). The user (200) may control the first control device (110) or the second control device (120) considering the environment around the avatar (130). The at least one processor (307) may obtain sixth data about the movement of the wearable device (100) from at least one sensor. The at least one processor (307) may generate the fourth data using the sixth data. An exemplary operation of generating fourth data for a second part of an avatar using a control device by at least one processor (307) is described, but the embodiment is not limited thereto. At least one processor (307) may generate fourth data for a second part of the avatar (130) by obtaining data on movement of a first part of the body of a user (200) using images acquired through a camera assembly (309).

[0088] The environment surrounding the avatar is described and illustrated in more detail with reference to Figure 11.

[0089] Figure 11 shows an example of the environment around an avatar.

[0090] Referring to FIG. 11, at least one processor (307) can display an environment surrounding an avatar (130) through a display assembly (308). The environment surrounding the avatar (130) can cause movement of a part of the avatar (130). For example, at least one processor (307) can display a scene of the avatar (130) conducting an orchestra through the display assembly (308). For example, the orchestra can include a violinist (e.g., 1110), a flutist (e.g., 1130), a trumpeter (e.g., 1150), and a celloist (e.g., 1170). For example, a first violinist (1110) and a second violinist (1120) can play the violin under the direction of the avatar (130). For example, a first flute player (1130) and a second flute player (1140) can play the flute under the direction of an avatar (130). For example, a first trumpet player (1150) and a second trumpet player (1160) can play the trumpet under the direction of an avatar (130). For example, a first cello player (1170) and a second cello player (1180) can play the cello under the direction of an avatar (130). Unlike the violin (210), the box (620), and the bow (630) of FIG. 6, the avatar (130) can avoid contact with the environment around the avatar (130). The environment around the avatar (130) can cause movement of the user (200). For example, the movement of the second part of the avatar (130) may be determined based on the environment around the avatar (130).

[0091] As a non-limiting example, the environment surrounding the avatar (130) may not cause movement of a part of the avatar (130). The environment surrounding the avatar (130) may not cause movement of the user (200). For example, if the environment surrounding the avatar (130) is a playground, the playground may not cause movement of the user (200). At least one processor (307) may generate fourth data using other information about the environment surrounding the avatar (130) and third data. At least one processor (307) may generate the fourth data using a model trained through machine learning (e.g., model (710)). At least one processor (307) may obtain sixth data about the movement of the wearable device (100) from at least one sensor (not shown). At least one processor (307) may further generate the fourth data using the sixth data. The fourth data may be described as information for generating movement for a second part of the avatar (130). For example, at least one processor (307) may determine the movement of the second part of the avatar (130) as the same movement as the first part. For example, at least one processor (307) may determine the movement of the second part of the avatar (130) as a movement that is symmetrical to the movement of the first part. For example, at least one processor (307) may determine the movement of the second part of the avatar (130) as one of the movements of the first part.

[0092] Referring back to FIG. 10, in operation 1050, at least one processor (307) may generate fifth data regarding global movement of the avatar (130) using the third data and the fourth data. At least one processor (307) may obtain sixth data regarding movement of the wearable device (100) from at least one sensor. At least one processor (307) may further generate the fifth data using the sixth data. Operation 1050 may correspond to operation 450 of FIG. 4.

[0093] In operation 1060, at least one processor (307) may display an avatar (130) moving relative to an object through the display assembly (308) using the fifth data. Operation 1060 may correspond to operation 460 of FIG. 4.

[0094] For example, at least one processor (307) may identify whether a mode for correcting the movement of the avatar (130) is activated based on receiving first data among first data regarding the movement of the first control device and second data regarding the movement of the second control device. Identifying whether the mode is activated is described and illustrated in more detail with reference to FIG. 12.

[0095] Figure 12 is a flowchart illustrating an exemplary method for identifying whether a mode for compensating avatar movement is activated.

[0096] Referring to FIG. 12, in operation 1210, at least one processor (307) may identify whether a mode for correcting the movement of an avatar (130) displayed through a display assembly (308) is activated based on receiving first data among first data about the movement of the first control device (110) transmitted from the first control device (110) and second data about the movement of the second control device (120) transmitted from the second control device (120). The mode for correcting the movement of the avatar (130) may be described as a mode for generating global movement of the avatar (130) when there is no reception of the second data transmitted by the second control device (120). An exemplary operation in which at least one processor (307) receives data about the movement of the control device from the control device is described, but the embodiment is not limited thereto. At least one processor (307) can identify whether a mode for correcting the movement of an avatar (130) displayed through a display assembly (308) is activated based on acquiring data on the movement of a first part of the user's (200) body and data on a second part of the user's (200) body among images acquired through a camera assembly (309).

[0097] At least one processor (307) in operation 1220 can identify whether a mode for generating global movement of the avatar (130) is activated. At least one processor (307) can execute operation 1230 on a condition that the mode for generating global movement of the avatar (130) is activated, and can execute operation 1222 on a condition that the mode for generating global movement of the avatar (130) is deactivated.

[0098] In operation 1230, at least one processor (307) may generate the third data using the first data. Operation 1230 may correspond to operation 430. An exemplary operation in which at least one processor (307) receives data on movement of the control device from the control device is described, but the embodiment is not limited thereto. At least one processor (307) may identify whether a mode for correcting the movement of the avatar (130) displayed through the display assembly (308) is activated based on obtaining data on the movement of the first part of the user's (200) body among data on the movement of the first part of the user's (200) body and data on the second part of the user's (200) body using images acquired through the camera assembly (309).

[0099] In operation 1240, at least one processor (307) may generate fourth data for a second part of the avatar (130) using third data and information about an object associated with the avatar (130). The wearable device (100) may further include at least one sensor. The at least one processor (307) may obtain data about movement of the wearable device (100). The at least one processor (307) may further generate the fourth data using the data. The at least one processor (307) may further generate the fourth data using other information about the environment around the avatar (130). The at least one processor (307) may generate the fourth data by providing the third data and the information to a model trained through machine learning. The at least one processor (307) may generate the fourth data by providing the third data and the other information to a model trained through machine learning. At least one processor (307) may generate the fourth data by providing further data on the movement of the wearable device (100) to a model trained through machine learning. The model may be included in an external electronic device distinct from the wearable device (100). Operation 1240 may correspond to operation 440.

[0100] In operation 1250, at least one processor (307) may generate fifth data about the global movement of the avatar (130) using the third data and the fourth data. At least one processor (307) may further generate the fifth data using data about the movement of the wearable device (100). At least one processor (307) may generate the fifth data using inverse kinematics. Operation 1250 may correspond to operation 450.

[0101] In operation 1260, at least one processor (307) may display the avatar (130) moving with respect to the object using the fifth data. An exemplary operation based on the at least one processor (307) receiving data on the movement of the first control device (110) from the first control device (110) is described, but the embodiment is not limited thereto. The at least one processor (307) may obtain data on the movement of the first part of the user's (200) body among data on the second part of the user's (200) body and data on the first part of the user's (200) body using images acquired through the camera assembly (309). The at least one processor (307) may perform operations 1220 to 1260 based on identifying that a mode for compensating the movement of the avatar (130) displayed through the display assembly (308) is activated. Action 1260 can correspond to action 460.

[0102] In operation 1222, at least one processor (307) may generate the third data using the first data.

[0103] In operation 1224, at least one processor (307) may skip (or bypass) (or bypass) (or omit) generating the fourth data using the third data and the information.

[0104] At least one processor (307) in operation 1226 may use the third data to generate sixth data about the global movement of the avatar (130).

[0105] In operation 1228, at least one processor (307) may display the avatar (130) moving relative to the object using the sixth data.

[0106] The above descriptions may relate to operations of the wearable device (100) performed based on receiving the first data among the first and second data. However, this is merely exemplary. At least one processor (307) may also generate global movements of the avatar (130) based on receiving both the first and second data. Such operations are described and illustrated in more detail with reference to FIG. 13.

[0107] FIG. 13 is a flowchart illustrating an exemplary method for generating global movement of an avatar using data received from a first control device and a second control device.

[0108] Referring to FIG. 13, in operation 1310, at least one processor (307) may display an avatar (130) including a first part controllable by a first control device (110) and a second part controllable by a second control device (120) paired with the first control device (110) and an object related to the avatar (130) through a display assembly (308). Operation 1310 may correspond to operation 410.

[0109] In operation 1320, at least one processor (307) may receive, through a communication circuit (305), first data on movement of the first control device (110) transmitted from the first control device (110) and data on movement of the second control device (120) transmitted from the second control device (120) while displaying the avatar (130) and the object. At least one processor (307) may be connected to the first control device (110) and the second control device (120).

[0110] In operation 1330, at least one processor (307) may generate third data for the first part of the avatar (130) using the first data based on the reception.

[0111] In operation 1340, at least one processor (307) may generate fourth data for the second part of the avatar (130) based on the reception using the second data. Since at least one processor (307) receives the second data from the second control device (120), it may refrain from using the model (710) to generate the fourth data.

[0112] In operation 1350, at least one processor (307) may generate fifth data about the global movement of the avatar (130) using the third data and the fourth data. For example, at least one processor (307) may generate the fifth data using inverse kinematics.

[0113] In operation 1360, at least one processor (307) may display the avatar (130) through the display assembly (308) using the fifth data.

[0114] Figure 14 is a flowchart illustrating an exemplary method for generating global motion of an avatar using images acquired through a camera assembly.

[0115] In operation 1410, at least one processor (307) may display an avatar (130) including a first part controllable by a first part of a body of a user (200) and a second part controllable by a second part of the body of the user (200) paired with the first part of the body of the user (200), and an object related to the avatar (130), through a display assembly (308). For example, the first part and the second part of the body may be a hand of the user (200). For example, the user (200) may control the movement of the avatar (130) using the hand of the user (200).

[0116] In operation 1420, at least one processor (307) may acquire first data among first data about movement of the first part of the body of the user (200) and second data about movement of the second part of the body of the user (200) using images acquired through the camera assembly (309) while displaying the avatar (130) and the object. For example, the second part of the body of the user (200) may be outside a range that the camera assembly (309) can capture. For example, the second part of the body may be outside the field of view of the user (200). For example, the second part of the body may be present in a place where the camera assembly (309) cannot capture the second part of the body. For example, the second part of the body may be obscured by an obstacle.

[0117] In operation 1430, at least one processor (307) may generate third data for the first part of the avatar (130) moving with respect to the object using the first data based on the acquisition.

[0118] In operation 1440, at least one processor (307) may generate fourth data (750) about the second part of the avatar (130) moving with respect to the object using the third data (720) and the information about the object (730). The wearable device (100) may further include at least one sensor. The at least one processor (307) may obtain data about the movement of the wearable device (100). The at least one processor (307) may further generate the fourth data (750) using the data about the movement of the wearable device (100). The at least one processor (307) may further generate the fourth data (750) using other information about the environment around the avatar (130). At least one processor (307) can generate the fourth data (750) by providing the third data (720) and information about the object (730) to a model (710) trained through machine learning. The model (710) can be stored in the memory (306). At least one processor (307) can generate the fourth data (750) by providing the third data (720) and other information about the environment around the avatar (130) to the model (710) trained through machine learning. At least one processor (307) can generate the fourth data by further providing data about the movement of the wearable device (100) to the model (710) trained through machine learning. The model can be included in an external electronic device distinct from the wearable device (100). At least one processor (307) may transmit the third data (720), information about the object (730), and other information about the environment surrounding the avatar to the external electronic device. At least one processor (307) may further transmit data about the movement of the wearable device (100) to the external electronic device.At least one processor (307) can receive the fourth data (750) from the external electronic device.

[0119] In operation 1450, at least one processor (307) may generate fifth data about global movement of the avatar (130) using the third data and the fourth data. At least one processor (307) may further generate the fifth data using data about movement of the wearable device (100). At least one processor (307) may generate the fifth data using inverse kinematics on the third data (720) and the fourth data (750). Operation 1450 may correspond to operation 450.

[0120] In operation 1460, at least one processor (307) can display the avatar (130) moving with respect to the object through the display assembly (308) using the fifth data.

[0121] FIG. 15 is a block diagram of an electronic device within a network environment according to various embodiments.

[0122] Referring to FIG. 15, in a network environment (1500), an electronic device (1501) may communicate with an electronic device (1502) via a first network (1598) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (1504) or a server (1508) via a second network (1599) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (1501) may communicate with the electronic device (1504) via the server (1508). According to one embodiment, the electronic device (1501) may include a processor (1520), a memory (1530), an input module (1550), an audio output module (1555), a display module (1560), an audio module (1570), a sensor module (1576), an interface (1577), a connection terminal (1578), a haptic module (1579), a camera module (1580), a power management module (1588), a battery (1589), a communication module (1590), a subscriber identification module (1596), or an antenna module (1597). In some embodiments, the electronic device (1501) may omit at least one of these components (e.g., the connection terminal (1578)), or may have one or more other components added. In some embodiments, some of these components (e.g., sensor module (1576), camera module (1580), or antenna module (1597)) may be integrated into a single component (e.g., display module (1560)).

[0123] The processor (1520) may, for example, execute software (e.g., a program (1540)) to control at least one other component (e.g., a hardware or software component) of the electronic device (1501) connected to the processor (1520) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (1520) may store commands or data received from other components (e.g., a sensor module (1576) or a communication module (1590)) in a volatile memory (1532), process the commands or data stored in the volatile memory (1532), and store result data in a non-volatile memory (1534). According to one embodiment, the processor (1520) may include a main processor (1521) (e.g., a central processing unit or an application processor) or an auxiliary processor (1523) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (1521). For example, when the electronic device (1501) includes the main processor (1521) and the auxiliary processor (1523), the auxiliary processor (1523) may be configured to use less power than the main processor (1521) or to be specialized for a given function. The auxiliary processor (1523) may be implemented separately from the main processor (1521) or as a part thereof.

[0124] The auxiliary processor (1523) may control at least a portion of functions or states associated with at least one component (e.g., the display module (1560), the sensor module (1576), or the communication module (1590)) of the electronic device (1501), for example, on behalf of the main processor (1521) while the main processor (1521) is in an inactive (e.g., sleep) state, or together with the main processor (1521) while the main processor (1521) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (1523) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (1580) or a communication module (1590)). In one embodiment, the auxiliary processor (1523) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (1501) where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (1508)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0125] The memory (1530) can store various data used by at least one component (e.g., the processor (1520) or the sensor module (1576)) of the electronic device (1501). The data can include, for example, software (e.g., the program (1540)) and input data or output data for commands related thereto. The memory (1530) can include volatile memory (1532) or non-volatile memory (1534).

[0126] The program (1540) may be stored as software in memory (1530) and may include, for example, an operating system (1542), middleware (1544), or an application (1546).

[0127] The input module (1550) can receive commands or data to be used in a component of the electronic device (1501) (e.g., a processor (1520)) from an external source (e.g., a user) of the electronic device (1501). The input module (1550) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0128] The audio output module (1555) can output audio signals to the outside of the electronic device (1501). The audio output module (1555) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0129] The display module (1560) can visually provide information to an external party (e.g., a user) of the electronic device (1501). The display module (1560) may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling the device. In one embodiment, the display module (1560) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0130] The audio module (1570) can convert sound into an electrical signal, or vice versa. According to one embodiment, the audio module (1570) can acquire sound through the input module (1550), output sound through the sound output module (1555), or an external electronic device (e.g., electronic device (1502)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (1501).

[0131] The sensor module (1576) can detect the operating status (e.g., power or temperature) of the electronic device (1501) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (1576) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0132] The interface (1577) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (1501) with an external electronic device (e.g., the electronic device (1502)). In one embodiment, the interface (1577) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0133] The connection terminal (1578) may include a connector through which the electronic device (1501) may be physically connected to an external electronic device (e.g., the electronic device (1502)). In one embodiment, the connection terminal (1578) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0134] The haptic module (1579) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (1579) may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0135] The camera module (1580) can capture still images and videos. In one embodiment, the camera module (1580) may include one or more lenses, image sensors, image signal processors, or flashes.

[0136] The power management module (1588) can manage the power supplied to the electronic device (1501). According to one embodiment, the power management module (1588) can be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

[0137] A battery (1589) may power at least one component of the electronic device (1501). In one embodiment, the battery (1589) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0138] The communication module (1590) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (1501) and an external electronic device (e.g., electronic device (1502), electronic device (1504), or server (1508)), and the performance of communication through the established communication channel. The communication module (1590) may operate independently from the processor (1520) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1590) may include a wireless communication module (1592) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (1594) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device (1504) via a first network (1598) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (1599) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network or a wide area network)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1592) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (1596) to verify or authenticate the electronic device (1501) within a communication network such as the first network (1598) or the second network (1599).

[0139] The wireless communication module (1592) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency communications (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (1592) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (1592) can support various technologies for securing performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (1592) can support various requirements specified in the electronic device (1501), an external electronic device (e.g., the electronic device (1504)), or a network system (e.g., the second network (1599)). According to one embodiment, the wireless communication module (1592) may support a peak data rate (e.g., 20 Gbps or more) for eMBB implementation, a loss coverage (e.g., 164 dB or less) for mMTC implementation, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC implementation.

[0140] The antenna module (1597) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (1597) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (1597) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (1598) or the second network (1599), may be selected from the plurality of antennas by, for example, the communication module (1590). A signal or power may be transmitted or received between the communication module (1590) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (1597).

[0141] According to various embodiments, the antenna module (1597) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0142] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0143] According to one embodiment, commands or data may be transmitted or received between the electronic device (1501) and an external electronic device (1504) via a server (1508) connected to a second network (1599). Each of the external electronic devices (1502 or 1504) may be the same or a different type of device as the electronic device (1501). According to one embodiment, all or part of the operations executed in the electronic device (1501) may be executed in one or more of the external electronic devices (1502, 1504, or 1508). For example, when the electronic device (1501) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (1501) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (1501). The electronic device (1501) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (1501) may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (1504) may include an Internet of Things (IoT) device. The server (1508) may be an intelligent server utilizing machine learning and / or a neural network.According to one embodiment, an external electronic device (1504) or server (1508) may be included within the second network (1599). The electronic device (1501) may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology and IoT-related technology.

[0144] FIG. 16A illustrates an example of a perspective view of a wearable device. FIG. 16B illustrates an example of one or more hardware components arranged within the wearable device. The wearable device (1501) may be an example of the wearable device (100). According to one embodiment, the wearable device (100) may have the form of glasses that are wearable on a body part (e.g., head) of a user. The wearable device (100) of FIGS. 16A and 16B may be an example of the wearable device (100) of FIG. 15. The wearable device (100) may include a head-mounted display (HMD). For example, the housing of the wearable device (100) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (100) may include one or more straps that can be twined around the user's head, and / or one or more temples that can be attached to the ears of the head.

[0145] Referring to FIG. 16A, according to one embodiment, a wearable device (100) may include at least one display (1650) and a frame (1600) supporting at least one display (1650).

[0146] According to one embodiment, the wearable device (100) can be worn on a part of a user's body. The wearable device (100) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to the user wearing the wearable device (100). For example, the wearable device (100) can display a virtual reality image provided from at least one optical device (1682, 1684) of FIG. 16B on at least one display (1650) in response to a user's designated gesture acquired through the motion recognition cameras (1660-2, 1660-3) of FIG. 16B.

[0147] According to one embodiment, at least one display (1650) may provide visual information to a user. For example, at least one display (1650) may include a transparent or translucent lens. At least one display (1650) may include a first display (1650-1) and / or a second display (1650-2) spaced apart from the first display (1650-1). For example, the first display (1650-1) and the second display (1650-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0148] Referring to FIG. 16B, at least one display (1650) can provide visual information transmitted from external light to the user through a lens included in the at least one display (1650), and other visual information distinct from the visual information. The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (1650) can include a first surface (1631) and a second surface (1632) opposite to the first surface (1631). A display area can be formed on the second surface (1632) of the at least one display (1650). When a user wears the wearable device (100), external light can be transmitted to the user by being incident on the first surface (1631) and transmitted through the second surface (1632). As another example, at least one display (1650) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (1682, 1684) on a real screen transmitted through external light, in a display area formed on the second surface (1632).

[0149] In one embodiment, at least one display (1650) may include at least one waveguide (1633, 1634) that diffracts light emitted from at least one optical device (1682, 1684) and transmits the diffracted light to a user. The at least one waveguide (1633, 1634) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (1633, 1634). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (1633, 1634) may be propagated to the other end of the at least one waveguide (1633, 1634) by the nano-pattern. At least one waveguide (1633, 1634) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (1633, 1634) may be arranged within the wearable device (100) to guide a screen displayed by at least one display (1650) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (1633, 1634).

[0150] The wearable device (100) can analyze an object included in a real image collected through a shooting camera (1660-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (1650). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (100) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (100) can execute spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using a multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (100) can view an image displayed on at least one display (1650).

[0151] According to one embodiment, the frame (1600) may be formed as a physical structure that allows the wearable device (100) to be worn on the user's body. According to one embodiment, the frame (1600) may be configured so that, when the user wears the wearable device (100), the first display (1650-1) and the second display (1650-2) can be positioned corresponding to the user's left and right eyes. The frame (1600) may support at least one display (1650). For example, the frame (1600) may support the first display (1650-1) and the second display (1650-2) to be positioned corresponding to the user's left and right eyes.

[0152] Referring to FIG. 16A, the frame (1600) may include a region (1620) that at least partially contacts a portion of the user's body when the user wears the wearable device (100). For example, the region (1620) of the frame (1600) that contacts a portion of the user's body may include a region that contacts a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (100) makes contact with. According to one embodiment, the frame (1600) may include a nose pad (1610) that contacts a portion of the user's body. When the wearable device (100) is worn by the user, the nose pad (1610) may contact a portion of the user's nose. The frame (1600) may include a first temple (1604) and a second temple (1605) that contact another part of the user's body that is distinct from the part of the user's body.

[0153] For example, the frame (1600) may include a first rim (1601) that surrounds at least a portion of the first display (1650-1), a second rim (1602) that surrounds at least a portion of the second display (1650-2), a bridge (1603) that is disposed between the first rim (1601) and the second rim (1602), a first pad (1611) that is disposed along a portion of the edge of the first rim (1601) from one end of the bridge (1603), a second pad (1612) that is disposed along a portion of the edge of the second rim (1602) from the other end of the bridge (1603), a first temple (1604) that extends from the first rim (1601) and is fixed to a portion of the ear of the wearer, and a second temple (1605) that extends from the second rim (1602) and is fixed to a portion of the ear opposite the ear. There are. The first pad (1611) and the second pad (1612) can be in contact with a part of the user's nose, and the first temple (1604) and the second temple (1605) can be in contact with a part of the user's face and a part of the user's ear. The temples (1604, 1605) can be rotatably connected to the rim through the hinge units (1606, 1607) of FIG. 16B. The first temple (1604) can be rotatably connected to the first rim (1601) through the first hinge unit (1606) disposed between the first rim (1601) and the first temple (1604). The second temple (1605) may be rotatably connected to the second rim (1602) via a second hinge unit (1607) disposed between the second rim (1602) and the second temple (1605). In one embodiment, the wearable device (100) may use a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (1600) to identify an external object (e.g., a user's fingertip) touching the frame (1600) and / or a gesture performed by the external object.

[0154] According to one embodiment, the wearable device (100) may include hardwares that perform various functions (e.g., hardwares to be described later based on the block diagram of FIG. 4). For example, the hardwares may include a battery module (1670), an antenna module (1675), at least one optical device (1682, 1684), speakers (e.g., speakers 1655-1, 1655-2), a microphone (e.g., microphones 1665-1, 1665-2, 1665-3), a light-emitting module (not shown), and / or a printed circuit board (PCB) (1690) (e.g., a printed circuit board). The various hardwares may be arranged within the frame (1600).

[0155] According to one embodiment, microphones (e.g., microphones 1665-1, 1665-2, 1665-3) of the wearable device (100) may be disposed on at least a portion of the frame (1600) to acquire sound signals. A first microphone (1665-1) disposed on the bridge (1603), a second microphone (1665-2) disposed on the second rim (1602), and a third microphone (1665-3) disposed on the first rim (1601) are illustrated in FIG. 16B, but the number and arrangement of the microphones (1665) are not limited to the embodiment of FIG. 16B. When the number of microphones (1665) included in the wearable device (100) is two or more, the wearable device (100) can identify the direction of a sound signal by using a plurality of microphones placed on different parts of the frame (1600).

[0156] According to one embodiment, at least one optical device (1682, 1684) may project a virtual object onto at least one display (1650) to provide various image information to a user. For example, at least one optical device (1682, 1684) may be a projector. At least one optical device (1682, 1684) may be disposed adjacent to at least one display (1650) or may be included within at least one display (1650) as a part of at least one display (1650). According to one embodiment, the wearable device (100) may include a first optical device (1682) corresponding to a first display (1650-1) and a second optical device (1684) corresponding to a second display (1650-2). For example, at least one optical device (1682, 1684) may include a first optical device (1682) positioned at an edge of a first display (1650-1) and a second optical device (1684) positioned at an edge of a second display (1650-2). The first optical device (1682) may transmit light to a first waveguide (1633) positioned on the first display (1650-1), and the second optical device (1684) may transmit light to a second waveguide (1634) positioned on the second display (1650-2).

[0157] In one embodiment, the camera (1660) may include a recording camera (1660-4), an eye tracking camera (ET CAM) (1660-1), and / or a motion recognition camera (1660-2, 1660-3). The recording camera (1660-4), the eye tracking camera (1660-1), and the motion recognition cameras (1660-2, 1660-3) may be positioned at different locations on the frame (1600) and may perform different functions. The eye tracking camera (1660-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (100). For example, the wearable device (100) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (1660-1). The wearable device (100) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (1660-1). The wearable device (100) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (100) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (1660-1). The wearable device (100) can render an image (or screen) displayed on at least one display (1650) based on the position of the user's eyes. For example, the visual quality of a first region related to the gaze within the image and the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI) of a second region distinguished from the first region may be different from each other.The wearable device (100) can obtain an image having a visual quality of a first area matching the user's gaze and a visual quality of a second area using foveated rendering. For example, if the wearable device (100) supports an iris recognition function, user authentication can be performed based on iris information obtained using a gaze tracking camera (1660-1). An example in which the gaze tracking camera (1660-1) is positioned toward the user's right eye is illustrated in FIG. 16B, but the embodiment is not limited thereto, and the gaze tracking camera (1660-1) can be positioned solely toward the user's left eye, or toward both eyes.

[0158] In one embodiment, the capturing camera (1660-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (1660-4) can be used to obtain a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (1660-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (1650). The at least one display (1650) can display a single image in which information about an actual image or background including an image of the specific object obtained using the capturing camera (1660-4) and a virtual image provided through at least one optical device (1682, 1684) are superimposed. The wearable device (100) can compensate for depth information (e.g., the distance between the wearable device (100) and an external object acquired through a depth sensor) using an image acquired through the capture camera (1660-4). The wearable device (100) can perform object recognition using an image acquired using the capture camera (1660-4). The wearable device (100) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the capture camera (1660-4). The wearable device (100) can perform a pass-through function to display an image acquired through the capture camera (1660-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (1650) while displaying the screen. In one embodiment, the capturing camera (1660-4) may be positioned on a bridge (1603) positioned between the first rim (1601) and the second rim (1602).

[0159] The gaze tracking camera (1660-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (100) and matching the user's gaze with visual information provided to at least one display (1650). For example, when the wearable device (100) looks straight ahead, the wearable device (100) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (1650). The gaze tracking camera (1660-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (1660-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (1660-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (1660-1) may be positioned within the first rim (1601) and / or the second rim (1602) to face the direction in which the user wearing the wearable device (100) is positioned.

[0160] The gesture recognition cameras (1660-2, 1660-3) can recognize the movement of the user's entire body, such as the user's torso, hands, or face, or a part of the body, and thereby provide a specific event on a screen provided on at least one display (1650). The gesture recognition cameras (1660-2, 1660-3) can recognize the user's gesture (gesture recognition), obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (1650). The processor can identify the signal corresponding to the gesture, and perform a designated function based on the identification. The gesture recognition cameras (1660-2, 1660-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 dof pose). The processor may perform gesture recognition and / or object tracking functions using the motion recognition cameras (1660-2, 1660-3). In one embodiment, the motion recognition cameras (1660-2, 1660-3) may be positioned on the first rim (1601) and / or the second rim (1602).

[0161] The camera (1660) included in the wearable device (100) is not limited to the above-described gaze tracking camera (1660-1) and motion recognition cameras (1660-2, 1660-3). For example, the wearable device (100) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (100) can identify an external object based on a sensor for identifying the distance between the wearable device (100) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (1660) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (100) may include a camera (1660) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (100).

[0162] Although not shown, in one embodiment, the wearable device (100) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being captured using the camera (1660). The light source may include an infrared wavelength LED. The light source may be disposed in at least one of the frame (1600) and the hinge units (1606, 1607).

[0163] According to one embodiment, the battery module (1670) may supply power to the electronic components of the wearable device (100). In one embodiment, the battery module (1670) may be disposed within the first temple (1604) and / or the second temple (1605). For example, the battery module (1670) may be a plurality of battery modules (1670). The plurality of battery modules (1670) may be disposed within each of the first temple (1604) and the second temple (1605). In one embodiment, the battery module (1670) may be disposed at an end of the first temple (1604) and / or the second temple (1605).

[0164] The antenna module (1675) can transmit signals or power to the outside of the wearable device (100), or receive signals or power from the outside. In one embodiment, the antenna module (1675) can be positioned within the first temple (1604) and / or the second temple (1605). For example, the antenna module (1675) can be positioned close to one surface of the first temple (1604) and / or the second temple (1605).

[0165] The speaker (1655) can output an audio signal to the outside of the wearable device (100). The audio output module may be referred to as a speaker. In one embodiment, the speaker (1655) may be positioned within the first temple (1604) and / or the second temple (1605) so as to be positioned adjacent to the ear of a user wearing the wearable device (100). For example, the speaker (1655) may include a second speaker (1655-2) positioned within the first temple (1604) and thus adjacent to the user's left ear, and a first speaker (1655-1) positioned within the second temple (1605) and thus adjacent to the user's right ear.

[0166] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (100) to the user. For example, when the wearable device (100) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (1601) and / or the second rim (1602).

[0167] Referring to FIG. 16B, according to one embodiment, a wearable device (100) may include a printed circuit board (PCB) (1690). The PCB (1690) may be included in at least one of the first temple (1604) or the second temple (1605). The PCB (1690) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (100) (e.g., hardwares illustrated by different blocks in FIG. 4) may be disposed on the PCB (1690). The wearable device (100) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0168] According to one embodiment, a wearable device (100) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (100) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (100). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (100) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (100) based on the IMU.

[0169] FIGS. 17A and 17B illustrate an example of an exterior appearance of a wearable device (e.g., a wearable device (100)). The wearable device (100) of FIGS. 17A and 17B may be an example of the wearable device (100) of FIG. 15. According to one embodiment, an example of an exterior appearance of a first side (1710) of a housing of a wearable device (100) is illustrated in FIG. 17A, and an example of an exterior appearance of a second side (1720) opposite to the first side (1710) may be illustrated in FIG. 17B.

[0170] Referring to FIG. 17A, according to one embodiment, a first surface (1710) of a wearable device (100) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (100) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (1604) and / or the second temple (1605) of FIGS. 16A and 16B). A first display (1650-1) for outputting an image to a left eye among the user's two eyes, and a second display (1650-2) for outputting an image to a right eye among the two eyes, may be disposed on the first surface (1710). The wearable device (100) may be formed on the first surface (1710) and may further include a rubber or silicone packing to prevent interference from light (e.g., ambient light) different from the light emitted from the first display (1650-1) and the second display (1650-2).

[0171] According to one embodiment, the wearable device (100) may include cameras (1660-1) for photographing and / or tracking both eyes of the user adjacent to each of the first display (1650-1) and the second display (1650-2). The cameras (1660-1) may be referred to as the gaze tracking camera (1660-1) of FIG. 16B. According to one embodiment, the wearable device (100) may include cameras (1660-5, 1660-6) for photographing and / or recognizing the face of the user. The cameras (1660-5, 1660-6) may be referred to as FT cameras. The wearable device (100) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (1660-5, 1660-6). For example, the wearable device (100) may change the texture and / or shape of a portion of an avatar (e.g., a portion of an avatar representing a human face) using information obtained by cameras (1660-5, 1660-6) (e.g., FT cameras) and representing a facial expression of a user wearing the wearable device (100).

[0172] Referring to FIG. 17b, a camera (e.g., cameras (1660-7, 1660-8, 1660-9, 1660-10, 1660-11, 1660-12)) and / or a sensor (e.g., a depth sensor (1730)) for obtaining information related to the external environment of the wearable device (100) may be disposed on a second surface (1720) opposite to the first surface (1710) of FIG. 17a. For example, the cameras (1660-7, 1660-8, 1660-9, 1660-10) may be disposed on the second surface (1720) for recognizing external objects. Cameras (1660-7, 1660-8, 1660-9, 1660-10) may be referenced to the motion recognition cameras (1660-2, 1660-3) of FIG. 16b.

[0173] For example, using cameras (1660-11, 1660-12), the wearable device (100) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (1660-11) can be placed on the second face (1720) of the wearable device (100) to obtain an image to be displayed through the second display (1650-2) corresponding to the right eye among the two eyes. The camera (1660-12) can be placed on the second face (1720) of the wearable device (100) to obtain an image to be displayed through the first display (1650-1) corresponding to the left eye among the two eyes. The cameras (1660-11, 1660-12) can be referred to as the shooting camera (1660-4) of FIG. 16B.

[0174] According to one embodiment, the wearable device (100) may include a depth sensor (1730) disposed on the second face (1720) to identify a distance between the wearable device (100) and an external object. Using the depth sensor (1730), the wearable device (100) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (100). Although not illustrated, a microphone may be disposed on the second face (1720) of the wearable device (100) to obtain a sound output from an external object. The number of microphones may be one or more depending on the embodiment.

[0175] Hereinafter, with reference to FIG. 18, the hardware or software configuration of the wearable device (100) is described.

[0176] Fig. 18 illustrates an example of a block diagram of a wearable device (e.g., a wearable device (100)). The wearable device (100) of Fig. 18 may be an example of the electronic device (1501) of Fig. 15 or the wearable device (100) of Figs. 16A to 17B.

[0177] Referring to FIG. 18, a wearable device (100) according to one embodiment may include a processor (1810), a memory (1815), a display (1650) (e.g., the first display (1650-1) and / or the second display (1650-2) of FIGS. 16A, 16B, 17A, and 17B), and / or a sensor (1820). The processor (1810), the memory (1815), the display (1650), and / or the sensor (1820) may be electrically and / or operatively connected to each other by electronic components such as a communication bus (1802). In the present disclosure, the operative connection of the electronic components may include a direct connection established between the electronic components and / or an indirect connection established between the electronic components such that a first electronic component among the electronic components is controlled by a second electronic component among the electronic components. The type and / or number of electronic components included in the wearable device (100) is not limited to those illustrated in FIG. 18. For example, the wearable device (100) may include only some of the electronic components illustrated in FIG. 18.

[0178] A processor (1810) of a wearable device (100) according to one embodiment may include a circuit (e.g., a processing circuit) for processing data based on one or more instructions. The circuit for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In one embodiment, the wearable device (100) may include one or more processors. The processor (1810) may have a multi-core processor structure such as a dual core, a quad core, a hexa core, and / or an octa core. The multi-core processor structure of the processor (1810) may include a structure (e.g., a big-little structure) based on a plurality of core circuits that are distinguished by power consumption, clock, and / or calculation amount per unit time. In one embodiment comprising a processor (1810) having a multi-core processor architecture, the operations and / or functions of the present disclosure may be performed individually or collectively by one or more cores included in the processor (1810).

[0179] A memory (1815) of a wearable device (100) according to one embodiment may include electronic components for storing data and / or instructions input to and / or output from a processor (1810). The memory (1815) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multi media card (eMMC). In one embodiment, memory (1815) may be referred to as storage.

[0180] In one embodiment, a display (1650) of a wearable device (100) can output visualized information to a user of the wearable device (100). The display (1650), which is arranged in front of the eyes of a user wearing the wearable device (100), can be arranged on at least a portion of a housing of the wearable device (100) (e.g., the first display (1650-1) and / or the second display (1650-2) of FIGS. 16A, 16B, 17A, and 17B). For example, the display (1650) can be controlled by a processor (1810) including circuits such as a CPU, a GPU (graphics processing unit), and / or a DPU (display processing unit), to output visualized information to the user. The display (1650) can include a flexible display, a flat panel display (FPD), and / or electronic paper. The display (1650) may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs may include organic LEDs (OLEDs). Embodiments are not limited thereto, and for example, if the wearable device (100) includes a lens for transmitting external light (or ambient light), the display (1650) may include a projector (or projection assembly) for projecting light onto the lens. In one embodiment, the display (1650) may be referred to as a display panel and / or a display module. The pixels included in the display (1650) may be arranged to face one of the user's eyes when the wearable device (100) is worn by the user.For example, the display (1650) may include display areas (or active areas) corresponding to each of the user's two eyes.

[0181] In one embodiment, the sensor (1820) of the wearable device (100) may generate electrical information that may be processed by the processor (1810) and / or the memory (1815) from non-electronic information related to the wearable device (100). For example, the sensor (1820) may include a global positioning system (GPS) sensor for detecting the geographic location of the wearable device (100). In addition to the GPS method, the sensor (1820) may generate information indicating the geographic location of the wearable device (100) based on a global navigation satellite system (GNSS) such as, for example, Galileo or Beidou (compass). The above information may be stored in memory (1815), processed by processor (1810), and / or transmitted to another electronic device distinct from the wearable device (100) via communication circuitry.

[0182] According to one embodiment, one or more instructions (or commands) representing data to be processed, calculations to be performed, and / or operations to be performed by the processor (1810) of the wearable device (100) may be stored in the memory (1815) of the wearable device (100). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or a software application (hereinafter, “application”). For example, the wearable device (100) and / or the processor (1810) may perform at least one of the operations of FIGS. 4, 10, 12, 13, and 14 when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, a program, and / or a software application is executed. Hereinafter, the fact that a software application is installed in a wearable device (100) may mean that one or more instructions provided in the form of a software application (or package) are stored in a memory (1815), and that the one or more applications are stored in a format executable by the processor (1810) (e.g., a file having an extension specified by the operating system of the wearable device (100)). For example, the application may include a program and / or a library related to a service provided to a user.

[0183] Referring to FIG. 18, programs installed in the wearable device (100) may be included in any one of different layers, including an application layer (1840), a framework layer (1850), and / or a hardware abstraction layer (HAL) (1880), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware (e.g., the display (1650), and / or the sensor (1820)) of the wearable device (100) may be included in the hardware abstraction layer (1880). The framework layer (1850) may be referred to as an XR framework layer from the perspective of including one or more programs for providing an XR (extended reality) service. For example, the layers illustrated in FIG. 18 may be logically (or for convenience of explanation) separated, and may not mean that the address space of the memory (1815) is separated by the layers.

[0184] For example, within the framework layer (1850), programs designed to target at least one of the hardware abstraction layer (1880) and / or the application layer (1840) (e.g., a position tracker (1871), a space recognizer (1872), a gesture tracker (1873), an eye-gaze tracker (1874), and / or a face tracker (1875)) may be included. The programs included in the framework layer (1850) may provide an application programming interface (API) that is executable (or callable) based on other programs.

[0185] For example, a program designed to target users of a wearable device (100) may be included within the application layer (1840). As an example of programs included in the application layer (1840), an extended reality (XR) system user interface (UI) (1841) and / or an XR application (1842) are exemplified, but the embodiment is not limited thereto. For example, programs (e.g., software applications) included in the application layer (1840) may call an API to cause execution of functions supported by programs included in the framework layer (1850).

[0186] For example, the wearable device (100) may display one or more visual objects on the display (1650) for performing interaction with the user based on the execution of the XR system UI (1841). A visual object may refer to an object that can be placed within a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (100) may provide the user with functions available within a virtual space based on the execution of the XR system UI (1841).

[0187] Referring to FIG. 18, a lightweight renderer (1843) and / or an XR plug-in (1844) are illustrated to be included within the XR system UI (1841), but are not limited thereto. For example, based on the XR system UI (1841), the processor (1810) may execute a lightweight renderer (1843) and / or an XR plug-in (1844) within the framework layer (1850).

[0188] For example, the wearable device (100) may acquire resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (1843). The lightweight renderer (1843) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline that allows partial changes. The lightweight renderer (1843) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (100) may acquire resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (1844). The XR plugin (1844) can be referred to as an open XR native client from the perspective of defining (or configuring) the entire rendering pipeline.

[0189] For example, the wearable device (100) may display a screen representing at least a portion of a virtual space on the display (1650) based on the execution of the XR application (1842). The XR plug-in (1844-1) included in the XR application (1842) may include instructions that support functions similar to those of the XR plug-in (1844) of the XR system UI (1841). Descriptions of the XR plug-in (1844-1) that overlap with those of the XR plug-in (1844) may be omitted. The wearable device (100) may cause the execution of the virtual space manager (1851) based on the execution of the XR application (1842).

[0190] For example, the wearable device (100) may display an image on the display (1650) in a virtual space based on the execution of the application (1845). The application (1845) may be configured to output image information for displaying a two-dimensional image. The wearable device (100) may cause the execution of the virtual space manager (1851) based on the execution of the application (1845). The wearable device (100) may generate dual image information to display the two-dimensional image in a three-dimensional virtual space based on the execution of the application (1845). Here, the dual image information may include first image information for the left eye and second image information for the right eye, taking into account binocular disparity. In order to display the two-dimensional image in the three-dimensional virtual space, the wearable device (100) may generate the dual image information based on the image information for displaying the two-dimensional image.

[0191] According to one embodiment, the wearable device (100) may provide a virtual space service based on the execution of the virtual space manager (1851). For example, the virtual space manager (1851) may include a platform for supporting the virtual space service. Based on the execution of the virtual space manager (1851), the wearable device (100) may identify a virtual space formed based on the user's location indicated by data acquired through the sensor (1830), and may display at least a portion of the virtual space on the display (1650). The virtual space manager (1851) may be referred to as a composition presentation manager (CPM).

[0192] For example, the virtual space manager (1851) may include a runtime service (1852). As an example, the runtime service (1852) may be referred to as an OpenXR runtime module (or an OpenXR runtime program). The wearable device (100) may execute at least one of a user's pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (1852). As an example, the wearable device (100) may perform rendering for a virtual space service for the user based on the execution of the runtime service (1852). For example, a function related to a virtual space, executable by the application layer (1840), may be supported based on the execution of the runtime service (1852).

[0193] For example, the virtual space manager (1851) may include a pass-through manager (1853). Based on the execution of the pass-through manager (1853), the wearable device (100) may display an image and / or video representing an actual space acquired through an external camera on at least a portion of the screen while displaying a screen representing a virtual space on the display (1650).

[0194] For example, the virtual space manager (1851) may include an input manager (1854). The wearable device (100) may identify data (e.g., sensor data) acquired by executing one or more programs included in the recognition service layer (1870) based on the execution of the input manager (1854). The wearable device (100) may use the acquired data to identify user input related to the wearable device (100). The user input may be related to a motion (e.g., a hand gesture), gaze, and / or speech of the user identified by a sensor (1820) (e.g., an image sensor (1830) such as an external camera). The user input may be identified based on an external electronic device connected (or paired) via a communication circuit.

[0195] For example, the perception abstract layer (1860) can be used for data exchange between the virtual space manager (1851) and the perception service layer (1870). From the perspective of being used for data exchange between the virtual space manager (1851) and the perception service layer (1870), the perception abstract layer (1860) can be referred to as an interface. For example, the perception abstract layer (1860) can be referenced as OpenPX. The perception abstract layer (1860) can be used for a perception client and a perception service.

[0196] According to one embodiment, the recognition service layer (1870) may include one or more programs for processing data acquired from the sensor (1820). The one or more programs may include at least one of a position tracker (1871), a space recognizer (1872), a gesture tracker (1873), and / or an eye tracker (1874). The type and / or number of the one or more programs included in the recognition service layer (1870) are not limited to those illustrated in FIG. 18.

[0197] For example, the wearable device (100) can identify the pose of the wearable device (100) using the sensor (1830) based on the execution of the position tracker (1871). The wearable device (100) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (100) using data acquired using an external camera (e.g., an image sensor (1821)) and / or an IMU (e.g., a motion sensor (1822) including a gyro sensor, an acceleration sensor, and / or a geomagnetic sensor) based on the execution of the position tracker (1871). The position tracker (1871) may be referred to as a head tracking (HeT) module (or head tracker, head tracking program).

[0198] For example, the wearable device (100) may obtain information for providing a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of the wearable device (100) (or the user of the wearable device (100)) based on the execution of the space recognizer (1872). The wearable device (100) may reproduce the surrounding environment of the wearable device (100) in three dimensions using data obtained using an external camera (e.g., an image sensor (1821)) based on the execution of the space recognizer (1872). The wearable device (100) may identify at least one of a plane, a slope, and stairs based on the surrounding environment of the wearable device (100) reproduced in three dimensions based on the execution of the space recognizer (1872). The space recognizer (1872) may be referred to as a scene understanding (SU) module (or a scene recognition program).

[0199] For example, the wearable device (100) may identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (100) based on the execution of the gesture tracker (1873). As an example, the wearable device (100) may identify a pose and / or gesture of a hand of a user using data acquired from an external camera (e.g., an image sensor (1821)) based on the execution of the gesture tracker (1873). As an example, the wearable device (100) may identify a pose and / or gesture of a hand of a user based on data (or images) acquired using an external camera based on the execution of the gesture tracker (1873). The gesture tracker (1873) may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or a gesture tracking module.

[0200] For example, the wearable device (100) may identify (or track) eye movements of a user of the wearable device (100) based on the execution of the gaze tracker (1874). As an example, the wearable device (100) may identify eye movements of the user using data acquired from a gaze tracking camera (e.g., an image sensor (1821)) based on the execution of the gaze tracker (1874). The gaze tracker (1874) may be referred to as an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module.

[0201] For example, the recognition service layer (1870) of the wearable device (100) may further include a face tracker (1875) for tracking the user's face. For example, the wearable device (100) may identify (or track) the movement of the user's face and / or the user's expression based on the execution of the face tracker (1875). The wearable device (100) may estimate the user's expression based on the movement of the user's face based on the execution of the face tracker (1875). As an example, the wearable device (100) may identify the movement of the user's face and / or the user's expression based on data (e.g., images and / or videos) acquired using a camera (1825) (e.g., a camera facing at least a portion of the user's face) based on the execution of the face tracker (1875).

[0202] Referring to FIG. 18, the renderer (1890) may include instructions for rendering images in a three-dimensional virtual space. The processor (1810) executing the renderer (1890) may obtain at least one image to be at least partially displayed in the display area of ​​the display (1650) in a software application. For example, the processor (1810) executing the renderer (1890) may determine the location of the area in which an application (e.g., XR application (1842), application (1845)) is to be rendered. The processor (1810) executing the renderer (1890) may generate an image of the application to be displayed on the display (1650). The renderer (1890) may synthesize images to generate a composite image to be displayed on the display (1650).

[0203] For example, the processor (1810) executing the renderer (1890) can divide the display area of ​​the display (1650) into a foveated portion (or may be referred to as the foveated area) and a peripheral portion (or may be referred to as the residual area) using the gaze position calculated using the position tracker (1871) and / or the gaze tracker (1874). For example, the processor (1810) detecting the coordinate values ​​of the gaze position can determine the portion of the display area including the coordinate values ​​as the foveated area. The DPU executing the renderer (1890) can obtain at least one image corresponding to each of the foveated area and the residual area, and having a size smaller than the size of the entire display area of ​​the display (1650) or a resolution smaller than the resolution of the display area.

[0204] The processor (1810) executing the renderer (1890) may obtain or generate a composite image to be displayed on the display (1650) by synthesizing an image corresponding to the foveated area and an image corresponding to the surrounding area. For example, the processor (1810) may perform upscaling to enlarge the image corresponding to the surrounding area to the size of the entire display area of ​​the display (1650). On the enlarged image, the processor (1810) may combine the image corresponding to the foveated area to generate a composite image to be displayed on the display (1650). Along the boundary line of the image corresponding to the foveated area, the processor (1810) may apply a visual effect, such as blur, to blend the enlarged image and the image corresponding to the foveated area.

[0205] Fig. 19 shows an example of a block diagram of an electronic device (e.g., electronic device (1501), wearable device (100)) for displaying an image in a virtual space. In Fig. 19, an example of executing multiple programs / instructions for displaying an image in a virtual space is described. The multiple programs / instructions may all be executed in one processor (e.g., AP) or may be executed by multiple processors (e.g., AP, GPU (graphics processing unit), NPU (neural processing unit)). The meaning of being executed by the multiple processors means that some programs / instructions may be executed by a first processor and other some programs / instructions may be executed by a second processor different from the first processor.

[0206] Referring to FIG. 19, the electronic device (1501) may execute a virtual space manager (1950) (e.g., the virtual space manager (1851) of FIG. 18, CPM) to render an image in a virtual space. For the virtual space manager (1950), at least some of the descriptions of the virtual space manager (1851) of FIG. 18 may be referenced. The virtual space manager (1950) may include a platform for supporting a virtual space service. The virtual space manager (1950) may include a runtime service (1951) (e.g., OpenXR Runtime), a panel rendering (1952) (e.g., 2D Panel Render), and an XR composition unit (1953) (XR Compositor). The electronic device (1501) may execute at least one of a user's pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (1951). For the runtime service (1951), at least some of the descriptions of the runtime service (1852) of FIG. 18 may be referred to. The electronic device (1501) may display at least one image (video) on a panel (e.g., a 2D panel) to implement a virtual space through the display based on the execution of the panel rendering (1952). For example, the electronic device (1501) may display a rendering image corresponding to RGB information (1966) for the panel from the spatialization manager (1940) described below through the display (e.g., the display (1650)). The electronic device (1501) may synthesize an image of an actual area captured by a camera in the virtual space (hereinafter, a pass-through image) with a virtual area image based on the execution of the XR compositor (1953). For example, the electronic device (1501) can generate a composite image by merging the pass-through image and the virtual area image based on the execution of the XR synthesis unit (1953).The electronic device (1501) may transmit the generated composite image to a display buffer so that the composite image is displayed. The electronic device (1501) may identify a virtual space through a virtual space manager (1950) and display at least a portion of the virtual space on the display (1650). The virtual space manager (1950) may be referred to as a CPM. The electronic device (1501) may execute the virtual space manager (1950) to render an image corresponding to at least a portion of the virtual space.

[0207] According to one embodiment, the electronic device (1501) may execute a spatialization manager (1940). The spatialization manager (1940) may perform processes for displaying an image in a three-dimensional virtual space. The electronic device (1501) may perform preprocessing based on the execution of the spatialization manager (1940) so that the image can be rendered in the three-dimensional virtual space through the virtual space manager (1950). For example, the electronic device (1501) may perform at least some of the functions of the renderer (1890) of FIG. 18 based on the execution of the spatialization manager (1940). The electronic device (1501) may process image information provided by an application (e.g., an XR application (1910), an application (1920) that provides a general 2D screen other than XR, and an application that provides a system UI (1930)) based on the execution of the spatialization manager (1940). A spatialization manager (1940) (e.g., Space Flinger) may include a system scene manager (1941) (e.g., System scene), an input manager (1942) (e.g., Input Routing), and a lightweight rendering engine (1943) (e.g., Impress Engine). The system scene manager (1941) may be executed to display a system UI (1930). System UI-related information (1964) may be transmitted to the system scene manager (1941) from a program (e.g., API) that provides the system UI (1930). The system UI-related information (1964) may be obtained through a spatializer API and / or a same-process private API. The spatialization manager (1940) may determine the layout (e.g., location, display order) of the screen of the system UI (1930) in a three-dimensional space through pre-allocated resources.The system screen manager (1941) may transmit image information (1967) for rendering the screen of the system UI (1930) to the virtual space manager (1950) according to the layout. The input manager (1942) may be configured to process user input (e.g., user input on a system screen or an app screen). The impression engine (1943) may be a renderer for image generation (e.g., a lightweight renderer (1843)). For example, the impression engine (1943) may be used to display the system UI (1930). According to one embodiment, the spatialization manager (1940) may include a lightweight rendering engine (1943) for rendering the system UI. According to one embodiment, when the lightweight rendering engine (1943) does not have sufficient resources to render an avatar used in the HMD, at least one external rendering engine may be used. At this time, to resolve compatibility issues with external rendering (e.g., 3rd party engines), an external rendering engine support module may be added within the spatialization manager (1940).

[0208] According to one embodiment, the electronic device can execute an application. For example, in response to the execution of an XR application (1910) (e.g., an XR application (1842), a 3D game, an XR map, or other immersive application), the electronic device can execute a virtual space manager (1950). The electronic device (1501) can provide dual image information (1961) provided from the XR application (1910) to the virtual space manager (1950). In order to display an image in a three-dimensional space, the dual image information (1961) can include two pieces of image information that take binocular parallax into account. For example, the dual image information (1961) can include first image information for the user's left eye and second image information for the user's right eye for rendering in a three-dimensional virtual space. Hereinafter, in the present disclosure, the term dual image information is used to refer to image information for displaying images for both eyes in a three-dimensional space. In addition to the dual image information, the above dual image information may also include binocular image information, dual image information, dual image data, dual images, binocular image data, stereoscopic image information, 3D image information, spatial image information, spatial image data, 16D-3D conversion data, dimensional conversion image data, binocular parallax image data, and / or equivalent technical terms. The electronic device (1501) can generate a composite image by merging image layers through a virtual space manager (1950). The electronic device (1501) can transmit the generated composite image to a display buffer. The composite image can be displayed on the display (1650) of the electronic device (1501).

[0209] According to one embodiment, the electronic device can execute at least one application among an XR application (1910) and other applications (1920) (e.g., a first application (1920-1), a second application (1920-2), ..., an Nth application (1920-N)). According to one embodiment, the application (1920) can be configured to output image information for displaying a two-dimensional image. In other words, the application (1920) can provide a two-dimensional image. For example, the application (1920) can be a video application, a schedule application, or an application (1920) can be an Internet browser application. If, in response to the execution of the application (1920), image information (1962) provided from the application (1920) is provided to the virtual space manager (1950). Since the image information (1962) only has x-coordinates and y-coordinates within a two-dimensional plane, it may be difficult to consider the chronological relationship (i.e., the distance from the user) between other applications centered on the user. Even when displaying an application (1920) that provides a general 2D screen, the electronic device (1501) may execute the spatialization manager (1940) to provide dual image information to the virtual space manager (1950). For example, based on the execution of the spatialization manager (1940), the electronic device (1501) may receive application-related information (1963) from the first application (1920-1). For example, the application-related information (1963) may include image information representing a two-dimensional image of the first application (1920-1) (e.g., information including RGB per pixel) and / or content information in the first application (1920-1) (e.g., characteristics of content executed in the first application, type of content). Application related information (1963) can be obtained via the spatializer API.Based on the execution of the spatialization manager (1940), the electronic device (1501) can identify information about the location of the area to be rendered by the first application (1920-1) and the size of the area to be rendered (hereinafter, location information). Based on the execution of the spatialization manager (1940), the electronic device (1501) can generate dual image information (1965, e.g., RGBx2) that takes into account the user's binocular disparity through the image information and the location information. Based on the execution of the spatialization manager (1940), the electronic device (1501) can provide the dual image information (1965) to the virtual space manager (1950). By converting a simple two-dimensional image into the dual image information (1965), a problem that occurs when the image information (1962) is directly transmitted to the virtual space manager (1950) can be resolved. Additionally, since at least some of the functions for displaying images in a virtual space are performed by the spatialization manager (1940) instead of the virtual space manager (1950), the burden on the virtual space manager (1950) can be reduced.

[0210] A wearable device as described above may include a memory storing instructions. The wearable device may include a communication circuit. The wearable device may include a display assembly including at least one display. The wearable device may include at least one processor. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display, through the display assembly, an avatar including a first portion controllable by a first control device and a second portion controllable by a second control device paired with the first control device, and an object associated with the avatar. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to receive, through the communication circuit, first data of movement of the first control device transmitted from the first control device and second data of movement of the second control device transmitted from the second control device while displaying the avatar and the object. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate, based on the reception, third data about the first part of the avatar moving relative to the object using the first data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate, based on the reception, fourth data about the second part of the avatar moving relative to the object using the third data and information about the object.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate fifth data about global movement of the avatar using the third data and the fourth data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the avatar moving relative to the object through the display assembly using the fifth data.

[0211] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify whether a mode for compensating for movement of the avatar displayed through the display assembly is activated based on receiving first data from the first control device regarding movement of the first control device and second data from the second control device regarding movement of the second control device. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate third data using the first data based on identifying that the mode is activated. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate fourth data using the third data and the information based on identifying that the mode is activated. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate the fifth data using the third data and the fourth data based on identifying that the mode is activated. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the avatar moving relative to the object using the fifth data based on identifying that the mode is activated. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate the third data using the first data based on identifying that the mode is deactivated.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to skip generating the fourth data using the third data and the information based on identifying that the mode is disabled. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate sixth data about global movement of the avatar using the third data based on identifying that the mode is disabled. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the avatar moving relative to the object using the sixth data based on identifying that the mode is disabled.

[0212] In one embodiment, the wearable device may further include at least one sensor. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to obtain sixth data regarding movement of the wearable device from the at least one sensor. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to further use the sixth data to generate the fourth data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to further use the sixth data to generate the fifth data.

[0213] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate the fourth data by providing the third data and the sixth data to a model trained through machine learning.

[0214] In one embodiment, the model may be included in an external electronic device distinct from the wearable device. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to transmit the third data, the information, and the sixth data to the external electronic device via the communication circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to receive the fourth data from the external electronic device via the communication circuit.

[0215] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate the fourth data by providing the third data and the information to a model trained through machine learning.

[0216] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate the fourth data using further information about the environment surrounding the avatar. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate the fifth data using the fourth data generated further using the third data and the other information.

[0217] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device (100) to generate the fourth data by providing the third data and the other information to a model trained through machine learning.

[0218] In one embodiment, the wearable device may further include at least one sensor. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to obtain sixth data regarding movement of the wearable device from the at least one sensor. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to further use the sixth data to generate the fifth data.

[0219] According to one embodiment, the fifth data can be generated by applying inverse kinematics to the third data and the fourth data.

[0220] A wearable device as described above may include a memory that stores instructions. The wearable device may include a camera assembly including a plurality of cameras. The wearable device may include a communication circuit. The wearable device may include a display assembly including at least one display. The wearable device may include at least one processor. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display, through the display assembly, an avatar including a first part controllable by a first part of a user's body and a second part controllable by a second part of the user's body paired with the first part of the user's body, and an object associated with the avatar. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to acquire first data of movement of the first part of the body of the user and second data of movement of the second part of the body of the user using images acquired through the camera assembly while displaying the avatar and the object. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate, based on the acquisition, third data of the first part of the avatar moving relative to the object using the first data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate, using the third data and information about the object, fourth data of the second part of the avatar moving relative to the object.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate fifth data about global movement of the avatar using the third data and the fourth data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the avatar moving relative to the object through the display assembly using the fifth data.

[0221] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify whether a mode for compensating for movement of the avatar displayed through the display assembly is activated based on acquiring first data of movement of the first part of the body of the user and second data of movement of the second part of the body of the user using the images acquired through the camera assembly. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate third data using the first data based on identifying that the mode is activated. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate fourth data using the third data and the information based on identifying that the mode is activated. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate the fifth data using the third data and the fourth data based on identifying that the mode is activated. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the avatar moving relative to the object using the fifth data based on identifying that the mode is activated.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate the third data using the first data based on identifying that the mode is disabled. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to skip generating the fourth data using the third data and the information based on identifying that the mode is disabled. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate sixth data about global movement of the avatar using the third data based on identifying that the mode is disabled. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the avatar moving relative to the object using the sixth data based on identifying that the mode is disabled.

[0222] In one embodiment, the wearable device may further include at least one sensor. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to obtain sixth data regarding movement of the wearable device from the at least one sensor. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to further use the sixth data to generate the fourth data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to further use the sixth data to generate the fifth data.

[0223] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate the fourth data by providing the third data and the sixth data to a model trained through machine learning.

[0224] In one embodiment, the model may be included in an external electronic device distinct from the wearable device. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to transmit the third data, the information, and the sixth data to the external electronic device via the communication circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to receive the fourth data from the external electronic device via the communication circuit.

[0225] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate the fourth data by providing the third data and the information to a model trained through machine learning.

[0226] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate the fourth data using further information about the environment surrounding the avatar. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate the fifth data using the fourth data generated further using the third data and the other information.

[0227] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device (100) to generate the fourth data by providing the third data and the other information to a model trained through machine learning.

[0228] In one embodiment, the wearable device may further include at least one sensor. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to obtain sixth data regarding movement of the wearable device from the at least one sensor. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to further use the sixth data to generate the fifth data.

[0229] According to one embodiment, the fifth data can be generated by applying inverse kinematics to the third data and the fourth data.

[0230] A method performed by a wearable device having a display assembly including a communication circuit and at least one display as described above may include an operation of displaying, through the display assembly, an avatar and an object related to the avatar, the avatar including a first part controllable by a first control device and a second part controllable by a second control device paired with the first control device. The method may include an operation of receiving, through the communication circuit, first data from among first data regarding movement of the first control device transmitted from the first control device and second data regarding movement of the second control device transmitted from the second control device, while displaying the avatar and the object. The method may include an operation of generating, based on the reception, third data regarding the first part of the avatar moving relative to the object using the first data. The method may include an operation of generating, using the third data and information regarding the object, fourth data regarding the second part of the avatar moving relative to the object. The method may include an operation of generating fifth data regarding the global movement of the avatar using the third data and the fourth data. The method may include an operation of displaying the avatar moving with respect to the object through the display assembly using the fifth data.

[0231] According to one embodiment, the method may include an operation of identifying whether a mode for correcting the movement of the avatar displayed through the display assembly is activated based on receiving first data among first data regarding movement of the first control device transmitted from the first control device and second data regarding movement of the second control device transmitted from the second control device. The method may include an operation of generating the third data using the first data based on identifying that the mode is activated. The method may include an operation of generating the fourth data using the third data and the information based on identifying that the mode is activated. The method may include an operation of generating the fifth data using the third data and the fourth data based on identifying that the mode is activated. The method may include an operation of displaying the avatar moving with respect to the object using the fifth data based on identifying that the mode is activated. The method may include an operation of generating the third data using the first data based on identifying that the mode is deactivated. The method may include an action of skipping generating the fourth data using the third data and the information based on identifying that the mode is deactivated. The method may include an action of generating sixth data about the global movement of the avatar using the third data based on identifying that the mode is deactivated. The method may include an action of displaying the avatar moving with respect to the object using the sixth data based on identifying that the mode is deactivated.

[0232] In one embodiment, the wearable device may further include at least one sensor. The method may include an operation of obtaining sixth data regarding movement of the wearable device from the at least one sensor. The method may further include an operation of generating the fourth data using the sixth data. The method may further include an operation of generating the fifth data using the sixth data.

[0233] According to one embodiment, the method may include an operation of generating the fourth data by providing the third data and the sixth data to a model trained through machine learning.

[0234] According to one embodiment, the model may be included in an external electronic device distinct from the wearable device. The method may include transmitting the third data, the information, and the sixth data to the external electronic device via the communication circuit. The method may include receiving the fourth data from the external electronic device via the communication circuit.

[0235] According to one embodiment, the method may include an operation of generating the fourth data by providing the third data and the information to a model trained through machine learning.

[0236] In one embodiment, the method may include generating the fourth data by further utilizing other information about the environment surrounding the avatar. The method may include generating the fifth data by utilizing the fourth data generated by further utilizing the third data and the other information.

[0237] In one embodiment, the method may include generating the fourth data by providing the third data and the other information to a model trained through machine learning.

[0238] In one embodiment, the wearable device may further include at least one sensor. The method may include an operation of acquiring sixth data regarding the movement of the wearable device from the at least one sensor. The method may further include an operation of generating the fifth data using the sixth data.

[0239] According to one embodiment, the fifth data can be generated by applying inverse kinematics to the third data and the fourth data.

[0240] A method performed by a wearable device having a camera assembly including a plurality of cameras, a communication circuit, and a display assembly including at least one display, as described above, may include an operation of displaying, through the display assembly, an avatar including a first part controllable by a first part of a body of a user and a second part controllable by a second part of the body of the user paired with the first part of the body of the user, and an object related to the avatar. The method may include an operation of acquiring, while displaying the avatar and the object, first data of first data about movement of the first part of the body of the user and second data about movement of the second part of the body of the user using images acquired through the camera assembly. The method may include an operation of generating, based on the acquisition, third data about the first part of the avatar moving with respect to the object using the first data. The method may include an operation of generating, using the third data and information about the object, fourth data about the second part of the avatar moving with respect to the object. The method may include an operation of generating fifth data regarding the global movement of the avatar using the third data and the fourth data. The method may include an operation of displaying the avatar moving with respect to the object through the display assembly using the fifth data.

[0241] According to one embodiment, the method may include an operation of identifying whether a mode for correcting the movement of the avatar displayed through the display assembly is activated based on acquiring first data among first data about movement of the first part of the body of the user and second data about movement of the second part of the body of the user using the images acquired through the camera assembly. The method may include an operation of generating third data using the first data based on identifying that the mode is activated. The method may include an operation of generating fourth data using the third data and the information based on identifying that the mode is activated. The method may include an operation of generating fifth data using the third data and the fourth data based on identifying that the mode is activated. The method may include an operation of displaying the avatar moving with respect to the object using the fifth data based on identifying that the mode is activated. The method may include an action of generating the third data using the first data based on identifying that the mode is deactivated. The method may include an action of skipping generating the fourth data using the third data and the information based on identifying that the mode is deactivated. The method may include an action of generating sixth data about global movement of the avatar using the third data based on identifying that the mode is deactivated. The method may include an action of displaying the avatar moving with respect to the object using the sixth data based on identifying that the mode is deactivated.

[0242] In one embodiment, the wearable device may further include at least one sensor. The method may include an operation of obtaining sixth data regarding movement of the wearable device from the at least one sensor. The method may further include an operation of generating the fourth data using the sixth data. The method may further include an operation of generating the fifth data using the sixth data.

[0243] According to one embodiment, the method may include an operation of generating the fourth data by providing the third data and the sixth data to a model trained through machine learning.

[0244] According to one embodiment, the model may be included in an external electronic device distinct from the wearable device. The method may include transmitting the third data, the information, and the sixth data to the external electronic device via the communication circuit. The method may include receiving the fourth data from the external electronic device via the communication circuit.

[0245] According to one embodiment, the method may include an operation of generating the fourth data by providing the third data and the information to a model trained through machine learning.

[0246] In one embodiment, the method may include generating the fourth data by further utilizing other information about the environment surrounding the avatar. The method may include generating the fifth data by utilizing the fourth data generated by further utilizing the third data and the other information.

[0247] In one embodiment, the method may include generating the fourth data by providing the third data and the other information to a model trained through machine learning.

[0248] According to one embodiment, the wearable device (100) may further include at least one sensor. The method may include an operation of acquiring sixth data regarding the movement of the wearable device from the at least one sensor. The method may further include an operation of generating the fifth data using the sixth data.

[0249] According to one embodiment, the fifth data may include a motion generated by applying inverse kinematics to the third data and the fourth data.

[0250] In a computer-readable storage medium having one or more programs stored thereon, as described above, the one or more programs may include instructions that, when executed by a wearable device having a display assembly including communication circuitry and at least one display, cause the wearable device to display, through the display assembly, an avatar and an object related to the avatar, the avatar including a first part controllable by a first control device and a second part controllable by a second control device paired with the first control device. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to receive, through the communication circuitry, first data regarding movement of the first control device transmitted from the first control device and second data regarding movement of the second control device transmitted from the second control device while displaying the avatar and the object. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to generate third data about the first part of the avatar moving relative to the object using the first data based on the reception. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to generate fourth data about the second part of the avatar moving relative to the object using the third data and information about the object. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to generate fifth data about global movement of the avatar using the third data and the fourth data.The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the avatar moving relative to the object through the display assembly using the fifth data.

[0251] In one embodiment, the one or more programs may include instructions that cause the wearable device to identify whether a mode for correcting the movement of the avatar displayed through the display assembly is activated based on receiving first data among first data regarding movement of the first control device transmitted from the first control device and second data regarding movement of the second control device transmitted from the second control device. The one or more programs may include instructions that cause the wearable device to generate third data using the first data based on identifying that the mode is activated. The one or more programs may include instructions that cause the wearable device to generate fourth data using the third data and the information based on identifying that the mode is activated. The one or more programs may include instructions that cause the wearable device to generate the fifth data using the third data and the fourth data based on identifying that the mode is activated. The one or more programs may include instructions that cause the wearable device to display the avatar moving relative to the object using the fifth data based on identifying that the mode is activated. The one or more programs may include instructions that cause the wearable device to generate the third data using the first data based on identifying that the mode is deactivated. The one or more programs may include instructions that cause the wearable device to skip generating the fourth data using the third data and the information based on identifying that the mode is deactivated.The one or more programs may include instructions that cause the wearable device to generate sixth data about global movement of the avatar using the third data based on identifying that the mode is disabled. The one or more programs may include instructions that cause the wearable device to display the avatar moving relative to the object using the sixth data based on identifying that the mode is disabled.

[0252] In one embodiment, the wearable device may further include at least one sensor. The one or more programs may include instructions for causing the wearable device to obtain sixth data regarding movement of the wearable device from the at least one sensor. The one or more programs may include instructions for causing the wearable device to further use the sixth data to generate the fourth data. The one or more programs may include instructions for causing the wearable device to further use the sixth data to generate the fifth data.

[0253] In one embodiment, the one or more programs may include instructions that cause the wearable device to generate the fourth data by providing the third data and the sixth data to a model trained through machine learning.

[0254] In one embodiment, the model may be included in an external electronic device distinct from the wearable device. The one or more programs may include instructions that cause the wearable device to transmit the third data, the information, and the sixth data to the external electronic device via the communication circuit. The one or more programs may include instructions that cause the wearable device to receive the fourth data from the external electronic device via the communication circuit.

[0255] In one embodiment, the one or more programs may include instructions that cause the wearable device to generate the fourth data by providing the third data and the information to a model trained through machine learning.

[0256] In one embodiment, the one or more programs may include instructions that cause the wearable device to generate the fourth data using further information about the environment surrounding the avatar. The one or more programs may include instructions that cause the wearable device to generate the fifth data using the fourth data generated further using the third data and the other information.

[0257] In one embodiment, the one or more programs may include instructions that cause the wearable device to generate the fourth data by providing the third data and the other information to a model trained through machine learning.

[0258] In one embodiment, the wearable device may further include at least one sensor. The one or more programs may include instructions causing the wearable device to obtain sixth data regarding movement of the wearable device from the at least one sensor. The one or more programs may further include instructions causing the wearable device to generate fifth data using the sixth data.

[0259] According to one embodiment, the fifth data can be generated by applying inverse kinematics to the third data and the fourth data.

[0260] In a computer-readable storage medium having one or more programs stored thereon, as described above, the one or more programs may include instructions that, when executed by a wearable device having a camera assembly including a plurality of cameras, a communication circuit, and a display assembly including at least one display, cause the wearable device to display, through the display assembly, an avatar including a first part controllable by a first part of a user's body and a second part controllable by a second part of the user's body paired with the first part of the user's body, and an object associated with the avatar. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to acquire, using images acquired through the camera assembly, first data regarding movement of the first part of the user's body and second data regarding movement of the second part of the user's body, while displaying the avatar and the object. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to generate third data about the first part of the avatar moving relative to the object using the first data based on the acquisition. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to generate fourth data about the second part of the avatar moving relative to the object using the third data and information about the object.The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to generate fifth data about global movement of the avatar using the third data and the fourth data. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display, through the display assembly, the avatar moving relative to the object using the fifth data.

[0261] In one embodiment, the one or more programs may include instructions that cause the wearable device to identify whether a mode for correcting movement of the avatar displayed through the display assembly is activated based on acquiring first data among first data about movement of the first part of the body of the user and second data about movement of the second part of the body of the user using the images acquired through the camera assembly. The one or more programs may include instructions that cause the wearable device to generate third data using the first data based on identifying that the mode is activated. The one or more programs may include instructions that cause the wearable device to generate fourth data using the third data and the information based on identifying that the mode is activated. The one or more programs may include instructions that cause the wearable device to generate the fifth data using the third data and the fourth data based on identifying that the mode is activated. The one or more programs may include instructions that cause the wearable device to display the avatar moving relative to the object using the fifth data based on identifying that the mode is activated. The one or more programs may include instructions that cause the wearable device to generate the third data using the first data based on identifying that the mode is deactivated.The one or more programs may include instructions that cause the wearable device to skip generating the fourth data using the third data and the information based on identifying that the mode is disabled. The one or more programs may include instructions that cause the wearable device to generate sixth data about global movement of the avatar using the third data based on identifying that the mode is disabled. The one or more programs may include instructions that cause the wearable device to display the avatar moving relative to the object using the sixth data based on identifying that the mode is disabled.

[0262] In one embodiment, the wearable device may further include at least one sensor. The one or more programs may include instructions for causing the wearable device to obtain sixth data regarding movement of the wearable device from the at least one sensor. The one or more programs may include instructions for causing the wearable device to further use the sixth data to generate the fourth data. The one or more programs may include instructions for causing the wearable device to further use the sixth data to generate the fifth data.

[0263] In one embodiment, the one or more programs may include instructions that cause the wearable device to generate the fourth data by providing the third data and the sixth data to a model trained through machine learning.

[0264] In one embodiment, the model may be included in an external electronic device distinct from the wearable device. The one or more programs may include instructions that cause the wearable device to transmit the third data, the information, and the sixth data to the external electronic device via the communication circuit. The one or more programs may include instructions that cause the wearable device to receive the fourth data from the external electronic device via the communication circuit.

[0265] In one embodiment, the one or more programs may include instructions that cause the wearable device to generate the fourth data by providing the third data and the information to a model trained through machine learning.

[0266] In one embodiment, the one or more programs may include instructions that cause the wearable device to generate the fourth data using further information about the environment surrounding the avatar. The one or more programs may include instructions that cause the wearable device to generate the fifth data using the fourth data generated further using the third data and the other information.

[0267] In one embodiment, the one or more programs may include instructions that cause the wearable device to generate the fourth data by providing the third data and the other information to a model trained through machine learning.

[0268] In one embodiment, the wearable device may further include at least one sensor. The one or more programs may include instructions causing the wearable device to obtain sixth data regarding movement of the wearable device from the at least one sensor. The one or more programs may further include instructions causing the wearable device to generate fifth data using the sixth data.

[0269] According to one embodiment, the fifth data can be generated by applying inverse kinematics to the third data and the fourth data.

[0270] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0271] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0272] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0273] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0274] Therefore, other implementations, other embodiments, and equivalents of the claims are also within the scope of the claims described below. According to one embodiment, the method according to the various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0275] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In a wearable device (100), A memory (306) storing instructions and including one or more storage media; Communication circuit (305); A display assembly (308) comprising at least one display; and At least one processor (307) comprising processing circuitry, The above instructions, when individually or collectively executed by the at least one processor, An avatar (130) including a first part controllable by a first control device (110) and a second part controllable by a second control device (120) paired with the first control device (110) and an object related to the avatar are displayed through the display assembly (308). While displaying the avatar (130) and the object, the first data among the first data on the movement of the first control device (110) transmitted from the first control device (110) and the second data on the movement of the second control device (120) transmitted from the second control device (120) is received through the communication circuit (305). Based on the above reception, third data (720) is generated for the first part of the avatar (130) moving with respect to the object using the first data, Using the third data (720) and the information about the object (730), fourth data (750) is generated for the second part of the avatar moving with respect to the object, Using the third data (720) and the fourth data (750), the fifth data for the global movement of the avatar is generated, To display the avatar (130) moving with respect to the object through the display assembly (308) using the fifth data, causing the above wearable device, Wearable devices.

2. In claim 1, The above instructions, when individually or collectively executed by the at least one processor, Based on receiving the first data among the first data on the movement of the first control device (110) transmitted from the first control device (110) and the second data on the movement of the second control device (120) transmitted from the second control device (120), it is identified whether a mode for correcting the movement of the avatar (130) displayed through the display assembly (308) is activated, Based on identifying that the above mode is activated: Generate the third data (720) using the first data; Generate the fourth data (750) using the third data (720) and the information (730); Generating the fifth data using the third data (720) and the fourth data (750); and Using the above fifth data, the avatar (130) moving with respect to the object is displayed, Based on identifying that the above mode is disabled: Generating the third data using the first data; Skip generating the fourth data (750) using the third data (720) and the information (730); Generating sixth data about the global movement of the avatar using the third data; and To display the avatar moving with respect to the object using the above sixth data, Further causing the above wearable device, Wearable devices.

3. In claim 1, Contains at least one further sensor, The above instructions, when individually or collectively executed by the at least one processor, Obtaining sixth data (740) about the movement of the wearable device (100) from at least one sensor, The fourth data (750) is generated by further using the sixth data (740), To further use the above 6th data (740) to generate the above 5th data, causing the above wearable device, Wearable devices.

4. In claim 3, The above instructions, when individually or collectively executed by the at least one processor, By providing the third data (720) and the sixth data (740) to a model (710) trained through machine learning, the fourth data (750) is generated. causing the above wearable device, Wearable devices.

5. In claim 4, the model (710) Included in an external electronic device distinct from the above wearable device (100), The above instructions, when individually or collectively executed by the at least one processor, Transmitting the third data (720), the information (730), and the sixth data (740) to the external electronic device through the communication circuit (305), To receive the fourth data (750) from the external electronic device through the communication circuit (305), causing the above wearable device, Wearable devices.

6. In claim 1, The above instructions, when individually or collectively executed by the at least one processor, By providing the third data (720) and the information (730) to a model (710) trained through machine learning, the fourth data (750) is generated. causing the above wearable device, Wearable devices.

7. In claim 1, The above instructions, when individually or collectively executed by the at least one processor, By further utilizing other information about the environment around the avatar (130), the fourth data (750) is generated, To generate the fifth data using the fourth data (750) generated by further utilizing the third data (720) and other information, causing the above wearable device, Wearable devices.

8. In claim 7, The above instructions, when individually or collectively executed by the at least one processor, By providing the third data (720) and other information to a model (710) trained through machine learning, the fourth data (750) is generated. causing the above wearable device, Wearable devices.

9. In claim 1, Contains at least one further sensor, The above instructions, when individually or collectively executed by the at least one processor, Obtaining sixth data (740) about the movement of the wearable device (100) from at least one sensor, To further use the above 6th data (740) to generate the above 5th data, causing the above wearable device, Wearable devices.

10. In claim 1, the fifth data is: Generated by applying inverse kinematics to the third data (720) and the fourth data (750). Wearable devices.

11. In wearable devices, A memory (306) storing instructions and including one or more storage media; A camera assembly (309) comprising multiple cameras; Communication circuit (305); A display assembly (308) comprising at least one display; and At least one processor (307) comprising processing circuitry, The above instructions, when individually or collectively executed by the at least one processor, An avatar (130) including a first part controllable by a first part of a body of a user (200) and a second part controllable by a second part of a body of the user (200) that is paired with the first part of the body of the user (200), and an object related to the avatar (130), are displayed through the display assembly (308). While displaying the avatar (130) and the object, the first data among the first data on the movement of the first part of the body of the user (200) and the second data on the movement of the second part of the body of the user (200) is acquired using the images acquired through the camera assembly (309), Based on the above acquisition, third data (720) is generated for the first part of the avatar (130) moving with respect to the object using the first data, Using the third data (720) and the information (730) about the object, fourth data (750) about the second part of the avatar (130) moving with respect to the object is generated, Using the third data (720) and the fourth data (750), the fifth data for the global movement of the avatar (130) is generated, To display the avatar (130) moving with respect to the object through the display assembly (308) using the fifth data, causing the above wearable device, Wearable devices.

12. In claim 11, The above instructions, when individually or collectively executed by the at least one processor, Based on obtaining the first data among the first data on the movement of the first part of the body of the user (200) and the second data on the movement of the second part of the body of the user (200) using the images acquired through the camera assembly (309), it is identified whether a mode for correcting the movement of the avatar (130) displayed through the display assembly (308) is activated, Based on identifying that the above mode is activated: Generating the third data using the first data; Generate the fourth data (750) using the third data (720) and the information (730); Generating the fifth data using the third data (720) and the fourth data (750); and Using the above fifth data, the avatar (130) moving with respect to the object is displayed, Based on identifying that the above mode is disabled: Generating the third data using the first data; Skip generating the fourth data (750) using the third data (720) and the information (730); Generating sixth data on the global movement of the avatar (130) using the third data; and To display the avatar (130) moving with respect to the object using the above sixth data, Further causing the above wearable device, Wearable devices.

13. In claim 11, Contains at least one further sensor, The above instructions, when individually or collectively executed by the at least one processor, Obtaining sixth data (740) about the movement of the wearable device (100) from at least one sensor, The fourth data (750) is generated by further using the sixth data (740), To further use the above 6th data (740) to generate the above 5th data, causing the above wearable device, Wearable devices.

14. In claim 13, The above instructions, when individually or collectively executed by the at least one processor, By providing the third data (720) and the sixth data (740) to a model (710) trained through machine learning, the fourth data (750) is generated. causing the above wearable device, Wearable devices.

15. In claim 14, the model (710) Included in an external electronic device distinct from the above wearable device (100), The above instructions, when individually or collectively executed by the at least one processor, Transmitting the third data (720), the information (730), and the sixth data (740) to the external electronic device through the communication circuit (305), To receive the fourth data (750) from the external electronic device through the communication circuit (305), causing the above wearable device, Wearable devices.

Images