Apparatus, method, and storage medium for performing tracking by using external electronic device

Abstract

This wearable device may comprise: a memory for storing instructions; a sensor; a communication circuit; and at least one processor. The instructions may, when executed by the at least one processor, cause the wearable device to: receive first sensor information of a first external electronic device and second sensor information of a second external electronic device; obtain a first similarity value and a second similarity value; identify a compensated motion value on the basis of the first sensor information and a value indicating the motion of the wearable device; identify a compensated motion value on the basis of the second sensor information and the value indicating the motion of the wearable device; and display a screen according to the compensated motion values.

Description

Device, method and storage medium for performing tracking using external electronic devices

[0001] The present disclosure relates to a device, method, and storage medium for performing tracking using an external electronic device.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide extended reality services that display computer-generated information in conjunction with external objects in the real world or virtual objects in the virtual world. The electronic devices may include wearable devices that can be worn by a user. For example, the electronic devices may include user equipment, augmented reality (AR) glasses, virtual reality (VR) glasses, and / or a head-mounted device (HMD) (e.g., video see-through (VST) HMD, optical see-through (OST) HMD).

[0003] The above information is provided solely as background information to aid understanding of the present disclosure. No determination has been made, and no claim is made, as to whether the above-described material constitutes prior art related to the present disclosure.

[0004] Aspects of the present disclosure address at least the problems and / or disadvantages mentioned above, and provide at least the advantages described below. Accordingly, aspects of the present disclosure provide a device, method, and storage medium for performing tracking using an external electronic device.

[0005] Additional aspects will be partly set forth in the following description, partly will be apparent from the description, or may be learned by practice of the embodiments presented.

[0006] According to one aspect of the present disclosure, a wearable device is provided. The wearable device may include a memory storing instructions and including one or more storage media. The wearable device may include a sensor. The wearable device may include a communication circuit. The wearable device may include at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to receive first sensor information of a first external electronic device and second sensor information of a second external electronic device through the communication circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to obtain a first similarity value between the movement of the wearable device and the movement of the first external electronic device based on a value representing the movement of the wearable device obtained through the sensor and the first sensor information. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to obtain a second similarity value between the movement of the wearable device and the movement of the second external electronic device based on the value and the second sensor information. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a compensated motion value of the wearable device based on the first sensor information and the value representing the motion of the wearable device, if the first similarity value exceeds a reference value and the second similarity value is less than or equal to the reference value.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a compensated motion value of the wearable device based on the second sensor information and the value representing the motion of the wearable device, if the first similarity value is less than or equal to the reference value and the second similarity value exceeds the reference value. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a screen according to the compensated motion value.

[0007] According to another aspect of the present disclosure, a wearable device is provided. The wearable device may include a memory storing instructions and including one or more storage media. The wearable device may include a sensor. The wearable device may include a communication circuit. The wearable device may include at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to establish a connection with an external electronic device via the communication circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to receive sensor information from the external electronic device, the sensor information including a sensor value and a first timing at which the sensor value was acquired. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to obtain, through the sensor, a value representing a movement of the wearable device at a second timing after the first timing. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to calculate a first vector for the movement of the wearable device based on another value and the value obtained through the sensor at a third timing prior to the first timing. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to calculate a second vector for the movement of the external electronic device based on another sensor value and the sensor value calculated for the third timing.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to obtain a similarity value between the movement of the external electronic device and the movement of the wearable device based on the first vector and the second vector. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to adjust the value representing the movement of the wearable device using the sensor value based on the similarity value exceeding a reference value.

[0008] According to another aspect of the present disclosure, a method performed by a wearable device is provided. The method may include an operation of receiving first sensor information of a first external electronic device and second sensor information of a second external electronic device. The method may include an operation of obtaining a first similarity value between the movement of the wearable device and the movement of the first external electronic device based on a value representing a movement of the wearable device and the first sensor information. The method may include an operation of obtaining a second similarity value between the movement of the wearable device and the movement of the second external electronic device based on the value and the second sensor information. The method may include an operation of adjusting the value representing the movement of the wearable device using the first sensor information based on the first similarity value exceeding a reference value and the second similarity value being below the reference value. The method may include an operation of identifying a compensated motion value of the wearable device based on the first sensor information and the value representing the motion of the wearable device when the first similarity value exceeds a reference value and the second similarity value is less than or equal to the reference value. The method may include an operation of identifying a compensated motion value of the wearable device based on the second sensor information and the value representing the motion of the wearable device when the first similarity value is less than or equal to the reference value and the second similarity value exceeds the reference value. The method may include an operation of displaying a screen according to the compensated motion value.

[0009] According to another aspect of the present disclosure, one or more non-transitory computer-readable storage media are provided that store one or more computer programs including computer-executable instructions that, when individually or collectively executed by at least one processor of a wearable device, cause a wearable device including a sensor and a communication circuit to perform operations. The operations may include receiving, via the communication circuit, first sensor information of a first external electronic device and second sensor information of a second external electronic device. The operations may include obtaining a first similarity value between the movement of the wearable device and the movement of the first external electronic device based on a value representing a movement of the wearable device obtained through the sensor and the first sensor information. The operations may include obtaining a second similarity value between the movement of the wearable device and the movement of the second external electronic device based on the value and the second sensor information. The above operations may include an operation of identifying a compensated motion value of the wearable device based on the first sensor information and the value representing the motion of the wearable device when the first similarity value exceeds a reference value and the second similarity value is less than or equal to the reference value. The above operations may include an operation of identifying a compensated motion value of the wearable device based on the second sensor information and the value representing the motion of the wearable device when the first similarity value is less than or equal to the reference value and the second similarity value exceeds the reference value. The above operations may include an operation of displaying a screen according to the compensated motion value.

[0010] Other aspects, advantages, or salient features of the present disclosure will become apparent to those skilled in the art from the following detailed description of various embodiments of the present disclosure taken in conjunction with the accompanying drawings.

[0011] The above and other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0012] FIG. 1 is a block diagram of an electronic device within a network environment according to an embodiment of the present disclosure;

[0013] FIG. 2A illustrates an example of a perspective view of a wearable device according to an embodiment of the present disclosure;

[0014] FIG. 2b illustrates an example of one or more hardware elements arranged within a wearable device according to an embodiment of the present disclosure;

[0015] FIGS. 3A and 3B illustrate examples of the appearance of a wearable device according to various embodiments of the present disclosure;

[0016] FIGS. 4A and 4B illustrate examples of a method for improving tracking drift using external electronic devices according to various embodiments of the present disclosure;

[0017] FIG. 5 illustrates a block diagram of a wearable device according to an embodiment of the present disclosure;

[0018] FIG. 6 illustrates an example of an operational flow for a method of adjusting a value indicating movement of a wearable device using sensor information obtained from an external electronic device according to an embodiment of the present disclosure;

[0019] FIG. 7A illustrates an example of a method for predicting a value representing the movement of an external electronic device using sensor information obtained from the external electronic device according to an embodiment of the present disclosure;

[0020] FIG. 7b illustrates an example of a method for calculating a similarity value between a movement of a wearable device and a movement of an external electronic device according to an embodiment of the present disclosure;

[0021] FIG. 7c illustrates an example of a method for adjusting a value representing the movement of a wearable device according to an embodiment of the present disclosure;

[0022] FIG. 8 illustrates an example of a wearable device that utilizes sensor information of a first external electronic device connected to a second external electronic device according to an embodiment of the present disclosure;

[0023] FIG. 9 illustrates an example of a visual object that inquires about the execution of a function that adjusts a value representing the movement of a wearable device according to an embodiment of the present disclosure; and

[0024] FIG. 10 illustrates an example of an operational flow for a method of adjusting a value representing the movement of a wearable device based on a similarity value obtained using sensor information of external electronic devices according to an embodiment of the present disclosure.

[0025] Identical reference numbers are used throughout the drawings to indicate identical components.

[0026] The following description, with reference to the accompanying drawings, is provided to facilitate a comprehensive understanding of various embodiments of the present disclosure as defined by the claims and their equivalents. While this disclosure includes numerous specific details to aid understanding, these are to be considered merely illustrative. Accordingly, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. Furthermore, well-known functions and configurations may be omitted for clarity and brevity.

[0027] The terms and words used in the following descriptions and claims are not intended to be limited in their bibliographical meanings, but are used by the inventors solely to facilitate a clear and consistent understanding of the present disclosure. Therefore, it should be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only and is not intended to limit the present disclosure as defined by the appended claims and their equivalents.

[0028] The singular forms "a," "an," and "the" should be understood to include plural referents unless the context clearly indicates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more such surfaces.

[0029] The various embodiments of the present disclosure described below illustrate a hardware-based approach as an example. However, since the various embodiments of the present disclosure include techniques utilizing both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0030] In addition, in the present disclosure, expressions such as "more than" or "less than" may be used to determine whether a specific condition is satisfied or fulfilled. However, this is merely a description for expressing an example and does not exclude descriptions such as "more than" or "less than." Conditions described as "more than" may be replaced with "more than," conditions described as "less than," and conditions described as "more than and less than" may be replaced with "more than and less than." In addition, hereinafter, "A" to "B" mean at least one of the elements from A (including A) to B (including B).

[0031] It should be recognized that the blocks and combinations of flowcharts within each flowchart can be performed by one or more computer programs containing instructions. The entirety of one or more computer programs may be stored in a single memory device, or the one or more computer programs may be divided into different parts stored in different memory devices.

[0032] Any of the functions or operations described in the present disclosure may be processed by a single processor or a combination of processors. A single processor or a combination of processors is a circuit that performs processing, and may include an application processor (AP) (e.g., a central processing unit (CPU), a communication processor (CP) (e.g., a modem), a graphics processing unit (GPU), a neural processing unit (NPU) (e.g., an artificial intelligence (AI) chip), a Wi-Fi chip, or a Bluetooth chip. ® It includes chips, GPS (global positioning system) chips, NFC (near field communication) chips, link chips, sensor controllers, touch controllers, fingerprint sensor controllers, display driver ICs (integrated circuits), audio codec chips, USB (universal serial bus) controllers, camera controllers, image processing ICs, microprocessor units (MPUs), SoCs (systems on chips), ICs, etc.

[0033] FIG. 1 is a block diagram of an electronic device within a network environment according to an embodiment of the present disclosure.

[0034] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to another embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0035] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to another embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0036] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In another embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.For example, the artificial neural network may be one of, but is not limited to, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above. An artificial intelligence model may additionally or alternatively include a software structure in addition to or in place of a hardware structure.

[0037] The memory (130) may be configured to store various data used by at least one component (e.g., a processor (120) or a sensor module (176)) of the electronic device (101). The data may include, for example, software (e.g., a program (140)) and input data or output data for commands related thereto. The memory (130) may include a volatile memory (132) or a non-volatile memory (134).

[0038] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0039] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0040] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In another embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0041] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In another embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0042] The audio module (170) can convert sound into an electrical signal, or vice versa. In another embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0043] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to another embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, a magnetometer, or an illuminance sensor.

[0044] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0045] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0046] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In another embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0047] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0048] The power management module (188) can manage the power supplied to the electronic device (101). In one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0049] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0050] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. For example, the communication module (190) may include one or more communication processors that operate independently from the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0051] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0052] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to another embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. Furthermore, according to another embodiment, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0053] According to some embodiments, the antenna module (197) may form a mmWave antenna module. The mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0054] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0055] Commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to another embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service by itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). For example, the electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. The external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0056] FIG. 2A illustrates an example of a perspective view of a wearable device according to an embodiment of the present disclosure. FIG. 2B illustrates an example of one or more hardware components arranged within a wearable device according to an embodiment of the present disclosure.

[0057] The wearable device (103) may have the form of glasses that can be worn on a body part of the user (e.g., head). The wearable device (103) of FIGS. 2A and 2B may be an example of the electronic device (101) of FIG. 1. The wearable device (103) may include a head-mounted display (HMD). For example, the housing of the wearable device (103) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). The housing of the wearable device (103) may include one or more straps that can be twined around the user's head, and / or one or more temples that can be detachably attached to the ears of the head.

[0058] Referring to FIG. 2A, according to one embodiment, a wearable device (103) may include at least one display (250) and a frame (200) supporting at least one display (250).

[0059] The wearable device (103) can be worn on a part of the user's body. The wearable device (103) can provide extended reality (XR) to the user wearing the wearable device (103). For example, the extended reality can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality. For example, the wearable device (103) can display a virtual reality image provided from at least one optical device (282, 284) of FIG. 2B on at least one display (250) in response to a user's designated gesture acquired through the motion recognition cameras (260-2, 263) of FIG. 2B.

[0060] In another embodiment, at least one display (250) may provide visual information to a user. For example, at least one display (250) may include a transparent or translucent lens. At least one display (250) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0061] Referring to FIG. 2B, at least one display (250) can provide a user with visual information transmitted from external light and other visual information distinct from the visual information through a lens included in the at least one display (250). The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. The at least one display (250) can include a first surface (231) and a second surface (232) opposite to the first surface (231). A display area can be formed on the second surface (232) of the at least one display (250). When a user wears the wearable device (103), external light can be transmitted to the user by being incident on the first surface (231) and transmitted through the second surface (232). For example, at least one display (250) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (282, 284) on a real screen transmitted through external light, in a display area formed on the second surface (232).

[0062] At least one display (250) may include at least one waveguide (233, 234) that diffracts light emitted from at least one optical device (282, 284) and transmits the diffracted light to a user. The at least one waveguide (233, 234) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (233, 234). For example, the nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (233, 234) may be propagated to the other end of the at least one waveguide (233, 234) by the nano-pattern. At least one waveguide (233, 234) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (233, 234) may be arranged within the wearable device (103) to guide a screen displayed by at least one display (250) to the user's eyes. In another example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (233, 234).

[0063] The wearable device (103) can analyze an object included in a real image collected through a shooting camera (260-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (250). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. For example, the wearable device (103) can analyze an object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (103) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using a multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (103) can view an image displayed on at least one display (250).

[0064] In one embodiment, the frame (200) may be configured as a physical structure that allows the wearable device (103) to be worn on the user's body. In another embodiment, the frame (200) may be configured so that, when the user wears the wearable device (103), the first display (250-1) and the second display (250-2) can be positioned corresponding to the user's left and right eyes. The frame (200) may support at least one display (250). For example, the frame (200) may support the first display (250-1) and the second display (250-2) to be positioned corresponding to the user's left and right eyes.

[0065] Referring to FIG. 2A, the frame (200) may include a region (220) that is at least partially in contact with a part of the user's body when the user wears the wearable device (103). For example, the region (220) of the frame (200) that is in contact with a part of the user's body may include a region that is in contact with a part of the user's nose, a part of the user's ear, and a part of the side of the user's face that the wearable device (103) makes contact with. The frame (200) may include a nose pad (210) that is in contact with a part of the user's body. When the wearable device (103) is worn by the user, the nose pad (210) may be in contact with a part of the user's nose. The frame (200) may include a first temple (204) and a second temple (205) that are in contact with another part of the user's body that is distinct from the part of the user's body.

[0066] The frame (200) may include a first rim (201) that surrounds at least a portion of the first display (250-1), a second rim (202) that surrounds at least a portion of the second display (250-2), a bridge (203) that is disposed between the first rim (201) and the second rim (202), a first pad (211) that is disposed along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) that is disposed along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) that extends from the first rim (201) and is fixed to a portion of the wearer's ear, and a second temple (205) that extends from the second rim (202) and is fixed to a portion of the ear opposite the ear. For example, the first pad (211) and the second pad (212) may be in contact with a portion of the user's nose, and the first temple (204) and the second temple (205) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through the first hinge unit (206) disposed between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through the second hinge unit (207) disposed between the second rim (202) and the second temple (205). In another embodiment, the wearable device (103) may use a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (200) to identify an external object (e.g., a user's fingertip) touching the frame (200) and / or a gesture performed by the external object.

[0067] The wearable device (103) may include hardwares that perform various functions (e.g., hardwares to be described later based on the block diagram of FIG. 5). The hardwares may include a battery module (270), an antenna module (275), at least one optical device (282, 284), speakers (e.g., speakers 255-1, 255-2), a microphone (e.g., microphones 265-1, 265-2, 265-3), a light-emitting module (not shown), and / or a printed circuit board (PCB) (290) (e.g., a printed circuit board). The various hardwares may be arranged within the frame (200).

[0068] In one embodiment, the microphones (e.g., microphones 265-1, 265-2, 265-3) of the wearable device (103) may be disposed on at least a portion of the frame (200) to acquire sound signals. A first microphone (265-1) disposed on the bridge (203), a second microphone (265-2) disposed on the second rim (202), and a third microphone (265-3) disposed on the first rim (201) are illustrated in FIG. 2B , but the number and arrangement of the microphones (265) are not limited to the embodiment of FIG. 2B . When the number of microphones (265) included in the wearable device (103) is two or more, the wearable device (103) may use multiple microphones disposed on different portions of the frame (200) to identify the direction of the sound signal.

[0069] In another embodiment, at least one optical device (282, 284) may project a virtual object onto at least one display (250) to provide various image information to a user. For example, at least one optical device (282, 284) may be a projector. At least one optical device (282, 284) may be disposed adjacent to at least one display (250) or may be included within at least one display (250) as a part of at least one display (250). In yet another embodiment, the wearable device (103) may include a first optical device (282) corresponding to a first display (250-1) and a second optical device (284) corresponding to a second display (250-2). For example, at least one optical device (282, 284) may include a first optical device (282) disposed at an edge of a first display (250-1) and a second optical device (284) disposed at an edge of a second display (250-2). The first optical device (282) may transmit light to a first waveguide (233) disposed on the first display (250-1), and the second optical device (284) may transmit light to a second waveguide (234) disposed on the second display (250-2).

[0070] The camera (260) may include a recording camera (260-4), an eye tracking camera (ET CAM) (260-1), and / or a motion recognition camera (260-2, 260-3). The recording camera (260-4), the eye tracking camera (260-1), and the motion recognition cameras (260-2, 260-3) may be positioned at different positions on the frame (200) and may perform different functions. The eye tracking camera (260-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (103). The wearable device (103) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (260-1). For example, the wearable device (103) can perform gaze interaction with at least one object using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (103) can express a portion corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (103) can render an image (or screen) displayed on at least one display (250) based on the position of the user's eyes. For example, the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI) of a first region related to the gaze and a second region distinguished from the first region within the image may be different from each other. For another example, if the wearable device (103) supports an iris recognition function, user authentication can be performed based on iris information acquired using the gaze tracking camera (260-1).An example in which the gaze tracking camera (260-1) is positioned toward the user's right eye is illustrated in FIG. 2B, but the embodiment is not limited thereto, and the gaze tracking camera (260-1) may be positioned solely toward the user's left eye, or may be positioned toward both eyes.

[0071] In one embodiment, the capturing camera (260-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (260-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (250). For example, the at least one display (250) can display a single image in which information about an actual image or background including an image of the specific object acquired using the capturing camera (260-4) is superimposed on a virtual image provided through at least one optical device (282, 284). The wearable device (103) can compensate for depth information (e.g., the distance between the wearable device (103) and an external object acquired using a depth sensor) using the image acquired using the capturing camera (260-4). For example, the wearable device (103) can perform object recognition using the image acquired using the capturing camera (260-4). The wearable device (103) may perform a pass-through function to display an image acquired through a photographing camera (260-4) by overlaying it on at least a portion of a screen representing a virtual space on at least one display (250). In one embodiment, the photographing camera (260-4) may be positioned on a bridge (203) positioned between a first rim (201) and a second rim (202).

[0072] The gaze tracking camera (260-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (103) and matching the user's gaze with visual information provided to at least one display (250). For example, when the wearable device (103) looks straight ahead, the wearable device (103) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (250). The gaze tracking camera (260-1) can be configured to capture an image of the user's pupil to determine the user's gaze. The gaze tracking camera (260-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In another embodiment, the gaze tracking camera (260-1) can be placed at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (260-1) may be positioned within the first rim (201) and / or the second rim (202) to face the direction in which the user wearing the wearable device (103) is positioned.

[0073] The gesture recognition camera (260-2, 260-3) can provide a specific event on a screen provided on at least one display (250) by recognizing the movement of the user's entire body or a part of the user's body, such as the user's torso, hand, or face. For example, the gesture recognition camera (260-2, 260-3) can recognize the user's gesture, obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (250). The wearable device (103) can identify the signal corresponding to the gesture, and perform a designated function based on the identification. The gesture recognition camera (260-2, 260-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 dof pose). The wearable device (103) can perform gesture recognition and / or object tracking functions using motion recognition cameras (260-2, 260-3). The motion recognition cameras (260-2, 260-3) can be placed on the first rim (201) and / or the second rim (202).

[0074] The camera (260) included in the wearable device (103) is not limited to the above-described gaze tracking camera (260-1) and motion recognition cameras (260-2, 260-3). For example, the wearable device (103) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (103) can identify an external object based on a sensor for identifying the distance between the wearable device (103) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (260) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. The wearable device (103) may include a camera (260) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (103).

[0075] Although not shown, in one embodiment, the wearable device (103) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (260). The light source may include an infrared wavelength light emitting diode (LED). The light source may be disposed in at least one of the frame (200) and the hinge units (206, 207).

[0076] The battery module (270) can supply power to electronic components of the wearable device (103). In one embodiment, the battery module (270) can be disposed within the first temple (204) and / or the second temple (205). For example, the battery module (270) can be a plurality of battery modules (270). The plurality of battery modules (270) can be disposed within each of the first temple (204) and the second temple (205). In another embodiment, the battery module (270) can be disposed at an end of the first temple (204) and / or the second temple (205).

[0077] The antenna module (275) can transmit signals or power to the outside of the wearable device (103), or receive signals or power from the outside. In one embodiment, the antenna module (275) can be positioned within the first temple (204) and / or the second temple (205). For example, the antenna module (275) can be positioned close to one surface of the first temple (204) and / or the second temple (205).

[0078] The speaker (255) can output an audio signal to the outside of the wearable device (103). The audio output module may be referred to as a speaker. In one embodiment, the speaker (255) may be positioned within the first temple (204) and / or the second temple (205) so as to be positioned adjacent to the ear of a user wearing the wearable device (103). The speaker (255) may include a second speaker (255-2) positioned within the first temple (204) and thus positioned adjacent to the user's left ear, and a first speaker (255-1) positioned within the second temple (205) and thus positioned adjacent to the user's right ear.

[0079] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (103) to the user. When the wearable device (103) requires charging, the wearable device (103) may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (201) and / or the second rim (202).

[0080] Referring to FIG. 2B, according to one embodiment, the wearable device (103) may include a printed circuit board (PCB) (290). The PCB (290) may be included in at least one of the first temple (204) or the second temple (205). For example, the PCB (290) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (103) (e.g., hardwares illustrated by different blocks in FIG. 5) may be disposed on the PCB (290). The wearable device (103) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0081] According to one embodiment, the wearable device (103) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (103) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (103). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (103) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (103) based on the IMU.

[0082]

[0083] FIGS. 3A and 3B illustrate examples of the appearance of a wearable device according to various embodiments of the present disclosure.

[0084] The wearable device (103) of FIGS. 3A and 3B may be an example of the electronic device (101) of FIG. 1. According to one embodiment, an example of the appearance of a first side (310) of a housing of the wearable device (103) may be illustrated in FIG. 3A, and an example of the appearance of a second side (320) opposite to the first side (310) may be illustrated in FIG. 3B.

[0085] Referring to FIG. 3A, according to one embodiment, a first surface (310) of a wearable device (103) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (103) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (204) and / or the second temple (205) of FIGS. 2A and 2B). A first display (250-1) for outputting an image to a left eye among the user's two eyes, and a second display (250-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (310). The wearable device (103) is formed on the first surface (310) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (250-1) and the second display (250-2).

[0086] The wearable device (103) may include cameras (260-1) for photographing and / or tracking the user's two eyes adjacent to each of the first display (250-1) and the second display (250-2). The cameras (260-1) may be referred to as the eye tracking camera (260-1) of FIG. 2B. According to one embodiment, the wearable device (103) may include cameras (260-5, 260-6) for photographing and / or recognizing the user's face. The cameras (260-5, 260-6) may be referred to as FT cameras. The wearable device (103) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (260-5, 260-6).

[0087] Referring to FIG. 3b, a camera (e.g., cameras (260-7, 260-8, 260-9, 260-10, 260-11, 260-12)) and / or a sensor (e.g., a depth sensor (330)) for obtaining information related to the external environment of the wearable device (103) may be disposed on a second surface (320) opposite to the first surface (310) of FIG. 3a. For example, the cameras (260-7, 260-8, 260-9, 260-10) may be disposed on the second surface (320) to recognize external objects. Cameras (260-7, 260-8, 260-9, 260-10) may be referenced to the motion recognition cameras (260-2, 260-3) of FIG. 2B.

[0088] Using cameras (260-11, 260-12), the wearable device (103) can acquire images and / or videos to be transmitted to each of the user's eyes. Camera (260-11) can be placed on the second face (320) of the wearable device (103) to acquire an image to be displayed through the second display (250-2) corresponding to the right eye among the two eyes. Camera (260-12) can be placed on the second face (320) of the wearable device (103) to acquire an image to be displayed through the first display (250-1) corresponding to the left eye among the two eyes. Cameras (260-11, 260-12) can be referred to as the shooting camera (260-4) of FIG. 2B.

[0089] In one embodiment, the wearable device (103) may include a depth sensor (330) disposed on the second face (320) to identify a distance between the wearable device (103) and an external object. Using the depth sensor (330), the wearable device (103) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (103). Although not shown, a microphone may be disposed on the second face (320) of the wearable device (103) to obtain sound output from an external object. The number of microphones may be one or more depending on the embodiment.

[0090] The wearable device (103) can recognize the movement of a body part (e.g., a head part) on which the wearable device (103) is worn. For example, the movement of the body part can be recognized based on a value obtained using at least one sensor of the wearable device (103). As another example, the value can include a sensor value indicating the movement. Recognizing the movement of the body part can be referred to as head tracking (HeT) technology.

[0091] In order to accurately perform the head tracking, the at least one sensor of the wearable device (103) may include a geomagnetic sensor, an acceleration sensor, a gyro sensor, or an image sensor (or a camera). In an example, the at least one sensor may include an IMU sensor including the geomagnetic sensor, the acceleration sensor, and the gyro sensor. A technology for analyzing an image acquired using the image sensor and extracting information (or vision information) within the image may be referred to as a vision technology.

[0092] When a user wearing a wearable device (103) is riding a vehicle such as an airplane or a car and the vehicle moves, the results of the tracking information identified through the IMU sensor may be different from the results of the tracking information identified through the image sensor. The values ​​(e.g., acceleration information or rotation information) acquired by the at least one sensor may indicate a change amount due to the movement of the vehicle, not a change amount due to the movement of the user (or the movement of the wearable device (103). Accordingly, the screen displayed by the wearable device (103) may change, contrary to the user's intention. The change of the screen that occurs regardless of the user's intention may be referred to as tracking drift. In contrast, if the user wearing the wearable device (103) is walking without a means of transportation, even if the user moves, the tracking drift may not occur because the results of the tracking information identified through the IMU sensor and the results of the tracking information identified through the image sensor are the same.

[0093] In addition, when correcting the tracking of the wearable device (103) using the sensor information and the vision information of the external electronic device, if the movements of the wearable device (103) and the external electronic device are different from each other, it may be difficult to accurately correct the tracking. Depending on the difference between the timing at which the sensor information of the external electronic device is acquired and the timing at which the tracking is corrected, the accuracy of the correction may be reduced. In addition, there is a problem of increased power consumption when continuously receiving the sensor information from the external electronic device.

[0094] Hereinafter, the device and method according to embodiments of the present disclosure can correct (or improve) the tracking drift by correcting the value acquired by the wearable device (103) using sensor information of an external electronic device connected to the wearable device (103). The device and method according to embodiments of the present disclosure can reduce the power consumption by obtaining the sensor information that satisfies a specified condition from the external electronic device. When the external electronic device is a plurality of external electronic devices, the device and method according to embodiments of the present disclosure can determine an application ratio for the correction according to the similarity between the movement of each of the plurality of external electronic devices and the movement of the wearable device (103), and perform the correction using the determined application ratio. Accordingly, the device and method according to embodiments of the present disclosure can more accurately perform correction for the tracking drift and perform tracking (e.g., HeT) of a body part of a user.

[0095] FIGS. 4A and 4B illustrate examples of a method for improving tracking drift using external electronic devices according to various embodiments of the present disclosure.

[0096] The wearable device (103) of FIGS. 4A and 4B may represent an example of the electronic device (101) of FIG. 1 and the wearable device (103) of FIGS. 2A, 2B, 3A, and 3B. In the examples (400, 450) of FIGS. 4A and 4B, the wearable device (103) provides an augmented reality (AR) environment, but the embodiments of the present disclosure are not limited thereto. For example, the wearable device (103) may provide an extended reality (XR) environment including a virtual reality (VR) environment.

[0097] Referring to FIGS. 4A and 4B , examples (400, 450) illustrate a case where a user (403) located within a vehicle uses a wearable device (103). For example, the vehicle may include an airplane or a vehicle. The wearable device (103) may be worn on a body part of the user (403). For example, the body part of the user (403) may include a head part of the user (403). The wearable device (103) may be connected to a first external electronic device (401) and a second electronic device (402). For another example, the first external electronic device (401) may represent an example of the electronic device (102) of FIG. 1 . For another example, the second external electronic device (402) may represent an example of the electronic device (104) of FIG. 1 . For example, the second external electronic device (402) may be a wearable device worn on another body part of the user (403). For example, the other body part may include a wrist part of the user (403).

[0098] Example (400) of FIG. 4A illustrates a case where the tracking drift does not occur due to the movement of the vehicle. For example, the user (403) may be wearing a wearable device (103) and a first external electronic device (401). The second external electronic device (402) may be positioned on an external object. For example, the second external electronic device (402) may not be affected by the movement of the user (403) when the user (403) is not wearing it. The movement (or change in position and posture) of the second external electronic device (402) may be caused by an external factor other than the user (403). For example, the external object may include a shelf within the vehicle. Referring to example (400), the wearable device (103) may provide an XR environment (405) through a screen (410). A user (403) can use a service through an XR environment (405) using a wearable device (103). For example, the wearable device (103) can display a virtual object (415) within a screen (410). For example, the virtual object (415) can be located in a portion of the screen (410), such as the center. For example, the virtual object (415) can include a web site. However, the embodiments of the present disclosure are not limited thereto.

[0099] Example (450) of FIG. 4B illustrates a case where the tracking drift occurs due to the movement of the moving means. If the movement of the moving means changes rapidly, the tracking drift may occur. For example, the wearable device (103), the first external electronic device (401), and the second external electronic device (402) may move according to the movement of the moving means. In another example, the wearable device (103) and the first external electronic device (401) may shake while being worn on a body part of the user (403). For example, the position of the second external electronic device (402) on the external object may change. At this time, the position of the virtual object (415) may change according to the movement of the moving means on the screen (460) provided through the wearable device (103). For another example, depending on the tracking drift occurring in the wearable device (103), the virtual object (415) may be positioned in another portion (e.g., the right) of the screen (460). The change from the portion of the screen (410) to the other portion of the screen (460) may be caused by the tracking drift. In other words, the wearable device (103) may recognize the movement of the means of transportation as the movement of the user (403), and display the virtual object (415) in the other portion of the changed portion based on the movement of the user (403).

[0100] The wearable device (103) can receive sensor information from at least one of the first external electronic device (401) or the second external electronic device (402). For example, the wearable device (103) can receive first sensor information acquired through at least one sensor of the first external electronic device (401) from the first external electronic device (401). For example, the first sensor information can include a first sensor value indicating movement of the first external electronic device (401) and information on the time at which the first sensor value was acquired. For example, the wearable device (103) can receive second sensor information acquired through at least one sensor of the second external electronic device (402) from the second external electronic device (402). For example, the second sensor information can include a second sensor value indicating movement of the second external electronic device (402) and information on the time at which the second sensor value was acquired. In the examples of FIGS. 4A and 4B , the wearable device (103) may receive the sensor information from the first external electronic device (401) and the second external electronic device (402), respectively, while being connected to the first external electronic device (401) and the second external electronic device (402). However, the embodiments of the present disclosure are not limited thereto. For example, the wearable device (103) may receive the first sensor information and the second sensor information together from the second external electronic device (402). The first external electronic device (401) may be connected to the second external electronic device (402). Specific details related thereto are described below in FIG. 8 .

[0101] According to one embodiment, the wearable device (103) can obtain a value indicating the movement of the wearable device (103) through at least one sensor of the wearable device (103). For example, the value indicating the movement of the wearable device (103) can represent a sensor value obtained through the at least one sensor. The wearable device (103) can obtain the value indicating the movement at a timing.

[0102] In one embodiment, the wearable device (103) may obtain a value indicating movement of the external electronic device expected (or predicted) at the timing by using time information included in sensor information obtained from the external electronic device. For example, the wearable device (103) may obtain a first value indicating movement of the first external electronic device (401) expected at the timing by using the time information of the first sensor information obtained from the first external electronic device (401). In addition, for example, the wearable device (103) may obtain a second value indicating movement of the second external electronic device (402) expected at the timing by using the time information of the second sensor information obtained from the second external electronic device (402). Specific details related thereto are described below with reference to FIGS. 6 and 7A.

[0103] According to another embodiment, the wearable device (103) can evaluate the similarity between the movement of the external electronic device and the movement of the wearable device (103). For example, the wearable device (103) can compare the first value indicating the movement of the first external electronic device (401) at the timing with the value indicating the movement of the wearable device (103). The wearable device (103) can obtain a first similarity value between the first value and the value. In addition, for example, the wearable device (103) can compare the second value indicating the movement of the second external electronic device (402) at the timing with the value indicating the movement of the wearable device (103). Accordingly, the wearable device (103) can obtain a second similarity value between the second value and the value. Specific details related thereto are described below with reference to FIGS. 6 and 7B.

[0104] According to another embodiment, the wearable device (103) may determine a ratio to be applied to the adjustment (or correction, compensation) of the value based on the acquired similarity value and the reference value. For example, the ratio may be referred to as an application ratio. For example, the wearable device (103) may determine to use the first value to adjust the value based on the first similarity value exceeding the reference value. In another example, the wearable device (103) may determine not to use the second value to adjust the value based on the second similarity value being less than the reference value. Accordingly, the wearable device (103) may perform the adjustment of the value using the first value. In other words, in the above example, the application ratio of the first value may be 1, and the application ratio of the second value may be 0.

[0105] An example is described in which adjustment of the value is performed using a certain value (e.g., the first value) based on a comparison of the reference value with the first value or the second value, but the embodiments of the present disclosure are not limited thereto. For example, when the second similarity value exceeds the reference value and the second similarity value exceeds the first similarity value, the wearable device (103) may use the first value and the second value to adjust the value. At this time, in the adjustment, the application ratio of the second value may be greater than the application ratio of the first value. When the application ratio of the first value is a first weight (or a first weight value) and the application ratio of the second value is a second weight (or a second weight value), the second weight (e.g., 0.7) may be greater than the first weight (e.g., 0.3). The second weight may be determined based on the difference between the reference value and the second similarity value, and the first weight may be determined based on the difference between the reference value and the first similarity value. Specific details related thereto are described below in FIGS. 6 and 7c.

[0106] As described above, the wearable device (103) can compensate for the tracking drift by adjusting the value representing the movement of the wearable device (103) using sensor information acquired from an external electronic device. Even when the means of transportation moves, as in example (450), the wearable device (103) can display the screen (410) instead of the screen (460) based on the adjustment. Even when a change in the external environment, such as when the means of transportation moves, occurs, the wearable device (103) can compensate for the tracking drift using the external electronic device, thereby displaying a screen such as the screen (410) in which there is no change in the external environment, instead of the screen (460) that would be displayed if there were no correction. The wearable device (103) can reduce the influence of the adjustment due to external factors that are unrelated to the intention of the user (403). Accordingly, the wearable device (103) can provide a more stable user experience to the user (403).

[0107] Referring to the above, examples (400, 450) are shown in which the tracking drift occurs in response to the movement of the transportation means when a user (403) wearing a wearable device (103) boards the transportation means, but the embodiments of the present disclosure are not limited thereto. For example, this may also be applied in cases where the user (403) does not board the transportation means. For example, the embodiments of the present disclosure may also be applied to the tracking drift occurring due to a malfunction of a sensor of the wearable device (103).

[0108] FIG. 5 illustrates a block diagram of a wearable device according to an embodiment of the present disclosure.

[0109] The wearable device (103) of FIG. 5 may be an example of the electronic device (101) of FIG. 1, the wearable devices (103) of FIGS. 2A, 2B, 3A, and 3B, or the wearable devices (103) of FIGS. 4A and 4B. The external electronic device (590) of FIG. 5 may be an example of the first external electronic device (401) or the second external electronic device (402) of FIGS. 4A and 4B.

[0110] Referring to FIG. 5, a situation is illustrated in which a wearable device (103) and an external electronic device (590) are connected to each other based on a wired network and / or a wireless network. The wired network may include a network such as the Internet, a local area network (LAN), a wide area network (WAN), or a combination thereof. For example, the wireless network may include a network such as long term evolution (LTE), 5g new radio (NR), wireless fidelity (WiFi), Zigbee, near field communication (NFC), Bluetooth, Bluetooth low-energy (BLE), or a combination thereof. Although the wearable device (103) and the external electronic device (590) are illustrated as being directly connected, the wearable device (103) and the external electronic device (590) may be indirectly connected via one or more routers and / or access points (APs).

[0111] Referring to FIG. 5, according to one embodiment, a wearable device (103) may include at least one of a processor (510), a display (520), a sensor (530), a communication circuit (540), and a memory (550). The processor (510), the display (520), the sensor (530), the communication circuit (540), and the memory (550) may be electrically and / or operably coupled with each other by a communication bus. The hardware components being operably coupled may mean that a direct connection or an indirect connection is established between the hardware components, either wired or wireless, such that a second hardware component is controlled by a first hardware component among the hardware components. Although illustrated based on different blocks, the embodiment is not limited thereto, and some of the hardware components illustrated in FIG. 5 (e.g., at least a portion of the processor (510), memory (550), and communication circuit (540)) may be included in a single integrated circuit such as a system on a chip (SoC). The type and / or number of hardware components included in the wearable device (103) is not limited to those illustrated in FIG. 5. For example, the wearable device (103) may include only some of the hardware components illustrated in FIG. 5.

[0112] The processor (510) of the wearable device (103) may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits including at least one processor, one or more of which may be configured to individually and / or collectively perform the various functions described below in a distributed manner. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms encompass, for example, and without limitation, situations where one processor performs some of the recited functions and other processor(s) perform other parts of the recited functions, and also situations where one processor may perform all of the recited functions. Additionally, the at least one processor may include a combination of processors that perform the various functions enumerated / disclosed, for example, in a distributed manner. At least one processor may execute program instructions to achieve or perform the various functions.

[0113] According to another embodiment, the processor (510) of the wearable device (103) may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), and a field programmable gate array (FPGA). As an example, the hardware component for processing data may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP), and / or a neural processing unit (NPU). The number of processors (510) may be one or more. For example, the processor (510) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core. The processor (510) of FIG. 5 may include the processor (120) of FIG. 1.

[0114] According to another embodiment, the display (520) of the wearable device (103) can output visualized information to the user. The number of displays (520) included in the wearable device (103) may be one or more. For example, the display (520) may be controlled by a processor (510) and / or a graphic processing unit (GPU) (not shown) to output visualized information to the user. For example, the display (520) may include a flat panel display (FPD) and / or electronic paper. The FPD may include a liquid crystal display (LCD), a plasma display panel (PDP), a digital mirror device (DMD), one or more light emitting diodes (LEDs), and / or micro LEDs. The LEDs may include organic LEDs (OLEDs). The display (520) of FIG. 5 may include the display module (160) of FIG. 1. For example, the display (520) of FIG. 5 can display a visual object as exemplified in FIG. 9.

[0115] The wearable device (103) may include a sensor (530). For example, the sensor (530) may include at least one sensor for measuring the movement of the wearable device (103). For example, the sensor may include an IMU (or IMU sensor). For example, the sensor may include a magnetometer, a gyro sensor, a gravity sensor, and / or an acceleration sensor. The sensor may include at least a portion of the sensor module (176) of FIG. 1.

[0116] In one embodiment, the communication circuit (540) of the wearable device (103) may include hardware for supporting transmission and / or reception of electrical signals between the wearable device (103) and an external electronic device (590). The communication circuit (540) may include, for example, at least one of a modem, an antenna, and an optical / electronic (O / E) converter. The communication circuit (540) may support transmission and / or reception of electrical signals based on various types of communication means, such as Ethernet, Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), and 5G new radio (NR). The communication circuit (540) of FIG. 5 may include the communication module (190) and / or the antenna module (197) of FIG. 1.

[0117] In another embodiment, the memory (550) of the wearable device (103) may include a hardware component for storing data and / or instructions input to and / or output from the processor (510). The memory (550) may include, for example, volatile memory such as random-access memory (RAM), and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multimedia card (eMMC). The memory (550) of FIG. 5 may include the memory (130) of FIG. 1.

[0118] Although not shown in FIG. 5, the wearable device (103) may include a camera. The camera may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. For example, the camera may be referred to as an image sensor. A plurality of optical sensors included in the camera may be arranged in the form of a two-dimensional array. The camera may acquire electrical signals from each of the plurality of optical sensors substantially simultaneously, and generate an image corresponding to light reaching the optical sensors of the two-dimensional array and including a plurality of pixels arranged two-dimensionally. The wearable device (103) may render an actual environment within the image acquired through the camera and display the rendered visual information. The number of cameras included in the wearable device (103) may be one or more, as described above with reference to FIGS. 2a and 2b, and / or FIGS. 3a and 3b.

[0119] Although not illustrated in FIG. 5, the wearable device (103) according to one embodiment may include an output means for outputting information in a form other than a visualized form. For example, the wearable device (103) may include a speaker for outputting an acoustic signal. As another example, the wearable device (103) may include a motor for providing haptic feedback based on vibration.

[0120] Referring to FIG. 5, according to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by a processor (510) of the wearable device (103) may be stored in the memory (550) of the wearable device (103). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or an application. Hereinafter, when an application is installed in an electronic device (e.g., a wearable device (103)), it may mean that one or more instructions provided in the form of an application are stored in the memory (550), and that the one or more applications are stored in a format executable by the processor of the electronic device (e.g., a file having an extension designated by the operating system of the wearable device (103)). The wearable device (103) can perform the operations of FIGS. 6 and 10 by executing one or more instructions stored in the memory (550).

[0121] Referring to FIG. 5, programs installed in the wearable device (103) may be classified into any one of different layers, including an application layer (560), a framework layer (570), and / or a hardware abstraction layer (HAL) (580), based on the target. For example, programs (e.g., drivers) designed to target the hardware of the wearable device (103) (e.g., the display (520), and / or the communication circuit (540)) may be classified within the hardware abstraction layer (580). Programs (e.g., sensor information reception module (571), sensor value prediction module (573), similarity verification module (575), application ratio determination module (577), and / or motion compensation module (579)) designed to target at least one of the hardware abstraction layer (580) and / or the application layer (560) may be classified within the framework layer (570). Programs classified as a framework layer (570) can provide an executable API (application programming interface) based on other programs.

[0122] Referring to FIG. 5, programs designed to target users controlling wearable devices (103) may be classified within the application layer (560). For example, a program classified within the application layer (560) may include at least one application that provides an XR environment. The embodiments of the present disclosure are not limited thereto. For example, a program classified within the application layer (560) may call an API to cause the execution of functions supported by programs classified within the framework layer (570).

[0123] Referring to FIG. 5, the wearable device (103) may receive sensor information from an external electronic device (590) based on the execution of the sensor information reception module (571) within the framework layer (570). For example, the wearable device (103) may request the sensor information from the external electronic device (590) when a specified condition is satisfied. For another example, the specified condition may include a case where an error occurs for a specified length of time or longer between vision information and a value indicating the movement of the wearable device (103) obtained using the sensor (530). The error may cause tracking drift. The specified condition may include a case where the movement of the wearable device (103) recognized based on the value of the sensor (530) is identified as an abnormal movement. The above abnormal movement may include a case where the movement of the wearable device (103) recognized based on the above value is not continuous. Or, for example, the above specified condition may include a case where the acceleration of the movement recognized based on the above value of the sensor (530) changes linearly. Or, for example, the above specified condition may include a case where the wearable device (103) (or a user wearing the wearable device (103)) is located within a means of transportation. For example, being located within the means of transportation may be recognized based on the wearable device (103) detecting vibration due to an external factor. Referring to the above, when the above specified condition is satisfied, the wearable device (103) may request the sensor information from the external electronic device (590). For example, the sensor information may include sensor values ​​and time information obtained through a sensor (593) of an external electronic device (590). For example, the sensor values ​​may include acceleration sensor values ​​or gyro sensor values.For example, the time information may include the timing (e.g., timestamp) at which the sensor value was acquired. The wearable device (103) and the external electronic device (590) may be synchronized with each other. When the sensor information is requested, the wearable device (103) may indicate an update cycle of the sensor information to be transmitted by the external electronic device (590). The update cycle may include a specified time interval (e.g., minutes, seconds) or a specified number of times. Alternatively, for example, the wearable device (103) may indicate an end point (timing) together with the update cycle. For example, the update cycle may be indicated for each sensor value. For example, when the amount of change in the acceleration of the wearable device (103) is large, the wearable device (103) may indicate a relatively short update cycle to the external electronic device (590) with respect to the acceleration sensor value. When the amount of change in the rotation information (or the value of the gyro sensor) of the wearable device (103) is small, the wearable device (103) can instruct the external electronic device (590) to have a relatively long update cycle with respect to the gyro sensor value.

[0124] According to one embodiment, the external electronic device (590) may transmit the sensor information to the wearable device (103) based on a comparison between the acquired sensor information and previous sensor information. For example, the external electronic device (590) may compare a difference between the sensor information acquired using the sensor (593) in the first period and another sensor information acquired using the sensor (593) in the second period prior to the first period with a reference difference. At this time, the other sensor information of the second period may be stored in a memory (not shown) of the external electronic device (590). At this time, the difference is the difference between a vector representing a sensor value of the sensor information (e.g., (x1, y1, z1)) and another vector representing a sensor value of the other sensor information (e.g., (x2, y2, z2)). may include. Or, for example, the difference may include a difference between a value of at least some axis (e.g., x-axis, y-axis, or z-axis) of the vector and a value of at least some axis of the other vector. Or, the external electronic device (590) may transmit the sensor information to the wearable device (103) when a value of at least some axis (e.g., x-axis, y-axis, or z-axis) of the vector is outside a specified range.

[0125] The sensor value included in the sensor information may be in text format or binary format. For example, the text format may include Plain, JSON, and XML. For example, the binary format may include Protobuf. When the connection status between the wearable device (103) and the external electronic device (590) is poor (e.g., data delay is detected), the data including the sensor value may be compressed and transmitted. In the above example, a compression transmission method of the data used when the connection status between the wearable device (103) and the external electronic device (590) is poor is described, but the embodiments of the present disclosure are not limited thereto. For example, the data transmitted and received between the wearable device (103) and the external electronic device (590) may be compressed and transmitted regardless of the connection status.

[0126] By transmitting and receiving the sensor information based on the comparison between the above difference and the above reference difference and / or the update cycle indicated by the wearable device (103), the power consumption of the wearable device (103) and the external electronic device (590) can be reduced.

[0127] According to another embodiment, the wearable device (103) may receive (or acquire) the sensor information from an external electronic device (590) identified according to a priority based on the execution of the sensor information receiving module (571). For example, the priority may indicate an order among one or more external electronic devices for requesting and receiving the sensor information. According to another embodiment, the priority may be determined by a user's input. For example, the wearable device (103) may set the priority among the one or more external electronic devices according to the user's state (e.g., walking state or riding in a means of transportation) based on the user's input. For example, the wearable device (103) may request the sensor information from an external electronic device having the highest priority among the one or more external electronic devices, and receive the sensor information in response to the request. The priority may be determined based on record information. For example, the wearable device (103) may determine the priority by using the record information including similarity values ​​for movements between the wearable device (103) and the one or more external electronic devices. For example, the record information may include the similarity values ​​prior to the current point in time. The priority may be determined according to the connection order between the one or more external electronic devices. For example, the wearable device (103) may determine the priority corresponding to the connection order between each of the one or more external electronic devices and the wearable device (103).

[0128] Referring to FIG. 5, the wearable device (103) may predict a sensor value at a specified timing based on the execution of the sensor value prediction module (573) within the framework layer (570). For example, the sensor value may be calculated based on sensor information acquired from an external electronic device (590) and may represent a movement of the external electronic device (590) predicted at the specified timing. The prediction may be to compensate for an error between the timing at which the external electronic device (590) acquires the sensor information and the timing at which the wearable device (103) acquires the value. The error may include a time delay while the sensor information is transmitted from the external electronic device (590) to the wearable device (103), or a time delay while the wearable device (103) acquires the value.

[0129] For example, the specified timing may indicate a timing at which the wearable device (103) acquires a value indicating the movement of the wearable device (103) using the sensor (530). The wearable device (103) may predict the sensor value at the specified timing using the sensor information acquired within the first period and the other sensor information acquired and stored within the second period prior to the first period. The specified timing may be included within the first period. For example, the predicted sensor value may be acquired (or calculated) using a computational algorithm or a statistical model (e.g., an artificial intelligence (AI) model, machine learning, or deep learning). Specific details related thereto are described below with reference to FIGS. 6 and 7A.

[0130] Referring to FIG. 5, the wearable device (103) can determine the similarity between the movement of the external electronic device (590) and the movement of the wearable device (103) based on the execution of the similarity determination module (575) within the framework layer (570). The wearable device (103) can calculate a similarity value indicating the similarity between the value acquired at the specified timing and the predicted sensor value. The similarity value can be acquired (or calculated) using a calculation algorithm or a statistical model. For example, the calculation algorithm can include a difference between a vector according to the value and a vector according to the predicted sensor value, or a cosine value between vectors. Specific details related thereto are described below with reference to FIGS. 6 and 7b.

[0131] Referring to FIG. 5, the wearable device (103) may determine whether to apply (or the application ratio) the sensor information of the external electronic device (590) based on the similarity value, based on the execution of the application ratio determination module (577) within the framework layer (570). For example, the wearable device (103) may determine whether to apply or the application ratio of the sensor information of the external electronic device (590) based on a comparison between the similarity value and a reference value. If the similarity value is less than or equal to the reference value, the wearable device (103) may determine not to use the sensor information to adjust (or correct, compensate) the value. Alternatively, for example, if the similarity value exceeds the reference value, the wearable device (103) may determine to use the sensor information to adjust (or correct, compensate) the value. When the first similarity value exceeds the reference value and the second similarity value exceeds the reference value, the wearable device (103) may determine to use first sensor information related to the first similarity value and second sensor information related to the second similarity value to adjust (or correct, compensate) the values. When the first sensor information and the second sensor information are used, an application ratio between the first sensor information and the second sensor information may be determined based on a first difference between the first similarity value and the reference value and a second difference between the second similarity value and the reference value. For example, when the first difference is greater than the second difference, the application ratio (or first weight) of the first sensor information may be greater than the application ratio (or second weight) of the second sensor information.

[0132] Referring to FIG. 5, the wearable device (103) may adjust the value based on the application ratio based on the execution of the motion compensation module (579) within the framework layer (570). For example, the wearable device (103) may perform an offset (or compensation, cancellation) between the predicted sensor value of the sensor information and the value. The wearable device (103) may compensate for the movement of the wearable device (103) based on information indicating the movement of the external electronic device (590) (e.g., the predicted sensor value). Specific details related thereto are described below in FIG. 6 and FIG. 7c. Tracking drift occurring with respect to the wearable device (103) may be eliminated or reduced.

[0133] In one embodiment, the wearable device (103) may display a screen according to the above correction of the movement of the wearable device (103). For example, the screen may be displayed with the tracking drift corrected, such as the screen (410) of FIG. 4A, unlike the screen (460) of FIG. 4B.

[0134] The external electronic device (590) of FIG. 5 may include a processor (591), a sensor (593), and a communication circuit (595). For example, the external electronic device (590) may be directly connected to the wearable device (103) using the communication circuit (595). The external electronic device (590) may be indirectly connected to the wearable device (103) via a server (not shown) using the communication circuit (595). The description of the communication circuit (595) may be substantially identical to the description of the communication circuit (540). The description of the processor (591) may be substantially identical to the description of the processor (510). The description of the sensor (593) may be substantially identical to the description of the sensor (530). Although not illustrated in FIG. 5, the external electronic device (590) may further include at least one of a display, a camera, and a memory.

[0135] FIG. 5 illustrates an example in which one external electronic device (590) is connected to a wearable device (103), but the embodiments of the present disclosure are not limited thereto. For example, the wearable device (103) may be connected to a plurality of external electronic devices, including the external electronic device (590) and another external electronic device. The wearable device (103) may use sensor information of the plurality of external electronic devices to correct the tracking drift. The wearable device (103) may also use sensor information of the external electronic device (590) and another external electronic device connected to the external electronic device (590) to correct the tracking drift. In this case, the other external electronic device may be connected to the external electronic device (590), but not connected to the wearable device (103).

[0136] FIG. 6 illustrates an example of an operational flow for a method of adjusting a value indicating movement of a wearable device using sensor information obtained from an external electronic device according to an embodiment of the present disclosure.

[0137] FIG. 7A illustrates an example of a method for predicting a value indicating the movement of an external electronic device using sensor information acquired from the external electronic device according to an embodiment of the present disclosure. FIG. 7B illustrates an example of a method for calculating a similarity value between the movement of a wearable device and the movement of the external electronic device according to an embodiment of the present disclosure. FIG. 7C illustrates an example of a method for adjusting a value indicating the movement of a wearable device according to an embodiment of the present disclosure.

[0138] At least some of the methods of FIG. 6 may be performed by the wearable device (103) of FIG. 5. For example, at least some of the methods may be controlled by the processor (510) of the wearable device (103). In the embodiments described below, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0139] In operation (600), the wearable device (103) may receive sensor information from an external electronic device (590). For example, the sensor information may include a sensor value acquired by the external electronic device (590) through a sensor (593) of the external electronic device (590) and information on the time at which the sensor value was acquired. The sensor value may include a value indicating the movement of the external electronic device (590). For example, the sensor (593) may include an acceleration sensor or a gyro sensor. For example, the time information may include a timing (e.g., a timestamp) at which the sensor value was acquired. The wearable device (103) and the external electronic device (590) may be synchronized with each other.

[0140] According to another embodiment, for example, the wearable device (103) may establish a connection with an external electronic device (590) via a communication circuit (540). For example, the wearable device (103) may transmit a signal requesting the sensor information to the external electronic device (590) when a specified condition is satisfied.

[0141] The above-mentioned specified condition may include a case where an error occurs for a specified length of time between vision information acquired through an image sensor (or camera) of the wearable device (103) and a value representing the movement of the wearable device (103) acquired using the sensor (530). The above-mentioned error may be referred to as tracking drift. The above-mentioned specified condition may include a case where the movement of the wearable device (103) recognized based on the value of the sensor (530) is identified as an abnormal movement. The abnormal movement may include a case where the movement of the wearable device (103) recognized based on the value is not continuous. Alternatively, for example, the above-mentioned specified condition may include a case where the acceleration of the movement recognized based on the value of the sensor (530) changes linearly. Alternatively, for example, the specified condition may include a case where the wearable device (103) (or a user wearing the wearable device (103)) is located within a vehicle. For example, being located within the vehicle may be recognized based on the wearable device (103) detecting vibration due to an external factor.

[0142] According to another embodiment, when requesting the sensor information, the wearable device (103) may indicate an update cycle of the sensor information to be transmitted by the external electronic device (590). For example, the update cycle may include a specified time interval (e.g., minutes, seconds) or a specified number of times. The wearable device (103) may indicate an end time (timing) together with the update cycle. For example, the update cycle may be indicated for each sensor value. For example, when the amount of change in acceleration of the wearable device (103) is large, the wearable device (103) may indicate an update cycle of a relatively short cycle to the external electronic device (590) with respect to the acceleration sensor value. When the amount of change in the rotation information (or the value of the gyro sensor) of the wearable device (103) is small, the wearable device (103) can instruct the external electronic device (590) to have a relatively long update cycle with respect to the gyro sensor value.

[0143] According to another embodiment, the external electronic device (590) may transmit the sensor information to the wearable device (103) based on a comparison between the acquired sensor information and previous sensor information. For example, the external electronic device (590) may compare a difference between the sensor information acquired using the sensor (593) in a first period and another sensor information acquired using the sensor (593) in a second period prior to the first period with a reference difference. At this time, the other sensor information of the second period may be stored in the memory of the external electronic device (590). At this time, the difference is the difference between a vector representing a sensor value of the sensor information (e.g., (x1, y1, z1)) and another vector representing a sensor value of the other sensor information (e.g., (x2, y2, z2)). may include. Or, for example, the difference may include a difference between a value of at least some axis (e.g., x-axis, y-axis, or z-axis) of the vector and a value of at least some axis of the other vector. Or, the external electronic device (590) may transmit the sensor information to the wearable device (103) when a value of at least some axis (e.g., x-axis, y-axis, or z-axis) of the vector is outside a specified range.

[0144] According to one embodiment, the sensor value included in the sensor information may be in a text format or a binary format. The text format may include Plain, JSON, or XML. For example, the binary format may include Protobuf. In cases where the connection between the wearable device (103) and the external electronic device (590) is poor (e.g., data delay is detected), the data including the sensor value may be compressed and transmitted.

[0145] By transmitting and receiving the sensor information based on the comparison between the above difference and the above reference difference and / or the update cycle indicated by the wearable device (103), the power consumption of the wearable device (103) and the external electronic device (590) can be reduced.

[0146] In the above example, an example is described for one external electronic device (590), but the present disclosure is not limited thereto. For example, the wearable device (103) may also receive the sensor information from one or more external electronic devices.

[0147] In one embodiment, the wearable device (103) may receive (or acquire) the sensor information from an external electronic device (590) identified according to a priority. For example, the priority may indicate an order among the one or more external electronic devices for requesting and receiving the sensor information. According to another embodiment, the priority may be determined by a user's input. For example, the wearable device (103) may set the priority among the one or more external electronic devices according to the user's state (e.g., walking state or riding in a means of transportation) based on the user's input. For example, the wearable device (103) may request the sensor information from an external electronic device with the highest priority among the one or more external electronic devices, and receive the sensor information in response to the request. Alternatively, according to one embodiment, the priority may be determined based on record information. For example, the wearable device (103) may determine the priority by using the record information including similarity values ​​for movements between the wearable device (103) and the one or more external electronic devices. For example, the record information may include the similarity values ​​before the current point in time. The priority may be determined according to the connection order between the one or more external electronic devices. The wearable device (103) may determine the priority corresponding to the connection order between each of the one or more external electronic devices and the wearable device (103).

[0148] In operation (605), the wearable device (103) may obtain a value indicating the movement of the wearable device (103) using at least one sensor (530). For example, the wearable device (103) may measure the value indicating the movement of the wearable device (103) using the sensor (530) at a specified timing. For example, the specified timing may be included in the first period and may be a timing after the timing at which the sensor information is received. For example, the sensor (530) may include an IMU sensor.

[0149] In operation (610), the wearable device (103) may predict a value indicating the movement of the external electronic device (590) based on the sensor information. For example, the wearable device (103) may calculate (or generate) a sensor value predicted at the specified timing using the sensor information obtained in the first period and the other sensor information obtained in the second period. For another example, the predicted sensor value may be a value indicating the movement of the external electronic device (590). The prediction may be to compensate for an error between the timing at which the external electronic device (590) obtains the sensor information and the timing at which the wearable device (103) obtains the value. The error may include a time delay while the sensor information is transmitted from the external electronic device (590) to the wearable device (103), or a time delay while the wearable device (103) obtains the value.

[0150] The predicted sensor values ​​can be obtained (or calculated) using a computational algorithm or statistical model (e.g., an artificial intelligence (AI) model, machine learning, or deep learning). For example, an example of a method for generating the predicted sensor values ​​can be referenced to FIG. 7a below.

[0151] Referring to FIG. 7A, an example (700) of a method for calculating the predicted sensor value at a timing (712) (T2) within a first cycle (710) by using the sensor information acquired in the first cycle (710) and the other sensor information acquired in the second cycle (720) is illustrated. The timing (712) (T2) may be referred to as the specified timing.

[0152] Referring to example (700), the wearable device (103) can receive the sensor information from the external electronic device (590) at the timing (711) (T1) of the first cycle (710). For example, the sensor information can include a sensor value acquired by the external electronic device (590) before the timing (711) (T1) and information on the time at which the sensor value was acquired. The wearable device (103) can acquire a value indicating the movement of the wearable device (103) using the sensor (530) of the wearable device (103) at the timing (712) (T2). The wearable device (103) can predict the movement of the external electronic device (590) at the timing (712) (T2). For example, the wearable device (103) can calculate the predicted sensor value representing the movement of the external electronic device (590) at timing (712) (T2). For example, the predicted sensor value can be calculated by an arithmetic algorithm such as the following mathematical equation.

[0153]

[0154] The above mathematical formula 1 is merely an example to aid understanding, and embodiments of the present disclosure may not be limited thereto. For example, the above mathematical formula 1 may be modified, applied, or expanded in various ways.

[0155] For example, the above S T2 is the predicted sensor value at timing (712) (T2), S T1 The sensor value of the sensor information received at the timing (711) (T1) is S T3 The sensor value of the other sensor information received at the timing (721) (T3) is T1 The time information of the sensor information received at the timing (711) (T1), T T3 The time information of the other sensor information received at the timing (721) (T3), T T2 may indicate the timing at which the above value representing the movement of the wearable device (103) is acquired.

[0156] According to one embodiment, the predicted sensor value may be calculated using the sensor information acquired at the timing (711) (T1) of the first cycle (710) and the other sensor information acquired at the timing (721) (T3) of the second cycle (720). For example, the wearable device (103) may receive the other sensor information from the external electronic device (590) at the timing (721) (T3) of the second cycle (720) prior to the first cycle (710). The other sensor information may include other sensor values ​​acquired by the external electronic device (590) prior to the timing (721) (T3) and other time information at which the other sensor values ​​were acquired. The wearable device (103) may acquire other values ​​indicating movement of the wearable device (103) at the timing (722) (T4) of the second cycle (720). At timing (722) (T4) of the second cycle (720), the wearable device (103) may perform prediction. The prediction for timing (722) (T4) may be performed based on the other sensor information of the second cycle (720) and the sensor information received in the third cycle (not shown).

[0157] In another embodiment, the wearable device (103) may store the predicted sensor value. For example, the wearable device (103) may store the predicted sensor value at timing (712) (T2) in the memory (550). For example, the wearable device (103) may store the predicted sensor value at timing (722) (T4) in the memory (550).

[0158] In the above example, an example of calculating the predicted sensor value based on a computational algorithm such as the mathematical formula described above is described, but the embodiments of the present disclosure are not limited thereto. For example, the wearable device (103) may obtain (or calculate) the predicted sensor value using a computational algorithm or a statistical model (e.g., an artificial intelligence (AI) model, machine learning, or deep learning).

[0159] Referring back to FIG. 6, in operation (615), the wearable device (103) may calculate a similarity value between the movement of the wearable device (103) and the movement of the external electronic device (590). For example, the wearable device (103) may calculate the similarity value, which represents the similarity between the value acquired at the specified timing and the predicted sensor value. An example of a method for generating the similarity value may be referred to FIG. 7B below.

[0160] Referring to FIG. 7b, an example (730) of how a wearable device (103) calculates the similarity value is illustrated.

[0161] Referring to example (730), examples are shown of vectors (741, 742) for values ​​acquired through a sensor (530) of a wearable device (103) according to movement (730-3) of the wearable device (103), examples of vectors (751, 752) for sensor values ​​acquired through a sensor of a first external electronic device according to movement (730-1) of the first external electronic device, and examples of vectors (761, 762) for sensor values ​​acquired through a sensor of a second external electronic device according to movement (730-2) of the second external electronic device. The first external electronic device and the second external electronic device may represent an example of an external electronic device (590).

[0162] The values ​​acquired through the sensor (530) of the wearable device (103) can be vectorized with respect to a virtual point (740). For example, the point (740) can be (0, 0, 0). For example, the vector (741)(V 31 ) may be a vector (e.g., 1, 1) for the values ​​obtained at the first timing. Also, for example, vector (742) (V 32 ) may be a vector (e.g., 2, 2, 2) for the values ​​obtained at the second timing after the first timing. For example, the first timing may be the timing (722) (T4) of the second period (720) of FIG. 7A. The second timing may be the timing (712) (T2) of the first period (710) of FIG. 7A.

[0163] The values ​​obtained through the sensor of the first external electronic device can be vectorized with respect to a virtual point (750). For example, the point (750) can be (0, 0, 0). For example, the vector (751)(V 11 ) may be a vector (e.g., 1, 1) for the sensor values ​​of the sensor information received at the first timing. Also, for example, vector (752) (V 12 ) may be a vector (e.g., 2, 1, 2) for the values ​​obtained at the second timing. For example, the first timing may be the timing (722) (T4) of the second period (720) of FIG. 7A. The second timing may be the timing (712) (T2) of the first period (710) of FIG. 7A.

[0164] The values ​​acquired through the sensor of the second external electronic device can be vectorized with respect to a virtual point (760). For example, the point (760) can be (0, 0, 0). For example, the vector (761)(V 21) may be a vector (e.g., -1, -1) for the sensor values ​​of the sensor information received at the first timing. Also, for example, vector (762) (V 22 ) may be a vector (e.g., -2, -2, -2) for the values ​​obtained at the second timing. For example, the first timing may be the timing (722) (T4) of the second period (720) of FIG. 7A. The second timing may be the timing (712) (T2) of the first period (710) of FIG. 7A.

[0165] For convenience of explanation, in example (730), points (740), (750), and (760) are illustrated as being the same points, but the embodiments of the present disclosure are not limited thereto. For example, since the wearable device (103), the first external electronic device, and the second external electronic device are located in different areas, points (740), (750), and (760) may be different from each other.

[0166] For example, the similarity evaluation (or calculation of a similarity value) based on the difference between the vectors of each of the wearable device (103), the first external electronic device, and the second external electronic device may refer to the following mathematical formula.

[0167]

[0168] The above mathematical formula 2 is merely an example to aid understanding, and embodiments of the present disclosure may not be limited thereto. For example, the above mathematical formula 2 may be modified, applied, or expanded in various ways.

[0169] Referring to the above, the similarity value between the wearable device (103) and the first external electronic device is approximately may be, and the similarity value between the wearable device (103) and the second external electronic device According to one embodiment, the wearable device (103) may recognize that the movement of the wearable device (103) is relatively more similar (or identical) to the movement of the second external electronic device and relatively less similar to the movement of the first external electronic device.

[0170] Alternatively, for example, the similarity evaluation (or calculation of the similarity value) based on the cosine value between the vectors of each of the wearable device (103), the first external electronic device, and the second external electronic device may refer to the following mathematical formula.

[0171]

[0172] The above mathematical formula 3 is merely an example to aid understanding, and the embodiments of the present disclosure may not be limited thereto. For example, the above mathematical formula 3 may be modified, applied, or expanded in various ways.

[0173] Referring to the above, the similarity value between the wearable device (103) and the first external electronic device may be about 0.962, and the similarity value between the wearable device (103) and the second external electronic device may be 1. According to one embodiment, the wearable device (103) may recognize that the movement of the wearable device (103) is relatively more similar (or identical) to the movement of the second external electronic device, and relatively more dissimilar to the movement of the first external electronic device.

[0174] In the above example, an example is described in which a similarity value is calculated based on vectors of two timings (e.g., the first timing and the second timing), but embodiments of the present disclosure are not limited thereto. For example, the similarity value may also be calculated using vectors of three or more timings. In addition, in the above example, an example is described in which the similarity value is calculated based on an operation algorithm such as the mathematical formula described above, but embodiments of the present disclosure are not limited thereto. For example, the wearable device (103) can obtain (or calculate) the similarity value using an operation algorithm or a statistical model.

[0175] Referring back to FIG. 6, in operation (620), the wearable device (103) may determine whether the similarity value exceeds a reference value. For example, the wearable device (103) may compare the similarity value calculated based on the difference between vectors with the reference value. For example, the similarity value between the wearable device (103) and the first external electronic device may be approximately and the reference value (e.g., 0.9) can be compared. For example, the wearable device (103) can determine that the similarity value for the first external electronic device is less than or equal to the reference value. In addition, for example, the similarity value between the wearable device (103) and the second external electronic device and can be compared with the above reference value (e.g., 0.9). For example, the wearable device (103) can determine that the similarity value for the second external electronic device exceeds the reference value.

[0176] In operation (620), the wearable device (103) may perform operation (625) upon determining that the similarity value exceeds the reference value. Alternatively, in operation (620), the wearable device (103) may perform operation (635) upon determining that the similarity value is less than or equal to the reference value.

[0177] In operation (625), the wearable device (103) may determine an application ratio. The wearable device (103) may determine the application ratio based on the magnitude (or absolute value) of the difference between the similarity value and the reference value. For example, the application ratio may represent a ratio to be applied to the adjustment of the value representing the movement of the wearable device (103). For example, the application ratio may be referred to as a reference weight.

[0178] It is assumed that the first similarity value between the wearable device (103) and the first external electronic device is about 0.82, the second similarity value between the wearable device (103) and the second external electronic device is 1, the third similarity value between the wearable device (103) and the third external electronic device is 0.95, and the reference value is 0.9. In the above example, the wearable device (103) may determine the application ratio of the first similarity as 0 for the first similarity value that is less than or equal to the reference value. The wearable device (103) may determine the application ratio of the second similarity as 0.66 (=(1-0.9) / {(1-0.9)+(1-0.95)}) for the second similarity value that exceeds the reference value. The wearable device (103) may determine the application ratio of the third similarity as 0.33 (=(1-0.95) / {(1-0.9)+(1-0.95)}) for the third similarity value exceeding the reference value. In this case, the application ratio may be used as a weight value for the adjustment (e.g., 0.66 or 0.33).

[0179] Referring to operations (620) and (625), an example is described in which the wearable device (103) compares the reference value with a plurality of external electronic devices (e.g., the first external electronic device, the second external electronic device, and the third external electronic device) and determines an application ratio among the plurality of external electronic devices, but the embodiments of the present disclosure are not limited thereto.

[0180] For example, if the wearable device (103) compares the similarity value for one external electronic device with the reference value, and the similarity value is less than or equal to the reference value, operation (625) may be omitted. In other words, if the similarity value for the one external electronic device is less than or equal to the reference value, the wearable device (103) may refrain from (or skip) adjusting the value representing the movement of the wearable device (103) based on sensor information acquired from the one external electronic device.

[0181] When the wearable device (103) compares the similarity value for one external electronic device with the reference value, and the similarity value exceeds the reference value, operation (625) may be omitted. In other words, when the similarity value for the one external electronic device exceeds the reference value, the wearable device (103) may adjust the value representing the movement of the wearable device (103) in operation (630) based on sensor information acquired from the one external electronic device.

[0182] A wearable device (103) may compare similarity values ​​for two external electronic devices (e.g., the first external electronic device and the second external electronic device) with the reference value, and adjust the value representing the movement of the wearable device (103) in an operation (630) based on sensor information acquired from one external electronic device (e.g., the second external electronic device). Sensor information acquired from the remaining external electronic devices (e.g., the first external electronic device) may not be used to adjust the value.

[0183] In operation (630), the wearable device (103) can adjust the value indicating the movement of the wearable device (103). The wearable device (103) can adjust the value based on the determined application ratio. For example, the wearable device (103) can perform an offset (or compensation, cancellation) between the predicted sensor value of the sensor information and the value. In other words, the wearable device (103) can compensate for the movement of the wearable device (103) based on information indicating the movement of the external electronic device (590) (e.g., the predicted sensor value). For example, an example of a method for adjusting the value may be referred to FIG. 7C below.

[0184] Referring to FIG. 7c, examples (770, 780) of how a wearable device (103) performs adjustment of the above values ​​are illustrated. In examples (770, 780), a method of performing adjustment of the above values ​​is illustrated based on sensor information of an external electronic device (590), which is the second external electronic device of FIG. 7b.

[0185] Referring to example (770), vectors (741, 742) related to the movement (730-3) of the wearable device (103) and vectors (761, 762) related to the movement (730-2) of the second external electronic device are depicted within a spatial domain. For convenience of explanation, points (740) and (760) may be identical to each other. Points (740) and (760) may be the origin (0, 0, 0) within the spatial domain.

[0186] For example, the wearable device (103) can perform an outer product on vectors. For example, the wearable device (103) can obtain an outer product vector (773) for vectors (741, 742). The wearable device (103) can obtain an outer product vector (772) for vectors (761, 772). The outer product vector (772) and the outer product vector (773) can be referenced as reference vectors. For example, the wearable device (103) can obtain rotation parameters to align the outer product vector (772) with the outer product vector (773). For example, the rotation parameters can include a matrix to change the outer product vector (772) into the outer product vector (773).

[0187] Referring to example (780), the wearable device (103) can match the external vector (772) and the external vector (773) based on the rotation parameters. Accordingly, the plane (or vector plane) formed by the vectors (761, 762) can be matched with the plane (or vector plane) formed by the vectors (741, 742). In other words, the vectors (761, 762) can be matched with the vectors (741, 742), respectively. For example, the vector (761)(V 21 ) is vector (741)(V 31 ) can be matched. Also, for example, vector (762)(V 22 ) is vector (742)(V 32 ) can be matched.

[0188] Referring to the above, the wearable device (103) generates a vector (741) (V) based on the rotation parameters. 31 ) can be changed. The changed vector (741) (V) based on the above rotation parameters 31 )(i.e. vector (761)(V 21 ) and the vector (741)(V 31 )) can be referenced as a compensation value. The compensation value can be generated from a sensor value of sensor information obtained from the second external electronic device.

[0189] In one embodiment, the wearable device (103) can offset (or compensate, cancel) the compensation value and the value representing the movement of the wearable device (103). Accordingly, the wearable device (103) can compensate for tracking drift due to factors different from the intention of the user of the wearable device (103).

[0190] Referring to the above, an example of generating a compensation value for correcting the value indicating the movement of the wearable device (103) from a sensor value of one external electronic device (e.g., the second external electronic device) and performing the correction is described, but the embodiments of the present disclosure are not limited thereto. The wearable device (103) may generate a compensation value for correcting the value indicating the movement of the wearable device (103) from the sensor values ​​of the plurality of external electronic devices (e.g., the second external electronic device and the third external electronic device), and perform the correction. For example, the compensation value may be calculated based on the sum of the sensor value of the second external electronic device and the sensor value of the third external electronic device using the determined application ratio in operation (625).

[0191] Referring again to FIG. 6, in operation (635), the wearable device (103) may display a screen based on the value representing the movement of the wearable device (103).

[0192] According to another embodiment, the wearable device (103) may display a screen according to the above compensation of the movement of the wearable device (103). For example, the wearable device (103) may display the screen based on the value representing the movement of the wearable device (103) adjusted using the compensation value calculated through the operation (630). For example, the screen may be displayed in a state in which the tracking drift is compensated, such as the screen (410) of FIG. 4A, unlike the screen (460) of FIG. 4B.

[0193] According to another embodiment, the wearable device (103) may display a screen in which the movement of the wearable device (103) is not corrected. For example, the wearable device (103) may display the screen based on the value representing the movement of the wearable device (103) acquired by the wearable device (103) when the similarity value is less than or equal to the reference value in operation (620). For example, the screen may be displayed in a state in which the tracking drift is not corrected, such as screen (410) of FIG. 4A.

[0194] According to another embodiment, the wearable device (103) may receive sensor information from an external electronic device (590), calculate the similarity value from the sensor information, and perform correction of the value representing the movement of the wearable device (103) at designated time intervals. For example, the designated time intervals may represent each of the first period (710) and the second period (720) of FIG. 7A. In other words, the wearable device (103) may perform the above-described operations at each period.

[0195] The wearable device (103) may change the mode of the wearable device (103) when another specified condition is satisfied. For example, the other specified condition may include when the value obtained by the wearable device (103) using the sensor (530) within the operation (605) is greater than or equal to a threshold value, or when the value cannot be obtained. For example, the threshold value may be used to indicate that the movement of the wearable device (103) recognized according to the value is abnormal. The wearable device (103) may switch the mode to a passthrough mode of the wearable device (103) when the other specified condition is satisfied. For example, the wearable device (103) may switch the mode from a mode that provides a VR environment or an AR environment to the passthrough mode. The above pass-through mode may indicate a mode that displays (or transmits) the external environment of the wearable device (103). Accordingly, the user of the wearable device (103) can ensure safety.

[0196] FIG. 8 illustrates an example of a wearable device that uses sensor information of a first external electronic device connected to a second external electronic device according to an embodiment of the present disclosure.

[0197] The wearable device (103) of FIG. 8 may be an example of the electronic device (101) of FIG. 1, the wearable devices (103) of FIGS. 2A, 2B, 3A, and 3B, the wearable devices (103) of FIGS. 4A and 4B, or the wearable device (103) of FIG. 5. The first external electronic device (401) of FIG. 8 may be an example of the first external electronic device (401) of FIGS. 4A and 4B. The second external electronic device (402) of FIG. 8 may be an example of the second external electronic device (402) of FIGS. 4A and 4B. For example, the first external electronic device (401) or the second external electronic device (402) may be an example of the external electronic device (590) of FIG. 5.

[0198] Referring to FIG. 8, an example (800) of a method for a wearable device (103) to correct tracking drift using sensor information of a first external electronic device (401) connected to a second external electronic device (402) is illustrated.

[0199] Referring to example (800), the wearable device (103) can establish a connection (820) with the second external electronic device (402). The wearable device (103) may not establish a connection with the first external electronic device (401). For example, the wearable device (103) may establish a connection with the second external electronic device (402) among the first external electronic device (401) and the second external electronic device (402), and may not establish a connection with the first external electronic device (401). The wearable device (103) may not be able to directly communicate with the first external electronic device (401). In this case, the first external electronic device (401) may establish a connection (810) with the second external electronic device (402).

[0200] According to one embodiment, the wearable device (103) may obtain second sensor information obtained through a sensor of the second external electronic device (402) from the second external electronic device (402) through a connection (820). For example, the second sensor information may include sensor values ​​and time information obtained through a sensor of the second external electronic device (402).

[0201] The wearable device (103) can obtain first sensor information along with second sensor information from a second external electronic device (402) via a connection (820). For example, the first sensor information can include sensor values ​​and time information obtained through a sensor of the first external electronic device (401).

[0202] According to one embodiment, the wearable device (103) may adjust a value indicating the movement of the wearable device (103) obtained through the sensor (530) of the wearable device (103) based on the first sensor information and the second sensor information. For example, the wearable device (103) may adjust the value indicating the movement of the wearable device (103) based on the second sensor information of the second external electronic device (402) directly connected to the wearable device (103) through the connection (820) and the first sensor information of the first external electronic device (401) connected to the second external electronic device (402) through the connection (810). In addition to the directly connected electronic device, the wearable device (103) may correct the tracking drift using another electronic device connected to the directly connected electronic device. Accordingly, the wearable device (103) can more accurately track a body part (e.g., head) of a user of the wearable device (103) by correcting the tracking drift.

[0203] FIG. 9 illustrates an example of a visual object that inquires about the execution of a function for adjusting a value representing the movement of a wearable device according to an embodiment of the present disclosure. For example, the wearable device (103) of FIG. 9 may be an example of the wearable device (103) of FIG. 5 .

[0204] FIG. 9 illustrates examples (900, 905) of a method for displaying a visual object (960) inquiring about the execution of the above function when a user (910) wearing a wearable device (103) enters a vehicle (920). It can be understood that the user (910) entering the vehicle (920) satisfies the above-mentioned condition. In the examples (900, 905) of FIG. 9, a vehicle (920) is illustrated as an example of a means of transportation, but the present disclosure is not limited thereto.

[0205] Referring to example (900), the wearable device (103) can be worn on a body part of the user (910). For example, the wearable device (103) can be worn on the head of the user (910). The wearable device (103), while worn on the body part, can determine whether the user (910) is located inside a vehicle (920). The wearable device (103) can recognize that the user (910) is located outside the vehicle (920) by detecting vibration due to an external factor.

[0206] Referring to examples (900) and (905), the user (910) may move inside the vehicle (920). Accordingly, the wearable device (103) may detect that the user (910) is located inside the vehicle (920), and thus may identify that the specified condition is satisfied. The wearable device (103) may recognize that there is a high possibility that the tracking drift will occur when the user (910) is located inside the vehicle (920). The wearable device (103) may inquire of the user (910) whether to execute the function to compensate for the tracking drift.

[0207] Referring to example (905), the wearable device (103) may display a visual object (960) to inquire the user (910) about the execution of the above function. For example, the wearable device (103) may display the visual object (960) in a state where at least a portion of the screen (950) overlaps. For example, the screen (950) may include a virtual environment (e.g., an AR environment) based on a running software application. Although example (905) of FIG. 9 illustrates the visual object (960) being displayed in a state where at least a portion of the screen (950) overlaps, the present disclosure is not limited thereto. In another example, the wearable device (103) may temporarily suspend the display of the screen (950) and display the visual object (960).

[0208] For example, the visual object (960) may include text that asks the user (910) to execute the function. For example, the text may include "Do you want to execute the screen calibration function?" However, the present disclosure is not limited thereto. The screen calibration function may represent a function for compensating for the tracking drift. For example, the visual object (960) may include icons (961, 962) for obtaining input from the user (910). For example, the icon (961) may be used to execute the screen calibration function. The icon (962) may be used to skip (or refrain from, delay) the execution of the screen calibration function.

[0209] For example, the wearable device (103) may execute the function for compensating for the tracking drift upon obtaining an input from the user (910) for the icon (961). For example, the wearable device (103) may obtain sensor information from an external electronic device (e.g., the external electronic devices (401, 402) of FIGS. 4A and 4B) to be connected to the wearable device (103) upon execution of the function. The wearable device (103) may determine the similarity of movement between the wearable device (103) and the external electronic device using the obtained sensor information. For example, the wearable device (103) may identify the compensated movement value of the wearable device (103) using the sensor information of the external electronic device determined based on the result of the determination of the similarity. The above compensated movement value can be identified using a value representing the movement of the wearable device (103) obtained through a sensor of the wearable device (103) (e.g., sensor (530) of FIG. 5) and the sensor information. The wearable device (103) can compensate (or correct) the tracking drift using the compensated movement value. Accordingly, the wearable device (103) can display a screen in which the tracking drift is compensated, such as screen (410) rather than screen (460) of FIG. 4b.

[0210] FIG. 10 illustrates an example of an operational flow for a method of adjusting a value representing the movement of a wearable device based on a similarity value obtained using sensor information of external electronic devices according to an embodiment of the present disclosure.

[0211] At least some of the methods of FIG. 10 may be performed by the wearable device (103) of FIG. 5. For example, at least some of the methods may be controlled by the processor (510) of the wearable device (103). In the embodiments described below, the operations may be performed sequentially, but are not necessarily performed sequentially. The order of the operations may be changed, and at least two operations may be performed in parallel.

[0212] In operation (1010), the wearable device (103) may receive first sensor information of a first external electronic device and second sensor information of a second external electronic device. For example, the wearable device (103) may receive the first sensor information and the second sensor information through the communication circuit (540). As another example, the first external electronic device may be an example of the first external electronic device (401) of FIG. 8 or the first external electronic device (401) of FIGS. 4A and 4B. For example, the second external electronic device may be an example of the second external electronic device (402) of FIG. 8 or the second external electronic device (402) of FIGS. 4A and 4B. The first external electronic device or the second external electronic device may be an example of the external electronic device (590) of FIG. 5.

[0213] According to one embodiment, the wearable device (103) can establish a connection with each of the first external electronic device and the second external electronic device through the communication circuit (540). For example, the wearable device (103) can establish a connection with the first external electronic device using a first communication technique. For example, the wearable device (103) can establish a connection with the second external electronic device using a second communication technique. The first communication technique and the second communication technique may be the same or different from each other. The wearable device (103) can receive the first sensor information from the first external electronic device and the second sensor information from the second external electronic device.

[0214] According to another embodiment, the wearable device (103), when the external electronic device is a wearable device, may determine whether to establish the connection based on whether the external electronic device is worn. For example, assume that the first external electronic device is a wearable device. The wearable device (103), when the first external electronic device is worn by the user, may establish the connection using the first communication technique. The wearable device (103) recognizing that the first external electronic device is worn by the user and establishing the connection may be understood as substantially the same as the wearable device (103) determining to perform the adjustment based on data of the first external electronic device.

[0215] According to another embodiment, the wearable device (103) may establish a connection with the second external electronic device through the communication circuit (540). At this time, the wearable device (103) may not establish a connection with the first external electronic device. In addition, the first external electronic device may establish a connection with the second external electronic device. The wearable device (103) may also receive the first sensor information together with the second sensor information through the second external electronic device.

[0216] According to one embodiment, the first sensor information may include a first sensor value indicating movement of the first external electronic device and time information at which the first sensor value was acquired. For example, the first sensor value may include a value indicating movement of the first external electronic device. The first sensor value may include data acquired through an acceleration sensor or a gyro sensor of the first external electronic device. For example, the time information may include a timing (e.g., a timestamp) at which the first sensor value was acquired. In this case, the wearable device (103) and the first external electronic device may be synchronized with each other.

[0217] The second sensor information may include a second sensor value indicating movement of the second external electronic device and time information at which the second sensor value was acquired. For example, the second sensor value may include a value indicating movement of the second external electronic device. For example, the second sensor value may include data acquired through an acceleration sensor or a gyro sensor of the second external electronic device. For example, the time information may include a timing (e.g., a timestamp) at which the second sensor value was acquired. The wearable device (103) and the second external electronic device may be synchronized with each other.

[0218] According to another embodiment, the wearable device (103) may transmit a signal requesting the sensor information to the first external electronic device or the second external electronic device when a specified condition is satisfied.

[0219] The above-mentioned specified condition may include a case where an error occurs for a specified length of time or longer between vision information acquired through an image sensor (or camera) of the wearable device (103) and a value representing the movement of the wearable device (103) acquired using the sensor (530). The error may be referred to as tracking drift. Alternatively, for example, the above-mentioned specified condition may include a case where the movement of the wearable device (103) recognized based on the value of the sensor (530) is identified as an abnormal movement. The abnormal movement may include a case where the movement of the wearable device (103) recognized based on the value is not continuous. The above-mentioned specified condition may include a case where the acceleration of the movement recognized based on the value of the sensor (530) changes linearly. Alternatively, for example, the specified condition may include a case where the wearable device (103) (or a user wearing the wearable device (103)) is located within a vehicle. For example, being located within the vehicle may be recognized based on the wearable device (103) detecting vibration due to an external factor.

[0220] According to one embodiment, when the sensor information is requested, the wearable device (103) may indicate an update cycle of sensor information (e.g., the first sensor information or the second sensor information) to be transmitted by the first external electronic device or the second external electronic device. For example, the update cycle may include a specified time interval (e.g., minutes, seconds) or a specified number of times. The wearable device (103) may indicate an end time (timing) together with the update cycle. For example, the update cycle may be indicated for each sensor value (e.g., the first sensor value or the second sensor value). For example, when the amount of change in acceleration of the wearable device (103) is large, the wearable device (103) may indicate an update cycle with a relatively short cycle with respect to the acceleration sensor value to the first external electronic device or the second external electronic device. When the amount of change in the rotation information (or the value of the gyro sensor) of the wearable device (103) is small, the wearable device (103) can instruct the first external electronic device or the second external electronic device to update the gyro sensor value with respect to a relatively long period.

[0221] According to one embodiment, the first external electronic device may transmit the first sensor information to the wearable device (103) based on a comparison between the acquired first sensor information and previous sensor information. For example, the first external electronic device may compare a difference between the first sensor information acquired using a sensor (e.g., sensor 593 of FIG. 5) in a first period and third sensor information acquired using the sensor in a second period prior to the first period with a reference difference. At this time, the third sensor information of the second period may be stored in the first external electronic device. At this time, the difference is a difference between a vector (e.g., (x1, y1, z1)) representing a sensor value of the first sensor information and another vector (e.g., (x2, y2, z2)) representing a sensor value of the third sensor information of the second period. may include. Or, for example, the difference may include a difference between a value of at least some axis (e.g., x-axis, y-axis, or z-axis) of the vector and a value of at least some axis of the other vector. Or, the first external electronic device may transmit the first sensor information to the wearable device (103) when a value of at least some axis (e.g., x-axis, y-axis, or z-axis) of the vector is outside a specified range.

[0222] According to one embodiment, the first sensor value included in the first sensor information may be in a text format or a binary format. For example, the text format may include Plain, JSON, and XML. For example, the binary format may include Protobuf. When the connection status between the wearable device (103) and the first external electronic device is poor (e.g., data delay is detected), the data including the first sensor value may be compressed and transmitted. In the above example, a compression transmission method of the data used when the connection status between the wearable device (103) and the first external electronic device is poor is described, but the embodiments of the present disclosure are not limited thereto. For example, the data transmitted and received between the wearable device (103) and the first external electronic device may be compressed and transmitted regardless of the connection status.

[0223] By transmitting and receiving the sensor information based on the comparison between the difference and the reference difference and / or the update cycle indicated by the wearable device (103), the power consumption of the wearable device (103) and the first external electronic device can be reduced. In the above example, an example is described for the first sensor information and the first external electronic device, but the example can be substantially equally applied to the second sensor information and the second external electronic device.

[0224] The wearable device (103) can obtain a value indicating the movement of the wearable device (103) using the sensor (530). For example, the wearable device (103) can measure the value indicating the movement of the wearable device (103) using the sensor (530) at a specified timing. The specified timing may be included in the first period and may be a timing after the timing at which the first sensor information (or the second sensor information) is received. For example, the sensor (530) may include an IMU sensor.

[0225] According to another embodiment, the wearable device (103) may predict a first value indicating movement of the first external electronic device based on the first sensor information. For example, the wearable device (103) may calculate (or generate) a sensor value predicted at the specified timing using the first sensor information acquired in the first period and the third sensor information acquired in the second period. For example, the first value may be a value indicating the movement of the first external electronic device. The prediction may be to compensate for an error between the timing at which the first external electronic device acquires the first sensor information and the timing at which the wearable device (103) acquires the value. For example, the error may include a time delay while the first sensor information is transmitted from the first external electronic device to the wearable device (103), or a time delay while the wearable device (103) acquires the value.

[0226] The wearable device (103) can predict a second value indicating the movement of the second external electronic device based on the second sensor information. For example, the wearable device (103) can calculate (or generate) a sensor value predicted at the specified timing using the second sensor information acquired in the first cycle and the fourth sensor information acquired in the second cycle. As another example, the second value may be a value indicating the movement of the second external electronic device.

[0227] According to one embodiment, the first value (or the second value) may be obtained (or calculated) using a computational algorithm or statistical model (e.g., an artificial intelligence (AI) model, machine learning, or deep learning). For example, reference may be made to FIGS. 6 and 7A described above for examples of a method for generating the predicted sensor value.

[0228] In operation (1020), the wearable device (103) may obtain a first similarity value between the movement of the wearable device (103) and the movement of the first external electronic device. The wearable device (103) may calculate the first similarity value between the movement of the wearable device (103) and the movement of the first external electronic device. For example, the wearable device (103) may calculate the first similarity value, which represents the similarity between the value obtained at the specified timing and the first value.

[0229] In operation (1030), the wearable device (103) may obtain a second similarity value between the movement of the wearable device (103) and the movement of the second external electronic device. The wearable device (103) may calculate the second similarity value between the movement of the wearable device (103) and the movement of the second external electronic device. For example, the wearable device (103) may calculate the second similarity value, which represents the similarity between the value obtained at the specified timing and the second value.

[0230] For example, for specific details on how to generate the similarity value (e.g., the first similarity value or the second similarity value), reference may be made to the above-described FIGS. 6 and 7b.

[0231] In operation (1040), the wearable device (103) can identify a compensated movement value of the wearable device (103) based on a result of a comparison between the first similarity value and the second similarity value and a reference value, respectively. The wearable device (103) can determine whether the similarity value exceeds the reference value. For example, the wearable device (103) can compare the similarity value calculated based on the difference between vectors with the reference value. For example, the first similarity value between the wearable device (103) and the first external electronic device can be compared with the reference value.

[0232] For example, the wearable device (103) may determine that the first similarity value for the first external electronic device is greater than the reference value. Further, for example, the wearable device (103) may compare the second similarity value between the wearable device (103) and the second external electronic device with the reference value. For example, the wearable device (103) may determine that the second similarity value for the second external electronic device is less than or equal to the reference value. When the first similarity value exceeds the reference value and the second similarity value is less than or equal to the reference value, the wearable device (103) may identify the compensated movement value of the wearable device (103) based on the first sensor information and the value representing the movement of the wearable device (103). The wearable device (103) may determine that the first similarity value for the first external electronic device is less than or equal to the reference value. For example, the wearable device (103) may determine that the second similarity value for the second external electronic device exceeds the reference value. For example, the wearable device (103) may identify the compensated movement value of the wearable device (103) based on the second sensor information and the value representing the movement of the wearable device (103) when the first similarity value is less than or equal to the reference value and the second similarity value exceeds the reference value.

[0233] For example, the compensated motion value may be adjusted from the value representing the movement of the wearable device (103) or may represent a value generated by the value. For example, the compensated motion value may be used to compensate for tracking drift. As another example, the compensated motion value may be the compensation value (e.g., the changed vector (741) (V) of FIG. 7b 31) and the offset (or compensation, cancellation) of the above values.

[0234] In one embodiment, the wearable device (103) may determine an application ratio. For example, the wearable device (103) may determine the application ratio based on the magnitude (or absolute value) of the difference between the similarity value and the reference value. The application ratio may represent a ratio to be applied to the adjustment of the value representing the movement of the wearable device (103). For example, the application ratio may be referred to as a reference weight.

[0235] It is assumed that the first similarity value between the wearable device (103) and the first external electronic device is 1, the second similarity value between the wearable device (103) and the second external electronic device is 0.8, and the reference value is 0.9. In the above example, the wearable device (103) may determine the application ratio of the first similarity as 1 for the first similarity value exceeding the reference value. The wearable device (103) may determine the application ratio of the second similarity as 0 for the second similarity value less than or equal to the reference value. The embodiments of the present disclosure are not to be construed as limited to the above example.

[0236] For example, if the wearable device (103) compares a similarity value for one external electronic device with the reference value, and the similarity value for the one external electronic device is less than or equal to the reference value, the wearable device (103) may refrain from (or skip) adjusting the value indicating the movement of the wearable device (103) based on sensor information obtained from the one external electronic device.

[0237] When the wearable device (103) compares a similarity value for one external electronic device with the reference value, and the similarity value for the one external electronic device exceeds the reference value, the wearable device (103) can adjust the value indicating the movement of the wearable device (103) based on sensor information obtained from the one external electronic device.

[0238] It is assumed that the first similarity value between the wearable device (103) and the first external electronic device is 1, the second similarity value between the wearable device (103) and the second external electronic device is about 0.82, the third similarity value between the wearable device (103) and the third external electronic device is 0.95, and the reference value is 0.9. The wearable device (103) may determine the application ratio of the first similarity as 0.66 (=(1-0.9) / {(1-0.9)+(1-0.95)}) for the first similarity value exceeding the reference value. The wearable device (103) may determine the application ratio of the second similarity as 0 for the second similarity value less than or equal to the reference value. The wearable device (103) may determine the application ratio of the third similarity as 0.33 (=(1-0.95) / {(1-0.9)+(1-0.95)}) for the third similarity value exceeding the reference value. The application ratio may be used as a weight value for the adjustment (e.g., 0.66 or 0.33).

[0239] According to one embodiment, the wearable device (103) can adjust the value indicating the movement of the wearable device (103). For example, the wearable device (103) can adjust the value based on the determined application ratio. For example, the wearable device (103) can perform an offset (or compensation, cancellation) between the first value of the first sensor information and the value. In other words, the wearable device (103) can compensate for the movement of the wearable device (103) based on information indicating the movement of the first external electronic device (e.g., the first value). For specific details on a method for adjusting the value, reference may be made to the above-described FIGS. 6 and 7c.

[0240] According to another embodiment, the wearable device (103) may offset (or compensate, cancel) the compensation value changed (or generated) from the first value and the value representing the movement of the wearable device (103). Accordingly, the wearable device (103) may correct for tracking drift due to factors different from the intention of the user of the wearable device (103).

[0241] Referring to the above, an example is described of generating a compensation value for correcting the value indicating the movement of the wearable device (103) from the first sensor value of one external electronic device (e.g., the first external electronic device), and performing the correction by generating the compensated movement value using the compensation value and the value, but the embodiments of the present disclosure are not limited thereto. For example, the wearable device (103) may generate a compensation value for correcting the value indicating the movement of the wearable device (103) from the sensor values ​​of the plurality of external electronic devices (e.g., the first external electronic device and the third external electronic device), and perform the correction. The compensation value may be calculated based on a sum of the first sensor value of the first external electronic device and the third sensor value of the third external electronic device using the application ratio.

[0242] In operation (1050), the wearable device (103) can display a screen according to the compensated movement value. For example, the wearable device (103) can display the screen according to the compensated movement value through the display (520).

[0243] According to one embodiment, the wearable device (103) may display a screen according to the compensation of the movement of the wearable device (103). For example, the wearable device (103) may display the screen according to the compensated movement value generated (or adjusted) based on the compensation value and the value representing the movement of the wearable device (103). Unlike the screen (460) of FIG. 4B, the screen may be displayed in a state in which the tracking drift is compensated, such as the screen (410) of FIG. 4A.

[0244] According to another embodiment, the wearable device (103) may receive the first sensor information from the first external electronic device, calculate the first similarity value from the first sensor information, and perform correction of the value indicating the movement of the wearable device (103) at designated time intervals. For example, the designated time intervals may represent each of the first period (710) and the second period (720) of FIG. 7A. In other words, the wearable device (103) may perform the above-described operations at each period.

[0245] According to another embodiment, the wearable device (103) may change the mode of the wearable device (103) when another specified condition is satisfied. For example, the other specified condition may include when the value obtained by the wearable device (103) using the sensor (530) is greater than or equal to a threshold value, or when the value cannot be obtained. For example, the threshold value may be used to indicate that the movement of the wearable device (103) recognized according to the value is abnormal. For example, the wearable device (103) may switch the mode to a passthrough mode of the wearable device (103) when the other specified condition is satisfied. The wearable device (103) may switch the mode from a mode that provides a VR environment or an AR environment to the passthrough mode. The passthrough mode may indicate a mode that displays (or transmits) the external environment of the wearable device (103). Accordingly, the user of the wearable device (103) can ensure safety.

[0246] Hereinafter, the device and method according to embodiments of the present disclosure can correct (or improve) the tracking drift by correcting the value acquired by the wearable device (103) using sensor information of an external electronic device connected to the wearable device (103). At this time, the device and method according to embodiments of the present disclosure can reduce the power consumption by obtaining the sensor information that satisfies a specified condition from the external electronic device. When the external electronic device is a plurality of external electronic devices, the device and method according to embodiments of the present disclosure can determine an application ratio for the correction according to the similarity between the movement of each of the plurality of external electronic devices and the movement of the wearable device (103), and perform the correction using the determined application ratio. Accordingly, the device and method according to embodiments of the present disclosure can more accurately perform correction for the tracking drift and perform tracking (e.g., HeT) of a body part of a user.

[0247] As described above, the wearable device (103) may include a memory (550) that stores instructions and includes one or more storage media. The wearable device (103) may include a sensor (530). The wearable device (103) may include a communication circuit (540). The wearable device (103) may include at least one processor (510) that includes a processing circuit. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to receive first sensor information of a first external electronic device and second sensor information of a second external electronic device through the communication circuit (540). The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to obtain a first similarity value between the movement of the wearable device (103) and the movement of the first external electronic device based on a value representing the movement of the wearable device (103) obtained through the sensor (530) and the first sensor information. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to obtain a second similarity value between the movement of the wearable device (103) and the movement of the second external electronic device based on the value and the second sensor information.The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to identify a compensated motion value of the wearable device based on the first sensor information and the value representing the motion of the wearable device (103) if the first similarity value exceeds a reference value and the second similarity value is less than or equal to the reference value. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to identify a compensated motion value of the wearable device based on the second sensor information and the value representing the motion of the wearable device (103) if the first similarity value is less than or equal to the reference value and the second similarity value exceeds the reference value. The above instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to display a screen according to the compensated motion value.

[0248] According to one embodiment, the first sensor information may include a first sensor value indicating the movement of the first external electronic device and information on the time at which the first sensor value was acquired. The second sensor information may include a second sensor value indicating the movement of the second external electronic device and information on the time at which the second sensor value was acquired.

[0249] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to establish a connection with the first external electronic device via the communication circuit (540). The second external electronic device may be connected to the first external electronic device. The second sensor information of the second external electronic device may be received together with the first sensor information via the first external electronic device.

[0250] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to transmit a signal requesting the first sensor information and the second sensor information when a specified condition is satisfied. The specified condition may include at least one of: when an error between vision data acquired through a camera of the wearable device (103) and the value exceeds a reference error; when an acceleration of the wearable device (103) based on the value changes linearly; or when a user wearing the wearable device (103) is located in a vehicle.

[0251] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to obtain the value representing the movement of the wearable device (103) through the sensor (530) at a timing within the first cycle after receiving the first sensor information and the second sensor information within the first cycle. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to obtain the first value representing the movement of the first external electronic device predicted at the timing based on the first sensor information and the third sensor information of the first external electronic device obtained within a second cycle prior to the first cycle. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to obtain a second value representing the movement of the second external electronic device predicted at the timing based on the fourth sensor information and the second sensor information of the second external electronic device obtained within the second period.

[0252] According to one embodiment, the first value may be obtained based on at least one of a difference between the first sensor information and the third sensor information or a statistical model for predicting the movement of the first external electronic device. The second value may be obtained based on at least one of a difference between the second sensor information and the fourth sensor information or a statistical model for predicting the movement of the second external electronic device.

[0253] In one embodiment, the difference between the first sensor information and the third sensor information may exceed a reference difference. The difference between the second sensor information and the fourth sensor information may exceed the reference difference.

[0254] In one embodiment, the instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to obtain another value obtained through the sensor (530) at another timing within the second cycle and a first vector between the values. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to obtain a second vector between the first value and a third value representing movement of the first external electronic device predicted for the other timing based on the third sensor information. The above instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to obtain a third vector between the second value and a fourth value representing movement of the second external electronic device predicted for the other timing based on the fourth sensor information.

[0255] In one embodiment, the instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to obtain the first similarity value based on the first vector and the second vector. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to obtain the second similarity value based on the first vector and the third vector.

[0256] In one embodiment, the instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to, in response to the first similarity value exceeding the reference value, obtain a first reference vector based on the other value and the value. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to, in response to the first similarity value exceeding the reference value, obtain a second reference vector based on the third value and the first value. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to obtain rotation parameters for changing the second reference vector to align with the first reference vector. The above instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to obtain a compensation value from the first sensor information based on the rotation parameters.

[0257] In one embodiment, the first reference vector may be obtained using a cross product of the other value and the first value. The second reference vector may be obtained using a cross product of the third value and the first value.

[0258] According to one embodiment, the compensated motion value can be identified based on a cancellation between the compensation value and the value.

[0259] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to receive third sensor information of a third external electronic device via the communication circuit (540). The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to obtain a third similarity value between the movement of the wearable device (103) and the movement of the third external electronic device based on the value and the third sensor information. The third similarity value may exceed the first similarity value. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to generate a first weight according to the first similarity value and the reference value and a second weight according to the third similarity value and the reference value. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to identify the first sensor information to which the first weight is applied, the third sensor information to which the second weight is applied, and a compensated other motion value based on the values. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to display a screen according to the compensated other motion value. The second weight may exceed the first weight.

[0260] According to one embodiment, the value representing the movement of the wearable device (103) may include acceleration information and rotation information obtained through the sensor (530).

[0261] As described above, the wearable device (103) may include a memory (550) that stores instructions and includes one or more storage media. The wearable device (103) may include a sensor (530). The wearable device (103) may include a communication circuit (540). The wearable device (103) may include at least one processor (510) that includes a processing circuit. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to establish a connection with an external electronic device through the communication circuit (540). The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to receive sensor information from the external electronic device, the sensor information including a sensor value and a first timing at which the sensor value was acquired. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to acquire, through the sensor (530), a value representing a movement of the wearable device (103) at a second timing after the first timing. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to calculate a first vector for the movement of the wearable device (103) based on the value and another value acquired through the sensor (530) at a third timing prior to the first timing.The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to calculate a second vector for the movement of the external electronic device based on another sensor value and the sensor value calculated for the third timing. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to obtain a similarity value between the movement of the external electronic device and the movement of the wearable device (103) based on the first vector and the second vector. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to adjust the value representing the movement of the wearable device (103) using the sensor value based on the similarity value exceeding a reference value.

[0262] In one embodiment, the instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to determine whether the similarity value exceeds the reference value. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to maintain the similarity value if the similarity value is less than or equal to the reference value. The value may be used for displaying contents provided by the wearable device (103).

[0263] As described above, the method performed by the wearable device (103) may include an operation of receiving first sensor information of a first external electronic device and second sensor information of a second external electronic device. The method may include an operation of obtaining a first similarity value between the movement of the wearable device (103) and the movement of the first external electronic device based on a value representing the movement of the wearable device (103) and the first sensor information. The method may include an operation of obtaining a second similarity value between the movement of the wearable device (103) and the movement of the second external electronic device based on the value and the second sensor information. The method may include an operation of adjusting the value representing the movement of the wearable device (103) using the first sensor information based on the first similarity value exceeding a reference value and the second similarity value being lower than or equal to the reference value. The method may include an operation of identifying a compensated motion value of the wearable device based on the first sensor information and the value representing the motion of the wearable device (103) when the first similarity value exceeds a reference value and the second similarity value is less than or equal to the reference value. The method may include an operation of identifying a compensated motion value of the wearable device based on the second sensor information and the value representing the motion of the wearable device (103) when the first similarity value is less than or equal to the reference value and the second similarity value exceeds the reference value. The method may include an operation of displaying a screen according to the compensated motion value.

[0264] According to one embodiment, the method may include an operation of obtaining the value indicating the movement of the wearable device (103) at a timing within the first period after receiving the first sensor information and the second sensor information within the first period. The method may include an operation of obtaining a first value indicating the movement of the first external electronic device predicted at the timing based on the third sensor information and the first sensor information of the first external electronic device obtained within a second period prior to the first period. The method may include an operation of obtaining a second value indicating the movement of the second external electronic device predicted at the timing based on the fourth sensor information and the second sensor information of the second external electronic device obtained within the second period.

[0265] According to one embodiment, the method may include an operation of obtaining another value obtained at another timing within the second period and a first vector between the values. The method may include an operation of obtaining a second vector between a third value representing movement of the first external electronic device predicted for the other timing based on the third sensor information and the first value. The method may include an operation of obtaining a third vector between a fourth value representing movement of the second external electronic device predicted for the other timing and the second value based on the fourth sensor information.

[0266] As described above, the non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by at least one processor (510) of a wearable device (103) including a sensor (530) and a communication circuit (540), cause the device to receive first sensor information of a first external electronic device and second sensor information of a second external electronic device through the communication circuit (540). The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor (510), cause the device to obtain a first similarity value between the movement of the wearable device (103) and the movement of the first external electronic device based on a value representing a movement of the wearable device (103) obtained through the sensor (530) and the first sensor information. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor (510), cause the at least one processor (510) to obtain a second similarity value between the movement of the wearable device (103) and the movement of the second external electronic device based on the value and the second sensor information. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor (510), cause the at least one processor (510) to identify a compensated movement value of the wearable device based on the first sensor information and the value representing the movement of the wearable device (103) when the first similarity value exceeds a reference value and the second similarity value is less than or equal to the reference value.The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor (510), cause the wearable device (103) to identify a compensated motion value based on the second sensor information and the value representing the motion of the wearable device (103), when the first similarity value is less than or equal to the reference value and the second similarity value exceeds the reference value. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor (510), cause the screen to be displayed according to the compensated motion value.

[0267] Electronic devices according to the various embodiments disclosed in this document may take various forms. The electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Depending on the embodiment, the electronic devices are not limited to the aforementioned devices.

[0268] The various embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In this document, phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can each include any one of the items listed together in the corresponding phrase among the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish the corresponding component from other corresponding components and do not limit the corresponding components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as being "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0269] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0270] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). A processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0271] In one embodiment, the method according to various embodiments disclosed herein may be provided as a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0272] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to other embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0273] It will be appreciated that the various embodiments of the present disclosure, as described in the claims and specification, may be implemented in the form of hardware, software, or a combination of hardware and software.

[0274] Any software may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores one or more computer programs (software modules). The one or more computer programs include computer-executable instructions (or instructions) that, when individually or collectively executed by one or more processors of an electronic device, cause the electronic device to perform the method of the present disclosure.

[0275] Any software may be stored in the form of a volatile or non-volatile storage medium, such as, for example, a storage device such as a read only memory (ROM), whether erasable or rewritable, or in the form of a memory such as, for example, a random access memory (RAM), memory chips, devices or integrated circuits, or in an optically or magnetically readable medium such as, for example, a compact disk (CD), a digital versatile disc (DVD), a magnetic disk or a magnetic tape or the like. It will be understood that various embodiments of storage devices and storage media, a computer program or non-transitory computer-readable storage suitable for storing computer programs comprising instructions that, when executed, implement various embodiments of the present disclosure, are provided. Accordingly, various embodiments provide programs comprising code for implementing an apparatus or method as recited in any of the claims of the present disclosure, and non-transitory machine-readable storage storing such programs.

[0276] While the present disclosure has been illustrated and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. In wearable devices, A memory that stores instructions and includes one or more storage media; sensor; communication circuit; and At least one processor comprising a processing circuit; The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Receiving first sensor information of a first external electronic device and second sensor information of a second external electronic device through the communication circuit; Based on a value representing the movement of the wearable device acquired through the sensor and the first sensor information, a first similarity value between the movement of the wearable device and the movement of the first external electronic device is acquired; Based on the above value and the second sensor information, a second similarity value is obtained between the movement of the wearable device and the movement of the second external electronic device; If the first similarity value exceeds the reference value and the second similarity value is less than or equal to the reference value, a compensated movement value of the wearable device is identified based on the first sensor information and the value representing the movement of the wearable device; If the first similarity value is less than or equal to the reference value and the second similarity value exceeds the reference value, identifying a compensated movement value of the wearable device based on the second sensor information and the value representing the movement of the wearable device; and Causing the screen to be displayed according to the above compensated movement value, Wearable devices.

2. In claim 1, The first sensor information includes a first sensor value indicating the movement of the first external electronic device and time information at which the first sensor value was acquired, and The second sensor information includes a second sensor value indicating the movement of the second external electronic device and time information at which the second sensor value was acquired. Wearable devices.

3. In claim 1, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Causing to establish a connection with the first external electronic device through the above communication circuit, The second external electronic device is connected to the first external electronic device, and The second sensor information of the second external electronic device is received together with the first sensor information through the first external electronic device. Wearable devices.

4. In claim 1, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: If a specified condition is satisfied, cause a signal to be transmitted requesting the first sensor information and the second sensor information, The above specified conditions are: If the error between the vision data acquired through the camera of the wearable device and the above value exceeds the standard error, If the acceleration of the wearable device based on the above value changes linearly, or Including at least one case where a user wearing the wearable device is located within a vehicle; Wearable devices.

5. In claim 1, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: After receiving the first sensor information and the second sensor information within the first period, the value representing the movement of the wearable device is obtained through the sensor at a timing within the first period; Based on the third sensor information of the first external electronic device acquired within the second period prior to the first period and the first sensor information, a first value representing the movement of the first external electronic device predicted at the timing is acquired; and Causing to obtain a second value representing the movement of the second external electronic device predicted at the timing based on the fourth sensor information of the second external electronic device obtained within the second period and the second sensor information. Wearable devices.

6. In claim 5, The first value is obtained based on at least one of a difference between the first sensor information and the third sensor information or a statistical model for predicting the movement of the first external electronic device, and The second value is obtained based on at least one of a difference between the second sensor information and the fourth sensor information or a statistical model for predicting the movement of the second external electronic device. Wearable devices.

7. In claim 5, The difference between the first sensor information and the third sensor information exceeds the reference difference, and The difference between the second sensor information and the fourth sensor information exceeds the reference difference. Wearable devices.

8. In claim 5, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Obtaining another value obtained through the sensor at another timing within the second period and a first vector between the values; Obtaining a second vector between the third value and the first value, which represents the movement of the first external electronic device predicted for the different timing based on the third sensor information; and causing a third vector to be obtained between the fourth value and the second value, which represents the movement of the second external electronic device predicted for the different timing based on the fourth sensor information; Wearable devices.

9. In claim 8, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Based on the first vector and the second vector, the first similarity value is obtained; and Based on the first vector and the third vector, causing the second similarity value to be obtained, Wearable devices.

10. In claim 8, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: In response to the first similarity value exceeding the above reference value, a first reference vector is obtained based on the other value and the above value; In response to the first similarity value exceeding the reference value, a second reference vector is obtained based on the third value and the first value; Obtaining rotation parameters for changing the second reference vector to align with the first reference vector; and Based on the above rotation parameters, causing a compensation value to be obtained from the first sensor information, Wearable devices.

11. In claim 10, The above first reference vector is obtained by using the cross product of the other value and the above value, and The second reference vector is obtained by using the outer product of the third value and the first value. Wearable devices.

12. In claim 10, The above compensated motion value is identified based on the cancellation between the compensation value and the above value. Wearable devices.

13. In claim 1, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Receive third sensor information of a third external electronic device through the communication circuit; Based on the above value and the third sensor information, a third similarity value is obtained between the movement of the wearable device and the movement of the third external electronic device, wherein the third similarity value exceeds the first similarity value; Generating a first weight according to the first similarity value and the reference value and a second weight according to the third similarity value and the reference value; and Identifying another compensated motion value based on the first sensor information to which the first weight is applied, the third sensor information to which the second weight is applied, and the value, and Causes the screen to be displayed according to the above compensated other movement values, The second weight is greater than the first weight. Wearable devices.

14. A method performed by a wearable device, The above method: An operation of receiving first sensor information of a first external electronic device and second sensor information of a second external electronic device; An operation of obtaining a first similarity value between the movement of the wearable device and the movement of the first external electronic device based on a value representing the movement of the wearable device and the first sensor information; An operation of obtaining a second similarity value between the movement of the wearable device and the movement of the second external electronic device based on the above value and the second sensor information; An operation of identifying a compensated movement value of the wearable device based on the first sensor information and the value representing the movement of the wearable device, when the first similarity value exceeds a reference value and the second similarity value is less than or equal to the reference value; An operation of identifying a compensated movement value of the wearable device based on the second sensor information and the value representing the movement of the wearable device, when the first similarity value is less than or equal to the reference value and the second similarity value exceeds the reference value; and Including an action of displaying a screen according to the above compensated movement value, method.

15. One or more non-transitory computer-readable storage media storing one or more computer programs including computer-executable instructions that, when individually or collectively executed by one or more processors of an electronic device, cause a wearable device including a sensor and a communication circuit to perform operations, said operations comprising: An operation of receiving first sensor information of a first external electronic device and second sensor information of a second external electronic device through the communication circuit; An operation of obtaining a first similarity value between the movement of the wearable device and the movement of the first external electronic device based on a value representing the movement of the wearable device obtained through the sensor and the first sensor information; An operation of obtaining a second similarity value between the movement of the wearable device and the movement of the second external electronic device based on the above value and the second sensor information; An operation of identifying a compensated movement value of the wearable device based on the first sensor information and the value representing the movement of the wearable device, when the first similarity value exceeds a reference value and the second similarity value is less than or equal to the reference value; An operation of identifying a compensated movement value of the wearable device based on the second sensor information and the value representing the movement of the wearable device, when the first similarity value is less than or equal to the reference value and the second similarity value exceeds the reference value; and Including an action of displaying a screen according to the above compensated movement value, A non-transitory computer-readable storage medium.

Images